Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suggestions for the Llama 2 container deployment example #449

Open
wagnersemantix opened this issue Apr 29, 2024 · 0 comments
Open

Suggestions for the Llama 2 container deployment example #449

wagnersemantix opened this issue Apr 29, 2024 · 0 comments

Comments

@wagnersemantix
Copy link

Hey everyone, thank you for the awesome work at creating these examples.

I was following the llama 2 deployment tutorial and I would like to suggest a few improvements for an easier flow:

  1. Indicate where should we run this example.
  • At the download section it is indicated to perform it at a Notebook session, however the rclone tool is not available at that enviroment. I managed to download it and set it up using conda install rclone and using this documentation.
  • During the image build/push section we are required to use Docker. However Docker is also not available at the notebook environment. I was not able to install it at the notebook environment and ended up doing it locally.
    As a newcomer, changing between environments gave me the impression I was performing those operations wrong. A hint here would be greatly appreciated.
  1. Indicate where to add policies. Another newcomer issue, but it is unclear where should we add the policies. The given documentation also exemplifies types of policies, but not where should we add them. I think that a direction or link would help here.

  2. During the build section the initial part is common to the TGI and vLLM builds. People going for the vLLM build might miss this necessary introductory part. Perhaps different section name?

Hopefully thats helps others when starting to use OCI through this example. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant