New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use continuous batching by default #882

Open

Wovchena wants to merge 89 commits into openvinotoolkit:master from Wovchena:use-continuos-batching-by-default

+27 −4

Collaborator

Wovchena commented Sep 19, 2024

No description provided.


          Use continuous batching by default

ec5f305

Wovchena requested a review from andrei-kochin

September 19, 2024 10:33

andrei-kochin added do_not_merge do_not_review labels


          Merge branch 'master' into use-continuos-batching-by-default

dd7a5cf

andrei-kochin marked this pull request as ready for review

September 19, 2024 14:52


          Merge branch 'master' into use-continuos-batching-by-default

229b7c5

andrei-kochin reviewed

View reviewed changes

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

andrei-kochin added 2 commits

September 20, 2024 16:41


          Update src/cpp/src/llm_pipeline.cpp

41d1fe7


          Update src/cpp/src/llm_pipeline.cpp

36150c4

andrei-kochin reviewed

View reviewed changes

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

andrei-kochin and others added 3 commits

September 20, 2024 17:32


          Update src/cpp/src/llm_pipeline.cpp

4a4a09e


          Update src/cpp/src/llm_pipeline.cpp

1a58b5e


          Reorder cout

90d81e6

andrei-kochin reviewed

View reviewed changes

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

andrei-kochin added 2 commits

September 20, 2024 18:48


          Update src/cpp/src/llm_pipeline.cpp

6dc43a3


          Update src/cpp/src/llm_pipeline.cpp

03e2f32

ilya-lavrenov added this to the 2024.5 milestone

andrei-kochin reviewed

View reviewed changes

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

andrei-kochin added 2 commits

September 20, 2024 19:03


          Update src/cpp/src/llm_pipeline.cpp

e561e93


          Update src/cpp/src/llm_pipeline.cpp

37ea2ad

andrei-kochin approved these changes

View reviewed changes

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

andrei-kochin added 2 commits

September 23, 2024 11:45


          Update src/cpp/src/llm_pipeline.cpp

b62aee9


          Update src/cpp/src/llm_pipeline.cpp

07505b3

andrei-kochin reviewed

View reviewed changes

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

andrei-kochin added 2 commits

September 23, 2024 11:57


          Update src/cpp/src/llm_pipeline.cpp

e078818


          Update src/cpp/src/llm_pipeline.cpp

001d3a0

andrei-kochin requested changes

View reviewed changes

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

andrei-kochin added 2 commits

September 25, 2024 18:32


          Update src/cpp/src/llm_pipeline.cpp

a0a964f


          Update src/cpp/src/llm_pipeline.cpp

3cb2105

ilya-lavrenov reviewed

View reviewed changes

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

ilya-lavrenov added 3 commits

November 12, 2024 16:45


          Apply suggestions from code review

d11db7e


          Update llm_pipeline.cpp

9acf368


          Update causal_lm_cpp.yml

3c835af

ilya-lavrenov added this to the 2025.0 milestone

github-actions bot removed the category: GHA label


          Merge branch 'master' into use-continuos-batching-by-default

046c017

ilya-lavrenov added a commit to ilya-lavrenov/openvino.genai that referenced this pull request


          Fixed endless generations in test_callback_one_string test (openvinot…

9509c9f

…oolkit#1189)

See
https://productionresultssa7.blob.core.windows.net/actions-results/1d9c7ed7-f60f-463a-8dba-2898f6f39e0a/workflow-job-run-2ad6080a-b02b-5d2d-feb5-675d9367578e/logs/job/job-logs.txt?rsct=text%2Fplain&se=2024-11-11T12%3A35%3A03Z&sig=cE1uxQm2H0TswczNt4jESUwM1%2BzN2uQBHxsm7Tqw0AI%3D&ske=2024-11-11T22%3A38%3A18Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2024-11-11T10%3A38%3A18Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2024-08-04&sp=r&spr=https&sr=b&st=2024-11-11T12%3A24%3A58Z&sv=2024-08-04

A lot of words are printed as max_new_tokens is quite huge number, while
exception is expected as streamer cannot work with batch.

To fix hanging of tests in this PR
openvinotoolkit#882


          Merge branch 'master' into use-continuos-batching-by-default

badaa5b

github-actions bot removed the category: sampling label


          Fix validation

c807011

Contributor

ilya-lavrenov commented Nov 25, 2024

for some reason only a single test is run:

======================== 1 failed, 3 warnings in 45.75s ========================

@Wovchena any ideas?

Collaborator Author

Wovchena commented Nov 25, 2024

It was meant to run a single test only: the command is python -m pytest -v ./tests/python_tests/test_chat_generate_api.py::test_set_chat_template. The full testing is in the next step.


          Update linux.yml

github-actions bot added the category: GHA label

ilya-lavrenov and others added 7 commits

November 25, 2024 11:07


          Update windows.yml

3538bbe


          Merge branch 'master' into use-continuos-batching-by-default

9eb4176


          Update linux.yml

a510e77


          Update windows.yml

eb0b0f4


          Merge branch 'master' into use-continuos-batching-by-default

d223042


          Merge branch 'master' into use-continuos-batching-by-default

cbd1ba8


          Merge branch 'master' into use-continuos-batching-by-default

62304ef

ScottZhang812 pushed a commit to ScottZhang812/_openvino.genai that referenced this pull request


          Generic fixes for CB integration via LLMPipeline (#961)

56b05c2

Extracted from
openvinotoolkit/openvino.genai#882

ScottZhang812 pushed a commit to ScottZhang812/_openvino.genai that referenced this pull request


          Fixed endless generations in test_callback_one_string test (#1189)

cffcb21

See
https://productionresultssa7.blob.core.windows.net/actions-results/1d9c7ed7-f60f-463a-8dba-2898f6f39e0a/workflow-job-run-2ad6080a-b02b-5d2d-feb5-675d9367578e/logs/job/job-logs.txt?rsct=text%2Fplain&se=2024-11-11T12%3A35%3A03Z&sig=cE1uxQm2H0TswczNt4jESUwM1%2BzN2uQBHxsm7Tqw0AI%3D&ske=2024-11-11T22%3A38%3A18Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2024-11-11T10%3A38%3A18Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2024-08-04&sp=r&spr=https&sr=b&st=2024-11-11T12%3A24%3A58Z&sv=2024-08-04

A lot of words are printed as max_new_tokens is quite huge number, while
exception is expected as streamer cannot work with batch.

To fix hanging of tests in this PR
openvinotoolkit/openvino.genai#882

ilya-lavrenov added 6 commits

December 23, 2024 16:43


          Merge branch 'master' into use-continuos-batching-by-default

44a9249


          Merge branch 'master' into use-continuos-batching-by-default

f7b2ac8


          Merge branch 'master' into use-continuos-batching-by-default

5f1417a


          Merge branch 'master' into use-continuos-batching-by-default

131f782


          Merge branch 'master' into use-continuos-batching-by-default

e517917


          Merge branch 'master' into use-continuos-batching-by-default

17d4333

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: GHA category: LLM do_not_merge do_not_review