Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stream executor back pressure #65

Open
amitschang opened this issue Mar 14, 2024 · 0 comments
Open

stream executor back pressure #65

amitschang opened this issue Mar 14, 2024 · 0 comments
Labels
question Further information is requested triage incoming issue needs tagging and a look over

Comments

@amitschang
Copy link
Member

There could be a case where upstream tasks generate faster than downstream can consume and end up just filling up queues - for example if downstream processes need special hardware and we have extra CPUs, the system will go ahead piling on upstreams. This is not itself a big problem, but could be if we end up filling object store

A simple back pressure could be like: stop if size(downstream_queue) > X * total_tasks_runnable / batch_size where total_tasks_runnable is the number that could fit on the current cluster if it were empty and X is some scaling factor.

@amitschang amitschang added question Further information is requested triage incoming issue needs tagging and a look over labels Apr 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested triage incoming issue needs tagging and a look over
Projects
None yet
Development

No branches or pull requests

1 participant