Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: new settings stream_consume_batch_size_hint #17102

Merged
merged 3 commits into from
Dec 24, 2024

Conversation

dantengsky
Copy link
Member

@dantengsky dantengsky commented Dec 23, 2024

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Introduce new settings stream_consume_batch_size_hint,

  • by default, it is set to 0. in this case, stream_consume_batch_size_hint has no effects

if stream_consume_batch_size_hint is set to a non-zero value, streams will reference this setting during the process of capturing change sets and will try to keep the number of rows in the captured change set close to the specified value.

  • MAX_BATCH_SIZE specified in the WITH clause have a higher priority

    e.g.

statement ok
create table t_settings (c int);

statement ok
CREATE STREAM s_t_settings ON TABLE t_settings;

statement ok
INSERT INTO t_settings values(1);

statement ok
INSERT INTO t_settings values(2);

statement ok
INSERT INTO t_settings values(3);

statement ok
set stream_consume_batch_size_hint = 2;

#########################################################
# max_batch_size specified in query has higher priority #
#########################################################

query I
select c from s_t_settings with (max_batch_size = 1);
----
1

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Dec 23, 2024
@dantengsky dantengsky changed the title feat: new settings stream_consume_batch_size feat: new settings stream_consume_batch_size_hint Dec 24, 2024
@dantengsky dantengsky marked this pull request as ready for review December 24, 2024 04:57
@BohuTANG BohuTANG merged commit cee1496 into databendlabs:main Dec 24, 2024
90 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants