Change batch size calculation from message count to message size #388
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
LucioDonda
force-pushed
the
enhancement/385-change-batch-size-calculation-from-message-count-to-message-size
branch
4 times, most recently
from
54f39c2 to
54e4f50
Compare
LucioDonda
force-pushed
the
enhancement/385-change-batch-size-calculation-from-message-count-to-message-size
branch
from
72baa17 to
7455be2
Compare
vikman90
reviewed
Dec 11, 2024
There was a problem hiding this comment.
@LucioDonda Can you clarify why these errors are happening?
On the other hand, please confirm that the maximum proposed batch size matches the maximum body size that the server would accept.
LucioDonda
force-pushed
the
enhancement/385-change-batch-size-calculation-from-message-count-to-message-size
branch
2 times, most recently
from
5b109f0 to
019468e
Compare
TomasTurina
reviewed
Dec 11, 2024
TomasTurina
reviewed
Dec 11, 2024
src/agent/configuration_parser/include/configuration_parser.hpp
Outdated
Show resolved
Hide resolved
TomasTurina
reviewed
Dec 11, 2024
TomasTurina
reviewed
Dec 11, 2024
TomasTurina
reviewed
Dec 11, 2024
LucioDonda
force-pushed
the
enhancement/385-change-batch-size-calculation-from-message-count-to-message-size
branch
5 times, most recently
from
c09cfd7 to
c21cc9a
Compare
LucioDonda
force-pushed
the
enhancement/385-change-batch-size-calculation-from-message-count-to-message-size
branch
from
898109b to
19556a0
Compare
TomasTurina
reviewed
Dec 18, 2024
TomasTurina
reviewed
Dec 18, 2024
TomasTurina
reviewed
Dec 18, 2024
LucioDonda
force-pushed
the
enhancement/385-change-batch-size-calculation-from-message-count-to-message-size
branch
3 times, most recently
from
7f944d6 to
f60dd92
Compare
LucioDonda
force-pushed
the
enhancement/385-change-batch-size-calculation-from-message-count-to-message-size
branch
from
f60dd92 to
40885d3
Compare
TomasTurina
reviewed
Dec 18, 2024
| { | ||
| if constexpr (std::is_convertible_v<decltype(key), std::string_view>) | ||
| { | ||
| // This is a workaround for parsing size units |
There was a problem hiding this comment.
Please, open an issue to improve this.
There was a problem hiding this comment.
@TomasTurina Issue created : #447
TomasTurina
approved these changes
Dec 18, 2024
TomasTurina
deleted the
enhancement/385-change-batch-size-calculation-from-message-count-to-message-size
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
After implementing the configuration options for event batching (#306), we need to change the way we build the batch. Instead of specifying
batch_sizeas the number of messages we want to retrieve from the queue to build the message batch, we want to specify the batch size in bytes.The mechanism that the communicator component uses to retrieve messages needs to be changed, as it will now need to calculate the size of each message when creating the batch.
Configuration options
For this, the already implemented
batch_sizesetting will be used, but will accept the following values instead:Details:
Tests