Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhancement(compactor): record potential oom risks in compactor components #15946

Closed
Li0k opened this issue Mar 27, 2024 · 2 comments
Closed
Assignees
Labels
type/enhancement Improvements to existing implementation. type/feature
Milestone

Comments

@Li0k
Copy link
Contributor

Li0k commented Mar 27, 2024

  1. We adjust the tcp_recv_buffer for the s3 backend implementation. However, this will break when we switch to the opendal_s3 backend.
  2. The memory usage estimation algorithm does not take into account different backend implementations.
  3. The memory usage estimation algorithm is unable to estimate the memory usage of in-flight uploading sst. (
    self.sst_outputs.push(SplitTableOutput {
    )
@github-actions github-actions bot added this to the release-1.8 milestone Mar 27, 2024
@Li0k Li0k added type/bug Something isn't working type/enhancement Improvements to existing implementation. and removed type/bug Something isn't working labels Mar 27, 2024
@Li0k Li0k changed the title bug(compactor): record potential omm risks in compactor components enhancement(compactor): record potential omm risks in compactor components Mar 27, 2024
@lmatz lmatz changed the title enhancement(compactor): record potential omm risks in compactor components enhancement(compactor): record potential oom risks in compactor components Mar 28, 2024
@Li0k Li0k modified the milestones: release-1.8, release-1.9 Apr 8, 2024
@fuyufjh fuyufjh modified the milestones: release-1.9, release-1.10 May 14, 2024
@Li0k
Copy link
Contributor Author

Li0k commented May 15, 2024

related to #16685

@hzxa21
Copy link
Collaborator

hzxa21 commented Jul 10, 2024

#1682 introduced compactor_concurrent_uploading_sst_count to control the upload concurrency, which can reduce the memory footprint when the object store backend is slow. We decide to rely on this configuration to avoid OOM on non-standard s3 backend to ensure that we will not introduce performance overhead for compaction on s3 backend.

@hzxa21 hzxa21 closed this as completed Jul 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement Improvements to existing implementation. type/feature
Projects
None yet
Development

No branches or pull requests

3 participants