Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support horaedb submit compaction task to remote #1563

Merged
merged 58 commits into from
Oct 30, 2024

Conversation

LeslieKid
Copy link
Contributor

@LeslieKid LeslieKid commented Sep 3, 2024

Rationale

The subtask to support compaction offloading. See #1545

Detailed Changes

Compaction node support remote compaction service

  • Define CompactionServiceImpl to support compaction rpc service.

  • Introduce NodeType to distinguish compaction node and horaedb node. Enable the deployment of compaction node.

  • Impl compaction_client for horaedb node to access remote compaction node.

Horaedb node support compaction offload

  • Introduce compaction_mode in analytic engine's Config to determine whether exec compaction offload or not.
  • Define CompactionNodePicker trait, supporting get remote compaction node info.
  • Impl RemoteCompactionRunner, supporting pick remote node and pass compaction task to the node.
  • Add docs (e.g. example-cluster-n.toml) to explain how to deploy a cluster supporting compaction offload.

Test Plan

src/cluster/src/cluster_impl.rs Outdated Show resolved Hide resolved
src/analytic_engine/src/setup.rs Outdated Show resolved Hide resolved
@LeslieKid LeslieKid changed the title feat(cluster): support horaedb submit compaction task to remote feat: support horaedb submit compaction task to remote Sep 20, 2024
@github-actions github-actions bot added the feature New feature or request label Sep 20, 2024
@Rachelint
Copy link
Contributor

It looks good to me.
I just helped to sort out codes, mainly move the compaction client to runner crate.

I am checking it in my local, let's merge it after the double check!

@LeslieKid
Copy link
Contributor Author

I am checking it in my local, let's merge it after the double check!

I check your commits in my local.
It's indeed better to move compaction client to runner crate.

I think we can merge it.

src/server/src/grpc/mod.rs Show resolved Hide resolved
src/server/src/grpc/compaction_service/mod.rs Outdated Show resolved Hide resolved
src/server/src/grpc/compaction_service/mod.rs Outdated Show resolved Hide resolved
src/analytic_engine/src/compaction/runner/remote_runner.rs Outdated Show resolved Hide resolved
src/analytic_engine/src/compaction/runner/remote_runner.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@Rachelint Rachelint left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Rachelint
Copy link
Contributor

Thanks @LeslieKid

@Rachelint Rachelint merged commit e47d9ae into apache:main Oct 30, 2024
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants