Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce SnapshotRepository and object store integration #2310

Merged
merged 22 commits into from
Dec 6, 2024
Merged

Introduce SnapshotRepository and object store integration #2310

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
ee6f413
Update paths to remove SK, simplify structure
pcholakov Nov 21, 2024
98e72e2
Move metadata serialization to repository
pcholakov Nov 21, 2024
afd6507
Maintain a latest.json pointer object
pcholakov Nov 21, 2024
18df90f
reformat
pcholakov Nov 21, 2024
ee9c22f
Addressing various PR comments
pcholakov Nov 25, 2024
723ade7
Update snapshot object store paths to include padded LSN and id
pcholakov Nov 25, 2024
56ade31
Prevent latest snapshot pointer from moving backwards
pcholakov Nov 25, 2024
03c1cfc
Reuse a BytesMut buffer for upload chunk data
pcholakov Nov 26, 2024
5252df6
Add PUT conditions to make the latest.json pointer update a CAS opera…
pcholakov Nov 26, 2024
cfd8652
Fix conditional put for latest to use UpdateVersion
pcholakov Nov 30, 2024
bc62ccc
Enable S3 conditional updates with latest object_store
pcholakov Nov 30, 2024
f1d1f8c
Allow git source for arrow-rs (no release of object_store includes S3…
pcholakov Nov 30, 2024
4f27d19
Add snapshot upload progress tracking and best-effort cleanup on error
pcholakov Dec 1, 2024
235565c
Update CreateSnapshot RPC request timeout to 90s
pcholakov Dec 1, 2024
063ba17
Reuse allocated upload buffer across files
pcholakov Dec 1, 2024
db9975e
Address latest PR feedback
pcholakov Dec 5, 2024
44f2c01
Bump pprof (RUSTSEC-2024-0408)
pcholakov Dec 5, 2024
6745ad9
Refactor S3/local file destination latest snapshot update test
pcholakov Dec 5, 2024
1a25a1d
Add comments to SnapshotRepository
pcholakov Dec 5, 2024
9889aa5
Revert to object_store 0.11 which does indeed work great with the lat…
pcholakov Dec 5, 2024
aee1a12
Validate object store uploads
pcholakov Dec 5, 2024
24abfde
Self-review updates
pcholakov Dec 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 94 additions & 33 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,8 @@ assert2 = "0.3.11"
async-channel = "2.1.1"
async-trait = "0.1.73"
axum = { version = "0.7.5", default-features = false }
aws-config = "1.5.10"
aws-credential-types = "1.2.1"
base64 = "0.22"
bitflags = { version = "2.6.0" }
bytes = { version = "1.7", features = ["serde"] }
Expand All @@ -102,7 +104,6 @@ datafusion = { version = "42.0.0", default-features = false, features = [
"regex_expressions",
"unicode_expressions",
] }
object_store = { version = "0.11.1"}
datafusion-expr = { version = "42.0.0" }
derive_builder = "0.20.0"
derive_more = { version = "1", features = ["full"] }
Expand Down Expand Up @@ -139,6 +140,7 @@ metrics-exporter-prometheus = { version = "0.15", default-features = false, feat
"async-runtime",
] }
moka = "0.12.5"
object_store = { version = "0.11.1", features = ["aws"] }
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we are not officially supporting anything other than S3, it was easier to just not compile the other providers. The file provider is always enabled, it seems.

once_cell = "1.18"
opentelemetry = { version = "0.24.0" }
opentelemetry-http = { version = "0.13.0" }
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ arc-swap = { workspace = true }
futures = { workspace = true }
futures-util = { workspace = true }
http = { workspace = true }
pprof = { version = "0.13", features = ["criterion", "flamegraph"] }
pprof = { version = "0.14", features = ["criterion", "flamegraph"] }
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated RUSTSEC update.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this :-)

reqwest = { workspace = true }
rlimit = { workspace = true }
serde_json = { workspace = true }
Expand Down
2 changes: 1 addition & 1 deletion crates/admin/src/cluster_controller/service.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -449,7 +449,7 @@ where
) -> anyhow::Result<SnapshotId> {
// todo(pavel): make snapshot RPC timeout configurable, especially if this includes remote upload in the future
let response = tokio::time::timeout(
Duration::from_secs(30),
Duration::from_secs(90),
self.create_snapshot_router.call(
&self.network_sender,
node_id,
Expand Down
Loading
Loading