storage: using concurrent data structure to incrementally maintain storage snapshot

[skyzh]

Currently, every operation on storage snapshot will clone and create a new full snapshot, which contains a list of rowsets. Indeed, we can use some concurrent data structure to maintain this, so as to reduce the need of cloning.

risinglight/src/storage/secondary/version_manager.rs

Lines 257 to 265 in 3e59c36

	// Add epoch number and make the modified snapshot available.
	let mut inner = self.inner.lock();
	assert_eq!(inner.epoch, current_epoch);
	inner.epoch += 1;
	let epoch = inner.epoch;
	inner.status.insert(epoch, Arc::new(snapshot));
	inner
	.rowset_deletion_to_apply
	.insert(epoch, rowset_deletion_to_apply);

https://github.com/ibraheemdev/seize looks like a good choice.

[BaymaxHWY]

we can use some concurrent data structure to maintain this, so as to reduce the need of cloning.

how to achieve this? make a snapshot of each epoch and only track the different from last?

[skyzh]

how to achieve this? make a snapshot of each epoch and only track the different from last?

Exactly, but I didn't come up with a detailed way to achieve this... Indeed, every operation to RisingLight's storage manifest is simply a diff operation.

[BaymaxHWY]

I'm sorry, i have no relevant experience about this. Unable to give helpful advice :)
Hope i can help you in the future.

[skyzh]

No worries 🤪