Optional persistance to object storage (S3 / GCP / etc) - like RocksDB cloud

[mindreframer]

Just a general question, would it make sense to implement persistence / "replication" via object storage?

• https://rocksdb.blogspot.com/2017/05/the-birth-of-rocksdb-cloud.html (reasoning)
• https://borthakur.com/ftp/rocksdb-cloud_dhruba_borthakur.pdf (slides)
• https://github.com/rockset/rocksdb-cloud (github)

Maybe it's not such a large stretch to add this on top fo Fjall.

I really like the fact, that your codebase is pure Rust and of very high quality. I'd rather use a Rust implementation, than C++. I'm not a Rust developer myself, so I cant offer my help here, just curious if this idea ever came to your mind.

Thanks and have a great weekend!
Roman

[marvin-j97]

I think I wouldn't want to integrate this directly but - like RocksDB-Cloud - have it be a separate repo, however, I think right now the APIs would be missing to hook into the journal (or replace it with a custom journal that works with e.g. Kinesis), or moving SST files around (especially on demand if it isn't cached on SSD already).

[mindreframer]

Thanks for the response!

So I guess this is not on the roadmap for the near future.

Maybe it would be possible to implement just this hook with some default behaviour (like writing to a log file, which SST table was generated), so that some process on the side could tail this log file and transparently upload the SST file to some obj. storage. So this would be the host processes responsibility to deal with the uploading / downloading of SST files for now. This would not allow the transparent fetching of the SST files from obj. storage for read request, but this would be a good first step into this direction. Otherwise one would have to watch on file created FS events and this is a little bit more fragile, I assume.