Recommended configuration for large database?

[MarcoLugo]

Thanks for creating kvrocks, it's a wonderful use of rocksdb. I do have a question as I may not yet understand every single kvrocks/rocksdb parameter there is.

I've been testing kvrocks and obtaining great results with the default configuration so far, and workers set to the number of CPU cores. Excellent throughput basically saturating the NVMe and pretty low latency for GET (i.e. P99 below 1ms). This is for small keys (32 bytes or less) and values of 4kb or less. This is for a cluster of 3 master nodes.

Now, up to about 200 million keys, all that performance holds up well but at 400 million keys things start to deteriorate: throughput is cut in more than half and latency doubles. I think the data directory is a little over 100GB at that point. Maybe there is some additional tuning that is required at this scale and above? Is there a recommended configuration for billions of keys?

FWIW, I tried these two parameters but it didn't change anything #366 (comment).

Thank you.

[git-hulk]

Hi @MarcoLugo

Want to know most of your operations are GET? I cannot tell what configurations could be helpful with the current information. But I think you can check the usage of CPU/Disk IO to see if they're close to the bottleneck. And for the GET operation, you can increase the value of rocksdb.block_cache_size if your memory is still enough which the default is 4096(4GiB).

[MarcoLugo]

@git-hulk , I am using:

~# ./kvrocks --version
kvrocks version 2.10.1 (commit dc4d5303)

[git-hulk]

@MarcoLugo Thank you! Are you also using this version before? we got the feedback from another user that 2.10.1(rocksdb 9.x) performed worse than 2.8(rocksdb is 8.x).

[MarcoLugo]

@git-hulk Interesting! I was using 2.10.1 in both cases. I'll try 2.8 today or tomorrow and report back.

[MarcoLugo]

@git-hulk I tried with kvrocks version 2.8.0 (commit 311398a2) this time and the result was roughly the same, unfortunately. Using the following non-default parameters:

--workers 32 --rocksdb.block_cache_size 16384 --rocksdb.subkey_block_cache_size 10240 --rocksdb.share_metadata_and_subkey_block_cache yes

Any other suggestions?

[git-hulk]

@MarcoLugo Yes, that's not related to it if you're using 2.10 in both cases. What's the instance throughput and P99 latency for now?

You can remove --rocksdb.subkey_block_cache_size 10240 --rocksdb.share_metadata_and_subkey_block_cache yes since they are deprecated parameters. And you can try to decrease worker threads to 16.

You can try profiling it on one of your instance:

Step 1: Enabled the profiling

redis-cli > CONFIG SET profiling-sample-commands "GET"
redis-cli > CONFIG SET profiling-sample-ratio 1
redis-cli > CONFIG SET profiling-sample-record-threshold-ms 100

Step 2: Investigate the profiling logs

# Get latest 10 profiling logs
redis-cli > perflog get 10