Captive-cores should use bucketlistdb

[leighmcculloch]

it may be worth enabling BUCKETLISTDB, it may help performance of captive core when it's using on-disk db, it's being done on rpc captive core config for example - https://github.com/stellar/quickstart/blob/horizon-captive/futurenet/soroban-rpc/etc/stellar-captive-core.cfg#L11-L12

Originally posted by @sreuland in #486 (comment)

---

At the moment we only use it in soroban-rpc, but we could apply it to horizon's too of all networks in the image.

[leighmcculloch]

@sreuland I gave this a go by enabling bucketlistdb on Horizon's captive core in the quickstart image. CPU usage was higher by ~5%. It's unclear to me if this is a better outcome as I couldn't see any other measure of improvement. There's a few resources we could measure impact on, but increased CPU seems pretty bad for devices that run on batteries like laptops.

What sort of performance improvements should we see on captive-core?

Does captive-core use on disk db? I thought it'd be in memory anyway?

Could you share some examples of the performance improvement you see?

[sreuland]

Hello @leighmcculloch ,

What sort of performance improvements should we see on captive-core?

when the captive core db is located on disk, which is done in quickstart by default for horizon, the savings are in less round-trips to the db from core process, as the bucklist acts like a read cache first, it was enabled on horizon cc config by default on horizon/4733 whenever 'use disk' is also enabled via CAPTIVE_CORE_USE_DB=true which is the default horizon prefers to use for captive core config rather than 'in memory', horizon/4733 went in as of release horizon 2.24.0

Does captive-core use on disk db? I thought it'd be in memory anyway?

yes, for horizon cc in quickstart, it runs on disk mode,

Could you share some examples of the performance improvement you see?

@urvisavla , has actually done some recent horizon benchmarking for docs updates which included cc on disk and bucketlistdb which may provide more data. the findings are posted here - stellar/go#4960.

[leighmcculloch]

@sreuland Are you able to open a PR or share a diff of what would change in the quickstart image to use this different setup you were proposing? I suspect when I tested it I was not using the configuration you proposed.

[sreuland]

@leighmcculloch ,

#504

i'm trying out two runs locally using --pubnet to observe, one with current quickstart:testing from dockerhub which has no bucketlist enabled, and the other built with this config, will watch activity monitor and see if any spikes on cpu/ram

horizon attempts to enable bucketlist by default, but it does so based on core's version string, looking for a semver of 19.6 or above, and since horizon built from source on quickstart image, I think it's not getting this automatically, so would have to add the enablement directly in the core .cfg

[leighmcculloch]

I gave this a whirl using testnet, which I think is more representative of a developers use of quickstart. Pubnet is more data than a user is likely to use. The difference between the two doesn't seem that meaningful. Here's the resource usage of the docker container:

Without bucket list:

With bucket list:

The only differences I see from this run is that:

• the bucket list resulted in more disk read/writes
• the bucket list resulted in less cpu usage

Seems fine to move ahead with this change then.

[sreuland]

ok, when you run quickstart are you including --enable-horizon-captive-core, I realized that by default, and w/o including that flag, quickstart will launch horizon ingestion with --stellar-core-db-url which skips captive core. maybe in tandem to adding bucketlist to captive configs in quickstart, we take the same time to just remove --enable-horizon-captive-core so that horizon always uses captive core, and avoid confusion, i hacked it out in the sample pr - 681cc5d

docker run --rm -it     -p "8000:8000"     --name stellar     stellar/quickstart:testing   --pubnet --enable-horizon-captive-core

[leighmcculloch]

Ah, I missed that. Thanks. When we merge #498 then this won't be such an easy mistake to make. I'll rerun and see what it is like on my system with it enabled.

[sreuland]

cool, #498 looks good, maybe can just add the bucketlist enablement in there as it's closely related to captive core anyways? - ac9088a

I ran with --testnet --enable-horizon-captive-core and bucketlistdb enabled and disabled, nothing spiked in resources on either, running with --pubnet and bucketlistdb, it did elevate on CPU to max and stayed there when it got to applying:
stellar-core: Catching up; Catching up to ledger 48382143: Applying buckets 0%. Currently on level 10, I need to try it with no bucketlistdb to see if any different.

[leighmcculloch]

Ok, I got similar results to what I mentioned the first time I tested this (#490 (comment)):

Running without bucket list change:

Running with bucket list change:

The differences appear to be:

• CPU usage is more choppy and generally a bit higher when bucket list is enabled.
• Memory usage is a tiny bit lower when using the bucket list.

[github-actions]

This issue is stale because it has been open for 30 days with no activity. It will be closed in 30 days unless the stale label is removed.