change(state): Stop using iterators on column families with many deletions

[teor2345]

Motivation

When we use a RocksDB iterator on a column family with lots of deleted keys, database performance decreases by up to 300x.

Closes #7618

Specifications

This is a known issue with RocksDB, here are similar issues in other projects:
https://tracker.ceph.com/issues/55324
https://jira.mariadb.org/browse/MDEV-19670

Solution

As a quick fix we can replace the iterator with get(&key).
The permanent fix in PR #7392 stops deleting these keys entirely.

Manual Testing

I did a local test of this fix, and Zebra is up to 1.2 million blocks in under 5 hours. This is faster than Marek's main branch checks on his machine, and about as fast as v1.2.0 in CI:
https://github.com/ZcashFoundation/zebra/actions/runs/6049184682/job/16453440408

I think v1.2.0 was about as fast on my machine, but I haven't done any detailed tests. (A manual check is ok, we can do detailed checks in #7649.)

Review

This is an urgent fix which is blocking CI from working.

Reviewer Checklist

• Will the PR name make sense to users?
  • Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
• Are the PR labels correct?
• Does the code do what the ticket and PR says?
  • Does it change concurrent code, unsafe code, or consensus rules?
• How do you know it works? Does it have tests?

Follow Up Work

Permanent fixes for all column families that delete keys:

• merge the permanent fix to these column families in PR fix(security): fix concurrency issues in tree key formats, and CPU usage in genesis tree roots #7392
• Set iterator read bounds on deleting column families to avoid very slow syncs #7664

Performance testing for this fix:

• Add a quick test for sampling initial sync performance #7649

[teor2345]

Once we've verified this performance fix, I think we should admin-merge this PR, because it is currently blocked by CI failures in #7659 and #7661.

[teor2345]

Manual Testing

I did a local test of this fix on my machine, and Zebra is up to 1.2 million blocks in under 5 hours. This is faster than Marek's main branch checks on his machine, and about as fast as v1.2.0 in CI:
https://github.com/ZcashFoundation/zebra/actions/runs/6049184682/job/16453440408

I think v1.2.0 was about as fast on my machine, but I haven't done any detailed tests. So it looks like this fixes the bug. (And the way we use iterators in other PRs doesn't cause similar bugs.)

But I'd appreciate someone else checking the performance as well. A manual check is ok, we can do detailed checks in #7649.

[upbqdn]

This PR is as fast as v1.2.0, so it resolves the slow sync.

I wonder if we could speed Zebra up even further by fixing a similar issue that degrades performance elsewhere, but we accepted it.

[arya2]

In the merge queue, the test passed but the job failed:

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 45 filtered out; finished in 10465.70s
tee: 'standard output': Broken pipe
Error: Process completed with exit code 1.

https://github.com/ZcashFoundation/zebra/actions/runs/6405648679/job/17392070905#step:9:5457

[arya2]

@Mergifyio refresh

[mergify]

refresh

✅ Pull request refreshed

[teor2345]

In the merge queue, the test passed but the job failed:

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 45 filtered out; finished in 10465.70s
tee: 'standard output': Broken pipe
Error: Process completed with exit code 1.
ZcashFoundation/zebra/actions/runs/6405648679/job/17392070905#step:9:5457

I reopened PR #7564 for this.

Since this PR is failing due to bug #7665, I'm going to admin-merge it.