raft: fix flaky leader index in waitLeader function

[nvanbenschoten]

Fixes #127413.

This commit bypasses the larger rebase in #122133 to pick up the test flake fix in etcd-io/raft#188. There was some discussion in etcd-io/raft#181 about alternatives for fixing this test. For now, we stick with a direct cherry-pick.

Release note: None

[cockroach-teamcity]

This change is 

[pav-kv]

Thanks for picking this up.

[pav-kv]

Could you remove the TODO / skip in the beginning of TestRestart?

NB: Back then I thought that this test was inherently flaky and the fix was incomplete. There was some analysis in etcd-io/raft#195 (comment) and other comments there. We can revisit if this flake comes up again after your PR.

[pav-kv]

Should this be reset to -1 on each iteration? Might be better to move the definition inside the loop then.

[nvanbenschoten]

Done. I also moved to clearing the map on each iteration, instead of re-allocating it.

[pav-kv]

NB: Back then I thought that this test was inherently flaky and the fix was incomplete.

I stressed this now locally 20k times with the fix applied (one with the lindex := -1 moved inside the loop, as commented), and it did not flake. Before the fix it flakes pretty eagerly. So this flake will be much less / not at all as noisy now.

[nvanbenschoten]

Could you remove the TODO / skip in the beginning of TestRestart?

Done.

[nvanbenschoten]

I stressed this now locally 20k times with the fix applied (one with the lindex := -1 moved inside the loop, as commented), and it did not flake. Before the fix it flakes pretty eagerly. So this flake will be much less / not at all as noisy now.

I stressed again for 20k iterations and still see that it is stable.

bors r=pav-kv

[craig]

Build succeeded:

• unit_tests
• docker_image_amd64
• macos_arm64_build
• check_generated_code
• linux_arm64_build
• local_roachtest_fips
• local_roachtest
• linux_amd64_fips_build
• windows_build
• examples_orms
• lint
• linux_amd64_build
• macos_amd64_build
• acceptance

[blathers-crl]

Based on the specified backports for this PR, I applied new labels to the following linked issue(s). Please adjust the labels as needed to match the branches actually affected by the issue(s), including adding any known older branches.

---

Issue #127413: branch-release-24.1, branch-release-24.2.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

error creating merge commit from bd9edcb to blathers/backport-release-24.1-130084: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 24.1.x failed. See errors above.

---

error creating merge commit from bd9edcb to blathers/backport-release-24.2-130084: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 24.2.x failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}