Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

raft/rafttest: TestRestart failed #127413

Closed
cockroach-teamcity opened this issue Jul 18, 2024 · 10 comments · Fixed by #130084
Closed

raft/rafttest: TestRestart failed #127413

cockroach-teamcity opened this issue Jul 18, 2024 · 10 comments · Fixed by #130084
Assignees
@pav-kv
Labels
A-testing Testing tools and infrastructure branch-master Failures and bugs on the master branch. branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-kv KV Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jul 18, 2024 •
edited by cockroach-jira-scripts
Loading

raft/rafttest.TestRestart failed with artifacts on master @ 46854026bc11b3870c8bbefc976f5550b18dd82c:

raft2024/07/18 04:23:33 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/07/18 04:23:33 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/07/18 04:23:33 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/07/18 04:23:33 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/07/18 04:23:33 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/07/18 04:23:33 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/07/18 04:23:33 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/07/18 04:23:33 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/07/18 04:23:33 INFO: 5 is starting a new election at term 2
raft2024/07/18 04:23:33 INFO: 5 became candidate at term 3
raft2024/07/18 04:23:33 INFO: 5 [logterm: 2, index: 6] sent MsgVote request to 1 at term 3
raft2024/07/18 04:23:33 INFO: 5 [logterm: 2, index: 6] sent MsgVote request to 2 at term 3
raft2024/07/18 04:23:33 INFO: 5 [logterm: 2, index: 6] sent MsgVote request to 3 at term 3
raft2024/07/18 04:23:33 INFO: 5 [logterm: 2, index: 6] sent MsgVote request to 4 at term 3
raft2024/07/18 04:23:33 INFO: raft.node: 5 lost leader 2 at term 3
raft2024/07/18 04:23:33 INFO: 5 received MsgVoteResp from 5 at term 3
raft2024/07/18 04:23:33 INFO: 5 has received 1 MsgVoteResp votes and 0 vote rejections
raft2024/07/18 04:23:33 INFO: 1 [term: 2] received a MsgVote message with higher term from 5 [term: 3]
raft2024/07/18 04:23:33 INFO: 1 became follower at term 3
raft2024/07/18 04:23:33 INFO: 1 [logterm: 2, index: 6, vote: 0] cast MsgVote for 5 [logterm: 2, index: 6] at term 3
raft2024/07/18 04:23:33 INFO: raft.node: 1 lost leader 2 at term 3
raft2024/07/18 04:23:33 INFO: 4 [term: 2] received a MsgVote message with higher term from 5 [term: 3]
raft2024/07/18 04:23:33 INFO: 4 became follower at term 3
raft2024/07/18 04:23:33 INFO: 4 [logterm: 2, index: 6, vote: 0] cast MsgVote for 5 [logterm: 2, index: 6] at term 3
raft2024/07/18 04:23:33 INFO: raft.node: 4 lost leader 2 at term 3
raft2024/07/18 04:23:33 INFO: 5 received MsgVoteResp from 1 at term 3
raft2024/07/18 04:23:33 INFO: 5 has received 2 MsgVoteResp votes and 0 vote rejections
raft2024/07/18 04:23:33 INFO: 3 [term: 2] received a MsgVote message with higher term from 5 [term: 3]
raft2024/07/18 04:23:33 INFO: 3 became follower at term 3
raft2024/07/18 04:23:33 INFO: 3 [logterm: 2, index: 6, vote: 0] cast MsgVote for 5 [logterm: 2, index: 6] at term 3
raft2024/07/18 04:23:33 INFO: raft.node: 3 lost leader 2 at term 3
raft2024/07/18 04:23:33 INFO: 2 [term: 2] received a MsgVote message with higher term from 5 [term: 3]
raft2024/07/18 04:23:33 INFO: 2 became follower at term 3
raft2024/07/18 04:23:33 INFO: 2 [logterm: 2, index: 6, vote: 0] cast MsgVote for 5 [logterm: 2, index: 6] at term 3
raft2024/07/18 04:23:33 INFO: raft.node: 2 lost leader 2 at term 3
raft2024/07/18 04:23:33 INFO: 5 received MsgVoteResp from 3 at term 3
raft2024/07/18 04:23:33 INFO: 5 has received 3 MsgVoteResp votes and 0 vote rejections
raft2024/07/18 04:23:33 INFO: 5 became leader at term 3
raft2024/07/18 04:23:33 INFO: raft.node: 5 elected leader 5 at term 3
raft2024/07/18 04:23:33 INFO: raft.node: 2 elected leader 5 at term 3
raft2024/07/18 04:23:33 INFO: raft.node: 1 elected leader 5 at term 3
raft2024/07/18 04:23:33 INFO: raft.node: 3 elected leader 5 at term 3
raft2024/07/18 04:23:33 INFO: raft.node: 4 elected leader 5 at term 3
    node_test.go:91: commits failed to converge!
I240718 04:23:38.520629 388 (gostd) node.go:100  [-] 8  raft.1: stop
I240718 04:23:38.520684 616 (gostd) node.go:100  [-] 9  raft.2: stop
I240718 04:23:38.520698 614 (gostd) node.go:100  [-] 10  raft.3: stop
I240718 04:23:38.520708 394 (gostd) node.go:100  [-] 11  raft.4: stop
I240718 04:23:38.521851 396 (gostd) node.go:100  [-] 12  raft.5: stop
--- FAIL: TestRestart (5.20s)
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-40428
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels Jul 18, 2024
@nvanbenschoten
Copy link
Member

This is the same flake that was discussed in etcd-io/raft#181. It was later fixed upstream by etcd-io/raft#188.

We are in the process of picking this fix up over in #122133. @pav-kv do we want to revive that PR and get it merged so that we can close this? If not, I'll import just that one commit.

@nvanbenschoten nvanbenschoten added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-testing Testing tools and infrastructure P-2 Issues/test failures with a fix SLA of 3 months and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Jul 18, 2024
@cockroach-teamcity
Copy link
Member Author

raft/rafttest.TestRestart failed with artifacts on master @ 4ae4a52772b4b6b2bf2c340e06919c090894dffb:

raft2024/08/03 00:03:03 INFO: 3 not forwarding to itself at term 2; dropping proposal
raft2024/08/03 00:03:03 INFO: 3 not forwarding to itself at term 2; dropping proposal
raft2024/08/03 00:03:03 INFO: 3 not forwarding to itself at term 2; dropping proposal
raft2024/08/03 00:03:03 INFO: 3 not forwarding to itself at term 2; dropping proposal
raft2024/08/03 00:03:03 INFO: 3 not forwarding to itself at term 2; dropping proposal
raft2024/08/03 00:03:03 INFO: 3 not forwarding to itself at term 2; dropping proposal
raft2024/08/03 00:03:03 INFO: 3 not forwarding to itself at term 2; dropping proposal
raft2024/08/03 00:03:03 INFO: 3 not forwarding to itself at term 2; dropping proposal
raft2024/08/03 00:03:03 INFO: 4 is starting a new election at term 2
raft2024/08/03 00:03:03 INFO: 4 became candidate at term 3
raft2024/08/03 00:03:03 INFO: 4 [logterm: 2, index: 6] sent MsgVote request to 1 at term 3
raft2024/08/03 00:03:03 INFO: 4 [logterm: 2, index: 6] sent MsgVote request to 2 at term 3
raft2024/08/03 00:03:03 INFO: 4 [logterm: 2, index: 6] sent MsgVote request to 3 at term 3
raft2024/08/03 00:03:03 INFO: 4 [logterm: 2, index: 6] sent MsgVote request to 5 at term 3
raft2024/08/03 00:03:03 INFO: raft.node: 4 lost leader 3 at term 3
raft2024/08/03 00:03:03 INFO: 4 received MsgVoteResp from 4 at term 3
raft2024/08/03 00:03:03 INFO: 4 has received 1 MsgVoteResp votes and 0 vote rejections
raft2024/08/03 00:03:03 INFO: 5 [term: 2] received a MsgVote message with higher term from 4 [term: 3]
raft2024/08/03 00:03:03 INFO: 5 became follower at term 3
raft2024/08/03 00:03:03 INFO: 5 [logterm: 2, index: 6, vote: 0] cast MsgVote for 4 [logterm: 2, index: 6] at term 3
raft2024/08/03 00:03:03 INFO: raft.node: 5 lost leader 3 at term 3
raft2024/08/03 00:03:03 INFO: 3 [term: 2] received a MsgVote message with higher term from 4 [term: 3]
raft2024/08/03 00:03:03 INFO: 3 became follower at term 3
raft2024/08/03 00:03:03 INFO: 3 [logterm: 2, index: 6, vote: 0] cast MsgVote for 4 [logterm: 2, index: 6] at term 3
raft2024/08/03 00:03:03 INFO: raft.node: 3 lost leader 3 at term 3
raft2024/08/03 00:03:03 INFO: 1 [term: 2] received a MsgVote message with higher term from 4 [term: 3]
raft2024/08/03 00:03:03 INFO: 1 became follower at term 3
raft2024/08/03 00:03:03 INFO: 1 [logterm: 2, index: 6, vote: 0] cast MsgVote for 4 [logterm: 2, index: 6] at term 3
raft2024/08/03 00:03:03 INFO: raft.node: 1 lost leader 3 at term 3
raft2024/08/03 00:03:03 INFO: 2 [term: 2] received a MsgVote message with higher term from 4 [term: 3]
raft2024/08/03 00:03:03 INFO: 2 became follower at term 3
raft2024/08/03 00:03:03 INFO: 2 [logterm: 2, index: 6, vote: 0] cast MsgVote for 4 [logterm: 2, index: 6] at term 3
raft2024/08/03 00:03:03 INFO: raft.node: 2 lost leader 3 at term 3
raft2024/08/03 00:03:03 INFO: 4 received MsgVoteResp from 3 at term 3
raft2024/08/03 00:03:03 INFO: 4 has received 2 MsgVoteResp votes and 0 vote rejections
raft2024/08/03 00:03:03 INFO: 4 received MsgVoteResp from 5 at term 3
raft2024/08/03 00:03:03 INFO: 4 has received 3 MsgVoteResp votes and 0 vote rejections
raft2024/08/03 00:03:03 INFO: 4 became leader at term 3
raft2024/08/03 00:03:03 INFO: raft.node: 4 elected leader 4 at term 3
raft2024/08/03 00:03:03 INFO: raft.node: 5 elected leader 4 at term 3
raft2024/08/03 00:03:03 INFO: raft.node: 3 elected leader 4 at term 3
raft2024/08/03 00:03:03 INFO: raft.node: 1 elected leader 4 at term 3
raft2024/08/03 00:03:03 INFO: raft.node: 2 elected leader 4 at term 3
    node_test.go:91: commits failed to converge!
I240803 00:03:08.972291 1723 (gostd) node.go:100  [-] 8  raft.1: stop
I240803 00:03:08.972370 1817 (gostd) node.go:100  [-] 9  raft.2: stop
I240803 00:03:08.972385 1815 (gostd) node.go:100  [-] 10  raft.3: stop
I240803 00:03:08.974438 1729 (gostd) node.go:100  [-] 11  raft.4: stop
I240803 00:03:08.975517 1731 (gostd) node.go:100  [-] 12  raft.5: stop
--- FAIL: TestRestart (5.21s)
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

raft/rafttest.TestRestart failed with artifacts on master @ 167a9af39d0c1fadb9de8ee3c40e616e6c98af78:

I240808 19:00:45.050083 1756 (gostd) node.go:100  [-] 9  raft.2: stop
I240808 19:00:45.050159 1754 (gostd) node.go:100  [-] 10  raft.3: stop
I240808 19:00:45.051214 1668 (gostd) node.go:100  [-] 11  raft.4: stop
I240808 19:00:45.051264 1670 (gostd) node.go:100  [-] 12  raft.5: stop
raft2024/08/08 19:00:45 INFO: 1 switched to configuration voters=()
raft2024/08/08 19:00:45 INFO: 1 became follower at term 0
raft2024/08/08 19:00:45 INFO: newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
raft2024/08/08 19:00:45 INFO: 1 became follower at term 1
raft2024/08/08 19:00:45 INFO: 1 switched to configuration voters=(1)
raft2024/08/08 19:00:45 INFO: 1 switched to configuration voters=(1 2)
raft2024/08/08 19:00:45 INFO: 1 switched to configuration voters=(1 2 3)
raft2024/08/08 19:00:45 INFO: 1 switched to configuration voters=(1 2 3 4)
raft2024/08/08 19:00:45 INFO: 1 switched to configuration voters=(1 2 3 4 5)
raft2024/08/08 19:00:45 INFO: 2 switched to configuration voters=()
raft2024/08/08 19:00:45 INFO: 2 became follower at term 0
raft2024/08/08 19:00:45 INFO: newRaft 2 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
raft2024/08/08 19:00:45 INFO: 2 became follower at term 1
raft2024/08/08 19:00:45 INFO: 2 switched to configuration voters=(1)
raft2024/08/08 19:00:45 INFO: 2 switched to configuration voters=(1 2)
raft2024/08/08 19:00:45 INFO: 2 switched to configuration voters=(1 2 3)
raft2024/08/08 19:00:45 INFO: 2 switched to configuration voters=(1 2 3 4)
raft2024/08/08 19:00:45 INFO: 2 switched to configuration voters=(1 2 3 4 5)
raft2024/08/08 19:00:45 INFO: 3 switched to configuration voters=()
raft2024/08/08 19:00:45 INFO: 3 became follower at term 0
raft2024/08/08 19:00:45 INFO: newRaft 3 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
raft2024/08/08 19:00:45 INFO: 3 became follower at term 1
raft2024/08/08 19:00:45 INFO: 3 switched to configuration voters=(1)
raft2024/08/08 19:00:45 INFO: 3 switched to configuration voters=(1 2)
raft2024/08/08 19:00:45 INFO: 3 switched to configuration voters=(1 2 3)
raft2024/08/08 19:00:45 INFO: 3 switched to configuration voters=(1 2 3 4)
raft2024/08/08 19:00:45 INFO: 3 switched to configuration voters=(1 2 3 4 5)
raft2024/08/08 19:00:45 INFO: 4 switched to configuration voters=()
raft2024/08/08 19:00:45 INFO: 4 became follower at term 0
raft2024/08/08 19:00:45 INFO: newRaft 4 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
raft2024/08/08 19:00:45 INFO: 4 became follower at term 1
raft2024/08/08 19:00:45 INFO: 4 switched to configuration voters=(1)
raft2024/08/08 19:00:45 INFO: 4 switched to configuration voters=(1 2)
raft2024/08/08 19:00:45 INFO: 4 switched to configuration voters=(1 2 3)
raft2024/08/08 19:00:45 INFO: 4 switched to configuration voters=(1 2 3 4)
raft2024/08/08 19:00:45 INFO: 4 switched to configuration voters=(1 2 3 4 5)
raft2024/08/08 19:00:45 INFO: 5 switched to configuration voters=()
raft2024/08/08 19:00:45 INFO: 5 became follower at term 0
raft2024/08/08 19:00:45 INFO: newRaft 5 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
raft2024/08/08 19:00:45 INFO: 5 became follower at term 1
raft2024/08/08 19:00:45 INFO: 5 switched to configuration voters=(1)
raft2024/08/08 19:00:45 INFO: 5 switched to configuration voters=(1 2)
raft2024/08/08 19:00:45 INFO: 5 switched to configuration voters=(1 2 3)
raft2024/08/08 19:00:45 INFO: 5 switched to configuration voters=(1 2 3 4)
raft2024/08/08 19:00:45 INFO: 5 switched to configuration voters=(1 2 3 4 5)
--- FAIL: TestRestart (5.20s)
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

raft/rafttest.TestRestart failed with artifacts on master @ bb2613e5348ea864c6a9cc348af73117d701f453:

raft2024/08/13 16:49:54 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/13 16:49:54 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/13 16:49:54 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/13 16:49:54 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/13 16:49:54 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/13 16:49:54 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/13 16:49:54 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/13 16:49:54 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/13 16:49:54 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/13 16:49:54 INFO: 4 is starting a new election at term 2
raft2024/08/13 16:49:54 INFO: 4 became candidate at term 3
raft2024/08/13 16:49:54 INFO: 4 [logterm: 2, index: 6] sent MsgVote request to 1 at term 3
raft2024/08/13 16:49:54 INFO: 4 [logterm: 2, index: 6] sent MsgVote request to 2 at term 3
raft2024/08/13 16:49:54 INFO: 4 [logterm: 2, index: 6] sent MsgVote request to 3 at term 3
raft2024/08/13 16:49:54 INFO: 4 [logterm: 2, index: 6] sent MsgVote request to 5 at term 3
raft2024/08/13 16:49:54 INFO: raft.node: 4 lost leader 2 at term 3
raft2024/08/13 16:49:54 INFO: 4 received MsgVoteResp from 4 at term 3
raft2024/08/13 16:49:54 INFO: 4 has received 1 MsgVoteResp votes and 0 vote rejections
raft2024/08/13 16:49:54 INFO: 1 [term: 2] received a MsgVote message with higher term from 4 [term: 3]
raft2024/08/13 16:49:54 INFO: 1 became follower at term 3
raft2024/08/13 16:49:54 INFO: 1 [logterm: 2, index: 6, vote: 0] cast MsgVote for 4 [logterm: 2, index: 6] at term 3
raft2024/08/13 16:49:54 INFO: raft.node: 1 lost leader 2 at term 3
raft2024/08/13 16:49:54 INFO: 5 [term: 2] received a MsgVote message with higher term from 4 [term: 3]
raft2024/08/13 16:49:54 INFO: 5 became follower at term 3
raft2024/08/13 16:49:54 INFO: 5 [logterm: 2, index: 6, vote: 0] cast MsgVote for 4 [logterm: 2, index: 6] at term 3
raft2024/08/13 16:49:54 INFO: raft.node: 5 lost leader 2 at term 3
raft2024/08/13 16:49:54 INFO: 3 [term: 2] received a MsgVote message with higher term from 4 [term: 3]
raft2024/08/13 16:49:54 INFO: 3 became follower at term 3
raft2024/08/13 16:49:54 INFO: 3 [logterm: 2, index: 6, vote: 0] cast MsgVote for 4 [logterm: 2, index: 6] at term 3
raft2024/08/13 16:49:54 INFO: raft.node: 3 lost leader 2 at term 3
raft2024/08/13 16:49:54 INFO: 4 received MsgVoteResp from 3 at term 3
raft2024/08/13 16:49:54 INFO: 4 has received 2 MsgVoteResp votes and 0 vote rejections
raft2024/08/13 16:49:54 INFO: 2 [term: 2] received a MsgVote message with higher term from 4 [term: 3]
raft2024/08/13 16:49:54 INFO: 2 became follower at term 3
raft2024/08/13 16:49:54 INFO: 2 [logterm: 2, index: 6, vote: 0] cast MsgVote for 4 [logterm: 2, index: 6] at term 3
raft2024/08/13 16:49:54 INFO: raft.node: 2 lost leader 2 at term 3
raft2024/08/13 16:49:54 INFO: 4 received MsgVoteResp from 2 at term 3
raft2024/08/13 16:49:54 INFO: 4 has received 3 MsgVoteResp votes and 0 vote rejections
raft2024/08/13 16:49:54 INFO: 4 became leader at term 3
raft2024/08/13 16:49:54 INFO: raft.node: 4 elected leader 4 at term 3
raft2024/08/13 16:49:54 INFO: raft.node: 1 elected leader 4 at term 3
raft2024/08/13 16:49:54 INFO: raft.node: 5 elected leader 4 at term 3
raft2024/08/13 16:49:54 INFO: raft.node: 2 elected leader 4 at term 3
raft2024/08/13 16:49:54 INFO: raft.node: 3 elected leader 4 at term 3
I240813 16:49:59.527205 1604 (gostd) node.go:102  [-] 8  raft.1: stop
I240813 16:49:59.527268 716 (gostd) node.go:102  [-] 9  raft.2: stop
I240813 16:49:59.527281 714 (gostd) node.go:102  [-] 10  raft.3: stop
I240813 16:49:59.527339 1610 (gostd) node.go:102  [-] 11  raft.4: stop
    node_test.go:91: commits failed to converge!
--- FAIL: TestRestart (5.19s)
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

raft/rafttest.TestRestart failed with artifacts on master @ 1993fc04b5116f20f4814d637c7ce87b003687e4:

raft2024/08/17 04:09:52 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/17 04:09:52 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/17 04:09:52 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/17 04:09:52 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/17 04:09:52 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/17 04:09:52 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/17 04:09:52 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/17 04:09:52 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/17 04:09:52 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/17 04:09:52 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/17 04:09:52 INFO: 5 is starting a new election at term 2
raft2024/08/17 04:09:52 INFO: 5 became candidate at term 3
raft2024/08/17 04:09:52 INFO: 5 [logterm: 2, index: 6] sent MsgVote request to 1 at term 3
raft2024/08/17 04:09:52 INFO: 5 [logterm: 2, index: 6] sent MsgVote request to 2 at term 3
raft2024/08/17 04:09:52 INFO: 5 [logterm: 2, index: 6] sent MsgVote request to 3 at term 3
raft2024/08/17 04:09:52 INFO: 5 [logterm: 2, index: 6] sent MsgVote request to 4 at term 3
raft2024/08/17 04:09:52 INFO: raft.node: 5 lost leader 2 at term 3
raft2024/08/17 04:09:52 INFO: 5 received MsgVoteResp from 5 at term 3
raft2024/08/17 04:09:52 INFO: 5 has received 1 MsgVoteResp votes and 0 vote rejections
raft2024/08/17 04:09:52 INFO: 2 [term: 2] received a MsgVote message with higher term from 5 [term: 3]
raft2024/08/17 04:09:52 INFO: 2 became follower at term 3
raft2024/08/17 04:09:52 INFO: 2 [logterm: 2, index: 6, vote: 0] cast MsgVote for 5 [logterm: 2, index: 6] at term 3
raft2024/08/17 04:09:52 INFO: raft.node: 2 lost leader 2 at term 3
raft2024/08/17 04:09:52 INFO: 4 [term: 2] received a MsgVote message with higher term from 5 [term: 3]
raft2024/08/17 04:09:52 INFO: 4 became follower at term 3
raft2024/08/17 04:09:52 INFO: 4 [logterm: 2, index: 6, vote: 0] cast MsgVote for 5 [logterm: 2, index: 6] at term 3
raft2024/08/17 04:09:52 INFO: raft.node: 4 lost leader 2 at term 3
raft2024/08/17 04:09:52 INFO: 1 [term: 2] received a MsgVote message with higher term from 5 [term: 3]
raft2024/08/17 04:09:52 INFO: 1 became follower at term 3
raft2024/08/17 04:09:52 INFO: 1 [logterm: 2, index: 6, vote: 0] cast MsgVote for 5 [logterm: 2, index: 6] at term 3
raft2024/08/17 04:09:52 INFO: raft.node: 1 lost leader 2 at term 3
raft2024/08/17 04:09:52 INFO: 5 received MsgVoteResp from 2 at term 3
raft2024/08/17 04:09:52 INFO: 5 has received 2 MsgVoteResp votes and 0 vote rejections
raft2024/08/17 04:09:52 INFO: 3 [term: 2] received a MsgVote message with higher term from 5 [term: 3]
raft2024/08/17 04:09:52 INFO: 3 became follower at term 3
raft2024/08/17 04:09:52 INFO: 3 [logterm: 2, index: 6, vote: 0] cast MsgVote for 5 [logterm: 2, index: 6] at term 3
raft2024/08/17 04:09:52 INFO: raft.node: 3 lost leader 2 at term 3
raft2024/08/17 04:09:52 INFO: 5 received MsgVoteResp from 1 at term 3
raft2024/08/17 04:09:52 INFO: 5 has received 3 MsgVoteResp votes and 0 vote rejections
raft2024/08/17 04:09:52 INFO: 5 became leader at term 3
raft2024/08/17 04:09:52 INFO: raft.node: 5 elected leader 5 at term 3
raft2024/08/17 04:09:52 INFO: raft.node: 1 elected leader 5 at term 3
raft2024/08/17 04:09:52 INFO: raft.node: 4 elected leader 5 at term 3
raft2024/08/17 04:09:52 INFO: raft.node: 2 elected leader 5 at term 3
raft2024/08/17 04:09:52 INFO: raft.node: 3 elected leader 5 at term 3
    node_test.go:91: commits failed to converge!
I240817 04:09:57.144339 948 (gostd) node.go:102  [-] 8  raft.1: stop
I240817 04:09:57.145540 1014 (gostd) node.go:102  [-] 9  raft.2: stop
I240817 04:09:57.145629 1012 (gostd) node.go:102  [-] 10  raft.3: stop
--- FAIL: TestRestart (5.16s)
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

raft/rafttest.TestRestart failed with artifacts on master @ 972ccd5d17c54f2cef84e38d14290e52b01bd70a:

I240827 19:44:11.128486 2418 (gostd) node.go:102  [-] 10  raft.3: stop
I240827 19:44:11.128499 2353 (gostd) node.go:102  [-] 11  raft.4: stop
I240827 19:44:11.128510 2355 (gostd) node.go:102  [-] 12  raft.5: stop
raft2024/08/27 19:44:11 INFO: 1 switched to configuration voters=()
raft2024/08/27 19:44:11 INFO: 1 became follower at term 0
raft2024/08/27 19:44:11 INFO: newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
raft2024/08/27 19:44:11 INFO: 1 became follower at term 1
raft2024/08/27 19:44:11 INFO: 1 switched to configuration voters=(1)
raft2024/08/27 19:44:11 INFO: 1 switched to configuration voters=(1 2)
raft2024/08/27 19:44:11 INFO: 1 switched to configuration voters=(1 2 3)
raft2024/08/27 19:44:11 INFO: 1 switched to configuration voters=(1 2 3 4)
raft2024/08/27 19:44:11 INFO: 1 switched to configuration voters=(1 2 3 4 5)
raft2024/08/27 19:44:11 INFO: 2 switched to configuration voters=()
raft2024/08/27 19:44:11 INFO: 2 became follower at term 0
raft2024/08/27 19:44:11 INFO: newRaft 2 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
raft2024/08/27 19:44:11 INFO: 2 became follower at term 1
raft2024/08/27 19:44:11 INFO: 2 switched to configuration voters=(1)
raft2024/08/27 19:44:11 INFO: 2 switched to configuration voters=(1 2)
raft2024/08/27 19:44:11 INFO: 2 switched to configuration voters=(1 2 3)
raft2024/08/27 19:44:11 INFO: 2 switched to configuration voters=(1 2 3 4)
raft2024/08/27 19:44:11 INFO: 2 switched to configuration voters=(1 2 3 4 5)
raft2024/08/27 19:44:11 INFO: 3 switched to configuration voters=()
raft2024/08/27 19:44:11 INFO: 3 became follower at term 0
raft2024/08/27 19:44:11 INFO: newRaft 3 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
raft2024/08/27 19:44:11 INFO: 3 became follower at term 1
raft2024/08/27 19:44:11 INFO: 3 switched to configuration voters=(1)
raft2024/08/27 19:44:11 INFO: 3 switched to configuration voters=(1 2)
raft2024/08/27 19:44:11 INFO: 3 switched to configuration voters=(1 2 3)
raft2024/08/27 19:44:11 INFO: 3 switched to configuration voters=(1 2 3 4)
raft2024/08/27 19:44:11 INFO: 3 switched to configuration voters=(1 2 3 4 5)
raft2024/08/27 19:44:11 INFO: 4 switched to configuration voters=()
raft2024/08/27 19:44:11 INFO: 4 became follower at term 0
raft2024/08/27 19:44:11 INFO: newRaft 4 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
raft2024/08/27 19:44:11 INFO: 4 became follower at term 1
raft2024/08/27 19:44:11 INFO: 4 switched to configuration voters=(1)
raft2024/08/27 19:44:11 INFO: 4 switched to configuration voters=(1 2)
raft2024/08/27 19:44:11 INFO: 4 switched to configuration voters=(1 2 3)
raft2024/08/27 19:44:11 INFO: 4 switched to configuration voters=(1 2 3 4)
raft2024/08/27 19:44:11 INFO: 4 switched to configuration voters=(1 2 3 4 5)
raft2024/08/27 19:44:11 INFO: 5 switched to configuration voters=()
raft2024/08/27 19:44:11 INFO: 5 became follower at term 0
raft2024/08/27 19:44:11 INFO: newRaft 5 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
raft2024/08/27 19:44:11 INFO: 5 became follower at term 1
raft2024/08/27 19:44:11 INFO: 5 switched to configuration voters=(1)
raft2024/08/27 19:44:11 INFO: 5 switched to configuration voters=(1 2)
raft2024/08/27 19:44:11 INFO: 5 switched to configuration voters=(1 2 3)
raft2024/08/27 19:44:11 INFO: 5 switched to configuration voters=(1 2 3 4)
raft2024/08/27 19:44:11 INFO: 5 switched to configuration voters=(1 2 3 4 5)
    node_test.go:91: commits failed to converge!
--- FAIL: TestRestart (5.16s)
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@github-project-automation github-project-automation bot moved this to roachtest/unit test backlog in KV Aug 28, 2024
@cockroach-teamcity
Copy link
Member Author

raft/rafttest.TestRestart failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/08/31 04:10:50 INFO: 2 not forwarding to itself at term 2; dropping proposal
    node_test.go:91: commits failed to converge!
I240831 04:10:55.441900 2411 (gostd) node.go:104  [-] 8  raft.1: stop
I240831 04:10:55.441975 2548 (gostd) node.go:104  [-] 9  raft.2: stop
I240831 04:10:55.441992 2546 (gostd) node.go:104  [-] 10  raft.3: stop
I240831 04:10:55.442054 2417 (gostd) node.go:104  [-] 11  raft.4: stop
I240831 04:10:55.442081 2451 (gostd) node.go:104  [-] 12  raft.5: stop
--- FAIL: TestRestart (5.17s)
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

raft/rafttest.TestRestart failed with artifacts on master @ 344fb32842be378bc82c2f3ed8ba12d6f3225be7:

raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
raft2024/09/02 19:40:56 INFO: 2 not forwarding to itself at term 3; dropping proposal
    node_test.go:91: commits failed to converge!
I240902 19:41:01.775546 1399 (gostd) node.go:104  [-] 8  raft.1: stop
I240902 19:41:01.775603 1463 (gostd) node.go:104  [-] 9  raft.2: stop
I240902 19:41:01.775618 1461 (gostd) node.go:104  [-] 10  raft.3: stop
I240902 19:41:01.775629 1405 (gostd) node.go:104  [-] 11  raft.4: stop
--- FAIL: TestRestart (5.25s)
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Sep 4, 2024
@nvanbenschoten
raft: fix flaky leader index in waitLeader function
c429d4e 
Fixes cockroachdb#127413.

This commit bypasses the larger rebase in cockroachdb#122133 to pick up the test
flake fix in etcd-io/raft#188. There was some
discussion in etcd-io/raft#181 about alternatives
for fixing this test. For now, we stick with a direct cherry-pick.

Release note: None
@nvanbenschoten nvanbenschoten mentioned this issue Sep 4, 2024
Merged
@cockroach-teamcity
Copy link
Member Author

raft/rafttest.TestRestart failed with artifacts on master @ 77606cbe1c98b248ea658105d7109479439a037f:

raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
raft2024/09/04 22:29:37 INFO: 2 not forwarding to itself at term 2; dropping proposal
    node_test.go:91: commits failed to converge!
--- FAIL: TestRestart (5.17s)
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

craig bot pushed a commit that referenced this issue Sep 6, 2024
@nvanbenschoten
Merge #130084
ea50475 
130084: raft: fix flaky leader index in waitLeader function r=pav-kv a=nvanbenschoten

Fixes #127413.

This commit bypasses the larger rebase in #122133 to pick up the test flake fix in etcd-io/raft#188. There was some discussion in etcd-io/raft#181 about alternatives for fixing this test. For now, we stick with a direct cherry-pick.

Release note: None

Co-authored-by: Nathan VanBenschoten <[email protected]>
@craig craig bot closed this as completed in bd9edcb Sep 6, 2024
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Sep 6, 2024

Based on the specified backports for linked PR #130084, I applied the following new label(s) to this issue: branch-release-24.1, branch-release-24.2. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl bot added branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 labels Sep 6, 2024
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Sep 6, 2024
@nvanbenschoten
raft: fix flaky leader index in waitLeader function
3780ceb 
Fixes cockroachdb#127413.

This commit bypasses the larger rebase in cockroachdb#122133 to pick up the test
flake fix in etcd-io/raft#188. There was some
discussion in etcd-io/raft#181 about alternatives
for fixing this test. For now, we stick with a direct cherry-pick.

Release note: None
@nvanbenschoten nvanbenschoten mentioned this issue Sep 6, 2024
Merged
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Sep 6, 2024
@nvanbenschoten
raft: fix flaky leader index in waitLeader function
86c8928 
Fixes cockroachdb#127413.

This commit bypasses the larger rebase in cockroachdb#122133 to pick up the test
flake fix in etcd-io/raft#188. There was some
discussion in etcd-io/raft#181 about alternatives
for fixing this test. For now, we stick with a direct cherry-pick.

Release note: None
@nvanbenschoten nvanbenschoten mentioned this issue Sep 6, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pav-kv pav-kv

Labels
A-testing Testing tools and infrastructure branch-master Failures and bugs on the master branch. branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-kv KV Team
Projects
No open projects
KV
Status: roachtest/unit test backlog
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

raft: fix flaky leader index in waitLeader function nvanbenschoten/cockroach
3 participants
@pav-kv @nvanbenschoten @cockroach-teamcity