Improper use of Raft::metrics() can block the RaftCore async task and underlying OS thread

[SteveLauC]

What kind of doc do you ask for?

• Code comments.
• Architecture design.
• Algorithm explanation.
• Guide.
• Examples.

Describe the information you want

Document that:

When your AsyncRuntime::Watch uses tokio::watch, you use Raft::metrics()/server_metrics()/data_metrics() interface and hold the acquired Watch::Ref instance in an improper way, this can block the RaftCore async task as well as the underlying OS thread. If you are using a single-threaded runtime, like Monoio, everything is blocked now.

Root cause: tokio::watch uses synchronous RwLock, Watch::Ref contains a read guard, and the first thing that the RaftCore async task does is flush_metrics(), which needs to acquire the write guard, if there is a Watch::Ref instance exists, flush_metrics() will wait, thus it blocks the RaftCore async task and the underlying OS thread.

Describe the status of the current doc

• Is there any doc yet?:
  Currently we don't have any doc on this.

• What other information should be added?
  I guess no.

• I'll open a pull request for it:)

[github-actions]

👋 Thanks for opening this issue!

Get help or engage by:

• /help : to print help messages.
• /assignme : to assign this issue to you.