Recover time may be very long if TiKV enables hibernate region

[sticnarf]

Enhancement

Considering one TiKV is unhealthy, all requests sending to that store will timeout. But due to hibernate region, the regions whose leaders are on the unhealthy store will not be re-elected until a request is sent to the follower.

If there are thousands of such regions, every time we are accessing a new region, it will send a request to the original leader on the unhealthy TiKV because the leader definitely does not change. And then, the user will always experience very long request duration until all regions are touched and elect new leaders.

We can use the health check service of TiKV to get the serving status of TiKV. By filtering out healthy nodes, we can recover from TiKV failure more quickly.

[windtalker]

@sticnarf if it is an enhancement, we should not add affects-xxx label

[sticnarf]

It's allowed to backport small enhancements, according to internal discussions in PingCAP release teams.

[cfzjywxk]

@sticnarf
Need a cherry-pick for the v6.1.1 release version.