BR failed caused by PD reboot

[WangLe1321]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. execute br command to backup cluster
2. reboot all the pd instances during backup

2. What did you expect to see? (Required)

br can continue backup after pd finished rebooting

3. What did you see instead (Required)

br failed.

[2023/07/20 11:41:48.158 +00:00] [ERROR] [main.go:60] ["br failed"] [error="failed to flush checkpoint: Failed to update checkpoint lock.: EOF"] [errorVerbose="EOF\ngithub.mirror.nvdadr.com/tikv/pd/client.(*client).processTSORequests\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20221031025758-80f0d8ca4d07/client.go:1090\ngithub.mirror.nvdadr.com/tikv/pd/client.(*client).handleDispatcher\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20221031025758-80f0d8ca4d07/client.go:847\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\ngithub.mirror.nvdadr.com/tikv/pd/client.(*tsoRequest).Wait\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20221031025758-80f0d8ca4d07/client.go:1313\ngithub.mirror.nvdadr.com/tikv/pd/client.(*client).GetTS\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20221031025758-80f0d8ca4d07/client.go:1333\ngithub.mirror.nvdadr.com/pingcap/tidb/br/pkg/checkpoint.(*CheckpointRunner).getTS.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/checkpoint/checkpoint.go:541\ngithub.mirror.nvdadr.com/pingcap/tidb/br/pkg/utils.WithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:56\ngithub.mirror.nvdadr.com/pingcap/tidb/br/pkg/checkpoint.(*CheckpointRunner).getTS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/checkpoint/checkpoint.go:539\ngithub.mirror.nvdadr.com/pingcap/tidb/br/pkg/checkpoint.(*CheckpointRunner).updateLock\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/checkpoint/checkpoint.go:600\ngithub.mirror.nvdadr.com/pingcap/tidb/br/pkg/checkpoint.(*CheckpointRunner).startCheckpointRunner.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/checkpoint/checkpoint.go:387\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\nFailed to update checkpoint lock.\nfailed to flush checkpoint"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:60\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]

4. What is your TiDB version? (Required)

v6.5.3

[WangLe1321]

/label component/br

[ti-chi-bot]

@WangLe1321: The label(s) component/br cannot be applied. These labels are supported: fuzz/sqlancer, challenge-program, compatibility-breaker, first-time-contributor, contribution, require-LGT3, good first issue, correctness, duplicate, proposal, security, needs-more-info, needs-cherry-pick-release-5.2, needs-cherry-pick-release-5.3, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.0, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.2, needs-cherry-pick-release-6.3, needs-cherry-pick-release-6.4, needs-cherry-pick-release-6.5, needs-cherry-pick-release-6.6, needs-cherry-pick-release-7.0, needs-cherry-pick-release-7.1, needs-cherry-pick-release-7.2, affects-5.2, affects-5.3, affects-5.4, affects-6.0, affects-6.1, affects-6.2, affects-6.3, affects-6.4, affects-6.5, affects-6.6, affects-7.0, affects-7.1, affects-7.2, may-affects-5.2, may-affects-5.3, may-affects-5.4, may-affects-6.0, may-affects-6.1, may-affects-6.2, may-affects-6.3, may-affects-6.4, may-affects-6.5, may-affects-6.6, may-affects-7.0, may-affects-7.1, may-affects-7.2.

In response to this:

/label component/br

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[Leavrth]

#43611 In v6.5.4