backup: fix retry of fine-grained backup (#43252)

[ti-chi-bot]

This is an automated cherry-pick of #43252

What problem does this PR solve?

Issue Number: close #43236

Problem Summary:
We are using the tikv.Backoffer to implement retry. Which isn't so customizable hence we cannot fully control the printed error messages and back off duration.

What is changed and how it works?

This PR added a new structure RetryWithBackoffer to adapt the retry logic with the TiDB backoffer logic, at the same time it has enhanced the error message.
This PR will also increase the max retry time of fine-grained backup. The reason is simple: given it begins the fine-grained backup, there must be some problems in the cluster. We need to be more patient.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Fixed a bug that may cause BR report false "resolve lock timeout" error.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2023-06-30 05:55:59.589679501 +0000 UTC m=+65042.964257351: ☑️ agreed by YuJuncen.
• 2023-06-30 05:56:54.579471841 +0000 UTC m=+65097.954049691: ☑️ agreed by Leavrth.

[YuJuncen]

/retest-required
🤔️

[tonyxuqqi]

/test

[ti-chi-bot]

@tonyxuqqi: The /test command needs one or more targets.
The following commands are available to trigger required jobs:

• /test build
• /test check-dev
• /test check-dev2
• /test mysql-test
• /test unit-test

Use /test all to run all jobs.

In response to this:

/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tonyxuqqi]

/test mysql-test

[tiprow]

@tonyxuqqi: No presubmit jobs available for pingcap/tidb@release-6.1

In response to this:

/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tiprow]

@tonyxuqqi: No presubmit jobs available for pingcap/tidb@release-6.1

In response to this:

/test mysql-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tonyxuqqi]

/merge

[ti-chi-bot]

@tonyxuqqi: We have migrated to builtin LGTM and approve plugins for reviewing.

Please use /approve when you want approve this pull request.

The changes announcement: LGTM plugin changes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[tonyxuqqi]

/approve

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3pointer, Leavrth, tonyxuqqi, YuJuncen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [3pointer,Leavrth,YuJuncen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Benjamin2037]

/retest