pdutil(ticdc): split `tidb_ddl_job` table #6673

overvenus · 2022-08-09T08:41:20Z

What problem does this PR solve?

Issue Number: close #4756

What is changed and how it works?

Split the table tidb_ddl_job.

Check List

Tests

Unit test
Manual test

Logs from PD leader.

[2022/08/09 08:33:44.514 +00:00] [INFO] [operator_controller.go:450] ["add operator"] [region-id=8978001] [operator="\"labeler-split-region {split: region 8978001 use policy USEKEY and keys [748000FFFFFFFFFFFFFE00000000000000F8 748000FFFFFFFFFFFFFF00000000000000F8]} (kind:split, region:8978001(4850, 5), createAt:2022-08-09 08:33:44.512985722 +0000 UTC m=+1131983.134539904, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, size:2, steps:[split region with policy USEKEY])\""] [additional-info="{\"region-end-key\":\"\",\"region-start-key\":\"7480000000000003FF5F5F728000000000FF436EF30000000000FA\"}"]
[2022/08/09 08:33:44.516 +00:00] [INFO] [operator_controller.go:652] ["send schedule command"] [region-id=8978001] [step="split region with policy USEKEY"] [source=create]
[2022/08/09 08:33:44.519 +00:00] [INFO] [cluster_worker.go:139] ["alloc ids for region split"] [region-id=10365001] [peer-ids="[10365002,10365003,10365004]"]
[2022/08/09 08:33:44.519 +00:00] [INFO] [cluster_worker.go:139] ["alloc ids for region split"] [region-id=10365005] [peer-ids="[10365006,10365007,10365008]"]
[2022/08/09 08:33:44.524 +00:00] [INFO] [region.go:580] ["region Version changed"] [region-id=8978001] [detail="StartKey Changed:{7480000000000003FF5F5F728000000000FF436EF30000000000FA} -> {748000FFFFFFFFFFFFFF00000000000000F8}, EndKey:{}"] [old-version=4850] [new-version=4852]
[2022/08/09 08:33:44.527 +00:00] [INFO] [operator_controller.go:566] ["operator finish"] [region-id=8978001] [takes=11.372527ms] [operator="\"labeler-split-region {split: region 8978001 use policy USEKEY and keys [748000FFFFFFFFFFFFFE00000000000000F8 748000FFFFFFFFFFFFFF00000000000000F8]} (kind:split, region:8978001(4850, 5), createAt:2022-08-09 08:33:44.512985722 +0000 UTC m=+1131983.134539904, startAt:2022-08-09 08:33:44.516296175 +0000 UTC m=+1131983.137850403, currentStep:1, size:2, steps:[split region with policy USEKEY]) finished\""] [additional-info="{\"region-end-key\":\"\",\"region-start-key\":\"7480000000000003FF5F5F728000000000FF436EF30000000000FA\"}"]
[2022/08/09 08:33:44.526 +00:00] [INFO] [cluster_worker.go:231] ["region batch split, generate new regions"] [region-id=8978001] [origin="id:10365001 start_key:\"7480000000000003FF5F5F728000000000FF436EF30000000000FA\" end_key:\"748000FFFFFFFFFFFFFE00000000000000F8\" region_epoch:<conf_ver:5 version:4852 > peers:<id:10365002 store_id:1 > peers:<id:10365003 store_id:4 > peers:<id:10365004 store_id:5 > id:10365005 start_key:\"748000FFFFFFFFFFFFFE00000000000000F8\" end_key:\"748000FFFFFFFFFFFFFF00000000000000F8\" region_epoch:<conf_ver:5 version:4852 > peers:<id:10365006 store_id:1 > peers:<id:10365007 store_id:4 > peers:<id:10365008 store_id:5 >"] [total=2]

$ pd-ctl region 10365001
{
  "id": 10365001,
  "start_key": "7480000000000003FF5F5F728000000000FF436EF30000000000FA",
  "end_key": "748000FFFFFFFFFFFFFE00000000000000F8",
  "epoch": {
    "conf_ver": 5,
    "version": 4852
  },
  "peers": [
    {
      "id": 10365002,
      "store_id": 1,
      "role_name": "Voter"
    },
    {
      "id": 10365003,
      "store_id": 4,
      "role_name": "Voter"
    },
    {
      "id": 10365004,
      "store_id": 5,
      "role_name": "Voter"
    }
  ],
  "leader": {
    "id": 10365002,
    "store_id": 1,
    "role_name": "Voter"
  },
  "written_bytes": 0,
  "read_bytes": 0,
  "written_keys": 0,
  "read_keys": 0,
  "approximate_size": 2,
  "approximate_keys": 10813
}

$ pd-ctl region 8978001
{
  "id": 8978001,
  "start_key": "748000FFFFFFFFFFFFFF00000000000000F8",
  "end_key": "",
  "epoch": {
    "conf_ver": 5,
    "version": 4852
  },
  "peers": [
    {
      "id": 8978002,
      "store_id": 1,
      "role_name": "Voter"
    },
    {
      "id": 8978003,
      "store_id": 4,
      "role_name": "Voter"
    },
    {
      "id": 8978004,
      "store_id": 5,
      "role_name": "Voter"
    }
  ],
  "leader": {
    "id": 8978002,
    "store_id": 1,
    "role_name": "Voter"
  },
  "written_bytes": 362,
  "read_bytes": 0,
  "written_keys": 6,
  "read_keys": 0,
  "approximate_size": 1,
  "approximate_keys": 0
}

Questions

Will it cause performance regression or break compatibility?

No.

Do you need to update user documentation, design documentation or monitoring documentation?

No.

Release note

None

Signed-off-by: Neil Shen <overvenus@gmail.com>

ti-chi-bot · 2022-08-09T08:41:23Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

3AceShowHand
CharlesCheung96

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

lonng · 2022-08-09T08:53:23Z

pkg/pdutil/pd_api_client_test.go

+// RegionLabel is the label of a region.
+// NOTE: This type is exported by HTTP API. Please pay more attention when modifying it.
+// Copied from github.com/tikv/pd/server/schedule/labeler
+type RegionLabel struct {


Why not import the github.com/tikv/pd/server/schedule/labeler directly? I saw it's a public type.

See: https://github.com/tikv/pd/blob/master/server/schedule/labeler/rules.go#L32

It make TiCDC depends on pd server which may bloat TiCDC binary size and dependency graph.

Actually, we have depends on it. see: https://github.com/pingcap/tiflow/blob/master/go.mod#L77

Tried, failed to compile tests

make tidy go mod tidy ./tools/check/check-tidy.sh go: finding module for package google.golang.org/grpc/naming github.com/pingcap/tiflow/pkg/pdutil tested by github.com/pingcap/tiflow/pkg/pdutil.test imports github.com/tikv/pd/server/schedule/labeler imports github.com/tikv/pd/server/storage/endpoint imports go.etcd.io/etcd/clientv3 tested by go.etcd.io/etcd/clientv3.test imports go.etcd.io/etcd/integration imports go.etcd.io/etcd/proxy/grpcproxy imports google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.48.0), but does not contain package google.golang.org/grpc/naming make: *** [Makefile:292: tidy] Error 1

We have to upgrade etcd version in PD or patch etcd in TiCDC side, I think it's not worth to do so, only complicates already complexed dependency.

lonng

Rest LGTM

Signed-off-by: Neil Shen <overvenus@gmail.com>

lance6716 · 2022-08-09T09:38:35Z

there are 3 tables newly introduced in 6.2

https://github.com/pingcap/tidb/blob/28c934cdae80d15f1433ad5948bbc46885532b8e/ddl/constant.go#L21-L27

overvenus · 2022-08-09T11:08:16Z

there are 3 tables newly introduced in 6.2

https://github.com/pingcap/tidb/blob/28c934cdae80d15f1433ad5948bbc46885532b8e/ddl/constant.go#L21-L27

TiCDC cares tidb_ddl_job only.

lonng · 2022-08-10T02:04:53Z

/merge

ti-chi-bot · 2022-08-10T02:04:55Z

This pull request has been accepted and is ready to merge.

Commit hash: 84e3a72

overvenus · 2022-08-10T04:13:31Z

/run-verify
/run-leak-test

overvenus · 2022-08-10T05:16:53Z

/merge

ti-chi-bot · 2022-08-10T05:16:55Z

This pull request has been accepted and is ready to merge.

Commit hash: da15a1d

overvenus · 2022-08-10T07:06:02Z

/run-verify
/run-leak-test

overvenus · 2022-08-10T07:27:52Z

/run-verify
/run-leak-test

Signed-off-by: Neil Shen <overvenus@gmail.com>

overvenus · 2022-08-10T07:57:08Z

/merge

ti-chi-bot · 2022-08-10T07:57:11Z

This pull request has been accepted and is ready to merge.

Commit hash: 8753323

overvenus · 2022-08-10T09:01:13Z

/run-verify

overvenus · 2022-08-10T09:21:44Z

/run-dm-integration-test

ti-chi-bot · 2022-08-10T10:16:58Z

@overvenus: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2022-08-10T11:11:26Z

In response to a cherrypick label: new pull request created: #6700.

ti-chi-bot · 2022-08-10T11:12:03Z

In response to a cherrypick label: new pull request created: #6701.

pdutil(ticdc): split tidb_ddl_job table

a76d80c

Signed-off-by: Neil Shen <overvenus@gmail.com>

overvenus added the area/ticdc Issues or PRs related to TiCDC. label Aug 9, 2022

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. labels Aug 9, 2022

overvenus requested review from CharlesCheung96 and 3AceShowHand August 9, 2022 08:41

ti-chi-bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Aug 9, 2022

overvenus added needs-cherry-pick-release-6.2 Should cherry pick this PR to release-6.2 branch. and removed needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. labels Aug 9, 2022

ti-chi-bot added the needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. label Aug 9, 2022

lonng reviewed Aug 9, 2022

View reviewed changes

add some comments

84e3a72

Signed-off-by: Neil Shen <overvenus@gmail.com>

CharlesCheung96 approved these changes Aug 9, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Aug 9, 2022

3AceShowHand approved these changes Aug 9, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Aug 9, 2022

lonng approved these changes Aug 10, 2022

View reviewed changes

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 10, 2022

Merge branch 'master' into split-ddl-table

da15a1d

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 10, 2022

overvenus force-pushed the split-ddl-table branch from c1d1bca to da15a1d Compare August 10, 2022 05:16

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Aug 10, 2022

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 10, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 10, 2022

fix leak tests

8753323

Signed-off-by: Neil Shen <overvenus@gmail.com>

overvenus force-pushed the split-ddl-table branch from b05c8ab to 8753323 Compare August 10, 2022 07:56

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Aug 10, 2022

ti-chi-bot added status/can-merge Indicates a PR has been approved by a committer. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 10, 2022

ti-chi-bot added 2 commits August 10, 2022 15:57

Merge branch 'master' into split-ddl-table

ab206d4

Merge branch 'master' into split-ddl-table

ec9b101

Merge branch 'master' into split-ddl-table

bd416a1

ti-chi-bot merged commit c3a1204 into pingcap:master Aug 10, 2022

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Aug 10, 2022

This is an automated cherry-pick of pingcap#6673

69df069

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Aug 10, 2022

pdutil(ticdc): split tidb_ddl_job table (#6673) #6700

Closed

ti-chi-bot mentioned this pull request Aug 10, 2022

pdutil(ticdc): split tidb_ddl_job table (#6673) #6701

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pdutil(ticdc): split `tidb_ddl_job` table #6673

pdutil(ticdc): split `tidb_ddl_job` table #6673

overvenus commented Aug 9, 2022 •

edited

Loading

ti-chi-bot commented Aug 9, 2022 •

edited

Loading

lonng Aug 9, 2022

overvenus Aug 9, 2022

lonng Aug 9, 2022

overvenus Aug 9, 2022

lonng Aug 10, 2022

lonng left a comment

lance6716 commented Aug 9, 2022

overvenus commented Aug 9, 2022

lonng commented Aug 10, 2022

ti-chi-bot commented Aug 10, 2022

overvenus commented Aug 10, 2022

overvenus commented Aug 10, 2022

ti-chi-bot commented Aug 10, 2022

overvenus commented Aug 10, 2022

overvenus commented Aug 10, 2022

overvenus commented Aug 10, 2022

ti-chi-bot commented Aug 10, 2022

overvenus commented Aug 10, 2022

overvenus commented Aug 10, 2022

ti-chi-bot commented Aug 10, 2022

ti-chi-bot commented Aug 10, 2022

ti-chi-bot commented Aug 10, 2022

pdutil(ticdc): split tidb_ddl_job table #6673

pdutil(ticdc): split tidb_ddl_job table #6673

Conversation

overvenus commented Aug 9, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Tests

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

ti-chi-bot commented Aug 9, 2022 • edited Loading

lonng Aug 9, 2022

Choose a reason for hiding this comment

overvenus Aug 9, 2022

Choose a reason for hiding this comment

lonng Aug 9, 2022

Choose a reason for hiding this comment

overvenus Aug 9, 2022

Choose a reason for hiding this comment

lonng Aug 10, 2022

Choose a reason for hiding this comment

lonng left a comment

Choose a reason for hiding this comment

lance6716 commented Aug 9, 2022

overvenus commented Aug 9, 2022

lonng commented Aug 10, 2022

ti-chi-bot commented Aug 10, 2022

overvenus commented Aug 10, 2022

overvenus commented Aug 10, 2022

ti-chi-bot commented Aug 10, 2022

overvenus commented Aug 10, 2022

overvenus commented Aug 10, 2022

overvenus commented Aug 10, 2022

ti-chi-bot commented Aug 10, 2022

overvenus commented Aug 10, 2022

overvenus commented Aug 10, 2022

ti-chi-bot commented Aug 10, 2022

ti-chi-bot commented Aug 10, 2022

ti-chi-bot commented Aug 10, 2022

pdutil(ticdc): split `tidb_ddl_job` table #6673

pdutil(ticdc): split `tidb_ddl_job` table #6673

overvenus commented Aug 9, 2022 •

edited

Loading

ti-chi-bot commented Aug 9, 2022 •

edited

Loading