Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Meta region merge may cause latency spike #4756

Closed
Tracked by #4757
overvenus opened this issue Mar 3, 2022 · 2 comments · Fixed by #4937 or #6673
Closed
Tracked by #4757

Meta region merge may cause latency spike #4756

overvenus opened this issue Mar 3, 2022 · 2 comments · Fixed by #4937 or #6673
Assignees
@CharlesCheung96
Labels
affects-5.4 area/ticdc Issues or PRs related to TiCDC. component/kv-client TiKV kv log client component. component/tikv TiKV component. type/enhancement The issue or PR belongs to an enhancement.

Comments

@overvenus
Copy link
Member

image

[2022/03/02 03:21:22.530 +08:00] [INFO] [apply.rs:2495] ["execute CommitMerge"] [source_region="id: 33380 end_key: 7480000000000000FF175F728000000000FF000F3C0000000000FA region_epoch { conf_ver: 6 version: 115 } peers { id: 33381 store_id: 1 } peers { id: 33382 store_id:
[2022/03/02 03:21:22.530 +08:00] [INFO] [peer.rs:2964] ["notify pd with merge"] [target_region="id: 33548 end_key: 7480000000000000FF175F728000000000FF000F7E0000000000FA region_epoch { conf_ver: 5 version: 116 } peers { id: 33549 store_id: 1 } peers { id: 33550 store_id:
[2022/03/02 03:35:13.765 +08:00] [INFO] [apply.rs:2495] ["execute CommitMerge"] [source_region="id: 33548 end_key: 7480000000000000FF175F728000000000FF000F7E0000000000FA region_epoch { conf_ver: 6 version: 117 } peers { id: 33549 store_id: 1 } peers { id: 33550 store_id:
[2022/03/02 03:35:13.766 +08:00] [INFO] [peer.rs:2964] ["notify pd with merge"] [target_region="id: 33480 end_key: 7480000000000000FF175F728000000000FF000FA50000000000FA region_epoch { conf_ver: 5 version: 118 } peers { id: 33481 store_id: 1 } peers { id: 33482 store_id:
[2022/03/02 03:37:30.519 +08:00] [INFO] [apply.rs:2495] ["execute CommitMerge"] [source_region="id: 33480 end_key: 7480000000000000FF175F728000000000FF000FA50000000000FA region_epoch { conf_ver: 6 version: 119 } peers { id: 33481 store_id: 1 } peers { id: 33482 store_id:
[2022/03/02 03:37:30.519 +08:00] [INFO] [peer.rs:2964] ["notify pd with merge"] [target_region="id: 33424 end_key: 7480000000000000FF175F728000000000FF000FDE0000000000FA region_epoch { conf_ver: 5 version: 120 } peers { id: 33425 store_id: 1 } peers { id: 33426 store_id:
[2022/03/02 04:12:06.786 +08:00] [INFO] [apply.rs:2495] ["execute CommitMerge"] [source_region="id: 33424 end_key: 7480000000000000FF175F728000000000FF000FDE0000000000FA region_epoch { conf_ver: 6 version: 121 } peers { id: 33425 store_id: 1 } peers { id: 33426 store_id:
[2022/03/02 04:38:00.557 +08:00] [INFO] [apply.rs:2495] ["execute CommitMerge"] [source_region="id: 33744 end_key: 7480000000000000FF175F728000000000FF0010270000000000FA region_epoch { conf_ver: 6 version: 123 } peers { id: 33745 store_id: 1 } peers { id: 33746 store_id:
[2022/03/02 04:40:03.330 +08:00] [INFO] [apply.rs:2495] ["execute CommitMerge"] [source_region="id: 33484 end_key: 7480000000000000FF175F728000000000FF0010500000000000FA region_epoch { conf_ver: 6 version: 125 } peers { id: 33485 store_id: 1 } peers { id: 33486 store_id:
[2022/03/02 04:40:38.384 +08:00] [INFO] [apply.rs:1370] ["execute admin command"] [command="cmd_type: PrepareMerge prepare_merge { min_index: 13454 target { id: 33736 end_key: 7480000000000000FF175F728000000000FF00107E0000000000FA region_epoch { conf_ver: 5 version: 126 }
[2022/03/02 04:40:38.394 +08:00] [INFO] [peer.rs:3080] ["merge finished"] [target_region="Some(id: 33736 end_key: 7480000000000000FF175F728000000000FF00107E0000000000FA region_epoch { conf_ver: 5 version: 126 } peers { id: 33737 store_id: 1 } peers { id: 33738 store_id: 5
[2022/03/02 04:40:41.902 +08:00] [INFO] [apply.rs:2495] ["execute CommitMerge"] [source_region="id: 33736 end_key: 7480000000000000FF175F728000000000FF00109B0000000000FA region_epoch { conf_ver: 6 version: 128 } peers { id: 33737 store_id: 1 } peers { id: 33738 store_id:
[2022/03/02 04:40:45.392 +08:00] [INFO] [apply.rs:2495] ["execute CommitMerge"] [source_region="id: 33732 end_key: 7480000000000000FF175F728000000000FF0010C90000000000FA region_epoch { conf_ver: 6 version: 130 } peers { id: 33733 store_id: 1 } peers { id: 33734 store_id:
[2022/03/02 04:41:20.468 +08:00] [INFO] [apply.rs:1370] ["execute admin command"] [command="cmd_type: PrepareMerge prepare_merge { min_index: 14040 target { id: 33684 end_key: 7480000000000000FF175F728000000000FF0010E50000000000FA region_epoch { conf_ver: 5 version: 131 }
[2022/03/02 04:41:20.479 +08:00] [INFO] [peer.rs:3080] ["merge finished"] [target_region="Some(id: 33684 end_key: 7480000000000000FF175F728000000000FF0010E50000000000FA region_epoch { conf_ver: 5 version: 131 } peers { id: 33685 store_id: 1 } peers { id: 33686 store_id: 5

TiCDC lag increases when meta region gets merged.
@overvenus overvenus added component/kv-client TiKV kv log client component. component/tikv TiKV component. type/enhancement The issue or PR belongs to an enhancement. area/ticdc Issues or PRs related to TiCDC. labels Mar 3, 2022
@overvenus overvenus mentioned this issue Mar 3, 2022
Closed
37 tasks
@overvenus overvenus changed the title Meta region merge may cause TiCDC lag spike Meta region merge may cause TiCDC latency spike Mar 3, 2022
@overvenus overvenus changed the title Meta region merge may cause TiCDC latency spike Meta region merge may cause latency spike Mar 3, 2022
@overvenus overvenus mentioned this issue Mar 3, 2022
Closed
@CharlesCheung96 CharlesCheung96 self-assigned this Mar 8, 2022
@CharlesCheung96 CharlesCheung96 mentioned this issue Mar 17, 2022
Merged
@CharlesCheung96 CharlesCheung96 mentioned this issue Mar 30, 2022
Closed
ti-chi-bot pushed a commit that referenced this issue May 7, 2022
@CharlesCheung96
owner(ticdc): Add support for region-label to enable meta-region isol…
dadad88 
…ation (#4937)

close #4756, close #4762
This was referenced May 7, 2022
Merged
Merged
Closed
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue May 7, 2022
@CharlesCheung96
owner(ticdc): Add support for region-label to enable meta-region isol…
0509638 
…ation (pingcap#4937)

    close pingcap#4756, close pingcap#4762
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue May 7, 2022
@CharlesCheung96
owner(ticdc): Add support for region-label to enable meta-region isol…
e0be6e8 
…ation (pingcap#4937)

    close pingcap#4756, close pingcap#4762
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue May 7, 2022
@CharlesCheung96
owner(ticdc): Add support for region-label to enable meta-region isol…
bbe7c3b 
…ation (pingcap#4937)

    close pingcap#4756, close pingcap#4762
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue May 7, 2022
@CharlesCheung96
owner(ticdc): Add support for region-label to enable meta-region isol…
6981429 
…ation (pingcap#4937)

    close pingcap#4756, close pingcap#4762
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue May 7, 2022
@CharlesCheung96
owner(ticdc): Add support for region-label to enable meta-region isol…
4a07cb7 
…ation (pingcap#4937)

    close pingcap#4756, close pingcap#4762
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue May 7, 2022
@CharlesCheung96
owner(ticdc): Add support for region-label to enable meta-region isol…
5fbb966 
…ation (pingcap#4937)

    close pingcap#4756, close pingcap#4762
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue Jun 20, 2022
@CharlesCheung96
owner(ticdc): Add support for region-label to enable meta-region isol…
de66f59 
…ation (pingcap#4937)

    close pingcap#4756, close pingcap#4762
@nongfushanquan
Copy link
Contributor

/label affects-5.4

CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue Jun 21, 2022
@CharlesCheung96
owner(ticdc): Add support for region-label to enable meta-region isol…
61bb40c 
…ation (pingcap#4937)

    close pingcap#4756, close pingcap#4762
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue Jun 21, 2022
@CharlesCheung96
owner(ticdc): Add support for region-label to enable meta-region isol…
789cc8b 
…ation (pingcap#4937)

    close pingcap#4756, close pingcap#4762
ti-chi-bot added a commit that referenced this issue Jun 21, 2022
@ti-chi-bot
owner(ticdc): Add support for region-label to enable meta-region isol…
01dc212 
…ation (#4937) (#5353)

close #4756, close #4762
ti-chi-bot added a commit that referenced this issue Jun 22, 2022
@ti-chi-bot
owner(ticdc): Add support for region-label to enable meta-region isol…
7575662 
…ation (#4937) (#5354)

close #4756, close #4762
@overvenus
Copy link
Member Author

TiDB 6.2 uses a new table mysql.tidb_ddl_job for storing DDL meta data which may be merged to other regions.

@overvenus overvenus reopened this Aug 9, 2022
@overvenus overvenus mentioned this issue Aug 9, 2022
Merged
ti-chi-bot pushed a commit that referenced this issue Aug 10, 2022
@overvenus
pdutil(ticdc): split tidb_ddl_job table (#6673)
c3a1204 
close #4756
This was referenced Aug 10, 2022
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@CharlesCheung96 CharlesCheung96

Labels
affects-5.4 area/ticdc Issues or PRs related to TiCDC. component/kv-client TiKV kv log client component. component/tikv TiKV component. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
4 participants
@overvenus @CharlesCheung96 @ti-chi-bot @nongfushanquan