Tracking issue for reducing high latency

[overvenus]

The status quo and the goal of the tracking issue.

Scenario	Status quo (2022/03/02)	Goal
Planned outage
TiCDC rolling upgrade	2-8 mins	<= 10s
TiKV/TiDB rolling upgrade	<= 1min	<= 30s
PD rolling upgrade	<= 1min	<= 1min
TiKV scale out/off	<= 1min	<= 30s
TiCDC scale out/off	2-8 mins	<= 5s
PD scale out/off	<= 1min	<= 1min
TiDB scale out/off	<= 1min	<= 30s
Unplaned outage (less than ⅓ total node number, outage last 5mins)
TiKV/TiDB outage (power down, disk failure etc)	2-4 mins	<= 2min
TiCDC  outage (power down, disk failure etc)	2-8 mins	2-8 mins
PD  outage (power down, disk failure etc)	2-8 mins	2-8 mins
No outage
Delay	<= 10s	<= 2s
Delay spikes (99%)	<= 1 min	<= 5s

Extreme high latency (>30 min)

• TiCDC take long time (may be a day) to recover from TiKV cluster failover #4516

Latency spike

• Meta region merge may cause latency spike #4756
• TiCDC reconnects TiKV blindly when region leader transfers #4761
• Meta region leader transfer may cause latency spike #4762
• TiCDC cluster suffers a round robin owner election during rolling update #3529
• region leaders can be banished in an election timeout when rolling upgrade a cluster tikv/tikv#12166
• reduce owner and processor flush interval to 50ms #7277

Two-phase scheduling

Two-phase scheduling aims to solve high latency spike (up to minutes) caused by moving table (move a table from a TiCDC to another).

• Cleanup scheduler v1 code *(ticdc): remove scheduler v1 #5362
• Cleanup schedulerv2 interfaces in owner and processor cdc: move scheduler and agent to package base  #5396
• Sketch two-phase scheduler (task based scheduling, move table task, add table task, remove table task) cdc: implement replication set #5450
• Implement two-phase replication_set state transition cdc: implement replication set #5450
• Implement replicationManager tp: implement capture manager #5562
• Implement two-phase scheduler coordinator cdc: integrate two phase scheduler #5710
• Implement new APIs of two-phase in table executor an table pipeline scheduler(2pc): agent for 2 phase scheduling #5593
• Implement two-phase processor agent scheduler(2pc): agent for 2 phase scheduling #5593
• Implement two-phase balance scheduler tp: support basic balance tables #5676
• Implement two-phase API scheduler (move table and rebalance) tp(cdc): support manual move table and rebalance #5711
• Checkpoint management refactor (per table based) tp: support per-table checkpoint #5709
• Support rebalance when adding a new TiCDC node tp: support balance when add a new capture #5760
• Support drain capture API fb/latency(cdc): 2 phase scheduling drain capture #5852
• Add metrics tp: add metrics #5823
• tp: add info provider #5844
• metrics(cdc): add scheduler metrics #6196

Graceful shutdown and upgrade

• cdc,pkg: accommodate systemd TimeoutStopSec and k8s terminationGracePeriodSeconds #6097
• Resign ownership during graceful shutdown cdc: drain waits for owner resign and table move out #6110
• Move out all tables during graceful shutdown cdc: drain waits for owner resign and table move out #6110
• cdc rolling upgrade / scale-in ultilize two-phase-scheduling tiup#1972
• Support graceful shutdown TiCDC node tidb-operator#4623
• ticdc: support graceful upgrade TiCDC pods tidb-operator#4647

Cross version grace upgrade

• relax version consistency check to support graceful upgrade #6450

Reduce p99 latency in normal cases

• cdc: collect slow table latency metrics #7323
• cdc: add slow table Grafana panels #7348
• Refactor owner and processor to event-driven processing #7368
• cdc: observe the slowest resolved ts lag tikv/tikv#13647
• Reuse map and set in check leader to save CPU tikv/tikv#13656
• Reduce memory allocation in check leader to save CPU tikv/tikv#13665
• Run CheckLeader tasks in a dedicate thread tikv/tikv#13774

[overvenus]

With the default settings in v6.5.0, changefeed replication lag is less 2s for both normal scenario and planned rolling restart/upgrade. Except the large table scenario, I think we have met the requirement described in this issue.
Let's track the large table scenario in #7720.