*: support the TiFlash replica of table #12453

crazycs520 · 2019-09-27T13:22:35Z

What problem does this PR solve?

Add DDL syntax to support add table flash replica.

ALTER TABLE [table_name] SET TIFLASH REPLICA [count] LOCATION LABELS [location_labels]
eg:
ALTER TABLE t SET TIFLASH REPLICA 3 LOCATION LABELS "rack", "host", "abc";

Add /flash/replica http api to query the all table flash replica infos.

▶ curl "http://$IP:10080/tiflash/replica"
[
 {
  "id": 84,
  "replica_count": 3,
  "location_labels": [
   "rack",
   "host",
   "abc"
  ],
  "status": false
 }
]%

The /flash/replicaHTTP API also use to let TiFlash to report the table replica status.

# update the replica status.
▶ curl -X POST -d '{"id":84,"region_count":3,"flash_region_count":3}' http://127.0.0.1:10080/tiflash/replica

▶ curl http://127.0.0.1:10080/tiflash/replica
[
 {
  "id": 84,
  "replica_count": 3,
  "location_labels": [
   "rack",
   "host",
   "abc"
  ],
  "status": true    # the status was true mean the flash replica is available now.
 }
]%

What is changed and how it works?

The flash replica information is store in the tableInfo, So when set/update the table flash replica information, I use DDL to change the tableInfo.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
Code changes
Has exported function/method change

Side effects

Related changes

Release note

codecov · 2019-09-27T13:42:29Z

Codecov Report

Merging #12453 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #12453   +/-   ##
===========================================
  Coverage   80.2271%   80.2271%           
===========================================
  Files           468        468           
  Lines        110303     110303           
===========================================
  Hits          88493      88493           
  Misses        15085      15085           
  Partials       6725       6725

zyxbest · 2019-09-28T11:50:44Z

/run-all-tests

zyxbest · 2019-09-28T11:54:47Z

/rebuild

…_region

server/http_handler.go

ddl/ddl_api.go

…_region

server/http_handler_test.go

ddl/db_test.go

…_region

ddl/ddl_api.go

zimulala · 2019-10-29T07:36:19Z

ddl/ddl_api.go

+	if err != nil {
+		return errors.Trace(infoschema.ErrTableNotExists.GenWithStackByArgs(ident.Schema, ident.Name))
+	}
+


How about alter table t_flash set tiflash replica 2 location labels 'a','a'; ?
If labels are the same, is it OK?

It's OK. PD will get the same labels and deduplicate.

ddl/table.go

crazycs520 · 2019-10-29T08:32:02Z

/run-all-tests

bb7133

LGTM

bb7133 · 2019-10-29T08:43:30Z

PTAL @zimulala @AilinKid

ddl/ddl_api.go

AilinKid

Rest LGTM

AilinKid · 2019-10-29T09:07:10Z

server/http_handler.go

+		tables := schema.SchemaTables(db.Name)
+		for _, tbl := range tables {
+			tblInfo := tbl.Meta()
+			if tblInfo.TiFlashReplica == nil {


So TiFlashReplica is in tableInfo, what will happen if tableInfo loses this meta, for example, admin repair table will reconstruct tableInfo from create table stmt without TiFlashReplica info.

em..., Great question, I have not consider this before...

zimulala

LGTM

less LGTM

crazycs520 · 2019-10-29T12:14:54Z

/run-unit-test

AilinKid · 2019-10-29T13:05:42Z

LGTM
additional consideration：
admin repair table could not repair table info with TiFlashReplica by now, cause create table stmt don't have TiFlashReplica info. Maybe it's not good choice to add it in tableInfo, I will add todo in my PR #12046 comment, please take care of this.

test: 1. create table t(id int); 2. ALTER TABLE t SET TIFLASH REPLICA 3 LOCATION LABELS "rack", "host", "abc"; // model.ActionSetTiFlashReplica 3. curl -X POST -d '{"id":45,"region_count":3,"flash_region_count":3}' https://127.0.0.1:10080/tiflash/replica // model.ActionUpdateTiFlashReplicaStatus 4. performance write on table t and check can still replicate. ref: pingcap/tidb#12453

* Skip the tiflash job. test: 1. create table t(id int); 2. ALTER TABLE t SET TIFLASH REPLICA 3 LOCATION LABELS "rack", "host", "abc"; // model.ActionSetTiFlashReplica 3. curl -X POST -d '{"id":45,"region_count":3,"flash_region_count":3}' https://127.0.0.1:10080/tiflash/replica // model.ActionUpdateTiFlashReplicaStatus 4. performance write on table t and check can still replicate. ref: pingcap/tidb#12453 * Specify status port for tidb * Try repl SetReplicate * check len

test: 1. create table t(id int); 2. ALTER TABLE t SET TIFLASH REPLICA 3 LOCATION LABELS "rack", "host", "abc"; // model.ActionSetTiFlashReplica 3. curl -X POST -d '{"id":45,"region_count":3,"flash_region_count":3}' https://127.0.0.1:10080/tiflash/replica // model.ActionUpdateTiFlashReplicaStatus 4. performance write on table t and check can still replicate. ref: pingcap/tidb#12453

crazycs520 added 3 commits September 27, 2019 16:01

*: suport alter table set flash replica syntax

4ec146d

add http api

5ceeb6f

add http api to report replica status

e1e46cc

crazycs520 added status/DNM type/new-feature labels Sep 27, 2019

crazycs520 mentioned this pull request Sep 27, 2019

*: suport alter table set flash replica syntax pingcap/parser#564

Merged

add test

3eaa30c

crazycs520 added 2 commits September 27, 2019 21:44

fmt code

5debf92

remove check

de738e8

crazycs520 added 2 commits October 8, 2019 10:47

only store available status in table info

5f0ab30

Merge branch 'master' of https://github.com/pingcap/tidb into tiflash…

e17d7bc

…_region

lzmhhh123 self-requested a review October 11, 2019 07:47

bb7133 reviewed Oct 16, 2019

View reviewed changes

server/http_handler.go Outdated Show resolved Hide resolved

bb7133 reviewed Oct 16, 2019

View reviewed changes

ddl/ddl_api.go Outdated Show resolved Hide resolved

crazycs520 and others added 8 commits October 21, 2019 13:38

Merge branch 'master' of https://github.com/pingcap/tidb into tiflash…

a99d991

…_region

update parser and rename variable

8826872

rename

3a8376a

update parser

ca5f264

Merge branch 'master' of https://github.com/pingcap/tidb into tiflash…

0d5381b

…_region

fix typo

05d02a6

Merge branch 'master' into tiflash_region

401df18

Merge branch 'master' into tiflash_region

a82fcff

bb7133 reviewed Oct 28, 2019

View reviewed changes

server/http_handler_test.go Outdated Show resolved Hide resolved

bb7133 reviewed Oct 28, 2019

View reviewed changes

server/http_handler_test.go Outdated Show resolved Hide resolved

bb7133 reviewed Oct 28, 2019

View reviewed changes

ddl/db_test.go Show resolved Hide resolved

Merge branch 'master' of https://github.com/pingcap/tidb into tiflash…

6f93629

…_region

zimulala reviewed Oct 29, 2019

View reviewed changes

crazycs520 added 2 commits October 29, 2019 16:24

address comment

fd30dc5

address comment

ea5275b

bb7133 removed the status/DNM label Oct 29, 2019

bb7133 reviewed Oct 29, 2019

View reviewed changes

bb7133 added the status/LGT1 Indicates that a PR has LGTM 1. label Oct 29, 2019

bb7133 added the component/DDL-need-LGT3 label Oct 29, 2019

zimulala reviewed Oct 29, 2019

View reviewed changes

ddl/ddl_api.go Outdated Show resolved Hide resolved

AilinKid reviewed Oct 29, 2019

View reviewed changes

crazycs520 added 3 commits October 29, 2019 17:22

add test and fix check

d6b36d0

Merge branch 'master' into tiflash_region

50f93cb

Merge branch 'master' into tiflash_region

76a9be5

zimulala previously approved these changes Oct 29, 2019

View reviewed changes

Merge branch 'master' into tiflash_region

9f0c9d3

AilinKid added status/LGT3 The PR has already had 3 LGTM. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Oct 29, 2019

AilinKid approved these changes Oct 29, 2019

View reviewed changes

crazycs520 merged commit a8ed950 into pingcap:master Oct 29, 2019

XiaTianliang pushed a commit to XiaTianliang/tidb that referenced this pull request Dec 21, 2019

*: support the TiFlash replica of table (pingcap#12453)

b43e4a8

lzmhhh123 pushed a commit to lzmhhh123/tidb that referenced this pull request Jan 19, 2020

*: support the TiFlash replica of table (pingcap#12453)

4fc82fa

you06 added the sig/sql-infra SIG: SQL Infra label Mar 4, 2020

july2993 mentioned this pull request Apr 3, 2020

Skip the tiflash job. pingcap/tidb-binlog#941

Merged

sre-bot mentioned this pull request Apr 3, 2020

Skip the tiflash job. (#941) pingcap/tidb-binlog#942

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: support the TiFlash replica of table #12453

*: support the TiFlash replica of table #12453

crazycs520 commented Sep 27, 2019 •

edited by AilinKid

Loading

codecov bot commented Sep 27, 2019 •

edited

Loading

zyxbest commented Sep 28, 2019

zyxbest commented Sep 28, 2019

zimulala Oct 29, 2019

lzmhhh123 Oct 29, 2019

crazycs520 commented Oct 29, 2019

bb7133 left a comment

bb7133 commented Oct 29, 2019

AilinKid left a comment

AilinKid Oct 29, 2019

crazycs520 Oct 29, 2019

zimulala left a comment

crazycs520 commented Oct 29, 2019

AilinKid commented Oct 29, 2019 •

edited

Loading

*: support the TiFlash replica of table #12453

*: support the TiFlash replica of table #12453

Conversation

crazycs520 commented Sep 27, 2019 • edited by AilinKid Loading

What problem does this PR solve?

What is changed and how it works?

Check List

codecov bot commented Sep 27, 2019 • edited Loading

Codecov Report

zyxbest commented Sep 28, 2019

zyxbest commented Sep 28, 2019

zimulala Oct 29, 2019

Choose a reason for hiding this comment

lzmhhh123 Oct 29, 2019

Choose a reason for hiding this comment

crazycs520 commented Oct 29, 2019

bb7133 left a comment

Choose a reason for hiding this comment

bb7133 commented Oct 29, 2019

AilinKid left a comment

Choose a reason for hiding this comment

AilinKid Oct 29, 2019

Choose a reason for hiding this comment

crazycs520 Oct 29, 2019

Choose a reason for hiding this comment

zimulala left a comment

Choose a reason for hiding this comment

crazycs520 commented Oct 29, 2019

AilinKid commented Oct 29, 2019 • edited Loading

crazycs520 commented Sep 27, 2019 •

edited by AilinKid

Loading

codecov bot commented Sep 27, 2019 •

edited

Loading

AilinKid commented Oct 29, 2019 •

edited

Loading