tikv: make region request can send to flash store #11652

lzmhhh123 · 2019-08-07T03:33:13Z

What problem does this PR solve?

Make region request can send to flash store.

What is changed and how it works?

Add global config to reveal tiflash store labels and some flash mark in tikv.Store.

Check List

Tests

Unit test
Integration test

Code changes

Has exported function/method change
Has exported variable/fields change
Has persistent data change

Side effects

Possible performance regression
Increased code complexity

Related changes

None

codecov · 2019-08-07T04:33:32Z

Codecov Report

Merging #11652 into master will increase coverage by 0.2002%.
The diff coverage is 34.4537%.

@@               Coverage Diff                @@
##             master     #11652        +/-   ##
================================================
+ Coverage   80.9013%   81.1015%   +0.2002%     
================================================
  Files           454        454                
  Lines        100138      98405      -1733     
================================================
- Hits          81013      79808      -1205     
+ Misses        13319      12847       -472     
+ Partials       5806       5750        -56

lysu

maybe need take care #11347...

and take care fail retry logic:

kv request fail should not switch and try to flash node
flash request fail's retry should not switch to kv node(it seem current code works for this item :D)
does tiflush has some return error logic like onRegionError?

store/tikv/region_request.go

lzmhhh123 · 2019-08-08T03:37:12Z

@lysu
answer for three suggestion:

OK.
Yes, that's my fault.
RegionErr return must keep same with tikv. I'll confirm this with tiflash team.

store/tikv/region_cache.go

config/config.go

store/tikv/region_request.go

coocood · 2019-08-09T05:19:28Z

@lzmhhh123
Seems the logic conflicts with this PR
#11347

config/config_test.go

lzmhhh123 · 2019-08-09T05:30:54Z

@lysu @coocood @zz-jason Because of some reasons that the hint to access flash doesn't toward the agreement with PM team. This PR needn't merge to master now. After testing with the flash team, I will consider the situation with follower read.

lzmhhh123 · 2019-08-19T07:48:52Z

It's ready for review.

store/tikv/region_cache.go

store/tikv/region_request.go

store/tikv/region_cache.go

config/config.go

config/config_test.go

store/tikv/region_request.go

store/tikv/coprocessor.go

store/tikv/region_cache.go

…hhh123/tidb into dev/add_theflash_access_flag

lysu

at last, maybe we should take care switchNextPeer

tidb/store/tikv/region_cache.go

Line 1064 in 534613f

    
           func (c *RegionCache) switchNextPeer(r *Region, currentPeerIdx int, err error) {

this method will be called after send request failure

tidb/store/tikv/region_request.go

Line 173 in 953ee8d

if e := s.onSendFail(bo, ctx, err); e != nil {

A request to TiKV leader should not switch current work index to TiFish, if so maybe will lead write request to send to TiFlash.

and A req to TiFlash maybe better not trigger "switch current work"(e.g. one TiFlash node has crashed, but TiKV's leader is work well and keep handling oltp load).

lysu · 2019-09-24T06:12:05Z

store/tikv/region_cache.go

+		s.storeType = TiKV
+		for _, label := range store.Labels {
+			if label.Key == "engine" && label.Value == "tiflash" {
+				s.storeType = TiFlash


we can break when label.Key == "engine" even if label.Value != "tiflash"

lysu · 2019-09-24T06:12:19Z

store/tikv/region_cache.go

+	storeType := TiKV
+	for _, label := range store.Labels {
+		if label.Key == "engine" && label.Value == "tiflash" {
+			storeType = TiFlash


lysu · 2019-09-24T07:16:38Z

store/tikv/region_cache.go

+	regionStore := cachedRegion.getStore()
+
+	tikvCnt := 0
+	for i, store := range regionStore.stores {


Do we have the situation that 1 region have multiple TiFlash?

if so we should take care failure in one of TiFlash store and loadbalance between multiple TiFlash node.

for example, region1 has 3 TiFlash node: a, b, c.

we should give chance to use b and c; and send request to b and c when a is failured?

lysu

LGTM

store/tikv/region_cache.go

lysu

LGTM

sre-bot · 2019-09-25T02:55:47Z

/run-all-tests

make region request can send to flash store

2bcb332

lzmhhh123 added priority/P1 The issue has P1 priority. component/tikv component/server type/new-feature labels Aug 7, 2019

fix ci and improve code

5e8e786

lysu reviewed Aug 8, 2019

View reviewed changes

store/tikv/region_request.go Outdated Show resolved Hide resolved

lysu requested a review from coocood August 8, 2019 03:27

zz-jason reviewed Aug 8, 2019

View reviewed changes

store/tikv/region_cache.go Show resolved Hide resolved

zz-jason reviewed Aug 8, 2019

View reviewed changes

config/config.go Outdated Show resolved Hide resolved

coocood reviewed Aug 9, 2019

View reviewed changes

store/tikv/region_request.go Outdated Show resolved Hide resolved

coocood reviewed Aug 9, 2019

View reviewed changes

config/config_test.go Outdated Show resolved Hide resolved

lzmhhh123 closed this Aug 9, 2019

lzmhhh123 removed the priority/P1 The issue has P1 priority. label Aug 19, 2019

lzmhhh123 reopened this Aug 19, 2019

lzmhhh123 requested review from lysu, zz-jason, XuHuaiyu and coocood August 19, 2019 07:48

lzmhhh123 force-pushed the dev/add_theflash_access_flag branch from 67ff8b3 to f73853b Compare August 19, 2019 08:12

zz-jason reviewed Aug 19, 2019

View reviewed changes

zz-jason reviewed Aug 20, 2019

View reviewed changes

lzmhhh123 force-pushed the dev/add_theflash_access_flag branch 3 times, most recently from 82f1f8c to 598f889 Compare August 23, 2019 03:43

lzmhhh123 added 7 commits September 16, 2019 16:10

change label key engine-type to engine

2739a68

Merge branch 'master' into dev/add_theflash_access_flag

102fd85

add tiflash store check

599ea2a

improve

534613f

Merge branch 'master' into dev/add_theflash_access_flag

f2b7ab1

delete useless code

953ee8d

Merge branch 'dev/add_theflash_access_flag' of https://github.com/lzm…

91d5584

…hhh123/tidb into dev/add_theflash_access_flag

lzmhhh123 requested review from SunRunAway and removed request for SunRunAway September 23, 2019 05:16

lysu reviewed Sep 24, 2019

View reviewed changes

address comments

ac5d5d9

lysu added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Sep 24, 2019

lysu previously approved these changes Sep 24, 2019

View reviewed changes

lysu reviewed Sep 24, 2019

View reviewed changes

store/tikv/region_cache.go Outdated Show resolved Hide resolved

improve

9480b8d

lzmhhh123 dismissed lysu’s stale review via 9480b8d September 24, 2019 11:49

zz-jason reviewed Sep 24, 2019

View reviewed changes

store/tikv/region_cache.go Outdated Show resolved Hide resolved

address

980ceb3

lysu reviewed Sep 24, 2019

View reviewed changes

store/tikv/region_cache.go Show resolved Hide resolved

lysu reviewed Sep 24, 2019

View reviewed changes

store/tikv/region_cache.go Outdated Show resolved Hide resolved

make load balance for flash store

c0ea508

lzmhhh123 requested a review from lysu September 25, 2019 02:46

lysu approved these changes Sep 25, 2019

View reviewed changes

lzmhhh123 added the status/can-merge Indicates a PR has been approved by a committer. label Sep 25, 2019

Merge branch 'master' into dev/add_theflash_access_flag

5fec66a

sre-bot merged commit b25b703 into pingcap:master Sep 25, 2019

lzmhhh123 deleted the dev/add_theflash_access_flag branch September 25, 2019 03:00

lzmhhh123 added a commit to lzmhhh123/tidb that referenced this pull request Jan 19, 2020

tikv: make region request can send to flash store (pingcap#11652)

07b1b29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tikv: make region request can send to flash store #11652

tikv: make region request can send to flash store #11652

lzmhhh123 commented Aug 7, 2019

codecov bot commented Aug 7, 2019 •

edited

Loading

lysu left a comment •

edited

Loading

lzmhhh123 commented Aug 8, 2019

coocood commented Aug 9, 2019

lzmhhh123 commented Aug 9, 2019

lzmhhh123 commented Aug 19, 2019

lysu left a comment

lysu Sep 24, 2019

lysu Sep 24, 2019

lysu Sep 24, 2019 •

edited

Loading

lysu left a comment

lysu left a comment

sre-bot commented Sep 25, 2019

tikv: make region request can send to flash store #11652

tikv: make region request can send to flash store #11652

Conversation

lzmhhh123 commented Aug 7, 2019

What problem does this PR solve?

What is changed and how it works?

Check List

codecov bot commented Aug 7, 2019 • edited Loading

Codecov Report

lysu left a comment • edited Loading

Choose a reason for hiding this comment

lzmhhh123 commented Aug 8, 2019

coocood commented Aug 9, 2019

lzmhhh123 commented Aug 9, 2019

lzmhhh123 commented Aug 19, 2019

lysu left a comment

Choose a reason for hiding this comment

lysu Sep 24, 2019

Choose a reason for hiding this comment

lysu Sep 24, 2019

Choose a reason for hiding this comment

lysu Sep 24, 2019 • edited Loading

Choose a reason for hiding this comment

lysu left a comment

Choose a reason for hiding this comment

lysu left a comment

Choose a reason for hiding this comment

sre-bot commented Sep 25, 2019

codecov bot commented Aug 7, 2019 •

edited

Loading

lysu left a comment •

edited

Loading

lysu Sep 24, 2019 •

edited

Loading