Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

non-dist-task fast-reorg ADD INDEX can't be canceled timely #56017

Open
lance6716 opened this issue Sep 11, 2024 · 0 comments
Open

non-dist-task fast-reorg ADD INDEX can't be canceled timely #56017

lance6716 opened this issue Sep 11, 2024 · 0 comments
Labels
component/ddl This issue is related to DDL of TiDB. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 severity/major type/bug The issue is confirmed as a bug.

Comments

@lance6716
Copy link
Contributor

lance6716 commented Sep 11, 2024 •
edited
Loading

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

It has been stuck in "cancelling" for a long time, and lightning's goroutine didn't exit.

1 @ 0x474cf87 0x474fbea 0x474e93b 0x474e9c5 0x474e9c5 0x472e3cc 0x472e38b 0x472d8aa 0x472dba5 0x476b42b 0x48a8970 0x48a83d3 0x481cb25 0x481c4e9 0x481c1a5 0x492d6f0 0x492b545 0x30f90ef 0x1f6c901
#	0x474cf86	github.com/pingcap/tidb/pkg/lightning/backend/local.(*Writer).flushKVs.func1+0xc6				/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/engine.go:1355
#	0x474fbe9	slices.partitionCmpFunc[...]+0xe49										/usr/local/go/src/slices/zsortanyfunc.go:154
#	0x474e93a	slices.pdqsortCmpFunc[...]+0x49a										/usr/local/go/src/slices/zsortanyfunc.go:114
#	0x474e9c4	slices.pdqsortCmpFunc[...]+0x524										/usr/local/go/src/slices/zsortanyfunc.go:121
#	0x474e9c4	slices.pdqsortCmpFunc[...]+0x524										/usr/local/go/src/slices/zsortanyfunc.go:121
#	0x472e3cb	slices.SortFunc[...]+0x8b											/usr/local/go/src/slices/sort.go:28
#	0x472e38a	github.com/pingcap/tidb/pkg/lightning/backend/local.(*Writer).flushKVs+0x4a					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/engine.go:1355
#	0x472d8a9	github.com/pingcap/tidb/pkg/lightning/backend/local.(*Writer).appendRowsUnsorted+0x5e9				/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/engine.go:1242
#	0x472dba4	github.com/pingcap/tidb/pkg/lightning/backend/local.(*Writer).AppendRows+0x264					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/engine.go:1279
#	0x476b42a	github.com/pingcap/tidb/pkg/ddl/ingest.(*writerContext).WriteRow+0x1aa						/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/ingest/engine.go:217
#	0x48a896f	github.com/pingcap/tidb/pkg/ddl.writeOneKVToLocal+0x1ef								/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/index.go:1781
#	0x48a83d2	github.com/pingcap/tidb/pkg/ddl.writeChunkToLocal+0xb92								/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/index.go:1740
#	0x481cb24	github.com/pingcap/tidb/pkg/ddl.(*indexIngestBaseWorker).WriteChunk+0x124					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/backfilling_operators.go:874
#	0x481c4e8	github.com/pingcap/tidb/pkg/ddl.(*indexIngestBaseWorker).HandleTask+0xa8					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/backfilling_operators.go:812
#	0x481c1a4	github.com/pingcap/tidb/pkg/ddl.(*indexIngestLocalWorker).HandleTask+0x124					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/backfilling_operators.go:763
#	0x492d6ef	github.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool.(*WorkerPool[...]).handleTaskWithRecover+0x1cf	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool/workerpool.go:142
#	0x492b544	github.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool.(*WorkerPool[...]).runAWorker.func1+0x84		/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool/workerpool.go:158
#	0x30f90ee	github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1+0x4e						/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/wait_group_wrapper.go:157

1 @ 0x474cf87 0x474fdc4 0x474e93b 0x474ea38 0x474ea38 0x474ea38 0x474ea38 0x474e9c5 0x474ea38 0x472e3cc 0x472e38b 0x472d8aa 0x472dba5 0x476b42b 0x48a8970 0x48a83d3 0x481cb25 0x481c4e9 0x481c1a5 0x492d6f0 0x492b545 0x30f90ef 0x1f6c901
#	0x474cf86	github.com/pingcap/tidb/pkg/lightning/backend/local.(*Writer).flushKVs.func1+0xc6				/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/engine.go:1355
#	0x474fdc3	slices.partitionCmpFunc[...]+0x1023										/usr/local/go/src/slices/zsortanyfunc.go:157
#	0x474e93a	slices.pdqsortCmpFunc[...]+0x49a										/usr/local/go/src/slices/zsortanyfunc.go:114
#	0x474ea37	slices.pdqsortCmpFunc[...]+0x597										/usr/local/go/src/slices/zsortanyfunc.go:125
#	0x474ea37	slices.pdqsortCmpFunc[...]+0x597										/usr/local/go/src/slices/zsortanyfunc.go:125
#	0x474ea37	slices.pdqsortCmpFunc[...]+0x597										/usr/local/go/src/slices/zsortanyfunc.go:125
#	0x474ea37	slices.pdqsortCmpFunc[...]+0x597										/usr/local/go/src/slices/zsortanyfunc.go:125
#	0x474e9c4	slices.pdqsortCmpFunc[...]+0x524										/usr/local/go/src/slices/zsortanyfunc.go:121
#	0x474ea37	slices.pdqsortCmpFunc[...]+0x597										/usr/local/go/src/slices/zsortanyfunc.go:125
#	0x472e3cb	slices.SortFunc[...]+0x8b											/usr/local/go/src/slices/sort.go:28
#	0x472e38a	github.com/pingcap/tidb/pkg/lightning/backend/local.(*Writer).flushKVs+0x4a					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/engine.go:1355
#	0x472d8a9	github.com/pingcap/tidb/pkg/lightning/backend/local.(*Writer).appendRowsUnsorted+0x5e9				/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/engine.go:1242
#	0x472dba4	github.com/pingcap/tidb/pkg/lightning/backend/local.(*Writer).AppendRows+0x264					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/engine.go:1279
#	0x476b42a	github.com/pingcap/tidb/pkg/ddl/ingest.(*writerContext).WriteRow+0x1aa						/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/ingest/engine.go:217
#	0x48a896f	github.com/pingcap/tidb/pkg/ddl.writeOneKVToLocal+0x1ef								/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/index.go:1781
#	0x48a83d2	github.com/pingcap/tidb/pkg/ddl.writeChunkToLocal+0xb92								/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/index.go:1740
#	0x481cb24	github.com/pingcap/tidb/pkg/ddl.(*indexIngestBaseWorker).WriteChunk+0x124					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/backfilling_operators.go:874
#	0x481c4e8	github.com/pingcap/tidb/pkg/ddl.(*indexIngestBaseWorker).HandleTask+0xa8					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/backfilling_operators.go:812
#	0x481c1a4	github.com/pingcap/tidb/pkg/ddl.(*indexIngestLocalWorker).HandleTask+0x124					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/backfilling_operators.go:763
#	0x492d6ef	github.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool.(*WorkerPool[...]).handleTaskWithRecover+0x1cf	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool/workerpool.go:142
#	0x492b544	github.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool.(*WorkerPool[...]).runAWorker.func1+0x84		/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool/workerpool.go:158
#	0x30f90ee	github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1+0x4e						/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/wait_group_wrapper.go:157

Then it begins to generateAndSendJob. Seems the ctx is not cancelled

4. What is your TiDB version? (Required)

+-----------------------------------------------------------+
| tidb_version()                                            |
+-----------------------------------------------------------+
| Release Version: v8.5.0-alpha                             |
| Edition: Community                                        |
| Git Commit Hash: b97baba693964ef0625eb7f0b4b321d40a16b3fe |
| Git Branch: heads/refs/tags/v8.5.0-alpha                  |
| UTC Build Time: 2024-09-09 04:40:51                       |
| GoVersion: go1.21.10                                      |
| Race Enabled: false                                       |
| Check Table Before Drop: false                            |
| Store: tikv                                               |
+-----------------------------------------------------------+
@lance6716 lance6716 added the type/bug The issue is confirmed as a bug. label Sep 11, 2024
@lance6716 lance6716 changed the title fast-reorg ADD INDEX can't be canceled timely non-dist-task fast-reorg ADD INDEX can't be canceled timely Sep 11, 2024
@jebter jebter added severity/major component/ddl This issue is related to DDL of TiDB. labels Sep 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component/ddl This issue is related to DDL of TiDB. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lance6716 @jebter