Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lightning might oom when mem/cpu ratio is low #43728

Closed
Tracked by #43740
D3Hunter opened this issue May 11, 2023 · 1 comment · Fixed by #43729
Closed
Tracked by #43740

lightning might oom when mem/cpu ratio is low #43728

D3Hunter opened this issue May 11, 2023 · 1 comment · Fixed by #43729
Labels
affects-6.1 affects-6.5 affects-7.1 component/lightning This issue is related to Lightning of TiDB. severity/major type/bug The issue is confirmed as a bug.

Comments

@D3Hunter
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  • each row of the target table is large, say 20k, and there's many int values cannot be stored correctly, so cause ErrDataOutOfRange warning when encoding
  • lightning might oom when memory is not that enough during sorting, say cpu/mem is 1:2, and we're using default region-concurrency

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

current master
@D3Hunter D3Hunter added type/bug The issue is confirmed as a bug. component/lightning This issue is related to Lightning of TiDB. affects-6.1 affects-6.5 affects-7.1 labels May 11, 2023
@D3Hunter D3Hunter mentioned this issue May 11, 2023
Merged
12 tasks
@D3Hunter D3Hunter changed the title lightning might oom when cpu/mem ratio is low lightning might oom when mem/cpu ratio is low May 11, 2023
@ti-chi-bot ti-chi-bot bot added may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. labels May 11, 2023
@D3Hunter D3Hunter removed may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. labels May 11, 2023
@D3Hunter
Copy link
Contributor Author

will not pick to older versions, user can use newer version directly

@lance6716 lance6716 mentioned this issue May 12, 2023
Open
16 tasks
ti-chi-bot bot pushed a commit that referenced this issue May 12, 2023
@D3Hunter
lightning: fix oom when mem/cpu ratio is low (#43729)
56dacd0 
close #43728
This was referenced May 12, 2023
Merged
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue May 12, 2023
@ti-chi-bot
lightning: fix oom when mem/cpu ratio is low (#43729) (#43748)
fe089de 
close #43728
ti-chi-bot bot pushed a commit that referenced this issue May 12, 2023
@ti-chi-bot
lightning: fix oom when mem/cpu ratio is low (#43729) (#43747)
d50dc76 
close #43728
ti-chi-bot bot pushed a commit that referenced this issue May 12, 2023
@ti-chi-bot
lightning: fix oom when mem/cpu ratio is low (#43729) (#43746)
b00cc84 
close #43728
@D3Hunter D3Hunter mentioned this issue May 15, 2023
Closed
@GMHDBJD GMHDBJD mentioned this issue Jun 25, 2023
Merged
16 tasks
@niubell niubell mentioned this issue Jul 6, 2023
Merged
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-6.1 affects-6.5 affects-7.1 component/lightning This issue is related to Lightning of TiDB. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

lightning: fix oom when mem/cpu ratio is low pingcap/tidb
2 participants
@lance6716 @D3Hunter