-
Notifications
You must be signed in to change notification settings - Fork 3.8k
-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sql: index backfill fails to complete with default settings #130939
Comments
Hi @andrewbaptist, please add branch-* labels to identify which branch(es) this C-bug affects. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
As part of the index backfill perturbation testing, three bugs were found in the backfill process and replication. Until those are fixed, parts of the test are disabled. Informs: cockroachdb#130934 Informs: cockroachdb#130902 Informs: cockroachdb#130939 Epic: none Release note: None
Could you clarify a bit more? What does "step 2" refer to? Which version were you using when this occurred? We recently merged #128201, which is meant to address a problem similar to what you may have seen. |
This is on |
As part of the index backfill perturbation testing, three bugs were found in the backfill process and replication. Until those are fixed, parts of the test are disabled. Informs: cockroachdb#130934 Informs: cockroachdb#130902 Informs: cockroachdb#130939 Epic: none Release note: None
During index backfill, the nodes can run out of memory with the default backfill batch size. This commit reduces the size by 10x to prevent the memory issue. Informs: cockroachdb#130939 Epic: none Release note: None
After creating a cluster and attempting to add an index, the index can get stuck continually failing due to memory settings. The only way to make progress is to manually change the
bulkio.index_backfill.batch_size
setting, however that should not be required and is not clear from the UI.Create a 12 node cluster:
Attempt to create an index on the cluster
Note that this will never complete and instead get stuck on step 2.
To unstick it, set the backfill batch size (this can be run either before the index creation or while it is stuck).
The index creation will now complete successfully.
Jira issue: CRDB-42302
The text was updated successfully, but these errors were encountered: