Fix the scheduler panic whenever the GPU is lost on node #306
Conversation
|
Welcome @william-wang! |
volcano-sh-bot
added
the
size/M
Travis tests have failedHey @william-wang, TravisBuddy Request Identifier: 3cb72fa0-a173-11e9-b4a0-0f4450928f2b |
There was a problem hiding this comment.
@william-wang Could you add more comments why this can cause a panic?
| ni.Releasing.Add(ti.Resreq) | ||
| ni.Idle.Sub(ti.Resreq) | ||
| case Pipelined: | ||
| ni.Releasing.Sub(ti.Resreq) |
There was a problem hiding this comment.
Does this face the same problem?
There was a problem hiding this comment.
Yes, So I deleted this line and replace it to line 191-193.
There was a problem hiding this comment.
@william-wang Could you add more comments why this can cause a panic?
In customer's environment, when the GPU was lost. the ni.Idle decreased and the value was less than ti.Resreq. Then the panic happened in Sub func. So we need to add a judgement before the execution of ni.Idle.Sub(ti.Resreq).
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: k82cn, william-wang The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
volcano-sh-bot
added
the
approved
|
@asifdxtreme , please help to cherry-pick this one to kube-batch :) |
Enable LeaderElect for both scheduler and controller
|
@william-wang Can you please take out some time and sign cncf-cla https://github.com/kubernetes/community/blob/master/CLA.md#the-contributor-license-agreement , this PR's needs to be cherry picked to kube-batch and kube-batch has mandatory check for cncf-cla |
|
@asifdxtreme , I think we also need cncf-cla check for volcano :) |
okay we will add it |
For CNCF Cla check we need webhook secret from CNCF, we contacted the admins to get the secret but they replied back saying they cannot enable webhook for external repo's as of now. |
What this PR does / why we need it:
The volcano-sh/scheduler panics when startup whever the GPU is lost on nodes.
The cause is that the GPU resource descreaced, the node.idle <task.Req. The panic happened
when ni.Idle.Sub(ti.Resreq). This PR is used to fix the issue.
Which issue(s) this PR fixes (optional, in fixes #(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #294
Special notes for your reviewer: