Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable preemption and borrowing based on flavor #849

Merged
merged 28 commits into from
Oct 9, 2023
Merged

Enable preemption and borrowing based on flavor #849

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
89dbdc9
add preempt and borrow based on flavor
KunWuLuan Jun 13, 2023
0a3717c
Add some details
KunWuLuan Jun 20, 2023
b1932a3
Skip the flavors which has been considered in last schedule
KunWuLuan Jun 27, 2023
1f77f6d
update crd
KunWuLuan Jul 4, 2023
2dce8f4
fix test
KunWuLuan Jul 4, 2023
5f3a495
add some tests
KunWuLuan Jul 14, 2023
c30dc96
fix last scheduling context
KunWuLuan Jul 19, 2023
cb6c6df
add test for last scheduling context
KunWuLuan Jul 19, 2023
fd57d27
add test cases
KunWuLuan Jul 26, 2023
a7ce098
replace usage state data in lastAssignmentState with generation
KunWuLuan Aug 22, 2023
27b18a9
add featuregate
KunWuLuan Aug 22, 2023
92246e5
update helm
KunWuLuan Aug 22, 2023
f82f4b4
enable feature gate in test
KunWuLuan Aug 22, 2023
a1c33e7
update comments and tests
KunWuLuan Sep 5, 2023
0ef0c8b
update helm
KunWuLuan Sep 5, 2023
02e37ee
rebase and update test
KunWuLuan Sep 5, 2023
182270f
add unit test for generation
KunWuLuan Sep 6, 2023
53154bc
update test
KunWuLuan Sep 13, 2023
7391e1f
fix test
KunWuLuan Sep 13, 2023
e1de448
modify according to commments
KunWuLuan Sep 14, 2023
fa06cfa
update AllocatableResourceGeneration logic
KunWuLuan Sep 15, 2023
3dbd26f
set last assignment to nil when assignment was skipped
KunWuLuan Sep 19, 2023
8a51167
add comments
KunWuLuan Sep 19, 2023
832c43c
add some tests
KunWuLuan Sep 21, 2023
fb85387
add in feature table
KunWuLuan Sep 25, 2023
dc5f167
modify test
KunWuLuan Sep 28, 2023
4cdc10f
fix test
KunWuLuan Sep 28, 2023
50475bd
move FlavorResourceQuantities to cache
KunWuLuan Oct 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions apis/kueue/v1beta1/clusterqueue_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,10 @@ type ClusterQueueSpec struct {
// If set to an empty selector `{}`, then all namespaces are eligible.
NamespaceSelector *metav1.LabelSelector `json:"namespaceSelector,omitempty"`

// flavorFungibility defines whether a workload should try the next flavor
// before borrowing or preempting in the flavor being evaluated.
FlavorFungibility *FlavorFungibility `json:"flavorFungibility,omitempty"`

// preemption describes policies to preempt Workloads from this ClusterQueue
// or the ClusterQueue's cohort.
//
Expand Down Expand Up @@ -274,6 +278,40 @@ const (
PreemptionPolicyLowerOrNewerEqualPriority PreemptionPolicy = "LowerOrNewerEqualPriority"
)

type FlavorFungibilityPolicy string

const (
Borrow FlavorFungibilityPolicy = "Borrow"
Preempt FlavorFungibilityPolicy = "Preempt"
TryNextFlavor FlavorFungibilityPolicy = "TryNextFlavor"
)

// FlavorFungibility determines whether a workload should try the next flavor
// before borrowing or preempting in current flavor.
type FlavorFungibility struct {
// whenCanBorrow determines whether a workload should try the next flavor
// before borrowing in current flavor. The possible values are:
//
// - `Borrow` (default): allocate in current flavor if borrowing
// is possible.
// - `TryNextFlavor`: try next flavor even if the current
// flavor has enough resources to borrow.
//
// +kubebuilder:validation:Enum={Borrow,TryNextFlavor}
// +kubebuilder:default="Borrow"
WhenCanBorrow FlavorFungibilityPolicy `json:"whenCanBorrow,omitempty"`
// whenCanPreempt determines whether a workload should try the next flavor
// before borrowing in current flavor. The possible values are:
//
// - `Preempt`: allocate in current flavor if it's possible to preempt some workloads.
// - `TryNextFlavor` (default): try next flavor even if there are enough
// candidates for preemption in the current flavor.
//
// +kubebuilder:validation:Enum={Preempt,TryNextFlavor}
// +kubebuilder:default="TryNextFlavor"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, the existing behavior for preemption is whenCanPreempt=Preempt. So by this PR, default behavior is changed?

Is there any reason to change the default behavior? If it is possible, keeping the default behavior might be better.

@alculquicondor @KunWuLuan WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When whenCanPreempt=Preempt, it means that we will pause at each flavor where usage+request<nominal+borrowingLimit and attempt to preempt. When whenCanPreempt=tryNextFlavor, it means that we will try to preempt after traversing all flavors and when mode=Preempt. whenCanPreempt=tryNextFlavor is the current default behavior because we will traverse all flavors and break the loop until the best mode is Fit or all flavors have been .

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right. The flavorassigner traverses all flavors if the mode is Preempt.
Thank you for the clarifications, and Sorry for the confusion.

kueue/pkg/scheduler/flavorassigner/flavorassigner.go

Lines 392 to 399 in c419743

if representativeMode > bestAssignmentMode {
bestAssignment = assignments
bestAssignmentMode = representativeMode
if bestAssignmentMode == Fit {
// All the resources fit in the cohort, no need to check more flavors.
return bestAssignment, nil
}
}

WhenCanPreempt FlavorFungibilityPolicy `json:"whenCanPreempt,omitempty"`
}

// ClusterQueuePreemption contains policies to preempt Workloads from this
// ClusterQueue or the ClusterQueue's cohort.
type ClusterQueuePreemption struct {
Expand Down
20 changes: 20 additions & 0 deletions apis/kueue/v1beta1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

29 changes: 29 additions & 0 deletions charts/kueue/templates/crd/kueue.x-k8s.io_clusterqueues.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,35 @@ spec:
Validation of a cohort name is equivalent to that of object names:
subdomain in DNS (RFC 1123)."
type: string
flavorFungibility:
description: flavorFungibility defines whether a workload should try
the next flavor before borrowing or preempting in the flavor being
evaluated.
properties:
whenCanBorrow:
default: Borrow
description: "whenCanBorrow determines whether a workload should
try the next flavor before borrowing in current flavor. The
possible values are: \n - `Borrow` (default): allocate in current
flavor if borrowing is possible. - `TryNextFlavor`: try next
flavor even if the current flavor has enough resources to borrow."
enum:
- Borrow
- TryNextFlavor
type: string
whenCanPreempt:
default: TryNextFlavor
description: "whenCanPreempt determines whether a workload should
try the next flavor before borrowing in current flavor. The
possible values are: \n - `Preempt`: allocate in current flavor
if it's possible to preempt some workloads. - `TryNextFlavor`
(default): try next flavor even if there are enough candidates
for preemption in the current flavor."
enum:
- Preempt
- TryNextFlavor
type: string
type: object
namespaceSelector:
description: namespaceSelector defines which namespaces are allowed
to submit workloads to this clusterQueue. Beyond this basic support
Expand Down
9 changes: 9 additions & 0 deletions client-go/applyconfiguration/kueue/v1beta1/clusterqueuespec.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

51 changes: 51 additions & 0 deletions client-go/applyconfiguration/kueue/v1beta1/flavorfungibility.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions client-go/applyconfiguration/utils.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

29 changes: 29 additions & 0 deletions config/components/crd/bases/kueue.x-k8s.io_clusterqueues.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,35 @@ spec:
Validation of a cohort name is equivalent to that of object names:
subdomain in DNS (RFC 1123)."
type: string
flavorFungibility:
description: flavorFungibility defines whether a workload should try
the next flavor before borrowing or preempting in the flavor being
evaluated.
properties:
whenCanBorrow:
default: Borrow
description: "whenCanBorrow determines whether a workload should
try the next flavor before borrowing in current flavor. The
possible values are: \n - `Borrow` (default): allocate in current
flavor if borrowing is possible. - `TryNextFlavor`: try next
flavor even if the current flavor has enough resources to borrow."
enum:
- Borrow
- TryNextFlavor
type: string
whenCanPreempt:
default: TryNextFlavor
description: "whenCanPreempt determines whether a workload should
try the next flavor before borrowing in current flavor. The
possible values are: \n - `Preempt`: allocate in current flavor
if it's possible to preempt some workloads. - `TryNextFlavor`
(default): try next flavor even if there are enough candidates
for preemption in the current flavor."
enum:
- Preempt
- TryNextFlavor
type: string
type: object
namespaceSelector:
description: namespaceSelector defines which namespaces are allowed
to submit workloads to this clusterQueue. Beyond this basic support
Expand Down
10 changes: 5 additions & 5 deletions keps/582-preempt-based-on-flavor-order/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -246,21 +246,21 @@ func assignFlavors(log logr.Logger, requests []workload.PodSetResources, podSets
var assignment Assignment
if lastAssignment != nil {
assignment = Assignment{
TotalBorrow: make(workload.FlavorResourceQuantities),
TotalBorrow: make(cache.FlavorResourceQuantities),
PodSets: make([]PodSetAssignment, 0, len(requests)),
LastState: *lastAssignment,
Usage: make(workload.FlavorResourceQuantities),
Usage: make(cache.FlavorResourceQuantities),
}
} else {
assignment = Assignment{
TotalBorrow: make(workload.FlavorResourceQuantities),
TotalBorrow: make(cache.FlavorResourceQuantities),
PodSets: make([]PodSetAssignment, 0, len(requests)),
LastState: workload.AssigmentClusterQueueState{
LastAssignedFlavorIdx: make([]map[corev1.ResourceName]int, 0),
LastAssignedFlavorIdx: make([]map[corev1.ResourceName]int, 0, len(podSets)),
CohortGeneration: 0,
ClusterQueueGeneration: cq.Generation,
},
Usage: make(workload.FlavorResourceQuantities),
Usage: make(cache.FlavorResourceQuantities),
}
if cq.Cohort != nil {
assignment.LastState.CohortGeneration = cq.Cohort.Generation
Expand Down
Loading