Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ipv6 prefix detection with aws-sdk-go-v2 #16809

Merged
merged 1 commit into from
Sep 5, 2024

Conversation

rifelpet
Copy link
Member

@rifelpet rifelpet commented Sep 5, 2024

AWS ipv6 jobs have been failing with this nodeup error:

Sep 05 03:58:28 i-002abc28507b7ab4f nodeup[4374]: I0905 03:58:28.588344    4374 executor.go:171] Continuing to run 1 task(s)
Sep 05 03:58:38 i-002abc28507b7ab4f nodeup[4374]: I0905 03:58:38.589379    4374 executor.go:113] Tasks: 182 done / 199 total; 1 can run
Sep 05 03:58:38 i-002abc28507b7ab4f nodeup[4374]: I0905 03:58:38.589427    4374 executor.go:214] Executing task "Prefix/prefix": Prefix: prefix
Sep 05 03:58:38 i-002abc28507b7ab4f nodeup[4374]: W0905 03:58:38.592417    4374 executor.go:141] error running task "Prefix/prefix" (6m1s remaining to succeed): failed to get "network/interfaces/macs/06:05:2f:0e:c6:eb/ipv6-prefix" from ec2 meta-data: operation error ec2imds: GetMetadata, http response error StatusCode: 404, request to EC2 IMDS failed
Sep 05 03:58:38 i-002abc28507b7ab4f nodeup[4374]: I0905 03:58:38.592440    4374 executor.go:171] Continuing to run 1 task(s)
Sep 05 03:58:48 i-002abc28507b7ab4f nodeup[4374]: I0905 03:58:48.593316    4374 executor.go:113] Tasks: 182 done / 199 total; 1 can run
Sep 05 03:58:48 i-002abc28507b7ab4f nodeup[4374]: I0905 03:58:48.593367    4374 executor.go:214] Executing task "Prefix/prefix": Prefix: prefix
Sep 05 03:58:48 i-002abc28507b7ab4f nodeup[4374]: W0905 03:58:48.596224    4374 executor.go:141] error running task "Prefix/prefix" (5m51s remaining to succeed): failed to get "network/interfaces/macs/06:05:2f:0e:c6:eb/ipv6-prefix" from ec2 meta-data: operation error ec2imds: GetMetadata, http response error StatusCode: 404, request to EC2 IMDS failed

This occurs in the task's Find method. We should just return nil (indicating the resource doesn't exist yet and needs to be created) but there was a bug in the error handling in aws-sdk-go-v2 and we weren't identifying the request as a 404.

The SDK's IMDS client uses different error types from normal services, so we have to unwrap the error differently:

https://github.com/aws/aws-sdk-go-v2/blob/f1d71c59a1499981873f70db8c92d07242779670/feature/ec2/imds/request_middleware.go#L185-L191

Once we confirm this fixes all of the failing ipv6 jobs then we should cherrypick this to 1.30

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. area/nodeup labels Sep 5, 2024
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 5, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 5, 2024
@rifelpet
Copy link
Member Author

rifelpet commented Sep 5, 2024

/retest

@k8s-ci-robot k8s-ci-robot merged commit 4bc07d7 into kubernetes:master Sep 5, 2024
23 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.31 milestone Sep 5, 2024
k8s-ci-robot added a commit that referenced this pull request Sep 6, 2024
…09-origin-release-1.30

Automated cherry pick of #16809: Fix ipv6 prefix detection with aws-sdk-go-v2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/nodeup cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants