Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Option to Use Host Network and Configure Ports #3895

Merged
merged 22 commits into from
Jul 17, 2024

Conversation

Orza
Copy link
Contributor

@Orza Orza commented Jul 9, 2024

What type of PR is this?
/kind feature

What this PR does / Why we need it:

This PR enables the use of the host network and allows changing the port number of the Agones controller and Agones extensions.
This change is necessary for certain managed Kubernetes clusters (e.g., AWS EKS) that use custom CNI plugins (such as Calico or Cilium). These environments require this adjustment because the AWS-managed control plane cannot communicate (e.g. webhooks) with pod IP CIDRs.

Which issue(s) this PR fixes:

Closes #2750

Special notes for your reviewer:

Tested on AWS EKS with Cilium CNI plugin.

Orza added 9 commits July 4, 2024 18:31
Required for compatibility with managed Kubernetes clusters (e.g., AWS EKS) using custom CNI plugins (e.g., Calico, Cilium).
Add httpPort variable to configure HTTP server port
Add WebhookPort variable to configure webhook HTTPs server port
controller:
  added HTTP_PORT variable
  removed unused webhooks port
extensions:
  added HTTP_PORT variable
  added WEBHOOK_PORT variable
extensions-service:
  webhooks.port will use value from values file
Copy link

google-cla bot commented Jul 9, 2024

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@github-actions github-actions bot added kind/feature New features for Agones size/M labels Jul 9, 2024
@agones-bot
Copy link
Collaborator

Build Failed 😱

Build Id: 15da4a83-35e4-49a0-a11f-5ea14409bd49

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@agones-bot
Copy link
Collaborator

Build Failed 😱

Build Id: 82a48814-fd9f-417a-8aa0-7b5e65bd821f

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@markmandel
Copy link
Member

We had the same conversation on the attached ticket. I'd like to understand why EKS can't support the standard K8s webhook setup. Requiring hostNetwork for Admission Controllers and/or APIService seems like a misconfiguration / issue with EKS that should be fixed at the EKS level.

Is there any documentation stating that you have to use hostNetwork for Admission Controllers?

@agones-bot
Copy link
Collaborator

Build Failed 😱

Build Id: e0fe7e2d-b52d-4f50-81bc-3906d16d404a

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@unlightable
Copy link

unlightable commented Jul 9, 2024

I'd like to understand why EKS can't support the standard K8s webhook setup.

It's indeed an issue of cluster networking setup. But rather popular one if you want to switch from ENI (and the likes) on EKS: cilium/cilium#21959

It stems from inability of control plane to reach in-cluster pod endpoints when data plane runs in separate network layer. And inability to actually reconfigure kube-apiserver as it is an AWS managed component.
And all available solutions are rather ugly:

  • Staying on ENI is not an option as moving away from is is the purpose of rolling out different CNI.
  • Exposing though LoadBalancer/NodePort service (in place of ClusterIP) seems to not help as kubernetes API aggregation layer extracts endpoints from it and goes there directly and it still is unable to reach inside separate network layer.
  • hostNetwok is the least bad here

@markmandel
Copy link
Member

Thanks for providing the links - this is starting to make more sense.

Once question though, I'm looking at the Cillium docs (after going through the PRs to find it):

https://docs.cilium.io/en/latest/installation/k8s-install-helm/#:~:text=Deploy%20Cilium%20release%20via%20Helm%3A

It does say (emphasis mine)

The EKS API Server is unable to route packets to the overlay network. This implies that any webhook which needs to be accessed must be host networked or exposed through a service or ingress.

Are the docs wrong? Shouldn't a Service still work?

@unlightable
Copy link

unlightable commented Jul 9, 2024

Shouldn't a Service still work?

It works if you run kube-apiserver with --enable-aggregator-routing=true. Then it won't "optimize" service exposing controller pods down to endpoints. But you can't control what EKS gives you, AFAIK.

It also works in a sense if you front your service through another one in a reverse-proxy setup, and that proxy could be hostNetwork one.

There probably is a way to abuse ExternalIP services too, but it all seems much more tedious than rolling a hostNetwork agones controller. Moreso since extensions controller is split from the main one?

@unlightable
Copy link

They also refer to webhooks (e.g. MutatingWebhookConfiguration) that have an option to provide url as a target. And that url could be an external ip of loadbalanced service.

You don't have such luxury with APIService that has to reference a service inside the cluster.

@agones-bot
Copy link
Collaborator

Build Failed 😱

Build Id: f34e969d-f4ba-4f0d-81f4-de2be2aff135

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@Orza
Copy link
Contributor Author

Orza commented Jul 10, 2024

Also I can reffer to this issues:
aws/containers-roadmap#2227
cilium/cilium#30111

@agones-bot
Copy link
Collaborator

Build Failed 😱

Build Id: dbd097ce-88ff-4dd5-8534-ff621eff98ce

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@agones-bot
Copy link
Collaborator

Build Failed 😱

Build Id: 867ea4df-54d8-4ad1-ac8a-b54464243c08

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@@ -307,6 +311,7 @@ func parseEnvFlags() config {
pflag.String(logLevelFlag, viper.GetString(logLevelFlag), "Agones Log level")
pflag.Duration(allocationBatchWaitTime, viper.GetDuration(allocationBatchWaitTime), "Flag to configure the waiting period between allocations batches")
pflag.String(podNamespace, viper.GetString(podNamespace), "namespace of current pod")
pflag.String(httpPort, viper.GetString(httpPort), "Port for the HTTP server. Defaults to 8080, can also use HTTP_PORT env variable")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is for webhooks, you only need this in extensions.

@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 5ddfc0a1-59c6-4768-bc7a-15a29e46749d

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/googleforgames/agones.git pull/3895/head:pr_3895 && git checkout pr_3895
  • helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.42.0-dev-c8eacb7-amd64

@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 6de35160-fdbf-4de7-ae3a-2edf8ebc953f

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/googleforgames/agones.git pull/3895/head:pr_3895 && git checkout pr_3895
  • helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.42.0-dev-6aecb70-amd64

@Orza Orza requested a review from zmerlynn July 11, 2024 16:04
Copy link
Collaborator

@zmerlynn zmerlynn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Thanks additionally for the cleanup of some remaining parameters that were clearly wrong after the extensions / controller split.

@agones-bot
Copy link
Collaborator

Build Failed 😱

Build Id: 616591b1-f323-42c4-9fd8-e9ad3d255df5

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@agones-bot
Copy link
Collaborator

Build Failed 😱

Build Id: 2a0df716-68dc-43c3-bfc6-c9ec745afe7a

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@agones-bot
Copy link
Collaborator

Build Failed 😱

Build Id: 0f7b5e9c-3f37-4fe1-84ca-ee85a93d015f

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 6b6fc027-7c8d-400b-bf20-bbb7d98229e3

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/googleforgames/agones.git pull/3895/head:pr_3895 && git checkout pr_3895
  • helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.42.0-dev-c8d7f72-amd64

@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 9c881289-5539-4ca6-921f-4e38d964e96c

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/googleforgames/agones.git pull/3895/head:pr_3895 && git checkout pr_3895
  • helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.42.0-dev-a327a7f-amd64

@Orza Orza requested a review from zmerlynn July 16, 2024 21:10
Copy link
Collaborator

@zmerlynn zmerlynn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! This just missed the 1.42 release (which we're prepping for right now), sorry for the slow review, but it'll be in the next.

ETA: I will merge tomorrow.

@zmerlynn zmerlynn enabled auto-merge (squash) July 17, 2024 13:26
@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 6ca0fa97-4fb9-4de6-9abc-eda09976f9e9

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/googleforgames/agones.git pull/3895/head:pr_3895 && git checkout pr_3895
  • helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.43.0-dev-a36cc75-amd64

@zmerlynn zmerlynn merged commit 37aafc0 into googleforgames:main Jul 17, 2024
4 checks passed
@markmandel
Copy link
Member

Gah - I didn't get to review this in time, my fault. This PR needs docs to round this work out. I just filed #3911 , and will discuss over there.

@zmerlynn
Copy link
Collaborator

Gah - I didn't get to review this in time, my fault. This PR needs docs to round this work out. I just filed #3911 , and will discuss over there.

Shoot, sorry, you mentioned it in our internal chat and I forgot.

spiceratops added a commit to spiceratops/k8s-gitops that referenced this pull request Aug 30, 2024
This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [agones](https://agones.dev)
([source](https://redirect.github.com/googleforgames/agones)) | minor |
`1.42.0` -> `1.43.0` |

---

> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.

---

### Release Notes

<details>
<summary>googleforgames/agones (agones)</summary>

###
[`v1.43.0`](https://redirect.github.com/googleforgames/agones/releases/tag/v1.43.0)

[Compare
Source](https://redirect.github.com/googleforgames/agones/compare/v1.42.0...v1.43.0)

This is the 1.43.0 release of Agones.

In this release, we updated the supported Kubernetes version, added a
new scheduled autoscaler, as well as Unity support for Counters and
Lists..

-   **Kubernetes 1.30 Support**
With this release, the Kubernetes support matrix for Agones is now 1.28,
1.29 and 1.30.

-   **Alpha support for ScheduledAutoscaler**
This provides the ability to have Fleet autoscaler scheduling with the
feature flag `ScheduledAutoscaler`, to allow you to schedule what level
of autoscaling you would like on a schedule or between dates, giving you
more control over your autoscaling needs.

-   **Unity SDK: Counters and Lists**
We now have support for `GameServer` [Counters and
Lists](https://agones.dev/site/docs/guides/counters-and-lists/) with our
Unity SDK!

-   **New Helm Installation Configuration Options**
    Two new installation options!

We made the parameters that control how fast we scale up and down
`GameServers` configurable, so you can see how fast your Kubernetes
control plane really is when spinning up lots of `GameServer` instances.

We also have new Helm configuration options that allow the use of the
host network for the Agones controller and extensions for AWS EKS when
using Cilium.

-   **Beta support for Passthrough PortPolicy on GKE Autopilot**
We’ve graduated Passthrough Port Policy support from Alpha to Beta on
GKE Autopilot, which you can now enable using the feature flag
`AutopilotPassthroughPort`.

<!-- Release notes generated using configuration in .github/release.yml
at main -->

#### What's Changed

##### Breaking changes

- Revert "Update Supported Kubernetes to 1.28, 1.29, 1.30
([#&#8203;3933](https://redirect.github.com/googleforgames/agones/issues/3933))"
by [@&#8203;gongmax](https://redirect.github.com/gongmax) in
[googleforgames/agones#3952

##### Implemented enhancements

- Add Option to Use Host Network and Configure Ports by
[@&#8203;Orza](https://redirect.github.com/Orza) in
[googleforgames/agones#3895
- Graduate Passthrough Port Policy to Beta on Autopilot by
[@&#8203;vicentefb](https://redirect.github.com/vicentefb) in
[googleforgames/agones#3916
- Agones Unity SDK development setup instructions + Agones Unity SDK
Ready test by [@&#8203;aallbrig](https://redirect.github.com/aallbrig)
in
[googleforgames/agones#3887
- feat: Add API Changes and Validation for FleetAutoscaler
Schedule/Chain Policy by
[@&#8203;indexjoseph](https://redirect.github.com/indexjoseph) in
[googleforgames/agones#3893
- feat: Adds autoscaling logic for new Chain and Schedule policies by
[@&#8203;indexjoseph](https://redirect.github.com/indexjoseph) in
[googleforgames/agones#3929
- Adds basic framework for the in place Agones upgrades test controller
by [@&#8203;igooch](https://redirect.github.com/igooch) in
[googleforgames/agones#3956
- \[Performance] - Added a new metric inside the allocator to track the
success retry rate inside the retry loop by
[@&#8203;vicentefb](https://redirect.github.com/vicentefb) in
[googleforgames/agones#3927
- Make the parameters that limits the number of GameServers to add
configurable by
[@&#8203;vicentefb](https://redirect.github.com/vicentefb) in
[googleforgames/agones#3950
- feat: Adds e2e tests for chain/schedule policy and bump
ScheduledAutoscaler to Alpha by
[@&#8203;indexjoseph](https://redirect.github.com/indexjoseph) in
[googleforgames/agones#3946
- Implement CountsAndLists for Unity SDK + Tests by
[@&#8203;ZeroParticle](https://redirect.github.com/ZeroParticle) in
[googleforgames/agones#3883

##### Fixed bugs

- Resolves `make site-server` issue
[#&#8203;3885](https://redirect.github.com/googleforgames/agones/issues/3885)
by [@&#8203;aallbrig](https://redirect.github.com/aallbrig) in
[googleforgames/agones#3914

##### Other

- Preparation for Release v1.43.0 by
[@&#8203;kamaljeeti](https://redirect.github.com/kamaljeeti) in
[googleforgames/agones#3910
- Introduce external resource(s) on multiplayer game programming to docs
by [@&#8203;aallbrig](https://redirect.github.com/aallbrig) in
[googleforgames/agones#3884
- Added line of code to update failure count details inside runscenario
by [@&#8203;vicentefb](https://redirect.github.com/vicentefb) in
[googleforgames/agones#3915
- updated golang upgrade template by
[@&#8203;ashutosji](https://redirect.github.com/ashutosji) in
[googleforgames/agones#3902
- Changes for GitHub/Cloud Build app integration by
[@&#8203;zmerlynn](https://redirect.github.com/zmerlynn) in
[googleforgames/agones#3918
- Meta: Contributor role by
[@&#8203;markmandel](https://redirect.github.com/markmandel) in
[googleforgames/agones#3922
- Fix allocator metrics endpoint by
[@&#8203;vicentefb](https://redirect.github.com/vicentefb) in
[googleforgames/agones#3921
- Meta: Contributor => Collaborator by
[@&#8203;markmandel](https://redirect.github.com/markmandel) in
[googleforgames/agones#3928
- Rewrite agones-bot, commit to Agones repo by
[@&#8203;zmerlynn](https://redirect.github.com/zmerlynn) in
[googleforgames/agones#3923
- Small cleanup of incorrect comment in features.go file by
[@&#8203;igooch](https://redirect.github.com/igooch) in
[googleforgames/agones#3944
- Update Supported Kubernetes to 1.28, 1.29, 1.30 by
[@&#8203;ashutosji](https://redirect.github.com/ashutosji) in
[googleforgames/agones#3933
- remove ctx within the condition func by
[@&#8203;peterzhongyi](https://redirect.github.com/peterzhongyi) in
[googleforgames/agones#3959
- Reapply "Update Supported Kubernetes to 1.28, 1.29, 1.30
([#&#8203;3933](https://redirect.github.com/googleforgames/agones/issues/3933))"
([#&#8203;3](https://redirect.github.com/googleforgames/agones/issues/3)…
by [@&#8203;gongmax](https://redirect.github.com/gongmax) in
[googleforgames/agones#3961
- change kubernetes API version to fix broken CI by
[@&#8203;peterzhongyi](https://redirect.github.com/peterzhongyi) in
[googleforgames/agones#3962
- docs(godot): add Agones x Godot third party example by
[@&#8203;andresromerodev](https://redirect.github.com/andresromerodev)
in
[googleforgames/agones#3938
- Link Unity Netcode for Gameobjects example in documentation by
[@&#8203;mbychkowski](https://redirect.github.com/mbychkowski) in
[googleforgames/agones#3937
- Docs: Use k8s-api-version for links by
[@&#8203;markmandel](https://redirect.github.com/markmandel) in
[googleforgames/agones#3963

#### New Contributors

- [@&#8203;Orza](https://redirect.github.com/Orza) made their first
contribution in
[googleforgames/agones#3895

**Full Changelog**:
googleforgames/agones@v1.42.0...v1.43.0

Images available with this release:

-
[us-docker.pkg.dev/agones-images/release/agones-controller:1.43.0](https://us-docker.pkg.dev/agones-images/release/agones-controller:1.43.0)
-
[us-docker.pkg.dev/agones-images/release/agones-sdk:1.43.0](https://us-docker.pkg.dev/agones-images/release/agones-sdk:1.43.0)
-
[us-docker.pkg.dev/agones-images/release/agones-ping:1.43.0](https://us-docker.pkg.dev/agones-images/release/agones-ping:1.43.0)
-
[us-docker.pkg.dev/agones-images/release/agones-allocator:1.43.0](https://us-docker.pkg.dev/agones-images/release/agones-allocator:1.43.0)
-
[us-docker.pkg.dev/agones-images/examples/allocation-endpoint-proxy:0.9](https://us-docker.pkg.dev/agones-images/examples/allocation-endpoint-proxy:0.9)
-
[us-docker.pkg.dev/agones-images/examples/autoscaler-webhook:0.14](https://us-docker.pkg.dev/agones-images/examples/autoscaler-webhook:0.14)
-
[us-docker.pkg.dev/agones-images/examples/cpp-simple-server:0.18](https://us-docker.pkg.dev/agones-images/examples/cpp-simple-server:0.18)
-
[us-docker.pkg.dev/agones-images/examples/crd-client:0.17](https://us-docker.pkg.dev/agones-images/examples/crd-client:0.17)
-
[us-docker.pkg.dev/agones-images/examples/nodejs-simple-server:0.10](https://us-docker.pkg.dev/agones-images/examples/nodejs-simple-server:0.10)
-
[us-docker.pkg.dev/agones-images/examples/rust-simple-server:0.13](https://us-docker.pkg.dev/agones-images/examples/rust-simple-server:0.13)
-
[us-docker.pkg.dev/agones-images/examples/simple-game-server:0.34](https://us-docker.pkg.dev/agones-images/examples/simple-game-server:0.34)
-
[us-docker.pkg.dev/agones-images/examples/supertuxkart-example:0.14](https://us-docker.pkg.dev/agones-images/examples/supertuxkart-example:0.14)
-
[us-docker.pkg.dev/agones-images/examples/unity-simple-server:0.3](https://us-docker.pkg.dev/agones-images/examples/unity-simple-server:0.3)
-
[us-docker.pkg.dev/agones-images/examples/xonotic-example:2.0](https://us-docker.pkg.dev/agones-images/examples/xonotic-example:2.0)

Helm chart available with this release:

- <a href="https://agones.dev/chart/stable/agones-1.43.0.tgz"
data-proofer-ignore>
    <code>helm install agones agones/agones --version 1.43.0</code></a>

> Make sure to add our stable helm repository using `helm repo add
agones https://agones.dev/chart/stable`

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Renovate
Bot](https://redirect.github.com/renovatebot/renovate).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOC41Ny4xIiwidXBkYXRlZEluVmVyIjoiMzguNTcuMSIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsicmVub3ZhdGUvaGVsbSIsInR5cGUvbWlub3IiXX0=-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature New features for Agones size/M size/S
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Failed to create fleet
5 participants