Skip to content

Latest commit

 

History

History
81 lines (55 loc) · 3.13 KB

kubernetes_events.md

File metadata and controls

81 lines (55 loc) · 3.13 KB
Raw

AWS Node Termination Handler Kubernetes events

AWS Node Termination Handler has the ability to emit a Kubernetes event every time an interruption signal is sent from AWS and also every time an operation is attempted on a node. More information on how to get events can be found here.

Configuration

There are two relevant parameters:

  • emit-kubernetes-events:

    If true, Kubernetes events will be emitted when interruption events are received and when actions are taken on Kubernetes nodes. Defaults to false.

  • kubernetes-events-extra-annotations:

    A comma-separated list of key=value extra annotations to attach to all emitted Kubernetes events. Example:

    "first=annotation,sample.annotation/number=two"

Event reasons

There are a number of events that can be emitted, each one with a reason that can be used to quickly identify the event nature and for filtering. Each event will also have a message with extended information. Here's a reasons summary:

AWS interruption event reasons:

  • RebalanceRecommendation
  • ScheduledEvent
  • SQSTermination
  • SpotInterruption

Node action reasons:

  • Cordon
  • CordonError
  • CordonAndDrain
  • CordonAndDrainError
  • PreDrain
  • PreDrainError
  • PostDrain
  • PostDrainError
  • Uncordon
  • UncordonError
  • MonitorError

Default IMDS mode annotations

If emit-kubernetes-events is enabled and enable-sqs-termination-draining is disabled (meaning we're operating in IMDS mode), AWS Node Termination Handler will automatically inject a set of annotations to each event it emits. Such annotations are gathered from the underlying host's IMDS endpoint and enrich each event with information about the host that emitted it.

NOTE: In Queue Processor mode, the default IMDS mode annotations will be disabled but you can still define a set of extra annotations.

The default IMDS mode annotations are:

Name Example value
account-id 123456789012
availability-zone us-west-2a
instance-id i-abcdef12345678901
instance-life-cycle spot
instance-type m5.8xlarge
local-hostname ip-10-1-2-3.us-west-2.compute.internal
local-ipv4 10.1.2.3
public-hostname my-example.host.net
public-ipv4 42.42.42.42
region us-west-2

If kubernetes-events-extra-annotations are specified they will be appended to the above. In case of collision, the user-defined annotation wins.

How to get events

All events are about Kubernetes Node objects so they belong in the default namespace. The event source is aws-node-termination-handler. From command line, use kubectl to get the events as follows:

kubectl get events --field-selector "source=aws-node-termination-handler"

To narrow down the search you can use multiple field selectors, like:

kubectl get events --field-selector "reason=SpotInterruption,involvedObject.name=ip-10-1-2-3.us-west-2.compute.internal"

Results can also be printed out in JSON or YAML format and piped to processors like jq or yq. Then, the above annotations can also be used for discovery and filtering.