Custom agent/etc/containerd/certs.d/* files are deleted by k3s on startup

[skirsten]

Environmental Info:
k3s version v1.30.4+k3s1 (98262b5)
go version go1.22.5

Node(s) CPU architecture, OS, and Version:
doesn't matter

Cluster Configuration:
doesn't matter

Describe the bug:
During generation of the containerd hosts config, the agent/etc/containerd/certs.d directory is completely deleted.
This happens here:

k3s/pkg/agent/containerd/config.go

Line 49 in fa6940d

os.RemoveAll(cfg.Containerd.Registry)

Steps To Reproduce:

• put your own files into the agent/etc/containerd/certs.d folder
• (optional) use k3s like this for a long time, upgrade k3s
• start k3s
• spend hours debugging stuff
• figure out the files are deleted
• be sad

Expected behavior:
I should be able to provide my own configs because I need full control over capabilities etc.

Actual behavior:
My own configs are deleted.

Additional context / logs:
At first glance I was thinking "why is there a os.RemoveAll in the code" but I can understand why this was implemented like this.
The writeContainerdHosts function needs to support deleting outdated files from previous configurations.

So my idea how to fix this would be instead of deleting all files, list all files and check if they start with # File generated by k3s. DO NOT EDIT.. And only delete them if they do.

What do you think?

Also, unfortunately its not possible to supply multiple config_path because then containerd won't start.

[brandond]

That does sound like a good potential solution. I think there might be some corner cases to work around, where we may drop certificates into that directory, but not create a hosts.toml - but that could probably be addressed by ensuring that we create a stub hosts.toml even if only placing certificates.

[fmoral2]

#10838 (comment)