WIP: make it so that journald logs are ascribed to containers #247
Conversation
When we start a container, stash a copy of the runtime spec that we generated for it in the container object. Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Make logger.Context into an object that also carries the cgroup path of the container for which it's logging, and initialize the field when we start a container. Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
|
Hmm, we don't have the "unified" controller on RHEL 7, and I suppose it's better to try to continue after failing to join the container's cgroup than to just exit and let the daemon log pipe-closed errors each time it tries to log some output. |
|
Factored the logging of error messages in the helper, and we're now joining the container's cgroup in all available controllers. |
|
RHEL system level integration testing for de52adfd9b5dc7d733f0f27e78fc1dcaca3a289e- FAIL Fedora system level integration testing for de52adfd9b5dc7d733f0f27e78fc1dcaca3a289e- FAIL |
|
RHEL system level integration testing for 927491872c8ff1b2232e9a9349d27b91bbf48b62- FAIL Fedora system level integration testing for 927491872c8ff1b2232e9a9349d27b91bbf48b62- FAIL |
|
Hmm, for some reason I'm seeing device-busy errors after exiting the container, which suggests that the log helper is keeping the container's filesystem busy, which is rather unexpected. |
|
RHEL system level integration testing for b5077e115714f5e130678da07cebe459a3bf5eee- FAIL Fedora system level integration testing for b5077e115714f5e130678da07cebe459a3bf5eee- FAIL |
|
Scratch that, I'm seeing that with json-file as well, so it's probably not a bug that's being introduced here. |
Previously, dockerd just relayed messages by itself from containers to the journal, which caused journald to apply rate limiting to messages across all containers as a single group. Here, we add another process to each container's cgroup, and have dockerd forward messages to that process over a pipe. That process, named "journal-logger", receives the messages and sends them on to the journal. As part of the container's cgroup, it's killed when the main process exits, so we only need to close the pipe and read its exit status when closing the logger. Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
|
RHEL system level integration testing for 62f4ebc42ee4bbdfc46a05750d8f515955c4e1ca- FAIL Fedora system level integration testing for 62f4ebc42ee4bbdfc46a05750d8f515955c4e1ca- FAIL |
| // to assume that we know how to compute the same path that runc is | ||
| // going to use, based on a value of the form "parent:docker:ID", where | ||
| // the "docker" is literal. | ||
| parts := strings.Split(scope, ":") |
There was a problem hiding this comment.
This will only work when cgroupdriver is set to systemd. We also need to handle cgroupfs (which is the upstream default) if we plan to try and upstream this.
There was a problem hiding this comment.
You could also look into using libcontainer cgroups library for doing this as it handles both the drivers.
There was a problem hiding this comment.
Doing the derivation right using libcontainer makes sense, since it's already a dependency. I'll have another look at that. Though that does leave open a question - how often do we see a running journald when systemd isn't managing cgroups?
|
I didn't read the code, but the concept of this change sounds good to me. |
Looking for feedback on this.
Currently, when we log container output to the journal, dockerd is doing the work of forwarding each piece of output to journald. For every piece of data that journald receives over its native socket, it looks up the unit of the process that sent that piece of data, and applies rate limiting logic to all of the data it receives from that unit. As a result, one container spewing massive numbers of log messages can effectively cause another container's output to be squelched due to rate limiting.
This PR modifies dockerd's
journaldlogging driver to add a new process to each container's scope. That process's sole job is to receive messages over a pipe and forward them to the journal. The dockerdjournaldlogging driver starts that process along with the container, feeds it each message, and stops it along with the container.In my limited testing with
journalctl -o exportafter this is applied, the journal is attributing log messages to the container's scope, sojournalctl -u docker-$CONTAINERID.scopeshould also produce the container's logs.