Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use "distroless" images for linkerd proxy components #5198

Closed
ericsuhong opened this issue Nov 9, 2020 · 11 comments
Closed

Use "distroless" images for linkerd proxy components #5198

ericsuhong opened this issue Nov 9, 2020 · 11 comments

Comments

@ericsuhong
Copy link

Feature Request

What problem are you trying to solve?

Reduce container image size + Reduce unnecessary security vulnerabiltiies flagged for base debian image.

How should the problem be solved?

All linkerd controller plane components EXCEPT proxy use distroless images, and I would like to have distroless images for proxy component as well (if possible).

Any alternatives you've considered?

N/A

@olix0r
Copy link
Member

olix0r commented Nov 9, 2020

I'm generally +1 on this idea, though I believe that it may not currently be possible. In order to build a static binary (i.e. that doesn't need an OS-provided libc), we need to build against musl, but this currently fails due to our TLS dependency:

:; (cd linkerd2-proxy && cargo build --target=x86_64-unknown-linux-musl)
   ...
   Compiling ring v0.16.12
error: failed to run custom build command for `ring v0.16.12`

Caused by:
  process didn't exit successfully: `/home/ver/b/l2-proxy/target/debug/build/ring-577d2b322a8c9caa/build-script-build` (exit code: 101)
 ...
  running "musl-gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-U_FORTIFY_SOURCE" "-DNDEBUG" "-c" "-o/home/ver/b/l2-proxy/target/x86_64-unknown-linux-musl/debug/build/ring-2d3302b00adab5b7/out/aes-x86_64-elf.o" "/home/ver/.cargo/registry/src/github.mirror.nvdadr.com-1ecc6299db9ec823/ring-0.16.12/pregenerated/aes-x86_64-elf.S"

  --- stderr
  thread 'main' panicked at 'failed to execute ["musl-gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-U_FORTIFY_SOURCE" "-DNDEBUG" "-c" "-o/home/ver/b/l2-proxy/target/x86_64-unknown-linux-musl/debug/build/ring-2d3302b00adab5b7/out/aes-x86_64-elf.o" "/home/ver/.cargo/registry/src/github.mirror.nvdadr.com-1ecc6299db9ec823/ring-0.16.12/pregenerated/aes-x86_64-elf.S"]: No such file or directory (os error 2)', /home/ver/.cargo/registry/src/github.mirror.nvdadr.com-1ecc6299db9ec823/ring-0.16.12/build.rs:638:9
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed

Some folks have expressed interest in building the proxy against alternate TLS implementations, so this may become possible in the future.

@olix0r
Copy link
Member

olix0r commented Nov 9, 2020

Also it seems like there's some progress on this in *ring* (our current dependency): briansmith/ring#713

@olix0r
Copy link
Member

olix0r commented Nov 9, 2020

Actually, I think that above error is a red herring. I was missing musl-gcc from my host :hurtrealbad:.

It looks like this should be doable. We'll need to vet that this is possible for ARM builds, but seems like a good goal for 2.10!

@olix0r
Copy link
Member

olix0r commented Nov 9, 2020

OK, while it's pretty simple to modify the build to produce static binaries for x86_64, but getting cross-compilation to work for arm builds seems to be substantially more complex. Given the state of the linkerd build/release process, it's not really feasible for us to support a one-off configuration only for x86_64... I'd love help figuring out how to get the ARM builds working with musl, though!

@briansmith
Copy link
Contributor

OK, while it's pretty simple to modify the build to produce static binaries for x86_64, but getting cross-compilation to work for arm builds seems to be substantially more complex. Given the state of the linkerd build/release process, it's not really feasible for us to support a one-off configuration only for x86_64... I'd love help figuring out how to get the ARM builds working with musl, though!

As of briansmith/ring#1118 there is a documented and mostly tested way of cross-compiling for ARM targets in particular. Look at how ring's build system does it, particularly its mk/cargo.sh. It's much better than using musl-gcc because you can install clang once and that one installation will work for all targets.

@olix0r
Copy link
Member

olix0r commented Nov 24, 2020

Thanks for the helpful pointer, @briansmith.

@olix0r olix0r added this to the stable-2.11 milestone Mar 22, 2021
@olix0r
Copy link
Member

olix0r commented Mar 22, 2021

We might be able to just avoid static compilation/musl for now and use https://github.com/GoogleContainerTools/distroless to provide a base image with glib

@ericsuhong
Copy link
Author

ericsuhong commented Mar 29, 2021

Did we get any traction on this issue? We are being flagged by internal security tool for base ubuntu image vulnerability for following three containers: proxy-init, proxy, and web

@olix0r
Copy link
Member

olix0r commented Apr 1, 2021

@ericsuhong This is scheduled for stable-2.11.0. The best way to make this happen quickly is to submit a PR, though :D

@olix0r olix0r unassigned hawkw Apr 5, 2021
@olix0r olix0r added the priority/P0 Release Blocker label Apr 8, 2021
alpeb added a commit that referenced this issue Apr 9, 2021
It only required removing `RUN mkdir -p app` from the Dockerfile, which
is not really needed.

This diminishes the image size and avoids having frequent vulnerability
reports by image scanners in clusters appearing in debian. We were
upgrading the base image whenever we could, but it's an issue for older
releases.

Addresses comment in #5198
alpeb added a commit that referenced this issue Apr 9, 2021
It only required removing `RUN mkdir -p app` from the Dockerfile, which
is not really needed.

This diminishes the image size and avoids having frequent vulnerability
reports by image scanners in clusters appearing in debian. We were
upgrading the base image whenever we could, but it's an issue for older
releases.

Addresses comment in #5198
Pothulapati pushed a commit that referenced this issue Apr 14, 2021
It only required removing `RUN mkdir -p app` from the Dockerfile, which
is not really needed.

This diminishes the image size and avoids having frequent vulnerability
reports by image scanners in clusters appearing in debian. We were
upgrading the base image whenever we could, but it's an issue for older
releases.

Addresses comment in #5198
alpeb added a commit to linkerd/linkerd2-proxy-init that referenced this issue Apr 14, 2021
I couldn't get rid of the base image (in an attempt to get rid of
unrelated CVE warnings thrown by cluster image scanners) because of the
need of iptables and the shared libs it depends on, so the best I could
do was to switch to Alpine. Still, that reduces the image size in amd64
  from 77.3MB to 13MB.

I also changed the registry from docker hub to ghcr.io for the tester
images, because `k3d image import` was complaining (note these images
aren't pushed though).

Finally, it's no longer necessary to install `procps` (used to run
`sysctl`) as it already comes installed in Alpine.

This was tested successfully in the ARM host.

Partially addresses comment in linkerd/linkerd2#5198
alpeb added a commit to linkerd/linkerd2-proxy-init that referenced this issue Apr 14, 2021
I couldn't get rid of the base image (in an attempt to get rid of
unrelated CVE warnings thrown by cluster image scanners) because of the
need of iptables and the shared libs it depends on, so the best I could
do was to switch to Alpine. Still, that reduces the image size in amd64
  from 77.3MB to 13MB.

I also changed the registry from docker hub to ghcr.io for the tester
images, because `k3d image import` was complaining (note these images
aren't pushed though).

Finally, it's no longer necessary to install `procps` (used to run
`sysctl`) as it already comes installed in Alpine.

This was tested successfully in the ARM host.

Partially addresses comment in linkerd/linkerd2#5198

Signed-off-by: Alejandro Pedraza <alejandro@buoyant.io>
jijeesh pushed a commit to jijeesh/linkerd2 that referenced this issue Apr 21, 2021
It only required removing `RUN mkdir -p app` from the Dockerfile, which
is not really needed.

This diminishes the image size and avoids having frequent vulnerability
reports by image scanners in clusters appearing in debian. We were
upgrading the base image whenever we could, but it's an issue for older
releases.

Addresses comment in linkerd#5198

Signed-off-by: Jijeesh <jijeesh.ka@gmail.com>
alpeb added a commit to linkerd/linkerd2-proxy-init that referenced this issue May 6, 2021
I couldn't get rid of the base image (in an attempt to get rid of
unrelated CVE warnings thrown by cluster image scanners) because of the
need of iptables and the shared libs it depends on, so the best I could
do was to switch to Alpine. Still, that reduces the image size in amd64
  from 77.3MB to 13MB.

I also changed the registry from docker hub to ghcr.io for the tester
images, because `k3d image import` was complaining (note these images
aren't pushed though).

Finally, it's no longer necessary to install `procps` (used to run
`sysctl`) as it already comes installed in Alpine.

This was tested successfully in the ARM host.

Partially addresses comment in linkerd/linkerd2#5198

Signed-off-by: Alejandro Pedraza <alejandro@buoyant.io>
kleimkuhler pushed a commit that referenced this issue May 12, 2021
It only required removing `RUN mkdir -p app` from the Dockerfile, which
is not really needed.

This diminishes the image size and avoids having frequent vulnerability
reports by image scanners in clusters appearing in debian. We were
upgrading the base image whenever we could, but it's an issue for older
releases.

Addresses comment in #5198
kleimkuhler pushed a commit that referenced this issue May 12, 2021
It only required removing `RUN mkdir -p app` from the Dockerfile, which
is not really needed.

This diminishes the image size and avoids having frequent vulnerability
reports by image scanners in clusters appearing in debian. We were
upgrading the base image whenever we could, but it's an issue for older
releases.

Addresses comment in #5198
@olix0r
Copy link
Member

olix0r commented May 24, 2021

Related: #6165

@olix0r
Copy link
Member

olix0r commented Jul 8, 2021

Recent proxy images use distroless

@olix0r olix0r closed this as completed Jul 8, 2021
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants