archive/tar: write too long error when using Docker Desktop containerd backend

[edmorley]

Summary

When Docker Desktop for macOS's containerd feature is enabled, pack build fails with an internal archive/tar: write too long error.

The containerd backend is essential when working on multi-architecture image workflows/development, so it's quite inconvenient to have to keep turning it off when using pack build and then back on again when working on other repos/projects.

---

Reproduction

Steps

1. Install Docker Desktop for macOS
2. Go to Docker Desktop settings -> General -> and tick "Use containerd for pulling and storing images"
3. Press "apply and restart" to save the config changes
4. git clone https://github.com/heroku/heroku-buildpack-python && cd heroku-buildpack-python
5. pack build --builder heroku/builder:24 --trust-builder --verbose testapp

Current behavior

$ pack build --builder heroku/builder:24 --trust-builder --verbose testapp
Builder heroku/builder:24 is trusted
Pulling image index.docker.io/heroku/builder:24
24: Pulling from heroku/builder
Digest: sha256:9c0b7fb35dfb729cb74d5994cabed2fb8b920c0b653212e9e874c1843fa75459
Status: Image is up to date for heroku/builder:24
CheckReadAccess succeeded for the run image heroku/heroku:24
Selected run image heroku/heroku:24
Pulling image heroku/heroku:24 with platform linux/arm64
24: Pulling from heroku/heroku
Digest: sha256:307a82519fb4a485da260d668b60e0c39d8ed2f578ab5f78510d938770b4d9bf
Status: Image is up to date for heroku/heroku:24
Creating builder with the following buildpacks:
-> heroku/go@0.3.1
-> heroku/java@6.0.0
-> heroku/gradle@6.0.0
-> heroku/jvm@6.0.0
-> heroku/maven@6.0.0
-> heroku/nodejs@3.2.3
-> heroku/nodejs-corepack@3.2.3
-> heroku/nodejs-engine@3.2.3
-> heroku/nodejs-npm-engine@3.2.3
-> heroku/nodejs-npm-install@3.2.3
-> heroku/nodejs-pnpm-engine@3.2.3
-> heroku/nodejs-pnpm-install@3.2.3
-> heroku/nodejs-yarn@3.2.3
-> heroku/procfile@3.1.1
-> heroku/python@0.10.0
-> heroku/ruby@3.0.0
-> heroku/scala@6.0.0
-> heroku/jvm@6.0.0
-> heroku/sbt@6.0.0
ERROR: failed to build: failed to write image to the following tags: [pack.local/builder/676b69686d656f796d71:latest: archive/tar: write too long]

Expected behavior

The pack build succeeds, like it does when the containerd backend is not enabled.

---

Environment

pack info

$ pack report
Pack:
  Version:  0.34.0+git-b8f5fc8.build-5993
  OS/Arch:  darwin/arm64

Default Lifecycle Version:  0.19.6

Supported Platform APIs:  0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10, 0.11, 0.12, 0.13

Config:
  default-builder-image = "[REDACTED]"
  experimental = true

  [[trusted-builders]]
    name = "[REDACTED]"

  [[trusted-builders]]
    name = "[REDACTED]"

  [[trusted-builders]]
    name = "[REDACTED]"

(side note: the redacted on the trusted builders list makes that information pretty useless - perhaps it should just filter any creds in repo URLs or custom (non-docker-io) repos etc?)

docker info

$ docker info
Client:
 Version:    26.1.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0-desktop.1
    Path:     /Users/emorley/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.0-desktop.2
    Path:     /Users/emorley/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.29
    Path:     /Users/emorley/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/emorley/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.23
    Path:     /Users/emorley/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /Users/emorley/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.1.0
    Path:     /Users/emorley/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/emorley/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.8.0
    Path:     /Users/emorley/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 26.1.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.26-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 15.6GiB
 Name: docker-desktop
 ID: <REDACTED>
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/emorley/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

[natalieparellano]

I was able to reproduce this with pack 0.34.0. On pack 0.33.2, I saw a different error:

Timer: Saving testapp... started at 2024-05-30T19:31:07Z
Saving testapp...
Timer: Saving testapp... ran for 114.913189ms and ended at 2024-05-30T19:31:07Z
Timer: Exporter ran for 1.931945425s and ended at 2024-05-30T19:31:07Z
ERROR: failed to export: saving image: failed to fetch base layers: open /tmp/imgutil.local.image.2257333352/blobs/sha256/193cc52087f3baeb8473b4149731a53e049b8b1510559a4f9a5e913fb7531bf6: no such file or directory
ERROR: failed to build: executing lifecycle: failed with status code: 62

[natalieparellano]

This error appears to be coming from imgutil. I will look into it...

[natalieparellano]

buildpacks/imgutil#275 fixes archive/tar: write too long but now I'm getting open /tmp/imgutil.local.image.2257333352/blobs/sha256/193cc52087f3baeb8473b4149731a53e049b8b1510559a4f9a5e913fb7531bf6: no such file or directory and I really REALLY don't understand it 🙀

Edit: linking to Slack thread, the second error looks like a bug in the daemon relating to switching back and forth between containerd storage and overlay2.