Clarification: --provenance false fails on Buildx 0.10.0, but works with 0.10.2?

[polarathene]

I looked at the release notes for buildx but there didn't seem to be any mention that would explain this.

Recently updated a Github CI action where it was required to opt out of attestations via provenance: false. When running buildx, our CI was inconsistent with versions used. 0.10.0 failed as shown below, while 0.10.2 was successful. There does not appear to be any difference in the logs beyond buildx version. Associated issue raised for Github Action build-push-action.

Docker Engine: 20.10.22
Buildx: 0.10.0

/usr/bin/docker buildx build --cache-from type=local,src=/tmp/.buildx-cache --iidfile /tmp/docker-build-push-AubJbU/iidfile --platform linux/amd64 --provenance false --tag mailserver-testing:ci --load --metadata-file /tmp/docker-build-push-AubJbU/metadata-file .

ERROR: attestations are not supported by the current buildkitd
Error: buildx failed with: ERROR: attestations are not supported by the current buildkitd

Docker Engine: 20.10.22
Buildx: 0.10.2

/usr/bin/docker buildx build --cache-from type=local,src=/tmp/.buildx-cache --iidfile /tmp/docker-build-push-ykKorz/iidfile --platform linux/amd64 --provenance false --tag mailserver-testing:ci --load --metadata-file /tmp/docker-build-push-ykKorz/metadata-file .

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 12.34kB done
#1 DONE 0.0s

---

Presumably this is a temporary issue, and buildx 0.10.2 will be more commonly used in future CI runs (several re-runs were done but the version on failed runners did not get bumped yet).

Any hints in the meantime as to why --provenance false is causing the error above? Is the option not properly handled in 0.10.0?

Failed attempt to resolve

I noticed this PR regarding --load, that was added to 0.10.1:

Detect invalid attestation configuration #1545

Perhaps we built with attestations previously as we were using the 3.3 version of docker/build-push-action where it seems it was enabled by default. That PR suggests the feature is not compatible with buildx --load? So 0.10.0 failed, and new releases of buildx handled it properly?

If that is it, clearing our CI image cache will probably resolve the issue 👍

UPDATE: Cleared entire CI cache and performed a full workflow re-run. No difference, still getting the same failure.

[polarathene]

Related issues (same error cited):

• Default image output in buildx v0.10 cannot be pushed to Yandex Cloud Container Registry #1513 (comment)
• error: dial unix /run/buildkit/buildkitd.sock #1570

---

I have observed prior to upgrading docker/build-push-action from v3.3 to v4 (and our addition of provenance: false), the same Docker Engine and buildx versions worked fine. Just the omission of --provenance false. We can probably remove that opt-out and the CI will not encounter the error.

Next week Github should be upgrading Docker Engine to v23 which will bring a newer BuildKit version. Perhaps by then buildx 0.10.2 will be more common in runs too and this issue will be technically resolved 🤷‍♂️

---

Forgot to mention that the CI has a build workflow that uses buildx with the docker-container driver, exports the build cache to pull down in a dependent workflow, which uses buildx with the plain docker driver, hence lower BuildKit version.

We only need to --load the image into the local Docker daemon for running a container to perform testing. I assume the BuildKit version isn't too relevant at this point due to the full image cache being available, but perhaps that change in driver and BuildKit support is relevant? (but newer buildx releases are smarter / relaxed to avoid failing with that error?)

[polarathene]

I am fairly certain I understand it now 👍

• Affected workflow used docker driver instead of docker-container.
• CI Docker Engine bundles BuildKit release from 2 years ago (soon to be resolved with v23 update).
• docker/build-push-action appends --provenance option to buildx command if explicitly configured in the workflow (or compatible BuildKit version detected).
• buildx 0.10.0 would fail early on attestation support with old BuildKit, before acknowledging opt-out with --provenance false. 0.10.1 or 0.10.2 presumably fixed that mishap to only fail when --provenance true (default, but presumably only when the BuildKit version supported it, hence no failure prior).

So, the only thing that changed was explicitly opting out with provenance: false in the action, added an explicit --provenance false option to buildx, mixed with bad timing of Github dependencies and the one workflow not using docker-container driver for newer BuildKit version 😅 (all while about to become a non-issue with Github as the CI in about a week)

Closing as I'm fairly confident in the troubleshooting and resolution 😎