Enable using docker build `--platform` switch (easily) #4388

richlander · 2023-02-06T01:12:34Z

There is no way to write a Dockerfile for .NET that is (A) succinct, (B) easy to build from the command line, (C) can equally produce Arm64 or x64 images, (D) that will run equally well on both Arm64 and x64 machines, and (E) avoids running the SDK in an emulator (since .NET doesn't support running in QEMU).

The following Dockerfile (which doesn't currently work) would satisfy all four of these requirements.

ARG BUILDARCH
FROM mcr.microsoft.com/dotnet/sdk:7.0-bullseye-slim-$BUILDARCH AS build
ARG TARGETARCH
WORKDIR /source

# copy csproj and restore as distinct layers
COPY aspnetapp/*.csproj .
RUN dotnet restore -r linux-$TARGETARCH

# copy everything else and build app
COPY aspnetapp/. .
RUN dotnet publish -c Release -r linux-$TARGETARCH --self-contained false --no-restore -o /app

# final stage/image
FROM mcr.microsoft.com/dotnet/aspnet:7.0-bullseye-slim
WORKDIR /app
COPY --from=build /app .
ENTRYPOINT ["./aspnetapp"]

Note: The two ENVs are set via Docker, not my invention. Context: #4387 (comment). The reason this Dockerfile doesn't work is because -r expects x64 not amd64 (which $TARGETARCH returns) and our container tags expect arm64v8 not arm64 (which $BUILDARCH returns). Funny enough, $TARGETARCH returns arm64 for Arm64, which -r likes fine and $BUILDARCH returns amd64 for x64, which our container tags like fine.

Note: Close readers may wonder why $TARGETARCH is not needed for the last FROM statement. That's because --platform is picking the correct image from the multi-arch tag. We could use $TARGETARCH if we wanted, but it is unnecessary since the underlying mechanics will do the right thing. How would one know that the tag is a multi-arch tag just by looking at it? It's because it doesn't include an architecture.

It would enable the following scenarios:

docker build -- will produce a native-arch result on both Arm64 and x64 and use the native arch SDK.
docker build --platform linux/arm64 -- Will produce an Arm64 container image on both Arm64 and x64 machines but run the SDK as native-arch, avoiding emulation (on x64 machines).
docker build --platform linux/amd64 -- Same as above, but will produce an x64 container images, and similarly avoid emulation (on Arm64 machines).
docker buildx build --platform linux/amd64,linux/arm64 -- Same as above, but will produce both Arm64 and x64 container images, and similarly avoid emulation.

Note that the resulting image will not run in emulation. That's the not the purpose of this proposal. Instead, the purpose is to reliably avoid emulation and to make it easy, for example, to build x64 container images on an Apple M1 box and push those to an x64 cloud. You'd be able to do that via the --platform switch and not need to do anything else (other than follow the pattern used in the Dockerfile).

It requires two things:

For Arm64, we need to push container tags as arm64 (matching $BUILDARCH) in addition to arm64v8.
For x64, teaching the CLI to replace linux-amd64 (matching $TARGETARCH) with linux-x64

The proposal is to make these changes for .NET 6+.

The alternative is the following.

ARG SDKARCH=amd64
FROM mcr.microsoft.com/dotnet/sdk:7.0-bullseye-slim-$SDKARCH AS build

ARG TARGETARCH
RUN arch=$TARGETARCH \
    && if [ "$TARGETARCH" = "amd64" ]; then arch="x64"; fi \
    && echo $arch > /tmp/arch

WORKDIR /source

# copy csproj and restore as distinct layers
COPY aspnetapp/*.csproj .
RUN dotnet restore -r linux-$(cat /tmp/arch)

# copy everything else and build app
COPY aspnetapp/. .
RUN dotnet publish -r linux-$(cat /tmp/arch) -c Release --self-contained false --no-restore -o /app

# final stage/image
FROM mcr.microsoft.com/dotnet/aspnet:7.0-bullseye-slim
WORKDIR /app
COPY --from=build /app .
ENTRYPOINT ["./aspnetapp"]

Inspiration: #4387 (comment)

SDKARCH needs to have a default, so you need to know to set the ARG on whatever platform isn't the default. Here, I'm choosing x64 (or really "amd64") as the default. Also, you cannot run any code before the first FROM since a Dockerfile isn't bash.

This pattern enables the following scenarios:

docker build -- only works correctly on x64 and breaks on Arm64 (dotnet restore just hangs),
docker build --build-arg SDKARCH=arm64v8 -- builds an Arm64 image on an Arm64 machine, not because the ARG is set, however. The ARG just enables the SDK to run with the native arch.
docker build --platform linux/arm64 -- Doesn't do anything useful, same for specifying linux/amd64.
docker build buildx --platform linux/amd64,linux/arm64 -- Doesn't work.
docker build buildx --platform linux/amd64,linux/arm64 --build-arg SDKARCH=arm64v8 -- Produces both Arm64 and x64 container images.

The text was updated successfully, but these errors were encountered:

richlander · 2023-02-06T01:14:49Z

Relates to:

@nagilson @baronfel @marcpopMSFT @elinor-fung

mthalman · 2023-02-06T19:38:54Z

[Triage]
Short-term we need to update our customer guidance for this scenario since the current guidance is incompatible with the functionality of .NET 7. The alternative Dockerfile that was provided above with the SDKARCH is probably the best approach identified at this point for a workable solution that's possible today.

Long-term we need to come up with a design that can work. Ideally it would be great if we could eliminate the need for a set of variant-less tags (e.g. arm64). There does exist a $TARGETVARIANT variable that may help with this. But it seems to only be set for Arm32 scenarios. Then we'd need to get buyoff from the CLI for implementing some kind of RID aliasing between x64 and amd64.

This also affects the scaffolded Dockerfiles produced by the VS Container Tools. They are generating the old "multi-arch" Dockerfile that no longer works with .NET 7 when targeting an architecture that differs from the host.

MichaelSimons · 2023-02-07T20:28:23Z

@richlander, In your proposed Dockerfile

ARG BUILDARCH
FROM mcr.microsoft.com/dotnet/sdk:7.0-bullseye-slim-$BUILDARCH AS build
ARG TARGETARCH
WORKDIR /source

# copy csproj and restore as distinct layers
COPY aspnetapp/*.csproj .
RUN dotnet restore -r linux-$TARGETARCH

# copy everything else and build app
COPY aspnetapp/. .
RUN dotnet publish -c Release -r linux-$TARGETARCH --self-contained false --no-restore -o /app

# final stage/image
FROM mcr.microsoft.com/dotnet/aspnet:7.0-bullseye-slim
WORKDIR /app
COPY --from=build /app .
ENTRYPOINT ["./aspnetapp"]

Have you considered utilizing the --platform option within the FROM statement? This feels like it could simplify the Dockerfile, work with the existing Dockerfile constructs, and not require any tagging derivations from the norm.

FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:7.0-bullseye-slim AS build
ARG TARGETARCH
WORKDIR /source

# copy csproj and restore as distinct layers
COPY aspnetapp/*.csproj .
RUN dotnet restore -r linux-$TARGETARCH

# copy everything else and build app
COPY aspnetapp/. .
RUN dotnet publish -c Release -r linux-$TARGETARCH --self-contained false --no-restore -o /app

# final stage/image
FROM mcr.microsoft.com/dotnet/aspnet:7.0-bullseye-slim
WORKDIR /app
COPY --from=build /app .
ENTRYPOINT ["./aspnetapp"]

richlander · 2023-02-07T22:15:15Z

That looks awesome! Ship it! I didn't know about that pattern.

That means we just need to fix the RID problem.

richlander · 2023-02-17T17:53:10Z

Just wonder about this and Native AOT. Will this pattern not work for that? I think it doesn't have a cross-arch cross-build story. Is that right @jkotas?

jkotas · 2023-02-17T18:27:33Z

NativeAOT has cross-arch cross-build story. It is documented here: https://github.com/dotnet/runtime/blob/main/src/coreclr/nativeaot/docs/compiling.md#cross-architecture-compilation . For Linux, it requires rootfs matching the target platform.

jkotas · 2023-02-17T18:33:53Z

If it helps, https://github.com/dotnet/samples/tree/main/core/nativeaot/HelloWorld is nativeaot sample that includes Linux x64 and Windows x64 docker files for building it (the sample does not have cross-arch build support).

mthalman · 2023-02-21T15:34:15Z

Issue for NativeAOT scenario investigation is at #4129

nagilson · 2023-03-06T18:52:57Z

That means we just need to fix the RID problem.

The RID fix is in, thanks to a new hire on our team @JL03-Yue who worked on it. 😄 Is there anything else required here @richlander ?

richlander · 2023-03-07T23:40:14Z

It works great! I tested it with dotnetapp and the following Dockerfile.

To recap, we want to be able to build Arm64 and x64 assets on an Apple Arm64 machine safetly and correctly. We now can (with .NET 8 Preview 3). We are going to look at backporting the change to .NET 7 as well.

The Dockerfile references a multi-platform tag that references Amd64, Arm64, and Arm32 images that include a .NET 8 Preview 3 SDK.

Note: This test repo will be deleted before long, so don't be surprised if you read this later and the sample doesn't work. You'll need to switch to the real .NET 8 SDK image. It's called "dotnetnonroot" since I was using for testing another feature.

# To learn about building .NET container images:
# https://github.com/dotnet/dotnet-docker/blob/main/samples/README.md
FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/nightly/sdk:8.0-preview AS build
ARG TARGETARCH
WORKDIR /source

# copy csproj and restore as distinct layers
COPY *.csproj .
RUN dotnet restore -a $TARGETARCH

# copy and publish app and libraries
COPY . .
RUN dotnet publish -a $TARGETARCH --self-contained false --no-restore -o /app

# To enable globalization:
# https://github.com/dotnet/dotnet-docker/blob/main/samples/enable-globalization.md
# final stage/image
FROM mcr.microsoft.com/dotnet/runtime:7.0-jammy
WORKDIR /app
COPY --from=build /app .
ENTRYPOINT ["./dotnetapp"]

Some quick tests on my Apple M1 machine.

% docker build -t dotnetapp .
% docker inspect dotnetapp -f "{{.Os}}\{{.Architecture}}"
linux\arm64
% docker run --rm dotnetapp | grep Arch        
OSArchitecture: Arm64
% docker build -t dotnetapp --platform linux/arm64 .
% docker inspect dotnetapp -f "{{.Os}}\{{.Architecture}}"
linux\arm64
% docker run --rm dotnetapp | grep Arch
OSArchitecture: Arm64
% docker build -t dotnetapp --platform linux/amd64  .
% docker inspect dotnetapp -f "{{.Os}}\{{.Architecture}}"
linux\amd64
% docker build -t dotnetapp --platform linux/arm .
% docker inspect dotnetapp -f "{{.Os}}\{{.Architecture}}"
linux\arm

I didn't run the generated Amd64 and Arm32 images on my Arm64 machine since they don't work in that environment. The key point is that you can build images for all the architectures with one Dockerfile using the --platform switch to control the outcome. The native arch is always the default.

I tested the same scenarios and they worked on my x64 machine as well.

We can have even more fun with docker buildx.

% docker buildx build -f Dockerfile.ubuntu --platform linux/amd64,linux/arm64,linux/arm -t dotnetnonroot.azurecr.io/dotnetapp --push .

On my Apple M1 machine:

% docker run --rm -it dotnetnonroot.azurecr.io/dotnetapp
         42                                                    
         42              ,d                             ,d     
         42              42                             42     
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM  
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42     
8b       42 8b       d8  42    42       42 8PP!!!!!!!   42     
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,    
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428  

.NET 7.0.3
Ubuntu 22.04.2 LTS

UserName: root
OSArchitecture: Arm64
ProcessorCount: 4
TotalAvailableMemoryBytes: 4124512256 (3.00 GiB)

On my x64 machine:

# docker run --rm -it dotnetnonroot.azurecr.io/dotnetapp
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP!!!!!!!   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 7.0.3
Ubuntu 22.04.2 LTS

UserName: root
OSArchitecture: X64
ProcessorCount: 12
TotalAvailableMemoryBytes: 33444470784 (31.00 GiB)
cgroup memory constraint: /sys/fs/cgroup/memory/memory.limit_in_bytes
cgroup memory limit: 9223372036854771712 (8589934591.00 GiB)
cgroup memory usage: 6713344 (6.00 MiB)
GC Hard limit %: 0

On my Arm64 Raspberry Pi:

 $ docker run --rm -it dotnetnonroot.azurecr.io/dotnetapp
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP!!!!!!!   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 7.0.3
Ubuntu 22.04.2 LTS

UserName: root
OSArchitecture: Arm64
ProcessorCount: 4
TotalAvailableMemoryBytes: 3978678272 (3.00 GiB)

richlander · 2023-03-15T23:08:50Z

This experience will ship in:

8.0 P3
7.0.300

richlander · 2023-03-15T23:54:32Z

If you want this experience now, you can use: mcr.microsoft.com/dotnet/nightly/sdk:8.0-preview

goncalo-oliveira · 2023-03-16T10:02:43Z

If you want this experience now, you can use: mcr.microsoft.com/dotnet/nightly/sdk:8.0-preview

Excellent news @richlander! Thanks for this. I've tried this on my M1 and on a x64 machine and was able to successfully build amd64 and arm64 with buildx on both.

docker buildx build --platform linux/amd64,linux/arm64 -t image:tag . --no-cache

campbellwray · 2023-03-16T20:37:07Z

@richlander I seem to be having some issues with the RUN dotnet lines in the Dockerfile you provided

The command I am using is similar to @goncalo-oliveira's docker buildx one but I suspect their Dockerfile might have omitted the -a $TARGETARCH switches.

dotnet restore doesn't appear to have an -a flag
dotnet publish appears to be expecting linux-x64 but instead it is getting linux-amd64 - Linux RIDs

Thanks for looking into this

richlander · 2023-03-17T00:22:50Z

Both your observations are 100% true. You've also identified what we fixed in 8.0 P3 and (in future) 7.0.300.

I updated the Dockerfile example to use a 8.0 P3 nightly build instead of the hacked build I was using before. That should work better. I just tested it. Please tell me if it doesn't.

campbellwray · 2023-03-17T03:15:48Z

Ahh apologies, I was unintentionally using mcr.microsoft.com/dotnet/nightly/aspnet:8.0-preview-bullseye-slim, I didn't realise that there was no bullseye version of P3 available, I must have been downloading a much older build.

Unfortunately even with this resolved I was still unable to compile my project, it is aspnet and not runtime, and it is complaining that version 8.0 P3 is not available for the Microsoft.NETCore.App.Host.linux-arm64 package on NuGet.

error NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-arm64 with version (= 8.0.0-preview.3.23165.10)
error NU1102:   - Found 117 version(s) in nuget.org [ Nearest version: 8.0.0-preview.2.23128.3 ]

I supposed that means that I will need to wait for either 8.0 P3 or 7.0.300 to be publicly released

Thanks again for all of your work on this @richlander, it is much appreciated

ptr727 · 2023-04-08T19:48:43Z

[I updated my original question as I figured out why it failed.]

In case it helps somebody else, it was not obvious to me that the builder and runtime platforms have to be the same platform kind, else the runtime library dependencies will not match.

I built Alpine using FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/nightly/sdk:8.0-preview-alpine AS builder

And dotnet publish told me it is using that it is using linux-musl-x64:

#12 [builder 4/5] RUN dotnet publish ./PlexCleaner.csproj     --arch amd64     --self-contained false     --output ./Publish     --configuration Release     -property:Version=1.0.0.0     -property:FileVersion=1.0.0.0     -property:AssemblyVersion=1.0.0.0     -property:InformationalVersion=1.0.0.0     -property:PackageVersion=1.0.0.0
#12 9.187   PlexCleaner -> /Builder/bin/Release/net7.0/linux-musl-x64/PlexCleaner.dll
#12 9.216   PlexCleaner -> /Builder/Publish/
#12 DONE 9.4s

When my deployment layer was built using FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:7.0, which is Debian, and I got a No such file or directory error on launching the binary.

The correct build should be linux-x64:

#15 [builder 4/5] RUN dotnet publish ./PlexCleaner/PlexCleaner.csproj     --arch amd64     --self-contained false     --output ./Publish     --configuration Release     -property:Version=1.0.0.0     -property:FileVersion=1.0.0.0     -property:AssemblyVersion=1.0.0.0     -property:InformationalVersion=1.0.0.0     -property:PackageVersion=1.0.0.0
#15 5.940   Restored /Builder/PlexCleaner/PlexCleaner.csproj (in 3.73 sec).
#15 6.075 /usr/share/dotnet/sdk/8.0.100-preview.3.23178.7/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(287,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/Builder/PlexCleaner/PlexCleaner.csproj]
#15 6.333 /root/.nuget/packages/microsoft.build.tasks.git/1.1.1/build/Microsoft.Build.Tasks.Git.targets(25,5): warning : Unable to locate repository with working directory that contains directory '/Builder/PlexCleaner'. [/Builder/PlexCleaner/PlexCleaner.csproj]
#15 6.378 /root/.nuget/packages/microsoft.build.tasks.git/1.1.1/build/Microsoft.Build.Tasks.Git.targets(48,5): warning : Unable to locate repository with working directory that contains directory '/Builder/PlexCleaner'. [/Builder/PlexCleaner/PlexCleaner.csproj]
#15 6.383 /root/.nuget/packages/microsoft.sourcelink.common/1.1.1/build/Microsoft.SourceLink.Common.targets(53,5): warning : Source control information is not available - the generated source link is empty. [/Builder/PlexCleaner/PlexCleaner.csproj]
#15 11.01   PlexCleaner -> /Builder/PlexCleaner/bin/Release/net7.0/linux-x64/PlexCleaner.dll
#15 11.06   PlexCleaner -> /Builder/Publish/
#15 DONE 11.2s

If you mix build platforms you will get a No such file or directory error as the binary fails to load due to missing dependencies.

richlander · 2023-04-10T20:35:36Z

@ptr727 100% correct. I have run into this same problem. However, it isn't specific to this new pattern. This has always been a pitfall. In fact, I just ran into this same problem (just a few minutes ago) building some Go code.

markmcgookin · 2023-04-21T13:27:16Z

Working through this thread with a version of @richlander 's great example but I am noticing something strange here. This is my docker file...

FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/nightly/sdk:8.0-preview AS build

ARG TARGETARCH

WORKDIR /src

# restore NuGet packages
COPY ["api/*.csproj", "api/"]
COPY ["tests/*.csproj", "tests/"]

RUN dotnet restore "api/some-project-api.csproj" -a $TARGETARCH
RUN dotnet restore "tests/some-project-tests.csproj" -a $TARGETARCH

# build project
COPY ["api/.", "api/"]
COPY ["tests/.", "tests/"]

WORKDIR /src/api

RUN dotnet build "some-project-api.csproj" -c Release -a $TARGETARCH

WORKDIR /src

RUN echo "Target: $TARGETARCH"
RUN echo "Build: $BUILDPLATFORM"

# run tests on docker build
RUN if [ "$TARGETARCH" = "$BUILDPLATFORM" ] ; then dotnet test "tests/some-project-tests.csproj" -a $TARGETARCH ; fi

# publish project
FROM build AS publish

WORKDIR /src/api

RUN dotnet publish "some-project-api.csproj" -c Release -o /app/publish -a $TARGETARCH 

# run app
FROM mcr.microsoft.com/dotnet/aspnet:7.0-bullseye-slim as final

WORKDIR /app
COPY --from=publish /app/publish .

ENTRYPOINT ["dotnet", "some-project-api.dll"]

I was running into issues with dotnet test and I assumed this was sort of 'oh it has to actually execute the code so that maybe won't work cross platform' ... so I fired in a conditional if, which never seemed to fire... so I echo'd out the $BUILDPLATFORM and it never seems to be set at all... am I being stupid here? I have tried with both normal docker build and docker buildx build (which DOES build cross platform at once for me!)

I will paste the full output below, but the relevant lines are:

 => CACHED [linux/arm64->amd64 build 12/14] RUN echo "Target: amd64"                                                                                    0.0s
 => CACHED [linux/arm64->amd64 build 13/14] RUN echo "Build: $BUILDPLATFORM"

for intel, and for arm:

 => CACHED [linux/arm64 build 12/14] RUN echo "Target: arm64"                                                                                           0.0s
 => CACHED [linux/arm64 build 13/14] RUN echo "Build: $BUILDPLATFORM"

Thus my tests are never getting run as the if statement never fires. Am I missing something here?

docker buildx build -f dockerfile.api --push -t markmcgookin/some-project:20230421.1 -t markmcgookin/some-project:latest --platform linux/amd64,linux/arm64  . 
[+] Building 72.5s (42/42) FINISHED                                                                                                                          
 => [internal] load .dockerignore                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                         0.0s
 => [internal] load build definition from dockerfile.api                                                                                                0.0s
 => => transferring dockerfile: 1.09kB                                                                                                                  0.0s
 => [linux/arm64 internal] load metadata for mcr.microsoft.com/dotnet/aspnet:7.0-bullseye-slim                                                          0.2s
 => [linux/amd64 internal] load metadata for mcr.microsoft.com/dotnet/aspnet:7.0-bullseye-slim                                                          0.2s
 => [linux/arm64 internal] load metadata for mcr.microsoft.com/dotnet/nightly/sdk:8.0-preview                                                           0.3s
 => [linux/arm64 build  1/14] FROM mcr.microsoft.com/dotnet/nightly/sdk:8.0-preview@sha256:d7da5c8aa2963c312ffeb43951755a4651ebdfde5f79230a37d2038cba5  0.0s
 => => resolve mcr.microsoft.com/dotnet/nightly/sdk:8.0-preview@sha256:d7da5c8aa2963c312ffeb43951755a4651ebdfde5f79230a37d2038cba547071                 0.0s
 => [linux/arm64 final 1/3] FROM mcr.microsoft.com/dotnet/aspnet:7.0-bullseye-slim@sha256:ff588d989020412cd2d0f2781a2c1e7a144811d405eb865d2280e285d861  0.0s
 => => resolve mcr.microsoft.com/dotnet/aspnet:7.0-bullseye-slim@sha256:ff588d989020412cd2d0f2781a2c1e7a144811d405eb865d2280e285d861de4d                0.0s
 => CACHED [linux/amd64 final 1/3] FROM mcr.microsoft.com/dotnet/aspnet:7.0-bullseye-slim@sha256:ff588d989020412cd2d0f2781a2c1e7a144811d405eb865d2280e  0.0s
 => => resolve mcr.microsoft.com/dotnet/aspnet:7.0-bullseye-slim@sha256:ff588d989020412cd2d0f2781a2c1e7a144811d405eb865d2280e285d861de4d                0.0s
 => [internal] load build context                                                                                                                       0.4s
 => => transferring context: 29.01MB                                                                                                                    0.3s
 => CACHED [linux/arm64 build  2/14] WORKDIR /src                                                                                                       0.0s
 => CACHED [linux/arm64 build  3/14] COPY [api/*.csproj, api/]                                                                                          0.0s
 => CACHED [linux/arm64 build  4/14] COPY [tests/*.csproj, tests/]                                                                                      0.0s
 => CACHED [linux/arm64->amd64 build  5/14] RUN dotnet restore "api/some-project-api.csproj" -a amd64                                                 0.0s
 => CACHED [linux/arm64->amd64 build  6/14] RUN dotnet restore "tests/some-project-tests.csproj" -a amd64                                             0.0s
 => CACHED [linux/arm64->amd64 build  7/14] COPY [api/., api/]                                                                                          0.0s
 => CACHED [linux/arm64->amd64 build  8/14] COPY [tests/., tests/]                                                                                      0.0s
 => CACHED [linux/arm64->amd64 build  9/14] WORKDIR /src/api                                                                                            0.0s
 => CACHED [linux/arm64->amd64 build 10/14] RUN dotnet build "some-project-api.csproj" -c Release -a amd64                                            0.0s
 => CACHED [linux/arm64->amd64 build 11/14] WORKDIR /src                                                                                                0.0s
 => CACHED [linux/arm64->amd64 build 12/14] RUN echo "Target: amd64"                                                                                    0.0s
 => CACHED [linux/arm64->amd64 build 13/14] RUN echo "Build: $BUILDPLATFORM"                                                                            0.0s
 => CACHED [linux/arm64->amd64 build 14/14] RUN if [ "amd64" = "$BUILDPLATFORM" ] ; then dotnet test "tests/some-project-tests.csproj" -a amd64 ; fi  0.0s
 => CACHED [linux/arm64->amd64 publish 1/2] WORKDIR /src/api                                                                                            0.0s
 => [linux/arm64->amd64 publish 2/2] RUN dotnet publish "some-project-api.csproj" -c Release -o /app/publish -a amd64                                 1.3s
 => CACHED [linux/arm64 final 2/3] WORKDIR /app                                                                                                         0.0s
 => CACHED [linux/arm64 build  5/14] RUN dotnet restore "api/some-project-api.csproj" -a arm64                                                        0.0s
 => CACHED [linux/arm64 build  6/14] RUN dotnet restore "tests/some-project-tests.csproj" -a arm64                                                    0.0s
 => CACHED [linux/arm64 build  7/14] COPY [api/., api/]                                                                                                 0.0s
 => CACHED [linux/arm64 build  8/14] COPY [tests/., tests/]                                                                                             0.0s
 => CACHED [linux/arm64 build  9/14] WORKDIR /src/api                                                                                                   0.0s
 => CACHED [linux/arm64 build 10/14] RUN dotnet build "some-project-api.csproj" -c Release -a arm64                                                   0.0s
 => CACHED [linux/arm64 build 11/14] WORKDIR /src                                                                                                       0.0s
 => CACHED [linux/arm64 build 12/14] RUN echo "Target: arm64"                                                                                           0.0s
 => CACHED [linux/arm64 build 13/14] RUN echo "Build: $BUILDPLATFORM"                                                                                   0.0s
 => CACHED [linux/arm64 build 14/14] RUN if [ "arm64" = "$BUILDPLATFORM" ] ; then dotnet test "tests/some-project-tests.csproj" -a arm64 ; fi         0.0s
 => CACHED [linux/arm64 publish 1/2] WORKDIR /src/api                                                                                                   0.0s
 => CACHED [linux/arm64 publish 2/2] RUN dotnet publish "some-project-api.csproj" -c Release -o /app/publish -a arm64                                 0.0s
 => CACHED [linux/arm64 final 3/3] COPY --from=publish /app/publish .                                                                                   0.0s
 => [linux/amd64 final 2/3] WORKDIR /app                                                                                                                0.0s
 => [linux/amd64 final 3/3] COPY --from=publish /app/publish .                                                                                          0.0s
 => exporting to image                                                                                                                                 70.5s
 => => exporting layers                                                                                                                                 0.3s
 => => exporting manifest sha256:126ec6a256c0b985b34890e294ca564185c2c5953cf801b4059f8b5130f22e9e                                                       0.0s
 => => exporting config sha256:91f49866a4cf7bbbc4015c52e20e2906418c940304e733a2f4c2b2c9a5ac684e                                                         0.0s
 => => exporting attestation manifest sha256:9b702e17c9536fefee0baebef6b0e441657875beb9a0d301eccffe4268e63bb4                                           0.0s
 => => exporting manifest sha256:804482a55a15a84769ac0fe837b586eb4f254d635073651751f4df36bf69226e                                                       0.0s
 => => exporting config sha256:f3ad2f6c8d294082c1c2670c627653beb8fb294fb7c477a1da4540cf0085bb98                                                         0.0s
 => => exporting attestation manifest sha256:ec0a6aa61f65a3d917e7b766caee3107dcde5d71935cce6465441cd105f3afd9                                           0.0s
 => => exporting manifest list sha256:c5b30f49682e9bdf619777119256092e1e18d6cd8751d2047335dba94b3884af                                                  0.0s
 => => pushing layers                                                                                                                                   1.8s
 => => pushing manifest for docker.io/markmcgookin/some-project:20230421.1@sha256:c5b30f49682e9bdf619777119256092e1e18d6cd8751d2047335dba94b3884af       1.9s
 => => pushing manifest for docker.io/markmcgookin/some-project:latest@sha256:c5b30f49682e9bdf619777119256092e1e18d6cd8751d2047335dba94b3884af           0.9s

goncalo-oliveira · 2023-04-21T13:45:51Z

@markmcgookin looking at all of those CACHED in the output, I'm guessing you should retry with the --no-cache argument.

markmcgookin · 2023-04-21T13:50:29Z

@markmcgookin looking at all of those CACHED in the output, I'm guessing you should retry with the --no-cache argument.

Sorry, I've run this a load and chopped and changed a lot of things, but its exactly the same. Just used --no-cache there.

 => [linux/arm64 build 11/14] WORKDIR /src                                                                                                              0.0s
 => [linux/arm64 build 12/14] RUN echo "Target: arm64"                                                                                                  0.0s
 => [linux/arm64 build 13/14] RUN echo "Build: $BUILDPLATFORM"                                                                                          0.0s
 => [linux/arm64 build 14/14] RUN if [ "arm64" = "$BUILDPLATFORM" ] ; then dotnet test "tests/battery-logger-tests.csproj" -a arm64 ; fi                0.0s
 => [linux/arm64 publish 1/2] WORKDIR /src/api                                                                                                          0.0s
 => [linux/arm64 publish 2/2] RUN dotnet publish "battery-logger-api.csproj" -c Release -o /app/publish -a arm64                                        1.3s
 => [linux/arm64->amd64 build 11/14] WORKDIR /src                                                                                                       0.0s
 => [linux/arm64->amd64 build 12/14] RUN echo "Target: amd64"                                                                                           0.0s
 => [linux/arm64->amd64 build 13/14] RUN echo "Build: $BUILDPLATFORM"

(The project name is different, because I replaced it last time, then realised this is a test app and no-one cares)

campbellwray · 2023-04-21T16:26:44Z

@markmcgookin your problem is a little different to mind, but I too noticed that dotnet test doesn't work properly.

For me, I think that because there is an SDK version mismatch (i.e. I am trying to test a .NET 7 .csproj file with SDK version 8), I was getting errors telling me to change the .NET version, in my .csproj or install the .NET 7 SDK. It seems that dotnet test is not backwards compatible like dotnet build is.

For now I am just waiting for 7.0.3xx to be released, since it will have this Docker fix too.

ptr727 · 2023-04-21T16:39:15Z

If the arg is not set, maybe explicitly declare the arg.
For running unit tests, builder layer will always be native platform so no need to test for platform, target layer will sometimes be native platform, so must test on target/final layer.
Need to force .NET 7 to run with .NET 8 preview.

E.g. this is what I ended up using: https://github.com/ptr727/PlexCleaner/blob/develop/Docker/Debian.dotNET.Dockerfile

E.g. builder layer:

ARG \
    TARGETPLATFORM \
    TARGETARCH \
    BUILDPLATFORM

ENV \
    DOTNET_ROLL_FORWARD=Major \
    DOTNET_ROLL_FORWARD_PRE_RELEASE=1

e.g. final layer

ARG \
    TARGETPLATFORM \
    BUILDPLATFORM
RUN if [ "$BUILDPLATFORM" = "$TARGETPLATFORM" ]; then \
        dotnet --info; \
        ffmpeg -version; \
        HandBrakeCLI --version; \
        mediainfo --version; \
        mkvmerge --version; \
        /PlexCleaner/PlexCleaner --version; \
    fi

richlander · 2023-04-21T18:01:22Z

Yes. The ARGS are not set by default. You have to set them via the pattern that @ptr727 is using. Same thing as here: https://gist.github.com/richlander/70cde3f0176d36862af80c41722acd47. The pattern in FROM is likely special-cased.

richlander · 2023-04-21T18:10:59Z

I am happy to create (or update) a sample that shows how to do unit testing as is being demonstrated here. I didn't think of that initially. Good scenario!

markmcgookin · 2023-04-21T20:37:17Z

Thanks @richlander that would be great. So the args are defined by default but I need to set them. Cool. I misunderstood. Thanks all for the help

richlander · 2023-04-21T22:17:05Z

No worries. It's simple once you know how it all works and quite mysterious before that. It took us a bit to figure out these patterns. And aspects of it are not intuitive because of the way Dockerfiles work, including various non-obvious scoping.

douglasg14b · 2023-04-21T23:59:59Z

This is closed (as are the dozens of similar issues), does that indicate the problem is now solved for .Net 7?

If so, what is the definitive solution?

richlander · 2023-04-22T01:15:25Z

Great question. The problem is fixed in .NET 8 Preview 3 and 7.0.300. The latter hasn't shipped yet. I don't have a date handy on when that will be. However, you can use .NET 8 Preview 3 to build .NET 7 code as a workaround in the time.

Also, you don't need the fix if you use -r or -a in your build. If you don't then the $BUILDPLATFORM pattern works already w/o any fixes.

douglasg14b · 2023-04-22T01:57:46Z

@richlander Gotcha, and by using .Net 8 Preview 3 only for the build step, that means using that are the image in the Dockerfile instead of the .Net 7 base image correct?

We are using --platform linux/amd64 in the docker build command and in the Dockerfile 🤔

We don't use the runtime flag for dotnet restore, but it sounds like that by itself is a workaround?

richlander · 2023-04-22T18:42:20Z

by using .Net 8 Preview 3 only for the build step, that means using that are the image in the Dockerfile instead of the .Net 7 base image correct?

Right. The SDK version you use doesn't matter for the final app/image.

We don't use the runtime flag

If you are not using -r then that's simpler.

iliashkolyar · 2023-05-04T22:15:21Z

Hello @richlander ,
I'm trying to use your technique to build a .NET7 application on an M1 apple machine.

Dockerfile:

FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/nightly/sdk:8.0-preview AS build-env
ARG TARGETARCH
WORKDIR /app

COPY . ./MyApp
WORKDIR /app/MyApp/MyApp.EntryPoint

RUN dotnet restore -a $TARGETARCH

###### publish ######
FROM build-env AS publish
RUN dotnet publish --no-restore -o out -a $TARGETARCH

###### runtime image ######
FROM mcr.microsoft.com/dotnet/aspnet:7.0.5-alpine3.17
WORKDIR /app
COPY --from=publish /app/MyApp/MyApp.EntryPoint/out .
ENTRYPOINT ["dotnet", "MyApp.EntryPoint.dll"]

Build command: docker build --platform linux/arm64 -t my-app -f ./Dockerfile .

Run command: docker run --rm -it --name my-app -p 5004:5004 my-app

I receive the following error:

Unhandled exception. System.IO.FileLoadException: Could not load file or assembly 'MyApp.EntryPoint, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'.
qemu: uncaught target signal 6 (Aborted) - core dumped

If I try to check the container contents using docker run --rm --entrypoint=sh -it my-app, I reach the /app folder, in which I can see the MyApp.EntryPoint.dll along with all other relevant DLLs, so I'm not sure what i'm missing here.

The container does load properly when I don't use the $TARGETARCH/$BUILDPLATFORM approach (although it does crash due to qemu segmentation fault when trying to use EFCore, which I assume is related to the fact the qemu doesn't really work with .NET as you stated before).

Thanks!

goncalo-oliveira · 2023-05-22T10:37:05Z

Wasn't this supposed to be fixed with 7.0.302 ?

I've just downloaded the SDK update and tried to build an image on my M1, but it fails with similar errors (sometimes it just freezes).

Reverting to 8.0 nightly or preview-4 works though.

To clarify, this is just for the build process, not the runtime. And yes, I did a --no-cache to ensure the newer image is downloaded and running a dotnet --version to confirm that. I also tried using SDK 7.0.302 explicitly, same result...

Retracting this... it does work as intended. When replacing the image, I removed the --platform argument. Apologies.

…dotnet-docker#4388) (#24)

vukasinpetrovic · 2023-08-03T21:50:01Z

@richlander Hi, I trying to solve this for days without success. I followed all of your steps but when I do that I get stuck on dotnet restore command which causes problem with package downgrade. It's just a small part of that error, it's a long list.

1.523   Determining projects to restore...
8.662   Restored /src/CorporateGames.Core/CorporateGames.Core.csproj (in 6.58 sec).
41.97 /src/CorporateGames.API/CorporateGames.API.csproj : error NU1605: Warning As Error: Detected package downgrade: System.Collections from 4.3.0 to 4.0.11. Reference the package directly from the project to select a different version.  [/src/CorporateGames.sln]

When I remove -a tags, then I guess it does not build for the proper platform. Here is my Dockerfile. Any help is appreciated.

# Defines SDK image to build the apliaction
FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/nightly/sdk:8.0-preview AS build-env
ARG TARGETARCH

# Creates folder "src" inside container and set it as current folder
WORKDIR /src

# Copies all csproj files as they are needed for restore function
COPY *.sln .
COPY CorporateGames.API/*.csproj CorporateGames.API/
COPY CorporateGames.Application/*.csproj CorporateGames.Application/
COPY CorporateGames.Core/*.csproj CorporateGames.Core/

# Restores all packages from main project (CorporateGames.API)
RUN dotnet restore -a $TARGETARCH

# Copies all files from project folder into src folder inside container
COPY . .

# Go into CorporateGames.API folder as we'll publish only that project
WORKDIR /src/CorporateGames.API

# Build the project
RUN dotnet build CorporateGames.API.csproj -c Staging --no-restore -a $TARGETARCH

# Publishes the project into publish folder inside container
RUN dotnet publish CorporateGames.API.csproj -c Staging -o /publish --no-build --no-restore -a $TARGETARCH

# Defines runtime image to run the application
FROM mcr.microsoft.com/dotnet/aspnet:7.0 as runtime

# Set current folder
WORKDIR /src/CorporateGames.API/publish

# Copy the publish directory from the build-env stage into the runtime image
COPY --from=build-env /publish .

ENV ASPNETCORE_URLS=http://+:5000
ENV ASPNETCORE_ENVIRONMENT=Staging

EXPOSE 5000
ENTRYPOINT ["dotnet", "CorporateGames.API.dll"]

richlander · 2023-08-04T15:53:36Z

@vukasinpetrovic -- Package downgrades should be unrelated. If you remove all this fancy architecture targeting stuff, I assume the package downgrades still exist. Is that true?

@iliashkolyar -- This pattern enables successful building. The resultant x64 image may or may not successfully run in QEMU, however. I just build a .NET 8 app/image with this pattern and (to my surprise) it ran as x64 on my M1 machine w/o issue.

Our samples are being updated to this pattern. #4742

Apparently, this ENV may also help: DOTNET_EnableWriteXorExecute=0, for development (don't disable for prod, as it is a security feature). Source: dotnet/runtime#88971 (comment)

vukasinpetrovic · 2023-08-06T11:10:38Z

@richlander You are right, that problem with downgrades is not related to docker, but restoring project packages on arm that targets amd64 platform (if I specify architecture/runtime while doting dotnet restore). I'll have to research that one. If you had that problem also, any info would be much appreciated.

richlander · 2023-08-06T17:09:38Z

I didn't run into that problem. If you have a repro, that would be useful. The repro won't need your app, just some subset of your project file.

wondertalik · 2023-09-19T09:02:28Z

Hi there, this doesnt work for me. I read this and all related subjects.

FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/runtime:7.0-jammy AS base
ARG TARGETARCH
ARG BUILD_CONFIGURATION
WORKDIR /app

FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:7.0-jammy AS build
ARG TARGETARCH
ARG BUILD_CONFIGURATION
WORKDIR /src
COPY ["TestDocker/TestDocker.csproj", "TestDocker/"]
RUN dotnet restore "TestDocker/TestDocker.csproj" -a $TARGETARCH
COPY . .
WORKDIR "/src/TestDocker"
RUN dotnet build "TestDocker.csproj" -c $BUILD_CONFIGURATION -o /app/build -a $TARGETARCH --self-contained false

FROM build AS publish
RUN dotnet publish "TestDocker.csproj" -c $BUILD_CONFIGURATION -a $TARGETARCH --self-contained false  -o /app/publish --no-restore

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "TestDocker.dll"]

on my m1 mac

docker buildx create --name testbuilder --bootstrap --use --platform linux/arm64,linux/amd64
docker buildx build --platform linux/amd64,linux/arm64 --builder testbuilder --no-cache --progress=plain --build-arg BUILD_CONFIGURATION=Debug --push -t myrepo.azurecr.io/testdocker:v1.0.0 -f TestDocker/Dockerfile .

run on m1

docker run myrepo.azurecr.io/testdocker:v1.0.0
Hello, World!

run on amd64

docker run myrepo.azurecr.io/testdocker:v1.0.0
exec /usr/bin/dotnet: exec format error

dotnet sdk: 7.0.401
Docker version 24.0.6, build ed223bc

leros1337 · 2024-05-16T12:20:24Z

Hi, stuck in problem: When build on my mac m1 via command docker buildx build --platform linux/arm64,linux/amd64 . -t test i can run it natively nice. But when push it to registry (in registry image shows both arch) and try to run on other amd64 machine (wsl2) getting error

Unhandled exception. System.IO.FileLoadException: Could not load file or assembly 'App, Version=1.7.9.0, Culture=neutral, PublicKeyToken=null'.
qemu: uncaught target signal 6 (Aborted) - core dumped

And reverse behavior, when build on amd64 and run on m1 have same error.

Dockerfile looks like this

FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:8.0-alpine AS pre-build
RUN apk add --no-cache gcompat protoc grpc-plugins

WORKDIR /src
COPY . .

FROM pre-build AS publish
ENV PROTOBUF_PROTOC=/usr/bin/protoc
ENV GRPC_PROTOC_PLUGIN=/usr/bin/grpc_csharp_plugin
ENV DOTNET_EnableWriteXorExecute=0
ARG TARGETARCH
RUN dotnet publish "src/Proj.csproj" -c Release -o /app/publish -a $TARGETARCH /p:UseAppHost=false

FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/aspnet:8.0-alpine
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "myapp.dll"]

Yes, workaround is build on amd64 machine > push > run on arm64 via emulation. May be have some missunderstanding and cant build on 1 machine 2 different arch?

richlander added the bug label Feb 6, 2023

dotnet-issue-labeler bot added area-infrastructure untriaged labels Feb 6, 2023

mthalman removed the untriaged label Feb 6, 2023

This was referenced Feb 6, 2023

Enable using docker build --platform switch (easily) dotnet/sdk#30369

Closed

Consider using arm64 instead of arm64v8 tags for .NET 8 #4400

Closed

richlander mentioned this issue Feb 22, 2023

Update samples for platform targeting #4441

Closed

hoodnoah mentioned this issue Mar 4, 2023

multiarch build hoodnoah/dynapork#27

Open

richlander mentioned this issue Mar 15, 2023

Theme: Cloud Native dotnet/aspnetcore#47240

Closed

11 tasks

richlander closed this as completed Mar 15, 2023

JL03-Yue mentioned this issue Apr 10, 2023

The specified RuntimeIdentifier 'linux-linux/amd64' is not recognized dotnet/sdk#31616

Closed

tam1m mentioned this issue Jun 9, 2023

Fix Dockerfile for cross building with dotnet SenexCrenshaw/StreamMaster#24

Merged

6 tasks

SenexCrenshaw pushed a commit to SenexCrenshaw/StreamMaster that referenced this issue Jun 9, 2023

Fix Dockerfile for cross building with dotnet. For reference (dotnet/…

f7cb86c

…dotnet-docker#4388) (#24)

austinlparker mentioned this issue Jul 18, 2023

Fix flaky cartservice build open-telemetry/opentelemetry-demo#984

Merged

alexyaang mentioned this issue Jul 21, 2023

Cannot debug dotnet Docker amd64 container on host OSX M2 microsoft/vscode-docker#4009

Closed

TheAxelander mentioned this issue Oct 31, 2023

Re-enable arm64 Docker Image build in Github Actions TheAxelander/OpenBudgeteer#131

Closed

zliebersbach mentioned this issue Dec 6, 2023

Some amd64 containers not running on arm64 host orbstack/orbstack#852

Closed

Enable using docker build --platform switch (easily) #4388

Enable using docker build --platform switch (easily) #4388

Comments

richlander commented Feb 6, 2023 • edited Loading

richlander commented Feb 6, 2023 • edited Loading

mthalman commented Feb 6, 2023 • edited Loading

MichaelSimons commented Feb 7, 2023

richlander commented Feb 7, 2023

richlander commented Feb 17, 2023

jkotas commented Feb 17, 2023

jkotas commented Feb 17, 2023

mthalman commented Feb 21, 2023

nagilson commented Mar 6, 2023 • edited Loading

richlander commented Mar 7, 2023 • edited Loading

richlander commented Mar 15, 2023

richlander commented Mar 15, 2023

goncalo-oliveira commented Mar 16, 2023

campbellwray commented Mar 16, 2023

richlander commented Mar 17, 2023 • edited Loading

campbellwray commented Mar 17, 2023 • edited Loading

ptr727 commented Apr 8, 2023 • edited Loading

richlander commented Apr 10, 2023

markmcgookin commented Apr 21, 2023

goncalo-oliveira commented Apr 21, 2023

markmcgookin commented Apr 21, 2023 • edited Loading

campbellwray commented Apr 21, 2023

ptr727 commented Apr 21, 2023

richlander commented Apr 21, 2023

richlander commented Apr 21, 2023

markmcgookin commented Apr 21, 2023

richlander commented Apr 21, 2023

douglasg14b commented Apr 21, 2023 • edited Loading

richlander commented Apr 22, 2023

douglasg14b commented Apr 22, 2023

richlander commented Apr 22, 2023

iliashkolyar commented May 4, 2023

goncalo-oliveira commented May 22, 2023 • edited Loading

vukasinpetrovic commented Aug 3, 2023

richlander commented Aug 4, 2023 • edited Loading

vukasinpetrovic commented Aug 6, 2023

richlander commented Aug 6, 2023

wondertalik commented Sep 19, 2023 • edited Loading

leros1337 commented May 16, 2024 • edited Loading

Enable using docker build `--platform` switch (easily) #4388

Enable using docker build `--platform` switch (easily) #4388

richlander commented Feb 6, 2023 •

edited

Loading

richlander commented Feb 6, 2023 •

edited

Loading

mthalman commented Feb 6, 2023 •

edited

Loading

nagilson commented Mar 6, 2023 •

edited

Loading

richlander commented Mar 7, 2023 •

edited

Loading

richlander commented Mar 17, 2023 •

edited

Loading

campbellwray commented Mar 17, 2023 •

edited

Loading

ptr727 commented Apr 8, 2023 •

edited

Loading

markmcgookin commented Apr 21, 2023 •

edited

Loading

douglasg14b commented Apr 21, 2023 •

edited

Loading

goncalo-oliveira commented May 22, 2023 •

edited

Loading

richlander commented Aug 4, 2023 •

edited

Loading

wondertalik commented Sep 19, 2023 •

edited

Loading

leros1337 commented May 16, 2024 •

edited

Loading