Proposal: Cross-platform Swarm cluster

[chanwit]

I'd like to thank you @bfirsh for inviting me to propose this feature to swarm.
According to my DZone article, a POC of the cross-platform hybrid cloud have been successfully built atop swarm.

To make swarm support the similar kind of feature mentioned in the article, i.e., being able to pull a certain image to its correct platform, it requires tweaking both swarm and docker.

In Docker, we need to put "platform": runtime.GOOS + "_" + runtime.GOARCH into the engine's labels.

In Swarm, we need to use the platform label to rewrite the name of an image during the container creation and the image pulling process. For example,

nginx would be rewritten to nginx:latest.linux_arm for the Linux/ARM platform, and to nginx:latest.windows_amd64 for the coming Windows/AMD64 platform.

This issue would serve as both proposal and the tracking for these changes.

Thoughts and feedback are very welcome.

/cc @vieux @aluzzardi @abronan

[aluzzardi]

Thank you so much for reporting back, I love the idea.

We'll talk with the Engine maintainers to figure out a path to achieve hybrid cluster management.

[chanwit]

Thank you, @aluzzardi
Please feel free to let me know if you'd like me to start any PR regarding this proposal.

[bpradipt]

This feature will be good for Power (ppc64) platform as well. Today swarm can be built and run on power architecture without any issues. However my understanding was that to really have a hybrid arch cluster, we would need the image registry to be arch aware so that the image 'pull' request automatically handles downloading of the appropriate arch image for the respective cluster members.

[chanwit]

@bpradipt a good thing in this design is that swarm will be taking care of platform-awareness. So the current registry does not need to change.

[bpradipt]

@chanwit , @aluzzardi , I was trying to code up a simple implementation of this proposal and wanted to take your inputs on the design.
Here are the key changes

1. Extend docker engine Info structure to report the platform Architecture (amd64, ppc64, arm64, s390x etc). Example 'docker info' o/p with proposed changes
   [snip]
   Name: swarm-vm1
   ID: HTLL:YJFZ:S6I4:ZTKA:A2H4:OSUX:L2ZU:ECVI:I5GY:2QL3:TICT:PF5P
   Architecture: amd64
   [snip]
2. Add architecture as a default label to swarm (similar to operatingsystem, kernelversion etc).
   Example o/p with proposed changes
   [snip]
   swarm-vm1: 192.168.122.51:2375
   └ Containers: 15
   └ Reserved CPUs: 0 / 1
   └ Reserved Memory: 0 B / 1.019 GiB
   └ Labels: architecture=amd64, executiondriver=native-0.2, kernelversion=3.16.0-30-generic, operatingsystem=Ubuntu 14.10, storagedriver=devicemapper
   [snip]
3. Architecture specific scheduling of a container will need to use the architecture constraint.

Do let me know your thoughts and I can send the PRs for docker and swarm accordingly.

[chanwit]

@bpradipt Thank you for looking into this!

1. architecture is not enough to handling all cases, IMO.
   We need the platform label, which is a concat of os and architecture.
2. Adding label to Swarm will not help much. I think you mean adding it to the engine?
3. There's several points in Swarm / Registry to intercept to make things flow smoothly. Maybe I need to have a bit better design before going through the PRs.

To sum things up, the first PR than can be safely done is to propose the platform label to the engine.

[bpradipt]

@chanwit, engine already provides for operating system label. I was of the opinion to just add architecture. Any concatenation can be done by the users of those label (swarm in this case).

[chanwit]

@bpradipt to me, the current operating system label in Docker is for human read, e.g. Ubuntu, Debian etc. We need a kind of uncertain parsing to detect which one the OS is (linux, windows, etc.). IMO, it's far better to just have the platform label using values providing by Go.

[bpradipt]

@chanwit, I see your point on the operating system type (Linux/Windows etc). I have submitted a new docker issue[1] to get thoughts from the engine pov. Based on their suggestion I'll submit the appropriate PRs. Thanks for your suggestions

[1] moby/moby#13634

[chanwit]

@bpradipt thank you for that!
I'll continue to update the design to this proposal in the mean time.

[bpradipt]

@chanwit while re-reading the proposal again, and trying some more hybrid (x86_64/ppc64le) cluster experimentation on my setup, I came across some challenges which I would like to discuss.

1. The requirement for having the image tagged like name:ver.arch puts a restriction on the way users would need to tag their images. Users might just decide to tag the image as name:arch or name:ver_arch or name:arch_ver etc. Ofcourse this can be added as one of the requirement to have a multi-arch swarm cluster.
2. I feel multi-arch support in v2 registry might be more intuitive and straightforward. If registry is indexed with arch data and the same is available as image query/pull/push option, then as long as the correct arch specific image is available in the registry the docker host can download the same. This avoids any explicit tagging with arch details.

[chanwit]

@bpradipt

1. This should be the problem IMO, When you call docker build via Swarm, it can detects the platform string of the underlying machine. After the build is successful, we can let Swarm issues another command to tag the image.
2. I am not familiar of how the registry work. Maybe you're right about this. But ultimately, I think the topology of the build system would be like this:

                                               +---------> [arm build node]
                                               |
[docker cli] --> [registry] --> [multi-arch swarm] ------> [x86 build node]
                                               |
                                               +---------> [ppc64le build node]

[chanwit]

Hi all,

We would like to test our cross-platform Swarm cluster on the following platforms:

• Linux / IBM Power8
• Linux / ARM64 (X-Gene)
• Windows / AMD64 (updated: have to wait for Microsoft to release a build of Windows 2016 this summer - https://twitter.com/Taylorb_msft/status/612244087358357505)

If any one could sponsor (or allow accessing) the above machines (and platform), please drop a comment on this issue.

[ahmetb]

@chanwit if you sign up to Azure, I believe there's a pretty good 90-day free trial. You can get quite Windows VMs easily from there. https://azure.microsoft.com/pricing/free-trial/

[shykes]

@chanwit the official Docker testing infrastructure is pretty advanced and well funded. I think it would be great if we could help test this setup! It would be very useful to the Docker project as well.

[chanwit]

Thanks for the prompt reply, @ahmetalpbalkan is it possible to try the new Windows kernel (with Docker support) on Azure? I have Swarm running fine on Windows now.

[chanwit]

@shykes Thank you very much. This will be fantastic!!

[ahmetb]

@chanwit Unfortunately not yet, containers will come in Windows Server 2016.

[chanwit]

@ahmetalpbalkan I see. Can't wait for it to be released :)

[zoobab]

armv7, armv5?

[chanwit]

@zoobab we already have a multi-arch cluster over ARMv7 and x86-64 at our lab. Listed above are hardware archs that we'd like to try Docker & Swarm on them.

[proegssilb]

I've got an older AMD64 machine you guys can use, but honestly, you can get a "good starting point" AMD64 machine for $300 ($200 if you've got a copy of windows) pretty trivially on newegg. Some assembly may be required.

Edit: I should point out that my knowledge of AMD64 is from the desktop market, where AMD64 is basically the same as x86_64, to the point where this happened because AMD was out first.

[chanwit]

@proegssilb thank you for that. I already have AMD64 machines for testing.

[aluzzardi]

ping @dmp42

[dmp42]

Thanks @aluzzardi
cc-ing @docker/distribution-maintainers for the record.

[chanwit]

@aluzzardi @dmp42 it seems you've got some moves on this topic. Let me know if I can be any help.

[stevvooe]

We are working on this as a part of the Distribution Working Group. The "platform" label is a common proposal but it has a number of drawbacks. In particular, ARM has a lot more variation than can be represented in a single label. We've discussed the notion of a "variant", "endian" and "abi" to correctly index different architectures.

Here is a similar discussion over at opencontainers: opencontainers/runtime-spec#39 (comment).

The problem here is that as we get more variation in architecture, we need to be able to represent that variation and build proper tables to make compatibility decisions (ie armv6-m could run on armv7-m). A single, cross-product of OS/Arch is not sufficient.

Any solutions that include the registry making image selections will break the security model of the new distribution system. All manifest and layers are content addressable in this scheme, so the approach we are following to maintain this property through multi-arch is to allow the client to select the correct manifest.

Here is the manifest proposal to start being able to support multi-arch from a distribution standpoint: distribution/distribution#62 (comment). The allows packing "fat tags", similar to fat binaries. When this gets implemented, we need to design a labeling scheme that we throughout docker, swarm and distribution. With this labeling scheme, clients can make manifest selections based on local support, compatibility tables and what's available.

cc @estesp @vbatts

[vbatts]

Is having the byteorder in there an immediate requirement?

[PatrickLang]

So where does this stand? Swarmkit in v1.12+ allows multiple platforms (OS/arch) to be in the same cluster. Can OS & Architecture at least be used in constraints when scheduling?

patrick@coldbrew:~$ docker node inspect --pretty coldbrew
ID:                     ...
Hostname:               coldbrew
Joined at:              2016-09-05 22:35:05.610826735 +0000 utc
Status:
 State:                 Ready
 Availability:          Active
Manager Status:
 Address:               192.168.1.143:2377
 Raft Status:           Reachable
 Leader:                Yes
Platform:
 Operating System:      linux
 Architecture:          x86_64
Resources:
 CPUs:                  2
 Memory:                1.945 GiB
Plugins:
  Network:              bridge, host, null, overlay
  Volume:               local
Engine Version:         1.12.1
patrick@coldbrew:~$ docker node inspect --pretty black-pearl
ID:                     ...
Hostname:               black-pearl
Joined at:              2016-09-11 03:11:34.184268901 +0000 utc
Status:
 State:                 Ready
 Availability:          Active
Platform:
 Operating System:      linux
 Architecture:          armv7l
Resources:
 CPUs:                  4
 Memory:                862 MiB
Plugins:
  Network:              bridge, host, null, overlay
  Volume:               local
Engine Version:         1.12.1

[aluzzardi]

Yes - in docker master (therefore in the next release, RCs coming soon)

I believe it's called node.platform.os and node.platform.arch
On Fri, Nov 4, 2016 at 10:12 PM Patrick Lang notifications@github.com
wrote:

So where does this stand? Swarmkit in v1.12+ allows multiple platforms
(OS/arch) to be in the same cluster. Can OS & Architecture at least be used
in constraints when scheduling?

patrick@coldbrew:$ docker node inspect --pretty coldbrew
ID: ...
Hostname: coldbrew
Joined at: 2016-09-05 22:35:05.610826735 +0000 utc
Status:
State: Ready
Availability: Active
Manager Status:
Address: 192.168.1.143:2377
Raft Status: Reachable
Leader: Yes
Platform:
Operating System: linux
Architecture: x86_64
Resources:
CPUs: 2
Memory: 1.945 GiB
Plugins:
Network: bridge, host, null, overlay
Volume: local
Engine Version: 1.12.1
patrick@coldbrew:$ docker node inspect --pretty black-pearl
ID: ...
Hostname: black-pearl
Joined at: 2016-09-11 03:11:34.184268901 +0000 utc
Status:
State: Ready
Availability: Active
Platform:
Operating System: linux
Architecture: armv7l
Resources:
CPUs: 4
Memory: 862 MiB
Plugins:
Network: bridge, host, null, overlay
Volume: local
Engine Version: 1.12.1

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#826 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAK9LzRavB0jABYMhHjJnJbcwlY0kkKPks5q7BAzgaJpZM4EjYiF
.

[PatrickLang]

Is there a way to dump all labels for a given node so I can confirm if it's working? I can move everything to v1.13-dev if needed.

[aluzzardi]

Platform OS/Arch will show up even in 1.12 on docker node inspect (under NodeDescription I believe).

1.13-dev adds constraints for those fields, so I guess you can verify it's showing up the right data directly on your current 1.12

[DieterReuter]

@PatrickLang You can build a multi-arch swarm with 1.12, I've already done that. Then using contraints to schedule tasks on specific arch nodes. You can use a) platform specific labels or b) you can additionally set engine specific labels (to be set in docker daemon start). The platform specific labels are standard can be listed with docker node inspect (thanks @aluzzardi for the tip):

Docker4Mac running Moby:

        "Description": {
            "Hostname": "moby",
            "Platform": {
                "Architecture": "x86_64",
                "OS": "linux"
            },

Raspberry Pi3 running HypriotOS:

"Description": {
            "Hostname": "black-pearl",
            "Platform": {
                "Architecture": "armv7l",
                "OS": "linux"
            },

So you can use Description/Platform/Architecture to address a specific arch node.

[nishanttotla]

Closing due to lack of activity. Please reopen if you wish to continue discussing it.