byte order info

[glevand]

I didn't see any way to know the byte order (endian) of the container image other than inspecting binaries in the image. Byte order info should be in the config file so that a system can determine compatibility with only the config, and not need to download the entire container image when testing.

Maybe this is a shortcoming of using GOARCH as the set of values for platform:arch.

[stevvooe]

@glevand Could you give an example of where this may matter but wouldn't be covered by the platform/architecture? For example, is ppc64le not sufficient for the architecture?

[glevand]

@stevvooe I think the issue is that the spec requires a platform have the go language ported to it (to have a GOARCH value). Is that what we want, a dependency on the go language? If so, that should be stated in the spec since it is a significant dependency. Looking at the FAQ, it states 'Container formats and runtime should not be bound to clients, to higher level frameworks...'. I think the go language could be considered a high level framework.

What prompted me to open this issue was support for aarch64_be, but I now see the problem is that the spec requires go language support, which aarch64_be does not have.

[stevvooe]

@glevand Understood.

I agree that we shouldn't be dependent on go language support. However, we should follow the conventions for GOARCH/GOOS since they have done a thorough, modern job of capturing this variation. The ease of cross-compilation with Go is evidence of this.

Either way, my question still stand. At this point, you've only pointed out an inconsistency but we really need to understand why it matters. Could you give an example of where this may matter but wouldn't be covered by the platform/architecture? What specific problem are you trying to solve that is not covered by architecture?

[jonboulle]

FWIW we went through a similar discussion in appc and ended up deciding against using raw GOARCH values (arm or arm64) for ARM architectures since they don't offer the necessary granularity (you need to combine it with GOARM but even that doesn't get you all the way). We instead corresponded with the folks at ARM Holdings and used their suggestions for the most appropriate values - which for the most part turns out to be the equivalent of utsname.machine on Linux - https://github.com/appc/spec/blob/master/schema/types/labels.go#L24

[stevvooe]

@jonboulle So, it sounds like some sort of ARCH is sufficient, but we need to adopt more granular values for certain architectures. Looking at utsname, there seem to be a number of caveats there, as well: http://www.gnu.org/software/libc/manual/html_node/Platform-Type.html. For example, some have manufacturer specified. Reading through the installation documentation, the Go developers broken architecture specific extensions into sub-variables. Is there a standard registry or specification for utsname machine definitions? I could not find one.

I'd hate to have us diverge into another solution that has the same problems as the current one.

[jonboulle]

@stevvooe Unfortunately no. The closest thing we came up with was mpasternacki's suggestion here to use autotools triplets, which I think ultimately might be the least-bad solution - at least it is comprehensive and consistent. Because at the end of the day there are some things that seem like a total coin-flip (e.g. amd64 vs x86_64) and some that have trickier/subtler implications (like the ARM stuff, which is why we went to the "source"). A good case in point is the Debian "port naming debates" for armhf

[stevvooe]

@jonboulle That is very unfortunate. I'm not even a fan of using autotools triplets, as they tend to be as obtuse.

I really think deviating from Go here is a bad idea. We can always augment with extra required information, as the need arises (ie GOARM and GO386). It looks like we've already headed down a half-baked path (x86_64 vs amd64 is already inconsistent :( ) without solving the open problems.

[jonboulle]

@stevvooe hmm but how exactly are you suggesting that the extra information would be augmented? I am also unclear how we'd solve glevand's use case here, and share his concern about the spec being a little too go-centric

[glevand]

Some problems with using GNU triplets and some general info on ABI specifiers is mentioned here: https://wiki.debian.org/Multiarch/Tuples

[stevvooe]

@jonboulle I'd like to clarify that I'm not saying we should use GOARCH/GOOS directly. We should mirror the technique and mapping, including the extra portions, rather than invent something new. Initially, we might simply define GOARCH as equal to OCP's arch. What I'd like to avoid is adopting another solution that is simply different.

[jonboulle]

Sure, I get that, was just unclear on how exactly you plan on incorporating those extra portions - e.g. another field? or mapping everything into OS and ARCH?

[stevvooe]

@jonboulle I would suggest having extra environment variables that are OS/ARCH specific. This will keep OS/ARCH clean while not trying to pack data too tightly into those variables.

[philips]

@stevvooe I think we should just leave it open too; this is what we did in appc. os and arch are well-known labels but we left it open for someone else to add a byte-order or even arm.com/byte-order label.

[glevand]

@philips The problem I see with not specifying how the ABI is described is that each publisher then is free to describe it as they see fit. The user of the config will then need to have knowledge of all the variations and the associated logic to figure out if a particular image is compatible.

[stevvooe]

@philips Agreed. Although, we should define a commonly used set to keep interop for the big ones. For example, use "amd64" instead of "x86_64".

Whatever we do, we should attempt to wrangle the sillyness of hardware manufacturers (just see the list in #69 😨). More than half of that variation is in ARM and the other half are duplicates. The danger here is that we make this complicated and verbose to handle a 1% case.

Perhaps, we should start a table in the wiki and see if we can get a handle on the required dimensions. The list at https://golang.org/doc/install/source#environment will be a good start.

@glevand And that sounds fine. It seems like we should define a set of "common" architectures and if you want to go off the map and be non-standard, you can define your own but interoperability of the images will be affected.

[petrosagg]

I also believe that defining a set of common architectures is a good idea to reduce the fragmentation of the built images but at the same time leaving a user to specify a custom one.

@stevvooe The list in #69 is very big indeed, but a lot of these items in the list are actually compatible with each other. For example, a CPU that has the ARMv7+NEON instruction set can run binaries that are compiled for ARMv7 (no NEON) just fine. The same thing happens with x86 where a binary can contain very specific CPU instructions (e.g Intel i7 AES extensions), or generic ones.

Distributions usually choose to compile their binaries with the most generic settings that make sense in order to support a broad spectrum of hardware. Since most container images are usually based on some distro base image it makes to include the architectures commonly used in popular distros.

Based on my experience these are the most common architectures used today: aarch64, armv5, armv6, armv6hf, armv7ahf. OpenWRT works a lot with MIPS processors so the following could also be added: mips, mips32, mips32el, mips64, mips64el.

Having a single token for ARM (arm) is certainly not going to do it. Even the Raspberry Pi, the most popular ARM board, has a different ARM version between versions 1 and 2.

[glevand]

@petrosagg We currently have aarch64, aarch64_be, aarch64_ipl32, aarch64_ipl32_be, or are those arm64, arm64_be, arm64_ipl32, arm64_ipl32_be. Was that aarch64_ipl32 with a byte-order field, or aarch64 with a byte-order and ilp32 fields? Maybe arm64_be_ipl32?

Each of the four ABIs are different and incompatible. Some systems will be designed to run binaries of any type, but others, like specialized or low cost systems, will only be able to run one type. I want to be able to have some simple logic that uses the config file info to tell me if a container image is compatible with my system.

[petrosagg]

@glevand Between arm64 and aarch64 I think the latter is more widely used.

I wasn't aware of ipl32. Reading a bit more about it, it looks like it's not a property of the CPU itself, please correct me if I'm wrong, but a property of the ABI you select when you compile your system. I'll assume that an AArch64 CPU can run a combination of kernel + userspace regardless of ipl32 as long as both or none have been compiled with ipl32. If this isn't true please ignore the following.

The problem with containers is that you don't want to end up running a ilp32 container that will try to do syscalls to a non-ilp32 kernel as this won't work [1]. Other than that, linking between the container binaries should work, since they all have the same ABI.

So we're talking about augmenting the CPU architecture field with a specifier that describes ABI variations. I'm not sure if this should be the same or separate fields, but if we go for the same field then <cpu_arch>_<abi variation> sounds good to me: aarch64, aarch64_be, aarch64_ipl32, aarch64_be_ipl32.

[1] http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dai0490a/ar01s04.html

[alexandrosm]

Just to add a tad of urgency to this conversation, at resin.io we continuously produce ARM and i32 containers that we run on real devices. We are constantly increasing the device types and CPU architectures we support. This is why we really need to see some progress on this issue, to avoid non-standard extensions.

[glevand]

@petrosagg Yes, IPL32 it is an ABI, not a CPU property. Some cost-down CPUs don't have the hardware support for the 32 bit A32 instructions though, and with those ILP32 is used to support 32 bit programs. The kernel can be configured with or without ILP32 support.

Your proposal of <cpu_arch>_ seems OK to me.

[stevvooe]

@glevand @petrosagg What is the value of exposing the ABI in the architecture field? Why don't we have a separate field for the ABI? Again, hiding this behind a platform tag (which is really what GOARCH is), is the right approach. There is too much variation in ARM (and MIPS) to do anything else. Effectively, if "platform" is "arm", check "architecture" (aarch64, armv7), "abi", "endianness" and others. This will provide the hooks to support the current variation in arm, as well as provide for the possibility of future extensions that will inevitably come.

With the complexity of ARM's ecosystem, I don't think anything else will suffice. With a simple string, compatibility might be based on a simple comparison. With broken down fields, tools could intelligently read the fields and match up compatible images on platforms, or even fix up code to ensure it can run on a platform it wasn't compiled for.

[petrosagg]

@stevvooe That's a good point on having a separate ABI field. I'm not sure what you mean by having the "platform" being "arm" since from what I see in the spec "platform" is a key-value object. Do you propose adding a platform property like so?

"platform": {
    "os": "linux",
    "platform": "arm",
    "arch": "armv7"
}

I would put everything that describes the CPU in a single architecture field, since there are schemes for it, and anything that depends on your toolchain configuration (like ABI) in a separate fields.

"platform": {
    "os": "linux",
    "arch": "aarch64_be",
    "abi": "ipl32"
}

[stevvooe]

@petrosagg My point was that the Go env var GOARCH is actually akin to a "platform" variable rather than CPU architecture. There isn't really an "arm64" architecture, but it defines a set of platforms variants as a group. Following that grouping model will help to reduce the clutter.

I'd prefer to avoid compound fields. There isn't really a solid specification on the meaning of `. If we leave this to chance, any manner of garbage will creep in. We can simply expand it like this:

"platform": {
    "os": "linux",
    "architecture": "arm64",
    "variant": "aarch64"
    "endian": "big",
    "abi": "ipl32"
}

This will let us group all of the arm variants under either "arm" or "arm64" and provide the necessary detail to cover all the platforms. We also have the convenience of matching GOARCH/GOOS with os/architecture but have a number of variables that could further specify the variant.

I hope this sounds like a sufficient compromise.

[vbatts]

this seems related to #302 in distinguishing the platform specifics for distributing, but as far as the runtime is concerned, the endianess of a container seems like it would be less of an hurdle.

[vbatts]

just talking to @jbouzane and even on golang, the architectures that can support varying endianness it is distinguishable (i.e. ppc64 and ppc64le) see https://golang.org/doc/install/source

I'm going to close this as we will solve these as needed.