osbuild manifest format transition

[achilleas-k]

This issue is for tracking the compromises or choices we make in composer for the transition between v1 and v2 of the osbuild manifest format. These were introduced in #1244.

ImageType package sets

The original image type defines two package sets, one for the final image delivered to the user (packages) and one for the build stage (buildPackages). The new image type can define any number of pipelines that can require more sets of packages. For the new rhel-edge-container type for instance we require three package sets: packages for the ostree commit (the "image"), buildPackages for the build stage of the commit, and a set of packages for the container that deliver the commit (currently, this is only httpd + dependencies).
For now, the httpd package is added to buildPackages for compatibility with the old ImageType interface.

Name of last pipeline

The new version of osbuild supports defining multiple artifacts to export through the --export option. At least one should be defined. When processing a v1 manifest, the value for the option must be "assembler" (or not defined at all) to achieve the old behaviouir.
For now, we define the name of the final pipelines for our new v2 image types to be "assembler" for simplicity.

osbuild result parsing

The format of osbuild's output depends on the format of the provided manifest. A custom UnmarshalJSON() method is implemented to handle both formats. The format of the output is likely to change going forward still.

---

Note that most relevant code is marked by a 1to2t (1-to-2 transition) comment.

[teg]

Thanks for opening this!

For the package sets, something we could consider is replacing the Packages() and BuildPackages() calls with one PackageSets() call that returns an array of package sets, each of which must be depsolved and passed back to Manifest(). It would then be up to each image type to keep track of the convention of which package sets means what. Too fragile?

[achilleas-k]

I don't think that's fragile no. That was my thinking exactly. Alternatively, it could be a map stage name -> package array to make it more explicit.

[teg]

I was thinking about the same about naming them (though by pipeline, rather than stage, right?). Either way, we could end up with several package sets per stage/pipeline down the line we might be back to square one. So I would put a (very small) vote on just doing the arrays.

[achilleas-k]

Right, good thinking ahead. And yes, it's pipeline name not stage name (keep making that mistake).

Keeping it as an array leaves it up to the image type to figure out the mapping between index and pipeline or index and stage, which also means the stage is more flexible to define package sets however it wants (e.g., same set in multiple pipelines or stages, sub-package sets for different situations, etc). I think that'll be useful in general.

[achilleas-k]

Bumping this issue with a few updates:

• Distro: replace Packages() / BuildPackages() with PackageSets() #1279 added support for defining named package sets (map[string]PackageSet). The names are arbitrary but internally the Manifest() builder uses them to identify which set to use with a pipeline.
• OSBuild result parsing: Currently results coming in from v2 manifests are converted to the v1 result type. I'd like to very soon change this to work the other way around (see OSBuild result parsing (v1 vs v2) #1493).
• With Hosted edge commit #1439 merged, all osbuild calls use the --export flag and the subdirectory of the --output directory to retrieve the artifact.

[thozza]

@achilleas-k is this still relevant? AFAIK we dropped v1 support from composer this year (IIRC).

[achilleas-k]

No, this is done. Closing. Thanks!