Merge existing image credentials into a fresh set

[hiddeco]

Fixes #1485

With the previous merging strategy the set of credentials which got
mutated could be assigned to other images, resulting in overwrites
where we would (and did) not expect them.

This new approach is the most direct fix to work around mutating the
credentials of other images. It creates a fresh set of (empty)
credentials and merges the others into it before assigning it to the
image.

Details

[hiddeco]

After deploying this to one of our clusters it seems to create a new issue where Flux is unable to pull images tied to HelmRelease manifests because it fails to extract credentials from this resource kind.

ts=2019-02-01T14:30:10.59171206Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-02-01T14:32:24.797392145Z caller=warming.go:192 component=warmer canonical_name=eu.gcr.io/<image> auth={map[]} err="requesting tags: denied: Failed to read tags for host 'eu.gcr.io', repository '/v2/<image>/tags/list'"
ts=2019-02-01T14:32:26.38038089Z caller=warming.go:192 component=warmer canonical_name=eu.gcr.io/<image> auth={map[]} err="requesting tags: denied: Failed to read tags for host 'eu.gcr.io', repository '/v2/<image>/tags/list'"
ts=2019-02-01T14:33:31.017196418Z caller=warming.go:192 component=warmer canonical_name=eu.gcr.io/<image> auth={map[]} err="requesting tags: denied: Failed to read tags for host 'eu.gcr.io', repository '/v2/<image>/tags/list'"

[squaremo]

Flux is unable to pull images tied to HelmRelease manifests because it fails to extract credentials from this resource kind.

fluxd doesn't try to extract secrets from HelmRelease resources (they don't have any); it relies on scanning images from resources created as part of the release. So, what is happening is fluxd doesn't pick up credentials for those.

[squaremo]

So, what is happening is fluxd doesn't pick up credentials for those.

No, I'm being silly -- what's happening is that the credentials extracted from HelmRelease (i.e., none) pre-empting the credentials from the other workloads.

This is a bug, but it's not clear what the correct solution is. Going back to merging credentials will solve it, but is not obviously correct either; excluding HelmRelease objects when scanning for images will give the right result, but seems like an arbitrary special case.

[squaremo]

Logically, I'm not sure how this changes the situation: before, credentials were merged meaning an entry for a particular registry can be overwritten (last entry wins); now, whichever workload is scanned first will supply the credentials, and no others can (first entry wins). Either way, if you have two distinct credentials against workloads using the same image, one of them will non-deterministically "win".

[squaremo]

Logically, I'm not sure how this changes the situation: before, credentials were merged meaning an entry for a particular registry can be overwritten (last entry wins); now, whichever workload is scanned first will supply the credentials, and no others can (first entry wins). Either way, if you have two distinct credentials against workloads using the same image, one of them will non-deterministically "win".

.. all of which doesn't go anywhere in countering your actual experience of it not working!

We found the problem with the existing code: https://github.com/weaveworks/flux/blob/master/cluster/kubernetes/images.go#L155 mutates a set of credentials which can be assigned to other images. The most direct fix would be to create a fresh set of credentials for each image, and merge the others into it; i.e.,

existingCreds, ok := allImageCreds[imageID]
if ok {
    mergedCreds := registry.NoCredentials()
    mergedCreds.Merge(existingCreds)
    mergedCreds.Merge(creds)
    allImageCreds[imageID] = mergedCreds
} else {
    allImageCreds[imageID] = creds
}

[hiddeco]

Force-pushed the fix mentioned above to solve the issue without leaving a trail of my original approach, which skipped the merge if there were already credentials present for an image.

[squaremo]

Looks right to me 🍎