Cannot configure securityContext of sync and cleanup jobs

[RGPosadas]

Summary
Sync and cleanup AppRepo jobs are currently not configureable to pass in container/pod SecurityContext.

Background and rationale
We are in the middle of PSP to PSA migration.
Currently, all core kubeapps components allow us to configure the container/pod SecurityContext - except for the Sync and Cleanup apprepository jobs.
Since we cannot configure the SecurityContext, the sync and cleanup jobs will be blocked from spawning pods if the namespace that kubeapps is running on has the restricted PodSecurity standard.

Description/Acceptance Criteria
Allow users to configure container/pod SecurityContext of the sync and cleanup jobs.

Extra Info
For apprepos that are defined as initialRepos, we are able to patch the AppRepository manifests, which allows us to define container/pod securityContext for the sync jobs. However, we do not have the same ability for the cleanup jobs.

[absoludity]

I just want to confirm that, when creating the AppRepository in Kubeapps, that you did specify the sync-job pod template? Asking because I can see in the code that we do intend to use that as the basis for the pod template used by the cron job that's created:

kubeapps/cmd/apprepository-controller/server/controller.go

Lines 627 to 656 in 23dac7b

	// Get the predefined pod spec for the apprepo definition if exists
	podTemplateSpec := apprepo.Spec.SyncJobPodTemplate
	// Add labels
	if len(podTemplateSpec.ObjectMeta.Labels) == 0 {
	podTemplateSpec.ObjectMeta.Labels = map[string]string{}
	}
	for k, v := range jobLabels(apprepo, config) {
	podTemplateSpec.ObjectMeta.Labels[k] = v
	}
	podTemplateSpec.ObjectMeta.Annotations = config.ParsedCustomAnnotations
	// If there's an issue, will restart pod until successful or replaced
	// by another instance of the job scheduled by the cronjob
	// see: cronJobSpec.concurrencyPolicy
	podTemplateSpec.Spec.RestartPolicy = corev1.RestartPolicyOnFailure
	// Populate container spec
	if len(podTemplateSpec.Spec.Containers) == 0 {
	podTemplateSpec.Spec.Containers = []corev1.Container{{}}
	}
	// Populate ImagePullSecrets spec
	podTemplateSpec.Spec.ImagePullSecrets = append(podTemplateSpec.Spec.ImagePullSecrets, config.ImagePullSecretsRefs...)
	podTemplateSpec.Spec.Containers[0].Name = "sync"
	podTemplateSpec.Spec.Containers[0].Image = config.RepoSyncImage
	podTemplateSpec.Spec.Containers[0].ImagePullPolicy = corev1.PullIfNotPresent
	podTemplateSpec.Spec.Containers[0].Command = []string{config.RepoSyncCommand}
	podTemplateSpec.Spec.Containers[0].Args = apprepoSyncJobArgs(apprepo, config)
	podTemplateSpec.Spec.Containers[0].Env = append(podTemplateSpec.Spec.Containers[0].Env, apprepoSyncJobEnvVars(apprepo, config)...)
	podTemplateSpec.Spec.Containers[0].VolumeMounts = append(podTemplateSpec.Spec.Containers[0].VolumeMounts, volumeMounts...)
	// Add volumes
	podTemplateSpec.Spec.Volumes = append(podTemplateSpec.Spec.Volumes, volumes...)

See https://kubeapps.dev/docs/latest/tutorials/managing-package-repositories/#modifying-the-synchronization-job for more info about customising the sync job pod template.

Can you please paste your AppRepository yaml to compare? (redacted if necessary)

Thanks

[RGPosadas]

Sample values definition:

  apprepository:
    containerSecurityContext:
      runAsUser: 10001
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
      runAsNonRoot: true
      seccompProfile:
        type: RuntimeDefault

Templated AppRepository:

apiVersion: kubeapps.com/v1alpha1
kind: AppRepository
metadata:
  labels:
    app.kubernetes.io/instance: apps
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kubeapps
    helm.sh/chart: kubeapps-12.4.6
  name: test-apprepo
  namespace: test-apprepo
spec:
  auth:
    header:
      secretKeyRef:
        key: authorizationHeader
        name: test-apprepo
  ociRepositories:
  - <redacted>
  syncJobPodTemplate:
    spec:
      securityContext:
        runAsUser: 10001
  type: oci
  url: <redacted>

Only the securityContext.runAsUser gets propagated down, as seen in https://github.com/vmware-tanzu/kubeapps/blob/main/chart/kubeapps/templates/apprepository/apprepositories.yaml#L42-L45

[RGPosadas]

Sorry I misread what you wrote, but I'll leave my previous comment.

Yes we already modify the AppRepository sync job via Kustomize so that is handled fine (although it would be nice to have this configureable in the helm chart instead of patching the CRD). However, we are unable to do the same patch with the cleanup job.

Asking because I can see in the code that we do intend to use that as the basis for the pod template used by the cron job that's created:

Did you mean that the cleanup and sync jobs both use the same podTemplate? Or does this only affect the sync job

[absoludity]

Yes - the chart options try to provide simple options that the user can use for specific tasks, and do not currently allow specifying a general syncPodTemplate, but as you've seen, you can specify that yourself by specifying the AppRepository in full (you don't need to patch the CR, most people just don't use the chart to define the CR in this case, but rather just write the CR - or generate it with kustomize - for their AppRepository).

And you are absolutely correct: the specified syncPodTemplate on the AppRepository CR is used successfully when generating the Job for the syncs, but is not currently used for generating the Job for the clean-ups. That is an easy fix that we should include :)

[RGPosadas]

Great, thanks for confirming!

[antgamdia]

That is an easy fix that we should include :)

Well... it turned out not to be that simple :P, the details of the affected AppRepo are no longer available in the cluster (wherefore it is clean-up job). I have proposed an alternative approach in #6605. See what you all think!

[antgamdia]

HI @RGPosadas, we ended up performing some refactor on the AppRepository controller, so instead of being the controller the responsible of creating clean-up jobs, it is a finalizer who will handle that. That means all the information already available in the AppRepo CR (like the pod template), can now be inherit by the clean-up jobs. In short: those jobs will have the security context set!

Feel free to use our dev chart (not Bitnami's, but the one in this repo) and the latest kubeapps/apprepository-controller container image if you want to give it a try now. Anyways, next Kubeapps release will include this change.

Thanks again for reporting the issue!

[RGPosadas]

@antgamdia Thanks for the quick work! Hopefully I'll have time to test it out before the official release to report back 🚀