Kubeadm does not provide mount point for scheduler policy-config-file

[gnavgire]

Is this a BUG REPORT

Choose one: BUG REPORT

Versions

kubeadm version (use kubeadm version): 1.8

Environment:

• Kubernetes version (use kubectl version):1.8
• Cloud provider or hardware configuration: intel nuc
• OS (e.g. from /etc/os-release): ubuntu-16.0.4
• Kernel (e.g. uname -a):4.10.0-37
• Others:

What happened?

kubeadm init can be provided a config file as below:
kubeadm init --config kubeadm-policy-new.yaml
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
schedulerExtraArgs:
policy-config-file: "/etc/kubernetes/sched_policy.json" <<< which tells kubeadm to start scheduler with policy.json file
This policy-config-file is not made available to the scheduler.

What you expected to happen?

The scheduler is succesfuly started with this configuration:
kube-scheduler --policy-config-file=/etc/kubernetes/sched_policy.json

however there is no /etc/kubernetes/sched_policy.json file mounted to the scheduler container. as a result the scheduler does not acknowledge the parameters in the policy file.

How to reproduce it (as minimally and precisely as possible)?

start kubeadm init with config file that contains parameter for scheduler args as shown below:
kubeadm init --config kubeadm-policy-new.yaml
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
schedulerExtraArgs:
policy-config-file: "/etc/kubernetes/sched_policy.json" <<< which tells kubeadm to start scheduler with policy.json file

Anything else we need to know?

After creating a new container image using the base scheduler and copying the policy file, made the policy to be available for the scheduler. Below is the sample dockerfile used to create the new scheduler image:
'FROM gcr.io/google_containers/kube-scheduler-amd64:v1.8.2
COPY sched_policy.json /etc/kubernetes/sched_policy.json'

and use the local repository image option in the kubeadm config file as below:
'apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
schedulerExtraArgs:
policy-config-file: "/etc/kubernetes/sched_policy.json"
imageRepository: localhost:5000'

[tamalsaha]

This issue if resolved by https://github.com/kubernetes/kubernetes

See discussion: #484

[gnavgire]

Does it require any other config to be used in the kubeadm-config.yaml ??

[gnavgire]

The scheduler container yaml still does not contain the additional volume mount:

---

spec:
containers:

• command:
  • kube-scheduler
  • --policy-config-file=/opt/k8s/cfg/sched_policy.json
  • --address=127.0.0.1
  • --leader-elect=true
  • --kubeconfig=/etc/kubernetes/scheduler.conf
    env:
    image: gcr.io/google_containers/kube-scheduler-amd64:v1.8.2
    livenessProbe:
    failureThreshold: 8
    httpGet:
    host: 127.0.0.1
    path: /healthz
    port: 10251
    scheme: HTTP
    initialDelaySeconds: 15
    timeoutSeconds: 15
    name: kube-scheduler
    resources:
    requests:
    cpu: 100m
    volumeMounts:
  • mountPath: /etc/kubernetes/scheduler.conf
    name: kubeconfig
    readOnly: true
    hostNetwork: true
    volumes:
• hostPath:
  path: /etc/kubernetes/scheduler.conf
  type: FileOrCreate
  name: kubeconfig
  status: {}

---

[gnavgire]

After providing the below configuration in the kubeadm config file:

---

schedulerExtraVolumes:

• name : "sched_policy"
  hostPath : "/opt/k8s/cfg/sched_policy.json"
  mountPath : "/opt/k8s/cfg/sched_policy.json

---

The below kube-scheduler.yaml got created, however kubeadm is not able to start the kube-scheduler

---

apiVersion: v1
kind: Pod
metadata:
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ""
creationTimestamp: null
labels:
component: kube-scheduler
tier: control-plane
name: kube-scheduler
namespace: kube-system
spec:
containers:

• command:
  • kube-scheduler
  • --policy-config-file=/opt/k8s/cfg/sched_policy.json
  • --address=127.0.0.1
  • --leader-elect=true
  • --kubeconfig=/etc/kubernetes/scheduler.conf
    env:
  • name: NO_PROXY
    value: 10.242.131.203
    image: gcr.io/google_containers/kube-scheduler-amd64:v1.8.2
    livenessProbe:
    failureThreshold: 8
    httpGet:
    host: 127.0.0.1
    path: /healthz
    port: 10251
    scheme: HTTP
    initialDelaySeconds: 15
    timeoutSeconds: 15
    name: kube-scheduler
    resources:
    requests:
    cpu: 100m
    apiVersion: v1
    kind: Pod
    metadata:
    annotations:
    scheduler.alpha.kubernetes.io/critical-pod: ""
    creationTimestamp: null
    labels:
    component: kube-scheduler
    tier: control-plane
    name: kube-scheduler
    namespace: kube-system
    spec:
    containers:
• command:
  • kube-scheduler
  • --policy-config-file=/opt/k8s/cfg/sched_policy.json
  • --address=127.0.0.1
  • --leader-elect=true
  • --kubeconfig=/etc/kubernetes/scheduler.conf
    env:
  • name: NO_PROXY
    value: 10.242.131.203
    image: gcr.io/google_containers/kube-scheduler-amd64:v1.8.2
    livenessProbe:
    failureThreshold: 8
    httpGet:
    host: 127.0.0.1
    path: /healthz
    port: 10251
    scheme: HTTP
    initialDelaySeconds: 15
    timeoutSeconds: 15
    name: kube-scheduler
    resources:
    requests:
    cpu: 100m
    volumeMounts:
  • mountPath: /opt/k8s/cfg/sched_policy.json
    name: sched_policy
    readOnly: true
  • mountPath: /etc/kubernetes/scheduler.conf
    name: kubeconfig
    readOnly: true
    hostNetwork: true
    volumes:
• hostPath:
  path: /etc/kubernetes/scheduler.conf
  type: FileOrCreate
  name: kubeconfig
• hostPath:
  path: /opt/k8s/cfg/sched_policy.json
  type: DirectoryOrCreate
  name: sched_policy
  status: {}

---

[gnavgire]

Making the below changes to the kubeadm config file:
schedulerExtraVolumes:

• name : "schedpolicy"
  hostPath : "/opt/k8s/cfg/"
  mountPath : "/opt/k8s/cfg/"

---

Made the kube-scheduler to be scheduled successfully.
Thanks for the Help!!