Merge pull request #18151 from rifelpet/kubescheduler-docs

Add some clarification to KubeSchedulerConfiguration

Author: k8s-ci-robot

docs/addon_objects.md

@@ -1,25 +1,71 @@
 # Passing additional configuration objects
 
-kOps has initial support for passing additional objects to the cluster, and recognizes a few "well known" objects.
+kOps supports passing additional objects to the cluster, and recognizes certain "well known" objects.
 
-This support is currently gated behind the `ClusterAddons` feature-flag (i.e. `export KOPS_FEATURE_FLAGS=ClusterAddons`)
+Objects that are not well-known will be applied directly to the cluster's k8s api.
 
-Objects that are not well-known will be applied to the cluster.  Well-known objects will have special handling.
+## Adding objects to a cluster
+
+To add an additional object to an existing cluster, use `kops create -f`:
+
+```
+kops create -f scheduler-config.yaml
+kops update cluster --name=my.cluster.k8s.local --yes --admin
+kops rolling-update cluster --name=my.cluster.k8s.local --yes
+```
+
+When creating a new cluster, you can use the `--add` flag (requires the `ClusterAddons` feature flag):
+
+```
+export KOPS_FEATURE_FLAGS=ClusterAddons
+kops create cluster --name=my.cluster.k8s.local --zones us-east-2a --add scheduler-config.yaml
+kops update cluster --name=my.cluster.k8s.local --yes --admin
+```
+
+To view the current additional objects for a cluster:
+
+```
+kops get all -o yaml
+```
 
 # Well-Known Objects
 
+Well-known objects receive special handling from kOps instead of being applied directly to the cluster.
+
 ## KubeSchedulerConfiguration (group: kubescheduler.config.k8s.io)
 
 KubeSchedulerConfiguration objects allow for custom configuration of
 kube-scheduler, the component responsible for assigning Pods to Nodes.
 
-Special handling:  the configuration will be written to a file on control plane nodes,
-and the kube-scheduler component will be configured to read from that file.
+Fields set in `spec.kubeScheduler` (see [cluster_spec.md](cluster_spec.md#kubescheduler)) are merged on top of the KubeSchedulerConfiguration object. 
+This allows you to use KubeSchedulerConfiguration for advanced settings like scheduler profiles and plugins, while using `spec.kubeScheduler` for simpler fields.
+
+Note: the `clientConnection.kubeconfig` field is managed by kOps and must not be set manually.
 
-Example usage:
+Example KubeSchedulerConfiguration file:
+
+```yaml
+# scheduler-config.yaml
+apiVersion: kubescheduler.config.k8s.io/v1
+kind: KubeSchedulerConfiguration
+profiles:
+  - schedulerName: default-scheduler
+    plugins:
+      score:
+        disabled:
+          - name: NodeResourcesBalancedAllocation
 ```
-export KOPS_FEATURE_FLAGS=ClusterAddons
-kops create cluster --name=kubescheduler.k8s.local --zones us-east-2a --add docs/examples/addons/kubescheduler.yaml
-kops update cluster --name=kubescheduler.k8s.local --yes --admin
-kops validate cluster --name=kubescheduler.k8s.local
-```
+
+Example usage with an existing cluster:
+
+```
+kops create -f scheduler-config.yaml
+kops update cluster --name=my.cluster.k8s.local --yes --admin
+kops rolling-update cluster --name=my.cluster.k8s.local --yes
+```
+
+# Other Kubernetes Objects
+
+Any Kubernetes object that is not a well-known type (such as ConfigMaps, Deployments,
+DaemonSets, Services, RBAC resources, etc.) will be applied directly to the cluster
+as a standard Kubernetes resource during `kops update cluster`.

docs/cluster_spec.md

@@ -867,11 +867,11 @@ This block contains configurations for `kube-scheduler`.  See https://kubernetes
 ```yaml
 spec:
   kubeScheduler:
-    usePolicyConfigMap: true
     enableProfiling: false
 ```
 
-Will make kube-scheduler use the scheduler policy from configmap "scheduler-policy" in namespace kube-system.
+For advanced scheduler configuration (e.g. custom profiles and plugins), see
+[addon_objects.md](addon_objects.md#kubeschedulerconfiguration-group-kubeschedulerconfigk8sio).
 
 ### LogFormat
 

docs/examples/addons/kubescheduler.yaml

@@ -1,6 +1,11 @@
-# This is an example "addon" object to provide additional/custom kube-scheduler configuration
-apiVersion: kubescheduler.config.k8s.io/v1beta2
+# This is an example "addon" object to provide custom kube-scheduler configuration.
+# Use with: kops create -f kubescheduler.yaml
+# Note: clientConnection.kubeconfig is managed by kOps and must not be set.
+apiVersion: kubescheduler.config.k8s.io/v1
 kind: KubeSchedulerConfiguration
-clientConnection:
-  burst: 100
-  qps: 3.1
+profiles:
+  - schedulerName: default-scheduler
+    plugins:
+      score:
+        disabled:
+          - name: NodeResourcesBalancedAllocation