Declarative Management of Kubernetes Objects Using Kustomize
Kustomize is a standalone tool to customize Kubernetes objects through a kustomization file.
Since 1.14, Kubectl also supports the management of Kubernetes objects using a kustomization file. To view Resources found in a directory containing a kustomization file, run the following command:
kubectl kustomize <kustomization_directory>
To apply those Resources, run kubectl apply
with --kustomize
or -k
flag:
kubectl apply -k <kustomization_directory>
Before you begin
Install kubectl
.
You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:
To check the version, enterkubectl version
.Overview of Kustomize
Kustomize is a tool for customizing Kubernetes configurations. It has the following features to manage application configuration files:
- generating resources from other sources
- setting cross-cutting fields for resources
- composing and customizing collections of resources
Generating Resources
ConfigMaps and Secrets hold configuration or sensitive data that are used by other Kubernetes objects, such as Pods. The source of truth of ConfigMaps or Secrets are usually external to a cluster, such as a .properties
file or an SSH keyfile.
Kustomize has secretGenerator
and configMapGenerator
, which generate Secret and ConfigMap from files or literals.
configMapGenerator
To generate a ConfigMap from a file, add an entry to the files
list in configMapGenerator
. Here is an example of generating a ConfigMap with a data item from a .properties
file:
# Create a application.properties file
cat <<EOF >application.properties
FOO=Bar
EOF
cat <<EOF >./kustomization.yaml
configMapGenerator:
- name: example-configmap-1
files:
- application.properties
EOF
The generated ConfigMap can be examined with the following command:
kubectl kustomize ./
The generated ConfigMap is:
apiVersion: v1
data:
application.properties: |
FOO=Bar
kind: ConfigMap
metadata:
name: example-configmap-1-8mbdf7882g
To generate a ConfigMap from an env file, add an entry to the envs
list in configMapGenerator
. Here is an example of generating a ConfigMap with a data item from a .env
file:
# Create a .env file
cat <<EOF >.env
FOO=Bar
EOF
cat <<EOF >./kustomization.yaml
configMapGenerator:
- name: example-configmap-1
envs:
- .env
EOF
The generated ConfigMap can be examined with the following command:
kubectl kustomize ./
The generated ConfigMap is:
apiVersion: v1
data:
FOO: Bar
kind: ConfigMap
metadata:
name: example-configmap-1-42cfbf598f
Note:
Each variable in the.env
file becomes a separate key in the ConfigMap that you generate. This is different from the previous example which embeds a file named application.properties
(and all its entries) as the value for a single key.ConfigMaps can also be generated from literal key-value pairs. To generate a ConfigMap from a literal key-value pair, add an entry to the literals
list in configMapGenerator. Here is an example of generating a ConfigMap with a data item from a key-value pair:
cat <<EOF >./kustomization.yaml
configMapGenerator:
- name: example-configmap-2
literals:
- FOO=Bar
EOF
The generated ConfigMap can be checked by the following command:
kubectl kustomize ./
The generated ConfigMap is:
apiVersion: v1
data:
FOO: Bar
kind: ConfigMap
metadata:
name: example-configmap-2-g2hdhfc6tk
To use a generated ConfigMap in a Deployment, reference it by the name of the configMapGenerator. Kustomize will automatically replace this name with the generated name.
This is an example deployment that uses a generated ConfigMap:
# Create a application.properties file
cat <<EOF >application.properties
FOO=Bar
EOF
cat <<EOF >deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
labels:
app: my-app
spec:
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: app
image: my-app
volumeMounts:
- name: config
mountPath: /config
volumes:
- name: config
configMap:
name: example-configmap-1
EOF
cat <<EOF >./kustomization.yaml
resources:
- deployment.yaml
configMapGenerator:
- name: example-configmap-1
files:
- application.properties
EOF
Generate the ConfigMap and Deployment:
kubectl kustomize ./
The generated Deployment will refer to the generated ConfigMap by name:
apiVersion: v1
data:
application.properties: |
FOO=Bar
kind: ConfigMap
metadata:
name: example-configmap-1-g4hk9g2ff8
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: my-app
name: my-app
spec:
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- image: my-app
name: app
volumeMounts:
- mountPath: /config
name: config
volumes:
- configMap:
name: example-configmap-1-g4hk9g2ff8
name: config
secretGenerator
You can generate Secrets from files or literal key-value pairs. To generate a Secret from a file, add an entry to the files
list in secretGenerator
. Here is an example of generating a Secret with a data item from a file:
# Create a password.txt file
cat <<EOF >./password.txt
username=admin
password=secret
EOF
cat <<EOF >./kustomization.yaml
secretGenerator:
- name: example-secret-1
files:
- password.txt
EOF
The generated Secret is as follows:
apiVersion: v1
data:
password.txt: dXNlcm5hbWU9YWRtaW4KcGFzc3dvcmQ9c2VjcmV0Cg==
kind: Secret
metadata:
name: example-secret-1-t2kt65hgtb
type: Opaque
To generate a Secret from a literal key-value pair, add an entry to literals
list in secretGenerator
. Here is an example of generating a Secret with a data item from a key-value pair:
cat <<EOF >./kustomization.yaml
secretGenerator:
- name: example-secret-2
literals:
- username=admin
- password=secret
EOF
The generated Secret is as follows:
apiVersion: v1
data:
password: c2VjcmV0
username: YWRtaW4=
kind: Secret
metadata:
name: example-secret-2-t52t6g96d8
type: Opaque
Like ConfigMaps, generated Secrets can be used in Deployments by referring to the name of the secretGenerator:
# Create a password.txt file
cat <<EOF >./password.txt
username=admin
password=secret
EOF
cat <<EOF >deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
labels:
app: my-app
spec:
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: app
image: my-app
volumeMounts:
- name: password
mountPath: /secrets
volumes:
- name: password
secret:
secretName: example-secret-1
EOF
cat <<EOF >./kustomization.yaml
resources:
- deployment.yaml
secretGenerator:
- name: example-secret-1
files:
- password.txt
EOF
generatorOptions
The generated ConfigMaps and Secrets have a content hash suffix appended. This ensures that a new ConfigMap or Secret is generated when the contents are changed. To disable the behavior of appending a suffix, one can use generatorOptions
. Besides that, it is also possible to specify cross-cutting options for generated ConfigMaps and Secrets.
cat <<EOF >./kustomization.yaml
configMapGenerator:
- name: example-configmap-3
literals:
- FOO=Bar
generatorOptions:
disableNameSuffixHash: true
labels:
type: generated
annotations:
note: generated
EOF
Runkubectl kustomize ./
to view the generated ConfigMap:
apiVersion: v1
data:
FOO: Bar
kind: ConfigMap
metadata:
annotations:
note: generated
labels:
type: generated
name: example-configmap-3
Setting cross-cutting fields
It is quite common to set cross-cutting fields for all Kubernetes resources in a project. Some use cases for setting cross-cutting fields:
- setting the same namespace for all Resources
- adding the same name prefix or suffix
- adding the same set of labels
- adding the same set of annotations
Here is an example:
# Create a deployment.yaml
cat <<EOF >./deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
EOF
cat <<EOF >./kustomization.yaml
namespace: my-namespace
namePrefix: dev-
nameSuffix: "-001"
labels:
- pairs:
app: bingo
includeSelectors: true
commonAnnotations:
oncallPager: 800-555-1212
resources:
- deployment.yaml
EOF
Run kubectl kustomize ./
to view those fields are all set in the Deployment Resource:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
oncallPager: 800-555-1212
labels:
app: bingo
name: dev-nginx-deployment-001
namespace: my-namespace
spec:
selector:
matchLabels:
app: bingo
template:
metadata:
annotations:
oncallPager: 800-555-1212
labels:
app: bingo
spec:
containers:
- image: nginx
name: nginx
Composing and Customizing Resources
It is common to compose a set of Resources in a project and manage them inside the same file or directory. Kustomize offers composing Resources from different files and applying patches or other customization to them.
Composing
Kustomize supports composition of different resources. The resources
field, in the kustomization.yaml
file, defines the list of resources to include in a configuration. Set the path to a resource's configuration file in the resources
list.
Here is an example of an NGINX application comprised of a Deployment and a Service:
# Create a deployment.yaml file
cat <<EOF > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80
EOF
# Create a service.yaml file
cat <<EOF > service.yaml
apiVersion: v1
kind: Service
metadata:
name: my-nginx
labels:
run: my-nginx
spec:
ports:
- port: 80
protocol: TCP
selector:
run: my-nginx
EOF
# Create a kustomization.yaml composing them
cat <<EOF >./kustomization.yaml
resources:
- deployment.yaml
- service.yaml
EOF
The Resources from kubectl kustomize ./
contain both the Deployment and the Service objects.
Customizing
Patches can be used to apply different customizations to Resources. Kustomize supports different patching
mechanisms through StrategicMerge
and Json6902
using the patches
field. patches
may be a file or
an inline string, targeting a single or multiple resources.
The patches
field contains a list of patches applied in the order they are specified. The patch target
selects resources by group
, version
, kind
, name
, namespace
, labelSelector
and annotationSelector
.
Small patches that do one thing are recommended. For example, create one patch for increasing the deployment
replica number and another patch for setting the memory limit. The target resource is matched using group
, version
,
kind
, and name
fields from the patch file.
# Create a deployment.yaml file
cat <<EOF > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80
EOF
# Create a patch increase_replicas.yaml
cat <<EOF > increase_replicas.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 3
EOF
# Create another patch set_memory.yaml
cat <<EOF > set_memory.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
template:
spec:
containers:
- name: my-nginx
resources:
limits:
memory: 512Mi
EOF
cat <<EOF >./kustomization.yaml
resources:
- deployment.yaml
patches:
- path: increase_replicas.yaml
- path: set_memory.yaml
EOF
Run kubectl kustomize ./
to view the Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 3
selector:
matchLabels:
run: my-nginx
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- image: nginx
name: my-nginx
ports:
- containerPort: 80
resources:
limits:
memory: 512Mi
Not all Resources or fields support strategicMerge
patches. To support modifying arbitrary fields in arbitrary Resources,
Kustomize offers applying JSON patch through Json6902
.
To find the correct Resource for a Json6902
patch, it is mandatory to specify the target
field in kustomization.yaml
.
For example, increasing the replica number of a Deployment object can also be done through Json6902
patch. The target resource
is matched using group
, version
, kind
, and name
from the target
field.
# Create a deployment.yaml file
cat <<EOF > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80
EOF
# Create a json patch
cat <<EOF > patch.yaml
- op: replace
path: /spec/replicas
value: 3
EOF
# Create a kustomization.yaml
cat <<EOF >./kustomization.yaml
resources:
- deployment.yaml
patches:
- target:
group: apps
version: v1
kind: Deployment
name: my-nginx
path: patch.yaml
EOF
Run kubectl kustomize ./
to see the replicas
field is updated:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 3
selector:
matchLabels:
run: my-nginx
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- image: nginx
name: my-nginx
ports:
- containerPort: 80
In addition to patches, Kustomize also offers customizing container images or injecting field values from other objects into containers
without creating patches. For example, you can change the image used inside containers by specifying the new image in the images
field
in kustomization.yaml
.
cat <<EOF > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80
EOF
cat <<EOF >./kustomization.yaml
resources:
- deployment.yaml
images:
- name: nginx
newName: my.image.registry/nginx
newTag: 1.4.0
EOF
Run kubectl kustomize ./
to see that the image being used is updated:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 2
selector:
matchLabels:
run: my-nginx
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- image: my.image.registry/nginx:1.4.0
name: my-nginx
ports:
- containerPort: 80
Sometimes, the application running in a Pod may need to use configuration values from other objects. For example,
a Pod from a Deployment object need to read the corresponding Service name from Env or as a command argument.
Since the Service name may change as namePrefix
or nameSuffix
is added in the kustomization.yaml
file. It is
not recommended to hard code the Service name in the command argument. For this usage, Kustomize can inject
the Service name into containers through replacements
.
# Create a deployment.yaml file (quoting the here doc delimiter)
cat <<'EOF' > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
command: ["start", "--host", "MY_SERVICE_NAME_PLACEHOLDER"]
EOF
# Create a service.yaml file
cat <<EOF > service.yaml
apiVersion: v1
kind: Service
metadata:
name: my-nginx
labels:
run: my-nginx
spec:
ports:
- port: 80
protocol: TCP
selector:
run: my-nginx
EOF
cat <<EOF >./kustomization.yaml
namePrefix: dev-
nameSuffix: "-001"
resources:
- deployment.yaml
- service.yaml
replacements:
- source:
kind: Service
name: my-nginx
fieldPath: metadata.name
targets:
- select:
kind: Deployment
name: my-nginx
fieldPaths:
- spec.template.spec.containers.0.command.2
EOF
Run kubectl kustomize ./
to see that the Service name injected into containers is dev-my-nginx-001
:
apiVersion: apps/v1
kind: Deployment
metadata:
name: dev-my-nginx-001
spec:
replicas: 2
selector:
matchLabels:
run: my-nginx
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- command:
- start
- --host
- dev-my-nginx-001
image: nginx
name: my-nginx
Bases and Overlays
Kustomize has the concepts of bases and overlays. A base is a directory with a kustomization.yaml
, which contains a
set of resources and associated customization. A base could be either a local directory or a directory from a remote repo,
as long as a kustomization.yaml
is present inside. An overlay is a directory with a kustomization.yaml
that refers to other
kustomization directories as its bases
. A base has no knowledge of an overlay and can be used in multiple overlays.
The kustomization.yaml
in a overlay directory may refer to multiple bases
, combining all the resources defined
in these bases into a unified configuration. Additionally, it can apply customizations on top of these resources to meet specific
requirements.
Here is an example of a base:
# Create a directory to hold the base
mkdir base
# Create a base/deployment.yaml
cat <<EOF > base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
EOF
# Create a base/service.yaml file
cat <<EOF > base/service.yaml
apiVersion: v1
kind: Service
metadata:
name: my-nginx
labels:
run: my-nginx
spec:
ports:
- port: 80
protocol: TCP
selector:
run: my-nginx
EOF
# Create a base/kustomization.yaml
cat <<EOF > base/kustomization.yaml
resources:
- deployment.yaml
- service.yaml
EOF
This base can be used in multiple overlays. You can add different namePrefix
or other cross-cutting fields
in different overlays. Here are two overlays using the same base.
mkdir dev
cat <<EOF > dev/kustomization.yaml
resources:
- ../base
namePrefix: dev-
EOF
mkdir prod
cat <<EOF > prod/kustomization.yaml
resources:
- ../base
namePrefix: prod-
EOF
How to apply/view/delete objects using Kustomize
Use --kustomize
or -k
in kubectl
commands to recognize Resources managed by kustomization.yaml
.
Note that -k
should point to a kustomization directory, such as
kubectl apply -k <kustomization directory>/
Given the following kustomization.yaml
,
# Create a deployment.yaml file
cat <<EOF > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80
EOF
# Create a kustomization.yaml
cat <<EOF >./kustomization.yaml
namePrefix: dev-
labels:
- pairs:
app: my-nginx
includeSelectors: true
resources:
- deployment.yaml
EOF
Run the following command to apply the Deployment object dev-my-nginx
:
> kubectl apply -k ./
deployment.apps/dev-my-nginx created
Run one of the following commands to view the Deployment object dev-my-nginx
:
kubectl get -k ./
kubectl describe -k ./
Run the following command to compare the Deployment object dev-my-nginx
against the state that the cluster would be in if the manifest was applied:
kubectl diff -k ./
Run the following command to delete the Deployment object dev-my-nginx
:
> kubectl delete -k ./
deployment.apps "dev-my-nginx" deleted
Kustomize Feature List
Field | Type | Explanation |
---|---|---|
bases | []string | Each entry in this list should resolve to a directory containing a kustomization.yaml file |
commonAnnotations | map[string]string | annotations to add to all resources |
commonLabels | map[string]string | labels to add to all resources and selectors |
configMapGenerator | []ConfigMapArgs | Each entry in this list generates a ConfigMap |
configurations | []string | Each entry in this list should resolve to a file containing Kustomize transformer configurations |
crds | []string | Each entry in this list should resolve to an OpenAPI definition file for Kubernetes types |
generatorOptions | GeneratorOptions | Modify behaviors of all ConfigMap and Secret generator |
images | []Image | Each entry is to modify the name, tags and/or digest for one image without creating patches |
labels | map[string]string | Add labels without automically injecting corresponding selectors |
namePrefix | string | value of this field is prepended to the names of all resources |
nameSuffix | string | value of this field is appended to the names of all resources |
patchesJson6902 | []Patch | Each entry in this list should resolve to a Kubernetes object and a Json Patch |
patchesStrategicMerge | []string | Each entry in this list should resolve a strategic merge patch of a Kubernetes object |
replacements | []Replacements | copy the value from a resource's field into any number of specified targets. |
resources | []string | Each entry in this list must resolve to an existing resource configuration file |
secretGenerator | []SecretArgs | Each entry in this list generates a Secret |
vars | []Var | Each entry is to capture text from one resource's field |