This primer is intended to give a brief overview of helm, how to use it, how it interacts with kubernetes, and some recommendations and best practices that make it easier to use.

The goal here is to fill the gap between A) a basic ‘hello world' example, and B) a detailed understanding of the finer points of helm. Additionally, the focus here is on using a helm chart, not building a helm chart.

In all things, feel free to go straight to the source: https://helm.sh/.

What is helm?

helm is a command line tool that greatly simplifies the deployment and management of kubernetes objects. Kubernetes objects are defined/managed via yaml manifest files. These files could all be managed by hand. However, suppose you wanted to make sure that each object had a common annotation set. You’d have to manually update each file. helm provides a way to generate the manifest files, and could allow a common annotation to be specified once and then automatically applied to all the kubernetes objects.

Additionally, helm can be used to deploy those manifest files directly to a kubernetes cluster. In fact, most helm commands don’t even bother to create local copies of the manifest files. The point is, helm isn’t doing anything ‘special’ - it’s creating manifests and applying them to kubernetes.

What is a helm chart?

At a basic level, a helm chart is instructions to build out a set of kubernetes objects. These instructions consist of two main parts:

1. Templates

2. Values

Templates are implemented via Go templates (https://pkg.go.dev/text/template ). They fill out the basic form of a particular kubernetes object (a deployment or a service, for example). Assuming your helm chart is called ‘my-chart’, the template files are all found in the my-chart/templates/ directory of your installed chart. Templates are filled out and turned into manifests based on the supplied values.

Values are the specific values that are used to fill out template files, turning them into valid kubernetes manifests, which can then be applied to a particular kubernetes cluster. Values are specified via one or more yaml files, with the chart defaults located in my-chart/values.yaml. Due to the flexible nature of templates themselves, values can be pretty much anything. They can include things like:

• a list of microservices that should be deployed

• environment variables that should be specified on particular pods

• annotations that should be placed on particular kubernetes objects

Are all helm charts alike?

Nope. This is part of the… fun. Any two helm charts can be as different from one another as any two java programs. They’ll all share similar syntax (templates and a yaml values file), but the content and style of each chart is entirely up to the creator of the chart.

Ok... but I don't want to install 'just the defaults'. 🤨

Good! That would be boring. Probably.

Fortunately, helm allows you to dynamically set different values. The easiest way to do this is to create a ‘values override file’. This is just another yaml file that specifies different/additional values that override whatever’s found in the default my-chart/values.yaml file. The format of an override file is exactly the same as the format of the default values file. The override file is specified on the helm command line via the -f option.

Consider the following defaults:

some:
  value: abc
  other: foo
my:
  list:
    - alpha
    - beta
  dictionary:
    rose: red
    violet: blue

Now consider the following override file:

some:
  value: xyz
my:
  list:
    - gamma
    - delta
  dictionary:
    chrysanthemum: yellow

When the given override file is applied over the top of the defaults, the values of some.value and my.list will be completely replaced, while the values of my.dictionary will be added:

some:
  value: xyz
  other: foo
my:
  list:
    - gamma
    - delta
  dictionary:
    rose: red
    violet: blue
    chrysanthemum: yellow

But... why don't I just modify the contents of my-chart/values.yaml?

You could! And it would all work fine. Until one day, when you decided you wanted to update to a newer version of your helm chart. Chances are, the new chart would have additions to its my-chart/values.yaml file. So you’d have to reconcile those updates with the local changes to your original my-chart/values.yaml. “Not much fun for little Harpo.” 🐟 By placing all your overridden values into a separate file, you are free to update to newer versions of a helm chart, and so long as the new chart is backwards compatible with your current version, you should be able to apply your override file over the top of the new chart.

Therefore:

• Always set/modify values in a separate override file

• Never modify the contents of the my-chart/values.yaml file

What is a 'release'?

Simply put, a release is the combination of:

1. a particular helm chart

2. a particular override file (optional)

3. a helm install command

The chart + override file tells helm how to produce the appropriate kubernetes manifests, and then those manifests are applied to the currently configured kubernetes cluster. The result is that there are now a number of kubernetes objects (deployments, services, secrets, configmaps, etc…) created in your cluster. At this point, helm's work is largely over. It’s up to kubernetes to create pods based on deployments, to properly wire up services to backend pods, to mount secrets on those pods, etc…

The last thing a helm install creates is a kubernetes secret that contains information about the current release. helm will keep a few of the recent ‘releases’ stored in various versioned kubernetes secrets for a couple reasons. One, there’s a helm rollback command that can be used to rollback to previous versions of a release. Two, and perhaps more importantly, the content of the most recent helm kubernetes secret is used when upgrading a release. Once you’ve run the first helm install command, subsequent modifications of the release are made with a helm upgrade command. Say you decide to add a new environment variable to a pod. You make the updates to your override file, and hit helm upgrade. helm does a 3 way merge of A) the new manifests generated by the latest update to the override file, B) the contents of the most recent helm kubernetes secret, and C) the actual manifests of the various deployments/services running in kubernetes. What this means is that helm attempts to respect local modifications to kubernetes objects otherwise managed by helm. For example, if a particular service’s type was originally set to ClusterIP, but later changed to LoadBalancer, not by modifying the override file, but by modifying the running manifest directly (via kubectl edit), a subsequent helm upgrade may leave the service’s type set to LoadBalancer, even though the chart might still want to set it to ClusterIP. The difference between the actual manifest running in kubernetes and the information stored in the kubernetes secret allows helm to recognize that local changes have been made, out of band from helm, and attempt to leave those changes in place, when possible.

Where should I store my override file?

The chart itself should be static and not require any changes. But the override file will, at least initially, see a lot of action, and will need to be updated frequently.

The most important bit of advice concerning managing your override file is to only have a single copy of the override file. As soon as you have multiple copies of the file, things get out of sync, one person runs a helm upgrade command against their updated version, but then someone else’s changes get lost. It’s a recipe for disaster.

Dogs and cats living together… mass hysteria! - Peter Venkman

The easiest way to satisfy the ‘single copy’ rule is to keep your override file in some sort of change management system - i.e. a git repository. Any time one wishes to affect the system, one would checkout the latest version of the file, make their changes, commit their changes, then run helm upgrade. This ensures that the environment always reflects the latest override file checked into the change management system. Furthermore, it becomes trivial to look at the history of that file, see who changed what, why they changed it, etc…

In a perfect world, the above flow would be managed by some sort of CI/CD system, where pushing a change to the override file would trigger a job that would pull in the latest version of the file and run the helm upgrade command. That way, the only way to interact with the system would be by interactions with the override file, and those interactions are tracked, etc…

In a slightly less perfect world, the only copy of the override file would be in a single, shared space (like a shared /home directory somewhere).

In a dystopian world, filled with weeping and gnashing of teeth, there would be multiple copies of the override file, all with a disjoint set of changes. Avoid this dystopian world.

This is awesome! But I don't want to store credentials in my override file.

For quick POCs and such, it’s sometimes handy to be able to specify credentials as environment variables directly in an override file. But that typically means you’re not going to want to check that override file into a git repository. Fortunately, the general answer here is to use some form of kubernetes secrets. When defining a deployment/pod, environment variable values can be specified by pointing to a particular key in a particular kubernetes secret. (Secrets are stored as a map of key/value pairs, so to find a particular value, you must specify both the name of the secret, and the key where the value is found). The actual secret values are stored in kubernetes, and the override file merely maintains pointers to where to find the values. That way, you can check in your override file without exposing any secret information.

I want helm to create the secrets for me.

Do you? For helm to create the secrets, it needs access to the secret values, which means they need to be in a file somewhere. Now you have to figure out how to manage that file such that it’s not checked into a git repo anywhere.

It’s generally best to create/populate kubernetes secrets from OUTSIDE of helm, via some other mechanism. Secrets aren’t likely to be changing too often, so making this a manual process shouldn’t be particularly prohibitive. In any case, different organizations often have wildly different requirements for how secrets must be handled - as such, it doesn’t make sense to try and shoehorn kubernetes secret creation into helm.

I want helm to create PVs and PVCs for me.

Kubernetes PVs and PVCs are, at the end of the day, implemented as yaml manifest files. As such, helm is perfectly capable of creating them. However, that means a helm upgrade or a helm uninstall can just as easily destroy them. An ill-conceived helm command can easily wipe away all your stored data. As such, it's better to create your PVs and PVCs outside of helm, and instead, simply reference them, when needed, in your override file. Just like we do for secrets.

Surely helm will create an ingress for me.

Yeah... maybe... but ingresses are very 'personal' - i.e. each environment typically has very different requirements for what all needs to be configured in an ingress. As such, the 'template' for creating an ingress becomes so flexible as to be largely useless - i.e. you're better off just building the ingress.yaml yourself and cutting out the middleman (helm in this case).

I’ve just modified something in my override file.  Now what?

Now you can run helm upgrade. Helm will comb through the existing manifests, check for things that need to change, and only update the necessary kubernetes objects. For example, if your only change was to add an environment variable to a particular deployment, only that deployment will be updated, and kubernetes will only restart that one associated pod.

I added new files to a configmap.  How come the services that use that configmap don’t see the updates?

You’ve just encountered a limitation of helm. Helm doesn’t keep track of the fact that a particular pod is mounting a particular configmap.  So if you change the contents of a configmap, helm doesn’t know to restart any pods.  As such, you have to manually restart the pods in question, so that when they’re restarted, the pull in the latest version of the configmap.

I have files ‘somewhere else’ and I want to wrap them up into a configmap.

That’s certainly possible, but due to another limitation of helm, those files have to be somewhere underneath the chart’s base directory structure. So files would have to be in my-chart/ somewhere. We recommend copying the files from ‘somewhere else’ into a subdirectory of the chart: cp -r /data/directory-of-handy-files my-chart/ (assuming your working directory is where the my-chart/ directory lives). You’ll now have a my-chart/directory-of-handy-files/ directory that can be referenced from your chart and used to populate a configmap.

It is also recommended that this cp be wrapped up into some sort of script or a Makefile so that it’s done each time a helm install or helm upgrade command is run.

What if the templates are 'wrong'?

It is hopefully unlikely, but theoretically possible, that particular environments or particular use cases require updates to be made to the template files themselves (the ones in my-chart/templates/). These files contain what look like normal kubernetes manifests, but they have a lot of extra {{ and }} sprinkled around, indicating where/how particular values from the my-chart/values.yaml and any override file should be feathered in to create the final manifests that get loaded up into kubernetes. In any case, in situations where there isn’t a way to affect the required changes via specific values, as a last resort, it’s possible to modify the contents of the templates. It’s not magic, and it’s well documented here: https://helm.sh/docs/chart_template_guide/getting_started/ .

However, the most important thing to do, should you decide to dive into the wonderful world of helm template modification, is to keep track of any and all changes you make. There are two reasons:

1. If/when you update to a newer version of the chart, you don’t want to lose any of your custom modifications. If the newer version of the chart still doesn’t provide the necessary, out of the box flexibility you require, you’ll likely have to make similar changes.

2. The good people who designed the original chart would probably love to know about the changes you’ve had to make, so that they can update the official versions of the chart moving forward, to provide the required features. In a perfect world, the chart templates are flexible enough so that everything you need to do can be specified in your override file, and no local modifications to the templates are required.

Example Directory Structure

.
├── Makefile
├── my-certs
│   └── my-cert.pem
├── my-chart
│   ├── templates
│   │   ├── _helpers.tpl
│   │   ├── abc-deployment.yaml
│   │   └── abc-service.yaml
│   └── values.yaml
└── override.yaml

Notes:

1. The my-certs/ directory is maintained outside the my-chart/ directory. This is so that we can update the chart by just replacing the contents of my-chart/.

2. The Makefile will contain commands that copy the my-certs/ directory into the my-chart/ directory before running any helm commands. In this way, the my-certs/ directory is now visible to helm, should it need to create a configmap based on the contents of that directory.

3. The override.yaml file is also outside the my-chart/ directory.

Makefile:

upgrade:
  rm -fr my-chart/my-certs # clear out the old certs directory
  cp -r my-certs my-chart/ # copy in the latest certs
  helm upgrade -i my-release-name my-chart -n my-namespace -f override.yaml

Notes on the helm command:

• The helm upgrade -i command will either install the release for the first time, or upgrade it based on the latest contents of the override.yaml file.

• my-release-name is just an arbitrary name to identify the release. You can run helm list to see a list of the current releases in the current namespace.

• my-chart is the relative path to the chart.

• -n my-namespace selects the kubernetes namespace where the installation should take place.

• -f override.yaml specifies that file as an override file.