Kubernetes Secret Management With External Secrets Operator

Secrets management is crucial for organizations of any size. However, it’s not as simple as you may think. You may be familiar with Kubernetes secrets. These are Kubernetes objects designed specifically to hold secrets, as the name suggests. But, in reality, Kubernetes secrets don’t give you everything you need to make your secrets secure. In this post, you’ll learn how to make your Kubernetes secrets management complete using the External Secrets Operator (ESO).

A Few Words About Kubernetes Secrets

Now, let’s start with the basics. Why aren’t Kubernetes secrets enough? You see, Kubernetes secrets are objects that let you store sensitive information. You can’t directly see the values of Kubernetes secrets, as they’re encoded. (However, it’s actually quite easy to decode them.) This is all good, but the problem is in getting the secrets into Kubernetes secrets in the first place.

Everything in Kubernetes can be (and usually is) defined in YAML configuration files. You create a YAML file, store it in a Git repository, and then apply these YAML files to your Kubernetes cluster, either manually or via some CI/CD system. The same applies to Kubernetes secrets. You can create a YAML file with Kubernetes secrets definition and then apply that configuration file on the cluster to create Kubernetes secrets.

Do you see the problem yet? That YAML definition file of Kubernetes secrets will need to have your secret values in plain text! Therefore, you can’t commit these files to a Git repository.

And that’s where the ESO comes to the rescue.

What Is ESO?

So, what is the ESO, and how does it solve the problem explained above? ESO is a Kubernetes operator that allows you to define your Kubernetes secrets in YAML files safely. It works in a clever way in that it uses an external (to Kubernetes) secret vault where the actual secrets can be safely stored, and then it automatically generates Kubernetes secret objects from the secrets in the vault.

So, instead of defining your secret value in a YAML file, you save it to a secret vault. It’s stored there and encrypted. Then, you create a YAML definition for the ESO, which basically says, “generate Kubernetes secret from this external vault entry.” Therefore, the ESO acts as a broker between your Kubernetes cluster and the external secrets vault. And the main point of the ESO is that the YAML definition doesn’t contain the actual secrets value, only the reference to vault entry. This means that it’s perfectly safe to store that YAML file in your Git repository.

The ESO supports a few different secrets vaults. It could be, for example, AWS Secret Manager, Azure Key Vault, or HashiCorp Vault. If you care about security, you’re probably already using some secrets vault. Therefore, it’s easy to start with the ESO. The big advantage of the ESO is that at the end of the process, you end up with a typical Kubernetes secret object, which means you don’t need to change anything in your existing applications.

How to Get Started With ESO

Now you know what the ESO is and how it works, let’s see how to use it in practice.

1. Install the ESO

The ESO installation is quite simple. As with many other Kubernetes tools, you can install it with Helm as follows:

2. Define Your SecretStore

Next, we need to tell the ESO where your secrets vault is. For that, the ESO uses a Kubernetes custom resource called SecretStore. So, we need to define it and apply it to the cluster. It will be then read by the ESO. Because the ESO supports multiple different secrets vaults, this step will differ based on which one you want to use. To find the exact steps for your vault, check the ESO documentation.

In our example, we’ll use Azure Key Vault. To use Azure Key Vault with the ESO, we need two things. First, we need to define credentials for accessing Azure Key Vault. We’ll use Azure service principal for that. What we need to do then is to create a standard Kubernetes secret with service principal details. The ESO will read that Kubernetes secret to connect to Azure Key Vault and do its job.

Once we have that, we can define and apply our SecretStore. We need to specify our Azure tenant ID and our Azure Key Vault URL. Then, we provide the name of Kubernetes secret where the ESO can find credentials for accessing that Azure Key Vault. It’s the name of the same Kubernetes secret that we applied just above.

The complete YAML definition will then look similar to this:

That’s it for the preparation part. The ESO is now ready to do what it was built for: be a go-between for your secrets vault and your Kubernetes cluster.

3. Create External Secrets

Now, whenever you need a Kubernetes secret, instead of creating an actual Kubernetes secret, you’ll create ExternalSecret instead. ExternalSecret is an ESO-specific resource that instructs the ESO where to find the actual secret value and what Kubernetes secret to create from it. Let’s see an example:

That’s how the ExternalSecret definition looks, and that’s what you can apply to your cluster instead of the actual Kubernetes secret object. And, as you can see, there are no secret values in plain text in that definition, so it’s safe to commit that file. So, what’s happening in that file? First, we define apiVersion, kind, metadata, and spec as in any other Kubernetes definition file. Then, we specify secretStoreRef—here, we need to point the ESO to which SecretStore to use. We just created one in the previous step, so we’ll reference that one. Then comes the important part. Target is where you define the name of the Kubernetes secret resource that will be created by the ESO. And in data you point the ESO to a specific secret entry in your Azure Key Vault.

So, what happens when you apply that definition? The ESO will pick it up and process it. It will connect to Azure Key Vault to read the value of secret/database-secret-test. It will then create an ordinary Kubernetes secret object called first-secret-from-eso with the value from secret/database-secret-test. That’s it. You can then use the Kubernetes secret in your application as usual, but you can also safely commit the ExternalSecret definition file in your Git repository. So, we solved the problem from the beginning.

Summary

Kubernetes comes with built-in secrets objects. However, it doesn’t come with any solution to how to bring those secrets onto your cluster securely. ESO is a great way of solving that problem. It helps to make your environment more secure.