We’ve already seen how to use ConfigMaps to pass non-sensitive data like configs and files to Pods. If you missed that, you can check it out here.

But now comes the real question:

• What about sensitive data?

• Passwords? Tokens? SSH keys?

That’s where Kubernetes Secrets come in. In this blog, we’ll cover:

• What are Secrets and why we need them

• How to create Secrets

• How Pods consume them

• Lifecycle & behavior

• Security best practices (this is where most people mess up 😅)

🤔 What are Kubernetes Secrets?

A Secret is a Kubernetes object used to store sensitive data like: Passwords , API tokens, SSH keys , Certificates.

Instead of hardcoding these inside your app or YAML, you decouple them using Secrets.

⚙️ How are they different from ConfigMaps?

They look similar to ConfigMaps however they have some distinct behaviour

• Stored in etcd (cluster DB)

• Mounted as tmpfs (RAM) → never written to disk 🧠

• Base64 encoded (⚠️ not encryption!)

💡 Important: Base64 ≠ Security. Anyone with access can decode it easily.

😌 Why do we need Secrets?

You already know the answer, but let’s reinforce:

• Keeps sensitive data out of code

• Helps follow 12-factor app principles

• Enables secure configuration per environment

You can think of it like this

• ConfigMap = app config

• Secret = app credentials

🛠️ How to Create Secrets

You can create Secrets in two ways:

• Inline Command

• Using YAML

Let's explore them in detail

⚡ 1. Inline Command (Quick & Dirty)

Example:

# Basic syntax
kubectl create secret <type> <name> --from-literal=<key>=<value>

# Example
kubectl create secret generic api-credentials \
  --from-literal=token=super-secret-123

Multi-value example:

kubectl create secret docker-registry private-reg-auth \
  --docker-username=my-user \
  --docker-password=my-password \
  --docker-server=my-registry.io \
  --docker-email=my-email@example.com

When to Choose this approach ?

Use this for:

• Quick testing

• PoCs

Note: Here you can directly pass the values without encoding, Kubernetes encode the value and store it in Etcd

📄 2. Using YAML (Production Way)

apiVersion: v1
kind: Secret
metadata:
  name: my-db-secret
type: Opaque
data:
  username: YWRtaW4=
  password: MWYyZDNoNGo1aw==

Here you must encode values manually using echo -n "admin" | base64

🧩 Secret Types (Quick Understanding)

Type	Use Case
Opaque	Default, anything
kubernetes.io/dockerconfigjson	Private registry auth
kubernetes.io/tls	TLS cert + key
kubernetes.io/basic-auth	Username/password
kubernetes.io/ssh-auth	SSH keys
service-account-token	Internal API access

💡 Tip: Use specific types when possible — Kubernetes validates structure.

🔌 How Pods Consume Secrets

Two main ways (same as ConfigMaps 👀):

• Environment Variables

• Volume Mounts

🛡️ Option 1: Environment Variables

This option is suitbale for for API keys or database passwords that the application expects to find in its environment. This is simple and clean approach

Example

apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
spec:
  containers:
    - name: my-app
      image: nginx
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-creds
              key: password

⚠️ Cons:

• Values are loaded only at startup

• If Secret changes → Pod restart required 🔄

📁 Option 2: Volume Mounts (Recommended)

This option is best for certificates, SSH keys, or large configuration files. Kubernetes mounts the secret as a directory where each key in the secret becomes a file.

Example:

apiVersion: v1
kind: Pod
metadata:
  name: secret-volume-pod
spec:
    containers:
    - name: ssh-container
      image: nginx 
      volumeMounts:
        - name: ssh-key-vol 
          mountPath: "/etc/ssh-keys" 
          readOnly: true 
volumes: 
    - name: ssh-key-vol 
      secret: 
        secretName: ssh-key-secret # The name of the Secret object

👉 Each key becomes a file:

/etc/ssh-keys/
  ├── private_key
  ├── public_key

💡 Why use Volumes over Environment Variables?

• Security: Secret volumes are stored in tmpfs (RAM). If the node is powered down, the data vanishes from the node's memory. 🧠

• Updates: If you update a Secret, Kubernetes automatically updates the files in the volume (though this takes a few seconds). Environment variables, however, are static—you would have to restart the Pod to see a change. 🔄

• Structure: Volumes are perfect for things that naturally exist as files, like SSL certificates or SSH id_rsa keys.

📁 Standard Mount vs. subPath

🔹 Standard Mount (Default Behavior)

When you mount a Secret as a volume:

• The entire target directory gets replaced

• Any existing files become invisible

• Only the mounted Secret content is available for your pod to use.

👉 If you mount another Secret to the same path, it will overwrite the previous one

This behavior is intentional in Kubernetes to Ensure data isolation, Improve security, Avoid unintended file conflicts

Now consider a scenario where you want to mount multiple Secrets into the same Pod path.

Using the standard approach:

• Mounting Secret A → works fine

• Mounting Secret B to the same path → overwrites Secret A

👉 Result: The first Secret becomes inaccessible.

To solve this, Kubernetes provides a feature called subPath.

💡 What is subPath?

subPath allows you to:

• Mount individual files or subdirectories

• Place them inside an existing directory

• Avoid overriding the entire volume path

To know in detail about subpath check here

✅ Using subPath for Multiple Secrets

With subPath, you can mount multiple Secrets into the same directory without conflicts.

Example:

volumeMounts:
  - name: secret-a
    mountPath: /etc/config/secret-a.txt
    subPath: secret-a.txt
  - name: secret-b
    mountPath: /etc/config/secret-b.txt
    subPath: secret-b.txt
volumes:
  - name: secret-a
    secret:
      secretName: secret-a
  - name: secret-b
    secret:
      secretName: secret-b

🔍 What happens here?

• secret-a is mounted as /etc/config/secret-a.txt

• secret-b is mounted as /etc/config/secret-b.txt

• No overwriting occurs

• Both Secrets coexist in the same directory 🎉

One thing to note here is that when using subPath we have to define path with filename and extension whereas in standardMount the mount path is sufficient enough.

LifeCycle for subPath
One more important thing is that when using subPath when the secret value is updated it will not update the file directly pod restart is required because here the file is a static bind-mount

🔄 Lifecycle of Secrets

In this section we will explore how changing a secret value will be handled in different Consumption option

Consumption Type	Behavior
Env Variables	Deployment Requires restart 🔁
Volume Mount	Auto updates ⏱️
subPath	Deployment Requires restart 🔁

Even though VolumeMount Auto Updates your secrets to the Mount file you app should be configured to pick the updated value.

Security Best Practices & RBAC

Now that we know how to create and consume Secrets, Let's talk about how to actually keep them "secret."

Even though Secrets are better than plain-text ConfigMaps, they have some vulnerabilities by default:

• Base64 is not encryption: It's just a way to store binary data as text. Anyone who can run kubectl get secret -o yaml can see your passwords. 🔓

• Etcd Storage: By default, Secrets are stored in plain text in the cluster's database (etcd).

Some of the best practices

Encryption at Rest (The Native Way)

You can configure the Kubernetes API server with an EncryptionConfiguration file. When this is enabled, the API server will mathematically encrypt the Secret data before it saves it to etcd. Even if someone steals the etcd hard drive, the data is unreadable garbage without the encryption key.

Strict RBAC (Role-Based Access Control)

As we discussed briefly before, you must restrict who can run kubectl get secrets. If a developer has the get or list permission for Secrets, they can just ask the API server to show them the Base64 data, bypassing all etcd encryption. You should only grant Secret access to the specific ServiceAccounts that the Pods use, not to human users.

To truly secure a Secret while still letting it be used:

• For Users (Developers): You generally want to deny get, list, and watch. If they can't get it, they can't see the password.

• For Pods: The Pod doesn't actually need the user to have permissions. The Pod uses a ServiceAccount. The Kubernetes controller-manager handles the mounting process, so as long as the Pod's YAML is allowed to be created, the system handles the rest.

External Secret Stores (The Enterprise Way)

In large companies, they don't even store the source-of-truth secrets in Kubernetes at all! They use tools like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault. They use plugins (like the Secrets Store CSI Driver or External Secrets Operator) to fetch the password from the cloud vault and inject it directly into the Pod at runtime.

Conclusion

Kubernetes Secrets might look simple at first, but how you use them makes all the difference. They help you keep sensitive data out of your code and configs—but they are not secure by default.

If you use them the right way prefer volumes over env vars, restrict access with RBAC, and integrate with external secret managers—you move from just “working setup” to a production-ready secure system 🔐

Thanks for reading. Keep learning, keep building, and keep breaking things (that’s how we grow) . See you in the next one ✌️