👋 The "Node-Lock" Problem

That "node-lock" is the biggest weakness of hostPath. If the node goes down or the Pod moves, the data is essentially "trapped" on the old node.

To solve this, Kubernetes uses a system that decouples the storage from the node entirely, often using network storage (like a cloud disk). This brings us to the "Gold Standard" of Kubernetes storage: PersistentVolumes (PV) and PersistentVolumeClaims (PVC).

Persistent Volumes(PV) and Persistent Volumes Claim (PVC)

The What ?

Persistent Volumes (PV):

A Persistent Volume (PV) represents a piece of storage provisioned in the cluster often backed by network or cloud storage that continues to exist even if Pods are deleted or moved.

You can think of a PV as block storage or file storage attached to the Kubernetes cluster, where the cluster has full control to allocate this storage to incoming requests.

Since this storage lives outside the Kubernetes cluster environment (for example, cloud disks or network-attached storage), it is not affected by Pod restarts, node failures, or most Kubernetes-level failures.

Persistent Volumes Claim (PVC):

A Persistent Volume Claim (PVC) is a request for storage made by an application running inside the Kubernetes cluster.

Instead of directly attaching a storage disk, a Pod asks for storage by specifying:

• How much space it needs

• The type of access required (read/write)

• The storage characteristics it expects

Kubernetes then looks for a suitable Persistent Volume (PV) that matches this request and binds it to the PVC.

You can think of a PVC as an interface between your application and the actual storage, allowing Pods to use persistent storage without needing to know where or how it is physically provisioned.

❓ Why PVC request is needed ?

PVCs introduce a clean separation between applications and storage.

Different applications have very different storage needs:

• Databases need high I/O and low latency

• Logs or backups can work with lower-performance storage

Instead of hard-coding storage details into applications, a PVC allows the app to simply say: “I need this much storage, with this access pattern.” Kubernetes then fulfills that request by binding the PVC to an appropriate PV.

This approach keeps applications portable, flexible, and storage-agnostic, while allowing storage performance to be controlled through PVC requests.

Here is a quick look at how they connect:

• Pod points to → PVC

• PVC binds to → PV

• PV points to → Actual Disk

Each layer has a clear responsibility, and none of them need to know the internal details of the others.

🔐 Access Modes

Both Persistent Volumes (PV) and Persistent Volume Claims (PVC) define something called access modes. Access modes describe how a volume can be accessed by Pods for example, whether it can be mounted as read-only, read-write, or shared across multiple nodes. The available access modes are explained below

• ReadWriteOnce (RWO): The volume can be mounted as read-write by a single node. (Like a USB drive plugged into one laptop).

• ReadOnlyMany (ROX): The volume can be mounted read-only by many nodes. (Like a shared CD-ROM).

• ReadWriteMany (RWX): The volume can be mounted as read-write by many nodes. (Like a shared network folder/Dropbox).

When you provision PV and PVC for a highly available application that runs across multiple nodes, choosing the right access mode becomes critical. For example, If the volume is read-only and shared across Pods across nodes, you can use ReadOnlyMany (ROX), If the volume needs to be read and written by multiple Pods across nodes, you must use ReadWriteMany (RWX)

What does “Many” mean here?

“Many” refers to the number of nodes, not Pods.

• RWO → Only one node can read/write to the volume at a time. i.e all the pods in the node can access this volume

• RWX → Multiple nodes can read/write to the same volume simultaneously. i.e pods across nodes can access the volume

An important detail to note here is a Persistent Volume (PV) can be configured with multiple access modes, indicating what it supports. A Persistent Volume Claim (PVC), on the other hand, requests a specific access mode based on the application’s requirement. If the requested access mode in the PVC is supported by the PV, Kubernetes approves the claim and binds them together.

🧠 Key Takeaway

PV advertises what it can support, and PVC asks for what it needs. Binding happens only when both agree on the access mode.

Example Kubernetes Resource Definition File

Persistent Volume (PV)

apiVersion: v1
kind: PersistentVolume
metadata:
  name: app-pv               # Name of the PV
spec:
  capacity:
    storage: 10Gi            # Size of the volume
  accessModes:
    - ReadWriteOnce          # Supported access mode
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  # Retain data even if PVC is deleted
  storageClassName: ebs-sc   # Must match PVC
  csi:
    driver: ebs.csi.aws.com  # AWS EBS CSI driver
    volumeHandle: vol-0abcd1234efgh5678
    # Actual EBS volume ID from AWS
  volumeMode: Filesystem    # Mount as a filesystem

Persistent Volume (PV)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-pvc              # Name of the PVC
spec:
  accessModes:
    - ReadWriteOnce          # Only one node can read/write
  resources:
    requests:
      storage: 10Gi          # Amount of storage requested
  storageClassName: ebs-sc   # Use this StorageClass

Consuming a PVC Inside a Pod

apiVersion: v1
kind: Pod
metadata:
  name: app-pod                 # Name of the Pod
spec:
  containers:
    - name: app-container
      image: nginx              # Sample container image
      volumeMounts:
        - name: app-storage
          mountPath: /data      # Path inside the container
          # The PVC will be mounted here
  volumes:
    - name: app-storage
      persistentVolumeClaim:
        claimName: app-pvc      # Name of the PVC to use

StorageClasses: Leveling Up to Dynamic Provisioning

In the real world the admin don’t want to worry about creating 100’s of PV’s(Persistent Volumes) with different types and sizes. Here is where StorageClasses (SC) comes in

How it works

• The administrator creates a StorageClass in the cluster

• The StorageClass is granted permission to create cloud provider disks (such as AWS EBS or Azure Disk) by installing the appropriate CSI driver in the cluster.

  • For example, in an Amazon EKS cluster, you install the EBS CSI driver, which enables Kubernetes to dynamically provision EBS volumes on demand whenever a PVC is created.

• The PVC specifies which StorageClass to use

• When the PVC is created, Kubernetes checks:

  • Is there an existing PV that matches this request?

• If no matching PV exists, Kubernetes uses the StorageClass to dynamically create a new PV.

• The newly created PV is bound to the PVC.

• When a Pod references the PVC, the volume is attached and mounted into the Pod.

For example:

• A Pod requests 50Gi

• No existing PV matches

• The StorageClass dynamically creates a new disk (EBS etc.)

• The disk is attached to the Pod as a volume

No manual PV creation needed 🎉

Reclaim Policies

Now comes an important question. Your application stores data in a PV via a PVC and that data can live forever.

But what if:

• You delete the application?

• You no longer need the data?

• You recreate the app with fresh data?

Continuously creating new volumes is not sustainable, especially when storage costs add up. This is where Reclaim Policies come into play.

Reclaim policies define what happens to a Persistent Volume after the PVC is deleted whether the data should be:

• Retained: The PV is not deleted. It stays in "Released” state and will not be assigned to any new PVC request. An admin must manually clean up the data. This is the safest for production databases.

• Deleted: The PV and the actual physical disk (AWS EBS, GCE PD) are deleted immediately. Should be choosen with extra caution as the data will be wiped once PVC is deleted

• Recycle(Deprecated): It performs a basic rm -rf /thevolume/* and makes the PV available again. (Rarely used now).

You control this with the persistentVolumeReclaimPolicy

The Binding Process

Kubernetes matches a PVC to a PV using these criteria:

1. Capacity: Does the PV have at least the amount requested?

2. Access Mode: Does the PV support the mode (RWO, RWX, etc.)?

3. StorageClass: Do they have the same StorageClass name?

4. Selectors: (Optional) Do the labels match?

Example Kubernetes Resource Definition File

StorageClass (Dynamic Provisioning)

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc               # Name of the StorageClass
provisioner: ebs.csi.aws.com # CSI driver that talks to AWS EBS
parameters:
  type: gp3                  # EBS volume type (gp2 / gp3 / io1, etc.)
reclaimPolicy: Delete        # What happens to the disk when PVC is deleted
volumeBindingMode: WaitForFirstConsumer
# Volume is created only when a Pod actually uses the PVC

Common Scenarios

The “Same Name” Scenario (ReclaimPolicy: Retain)

Assume you have an application using a Persistent Volume (PV) named pv-x, and the reclaim policy is set to Retain. You deploy the app, it creates a PVC, and everything works fine. After some experimentation, you delete the application (which deletes the PVC).

Later, you:

• Create a new application with a different PVC name, but want to use the same data

• Or simply rename the PVC for the same application

So… what happens?

When the PVC is deleted and the reclaim policy is Retain:

• The PV is NOT deleted. The underlying disk is preserved

• The PV moves into a Released state. The PV still remembers the old PVC it was bound to

This is the important part 👇

A PV in the Released state cannot be automatically reused by a new PVC—even if:

• The storage size matches

• The access modes match

• The StorageClass matches

Kubernetes intentionally blocks this to prevent accidental data leaks between workloads.

❓ Why Can’t a New PVC Reuse the Same PV Automatically?

Because the PV still contains Old application data and Metadata pointing to the previous PVC. Kubernetes assumes this data belongs to someone else. I won’t reattach it unless a human explicitly says so. That’s a safety feature, not a limitation.

So How Can You Recover the Data?

There are two correct and safe ways to recover data from a retained PV.

✅ Option 1: Manually Rebind the Existing PV (Most Common)

If you want the same data to be used by a new or renamed PVC:

• Manually edit the PV and remove the old claimRef

• Set the PV back to Available

  • Create a new PVC that matches: StorageSize, AccessMode, and Storage class

• Once this is done, Kubernetes will bind the new PVC to the old PV—and your data is recovered intact.

This is the intended recovery flow for Retain.

✅ Option 2: Create a New PV Pointing to the Same Disk

This is useful when you don’t want to touch the old PV object and you want more control over the new binding.

• Identify the underlying disk (EBS volume, Azure Disk, etc.)

• Create a new PV definition

• Point it to the same physical disk

• Create a new PVC that binds to this new PV

The data remains untouched because the disk never changed—only the Kubernetes objects did.

Do you know AWS EBS Does NOT Support RWX

AWS Elastic Block Storage (EBS) does not support ReadWriteMany (RWX) access mode. why ?

Because an EBS volume behaves like a physical hard drive. Once you attach a hard drive to one machine, you cannot plug that same drive into multiple machines at the same time and expect it to work safely. This is exactly how EBS works:

• One EBS volume → one node

• Read/write access → single node only

That’s why EBS supports ReadWriteOnce (RWO).This limitation is not a Kubernetes issue — it’s a block storage limitation.

So What If You Need RWX Across Multiple Nodes?

If your application needs to run on multiple nodes and read and write the same data simultaneously, then block storage is the wrong tool. This is where file-based network storage comes in.

EFS: The RWX-Friendly Storage

Amazon Elastic File System (EFS) is designed to solve exactly this problem. EFS behaves like a shared network drive. Multiple nodes can mount it and all nodes see the same file and Read/Write operations happens concurrently

This wraps up our deep dive into PV, PVC, and StorageClasses — what they are, why they exist, and how Pods actually consume them.

From here, you can start mapping these concepts to real-world scenarios and production use cases, not just lab setups.