If you're just joining this series, I highly recommend reading the previous blog first, as the concepts we'll discuss here build directly on top of it.

With that said, let's move on to the next building blocks of AI Context Window and Attention Horizon. These two concepts determine how much information an AI model can remember during a conversation and how effectively it can use that information to generate meaningful responses.

Let's dive in.

Context Window

Now that we know how data enters an AI model in the form of tokens let's explore where those tokens live while the model is processing your request.

Think of the Context Window as the model's working memory. It contains everything the model can "see" at a given moment before generating a response.

Every AI model has a hard limit on the number of tokens it can process in a single interaction. This limit is called the Context Window.

For example:

• Older models could process around 8,000 tokens.

• Modern frontier models can process 128K, 200K, or even over 1 million tokens in a single context window.

When you type a prompt, it is first split into tokens. These tokens are then assigned token IDs and converted into numerical vectors that the AI model can understand and process. In simple terms, tokens are just small pieces of text that are transformed into numbers before the model begins generating a response.

Almost every AI model specifies its context window in its official documentation. For example, here's the documentation for Gemma, where you can see its supported context window:

Why the Context Window Matters

The quality of your input has a direct impact on the quality of the model's output.

A common misconception is that the Context Window refers only to the input prompt. It doesn't.

The context window includes:

• Your input tokens

• Previous conversation (if applicable)

• The model's generated output

For example, if a model supports a 128K token context window and generates a 4K token response, then approximately 124K tokens remain available for your input and conversation history.

This is why context management becomes increasingly important when building AI applications.

Now that you understand what a context window is, the next question you're probably asking is:

Does a Larger Context Window Mean Better Memory?

Not necessarily. Just because a model can accept a massive log file or an entire codebase doesn't mean it understands or remembers every part of it equally well.

This brings us to an interesting phenomenon known as Lost in the Middle.

Lost in the Middle

Research has shown that many Large Language Models exhibit a behavior called Lost in the Middle.

They are generally very good at remembering information placed:

• At the beginning of a prompt

• At the end of a prompt

However, information buried somewhere in the middle of a very long prompt is more likely to receive less attention, making it easier for the model to overlook.

Let's take a real world example.

Imagine one of your production workflows fails. Instead of extracting only the relevant error logs, you send the entire log file to the LLM. The log contains tens of thousands of lines, and the actual error occurred somewhere in the middle.

There's a good chance the model may miss the most important error or fail to connect it with the surrounding events. As a result, it might provide an incomplete diagnosis or even suggest the wrong root cause.

This behavior is commonly referred to as Lost in the Middle.

Note: This doesn't happen every time, and newer models are continuously improving. However, it is still a well-observed limitation when working with very large contexts.

How the Context Window Impacts Performance

When working with large inputs, there are two important limits to understand.

Context Window (Input + Output Limit)

This is the total number of tokens the model can process during a single interaction, including both the prompt and the generated response.

Maximum Output Tokens

This is the maximum number of tokens the model is allowed to generate in its response.

For example, a model might accept 128,000 tokens of context but be configured to generate only 4,000 output tokens. These are two different limits and are often confused.

If you exceed the available context window, the model cannot process the extra information. Depending on the model, it may reject the request, truncate older content, or lose important context. Even before reaching the hard limit, extremely large prompts can reduce response quality because of the Lost in the Middle effect.

Interestingly, you don't always need to reach the hard limit to notice a drop in quality. With very large prompts, models can suffer from the "Lost in the Middle" effect, where information placed in the middle of a long context receives less attention than content at the beginning or the end. This can lead to missed details or less accurate responses.

Have you ever been in a long conversation with an AI where everything starts off perfectly, but after dozens of prompts it begins forgetting details, asks questions you've already answered, or even stops accepting more input? That's usually because the conversation is approaching the model's context window limit.

Every prompt you send and every response the AI generates consumes part of this limited space. Once it's full, the application must either remove older messages, replace them with a summary, or reject additional input altogether. So, the AI isn't getting "confused" it's simply running out of room to keep the entire conversation in memory.

So, How Do We Avoid Lost in the Middle?

Fortunately, there are several techniques that significantly improve both accuracy and cost.

Trimming and Slicing

Instead of sending a raw 50,000 line log file, extract only the relevant sections.

For example, search for keywords such as: ERROR CRITICAL FAIL EXCEPTION

Then include a few lines before and after the matching entries to preserve context.

This gives the model exactly what it needs without overwhelming it.

Place Important Information Strategically

If you must include a large amount of reference data, place your most important instructions or questions at the beginning or end of the prompt rather than burying them in the middle.

This increases the likelihood that the model will pay attention to the information that matters most.

Use Retrieval Instead of Sending Everything

In production AI applications, it's usually better to retrieve only the relevant documents, log snippets, or code sections before sending them to the LLM.

This approach reduces token usage, lowers cost, improves accuracy, and minimizes the chances of the model getting "lost in the middle."

There are many more advanced techniques used to manage context efficiently, which we'll explore in upcoming blogs.

Attention Horizon

At this point, you might be wondering:

If my file is only 100K tokens and the model supports a 128K token context window, shouldn't it be able to read the entire file? If so, why does it sometimes miss information that's right in the middle?

The answer lies in something called the Attention Horizon.

While the Context Window defines how much information the model can see, the Attention Horizon describes how effectively the model can focus on different parts of that information

Think of it this way:

• Context Window = Everything the model can read.

• Attention Horizon = What the model can effectively focus on while generating the next token.

Imagine someone pours 10,000 balls onto a football field. Only 30 of them are red, and the remaining 9,970 are white and blue. The red balls are there you haven't lost them but spotting all 30 quickly is difficult because they're scattered among thousands of others. Now imagine there are only 100 balls with the same 30 red balls. Suddenly, finding the red ones becomes much easier because there are far fewer distractions.

Large Language Models work in a similar way. Even if all the information fits inside the context window, the model has to distribute its attention across thousands of tokens. As the context becomes larger, important pieces of information especially those buried in the middle have to compete with everything else for attention. They aren't missing; they're simply harder for the model to focus on. This is why models can overlook details that are technically within the context window, this is one of the reasons why models sometimes exhibit the Lost in the Middle behavior.

Attention Horizon is not an official metric like the Context Window. Instead, it's a conceptual way of describing how effectively a model can focus on and make use of information distributed across its available context.

Since it's a conceptual term rather than a published specification, you won't find it listed in a model's documentation so don't bother searching for it. 🙂

A Real-World Example

Suppose you provide an LLM with:

• 40,000 lines of application logs

• Kubernetes manifests

• Terraform code

• CI/CD pipeline output

• Monitoring alerts

In this file, the actual root cause is hidden around line 22,000: java.lang.OutOfMemoryError: Java heap space

Although the model can technically read all of this information, its attention is spread across thousands of tokens. The important error may not receive enough focus, causing the model to miss it or provide a less accurate diagnosis.

Now imagine instead that you extract only:

• The error message

• 20 lines before it

• 20 lines after it

Now, almost every token in the context is relevant. The model can focus its attention much more effectively, resulting in a faster, cheaper, and more accurate response.

This is why a larger Context Window doesn't automatically guarantee better performance. The model still needs to decide where to focus its attention, and that's where the concept of Attention Horizon becomes important.

Summary

Let's quickly recap what we've learned:

• Context Window is the maximum amount of information (measured in tokens) that a model can process in a single interaction. It includes both your input and the model's generated output.

• A larger context window allows you to provide more information, but it doesn't guarantee that the model will understand or utilize every part of it equally well.

• Models can exhibit a behavior known as Lost in the Middle, where important information buried deep within a long prompt may receive less attention than information placed at the beginning or the end.

• Attention Horizon is a conceptual way to think about how effectively a model can focus on different parts of the context. While it's not an official model specification, it helps explain why simply increasing the context window doesn't always improve response quality.

• To get better results, send only the most relevant information. Trim unnecessary content, place critical instructions strategically, and retrieve only the sections the model actually needs instead of overwhelming it with massive inputs.

The key takeaway is simple:

A bigger Context Window lets the model see more, but a well-structured prompt helps the model understand more.

But before we start building AI-powered applications or experimenting with the latest models, it's important to understand the core mechanics. A strong foundation makes learning advanced concepts much easier and helps you understand why things work instead of just how to use them.

That’s why, in this multi part series, we are going to cover all the essential AI fundamentals step by step, giving you a solid launchpad for your engineering journey.

Let’s kick things off with the absolute baseline of how language models process information. Part 1: Tokens.

Token

Language models do not read whole words like humans do, nor do they look at individual characters one by one. Instead, they process text in chunks called tokens. A token can be an entire word, a part of a word (like a syllable), or even a single punctuation mark or space.

Think of it like LEGO bricks. Instead of manufacturing a unique brick for every single object in the world (words), or forcing you to build everything out of microscopic 1x1 pegs (characters), we use a standard set of pre made bricks of various sizes (tokens) to build everything efficiently.

Byte Pair Encoding

Modern models use an algorithm called Byte-Pair Encoding (BPE) to build their vocabulary of tokens. Here is the simple intuition of how BPE works from scratch:

• It starts by treating every individual character as a token.

• It looks at a massive dataset of text and finds the most frequently occurring pair of characters side-by-side (e.g., "t" and "h").

• It merges that frequent pair into a new, single token: "th".

• It repeats this process millions of times, constantly merging the most common adjacent pieces into larger tokens (like "the", or "ing").

Because of this, common words like the or infrastructure usually become a single token. Rare words, typos, or complex code blocks get broken down into multiple smaller tokens.

🪙 The Golden Rule of AI Cost

As an engineer, this is where the architecture meets the budget: Models bill you per token, not per word or character.

Generally, 1 token is roughly equal to 0.75 words in English. However, unexpected things can inflate your token count and therefore your cloud bill.

To see this in action, let's look at how spaces and punctuation change things. Consider these two inputs:

• Input A: Identify the error.

• Input B: Identify the error . (with extra spaces before and after the period)

In Input A, the punctuation might be cleanly bundled. In Input B, the tokenizer has to create distinct, individual tokens for each unexpected space and isolated punctuation mark because that specific sequence isn't a common "merged brick" in its vocabulary.

💳 The Token-to-Cost Ratio

When building production infrastructure, token counts translate directly to operational costs. LLM providers bill you based on two distinct metrics:

1. Input Tokens (Prompt): The text you send to the model.

2. Output Tokens (Completion): The text the model generates back to you.

Output tokens are almost always more expensive than input tokens because the model has to expend more compute power generating words one by one.

🔍 How Spaces and Indentations Multiply Costs

In standard code formatting, engineers use spaces or tabs for readability. However, many tokenizers process individual spaces or pairs of spaces as distinct tokens if they don't match a common pattern. Look at these two ways of sending the exact same variable payload to an LLM:

Format A (Pretty Printed JSON):

{
    "status": "unhealthy",
    "replica_count": 0
}

Format B (Minified JSON):

{"status":"unhealthy","replica_count":0}

Format A includes multiple newline characters and 4-space indentations. In many tokenizers (like the one used for GPT-4), Format A takes 19 tokens, while Format B takes only 11 tokens.

By stripping whitespace (minifying) before sending data to an API, you can cut your input token footprint by 40% or more for large data structures like JSON logs or configuration dumps.

🍓 The Missing Example: The "Strawberry" Mystery

Every AI user eventually discovers that if you ask an LLM, "How many 'r's are in the word strawberry?", the model will confidently answer "Two." People assume the AI is hallucinating or bad at math. But with the foundation you’ve already built in your blog, you can give your readers the real engineering answer: It is a tokenization limitation.

You can add a section like this:

🍓 Why AI Can’t Count the "R"s in Strawberry

When a human reads the word "strawberry," we see a sequence of 10 letters: s-t-r-a-w-b-e-r-r-y.

But an LLM never sees the letters. Because of Byte-Pair Encoding, the tokenizer bundles the word into chunks. To the model's vocabulary, "strawberry" is split into just two tokens: ["straw", "berry"].

Once it gets converted into token numbers say [301, 8396]the raw spelling structure is completely hidden. The model is trying to count threads in a rope without untwisting it first. There is simply no "letter R" in its active memory to inspect!

• Token Blindness: The LLM doesn't see the letters; it only sees two opaque boxes: [Token: "straw"] and [Token: "berry"].

• The Hidden 'R': The model has no idea what letters are hidden inside the "straw" token. To the AI, that token is just a random identification number (like 301). It registers 0 'r's here.

• The Known 'R's: However, during its massive training, the model has seen plenty of text discussing the spelling of the standalone word "berry" (b-e-r-r-y). Its neural network has a strong statistical association that the "berry" token involves 2 'r's.

• The Flawed Math: The model does the only math it can with the pieces it sees:

0 (from "straw") + 2 (from "berry") = 2

Vocabulary Size (The GPT-4 vs GPT-4o Shift)

When OpenAI released GPT-4o, they upgraded their tokenizer algorithm from a vocabulary size of roughly 100,000 tokens (cl100k_base) to 200,000 tokens (o200k_base).

By doubling the vocabulary "brick set," the model became massively more efficient, especially at processing code and non English text. For instance, the token footprint for South Asian languages like Tamil or Hindi dropped by over 60%, drastically lowering the cloud bill for developers building global apps.

Special Tokens (The "Invisible" Code)

It’s worth noting that not all tokens represent human language. Tokenizers include Special Tokens that act as architectural control signals. For example:

• <|endoftext|> tells the model the document or prompt is officially over.

• <|im_start|> and <|im_end|> separate the developer’s system instructions from the user's prompt.

If a user manages to trick your app into outputting or passing these special tokens unexpectedly, it can bypass security guardrails (a concept known as token smuggling or prompt injection).

🚀 Going Deeper: Beyond the Basics

I don't want to bombard you with more technical jargon as we are just getting started. If you are more interested, just explore these topics as well:

• Token IDs (From Strings to Numbers): While we talk about tokens as text chunks (like "straw" or "berry"), the AI model itself still doesn't understand text. Once the tokenizer chops up your prompt, it maps every single token to a unique integer ID (e.g., "the" becomes 464). It is this sequence of numbers that is actually passed into the neural network.

• The Context Window Wall: Every model has a hard limit on the total number of tokens it can hold in its memory at one time (input + output combined). If a model has a 128k token context window, exceeding that limit will cause the application to crash or forget the earliest parts of the conversation. (We will cover this in upcoming blogs)

• "Glitch Tokens": Because tokenizers are trained on massive web datasets, their vocabularies sometimes include highly specific, bizarre strings (like specific Reddit usernames or obscure code strings) that the actual AI model rarely saw during its training phase. When an LLM encounters these "glitch tokens," it can get confused and behave erratically!

That being said, if you're already familiar with Kubernetes and Kubernetes networking, grab a cup of coffee ☕ and join me as we explore the Gateway API.

Why was Gateway API introduced ?

Before we understand what Gateway API is, let's first answer an important question.

Why did Kubernetes introduce another API when Ingress was already working?

If you've read my previous blog, you know that Ingress solved one of the biggest problems with exposing applications.

Without Ingress, every application needed its own LoadBalancer Service. In a production environment running dozens or even hundreds of applications, this quickly becomes expensive and difficult to manage.

Ingress solved this by allowing multiple applications to share a single Load Balancer. Using host-based and path-based routing, one Load Balancer could expose an entire fleet of applications.

Sounds perfect, right? Well... not exactly.

The Problem with Ingress

Ingress mainly consists of two components:

• Ingress Resource – A Kubernetes object where you define routing rules.

• Ingress Controller – The component that watches those rules and configures the underlying proxy (NGINX, Cilium, Traefik, HAProxy, etc.).

Technically, Ingress works very well. The challenge isn't the technology it's who manages it.

Let's understand this with a simple example. Imagine your company has an e-commerce application with multiple teams.

• Team A owns the User Service

• Team B owns the Product Service

• Team C owns the Payment Service

All of these services are exposed through a single Ingress.

shop.example.com/user -> user-service

shop.example.com/product -> product-service

shop.example.com/payment -> payment-service

Example ingress manifest:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ecommerce-ingress
spec:
  ingressClassName: nginx
  rules:
  - host: shop.example.com
    http:
      paths:
      - path: /user
        pathType: Prefix
        backend:
          service:
            name: user-service
            port:
              number: 80

      - path: /product
        pathType: Prefix
        backend:
          service:
            name: product-service
            port:
              number: 80

      - path: /payment
        pathType: Prefix
        backend:
          service:
            name: payment-service
            port:
              number: 80

Notice something interesting? Every application's routing configuration is stored in a single Ingress resource.

Now imagine the Payment team wants to expose a new endpoint. To do that, they need to modify the same Ingress resource that everyone else uses. Likewise, if the Product team wants to modify its routing, it still needs access to this shared Ingress resource.

This creates a few problems:

• Every team needs permission to edit the same Ingress resource.

• A small mistake can accidentally impact another team's routes.

• Most organizations don't allow application teams to modify shared networking resources.

• As a result, every routing change has to go through the DevOps or Platform team.

Now imagine dozens of teams raising tickets every day just to add or update a route.

Instead of developers deploying independently, they now depend on the Platform or DevOps team for even small networking changes. This slows down releases and creates an unnecessary operational bottleneck. This is one of the biggest reasons Gateway API was introduced.

It separates infrastructure ownership from application routing, allowing each team to manage only what they own while the Platform team continues to control the shared networking infrastructure.

Ingress was designed to expose applications. Gateway API was designed to expose applications at scale across multiple teams.

What Gateway API is ?

Now that we understand the problem, let's see how Gateway API solves it.

Instead of having one Ingress resource that everyone edits, Gateway API splits the responsibilities into different resources.

Think of it this way:

• The Platform (DevOps) Team manages the networking infrastructure.

• The Application Teams manage only their application's routing.

This separation is exactly what was missing in Ingress.

Let's revisit our ecommerce application. Previously, we had a single Ingress like this:

shop.example.com/user -> user-service

shop.example.com/product -> product-service

shop.example.com/payment -> payment-service

Every team had to edit the same Ingress resource whenever they wanted to expose a new endpoint.

With Gateway API, things work differently. The Platform team first creates a Gateway.

Gateway ?

Think of the Gateway as the main entry point into your Kubernetes cluster. It defines things like:

• Which ports should be exposed (HTTP/HTTPS)

• Which hostname or domain should be accepted

• Which teams are allowed to attach routes

• TLS certificates

• Listener configuration

Notice something? There are no application routes here.

The Platform team is responsible only for setting up the networking infrastructure. Now comes the interesting part.

Each application team creates its own HTTPRoute. For example,

• the User team creates: /user - user-service

• The Product team creates: /product - product-service

• etc

Each team owns only its own HTTPRoute resource. If the Payment team wants to add a new endpoint tomorrow, they only modify their HTTPRoute. They don't touch the User team's routes. They don't touch the Product team's routes. They don't even need permission to edit the Gateway.

The Platform team continues managing the shared networking infrastructure, while every application team manages only its own routing rules.

This removes the operational bottleneck we discussed earlier. Developers can independently deploy routing changes without waiting for the DevOps team, and the Platform team doesn't have to review every small routing update.

This separation of responsibilities is one of the biggest design improvements introduced by Gateway API.

Now that we've learned how Gateway API solves the problems of Ingress, let's dive a little deeper and understand its core components.

Components of Gateway API

Gateway API defines multiple resources, but for exposing HTTP applications, there are three important resources that you'll work with most of the time.

• Gateway Class

• Gateway

• HTTPRoute

One of the main goals of Gateway API is separation of responsibilities. Instead of having one resource that everyone edits (like Ingress), each resource has a specific purpose and is usually managed by a different team.

To understand these resources, let's use a simple real world analogy. Imagine your company is constructing a new office building.

• First, the architects decide which construction company will build the office.

• Once that's decided and office is built, the infrastructure team designs and manages the building's main entrance, deciding who can enter and through which doors.

• Finally, each department places its own signboards, directing visitors to the correct department enterance.

Gateway API follows the same idea.

• GatewayClass decides which gateway implementation will be used.

• Gateway creates the main entry point into your Kubernetes cluster.

• HTTPRoute defines where incoming requests should be routed.

Let's understand each one in detail.

Gateway Class

GatewayClass is the top-most resource in Gateway API.

Its job is to tell Kubernetes which Gateway implementation (provider) should manage your Gateways. Think of it like hiring a construction company before building your office.

You first decide who is going to build it. Will it be Company A? or Company B. Once you've made that decision, every building constructed under that contract follows that company's standards.

GatewayClass works in exactly the same way. You choose which Gateway provider will implement your Gateway resources. Some popular Gateway providers include:

• Cilium Gateway

• NGINX Gateway Fabric

• Envoy Gateway

• Kong Gateway

• HAProxy Kubernetes Ingress Controller

• Traefik

Each provider has its own Gateway controller that watches Gateway API resources and configures the underlying data plane accordingly.

For example, if you create a GatewayClass for Cilium, the Cilium Gateway controller manages all Gateways that reference that class. Likewise, if you create a GatewayClass for NGINX Gateway Fabric, the NGINX Gateway controller becomes responsible for those Gateways.

Since GatewayClass defines the infrastructure implementation for the entire cluster, it is a cluster scoped resource and is typically managed by the Platform or Infrastructure team.

Although Kubernetes allows you to create multiple GatewayClasses in the same cluster for example, one using Cilium and another using NGINX Gateway Fabric in most production environments, organizations standardize on a single provider unless there is a specific business requirement.

Example manifest of gateway class

apiVersion: gateway.networking.k8s.io/v1 
kind: GatewayClass 
metadata: 
  name: cilium 
spec: 
  controllerName: io.cilium/gateway-controller

Let's understand what each field means:

• apiVersion – Specifies the version of the Gateway API.

• kind – Tells Kubernetes that we're creating a GatewayClass resource.

• metadata.name – The name of the GatewayClass. We'll reference this later while creating a Gateway.

• spec.controllerName – Identifies the Gateway controller that will manage this GatewayClass. Since we're using Cilium, the controller name is io.cilium/gateway-controller.

Gateway

Now that we've selected our Gateway provider using a GatewayClass, the next step is to create a Gateway.

Continuing with our office building analogy...

We've already decided which construction company is going to build and manage our building. Now, it's time to design the main entrance. Think of the Gateway as the main entrance to your office building.

The infrastructure team decides:

• Which entrances should be open (HTTP, HTTPS, or both)

• Which domain names should visitors use

• Which TLS certificate should secure the entrance

• Which application teams are allowed to attach their routes

Notice that the Gateway doesn't know anything about your applications. It doesn't know where the User Service is. It simply defines how traffic is allowed to enter the Kubernetes cluster.

Once the Gateway is created, multiple application teams can attach their own routing rules without modifying the Gateway itself. This is one of the biggest improvements over Ingress.

With Ingress, networking configuration and application routing were often defined in the same resource. With Gateway API, those responsibilities are separated.

The Platform or Infrastructure team owns the Gateway, while application teams own their HTTPRoute resources. This allows each team to work independently without interfering with one another.

Let's look at what a Gateway resource looks like.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: web-gateway
spec:
  gatewayClassName: cilium
  listeners:
  - name: http
    protocol: HTTP
    port: 80

Let's understand the important fields in the Gateway manifest.

• apiVersion – Specifies the version of the Gateway API that Kubernetes should use.

• kind – Indicates that we're creating a Gateway resource.

• metadata.name – The name of the Gateway. Other resources, such as HTTPRoute, will reference this Gateway to attach their routing rules.

• gatewayClassName – Specifies which GatewayClass should manage this Gateway. In our example, we're using the cilium GatewayClass that we created earlier. This tells Kubernetes to let the Cilium Gateway controller manage this Gateway.

• listeners – Defines how the Gateway accepts incoming traffic. A Gateway can have one or more listeners.

  In our example, the listener is configured with:

  • name – A unique name for the listener.

  • protocol – The protocol that the Gateway accepts, such as HTTP, HTTPS, TLS, or TCP.

  • port – The port on which the Gateway listens for incoming traffic.

HTTPRoute

So far, we've selected our Gateway provider using a GatewayClass and created a Gateway to act as the main entry point into our Kubernetes cluster.

But there's still one important question. Once a request enters the cluster, how does it know which application to reach?

This is where HTTPRoute comes into the picture.

Let's continue with our office building analogy.

A visitor enters the building through the main entrance. Now they need directions. They need to go to the Finance department which room they want to enter ?

This is exactly what the signboards inside a building do—they guide visitors to the correct destination.

An HTTPRoute works in the same way. It defines how incoming HTTP requests should be matched and which backend service they should be forwarded to.

Unlike Ingress, each application team can create and manage its own HTTPRoute without modifying the shared Gateway. For our ecommerce application, each team owns its own HTTPRoute.

The User team creates an HTTPRoute for /user.

The Product team creates an HTTPRoute for /product.

The Payment team creates an HTTPRoute for /payment.

Each team is responsible only for its own routing configuration. This separation allows multiple teams to work independently while sharing the same Gateway.

Let's look at an example HTTPRoute.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: user-route
spec:
  parentRefs:
  - name: web-gateway

  hostnames:
  - shop.example.com

  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /user

    backendRefs:
    - name: user-service
      port: 80

Let's understand the important fields:

• metadata.name – The name of the HTTPRoute resource.

• parentRefs – Specifies which Gateway this route should attach to. In our example, this route is attached to the web-gateway.

• hostnames – Defines the hostname that this route should match.

• rules – Contains one or more routing rules.

• matches – Specifies the conditions that incoming requests must satisfy. Here, we're matching requests whose path starts with /user.

• backendRefs – Specifies the backend Service that should receive the traffic once the request matches the rule.

Overall Flow

• A request to shop.example.com/user/profile

• Arrives at the Gateway

• The Gateway forwards it to the attached HTTPRoute

• The HTTPRoute matches the /user path

• The request is finally sent to the user-service

Hands On Lab

We've covered the concepts behind Gateway API and understood how its components work together. But the best way to learn is by seeing it in action.

In this hands on lab, we'll deploy a Gateway, create an HTTPRoute, and expose our application using Gateway API.

Feel free to follow along using the README in the GitHub repository. By the end of this lab, you'll have a working Gateway API setup and a much clearer understanding of everything we've learned so far.

Let's get our hands dirty!

Conclusion

Congratulations! 🎉 If you've made it this far.

In this blog, we learned why the Kubernetes community introduced Gateway API and how it addresses some of the limitations of Ingress, especially in environments where multiple teams share the same cluster.

We explored the three core resources you'll use most often:

• GatewayClass – Specifies which Gateway provider manages your Gateways.

• Gateway – Defines how traffic enters your Kubernetes cluster.

• HTTPRoute – Defines where that traffic should be routed.

Most importantly, we learned that Gateway API isn't just another way to expose applications. It introduces a clear separation of responsibilities, allowing Platform teams to manage the networking infrastructure while application teams independently manage their own routing rules.

This makes Gateway API more scalable, easier to manage, and a better fit for modern production environments.

I hope this blog helped you understand not just how Gateway API works, but also why it was introduced in the first place.

In the next blog, we'll dive deeper into the networking concepts of kubernetes.

Until then, keep learning, keep experimenting, and as always...

Happy Kubernetes! 🚀

That being said, if you already have a basic understanding of Kubernetes and networking, grab a cup of coffee ☕ and let's explore how Ingress works in Kubernetes.

Ingress Basics:

This is probably the most common and heavily used method in production environments to expose applications to the outside world.

At this point, we already know that Kubernetes provides multiple ways to expose our applications.

NodePort

With NodePort, applications are exposed through a port on every cluster node.

While it is great for learning and testing, managing NodePorts at scale can quickly become painful. This is one of the reasons why NodePort is rarely used directly in production environments.

LoadBalancer

A LoadBalancer Service solves many of the limitations of NodePort. The cloud provider automatically provisions a Load Balancer and routes traffic to your application.

Sounds great, right? But there is one problem.

Every application exposed using a LoadBalancer Service typically gets its own cloud Load Balancer. As the number of applications grows, so does the cost 💸.

Imagine running 20 applications and provisioning 20 separate Load Balancers. Your cloud bill might not be very happy.

Ingress

Instead of creating a dedicated Load Balancer for every application, we can place a single Ingress Controller in front of multiple applications.

Typically, a single cloud Load Balancer forwards all incoming traffic to the Ingress Controller. The Ingress Controller then examines the incoming request and decides where it should go based on the routing rules we define.

For example:

• Requests for shop.example.com can be routed to the shopping application.

• Requests for api.example.com can be routed to the API service.

• Requests for /admin can be routed to a completely different backend.

In simple terms, the Ingress Controller acts as the gatekeeper between the outside world and your Kubernetes cluster.

One important things to keep in mind that,

Ingress routes traffic to Services, not directly to Pods.

Components of Ingress

When people talk about Ingress, they are usually referring to two different components:

1. Ingress

2. Ingress Controller

Let's understand the difference because this is where many beginners get confused.

Ingress

Ingress is a Kubernetes resource, just like a Deployment or a Service.

It contains the routing rules that describe:

• Which hostname should map to which Service

• Which URL path should route to which application

• How incoming traffic should be handled

Think of an Ingress resource as a traffic routing configuration file.

By itself, the Ingress resource does absolutely nothing. It simply stores the desired routing rules.

Ingress Controller

The Ingress Controller is the actual application that reads the Ingress configuration and performs the routing.

Without an Ingress Controller, creating an Ingress resource is like writing traffic rules on a piece of paper and expecting cars to follow them automatically.

Some popular Ingress Controllers include:

• NGINX Ingress Controller (one of the most widely used options, but officially retiring in March 2026)

• Traefik

• HAProxy

• Istio Ingress Gateway

• Kong Ingress Controller

Note: The NGINX Ingress Controller project announced its retirement and reached end-of-life in March 2026. New Kubernetes environments are increasingly adopting alternatives such as Gateway API implementations.

The Ingress Controller does much more than simple traffic routing. Depending on the implementation, it can also provide:

• TLS Termination: Users access applications securely using HTTPS, while Ingress handles SSL/TLS for us.

• Authentication Integrations: Can integrate with OAuth, SSO, Identity Providers, and more.

• Rate Limiting: Prevent abuse by limiting requests. i.e Stop someone from sending 10 million requests in 5 minutes 😅

• Traffic Filtering: Block unwanted traffic based on headers, IPs, rules, etc.

• Traffic redirection: Redirect traffic to different endpoints based on the configuration

• Observability and metrics: Provides network related Metrics and logs for observability such as API response time, etc

In short, an Ingress Controller is much more than a router—it's often the first line of traffic management and security for applications running inside a Kubernetes cluster.

How Does Traffic Enter the Cluster?

Now that we understand what an Ingress Controller is, let's see how traffic actually flows through the cluster.

Internet → External Load Balancer → Ingress Controller → Application Service → Pod

In simple terms:

1. A user sends a request from the internet.

2. The request reaches the external Load Balancer.

3. The Load Balancer forwards the request to the Ingress Controller.

4. The Ingress Controller examines the request and determines where it should go.

5. The request is forwarded to the appropriate Kubernetes Service.

6. Finally, the Service routes the request to one of the application's Pods.

So the real question becomes:

How does the Ingress Controller know where to send the request? Great question 😄

How Does Ingress Know Where to Route Traffic?

We define routing behavior using an Ingress resource.

Just like Deployments define how Pods should run and Services define how applications should be exposed internally, an Ingress resource defines how external traffic should be routed.

Typically, an Ingress manifest is deployed alongside the application's Deployment and Service resources.

The Ingress resource tells Kubernetes:

• Which hostname should be matched

• Which URL path should be matched

• Which Service should receive the traffic

• Which Ingress Controller should process the request

The Ingress Controller continuously watches these Ingress resources and updates its routing configuration accordingly.

Traffic Routing Strategies

Ingress commonly routes traffic using two strategies:

1. Path-Based Routing

2. Host-Based Routing

Let's look at both with real-world examples.

Path-Based Routing

In Path Based Routing, multiple applications share the same hostname, but different URL paths are routed to different services.

Imagine we have an ecommerce platform with three applications:

• Frontend Service

• Product Service

• Admin Service

We want all traffic to come through the same domain: example.com

But route requests based on the URL path.

example.com/products   → Product Service
example.com/admin      → Admin Service
example.com/           → Frontend Service

Visually:

                   example.com
                         │
                         ▼
                Ingress Controller
                    │     │     │
                    │     │     │
                    ▼     ▼     ▼
                /       /products   /admin
                 │           │         │
                 ▼           ▼         ▼
          Frontend     Product Svc   Admin Svc

This approach is useful when multiple applications belong to the same domain but serve different functions.

Host-Based Routing

In Host Based Routing, traffic is routed based on the hostname requested by the user.

For example:

shop.example.com   → Shopping Service
api.example.com    → API Service
admin.example.com  → Admin Service

The Ingress Controller examines the Host header in the incoming request and forwards traffic to the appropriate Service.

Visually:

                Ingress Controller
                       │
       ┌───────────────┼────
       │               │               │
       ▼              ▼              ▼

shop.example.com  api.example.com  admin.example.com
       │               │               │
       ▼              ▼              ▼

 Shopping Svc      API Svc        Admin Svc

This is one of the most common routing patterns used in production environments because different applications can have their own dedicated subdomains while still sharing the same Ingress Controller and Load Balancer.

In practice, most production environments use a combination of both Host-Based and Path-Based Routing. For example:

api.example.com/v1      → API v1 Service
api.example.com/v2      → API v2 Service
shop.example.com        → Shopping Service
admin.example.com       → Admin Service

This provides a flexible and cost-effective way to expose multiple applications without provisioning a separate Load Balancer for each one.

Enough Theory, Let's Build It!

So far, we've covered a lot of concepts around Ingress, Ingress Controllers, and traffic routing. But as engineers, we all know one thing:

The real learning starts when we get our hands dirty. 😄

Let's move beyond the theory and see Ingress in action.

For this demo, we'll use Traefik, one of the most popular Kubernetes Ingress Controllers. We'll deploy Traefik, expose a couple of sample applications, and then configure Ingress rules to route traffic based on hostnames and URL paths.

Rather than walking through a simple demo here, I've put together a dedicated hands-on lab repository where you can practice everything we've learned in this blog.

The repository includes step-by-step exercises covering:

• Installing and configuring Traefik

• Creating Ingress resources

• Path-Based Routing

• Host-Based Routing

• TLS configuration

• Traffic validation and troubleshooting

• Real-world Ingress scenarios

👉 Follow along using the hands-on labs

My recommendation is simple: don't just read the manifests deploy them, modify them, break them, and fix them. That's the fastest way to understand how Ingress works under the hood.

Conclusion

In this blog, we explored one of the most important concepts in Kubernetes networking: Ingress.

We started by understanding why Ingress was introduced and the challenges it solves compared to NodePort and LoadBalancer Services. We then learned the two key components involved:

• Ingress Resource

• Ingress Controller

We also looked at how traffic flows from the internet into a Kubernetes cluster and explored two common routing strategies:

• Path-Based Routing

• Host-Based Routing

By now, you should have a solid understanding of how a single Ingress Controller can intelligently route traffic to multiple applications while reducing infrastructure costs and simplifying traffic management.

Of course, what we've covered here is just the beginning. Modern Ingress Controllers such as Traefik provide many advanced capabilities that are commonly used in production environments.

Some great next steps to explore are:

• TLS/SSL – Secure your applications with HTTPS and TLS certificates.

• Authentication – Protect applications using Basic Authentication, OAuth, or external identity providers.

• Rate Limiting – Prevent abuse and protect services from excessive traffic.

• Canary Deployments – Gradually roll out new versions using weighted traffic routing.

• Monitoring & Observability – Integrate Prometheus and Grafana to monitor traffic patterns and controller health.

Most importantly, don't stop with the theory. Take the examples from the hands on repository, experiment with them, break them intentionally, and troubleshoot the issues. That's where Kubernetes networking truly starts to make sense.

Until then, happy learning and happy Kubernetes-ing! 🚀

If you are new to this series, I highly recommend checking out the previous networking blogs first. At least having a basic understanding of Kubernetes networking and Services will help you follow along much better in this blog. That said, if you already have some experience and know the basics of networking and Kubernetes Services, feel free to continue from here 🚀

In this blog, we will explore one of the most important topics in Kubernetes networking:

How do we expose applications to the outside world?

Because let’s be honest 😄 Running an application inside a Kubernetes cluster is one thing. But making it accessible to users securely and efficiently? That is where things start getting interesting.

We will explore the different ways Kubernetes allows us to expose applications, including:

• NodePort — the simplest way to expose applications

• LoadBalancer — cloud-native exposure using external Load Balancers

• Ingress — the production-friendly and cost-effective approach

• Gateway API — the modern evolution of Kubernetes traffic management

We will also discuss:

• When to use each option

• Their pros and cons

• Why some approaches work better than others in production environments

By the end of this blog, you will have a solid understanding of the different ways applications are exposed in Kubernetes and when to choose each approach.

So grab your coffee ☕, let’s get started!

How Do We Expose Applications to the Outside World?

Let’s start with the most basic and simplest approach.

1. Node Port:

This is one of the simplest ways to expose your application to the outside world.

What it is and How it works ?

As the name suggests, Kubernetes exposes your application using a port on the cluster node itself.

Here, we use a Service of type NodePort. Kubernetes opens a specific port on every node in the cluster, and traffic hitting that port gets forwarded to your application.

For example, imagine your application is exposed on port 30080. This means: <NodeIP>:30080 can be used to access your application.

One important thing to remember here is: Even if your application is running only on one node, Kubernetes exposes the same port on all nodes in the cluster.

So what happens if traffic reaches the “wrong” node where your application is not running? No worries our friend kube-proxy takes care of this for us 🚀

• kube-proxy is responsible for forwarding the traffic to the correct Pod, even if the request initially lands on a node where your application is not hosted.

• In short: Request → Any Cluster Node → kube-proxy → Correct Pod

Sounds simple, right? Well… yes, but there are a few catches 😄. You will get to know in Cons section.

Pros

• Easy to implement

• Useful for local testing and quick demos

• No external Load Balancer required, making it cost-effective

Cons

• Not ideal for production grade deployments

• Port conflicts can become a headache

• Managing and remembering ports is difficult as applications grow

• Exposes high numbered ports, which is not very user friendly

• Rarely used in production; mostly useful for local testing or debugging

So while NodePort is great to get started, it is usually not the preferred approach for production environments. And this is where more advanced approaches like LoadBalancer, Ingress, and eventually Gateway API start becoming important.

2. Load Balancer

After NodePort, the next common option to expose applications is using a LoadBalancer Service. This is one of the more cloud native ways of exposing applications to the outside world.

But there is one catch here 👀

This approach mainly works when your Kubernetes cluster is hosted on a cloud provider such as AWS, Azure, or GCP because Kubernetes depends on the cloud provider to provision an external Load Balancer for you.

What is it and How does it work?

The idea here is pretty straightforward.

Instead of exposing your application directly through a node port, Kubernetes creates a Cloud Provider Load Balancer and connects external traffic to your application through it.

For this, we use a Service of type LoadBalancer.

Once you create a LoadBalancer Service, Kubernetes communicates with the cloud provider and automatically provisions a Load Balancer for your application.

Behind the scenes, the flow looks something like this:

Internet → Cloud Load Balancer → Kubernetes Service → Pod

For example:

• In AWS, Kubernetes can provision an NLB (Network Load Balancer) by default

• In Azure, it creates an Azure Load Balancer

• In GCP, it provisions a Google Cloud Load Balancer

The best part?

You don’t have to manually create or configure the Load Balancer yourself. Kubernetes handles it automatically (provided your cluster has the necessary permissions).

Sounds perfect, right? Well… not exactly 😄

Here comes the biggest downside.

The Cost Problem 💸

For every application exposed using a LoadBalancer Service, Kubernetes generally creates a separate external Load Balancer.

Let’s say:

• You have 10 applications → you may end up with 10 Load Balancers

• You have 100 applications → yes, potentially 100 Load Balancers 😅

And as we all know, cloud Load Balancers are not cheap. This makes the approach expensive and difficult to scale for large environments.

Pros

• Easy to implement

• Production-ready and widely used

• Direct real-world connectivity to applications

• Secure when configured properly (TLS, security groups, firewall rules, etc.)

Cons

• Can become very expensive at scale

• Each application may get its own Load Balancer

• Managing many Load Balancers becomes operationally difficult

• Less cost-efficient for microservices architectures with many apps

So while LoadBalancer Services are great and production ready, using them for every application is usually not the most scalable or cost effective approach.

And this is exactly where solutions like Ingress and later Gateway API start becoming really interesting 🚀

3. Ingress:

This is probably the most common and heavily used method in production environments to expose applications to the outside world.

In the previous section, we saw one major problem with LoadBalancer Services: Every application gets its own Load Balancer. And as we know, cloud Load Balancers are expensive 💸

So naturally, the question becomes:

What if we use one Load Balancer for multiple applications?

Well… that is exactly the problem Ingress solves 😄

What is it and How does it work?

Before we understand Ingress, there is one important thing to clarify.

Ingress ≠ Ingress Controller

These are two different things.

Ingress:

Ingress is a Kubernetes resource (just like Deployment or Service).

It contains the routing rules:

• Which application should receive traffic?

• Which hostname should map to which service?

• Which path should go to which application?

Think of it as a traffic routing configuration file.

Ingress Controller

The Ingress Controller is the actual application that reads the Ingress configuration and performs the routing. Instead of relying completely on cloud native routing logic, the Ingress Controller uses technologies like:

• NGINX Ingress Controller (one of the most widely used options, but officially retiring in March 2026)

• Traefik

• HAProxy

• Istio Ingress Gateway

• And many more...

These tools take care of traffic routing, TLS termination, filtering, rate limiting, and much more.

Note: The Kubernetes community announced the retirement of the Ingress-NGINX project. Best-effort maintenance continues until March 2026, after which there will be no new releases, bug fixes, or security updates. Existing deployments will continue to work, but organizations are encouraged to plan migrations to alternatives such as Gateway API-compatible solutions.

But Wait… How Does Traffic Enter the Cluster?

Good question 👀

You might remember:

By default, applications inside Kubernetes use ClusterIP, which cannot be accessed from outside the cluster.

So how does Ingress solve this? Here is the trick:

The Ingress Controller itself is exposed using a LoadBalancer Service. That means: Instead of creating 100 Load Balancers for 100 applications, we create:

1 Load Balancer → 1 Ingress Controller → Many Applications

Much cheaper 😄

The traffic flow looks like this:

Internet → External Load Balancer → Ingress Controller → Application Service → Pod

So all incoming traffic enters through the Ingress Controller, and then the controller decides: Which application should receive this request?

How Does Ingress Know Where to Route Traffic? Another great question 😄

We define this using an Ingress resource which is deployed along with the application resources such as deployment or service as one of the resource. Each application can have its own Ingress manifest, which tells Kubernetes:

• Which hostname to listen for

• Which path to match

• Which service should receive traffic

• Which Ingress Controller to use

Traffic routing in an Ingress is typically handled in two ways: Path based Routing and Host based Routing. These routing strategies help direct incoming requests to the appropriate services based on the URL path or hostname.

For now, just understand that an Ingress routes incoming traffic to the correct application using one of these techniques. We’ll explore both routing methods in more detail in a separate blog post.

Simple Ingress Manifest Example

# API Version of the Ingress resource
apiVersion: networking.k8s.io/v1

# We are creating an Ingress object
kind: Ingress

metadata:
  # Name of the Ingress
  name: demo-ingress

spec:
  # Which ingress controller should handle this
  ingressClassName: nginx

  rules:
    # Hostname to match
    - host: app.example.com
      http:
        paths:
          # URL path to match
          - path: /

            # Match everything under "/"
            pathType: Prefix

            backend:
              service:
                # Service to route traffic to
                name: my-app-service

                port:
                  # Service port number
                  number: 80

What this manifest says is: If traffic comes to app.example.com, send it to my-app-service on port 80.

What Else Can Ingress Do?

Ingress is not just about routing traffic. It also provides many production grade features like:

• TLS Termination: Users access applications securely using HTTPS, while Ingress handles SSL/TLS for us.

• Rate Limiting: Prevent abuse by limiting requests. i.e Stop someone from sending 10 million requests in 5 minutes 😅

• Traffic Filtering: Block unwanted traffic based on headers, IPs, rules, etc.

• Authentication Integrations: Can integrate with OAuth, SSO, Identity Providers, and more.

Basically, Ingress becomes the front door of your Kubernetes cluster 🚪

Pros

• Very cost-effective compared to individual LoadBalancers

• One external Load Balancer can serve many applications

• Production-ready and widely adopted

• Supports TLS termination

• Supports rate limiting and traffic filtering

• Multiple controller options available

Cons

• One component ends up handling many responsibilities

• Can become a bottleneck if not designed properly

• If the Ingress Controller fails, many applications may be affected

• Limited delegation model — platform teams manage controllers while developers often need routing changes, causing dependency

And this exact limitation around ownership, flexibility, and delegation is one of the biggest reasons Gateway API was introduced 🚀

4. Gateway API

So far, we explored:

• NodePort → Simple but not production friendly

• LoadBalancer → Production ready but expensive

• Ingress → Cost-effective and production-ready

At this point, you might be wondering:

If Ingress already solves the problem, why do we even need Gateway API? 🤔

Fair question 😄

What is Gateway API?

Gateway API is the modern and more flexible way of managing traffic into Kubernetes clusters.

At a high level, it still follows a similar architecture as Ingress: under the hood, it still uses a small number of Load Balancers and routes traffic through them.

Then what changed? The answer is simple:

Better separation of responsibilities and better delegation.

And trust me this becomes a huge deal in real production environments 😄

Why Was Gateway API Introduced?

Ingress works. No doubt about that. But over time, teams realized it has some important limitations.

• One Component Handles Too Many Responsibilities

  • Ingress Controllers usually handle: Load balancing, TLS termination, Rate limiting, Traffic filtering, Authentication, Routing

  • Basically One component doing many jobs.

  • This is manageable initially, but at scale it becomes harder to maintain and customize.

• Limited Delegation Model (The Real Production Pain 😅)

  • Let’s understand this with a real-world example. Assume you are a DevOps Engineer. You:

    • Deploy the Ingress Controller

    • Configure networking rules

    • Create Ingress resources for applications

  • Now imagine a developer comes and says: "Hey, I need to update my route."

  • For Example

    • /payments → /payments-v2

    • (Or) api.company.com → payments.company.com

  • Can the developer directly update the Ingress configuration? Usually… No 👀

  • Why ? Because Ingress is shared infrastructure.

  • If routing is path-based, You must ensure:

    • No other application is already using the same path

    • Existing routes do not break

  • If routing is host-based, You must ensure:

    • The hostname is unique

    • TLS and DNS configurations remain correct

  • And here comes the scary part 😅. A small mistake by one team can accidentally impact many other applications, because everyone is sharing the same Ingress Controller.

  • This creates: Developer → Raise Request → DevOps Review → Cross Verification → Deployment

  • And suddenly even a small routing change becomes a ticket, approval process, and deployment delay 😅 . This is one of the biggest pain points Gateway API was designed to solve.

How Gateway API Solves This ?

Gateway API separates responsibilities into independent resources.

Instead of one big Ingress object handling everything, Gateway API breaks things into smaller, manageable components. The main components are:

• GatewayClass

• Gateway

• HTTPRoute

Each component can be owned and managed by different teams. This separation is what makes Gateway API powerful.

GatewayClass

This defines: Which Gateway implementation/tool should be used

Just like Ingress had multiple controllers (NGINX, Traefik, etc.), Gateway API also supports different implementations. Some popular options include:

• Cilium

• Istio

• Calico

• Envoy Gateway

• Kong

Think of this as: "Which engine are we using under the hood?". For example: We chose Cilium as our Gateway implementation.

You can technically have multiple GatewayClasses for different teams or business units. Although possible, keeping too many GatewayClasses is usually not considered best practice unless there is a strong reason.

Ownership: This is usually managed by the Infrastructure Team. Because platform wide networking decisions are generally centralized.

Gateway

This is similar to the Ingress Controller layer. The Gateway defines:

• Which listeners to expose (HTTP, HTTPS)

• Which ports to open

• Which domains to allow

• Which GatewayClass to use

Think of this as: The actual traffic entry point into the cluster.

Example:

You may have:

Public Gateway → Internet-facing → Strong TLS → Rate limiting

and

Internal Gateway → Internal traffic only → Relaxed restrictions

Ownership: Usually managed by the Platform Team or DevOps Team. Because this still affects cluster wide networking behavior.

HTTPRoute

This is where things get interesting 😄

HTTPRoute is the equivalent of the Ingress resource, but with one major improvement: It can be safely delegated to developers.

Here developers define:

• Hostnames

• Paths

• Backend Services

• Traffic rules

And simply attach the route to an approved Gateway.

Example: /payments → payment-service . The developer can manage their own routing without touching shared infrastructure. And trust me… this is huge 🚀 Because now:

Developers Can Move Independently: Developers no longer need to depend on DevOps teams for every small routing update.

Want to add a new endpoint?, Want to change a path?, Want to update service mapping?

They can manage it themselves through their own HTTPRoute resource.

Reduced Blast Radius:

But wait…

How exactly does Gateway API reduce the blast radius? This will also

Let’s understand this with a real world example 😄. This will also help you to understand how Developers Can Move Independently

In most organizations using traditional Ingress, teams commonly share host such as: api.company.com and use path-based routing to route traffic to different applications. Something like this:

/payments → payment-service

/users → user-service

/orders → order-service

Now here comes the problem 👀. In many cases, all these routes live inside the same shared Ingress resource file. That means if the Payments team wants to make a change, they are editing a file that also contains routes for: Users Service, Order Service etc.

Now imagine someone accidentally:

• Updates the wrong route

• Creates a conflicting path

• Adds /orders without realizing it already exists

• Changes path priority incorrectly

Suddenly: Payments issue → Orders affected → Users affected → Unexpected production issue 😨

Why? Because everything is tightly coupled inside a shared Ingress configuration. This increases the blast radius. A small mistake from one team can potentially impact multiple services. And trust me this happens more often than people admit 😅

How Gateway API Fixes This

With Gateway API, teams are encouraged (and mostly forced by design) to create separate HTTPRoute resources for each application.

Even if everyone is still using: api.company.com with path-based routing, each application owns its own independent routing file. Example

payment-httproute.yaml

users-httproute.yaml

orders-httproute.yaml

Now if the Payments team makes a bad change they only touch payment-httproute.yaml if any issue only gets payment service gets affected. The users and orders routes continue working normally because their configurations are completely isolated. So now the blast radius becomes:

Ingress → One bad config can affect many apps 😨

Ingress → One bad config can affect many apps 😨 Gateway API → Mostly isolated to one app 😌

This is one of the easiest ways to remember why Gateway API improves both delegation and reliability in shared Kubernetes environments.

Pros

• Better separation of responsibilities

• Clear ownership model across teams

• Safer delegation to developers

• Reduced blast radius during failures

• More flexible and expressive routing rules

• Standardized Kubernetes-native API

• Supports advanced traffic management

• Easier to scale for large organizations

Cons

• More components to understand initially

• Slight learning curve compared to Ingress

• Ecosystem is still evolving in some areas

• Not every controller supports all Gateway API features equally

Should Gateway API Replace Ingress?

Not necessarily. Ingress is still heavily used and works perfectly fine for many organizations.

But if you are dealing with:

• Multiple teams

• Shared clusters

• Complex routing needs

• Delegation challenges

Gateway API becomes a much cleaner and scalable option.

Wrapping Up: Choosing Your Front Door 🚪

Exposing your applications to the outside world isn't a one size fits all decision. As we've seen, Kubernetes networking has evolved to solve real world operational headaches:

• NodePort is your quick and dirty tool for local debugging and testing.

• LoadBalancer gives you robust, cloud native connectivity but comes with a hefty price tag at scale.

• Ingress solves the cost problem by putting multiple apps behind a single cloud load balancer, but it can introduce operational bottlenecks and shared configuration risks.

• Gateway API is the modern evolution, breaking down those bottlenecks by splitting responsibilities so infrastructure, platform, and development teams can move independently without stepping on each other's toes.

If you are running a small project or a single app, Ingress or a simple LoadBalancer might be all you ever need. But if you are scaling across multiple teams in a shared cluster, adopting the Gateway API will save you from a lot of configuration conflicts and deployment delays down the road. 🚀

What’s Next? 🛠️

We’ve covered the high level architecture and the why behind each approach, but we’re just scratching the surface!

To really get deep into these tools, we need to look under the hood. In our upcoming posts, we will explore each of these options in deep, technical detail in separate blogs, including step by step implementation guides, YAML manifests, and production best practices.

Stay tuned for the next part, where we’ll roll up our sleeves and dive deep into each of these options! Until then, happy routing! ☕

That being said, in this blog, we will dive deeper into the different types of Services Kubernetes provides and understand when and why to use them in real world scenarios.

Kubernetes Services Types

We already know the purpose of a Kubernetes Service it provides a stable DNS name and a stable IP address, allowing communication to happen seamlessly without interruption.

We also learned that Pods cannot have stable IPs because of their ephemeral nature. A Pod can be terminated, recreated, or rescheduled at any time, which means its IP address can change. This is exactly why we place a Service on top of a Pod or Deployment to provide a stable entry point (stable IP and DNS) so applications can continue communicating regardless of what happens at the Pod or Deployment level.

Now that we understand why Services are needed, let’s look at the different Service types Kubernetes provides.

Kubernetes offers five types of Services, and each of them is designed for a specific purpose and use case. At the time of writing, Kubernetes provides the following Service types:

• ClusterIP (Default)

• NodePort

• Load Balancer

• Headless Service

• ExternalName

Each Service type solves a different networking problem. Some are meant for internal communication within the cluster, while others help expose applications externally.

In the upcoming sections, we will explore each Service type in detail, understand how they work behind the scenes, and discuss when to use them in real-world scenarios.

Cluster IP

This is the default Service type available in Kubernetes. If you do not explicitly specify a Service type, Kubernetes automatically creates it as a ClusterIP Service.

What it is ?

ClusterIP is a Service type that enables internal communication within the Kubernetes cluster. Instead of directly communicating with Pods using their changing Pod IPs, applications communicate through a stable Service IP and DNS name.

As we learned earlier, Pods are ephemeral in nature they can be recreated, restarted, or rescheduled at any time, which means their IP addresses may change. Kubernetes solves this problem by placing a ClusterIP Service in front of the Pods.

Behind the scenes, kube-proxy updates networking rules so traffic sent to the Service IP is automatically routed to the healthy backend Pods.

One important caveat here is that the application is not exposed to the public internet. Communication through a ClusterIP Service is restricted only within the cluster.

When this is used?

This is the most commonly used Service type in Kubernetes. In real world deployments, most applications run internally using ClusterIP Services.

But wait… if applications are internal, how do users access them?

This is where components like Ingress, Gateway API, Load Balancers, or NGINX come into the picture. Instead of exposing every application individually, organizations usually expose a common entry point that routes traffic internally to multiple ClusterIP Services.

Why is it done this way? There are multiple reasons for this approach:

• Cost optimization – Exposing every Service through a LoadBalancer can significantly increase infrastructure costs.

• Better traffic control – A central entry point gives teams more control over routing, authentication, rate limiting, and security.

• Avoid operational issues – Using NodePort for every application can lead to port management and conflict issues.

Because of these reasons, most organizations keep applications internal using ClusterIP and expose only selected endpoints externally through Ingress Controllers, Gateway APIs, or Load Balancers.

Try it yourself :

You can follow the practical example here to see how a ClusterIP Service works in action and understand how Pods communicate through a stable Service endpoint.

Node Port

This type of Service is mainly used for local testing, lab setups, or Proof of Concepts (PoCs). It is not very common in production environments, although there are some edge cases where teams still use it.

NodePort exposes an application outside the Kubernetes cluster by opening a specific port on every node in the cluster.

What it is ?

When you create a Service with the NodePort type, Kubernetes exposes your application externally using a port on the node.

This means you can access the application using: <NodeIP>:<NodePort>. For example: 192.168.1.10:30080.

By default, Kubernetes allocates a port from the 30000–32767 range, although you can manually specify a port if needed.

Behind the scenes, Kubernetes creates a ClusterIP Service first, and then the NodePort Service extends it by exposing that application externally through the node.

But wait… what if my Pods are running on different nodes?

Good question 🙂

Imagine your application Pod is running only on Node 2, but you accidentally access the NodePort of Node 1. Will it fail? No.

When Kubernetes creates a NodePort Service, the same port is opened on all nodes in the cluster. If traffic reaches a node where the Pod is not running, kube-proxy forwards the request internally to a healthy Pod running on another node.

For example:

Node 1 → No Pod running
Node 2 → Pod running
Node 3 → No Pod running

If you access: http://Node1IP:30080.The request will still work because Kubernetes internally forwards it to the Pod running on Node 2. This is one of the reasons why NodePort feels simple to use during testing.

One important caveat:

Although NodePort exposes applications externally, it is usually not ideal for internet facing production applications.

Why?

• Ports are limited and need manual management.

• You expose node level networking directly.

• Security becomes harder to manage.

• You need to know node IP addresses.

• Scaling and routing become difficult compared to using Ingress or Load Balancers.

When this is used?

NodePort is mostly used for:

• Local testing

• Learning Kubernetes

• Lab environments

• Proof of Concepts (PoCs)

• Temporary external access during debugging

In production environments, teams usually prefer LoadBalancer Services, Ingress Controllers, or Gateway APIs instead of directly exposing applications through NodePort.

Try it yourself :

Let’s create a simple NodePort Service and see how it works.

Step 1: Create an NGINX Deployment

#Create a file called `nginx-deployment.yaml`
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-app
  template:
    metadata:
      labels:
        app: nginx-app
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80

---
#Create a file called nginx-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-nodeport
spec:
  type: NodePort
  selector:
    app: nginx-app
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30080
---

#Apply it using following command
kubectl apply -f nginx-nodeport.yaml

#Check the Service
kubectl get svc

#Check nodes
kubectl get nodes -o wide

#Now Access
http://<NodeIP>:30080

# Try different node IPs, even nodes where the Pod is not running.

You can inspect the Service: kubectl describe svc nginx-nodeport

Check endpoints: kubectl get endpoints

This helps you understand which Pods are backing the Service.

Load Balancer

This is one of the most commonly used Service types in production grade Kubernetes environments to expose applications to the outside world.

But wait… does that mean every application gets its own Load Balancer? Not really.

As we discussed earlier, exposing every application using a LoadBalancer Service can become very expensive because each Load Balancer typically provisions a public IP and cloud resources.

You might be wondering: Then how do companies expose applications externally in production? Good question 🙂

In the real world, organizations usually expose only a few components such as Ingress Controllers (NGINX, Traefik, HAProxy, etc.) or Gateway APIs using a LoadBalancer Service.

Then, all application traffic is routed internally through these components to the actual backend services running as ClusterIP Services.

This approach helps organizations:

• Reduce infrastructure cost

• Centralize traffic management

• Improve security

• Manage routing in a scalable way

Instead of provisioning 50 Load Balancers for 50 applications, companies usually use one or a few Load Balancers and route traffic internally.

What it is ?

A LoadBalancer Service exposes your Kubernetes application to the public internet.

When you create a Service with the type LoadBalancer, Kubernetes works with the underlying cloud provider (AWS, Azure, GCP, etc.) to provision an external Load Balancer and assign a public IP address (or DNS endpoint).

This public endpoint becomes the entry point through which users can access your application. For example: http://<External-IP> or http://my-app.company.com

Behind the scenes, Kubernetes creates:

LoadBalancer → NodePort → ClusterIP → Pod

Did you notice it ? Yes, you read that right 🙂. A LoadBalancer Service internally uses NodePort.

Here is how the traffic flow works:

1. The external Load Balancer receives incoming traffic.

2. It forwards traffic to the NodePort opened on cluster nodes.

3. kube-proxy routes the request to the appropriate ClusterIP Service.

4. Finally, the traffic reaches a healthy backend Pod.

This is why when you create a LoadBalancer Service, you may notice a NodePort also getting created automatically.

One important thing to remember. A LoadBalancer Service works differently depending on where Kubernetes is running. In cloud managed Kubernetes environments such as: AWS EKS, Azure AKS, Google GKE

When is this used?

Use a LoadBalancer Service whenever you want to expose an application externally.

Typical use cases include:

• Public facing web applications

• APIs exposed to external consumers

• Applications requiring external access

However, in production, teams usually avoid assigning a separate Load Balancer to every application. Instead, they expose Ingress Controllers or Gateway APIs through a LoadBalancer and route traffic internally to backend services.

Try it yourself

You can follow the practical guide linked below to create a LoadBalancer Service, understand how it works internally, and see how Kubernetes exposes applications externally.

Headless Service

This is one of the most misunderstood Service types in Kubernetes. It is called Headless service because it doesn't have a stable Service IP which we used to have in all other services. At first glance, you may wonder:

Wait… why would anyone want a Service without a Service IP?

Good question 🙂

A Headless Service is mainly used when applications need to directly communicate with individual Pods, instead of a load-balanced Service endpoint.

What is it?

In a normal Kubernetes Service (ClusterIP), traffic flows like this: Application → Service IP → One of the Pods

Kubernetes automatically load balances requests across Pods. But in some cases, applications want to know:

Which exact Pod am I talking to?

For such scenarios, Kubernetes provides Headless Services. A Headless Service is created by setting: clusterIP: None

This tells Kubernetes: “Do not assign a Cluster IP. Give me direct Pod-level access.”

Instead of returning a single Service IP, Kubernetes DNS returns the individual Pod IPs behind the Service. For example:

Instead of:

my-app.default.svc.cluster.local → 10.96.10.20

You may get:

pod-1.my-app.default.svc.cluster.local
pod-2.my-app.default.svc.cluster.local
pod-3.my-app.default.svc.cluster.local

Why do we need this?

Let’s understand this with a simple Redis example. Assume you are running Redis in Kubernetes with:

redis-0 → Primary (Leader)
redis-1 → Replica
redis-2 → Replica

Now imagine you expose Redis using a normal ClusterIP Service.

What happens? A normal Service will load balance traffic randomly across all Redis Pods. So your application may connect like this:

Request 1 → redis-0 (Primary)
Request 2 → redis-2 (Replica)
Request 3 → redis-1 (Replica)

This becomes a problem.

Why? Because in Redis:

• Writes must go to the Primary node

• Reads may go to Replicas

• Replicas synchronize data from the Primary

If Kubernetes randomly sends traffic to any Redis Pod through a normal Service, your write requests may accidentally land on a Replica, which is not what we want.

So how do we solve this?

This is where Headless Services become useful.

Instead of giving one load-balanced Service IP, Kubernetes exposes the individual Pod DNS records.

For example:

redis-0.redis.default.svc.cluster.local
redis-1.redis.default.svc.cluster.local
redis-2.redis.default.svc.cluster.local

Now the Redis cluster (or your application) can directly talk to the specific Pod it wants. So here the app or the Redis Cluster will decide which one to use rather Kubernetes deciding for you.

For example:

• Write traffic → redis-0 (Primary)

• Read traffic → redis-1 or redis-2 (Replicas)

This gives applications full control over which Pod they communicate with.

And this is exactly why Headless Services are commonly used with StatefulSets and databases.

In short:

If your application needs Pod level awareness, a normal Service may not be enough and that is where Headless Services shine 🙂

When is this used?

Headless Services are commonly used with:

• StatefulSets

• Distributed databases

• Clustered applications

• Messaging systems

• Peer-to-peer communication between Pods

In most regular stateless applications, you probably do not need Headless Services. A standard ClusterIP Service is usually enough.

Try it yourself

You can follow the practical guide to create a Headless Service, understand how it works.

ExternalName

This is probably the least commonly used Service type in Kubernetes, but it can still be very useful in certain situations. Unlike other Service types, ExternalName does not create a ClusterIP, NodePort, or LoadBalancer.

Wait… then what exactly does it do? An ExternalName Service simply acts as a DNS alias (mapping) inside Kubernetes. Instead of routing traffic to Pods inside the cluster, it redirects requests to an external DNS name.

What is it?

Imagine your application running inside Kubernetes needs to communicate with an external database, API, or third-party service.

Normally, your application would directly call: mydb.company.com

But what if the endpoint changes later? You would need to update configurations in multiple places.

This is where ExternalName becomes useful. You can create a Kubernetes Service that maps:

database-service.default.svc.cluster.local

to:

mydb.company.com

Now your application continues using the Kubernetes Service name, while Kubernetes redirects the request to the external DNS.

Behind the scenes, Kubernetes simply returns a CNAME DNS record. There is no proxying, no kube-proxy rules, and no Service IP involved.

Example

Suppose your payment service db is running outside Kubernetes: mydb.abc123.us-east-1.rds.amazonaws.com. Instead of hardcoding it in multiple applications, create an ExternalName Service:

apiVersion: v1
kind: Service
metadata:
  name: payment-service-db
spec:
  type: ExternalName
  externalName: mydb.abc123.us-east-1.rds.amazonaws.com

Now applications inside Kubernetes can simply access: payment-service-db.default.svc.cluster.local. Kubernetes will automatically redirect traffic to: mydb.abc123.us-east-1.rds.amazonaws.com

When is this used?

ExternalName is commonly used when:

• Connecting Kubernetes apps to external databases

• Connecting to third-party APIs

• Migrating workloads gradually to Kubernetes

• Avoiding hardcoded external URLs in applications

• Creating consistent service naming conventions

One important caveat:

Since this only works using DNS mapping, Kubernetes does not perform load balancing or health checks. If the external service is unavailable, Kubernetes cannot help much because the traffic is going directly to the external endpoint.

So think of this more like a DNS shortcut rather than a traditional Kubernetes Service.

You might now wonder: If this is useful, why do many teams not use it often?

Good question 🙂 In real-world production environments, many organizations prefer using:

• Environment variables

• Centralized configuration management

• Service discovery tools

• API gateways or service meshes

instead of relying on ExternalName. Why?

Because ExternalName has a few limitations:

• No traffic control – Kubernetes cannot route or balance traffic.

• No observability – Monitoring and tracing become harder.

• No failover mechanism – If the external endpoint fails, Kubernetes cannot automatically redirect traffic.

• DNS dependency – Everything depends on proper DNS resolution.

Also, many modern architectures already manage external endpoints through configuration management systems, making ExternalName less necessary.

Conclusion

That brings us to the end of this blog. In this blog, we explored the different types of Kubernetes Services and understood why each Service type exists and what problem it solves.

We started with ClusterIP, the default Service type used for internal communication inside the cluster. Then we looked at NodePort, which helps expose applications externally for testing and lab environments. After that, we explored LoadBalancer, one of the most common approaches used in production to expose applications to the outside world.

We also discussed ExternalName, which acts more like a DNS mapping for external dependencies, and finally Headless Services, which are useful when applications need direct Pod-level communication instead of load balancing.

One important thing to remember is:

There is no “best” Service type in Kubernetes.

The right Service depends entirely on your use case, traffic pattern, security requirements, and architecture design.

For example:

• Need internal communication? → ClusterIP

• Need quick testing access? → NodePort

• Need external exposure? → LoadBalancer

• Need external DNS mapping? → ExternalName

• Need Pod-level discovery? → Headless Service

Also, in real world production environments, you will rarely see applications directly exposed using NodePort or individual LoadBalancers. Most organizations combine these Service types with Ingress Controllers or Gateway APIs to manage traffic in a more scalable and cost-effective way.

Now that we understand how Services work and the different ways applications are exposed, in the upcoming blogs we will dive deeper into Ingress, Gateway API, kube-proxy, and Kubernetes traffic flow behind the scenes.

Until then, try out the practical examples, break things, experiment, and most importantly learn by doing 🙂

In this series, we will dive deep into Kubernetes networking such as: how Pods communicate with each other, why Services are needed, the different types of Services, and much more.

So, buckle up: things are about to get interesting. Let’s get started 🚀

Phase 1: How Do Pods Communicate with Each Other?

From our previous blog, we learned that a Pod is the smallest deployable unit in Kubernetes. A Pod can have one or more containers, and this is where our application actually runs.

Now, if our application is running inside a Pod, we need some way to access it, right?

This is where IP addresses come into the picture. Every Pod in Kubernetes gets its own unique IP address. And here comes one of the coolest things about Kubernetes networking Pods can communicate with other Pods in the cluster directly. No NAT. No complex routing setup. No communication barriers (by default). This concept is called the Flat Network Model.

Why is it called a Flat Network Model?

We call it a flat network because:

• Every Pod gets a unique IP address

• Pods can communicate with each other across nodes

• No need for port mapping between Pods

• Pods can directly reach other Pods without an additional networking layer

Mental model

Pod A (10.244.1.2) → Pod B (10.244.2.5)

👉 Works directly even if on different nodes

🔥 Why Did Kubernetes Choose This Networking Model?

Before we understand why Kubernetes chose this approach, let’s first see how networking usually worked in traditional container environments.

Traditional Approach: Container → NAT → Host → NAT → Another container. This adds extra networking layers, port mappings, and sometimes additional complexity.

Kubernetes Approach: Pod → Pod (direct). No manual port mapping. No complicated networking setup between Pods. Kubernetes networking is designed so Pods can directly reach each other using their IP addresses.

✅ Benefits of This Approach

• Simpler service discovery - Pods and Services can communicate more predictably inside the cluster.

• No port conflict - Since every Pod gets its own IP address, multiple applications can use the same port without conflicts.

• Cleaner Architecture - Networking becomes easier to reason about, making application communication more straightforward.

This is one of the key reasons Kubernetes is designed around a direct Pod-to-Pod communication model to make distributed systems easier to build and operate.

🔌 How Pod Networking Actually Works

So far, we learned that every Pod gets its own IP address and can communicate with other Pods in the cluster. Sounds cool, right?

But have you ever wondered what actually happens behind the scenes? How are these Pods connected? Who manages all this networking magic? Let’s break it down.

🛠️ The Behind-the-Scenes Magic

• Every Pod Gets Its Own Network Namespace

  • Every pod gets: Its own network namespace and Its own unique IP address?

  • Network Namespace ? Think of a network namespace as a mini network environment for a Pod. It gets its own network interfaces, routing table, and network stack, isolated from other Pods. This is the reason why each Pod behaves almost like a separate machine in the cluster.

• How Are Pods Connected?

  • Now comes the interesting part how do Pods actually talk to each other?

  • Pods are typically connected using something called veth pairs (Virtual Ethernet pairs).

  • You can think of a veth pair like a virtual network cable: One end connects to the Pod and the other end connects to the host network.

  • From there, traffic is handled using mechanisms such as Linux bridges, routing, overlay networks, or eBPF, depending on the networking solution being used. Don’t worry if this sounds complicated right now we will simplify this step by step in upcoming blogs 😄

• Who Manages All of This?

  • This is where the real rockstar enters the stage the CNI Plugin (Container Network Interface) 🎸

  • The CNI plugin is responsible for:

    • Assigning an IP address to Pods

    • Setting up the network interfaces

    • Creating required routing/network rules

    • Handling Pod-to-Pod communication across nodes

  • In short, without a CNI plugin, Kubernetes networking simply would not work.

  • 👀 Who Are These CNI Plugins? You might have heard names like: Flannel, Calico, Cilium. These are popular CNI plugins used in Kubernetes for networking.

  • Different companies choose different networking solutions based on their needs. Some prefer simplicity, while others need advanced networking, security policies, or high performance.

Curious to know what CNI plugin your Kubernetes cluster is using? Run the below command:

kubectl get pods -n kube-system

Look for Pods with names related to networking. You might notice something familiar:

• calico-* → Using Calico

• cilium-* → Using Cilium

• kube-flannel-* → Using Flannel

• aws-node → Usually AWS VPC CNI (common in EKS)

For example, if you see Pods like cilium-xyz123, congratulations your cluster is most likely using Cilium 😄. Here is my example

🧪 Want to Try It Yourself?

If you want to do a hands on practical and see how Pod-to-Pod communication actually works, follow the document to test Pod-to-Pod communication in Kubernetes.

The BIG problem this creates

We learned in our previous blog that Pods are ephemeral in nature.

Ephemeral? This means Pods are temporary if a Pod crashes, gets deleted, or is rescheduled, Kubernetes creates a new one (usually through controllers like Deployments).

But here’s the catch when a new Pod gets created, its IP address changes. Now imagine the headache this creates 😅

If Pod IPs keep changing constantly, how can other Pods reliably communicate with them? Tracking Pod IP addresses manually would be a tedious and messy job.

This is exactly where Kubernetes Services come to the rescue. Instead of talking directly to constantly changing Pod IPs, Services provide a stable way to access Pods, even when Pods come and go behind the scenes.

Phase 2: Services — The Backbone of Kubernetes Networking

We understood one important problem with Pods: they are ephemeral.

Luckily, Kubernetes has a smart solution for this problem Services. But the big question is: How do Services actually solve this issue?

How Services Resolve This Problem 🤔 ?

A Kubernetes Service provides:

• A Stable Virtual IP

• A Stable DNS name

This stable identity dynamically maps to the Pods running your application. Think of it like this:

• Instead of applications directly talking to Pod IPs (which keep changing), they talk to a Service.

• The Service acts like a stable entry point and forwards requests to the correct Pods behind the scenes.

So, your setup looks something like this: pod-b → Service → pod-a (new Pod)

Every Pod created by your application (usually managed by a Deployment) gets connected to the Service through labels and selectors.

The important part? The Service IP and DNS name stay stable, even if Pods are destroyed and recreated. We will learn how Kubernetes actually does this magic behind the scenes shortly 😄

Why Is This Powerful?

No matter how many times your Pods restart… No matter what new IP addresses your Pods get… Your application always talks to the Service, not directly to the Pods. The Service takes care of routing your request to the correct Pod and returning the response.

✨ How Kubernetes Services Actually Do the Magic

Now comes the interesting part how does a Kubernetes Service actually take your request to the right Pod? Until now, we learned that Services provide a stable IP and DNS name. But what’s happening behind the scenes that makes this possible? Let’s break it down step by step.

The Basics

We already know what a Deployment is. A Deployment manages multiple replicas of your application and takes care of things like: Auto healing (self-recovery), Rolling Updates and Scaling

Let’s say you deploy your application with 3 replicas. What happens next?

The Flow Behind the Scenes

API Server → ETCD → Controller Manager → Scheduler → Kubelet → Container Runtime + CNI

Let’s understand this one by one:

You create a Deployment: You apply your Deployment manifest.

• API Server updates ETCD: The API Server stores the desired state of the Deployment inside ETCD (Kubernetes' database).

• Controller Manager notices the Deployment: The Controller Manager sees that 3 replicas are required and starts creating them.

• Scheduler assigns Nodes: The Scheduler notices new Pods are created but not assigned to any Node yet, so it decides where each Pod should run.

• Kubelet starts Pod creation: The Kubelet running on each Node notices that a Pod has been assigned to it and starts the Pod creation process.

• Container Runtime creates containers: The container runtime (like containerd or CRI-O) creates the containers.

• CNI Plugin joins the party 🎉: The CNI plugin: Assigns IP addresses to Pods, Configures networking, Creates routes required for Pod communication

Parallely Services are created as per the manifest. And now… this is where the real magic starts 👀 Enter the EndpointSlice Controller

EndpointSlice Controller

Kubernetes has something called the EndpointSlice Controller. Its job is to continuously watch for newly created Services. When a Service is created, the controller:

• Reads the selector labels from the Service definition

• Finds Pods with matching labels in the same namespace

• Collects the Pod IP addresses

• Creates an EndpointSlice object

Think of an EndpointSlice as a list of healthy Pod IPs behind a Service. Pretty cool, right?

But Wait… It Doesn't Add Every Pod Immediately. The EndpointSlice Controller is smart. It does not blindly add Pod IPs. Instead, it checks whether the Pod is actually:

• Running

• Healthy

• Ready to accept traffic

If you configure a Readiness Probe, Kubernetes waits for the Pod to become Ready = True before adding that Pod IP to the EndpointSlice. This prevents traffic from going to unhealthy Pods.

What Happens Next ?

The Kube-Proxy Show

Now kube-proxy takes over. kube-proxy watches these EndpointSlices and creates networking rules (usually using iptables or IPVS, depending on configuration). These rules make sure traffic sent to the Service gets forwarded to one of the healthy Pod IPs behind it.

In simple words: Service → kube-proxy → EndpointSlice → Healthy Pod

A Small but Important Detail: Whenever you create a Service, Kubernetes automatically creates an EndpointSlice for it.

You can actually see them using: kubectl get endpointslices

You’ll notice names similar to your Service name with some random suffix added.

What If a Pod Goes Down?

Imagine one Pod crashes unexpectedly. No worries — Kubernetes has your back 😄

The EndpointSlice Controller notices that the Pod is no longer healthy and removes that Pod IP from the EndpointSlice. This means traffic will stop going to the failed Pod.

When a healthy replacement Pod comes up, Kubernetes automatically adds the new Pod IP back.

So, What’s the Final Magic?

When your application receives a request:

1. The request first hits the Service

2. kube-proxy checks the available healthy endpoints

3. Traffic gets routed to one of the healthy Pod IPs

4. Your Pod processes the request and sends the response back

And the best part?

You never have to care about changing Pod IPs.

Kubernetes handles all the heavy lifting behind the scenes 🚀

🧪 Want to Try It Yourself?

If you want to do a hands on practical and see how Pod and Service actually works, follow the document to test Pod and Service communication in Kubernetes.

Trust me trying it practically makes Kubernetes networking much easier to understand than just reading theory 😄

🛠️ Troubleshooting Communication Issues

Let’s see some high-level troubleshooting steps to debug Kubernetes Service related issues.

Note: These steps only cover troubleshooting from the Service layer level.

1. Check if the Service Exists

kubectl get svc

Make sure:

• The Service exists

• The correct port is exposed

• The ClusterIP is assigned

2. Check Service Details

kubectl describe svc

Look for:

• Correct selector labels

• Correct targetPort

• Correct exposed port

A label mismatch is one of the most common issues beginners face.

3. Check if Pods Are Running

kubectl get pods -o wide

Verify:

• Pods are in Running state

• Pods are READY

• Pod IPs are assigned

4. Check EndpointSlices (Very Important 👀)

kubectl get endpointslices

kubectl describe endpointslice

This is where the real truth lives 😄 Check whether Pod IPs are actually present.

If you don’t see Pod IPs here, Kubernetes has no healthy backend to route traffic to.

5. Check Pod Labels

kubectl get pods --show-labels

Sometimes Pods exist, but labels don’t match the Service selector.

No matching labels = No endpoints.

6. Check Readiness Probe Status

kubectl describe pod

Look for:

• Ready: True

• Probe failures

• Events section

If readiness checks fail, the Pod will not be added to EndpointSlice, meaning Services won’t send traffic to it.

7. Test Connectivity from Another Pod

kubectl exec -it -- sh

# Then test:

curl http://<service-name>

#or

nslookup <service-name>

This helps verify DNS and Service connectivity. Most Kubernetes networking issues can be debugged by checking these few things:

Service → Labels → Pods → EndpointSlice → kube-proxy → Connectivity

🎯 Wrapping Up — What We Learned

And there you go we just uncovered the fundamentals of Kubernetes Networking 🚀

We started with understanding how Pods communicate with each other, why Kubernetes follows a Flat Network Model, and how CNI plugins quietly handle all the networking magic behind the scenes.

Then we explored the big problem with Pods they are ephemeral, meaning their IPs keep changing. This led us to one of the most important building blocks in Kubernetes networking: Services.

We learned how Services provide:

• A stable virtual IP

• A stable DNS name

• A reliable way for applications to communicate without worrying about changing Pod IPs

And finally, we went under the hood to understand the real magic:

Service → EndpointSlice → kube-proxy → Healthy Pod

Now we know how Kubernetes Services actually route traffic to the right Pods behind the scenes 😄

⏭️ What’s Next?

So far, we focused on the fundamentals and understood why Services exist and how they work behind the scenes. But Kubernetes Services come in different flavors 😄

In the next blog, we will deep dive into the Types of Kubernetes Services. Until then, happy learning and keep experimenting 🚀

But if you are someone like me, you might have paused for a second and wondered:

“Wait… what exactly are all these tools?”
“Where are they actually used?”
“Why do we even need so many networking components in Kubernetes?”

Trust me, I had the exact same questions.

At first, terms like Ingress, Gateway API, CNI, Service Mesh, Cilium, Istio, Envoy, MetalLB, and Load Balancer can feel like a giant networking soup 🍜.

But here’s the interesting part: Most of these tools are not competing with each other. Instead, they solve different networking problems at different layers of Kubernetes.

So in this blog, we are going to take a fun and practical deep dive into:

✅ Kubernetes networking basics
✅ What happens when traffic enters a Kubernetes cluster
✅ Which networking tools are used where
✅ When to use what (without getting confused!)

By the end of this blog, whenever someone mentions a Kubernetes networking tool, you should be able to say:

“Ah okay, I know exactly where this fits in.”

So grab your coffee ☕ (or tea 😄), and let’s get started.

The Basics: How traffic flows in Kubernetes ?

Before we start throwing around fancy names like Cilium, Istio, Gateway API, Envoy, MetalLB, or Service Mesh, let’s first understand something important:

How does a normal request travel inside Kubernetes? Because once we understand the journey of a request, understanding networking tools becomes much easier.

When a request comes from the internet to your application running inside Kubernetes, it goes through multiple layers before reaching the actual Pod.

A request flowing inside Kubernetes typically goes through the following stages:

Internet → Ingress / Gateway Layer → Service Layer → Finds the Correct Backend Pod → Pod Networking Layer → Delivers Traffic to the Pod → Application Pod → Serves the Request

The request originates from an external user or application. For example: https://myapp.com

At this point, Kubernetes needs a way to expose your application to the outside world. So the traffic enters the cluster through the:

Ingress / Gateway Layer

• Ingress / Gateway Layer → Controls External Traffic.

• This layer acts as the entry point of the cluster.

Here Kubernetes decides:

• Which application/service the request should go to URL/path-based routing (/payment, /user, /orders)

• Authentication and authorization

• TLS/HTTPS termination

• Rate limiting and traffic filtering

Once the request rules are evaluated, the traffic is forwarded to the:

Service Layer

• Service Layer → Finds the Correct Backend Pod

• Services act as a stable endpoint for Pods. Since Pods are temporary and their IPs can change, Services help Kubernetes reliably find the correct application Pods.

Here Kubernetes:

• Finds matching Pods using labels

• Performs load balancing Routes traffic to healthy Pods

• Enables service-to-service communication

Once the correct Pod endpoint is identified, traffic is forwarded through:

Pod Networking Layer

• Pod Networking Layer → Delivers Traffic to the Pod

• This is where the actual packet routing happens. The networking layer ensures Pods can communicate: Within the same node, Across different nodes, Across namespaces, Across the cluster.

Here Kubernetes networking components:

• Assign IP addresses to Pods,

• Route traffic between Pods

• Handle inter-node communication

• Apply networking rules

Finally, the request reaches:

Application Pod

• Application Pod → Serves the Request

• This is where your actual application runs. The Pod processes the request and sends the response back through the same networking path.

Security & Observability → Monitors the Entire Journey

While the request travels through Kubernetes, security and monitoring tools continuously observe the traffic.

This layer helps with:

• Restricting communication between services

• Monitoring traffic flow Tracking latency and failures

• Applying network policies

• Observability and troubleshooting

This happens across all layers, not just at one specific point.

Now that we understand how traffic flows inside Kubernetes, the next obvious question is:

What tools are used at each layer, and what problem do they solve?

Deep Dive into Kubernetes Networking Tools

As we saw earlier, a request travels through different networking layers before reaching the application Pod.

Each layer has its own responsibility, and naturally, different tools are built to solve problems at those specific layers. So let’s deep dive into the tools layer by layer and understand:

• What does this layer handle?

• Popular Tools Available

1. Ingress/Gateway Layer:

As discussed earlier, the Ingress/Gateway Layer acts as the entry point into the Kubernetes cluster. This layer is mainly responsible for handling incoming external traffic and deciding how requests should enter and move inside the cluster.

What does this layer handle?

This layer controls:

• Which application/service the request should go to

• URL / Path-based routing

• Authentication & Authorization

• TLS / HTTPS Termination - Handling HTTPS Encryption

• Rate Limiting & Traffic Filtering

Popular Tools Used in the Ingress / Gateway Layer

• Ingress Controllers: NGINX Ingress Controller, HAProxy Ingress, Traefik Ingress, AWS Load Balancer Controller, Azure Application Gateway Ingress Controller

• NGINX Gateway / Gateway API: Modern implementation for managing external traffic using the Kubernetes Gateway API.

• Traefik: A lightweight and cloud-native ingress/gateway solution.

• Cilium Gateway: Part of the Cilium ecosystem. Uses eBPF-powered networking and implements Gateway API capabilities.

• Istio Gateway: Part of the Istio Service Mesh ecosystem.

2. Service Layer Tools

Once the traffic enters the cluster through the Ingress/Gateway Layer, it is forwarded to a Kubernetes Service.

But here’s the challenge: Pods in Kubernetes are temporary (ephemeral). So how does Kubernetes make sure traffic always reaches the correct Pod? This is where the Service Layer comes into play.

The Service Layer acts as the traffic manager inside the cluster and ensures requests are routed reliably to healthy application Pods.

What Does the Service Layer Handle?

• Service Discovery: Services help applications find each other without hardcoding Pod IPs.

• Load Balancing Across Pods: Traffic is distributed across Pods.

• Service-to-Service Communication: Inside Kubernetes, services constantly communicate with each other.

• Internal Traffic Encryption (mTLS): Normally, traffic inside the cluster is Plain Text. Because TLS is often terminated at the Ingress layer. But industries like Banking, Healthcare etc may require Encrypted communication between services inside the cluster as well. This is where mTLS (Mutual TLS) becomes important.

Popular Tools Used in the Service Layer

• kube-proxy: This is the default Kubernetes networking component responsible for service routing.

• Istio: One of the most popular Service Mesh tools. Istio adds advanced traffic management between services.

• Linkerd: A lightweight Service Mesh alternative.

• Consul Service Mesh: Another Service Mesh offering. It works beyond Kubernetes as well.

• Cilium Service Mesh: If you are already using Cilium, it can also provide Service Mesh capabilities without traditional sidecars.

Note: One important thing to understand is that not all tools provide the same capabilities.

For example, kube-proxy handles service routing and load balancing, but it does not provide features like mTLS (Mutual TLS), advanced traffic management, or observability.

If your requirement includes secure service-to-service communication (mTLS), traffic control, retries, or advanced visibility, you would typically need a Service Mesh like Istio, Linkerd, Consul, or solutions like Cilium Service Mesh.

The tools listed above are some of the popular options, and the right choice depends on your use case, complexity, security requirements, and operational needs.

3. Pod Networking Tools

Now the request knows which Pod it should go to. But another important question comes up:

How does Kubernetes know:

• Where the Pod is?

• What its IP address is?

• How to route traffic across nodes?

• How Pods communicate reliably?

This is where Pod Networking comes into play. Honestly, this is one of the most important layers in Kubernetes networking because:

Without Pod Networking, Pods simply cannot talk to each other.

What Does Pod Networking Handle?

• Assigning IP Addresses to Pods: Every Pod gets its own IP address

• Pod-to-Pod Communication: Within the same node, Across different nodes, Across namespaces without requiring NAT or complicated translations.

• Cross-Node Networking: A CNI((Container Network Interface)) plugin handles the actual networking implementation.

• Traffic Routing Between Nodes: This decide, how traffic moves, which node owns which pod IP range, how traffic gets routed

• Network Policies: This layer can also enforce rules. Just because Pods can communicate doesn’t mean they always should. Security restrictions matter.

Popular Tools Used in Pod Networking

• Flannel: One of the simplest Kubernetes networking tools. Flannel focuses mainly on Pod-to-Pod communication. This doesn't have all the above capabilities.

• Calico: One of the most widely used Kubernetes networking solutions. This has Strong Network Policy support. Preferred choice in Enterprise, Prod Cluster and Security-focused environments

• Cilium: Probably one of the hottest Kubernetes networking tools right now. It uses eBPF for traffic communication instead of traditional Linux Networking. Cilium is a complete ecosystem which span across all layers

• AWS VPC CNI / Cloud Provider CNIs: Cloud providers also offer native networking plugins. AWS VPC CNI, Azure CNI, Google Kubernetes Engine CNI

Note: In real world Kubernetes environments, it is common to use multiple networking tools together.

For example, in cloud-managed Kubernetes services like EKS, companies often use the cloud-native CNI plugin (such as AWS VPC CNI) for Pod communication and add tools like Calico or Cilium on top for additional features such as Network Policies, security, and observability.

4. Security & Observability Tools

At this point, traffic is successfully flowing inside the cluster. Requests are reaching the correct Pods. But here comes an important question:

How do we control who can talk to whom?
How do we monitor what is happening inside the cluster?

This is where Security & Observability tools come into the picture.

This layer helps us:

• Restrict communication between Pods and services

• Monitor traffic flow

• Debug networking problems

• Understand service dependencies

• Improve visibility into cluster communication

What Does This Layer Handle?

• Network Policies: Who can communicate with whom

• Traffic Observability: Adding detailed visibility to the network flows

• Security Monitoring: Identifying unusual traffic, Policy violations etc

Popular Tools Used in This Layer

• Kubernetes Network Policies: This is the native Kubernetes way to restrict Pod communication. Kubernetes Network Policies alone do not enforce anything you need compatible CNI like cilium or calico

• Calico: Apart from Pod networking, Calico provides strong Network Policy enforcement, Traffic Filtering, and Security Isolation

• Cilium + Hubble: Cilium takes security and observability to another level, along with many of the features offered by Calico. It provides advanced networking, security, and traffic visibility capabilities. Hubble is Cilium’s visualization tool

• Istio Observability: Similar to Cilium, Istio also provides security and observability features. If you are already using Istio as your Service Mesh, you can make use of its built-in capabilities.

Conclusion

By now, you should have a better understanding of the different Kubernetes networking tools, where they fit in the networking flow, and what problem each of them is trying to solve.

Some tools help with:

• Exposing applications to the outside world (Ingress / Gateway)

• Service-to-service communication (Service Layer / Service Mesh)

• Pod communication across nodes (CNI / Pod Networking)

• Security, visibility, and traffic monitoring (Observability & Network Policies)

So the next time you hear terms like Cilium, Calico, Istio, NGINX Gateway, or Traefik, hopefully you will have a clearer idea of:

Where they fit and what their purpose is.

For a quick recap and better understanding, feel free to refer to the Flow Diagram.

I hope this blog helped simplify Kubernetes networking a little 😄

Happy Learning! 🚀
See you in the next one.

In this blog, we’ll explore some important Terraform insights and nitty-gritties that are extremely useful for day to day work. These are the kind of concepts that not only make you better at Terraform but also give you an edge in technical discussions and interviews.

The blog is structured in a simple and practical way. Each topic will cover:

• What it is – A clear explanation of the concept

• Why and Where it is used – Real-world scenarios where it applies

• How to use it – Practical usage with Terraform

• Example – So you can relate and apply it immediately

This approach ensures that you don’t just understand the concept, you know exactly how to use it.

Let’s get started 🚀

How do we structure the terraform code for large project

One of the most common questions we encounter is: “How do we structure Terraform code for a large project?”

This becomes even more critical when working on a greenfield setup, where everything is being built from scratch. The decisions you make at this stage will directly impact how well your infrastructure scales, how easy it is to maintain, and how effectively teams can collaborate.

So, how do we design a Terraform project that not only works today but also scales seamlessly in the future? Let’s dive deeper into this.

What it is ?

Before diving into how to structure a Terraform project, let’s first get the basics clear: What exactly are we structuring? What are we really talking about here?

Every Terraform project is made up of a set of core files and components. The real challenge is not creating these files, it’s about organizing them in a way that scales with your infrastructure.

Core Components in a Terraform Project

• Modules: Modules are the building blocks of any Terraform project. It helps you to Reuse Code, Maintain Consistency and Avoid Duplication. A typical module consists of:

  • main.tf → Defines the resources

  • variables.tf → Input variables.

  • outputs.tf → Exposes values

  • provider.tf → Provider configuration

  • versions.tf → Terraform & provider version constraints

  • data.tf → Data sources

  • backend.tf → Remote state configuration (sometimes kept separate mostly added with versions.tf)

• State File Management: Terraform maintains a state file to track the current state of your infrastructure. Proper state management is critical for Preventing Conflicts, Ensuring Consistency and Enabling team collaboration.

  • Key considerations:

    • Where is the state stored? (local vs remote),

    • How is state locking handled?

    • How do multiple users safely interact with it?

    • ⚠️ Pain Point: Large State File Slowing Down Terraform: When all resources are managed in a single Terraform state file, it grows significantly in size and slows down deployments because Terraform must evaluate the entire state during every run; to avoid this, it’s a best practice to split state files logically—either by resource type or architectural design to improve performance, scalability, and maintainability.

• Multi-Environment & Multi-Region Deployment: Real-world infrastructure is rarely single-environment. This involves Structuring Folders properly, Passing Environment specific variables, Managing isolated State files

  • You need to manage deployments across Environments (dev, stag and prod etc..) and across region (us-east-1, eu-west-1 etc..)

• Input Variables & Defaults Management: Variables make your Terraform code dynamic and reusable. Doing a right setup here will allow our code base to work across different region and enviornment

  • Things to Consider

    • Defining clear input variables

    • Setting sensible defaults

    • Overriding values per environment

So, when we talk about structuring Terraform, we are essentially talking about how we organize these building blocks effectively.

In Simple Terms

Structuring Terraform is about:

Organizing modules, state, variables, and environments in a way that your code scales along with your infrastructure without breaking performance, security, or maintainability.

Now we understand the what part let's discuss the why part.

Why?

Let's talk about why this is needed, As your infrastructure grows, your Terraform code should be able to:

• Handle complexity without becoming messy: As more components (infra resource) are added, the codebase should remain clean, well organized, and easy to debug or navigate.

• Support multiple environments and regions: The same codebase should support deployments across different environments (dev, staging, prod) and regions with minimal or no changes to the core logic.

• Maintain performance and security: Execution performance and security standards should remain consistent or improve as the infrastructure scales, without degradation.

• Enable easy collaboration across teams: As infrastructure grows, more team members will contribute. The code should be structured in a way that enables seamless collaboration and reduces conflicts.

By arranging them in this way helps you to grow your IaC seamlessly as your infrastructure grows.

How ?

When starting a greenfield Infrastructure as Code (IaC) project, there are multiple ways to structure and manage your infrastructure. Choosing the right approach early helps improve scalability, maintainability, collaboration, and operational efficiency as the infrastructure grows.

In this section, we will discuss two commonly used approaches for organizing Terraform based IaC projects:

1. Terraform Workspace Approach

2. Modular Folder-Based Approach

Terraform Workspace Approach

What is Terraform Workspace?

Terraform Workspaces allow you to manage multiple instances of the same infrastructure configuration using a single Terraform codebase.

Each workspace maintains its own separate state file, enabling you to deploy the same infrastructure into different environments such as:

• Development (dev), Testing (test), Staging (stage), Production (prod)

Instead of maintaining separate folders or separate state files manually, Terraform automatically isolates the infrastructure state per workspace.

Why Use Terraform Workspaces?

As infrastructure grows, managing multiple environments becomes challenging. Teams often need identical infrastructure across environments with only minor differences such as:

• Instance size, Number of replicas, Database sizing

Terraform Workspaces help reduce duplication by allowing the same code to be reused across environments while keeping the infrastructure states isolated.

This approach improves:

• Code reusability

• Consistency across environments

• Simpler maintenance

• Faster onboarding

Where Can Terraform Workspaces Be Used?

1. Multiple Similar Environments Exist - Dev, Qa, Prod where you will use same infra setup while only minor changes are required

2. Platform or Shared Services Teams: Teams managing reusable infrastructure templates can deploy the same stack for different customers, regions, or business units.

3. Temporary or Feature Environments: Useful fof Feature branch testing, Sandbox testing

How Terraform Workspace Works

Consider a hotel where:

• The kitchen prepares a common base meal for all guests.

• Every guest stays in a separate room.

• Guests can customize their food in their own room by adding available ingredients such as:

  • Pepper

  • Salt

  • Sauces

  • Spices

The chef does not cook completely different meals for every room. Instead, one common meal is prepared and each room customizes it based on individual preferences.

Relating This to Terraform Workspaces

In Terraform:

• The main Terraform code acts like the hotel kitchen.

• Each workspace acts like a separate room/environment.

• The common infrastructure code remains the same for all environments.

• Each workspace can customize values such as:

  • Instance size

  • Resource count

  • Naming conventions

  • Scaling configuration

  • Environment variables

without changing the core infrastructure code.

How to create workspace and work with that?

• To list the available workspace you use: terraform workspace list

• To Create new workspace : terraform workspace new <workspace name>

  • terraform workspace new dev

  • terraform workspace new prod

• Switch between workspace: terraform workspace select dev

In a real-world setup, Terraform Workspaces are commonly used by creating a separate workspace for each environment such as dev, test, and prod. The typical workflow looks like this:

• Create and maintain separate workspaces for each environment

• Select the required workspace before performing any operation terraform workspace select dev

• Verify that you are in the correct workspace terraform workspace show

• Run Terraform commands, which will apply the infrastructure using the variables and state associated with that workspace terraform apply

Pros:

• Reduced Code Duplication

• Easier Maintenance

• Faster Environment Creation

• Consistency Across Environments

• Simplified State Isolation

Cons:

• Shared Backend Complexity: All workspaces typically share the same backend configuration. So all the backed files are in same location so you need to create a complex rules to restrict user to access the backend statefiles.

• Risk of Human Error: Accidentally applying changes in the wrong workspace can impact production.

• Difficult CI/CD Integration at Scale: As the number of environment increases Pipeline management becomes harder, Environment-specific approvals become complex, State access management becomes difficult

• Limited Environment Customization: Workspaces work best when environments are nearly identical.

Use Workspaces When

✅ Infrastructure is mostly identical
✅ Environment differences are minimal
✅ Small-to-medium scale projects
✅ Rapid environment provisioning is needed

If that is the case how the other modular folder based approach solve the problem let's discuss that.

Modular Folder-Based Approach

The Modular Folder-Based Approach is a Terraform project structure where infrastructure is organized into:

• Reusable Modules

• Environment-specific folders

Instead of using a single Terraform configuration for all environments, each environment has its own dedicated folder and configuration while sharing reusable infrastructure modules. This approach separates:

• Reusable infrastructure logic

• Environment-specific configurations

making the infrastructure easier to scale, maintain, and manage. This is the industry wide popular approach used to manage large projects

Why Use the Modular Folder-Based Approach?

As infrastructure grows, environments usually become different from each other. For example:

• Production may require: Multi-region deployment, High Availability, Autoscaling etc which is not needed for Dev

• For Dev we can use low cost instances and minimum setup to test the code.

Managing these differences using only Terraform Workspaces can become complicated. The Modular Folder-Based Approach solves this by:

• Separating environments completely

• Reusing infrastructure components through modules

• Allowing each environment to evolve independently

Where Can This Approach Be Used?

As mentioned this is the industry wide popular approach,

• Enterprise Infrastructure

• Production-Grade Platforms

• Multi-Region Deployments

• Microservices Platforms

How we Design this approach ?

We will create a folder structure as mentioned below :

terraform/
├── modules/
│   ├── vpc/
│   ├── ec2/
│   ├── rds/
│   └── eks/
│
└── environments/
    ├── dev/
    │   ├── main.tf
    │   ├── variables.tf
    │   └── terraform.tfvars
    │
    ├── test/
    │   ├── main.tf
    │   ├── variables.tf
    │   └── terraform.tfvars
    │
    └── prod/
        ├── main.tf
        ├── variables.tf
        └── terraform.tfvars

How It Works

• Modules Directory: Contains reusable infrastructure components. Each module is written once and reused across environments.

• Environment Directory: Each environment

  • Maintains its own Terraform state

  • Has its own backend configuration

  • Uses its own variable values

  • Can independently deploy infrastructure

Pros

• One of the major advantages of the Modular Folder-Based Approach is that it helps reduce human error compared to Terraform Workspaces. In the workspace approach, users must manually switch between environments, and in real-world scenarios engineers may accidentally apply changes to the wrong workspace assuming they are already in the correct one. With the folder-based approach, each environment has its own dedicated directory such as dev, test, and prod, and users must explicitly navigate to the corresponding folder before running Terraform commands. This clear separation between environments greatly reduces the chances of accidental deployments and improves operational safety, especially for production infrastructure.

• Better Scalability: Works well with Large infrastructure, Multi-team organizations

• Reusable Infrastructure Components: Modules reduce duplication while keeping flexibility.

• Easier Customization: Each environment can use different modules, different architecture and can scale as per the need

• Better Security and Access Control

• Easier CI/CD Integration

Cons:

• More Folder Management

• Slightly More Initial Setup

• Risk of Module Over-Engineering

• Version Management Complexity

Best Practices

• Keep Modules Small and Focused

• Maintain Independent State Per Environment

• Use Versioned Modules

source = "git::https://github.com/org/vpc-module.git?ref=v1.0.0"

When to Use This Approach

Use the Modular Folder-Based Approach when:

✅ Infrastructure differs across environments
✅ Large-scale systems are involved
✅ Strong isolation is required
✅ Multiple teams manage infrastructure
✅ Enterprise-grade CI/CD pipelines exist
✅ Security boundaries are important

Conclusion

The Modular Folder-Based Approach is one of the most scalable and production-friendly ways to organize Terraform infrastructure.

It provides:

• Reusable infrastructure through modules

• Strong environment isolation

• Better scalability

• Easier customization

• Safer production deployments

While it introduces slightly more structure and setup effort initially, it becomes significantly easier to manage as infrastructure and teams grow.

At first, I was just looking for discount Like, is there some hidden setting or plan AWS offers that I’m missing?. But as I dug deeper, I got introduced to a whole new world FinOps. And honestly, this is something many of us (especially in small companies) either:

• don’t know about, or

• know, but haven’t really implemented properly.

Here’s the surprising part, when I checked many organisation are still not aware of this. When I started looking into it seriously, I realized, We could potentially save up to ~15% of our AWS bill just by enabling few things. That’s not small.

So in this blog, I’ll walk through:

• what options AWS gives us to reduce cost

• where most people miss out

• and some practical ways you can actually start saving

Let’s get started 👇

Even though there are many ways to optimize AWS costs, in this blog I’m going to focus on one of those: The Savings Plan

💡 Savings Plans

AWS really wants you to use Savings Plans. At a high level, Savings Plans are AWS’s way of saying:

👉 “Commit to using a certain amount of compute, and I’ll give you a discount.”

Simple idea. But where most people get confused is what exactly are we committing to?

🧠 What are you actually committing? You’re not committing to:

• a specific Instance

• a specific region

• or even a specific service

Instead you are committing to:

👉 A fixed hourly spend (in $/hour) for 1 or 3 years.

Think of it like telling AWS: “Hey, I’m committing to spend around $10/hour for the next year what kind of discount can you give me?”. AWS will offer you discount and as long as your usage matches that commitment you save money.

🔍 Types of Savings Plans

At the time of writing this blog, AWS offers four types of Savings Plans. But honestly, you only need to care about two of them right now because that’s where you’ll see the biggest savings (highlighted below):

• Compute Savings ✅

• EC2 Instance Savings Plan ✅

• Database Savings plan

• SageMaker Savings plan

Compute Savings

This is the most flexible option you have and honestly, this is where most teams start. Why? Because there are very few restrictions. The trade off is simple: you get slightly lower discounts compared to other plans, but a lot more freedom.

You don’t need to worry about

• instance family,

• instance type,

• or even region.

Just commit to a certain hourly spend, and AWS automatically applies the discount across your usage.

Another big advantage is coverage. It applies to a wide range of services like EC2, Fargate, Lambda, and more so you’re not locked into a single service. This makes it a great fit when your workload is unpredictable or when you're using a mix of different services across your architecture. In this options the discounts will be in the range of around 10% to 15% approx at the time of writing this blog.

Example:

Let’s say:

• Your average usage = $10/hour --> This is what you commit to AWS

• AWS Says I will give you 15% discount

So instead of paying \(10/hour, you effectively pay: 👉 \)8.5/hour

Over a month (~730 hours):

• Without Savings Plan → \(10 × 730 = \)7,300

• With Compute Plan → \(8.5 × 730 = \)6,205

💡 You save ~$1,095/month

And the best part? This applies across services like EC2, Fargate, Lambda—no restrictions.

EC2 Instance Savings Plans

As the name suggests, this plan is specifically for EC2 instances. It comes with a few more restrictions compared to Compute Savings Plans but in return, you get better discounts.

When I say “restrictions,” here’s what I mean:

• You’re tied to a specific instance family (e.g., t3, m5)

• You’re tied to a specific region

That said, AWS still gives you some flexibility where it matters. You can freely change:

• Instance size (e.g., t3.small → t3.large)

• Operating system (Linux, Windows, etc.)

• Tenancy (shared, dedicated)

So you’re not completely locked in just scoped within a boundary. In simple terms, you’re telling AWS:

“Hey, I commit to using this EC2 instance family in this region for the next year (or three) for X amount of usage—what discount can you give me?”

Because of this tighter commitment, the discounts here are higher compared to Compute Savings Plans.

Example:

Now let’s take the same scenario:

• You commit to $10/hour

• But this time for a specific EC2 family + region

• AWS gives you 40% discount (Discount rate is an example may vary in real time)

So you effectively pay: 👉 $6/hour

Over a month:

• Without Savings Plan → $7,300

• With EC2 Plan → \(6 × 730 = \)4,380

💡 You save ~$2,920/month

What happens if your usage goes above $10/hour?

So what happens when you use beyond your commitment? AWS keeps track of your usage every single hour. As long as your total eligible usage in that hour is within $10/hour, you’ll get the applicable discount.

But the moment you go beyond that: The extra usage (above $10/hour) is charged at On-Demand pricing(no discount).

👉 Example:

• You committed → $10/hour

• Actual usage → $12/hour

Then:

• First $10/hour → discounted (15%)

• Remaining $2/hour → normal On-Demand pricing

But AWS does give you flexibility in one direction: You can increase your total $/hour commitment anytime by purchasing additional Savings Plans even in the middle of your current plan. You cannot decrease or edit your existing commitment. This is applicable for both the savings plans. This gives us flexiblity to increase our commitment cost if we find out that our average usage is more than what we commit.

Now that we understand what Savings Plans are and the value they bring, the next obvious question is:

👉 How much should I commit?

Committing the right number gives you the maximum advantage.

• If you overcommit → you’ll end up paying for unused capacity

• If you undercommit → you won’t get the full discount benefit ❌

So getting this number right is key.

How to decide the right number?

AWS already gives you a built in helper for this.

👉 It’s called “Recommendations” under Savings Plans.

This tool analyzes your historical usage and suggests an optimal $/hour commitment based on your actual workload.

How to use it

• Login to your AWS account >> Go to "Billing and Cost Management" >> From Left Side panel Expand Saving Plan >> Select Recommendations

• To Run recommendations

  • Choose Compute Savings Plan (recommended for most cases) or Or EC2 Instance Savings Plan

  • Select your preferences:

    • Term → 1 year or 3 years

    • Payment option → No upfront / Partial / All upfront (Upfront will give you more discount but you can also opt for No upfront which falls on your normal billing cycle)

    • Lookback period → 7, 30, or 60 days (This is the sample peroid)

  • AWS will automatically generate a recommendation for you. And it looks something like below.

Note: The menu may vary when you check them this is taken at the time of writing the blog

• 👉 Start with 30 days lookback + 1-year term + No upfront. This tells us that if we commit US\(0.750/hour for a 1-year term you could save an average of US\)0.15/hour that is 19% savings.

Now once you conclude with the plan you can click on Add Savings plan to cart and purchase it. It will get reflected on your next billing cycle

You can also use purchase analyzer to see what is the right amount for your current environment and many other things.

Purchase Analyzer:

The Purchase Analyzer gives you a deeper view before you actually buy a Savings Plan. It helps you understand things like:

• Cost comparison → How your current spend compares with the Savings Plan

• Coverage % → How much of your current usage will be covered by the plan

• Projected savings → What you actually save with this commitment

👉 It helps you validate your decision before committing. To use this tool

• Login to your AWS account >> Go to "Billing and Cost Management" >> From Left Side panel Expand Saving Plan >> Select Purchase Analyzer

• Choose Compute Savings Plan (recommended for most cases) or Or EC2 Instance Savings Plan

• Select your preferences:

  • Term → 1 year or 3 years

  • Payment option → No upfront / Partial / All upfront (Upfront will give you more discount but you can also opt for No upfront which falls on your normal billing cycle)

  • Lookback period → 7, 30, or 60 days (This is the sample peroid)

• Click on Run Analysis

Making the final decision

With these insights, you can make a much more informed decision on choosing the right commitment.

And if you’re still unsure, you can always reach out to the AWS Support team they can help you arrive at a number that fits your usage pattern.

Conclusion

At the end of the day, just by purchasing a Savings Plan and committing to the right $/hour value, you can easily save around 10–15% on your overall bill (or even more, depending on the plan).

The best part?

👉 You don’t need to pay anything upfront
👉 You continue with your normal billing cycle
👉 You’re simply committing: “I’ll use X $/hour for the next 1 year”

This makes it a great starting point for many teams especially since Compute Savings Plans come with minimal restrictions, so you don’t have to worry about how you provision your resources.

What’s next?

If you haven’t tried this yet in your organization, it’s definitely worth exploring you might be leaving easy savings on the table 💰

Since this blog is already getting long, in the next one we’ll dive into Reserved Instances, where you can get even better savings—but with a few more restrictions.

But now comes the real question:

• What about sensitive data?

• Passwords? Tokens? SSH keys?

That’s where Kubernetes Secrets come in. In this blog, we’ll cover:

• What are Secrets and why we need them

• How to create Secrets

• How Pods consume them

• Lifecycle & behavior

• Security best practices (this is where most people mess up 😅)

🤔 What are Kubernetes Secrets?

A Secret is a Kubernetes object used to store sensitive data like: Passwords , API tokens, SSH keys , Certificates.

Instead of hardcoding these inside your app or YAML, you decouple them using Secrets.

⚙️ How are they different from ConfigMaps?

They look similar to ConfigMaps however they have some distinct behaviour

• Stored in etcd (cluster DB)

• Mounted as tmpfs (RAM) → never written to disk 🧠

• Base64 encoded (⚠️ not encryption!)

💡 Important: Base64 ≠ Security. Anyone with access can decode it easily.

😌 Why do we need Secrets?

You already know the answer, but let’s reinforce:

• Keeps sensitive data out of code

• Helps follow 12-factor app principles

• Enables secure configuration per environment

You can think of it like this

• ConfigMap = app config

• Secret = app credentials

🛠️ How to Create Secrets

You can create Secrets in two ways:

• Inline Command

• Using YAML

Let's explore them in detail

⚡ 1. Inline Command (Quick & Dirty)

Example:

# Basic syntax
kubectl create secret <type> <name> --from-literal=<key>=<value>

# Example
kubectl create secret generic api-credentials \
  --from-literal=token=super-secret-123

Multi-value example:

kubectl create secret docker-registry private-reg-auth \
  --docker-username=my-user \
  --docker-password=my-password \
  --docker-server=my-registry.io \
  --docker-email=my-email@example.com

When to Choose this approach ?

Use this for:

• Quick testing

• PoCs

Note: Here you can directly pass the values without encoding, Kubernetes encode the value and store it in Etcd

📄 2. Using YAML (Production Way)

apiVersion: v1
kind: Secret
metadata:
  name: my-db-secret
type: Opaque
data:
  username: YWRtaW4=
  password: MWYyZDNoNGo1aw==

Here you must encode values manually using echo -n "admin" | base64

🧩 Secret Types (Quick Understanding)

💡 Tip: Use specific types when possible — Kubernetes validates structure.

🔌 How Pods Consume Secrets

Two main ways (same as ConfigMaps 👀):

• Environment Variables

• Volume Mounts

🛡️ Option 1: Environment Variables

This option is suitbale for for API keys or database passwords that the application expects to find in its environment. This is simple and clean approach

Example

apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
spec:
  containers:
    - name: my-app
      image: nginx
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-creds
              key: password

⚠️ Cons:

• Values are loaded only at startup

• If Secret changes → Pod restart required 🔄

📁 Option 2: Volume Mounts (Recommended)

This option is best for certificates, SSH keys, or large configuration files. Kubernetes mounts the secret as a directory where each key in the secret becomes a file.

Example:

apiVersion: v1
kind: Pod
metadata:
  name: secret-volume-pod
spec:
    containers:
    - name: ssh-container
      image: nginx 
      volumeMounts:
        - name: ssh-key-vol 
          mountPath: "/etc/ssh-keys" 
          readOnly: true 
volumes: 
    - name: ssh-key-vol 
      secret: 
        secretName: ssh-key-secret # The name of the Secret object

👉 Each key becomes a file:

/etc/ssh-keys/
  ├── private_key
  ├── public_key

💡 Why use Volumes over Environment Variables?

• Security: Secret volumes are stored in tmpfs (RAM). If the node is powered down, the data vanishes from the node's memory. 🧠

• Updates: If you update a Secret, Kubernetes automatically updates the files in the volume (though this takes a few seconds). Environment variables, however, are static—you would have to restart the Pod to see a change. 🔄

• Structure: Volumes are perfect for things that naturally exist as files, like SSL certificates or SSH id_rsa keys.

📁 Standard Mount vs. subPath

🔹 Standard Mount (Default Behavior)

When you mount a Secret as a volume:

• The entire target directory gets replaced

• Any existing files become invisible

• Only the mounted Secret content is available for your pod to use.

👉 If you mount another Secret to the same path, it will overwrite the previous one

This behavior is intentional in Kubernetes to Ensure data isolation, Improve security, Avoid unintended file conflicts

Now consider a scenario where you want to mount multiple Secrets into the same Pod path.

Using the standard approach:

• Mounting Secret A → works fine

• Mounting Secret B to the same path → overwrites Secret A

👉 Result: The first Secret becomes inaccessible.

To solve this, Kubernetes provides a feature called subPath.

💡 What is subPath?

subPath allows you to:

• Mount individual files or subdirectories

• Place them inside an existing directory

• Avoid overriding the entire volume path

To know in detail about subpath check here

✅ Using subPath for Multiple Secrets

With subPath, you can mount multiple Secrets into the same directory without conflicts.

Example:

volumeMounts:
  - name: secret-a
    mountPath: /etc/config/secret-a.txt
    subPath: secret-a.txt
  - name: secret-b
    mountPath: /etc/config/secret-b.txt
    subPath: secret-b.txt
volumes:
  - name: secret-a
    secret:
      secretName: secret-a
  - name: secret-b
    secret:
      secretName: secret-b

🔍 What happens here?

• secret-a is mounted as /etc/config/secret-a.txt

• secret-b is mounted as /etc/config/secret-b.txt

• No overwriting occurs

• Both Secrets coexist in the same directory 🎉

One thing to note here is that when using subPath we have to define path with filename and extension whereas in standardMount the mount path is sufficient enough.

LifeCycle for subPath
One more important thing is that when using subPath when the secret value is updated it will not update the file directly pod restart is required because here the file is a static bind-mount

🔄 Lifecycle of Secrets

In this section we will explore how changing a secret value will be handled in different Consumption option

Even though VolumeMount Auto Updates your secrets to the Mount file you app should be configured to pick the updated value.

Security Best Practices & RBAC

Now that we know how to create and consume Secrets, Let's talk about how to actually keep them "secret."

Even though Secrets are better than plain-text ConfigMaps, they have some vulnerabilities by default:

• Base64 is not encryption: It's just a way to store binary data as text. Anyone who can run kubectl get secret -o yaml can see your passwords. 🔓

• Etcd Storage: By default, Secrets are stored in plain text in the cluster's database (etcd).

Some of the best practices

Encryption at Rest (The Native Way)

You can configure the Kubernetes API server with an EncryptionConfiguration file. When this is enabled, the API server will mathematically encrypt the Secret data before it saves it to etcd. Even if someone steals the etcd hard drive, the data is unreadable garbage without the encryption key.

Strict RBAC (Role-Based Access Control)

As we discussed briefly before, you must restrict who can run kubectl get secrets. If a developer has the get or list permission for Secrets, they can just ask the API server to show them the Base64 data, bypassing all etcd encryption. You should only grant Secret access to the specific ServiceAccounts that the Pods use, not to human users.

To truly secure a Secret while still letting it be used:

• For Users (Developers): You generally want to deny get, list, and watch. If they can't get it, they can't see the password.

• For Pods: The Pod doesn't actually need the user to have permissions. The Pod uses a ServiceAccount. The Kubernetes controller-manager handles the mounting process, so as long as the Pod's YAML is allowed to be created, the system handles the rest.

External Secret Stores (The Enterprise Way)

In large companies, they don't even store the source-of-truth secrets in Kubernetes at all! They use tools like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault. They use plugins (like the Secrets Store CSI Driver or External Secrets Operator) to fetch the password from the cloud vault and inject it directly into the Pod at runtime.

Conclusion

Kubernetes Secrets might look simple at first, but how you use them makes all the difference. They help you keep sensitive data out of your code and configs—but they are not secure by default.

If you use them the right way prefer volumes over env vars, restrict access with RBAC, and integrate with external secret managers—you move from just “working setup” to a production-ready secure system 🔐

Thanks for reading. Keep learning, keep building, and keep breaking things (that’s how we grow) . See you in the next one ✌️

But let’s pause and think for a moment 🤔

• What if we just want to pass configuration values to a Pod? For example:

  • Database URL

  • Application mode (dev / prod)

  • Feature flags

  • External service endpoints

We don’t want to hardcode these values inside the container image. Because configuration changes frequently, but images should remain stable. So how do we handle this in Kubernetes? 👉 This is exactly where ConfigMaps come into the picture.

In this blog we will explore

• What is ConfigMaps

• How and Where to use it

• Creation Methods: Understanding the various ways to build ConfigMaps using kubectl literals, files, or YAML manifests.

• Consumption Strategies: Discovering how Pods actually "read" this data, whether through environment variables or by mounting them as files.

• Lifecycle & Updates: Exploring what happens when configuration changes and how to handle updates without breaking your services.

What is a ConfigMap? 📦

A ConfigMap is a Kubernetes Object is used to store non-sensitive configuration data as key-value pairs. It allows you to:

• Decouple configuration from container images. This means you can use the exact same image in development, staging, and production, simply by swapping out the ConfigMap associated with it.

• Pass environment variables to pods

• Provide configuration files to containers

• Update configs without rebuilding images

In simple words:

ConfigMap = A way to inject configuration into your Pod dynamically.

How ConfigMap Works 🔍

A ConfigMap stores configuration as:

• Key-value pairs

• Entire configuration files (like an nginx.conf or settings.py)

• Environment Variable

Then Kubernetes allows you to: Inject them as environment variables or Mount them as files inside a container.

Remember that Configmaps are not intended for sensitive information like passwords or API keys—for those, we use Secrets

Complete Internal Flow Summary:

• kubectl sends ConfigMap definition to API Server

• API Server validates and stores ConfigMap in etcd

• ConfigMap now exists as a cluster object

• Pod referencing ConfigMap is scheduled to a node

• kubelet detects Pod and fetches ConfigMap

• kubelet injects ConfigMap into container (env or volume)

• Container starts

• Application consumes configuration

ConfigMap - Creation Methods

Method 1: Literal Key-Value Pairs 🔑

The first method is using kubectl command and creating a config map using literals directly from terminal

# Syntax
kubectl create configmap <map-name> --from-literal=<key>=<value> 

# Example
kubectl create configmap app-settings --from-literal=ui_color=blue --from-literal=debug_mode=true

When to use this method:

You should use Literal Key-Value Pairs when quick testing/debugging is needed. If you want to:

• Quickly toggle a feature flag

• Change a log level

• Test a temporary configuration

• Small, Simple Configuration

While literals are great for quick flags, real-world applications often have dozens of settings stored in configuration files (like .properties, .env, or .yaml). Kubernetes allows you to take an entire file and shove it into a ConfigMap in one go.

Method 2: From a File 📄

In this method instead of passing individual Key Value pairs we can pass the whole file as an input. This is very useful when we have a need to pass a config file to the Container image.

When you use the --from-file flag, Kubernetes uses the filename as the key and the file contents as the value.

Imagine you have a file named connection.conf with this content:

# Filename: connection.conf

db_host=prod-db.example.com
db_port=5432

To configure this file in Config map we will use below command

kubectl create configmap db-config --from-file=connection.conf

Inside the ConfigMap, it looks like this:

• Key: connection.conf

• Value: (The entire multiline string of the file)

When to use this Method:

This is similar to the above literals the only difference is that here we can pass the whole file. This method is also useful for short term testing or for temporary change

Method 3: Declarative YAML 📝

This is the widely used industry standard method for declaring ConfigMaps. All the ConfigMaps are written in a YAML file in declarative way and then we apply it. This gives us an option to maintain desired state and also help us to follow GitOps. This allows us to track changes in Git (Infrastructure as Code).

A ConfigMap in YAML looks like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: test-config
data:
  # Simple key-value pair
  log_level: "INFO"
  DB_TYPE: "postgres"
  # Multiline configuration file
  nginx.conf: |
    server {
      listen 80;
      server_name localhost;
    }

To apply this to our K8 cluster we use below default apply command

kubectl apply -f <filename>

When to use this Method:

This is the most commonly used best practice method. Apart from short term testing it is recommended to use Declarative YAML method while creationg configmaps.

We understood the methods of creating config maps now lets explore how we can consume it.

ConfigMap - Consumption Strategies

Whenever you create a configmap using any of the above method the configmaps are stored in the cluster now your application needs to read it. There are two methods to consume the configmaps

• Environment Variables: Good for individual settings (like DB_TYPE).

• Volume Mounts: Good for configuration files (like nginx.conf).

Method 1: Environment Variables:

In this method we will fecth the ConfigMap and use it as an Environment variable in the pod.

Let's take the below Pod Definition example

apiVersion: v1
kind: Pod
metadata:
  name: sample-app-pod
spec:
  containers:
    - name: sample-container
      image: nginx:latest
      env:
        - name: DATABASE_KIND   # Environment variable inside container
          valueFrom:
            configMapKeyRef:
              name: test-config   # ConfigMap name
              key: DB_TYPE         # Key inside ConfigMap

When to use this Method:

When your config maps is just are just Key Value pair and your applications can consume it via environment variable you can use this method.

Method 2: Volume Mounts:

In this method we will mount the ConfigMaps as a volume. So the keys in the configMap will be file name in the Volume and value will be content inside the file.

Example:

# weather-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: weather-config
data:
  city: "Chennai"
  temperature-unit: "Celsius"
  app.properties: |
    forecast.days=5
    refresh.interval=30

# weather-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: weather-app-pod
spec:
  containers:
    - name: my-container
      image: nginx:latest
      volumeMounts:
        - name: config-dir              # Must match volume name below
          mountPath: /etc/config        # Files appear here inside container
          readOnly: true
  volumes:
    - name: config-dir
      configMap:
        name: weather-config            # ConfigMap name

How the config maps are stored in this method ?

Kubernetes will read the weather-config ConfigMap, convert each key into a file, mount those files inside /etc/config

Inside your container you will be seing the Config map as file like this : /etc/config/city , /etc/config/temperature-unit

Everything is a "File" in a Volume:

[CREATION]: Whether you create a ConfigMap from a literal (--from-literal) or a file (--from-file) or using manifest files, Kubernetes stores them the exact same way key-value pairs

[CONSUMPTION]: But, when you mount a ConfigMap as a volume, Kubernetes treats every key as a filename and every value as the content of that file. It doesn't care if the original source was a literal string or a 500-line config file.

Remember the difference between creating a config maps(which we discussed in previous section) and consuming the configmaps(which we are discussing now). This above point is when you consume configMaps as volume

There is a small catch here 👀. Whenever you mount a ConfigMap as a volume, Kubernetes mounts it on the specified path inside the container.

👉 But here is the important part:

When a volume is mounted to a path, any existing content in that path will be hidden. Only the data from the latest mounted volume will be visible.

🔎 Let’s Understand With Example

Assume that you have two configMaps named xyz-config and weather-config . If you want to mount both in the same path mountPath: /etc/config, then

• The second mount weather-config will override the first one

• Only the latest mounted config map will be visible

• Other files in that path become unavailable to the pod

🤔 Why Is It Like This?

This is Kubernetes’ default behavior. When you mount a volume to a path,

• Kubernetes overlays that volume on that directory. The original content is hidden. This ensures clean and predictable file mapping. Also improves isolation and avoids accidental file mixing (security + consistency reasons).

So what is the solution if I want to mount all configMaps to the specific Path Or map my ConfigMap to a specific path and also use other contents from the path. This is where a new parameter comes in called subPath

subPath

If you want to mount multiple ConfigMaps to the same directory without hiding each other, you can use subPath

👉 what subPath tells Kubernetes:

Mount this specific file inside the directory, not override the entire directory.

Example - without subPath

volumeMounts:
  - name: xyz-volume
    mountPath: /etc/config   
  - name: weather-volume
    mountPath: /etc/config # This will override the previous one ❌
  

Example - with subPath

apiVersion: v1
kind: Pod
metadata:
  name: multi-config-pod
spec:
  containers:
    - name: my-container
      image: nginx:latest
      volumeMounts:
        - name: weather-volume
          mountPath: /etc/config/weather.properties
          subPath: weather.properties
          readOnly: true

        - name: xyz-volume
          mountPath: /etc/config/xyz.properties
          subPath: xyz.properties
          readOnly: true

  volumes:
    - name: weather-volume
      configMap:
        name: weather-config

    - name: xyz-volume
      configMap:
        name: xyz-config

🧠 Important Note

• When using normal mount you only specify directory path. mountPath: /etc/config . This is because by default, when you mount a volume to a path, Kubernetes creates a directory at that path. If you mount it exactly at /app/static/theme.css, that path will become a directory, and your file would actually live at /app/static/theme.css/theme.css. This is why when you are not using subpath you have to define only the path without filename

• But when using subPath, You must specify the complete file path including filename. mountPath: /etc/config/weather.properties. This is because now you are mounting a specific file, not the whole directory.

ConfigMap - Lifecycle & Updates 🔄

In this section, let’s understand something very practical:

👉 What happens when you update a ConfigMap?
👉 How does Kubernetes handle those updates?
👉 Does your application automatically see the changes?

The behavior actually depends on how you consume the ConfigMap. There are two main consumption strategies which we discussed, as Environment Variables and as Volume Mounts. Let's see what happens on both clearly

🧪 Lifecycle of ConfigMap as Environment Variable

When you consume a ConfigMap as an environment variable, and later you update the ConfigMap, The running Pods will NOT see the changes.

🤔 Why is this? - Because, ConfigMap values are injected into the container only at startup. Once the container starts Environment variables are set. They become part of the container’s runtime. Kubernetes does NOT re-inject updated values.

So even you edit your config map using kubectl edit configmap my-config the values of ConfigMaps will be updated but your pods will not use the updated values.

✅ So What Should You Do? You must restart the Pod. There are multiple ways to do it

• Delete the Pod (ReplicaSet will recreate it)

• Trigger a Deployment rollout

Once the container restarts, the updated ConfigMap values will be injected.

📂 Lifecycle of ConfigMap as Volume Mount

Now this is interesting 👀

When you consume a ConfigMap as a Volume Mount, updates behave differently. When you update the ConfigMap, Kubernetes automatically updates the files inside the mounted volume. Yes — without restarting the Pod.

🤔 But there is a Catch? - The catch is in your application design.

Let's take a example, Your app reads /etc/config/app.properties , It reads it only once during startup. Now you update the ConfigMap. The file inside the container gets updated. But your app still uses old values. Because your application is not re-reading the file.

🧠 What Should Be Done? - If you consume ConfigMap via Volume, Your application should Watch the file for changes or periodically reload configuration or support dynamic configuration refresh Otherwise, the update exists at the file level — but not at the application level.

🛡️ Immutable ConfigMaps

Now let’s talk about something advanced and powerful.

Imagine, You have 300 Pods all are consuming the same ConfigMap via Volume Mount. They are watching for changes.

Whenever you update that ConfigMap, Kubernetes API server must notify all watchers, the control plane processes multiple update events, this increases API server load. It may cause unexpected behavior if files update mid-process

Now think practically, What if this configuration NEVER changes or changes very rarely ?

In such cases, Kubernetes gives you something called Immutable ConfigMaps

You can mark a ConfigMap as immutable. This tells Kubernetes that the values in the configMaps will not change. So that Kubernetes will not watch these configMaps.

Example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: static-config
immutable: true # This is the magic
data:
  api_key: "12345"

✅ Benefits of Immutability

• Safety: Prevents accidental configuration changes that could break production.

• Performance: Kubernetes stops "watching" these files, reducing the load on the API server.

❓ But What If I Need to Change an Immutable ConfigMap?

Good question 😉. Once a ConfigMap is marked as immutable, you CANNOT Modify the data, Remove the immutable flag, Edit it directly, Kubernetes strictly prevents this.

You could delete and recreate it, but that is risky. If a Pod restarts during the split second where the ConfigMap doesn’t exist Pod will fail to start.

The Expert Way: Versioning (Rolling Forward Pattern):

Instead of editing the old ConfigMap, create a new one

• Create New: You create a new ConfigMap named static-config-v2 with your updated database URL.

• Update Deployment: You edit your Deployment manifest to point to this new name.

• Rollout: Kubernetes sees the Deployment change and automatically performs a rolling update, creating new Pods with the new config and terminating the old ones safely.

At this point, you might have one more doubt. If I’m using versioned ConfigMaps and I have 100 applications, do I need to manually update all 100 Deployment files every time the config changes?

Yes… technically you could. But that would be Painful 😅, Error Prone, Not Scalable. This is where tools like Kustomize come into play.

⚙️ How Kustomize Solves This Problem

Kustomize helps you manage Kubernetes configurations in a smarter way. When you use Kustomize to generate ConfigMaps:

• It automatically hashes the content.

• It appends that hash to the ConfigMap name.

Example:

• Instead of static-config it creates hashed value something like this static-config-m9t8f5c92k

We will explore Kustomize deeply in another blog (because that itself deserves a full discussion 😄).

📋 Putting It All Together: The Expert Checklist

Now we have explored ConfigMaps in detail:

• What they are

• How they are consumed

• Update behavior

• Immutable pattern

Let’s finish with a simple, practical checklist. Because in production, the real question is: When should I use what?

Common Doubts

If My Config Never Changes, Why Use ConfigMap At All why do we need Immutable?

At first glance, this seems logical as I also had the same doubt. If it never changes, why not just hardcode it inside the application?

But here’s why ConfigMaps still matter.

Separation:

Even if configuration never changes, we still separate Application Code and Configuration, why because Same image can run in Dev, QA, Prod. Only configuration changes between environments. ou don’t rebuild image for every small config variation

So Even if production config rarely changes, it is still environment-specific. That separation is the core design principle.

Image Immutability Principle:

In Kubernetes world, Container images should be immutable. If we bake config inside the image any changes to the config maps will force us to rebuild the image which increment the image version for small change.

practical list of common ConfigMap-related errors

ConfigMap Not Found Errors:

Error: configmap "<Configmap name>" not found

Why it happens: Typo in name, Wrong namespace, ConfigMap not created before Pod, Deleted accidentally

Fix: Verify namespace: kubectl get configmap -n <ns> , Ensure order: ConfigMap must exist before Pod starts

Wrong Namespace Issue:

Why it happens: Very common mistake. ConfigMap is created in default namespace as we forget to mention the namespace

Fix: Create ConfigMap in the same namespace as Pod.

Key Not Found Error

Error: couldn't find key DB_HOST in ConfigMap

Why it happens: Wrong key name, Case mismatch (db_host vs DB_HOST), Key deleted during update

Fix: Make sure you are using same KEY across the namespace. Validate it with Configmap when you are consuming it

Pod Stuck in CreateContainerConfigError

Error: CreateContainerConfigError

Why it happens: Missing ConfigMap, Missing key, Invalid reference. This usually means Kubernetes cannot inject the ConfigMap.

Fix: Validate the configmap in correct namespace and the names are matched across the deployments. Validate Configmap is created or not

Environment Variable Not Updating

Error: You update ConfigMap. But app still shows old value.

Why it happens: Env variables are injected only at container startup.

Fix: Restart Pod - kubectl rollout restart deployment app

Volume Mount Not Updating Immediately

Error: You update ConfigMap. File inside container still shows old content for some time.

Why it happens: Update propagation delay, Kubelet sync period (usually ~1 min)

Fix: This is normal behavior. No fix needed

Application Not Picking Updated File

Error: File updated but app still behaves same.

Why it happens: Application reads config only at startup. Kubernetes updated file but app never reloaded it.

Fix: This is an application design issue, not Kubernetes issue. Need to design app to check config maps periodically or take changes whenever happens

subPath Does NOT Auto-Refresh

Error: If we use subPath then configMap does not auto refresh

Why it happens: Because subPath mounts a single file copy not the live symlink.

Accidentally Overriding Directory

Error: When you mount configmap as Volume it hides other files in the volume

Why it happens: This is the default behaviour of Kubernetes to have clean mount and saftey.

Fix: If you want to mount the configMap where you also needs to use other files from volume use subPath to mount configmap.

Large ConfigMap Size Limit

Error: etcd request too large

Why it happens: ConfigMap size limit: ~1MB.

Fix: Makesure you manage size of configmap in right way. ConfigMaps are not meant for Huge Binaries or Large Json datasets

Watch Load on API Server

Error: API server getting overloaded

Why it happens: Hundreds of Pods watching a frequently updated ConfigMap: High API server load, Performance degradation

Fix: Use immutable configmap for static configs

🧠 Production-Level Issues (Advanced)

These are not syntax errors but architectural mistakes:

• Using ConfigMap for dynamic runtime feature toggles (bad design)

• Updating ConfigMap frequently in high-scale clusters

• Not versioning production config

• Editing ConfigMap directly in production (breaks GitOps)

• Using subPath when expecting dynamic updates

Conclusion

ConfigMaps are a simple, powerful Kubernetes primitive for decoupling non-sensitive configuration from container images. They let you store key-value configuration outside of your images and supply those values to Pods either as environment variables or as mounted files — giving you flexibility to use the same image across environments and change configuration independently.

Key takeaways:

• Use ConfigMaps for non-sensitive, environment-specific values (use Secrets for sensitive data).

• Create ConfigMaps via kubectl literals, files, or declarative YAML manifests depending on your workflow and need for version control.

• Consume ConfigMaps as environment variables for small, simple values or mount them as files when applications expect file-based configs.

• Be mindful of update behavior: mounted files are updated automatically (with caveats), while env-based values require Pod restart; manage updates via rollout restarts, immutable ConfigMaps, or reloader sidecars/controllers when appropriate.

• Follow best practices: keep configs declarative (GitOps), name and label ConfigMaps clearly, avoid embedding large binaries, and consider using immutable ConfigMaps for stable deployments.

If you’ve worked with Kubernetes volumes, you’ve probably seen this statement:

“EBS (or any block storage) only supports RWO (ReadWriteOnce).
For RWX (ReadWriteMany), you need something like EFS.”

If you’ve ever wondered why that is, you’re not alone. I had the same question when I first learned it. Let’s break it down properly.

Before the Crux — Let’s Get the Basics Right

What Is Block Storage?

Block storage is essentially a raw disk. When you create an EBS volume, Kubernetes (or the operating system) sees it as:

“Here’s an empty hard drive. You decide how to use it.”

Since it’s raw, it must be formatted with a filesystem (like ext4 or xfs) before use.

Formatting what is it ?

Formatting means creating a filesystem structure on the disk. Think of it like this:

• You have a blank notebook (raw disk)

• You draw structured grids inside it. Think of it like your Microsoft Excel

• Now you can store and retrieve values in specific locations by mapping the Row and Column.

In the same way the disk gets divided into blocks, and the filesystem keeps track of:

• Which block belongs to which file

• Where data is stored

• How to retrieve it quickly

Do you know the Block Storage have Fast I/O. You know why ?'

Because block storage allows direct access to data blocks. When an application wants data, the operating system can say: “Give me block #504.” it retrieve it directly from the Block storage Disk nothing between the storage and the Operating System.

That’s why block storage is:

• Low latency

• High performance

• Ideal for databases

Then Why Can’t Block Storage Support RWX?

A traditional filesystem (like ext4 or xfs) is not cluster-aware. If you attach the same block disk to multiple nodes and both try to write:

• Filesystem corruption can occur

• There’s no built-in distributed locking (So that when one node is using it it locks other node)

• Metadata can conflict (If some node tries to write and other tries to modify the metadata conflict occurs.)

  • Metadata is nothing but the information about the data stored such as when it is created what is the size etc.

Block storage assumes: “One disk → one machine managing the filesystem.”

That’s why:

• EBS supports ReadWriteOnce (RWO)

• It cannot safely support ReadWriteMany (RWX)

What Is File Storage?

File storage (like EFS) is different. It is

• Already formatted

• Managed by a file server

• Exposed over the network (NFS in EFS case)

Instead of giving you raw blocks, it gives you A shared file system accessible over the network.

You may ask How Is File Storage Different ?

Even file storage uses blocks internally but the key difference is:

• A central file server manages metadata

• It handles locking

• It coordinates concurrent access

• It ensures consistency

Every file contains metadata like:

• Permissions

• Ownership

• Size

• Timestamps

• Access rules

What happens when multiple nodes try to connect ?

• They don’t manage blocks directly

• They talk to the file server

• The file server manages read/write coordination

That’s what enables RWX.

Is File Storage Slower?

Generally, yes — compared to block storage. Why?

Because:

• It involves network communication

• Metadata validation happens

• There’s coordination overhead

But the trade-off is: You get safe, shared, multi-node access.

The Real Crux

Block storage:

• Attached to one node

• Filesystem managed locally

• No distributed locking

• High performance

• Supports RWO

File storage:

• Managed by centralized file servers

• Supports distributed locking

• Safe concurrent access

• Supports RWX

• Slightly higher latency

The Mental Model

Think of it like this:

Block Storage (EBS)

Like a USB drive plugged into one laptop. Only that laptop controls it.

File Storage (EFS)

Like a shared Google Drive folder. Multiple machines can access it at the same time.

That "node-lock" is the biggest weakness of hostPath. If the node goes down or the Pod moves, the data is essentially "trapped" on the old node.

To solve this, Kubernetes uses a system that decouples the storage from the node entirely, often using network storage (like a cloud disk). This brings us to the "Gold Standard" of Kubernetes storage: PersistentVolumes (PV) and PersistentVolumeClaims (PVC).

Persistent Volumes(PV) and Persistent Volumes Claim (PVC)

The What ?

Persistent Volumes (PV):

A Persistent Volume (PV) represents a piece of storage provisioned in the cluster often backed by network or cloud storage that continues to exist even if Pods are deleted or moved.

You can think of a PV as block storage or file storage attached to the Kubernetes cluster, where the cluster has full control to allocate this storage to incoming requests.

Since this storage lives outside the Kubernetes cluster environment (for example, cloud disks or network-attached storage), it is not affected by Pod restarts, node failures, or most Kubernetes-level failures.

Persistent Volumes Claim (PVC):

A Persistent Volume Claim (PVC) is a request for storage made by an application running inside the Kubernetes cluster.

Instead of directly attaching a storage disk, a Pod asks for storage by specifying:

• How much space it needs

• The type of access required (read/write)

• The storage characteristics it expects

Kubernetes then looks for a suitable Persistent Volume (PV) that matches this request and binds it to the PVC.

You can think of a PVC as an interface between your application and the actual storage, allowing Pods to use persistent storage without needing to know where or how it is physically provisioned.

❓ Why PVC request is needed ?

PVCs introduce a clean separation between applications and storage.

Different applications have very different storage needs:

• Databases need high I/O and low latency

• Logs or backups can work with lower-performance storage

Instead of hard-coding storage details into applications, a PVC allows the app to simply say: “I need this much storage, with this access pattern.” Kubernetes then fulfills that request by binding the PVC to an appropriate PV.

This approach keeps applications portable, flexible, and storage-agnostic, while allowing storage performance to be controlled through PVC requests.

Here is a quick look at how they connect:

• Pod points to → PVC

• PVC binds to → PV

• PV points to → Actual Disk

Each layer has a clear responsibility, and none of them need to know the internal details of the others.

🔐 Access Modes

Both Persistent Volumes (PV) and Persistent Volume Claims (PVC) define something called access modes. Access modes describe how a volume can be accessed by Pods for example, whether it can be mounted as read-only, read-write, or shared across multiple nodes. The available access modes are explained below

• ReadWriteOnce (RWO): The volume can be mounted as read-write by a single node. (Like a USB drive plugged into one laptop).

• ReadOnlyMany (ROX): The volume can be mounted read-only by many nodes. (Like a shared CD-ROM).

• ReadWriteMany (RWX): The volume can be mounted as read-write by many nodes. (Like a shared network folder/Dropbox).

When you provision PV and PVC for a highly available application that runs across multiple nodes, choosing the right access mode becomes critical. For example, If the volume is read-only and shared across Pods across nodes, you can use ReadOnlyMany (ROX), If the volume needs to be read and written by multiple Pods across nodes, you must use ReadWriteMany (RWX)

What does “Many” mean here?

“Many” refers to the number of nodes, not Pods.

• RWO → Only one node can read/write to the volume at a time. i.e all the pods in the node can access this volume

• RWX → Multiple nodes can read/write to the same volume simultaneously. i.e pods across nodes can access the volume

An important detail to note here is a Persistent Volume (PV) can be configured with multiple access modes, indicating what it supports. A Persistent Volume Claim (PVC), on the other hand, requests a specific access mode based on the application’s requirement. If the requested access mode in the PVC is supported by the PV, Kubernetes approves the claim and binds them together.

🧠 Key Takeaway

PV advertises what it can support, and PVC asks for what it needs. Binding happens only when both agree on the access mode.

Example Kubernetes Resource Definition File

Persistent Volume (PV)

apiVersion: v1
kind: PersistentVolume
metadata:
  name: app-pv               # Name of the PV
spec:
  capacity:
    storage: 10Gi            # Size of the volume
  accessModes:
    - ReadWriteOnce          # Supported access mode
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  # Retain data even if PVC is deleted
  storageClassName: ebs-sc   # Must match PVC
  csi:
    driver: ebs.csi.aws.com  # AWS EBS CSI driver
    volumeHandle: vol-0abcd1234efgh5678
    # Actual EBS volume ID from AWS
  volumeMode: Filesystem    # Mount as a filesystem

Persistent Volume (PV)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-pvc              # Name of the PVC
spec:
  accessModes:
    - ReadWriteOnce          # Only one node can read/write
  resources:
    requests:
      storage: 10Gi          # Amount of storage requested
  storageClassName: ebs-sc   # Use this StorageClass

Consuming a PVC Inside a Pod

apiVersion: v1
kind: Pod
metadata:
  name: app-pod                 # Name of the Pod
spec:
  containers:
    - name: app-container
      image: nginx              # Sample container image
      volumeMounts:
        - name: app-storage
          mountPath: /data      # Path inside the container
          # The PVC will be mounted here
  volumes:
    - name: app-storage
      persistentVolumeClaim:
        claimName: app-pvc      # Name of the PVC to use

StorageClasses: Leveling Up to Dynamic Provisioning

In the real world the admin don’t want to worry about creating 100’s of PV’s(Persistent Volumes) with different types and sizes. Here is where StorageClasses (SC) comes in

How it works

• The administrator creates a StorageClass in the cluster

• The StorageClass is granted permission to create cloud provider disks (such as AWS EBS or Azure Disk) by installing the appropriate CSI driver in the cluster.

  • For example, in an Amazon EKS cluster, you install the EBS CSI driver, which enables Kubernetes to dynamically provision EBS volumes on demand whenever a PVC is created.

• The PVC specifies which StorageClass to use

• When the PVC is created, Kubernetes checks:

  • Is there an existing PV that matches this request?

• If no matching PV exists, Kubernetes uses the StorageClass to dynamically create a new PV.

• The newly created PV is bound to the PVC.

• When a Pod references the PVC, the volume is attached and mounted into the Pod.

For example:

• A Pod requests 50Gi

• No existing PV matches

• The StorageClass dynamically creates a new disk (EBS etc.)

• The disk is attached to the Pod as a volume

No manual PV creation needed 🎉

Reclaim Policies

Now comes an important question. Your application stores data in a PV via a PVC and that data can live forever.

But what if:

• You delete the application?

• You no longer need the data?

• You recreate the app with fresh data?

Continuously creating new volumes is not sustainable, especially when storage costs add up. This is where Reclaim Policies come into play.

Reclaim policies define what happens to a Persistent Volume after the PVC is deleted whether the data should be:

• Retained: The PV is not deleted. It stays in "Released” state and will not be assigned to any new PVC request. An admin must manually clean up the data. This is the safest for production databases.

• Deleted: The PV and the actual physical disk (AWS EBS, GCE PD) are deleted immediately. Should be choosen with extra caution as the data will be wiped once PVC is deleted

• Recycle(Deprecated): It performs a basic rm -rf /thevolume/* and makes the PV available again. (Rarely used now).

You control this with the persistentVolumeReclaimPolicy

The Binding Process

Kubernetes matches a PVC to a PV using these criteria:

1. Capacity: Does the PV have at least the amount requested?

2. Access Mode: Does the PV support the mode (RWO, RWX, etc.)?

3. StorageClass: Do they have the same StorageClass name?

4. Selectors: (Optional) Do the labels match?

Example Kubernetes Resource Definition File

StorageClass (Dynamic Provisioning)

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc               # Name of the StorageClass
provisioner: ebs.csi.aws.com # CSI driver that talks to AWS EBS
parameters:
  type: gp3                  # EBS volume type (gp2 / gp3 / io1, etc.)
reclaimPolicy: Delete        # What happens to the disk when PVC is deleted
volumeBindingMode: WaitForFirstConsumer
# Volume is created only when a Pod actually uses the PVC

Common Scenarios

The “Same Name” Scenario (ReclaimPolicy: Retain)

Assume you have an application using a Persistent Volume (PV) named pv-x, and the reclaim policy is set to Retain. You deploy the app, it creates a PVC, and everything works fine. After some experimentation, you delete the application (which deletes the PVC).

Later, you:

• Create a new application with a different PVC name, but want to use the same data

• Or simply rename the PVC for the same application

So… what happens?

When the PVC is deleted and the reclaim policy is Retain:

• The PV is NOT deleted. The underlying disk is preserved

• The PV moves into a Released state. The PV still remembers the old PVC it was bound to

This is the important part 👇

A PV in the Released state cannot be automatically reused by a new PVC—even if:

• The storage size matches

• The access modes match

• The StorageClass matches

Kubernetes intentionally blocks this to prevent accidental data leaks between workloads.

❓ Why Can’t a New PVC Reuse the Same PV Automatically?

Because the PV still contains Old application data and Metadata pointing to the previous PVC. Kubernetes assumes this data belongs to someone else. I won’t reattach it unless a human explicitly says so. That’s a safety feature, not a limitation.

So How Can You Recover the Data?

There are two correct and safe ways to recover data from a retained PV.

✅ Option 1: Manually Rebind the Existing PV (Most Common)

If you want the same data to be used by a new or renamed PVC:

• Manually edit the PV and remove the old claimRef

• Set the PV back to Available

  • Create a new PVC that matches: StorageSize, AccessMode, and Storage class

• Once this is done, Kubernetes will bind the new PVC to the old PV—and your data is recovered intact.

This is the intended recovery flow for Retain.

✅ Option 2: Create a New PV Pointing to the Same Disk

This is useful when you don’t want to touch the old PV object and you want more control over the new binding.

• Identify the underlying disk (EBS volume, Azure Disk, etc.)

• Create a new PV definition

• Point it to the same physical disk

• Create a new PVC that binds to this new PV

The data remains untouched because the disk never changed—only the Kubernetes objects did.

Do you know AWS EBS Does NOT Support RWX

AWS Elastic Block Storage (EBS) does not support ReadWriteMany (RWX) access mode. why ?

Because an EBS volume behaves like a physical hard drive. Once you attach a hard drive to one machine, you cannot plug that same drive into multiple machines at the same time and expect it to work safely. This is exactly how EBS works:

• One EBS volume → one node

• Read/write access → single node only

That’s why EBS supports ReadWriteOnce (RWO).This limitation is not a Kubernetes issue — it’s a block storage limitation.

So What If You Need RWX Across Multiple Nodes?

If your application needs to run on multiple nodes and read and write the same data simultaneously, then block storage is the wrong tool. This is where file-based network storage comes in.

EFS: The RWX-Friendly Storage

Amazon Elastic File System (EFS) is designed to solve exactly this problem. EFS behaves like a shared network drive. Multiple nodes can mount it and all nodes see the same file and Read/Write operations happens concurrently

This wraps up our deep dive into PV, PVC, and StorageClasses — what they are, why they exist, and how Pods actually consume them.

From here, you can start mapping these concepts to real-world scenarios and production use cases, not just lab setups.

If you ever wondered “Where my data go when a pod restarts“ you are in a right place. Lets dive in 🚀

🤔 Why Do We Even Need Volumes?

Let’s start with a simple question: why does Kubernetes need volumes at all?

We know that a Pod is made up of one or more containers. And we also know an important (and slightly scary) fact about containers: Containers are ephemeral. This means when a container crashes, restarts, or gets recreated, its filesystem is wiped clean. Any files written inside the container? 💥 Gone.

This is where Volumes come to the rescue. A Kubernetes Volume provides a way to decouple storage from the container’s lifecycle. So to answer the question :

🎯 Why We Use Volumes

• Protect application data from container crashes

• Share data between containers in a Pod

• Make applications more reliable and production-ready

Kubernetes offers multiple volume types, each designed for different use cases—temporary storage, shared storage, cloud disks, network storage, and more.

In the next sections, we’ll explore these volume options one by one, understand when to use what, and avoid common mistakes along the way.

📦 emptyDir — The Simplest Kubernetes Volume

The most basic volume type in Kubernetes is emptyDir. An emptyDir volume is created at the Pod level, not at the container level.

The What ?

👉 But what does Pod level actually mean?

When a Pod is created: Kubernetes creates the emptyDir volume once. Every container inside that Pod can mount and access the same volume.

So, the volume does not belong to a single container, it belongs to the pod itself. As long as the pod exists the volume exists

🤝 How Do Containers Share an emptyDir?

Let’s make this real with a common and very practical scenario. Imagine a Pod with two containers:

• App Container: Runs your main application and generate logs

• Sidecar Container: Collect those logs and ship them to a central logging system. You don’t want to add extra load or logging logic inside your main app, so you offload that responsibility to a sidecar container.

Both containers mount the same emptyDir volume as a result the app writes the logs to the volume and sidecar reads logs from the same volume. They’re isolated containers but sharing data seamlessly.

The important catch is data in the emptyDir volume will be lost when the pod restarts or crashed. But the data will be preserved even if any of the container in the pod crashes or restarted. Because emptyDir lives only as long as the Pod lives.

So in simple terms emptyDir:

• Container lifecycle ❌ does NOT affect data

• Pod lifecycle ✅ DOES affect data

The When ?

emptyDir is ideal for:

• Temporary files

• Cache data

• Shared workspace between containers

• Log sharing (like our sidecar example)

🚫 It is not meant for long-term or critical data storage.

The How ?

Here’s a minimal working example of exactly the scenario we discussed.

apiVersion: v1
kind: Pod
metadata:
  name: shared-workspace
spec:
  containers:
    - name: producer
      image: busybox
      # Writes data to the volume
      volumeMounts:          # <--- VolumeMount block (Producer)
        - name: shared-data
          mountPath: /app/data
    - name: consumer
      image: busybox
      # Reads data from the volume
      volumeMounts:          # <--- VolumeMount block (Consumer)
        - name: shared-data
          mountPath: /app/input
  volumes:                  # <--- Volume definition
    - name: shared-data
      emptyDir: {}           # <--- The magic keyword

🔍 Let’s Dissect the Example Step by Step

• Let’s begin at the bottom of the Pod spec—the volumes block. This is where we define

  • The volume name - shared-data

  • The volume type - emptyDir

• This tells Kubernetes: Create an empty directory when the Pod starts and attach it to this Pod. At this point, the volume exists—but no container is using it yet.

• Now look at the two containers inside the Pod: Each container has its own volumeMounts block. Both containers mount the same volume (shared-data), but at different paths. This is intentional—and this is where things get interesting.

🤔 The Common Doubt (Very Natural One!)

You might be thinking: If the producer writes logs to /app/data, how does the consumer read them from /app/input?

Think of the volume as a room. And think of mountPath as a door. A room can have multiple doors, no matter which door you enter, you end up in the same room

With this analogy in our example /app/data is one door and /app/input is another door, both doors lead to the same emptyDir volume, so The producer writes logs into the room through the /app/data door and the consumer reads those same logs from the room through the /app/input door

With emptyDir, data survives container crashes and restarts. However, once the Pod is restarted or recreated, all the data in the volume is lost.

But what if we want our data to survive even Pod restarts or crashes? That’s where hostPath comes in. Let’s take a look.

📦 hostPath

The main limitation of emptyDir is that the data is lost when the Pod restarts. This problem is addressed by hostPath.

The What ?

With hostPath, the data is preserved no matter what happens to the Pod, as long as the Pod is scheduled on the same node.

🤔 How does this work?

The trick is simple: hostPath mounts a specific file or directory from the node’s filesystem directly into your Pod. So even if the Pod dies and a new Pod starts on the same node, the data is still there—because it never left the node in the first place.

The When?

Use hostPath when:

• You want data to survive Pod restarts

• You are okay with the Pod running on the same node

• You are working in local development, single-node clusters, or testing environments

• You need access to node-level files (logs, sockets, configs)

⚠️ Not recommended for multi-node production workloads due to portability and security concerns.

⚠️ During node maintenance or a node crash, extra care is required. Any Pod using a hostPath volume will lose its data if it gets drained and rescheduled onto a different node. The data is preserved only as long as the Pod remains on the same node and the node is up and running.

The How ?

apiVersion: v1
kind: Pod
metadata:
  name: shared-workspace
spec:
  containers:
    - name: producer
      image: busybox
      volumeMounts:
        - name: shared-data
          mountPath: /app/data
    - name: consumer
      image: busybox
      volumeMounts:
        - name: shared-data
          mountPath: /app/input
  volumes:
    - name: shared-data
      hostPath: # <--- The magic keyword hostPath
        path: /tmp/shared-data
        type: DirectoryOrCreate

🔍 Let’s Dissect the Example Step by Step

• Instead of emptyDir, we now use hostPath

• The data is stored on the node at /tmp/shared-data

• The type field tells Kubernetes what it should expect at the given path on the node and what to do if it doesn’t exist. If it does not exist create the directory automatically

• 📦 Common hostPath Types

🤔 Common Doubts

You might have this question in mind: We know that hostPath is tied to a specific node. What happens if I restart the Pod or apply a rollout that recreates Pods? Is there any guarantee that the Pod will be scheduled on the same node where my data exists? And if it gets scheduled on a different node, will the data be lost?

Yes—if the Pod is scheduled on a different node, the data will be lost. In most cases, Kubernetes tries to reschedule the Pod onto the same node it was previously running on. This is why, during normal Pod restarts or rollouts, you often see the Pod coming back on the same node and your data appearing to be “safe.”

❓ Why Does This Happen?

When the scheduler evaluates where to place a Pod, it considers: Existing node assignments, Node availability, Resource constraints. If the node is Healthy, Not Drained, and has sufficient resources Kubernetes will typically place the pod back on the same node. ‘

However There is no strict guarantee. If the node is crashed, drained, under maintenance, or out of resources the pod will be rescheduled to different node.

So far, we’ve seen how hostPath and emptyDir volumes work—great for temporary data, experiments, caching, and understanding how Kubernetes handles storage inside a Pod or a node. But what if you want your data to live beyond Pod restarts, survive node failures, and stay safe from all the usual Kubernetes chaos? 🤯

That’s exactly where Persistent Volumes (PV) come into the picture. They solve the problem of long-lived, reliable storage in Kubernetes. I’ve covered Persistent Volumes in detail in the next blog, breaking down how they work, why they matter, and when you should use them in real-world setups.

If you’ve made it this far — great job 👏 You now have a solid understanding of Kubernetes’ ephemeral storage story.

👉 Continue the journey here: Persistent Volumes in Kubernetes — and let’s level up your storage game 🚀

If you're just joining the journey, you can follow the entire series from here:

🧠 Setting the Context: The Core Logic of Our App

The core functionality of our ToDo application includes:

• ✅ Creating a ToDo item

• ✏️ Updating a ToDo item

• ❌ Deleting a ToDo item

• 📄 Reading (fetching) ToDo items

In this section, we’ll implement the application logic that powers these features.

We'll be writing code that defines what the app should do when a user makes a specific request — and what kind of response the app should return.

🛠️ How Are We Going to Do This?

Just like in the previous blogs of this series, we’ll follow a two-step approach:

1. Start with the basic (native) approach to understand how things work at the core level.

2. Evaluate and adopt a better alternative library that simplifies and enhances the process.

3. Use the selected method to implement the entire logic and test it end-to-end.

Let’s get started and build the Application Layer of our ToDo app! 🚀

🌐 Understanding HTTP in Our Web Application

Since we’re building a web application, all user interactions will happen through HTTP requests. We should handle this request to serve the user.

💡 Wait… what do you mean by handling HTTP requests?

Great question! In a web app, users interact with the backend using HTTP — the protocol behind the web. For example, if a user wants to create a ToDo item, they would make a request like:

curl -X POST http://mytodoapp/create \
  -H "Content-Type: application/json" \
  -d '{
    "task_name": "Write blog post",
    "description": "Complete the blog post for the Golang ToDo app",
    "status": "NotStarted",
    "end_date": "2024-06-30T23:59:00Z"
  }'

You might be wondering: “What exactly is that?” Don’t worry — we’ll cover it in more detail soon. For now, just understand that users will interact with our app using HTTP URLs and endpoints (like /create, /update, etc.) to trigger certain actions. And when we say "handling HTTP requests", we mean:

The application should be able to receive the request, extract any information from it (like request body or URL parameters), and perform the appropriate action in response.

How Do We Handle HTTP in Golang? To handle HTTP in Go, there are several libraries available. One of the most basic and native options is: net/http

🛠 Let’s Start with Creating a ToDo Item

To begin, we’ll focus on one core operation: creating a new ToDo item.

We’ll implement an HTTP handler using the net/http package that listens for a POST request, extracts data from the request body, and passes it to our application logic (which we already connected to the database layer).

In the next section, we’ll walk through how to:

1. Define an HTTP handler function

2. Read and parse the JSON body from the client

3. Validate and use the data to create a new ToDo

4. Return a meaningful response back to the user

Let’s dive in and build the POST /todo/create endpoint using net/http.

🚦 Recollecting what we will be doing

From the previous step, we now understand that:

• ✅ We need to import an HTTP handler to allow our Go application to serve web requests

• ✅ To read and parse the incoming request body (usually in JSON), we need a package for decoding JSON. A quick Google search shows that Go provides a standard library for this:"encoding/json"

• ✅ We need to write the logic to:

  1. Receive incoming HTTP requests

  2. Read the data (like a new ToDo item) from the request body

  3. Pass it to our business logic (e.g., CreateToDo)

  4. Return a proper response to the user

▶️ Let’s Code from Where We Left

If you're following along, continue from the code snippet in the last blog. We'll now build the ToDoHanlderCreate using the net/http and encoding/json packages.

Let’s plug that into our main.go file from our last blog and get started 🚀

package main

import (
    "fmt"
    "gorm.io/driver/mysql"
    "gorm.io/gorm"
)

var (
    // Declare a global variable to hold the DB connection
    db *gorm.DB // Type: pointer to gorm.DB
)

type ToDo struct{
    gorm.Model
    TaskName    string     `gorm:"not null" json:"title"`
    Description string     `gorm:"type text" json:"description"`
    Status      string     `gorm:"default:NotStarted" json:"status" validate:"required,oneof=NotStarted InProgress Pending Completed"`
    EndDate     *time.Time `json:"end_date"`
}

// ConnectDB establishes the connection to the MySQL database
func ConnectDB() {
    // Data Source Name (DSN): contains database connection info
    dsn := "root:password@tcp(127.0.0.1:3306)/todo?charset=utf8&parseTime=True&loc=Local"

    // Connect to the DB using GORM and MySQL driver
    db_conn, err := gorm.Open(mysql.Open(dsn), &gorm.Config{})
    if err != nil {
        // If connection fails, panic and log the error
        panic("Failed to connect to the database: " + err.Error())
    }

    /*Why I'm passing value here why can't I pass it like db,err. 
    If I do that I have to either add err variable globally or remove db global var as I'm using := so I used a var to exchnage value
    */
    db = db_conn 
}

func InitDB(){
    ConnectDB() // This will create new connection and the value is stored in DB variable

    // Here we are saying that Use the Schema &ToDo and create table using ou DB connection 'db'
    //'{}' defines that create empty table with that ToDo Struct Schema
    db.AutoMigrate(&ToDo{})
}

// CreateToDo inserts a new ToDo item into the database.
// It takes a pointer to a ToDo struct as input and returns the same pointer after insertion.
func CreateToDo(todo *ToDo) *ToDo {
    // Use GORM's Create method to insert the new record into the database.
    // The &todo tells GORM to insert the data from the memory location of the passed ToDo.
    db.Create(&todo)
    // Return the same pointer, now updated with DB-generated fields (like ID, CreatedAt, etc.)
    return todo
}

// .... Truncating other code you can refer from the link provided above
...
...
...

Let’s import and modify the code

package main

import (
    "fmt"
    "gorm.io/driver/mysql"
    "gorm.io/gorm"
    "time"
    "net/http"
    "encoding/json"
)

var (
    // Declare a global variable to hold the DB connection
    db *gorm.DB // Type: pointer to gorm.DB
)

type ToDo struct{
    gorm.Model
    TaskName    string     `gorm:"not null" json:"title"`
    Description string     `gorm:"type text" json:"description"`
    Status      string     `gorm:"default:NotStarted" json:"status" validate:"required,oneof=NotStarted InProgress Pending Completed"`
    EndDate     *time.Time `json:"end_date"`
}

func ConnectDB() {...}
func InitDB(){...}

func ToDoHanlderCreate(w http.ResponseWriter, r *http.Request){
    // step1: Ensure it's a Post request
    if r.Method != http.MethodPost {
     http.Error(w, "Only Post method is allowed", http.StatusMethodNotAllowed)
    return
    }
    // Step 2: Parse the incoming JSON body into a ToDo struct
    var todo ToDo
    err := json.NewDecoder(r.Body).Decode(&todo)
    if err != nil {
        http.Error(w, "Invalid Json Input", http.StatusBadRequest)
        return
    }
    // Step 3: Save the new ToDo item to the database
    db.Create(&todo) // Using Gorm Native create object option
    // Step 4: Set the content type and return the created object as JSON
    w.Header().Set("Content-Type", "application/json") // Set response type as JSON
    w.WriteHeader(http.StatusCreated) // Set HTTP status code to 201 (Created)
    json.NewEncoder(w).Encode(todo) // Write the todo object as JSON in response


}

func main() {
    InitDB() // Initialize DB

    // Register the route and handler
    http.HandleFunc("/todo/create", ToDoHanlderCreate)

    fmt.Println("🚀 Server running at http://localhost:8080")
    err := http.ListenAndServe(":8080", nil)
    if err != nil {
        fmt.Println("Error starting server:", err)
}

Lets breakdown the ToDoHanlderCreate function and how we change main.

Step : 1 - ToDoHandlerFunction

• func ToDoHanlderCreate(w http.ResponseWriter, r *http.Request){…}

  📥 r *http.Request

  • This parameter is used to receive and read the incoming HTTP request from the user.

  • It contains:

    • Request method (e.g., GET, POST)

    • Headers

    • Query parameters

    • Request body (like form data or JSON payload)

  • We're using a pointer (*http.Request) so we’re accessing the actual request data without copying it.

  • We store it in a variable named r so we can access its fields inside the function.

Example: To read the JSON request body, we'll use r.Body.

📤 w http.ResponseWriter

• This parameter is used to send the response back to the user.

• It lets us:

  • Write text or JSON data as the response

  • Set response status codes (like 200 OK or 400 Bad Request)

  • Set response headers (like Content-Type)

• Anything you write to w is what the client receives.

Example: To send back a success message, we can use w.Write([]byte("ToDo created")).

Step : 2 - Json Decoding

    // Step 2: Parse the incoming JSON body into a ToDo struct
    var todo ToDo
    err := json.NewDecoder(r.Body).Decode(&todo)
    if err != nil {
        http.Error(w, "Invalid Json Input", http.StatusBadRequest)
        return
    }

• Here, we create a variable called todo, which is of type ToDo struct. We need this variable because once we extract the content from the HTTP request body, we store it in this variable so we can use it later—for example, to insert it into the database.

• err := json.NewDecoder(r.Body).Decode(&todo) - This line uses Go’s encoding/json package to decode the JSON data from the request body and assign it to the todo variable.

• After decoding, we handle any potential error—this helps us ensure that the request body is in the correct format. If it's not, we can return a proper error message to the client.

Step : 3 - Create ToDo

db.Create(&todo) - We use GORM’s built-in Create method to add a new entry to our database. The data we’re inserting comes from the todo variable, which already holds the content we extracted and decoded from the incoming HTTP request. By passing &todo, we're telling GORM to insert the data from that memory location into the corresponding table.

Step : 4 - Response to user

w.Header().Set("Content-Type", "application/json") // Set response type as JSON
w.WriteHeader(http.StatusCreated)                 // Set HTTP status code to 201 (Created)
json.NewEncoder(w).Encode(todo)                   // Write the todo object as JSON in response

• w.Header().Set("Content-Type", "application/json") - Tells the client that you're sending back JSON data.

• w.WriteHeader(http.StatusCreated)- Sends an HTTP status code 201 Created, which is the correct status code for successful resource creation.

• json.NewEncoder(w).Encode(todo) - Converts the todo struct to JSON and writes it to the response body.

main() Function:

InitDB() // Initialize DB

    // Register the route and handler
    http.HandleFunc("/todo/create", ToDoHanlderCreate)

    fmt.Println("🚀 Server running at http://localhost:8080")
    err := http.ListenAndServe(":8080", nil)
    if err != nil {
        fmt.Println("Error starting server:", err)

• InitDB() This function initializes the database connection using GORM and MySQL. It also runs the AutoMigrate() function, which ensures that our ToDo model is translated into a table in the database.

• http.HandleFunc("/todo/create", ToDoHandlerCreate) Registering the Route. This means that any POST request to /todo/create will be handled by the ToDoHandlerCreate function. This is the entry point for users to create new ToDo items.

• http.ListenAndServe(":8080", nil) Starting the HTTP Server. This tells Go to start an HTTP server on port 8080. As long as the server is running, it will listen for incoming HTTP requests on that port.

✅ Summary

• Database: Initialized and migrated with InitDB()

• Route: /todo/create handles new ToDo creation

• Server: Listens on http://localhost:8080

This covers the full operation of create ToDo item on our ToDo app using native http request. Here is the complete code snippet.

package main

import (
     "fmt"
    "gorm.io/driver/mysql"
    "gorm.io/gorm"
    "time"
    "net/http"
    "encoding/json"


)

var db *gorm.DB // Global DB connection

type ToDo struct {
    gorm.Model
    TaskName    string     `gorm:"not null" json:"title"`
    Description string     `gorm:"type:text" json:"description"`
    Status      string     `gorm:"default:NotStarted" json:"status"`
    EndDate     *time.Time `json:"end_date"`
}

// ConnectDB sets up the MySQL connection and assigns it to the global db variable
func ConnectDB() {
    dsn := "root:password@tcp(127.0.0.1:3306)/todo?charset=utf8&parseTime=True&loc=Local"
    dbConn, err := gorm.Open(mysql.Open(dsn), &gorm.Config{})
    if err != nil {
        panic("Failed to connect to the database: " + err.Error())
    }
    db = dbConn
}

// InitDB initializes the DB and migrates the schema
func InitDB() {
    ConnectDB()
    db.AutoMigrate(&ToDo{}) // Creates the 'todos' table based on our struct
}

// CreateToDoHandler handles HTTP POST requests to create a new ToDo item
func ToDoHanlderCreate(w http.ResponseWriter, r *http.Request){
    // step1: Ensure it's a Post request
    if r.Method != http.MethodPost {
     http.Error(w, "Only Post method is allowed", http.StatusMethodNotAllowed)
    return
    }
    // Step 2: Parse the incoming JSON body into a ToDo struct
    var todo ToDo
    err := json.NewDecoder(r.Body).Decode(&todo)
    if err != nil {
        http.Error(w, "Invalid Json Input", http.StatusBadRequest)
        return
    }
    // Step 3: Save the new ToDo item to the database
    db.Create(&todo) // Using Gorm Native create object option
    // Step 4: Set the content type and return the created object as JSON
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(http.StatusCreated)
    json.NewEncoder(w).Encode(todo)

}

func main() {
    InitDB() // Initialize DB

    // Register the route and handler
    http.HandleFunc("/todo/create", ToDoHanlderCreate)

    fmt.Println("🚀 Server running at http://localhost:8080")
    err := http.ListenAndServe(":8080", nil)
    if err != nil {
        fmt.Println("Error starting server:", err)
    }  
}

▶️ How to test the code

• Create Database if you don’t have one by following the steps mentioned here

• Create a file called main.go and copy the above content and paste it.

• Initialise your go app by running go mod init example.com/todo

• Run go mod tidy to download all the dependencies

• Finally run go run main.go start your application. You will see the below response on your terminal

  

• Now open new terminal and use below curl command to create an entry in your ToDo app

curl -X POST http://localhost:8080/todo/create \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Write Go server",
    "description": "Build a simple ToDo API with Go and GORM",
    "status": "InProgress",
    "end_date": "2025-06-30T17:00:00Z"
  }'

• You will get the response like this

• To check it on database run the below commands

# Open CMD and run
docker exec -it mysql /bin/bash
mysql -u root -p
# <Enter your password when it asks i.e "password">
# Now you will be in Mysql db
USE todo;
SELECT * FROM to_dos

• This command will show you the newly created Entry

📝 Conclusion

That wraps up our implementation of a simple ToDo API using Go’s net/http package, GORM, and MySQL. In this post, we created the model, set up the database connection, handled a POST request, and tested it with curl.

In the next blog post, we’ll explore alternatives to the net/http standard library—specifically, why developers often choose web frameworks like Gin or Echo for larger or more maintainable applications. We'll also complete the Application Layer, continuing our journey toward building a clean and modular Go backend.

Stay tuned!

Before diving into the actual problem and its solution, let’s take a moment to understand what OpenSearch is and where it's commonly used.

📌 What is OpenSearch? Where is it Used?

OpenSearch is an AWS-managed service based on the open-source fork of Elasticsearch. It offers powerful capabilities for indexing, searching, and analyzing large volumes of data in near real time.

OpenSearch is widely used in infrastructure for a variety of use cases, including:

• Log analytics

• Full-text search

• Application performance monitoring

• Security information and event management (SIEM)

In recent years, many companies managing their infrastructure on AWS have been shifting towards AWS-managed services. This transition helps reduce the operational burden of managing infrastructure, allowing IT teams to focus on solving business-critical problems rather than maintaining and scaling services themselves.

AWS OpenSearch has become one of the most widely adopted services in this category, providing a fully managed, scalable, and secure alternative to running Elasticsearch on self-managed instances.

⚙️ How is OpenSearch Configured?

When using AWS OpenSearch Service, you typically configure the following key components:

🌐 Exposing OpenSearch to External Networks Using ALB

When you create an AWS-managed OpenSearch cluster, AWS provides a domain endpoint that is accessible only from within the same VPC, depending on your VPC settings and security group rules. However, resources or users outside the VPC—such as developers, third-party services, or monitoring tools—cannot access this endpoint directly.

In real-world setups, it's quite common for some users or systems to live outside the VPC. For example, accessing the OpenSearch Dashboard from an external network becomes difficult since direct connectivity isn’t allowed.

To solve this problem, we can expose OpenSearch securely through an AWS Application Load Balancer (ALB).

1. Create an ALB
   An internet-facing ALB is created to act as a public entry point. This ALB provides a public DNS name that external users and resources can access.

2. Register OpenSearch Node in a Target Group
   We then create a Target Group and register the IP address of the OpenSearch node behind the domain endpoint.

   🔹 Note: Even if your OpenSearch domain has multiple nodes, AWS internally exposes only one node behind the endpoint. Other nodes are used for redundancy and failover.

3. Connect Target Group to ALB
   The Target Group is attached to the ALB. This setup ensures that whenever a request hits the ALB, it is forwarded to the registered OpenSearch node in the Target Group.

4. Route 53 for Domain Management
   AWS Route 53 manages the public DNS name of the ALB. If you're using HTTPS, SSL certificates issued via ACM (AWS Certificate Manager) are also managed and linked here.

This setup enables external access to OpenSearch in a secure and controlled way. The ALB acts as a bridge between the external network and the private OpenSearch cluster inside the VPC.

⚠️ The Problem Statement: Why the ALB Connection Breaks

Here comes the real challenge.

In most setups, we do not maintain static (sticky) IP addresses for OpenSearch cluster nodes. When an OpenSearch cluster restarts—either due to scaling, upgrades, or internal AWS maintenance—the IP address of the node exposed via the endpoint can change.

Now, since the ALB target group is manually registered with the IP of the previously exposed node, it still tries to forward traffic to that old, now-invalid IP. But OpenSearch is now responding from a new node with a different IP. And here's the problem:

• The ALB doesn’t have any native integration with OpenSearch to update its target group dynamically.

• As a result, the target group points to a stale IP, and the target becomes unhealthy, breaking the connection between the ALB and OpenSearch.

This leads to downtime or failed requests from all external clients relying on the ALB for access.

🔁 The Need for Automation

To prevent this, we need a robust, automated mechanism to keep the ALB target group in sync with the currently exposed IP of the OpenSearch endpoint.

There are several ways to implement this, and in the next section, I’ll walk you through one of the approaches I recently implemented in a production environment.

✅ Quick Summary: How I Solved the ALB & OpenSearch Restart Issue

Before we dive into the detailed steps, here’s a quick overview of how this issue was resolved.

Assumption: You already have an OpenSearch cluster running and exposed via an ALB to allow access from external networks.

The core idea is to detect when the OpenSearch cluster restarts and automatically update the ALB target group with the currently active OpenSearch node IP. This ensures that the ALB always routes traffic to a healthy target, avoiding downtime or broken access for external users.

To achieve this:

• I set up a CloudWatch alarm that monitors the number of nodes in the OpenSearch cluster.

• Using EventBridge, I track when the alarm transitions from an ALARM state back to OK, indicating that the cluster has recovered after a restart.

• This triggers a Lambda function, which:

  • Fetches the active OpenSearch node IP.

  • Updates the ALB target group with the new IP.

  • Removes any stale IPs no longer part of the cluster.

This automation ensures that your ALB always points to a healthy OpenSearch node, even after a restart — maintaining uninterrupted access for external systems.

In the next section, I’ll walk you through each step of this setup with screenshots and configuration details.

🛠️ Detailed Procedure:

The steps outlined below can be implemented using any Infrastructure-as-Code (IaC) or configuration management tool of your choice. In this guide, the focus is on what needs to be implemented rather than how it's implemented with a specific tool. You’re free to use the tool or framework that best fits your environment — this guide should help you understand the logic and flow regardless of the platform.

🔹 Step 1: Get the Total Number of Nodes in Your OpenSearch ClusterYou can find the details in Cloud watch

The first step is to identify the total number of nodes (both master and data nodes) in your OpenSearch cluster. This information is essential for setting up a reliable CloudWatch alarm.

You can find this detail using Amazon CloudWatch:

1. Log in to the AWS Management Console.

2. In the search bar, type and select CloudWatch.

3. From the left-hand menu, click on "All metrics."

4. 

   In the search bar, type “ES” to filter OpenSearch-related metrics, From the results, select “ES → Per-Domain, Per-Client Metrics.”

   

5. Choose your OpenSearch domain and look for the Nodes metric. This will show the number of nodes currently active in your cluster.

6. 

   Check the checkbox for Nodes corresponding to your specific OpenSearch domain. This will display a graph showing the number of active nodes over time.

7. While there are certainly easier ways to retrieve the total number of nodes in an OpenSearch cluster, this approach has a key advantage — it allows us to directly create a CloudWatch alarm based on this metric.

8. So, as you might have guessed — the next step is to click on “Create alarm.”

9. 

   Configure the alarm setting as follow

10. 

    In the next step, click “Remove” under the notification section to delete any default SNS notification settings — since we’ll be using EventBridge to trigger the action instead of relying on SNS alerts.

11. 

    Add your preferred name to the Alarm

    

12. Finally Click on Create Alarm

13. Now, follow Steps 3 to 5 in the alarm creation flow. Once completed, you should see 1 alarm listed under your OpenSearch domain name in the CloudWatch metrics dashboard.

14. Select the Node Checkbox, and then select the Graphed Metrics

15. 

    Select Add Math » Conditional » Equals

16. 

    From the expression click Edit icon

17. 

    Change the value to m1 == <your total number of nodes>, where m1 refers to the metric ID for Nodes in your alarm, which you can find in the same menu under ID, and <your total number of nodes> is the value you noted in Step 8. Click on Apply

18. 

    Voila! The alarm is now set. It will return a value of 1 when the current node count is exactly equal to the expected total (in our case, 6), and 0 when it’s either greater or less than that.

19. This is exactly what we need — because any change in node count usually indicates that something has happened with the OpenSearch cluster (such as a restart or scaling event). At this point, we need to verify the ALB Target Group IP and update it if it no longer points to a healthy OpenSearch node.

20. Now, let’s move on to setting up EventBridge to trigger a Lambda function whenever this happens.

🔹 Step 2: Configure a Lambda Function to Update the Target Group with the OpenSearch Endpoint Node IP

There are plenty of resources — including official documentation and YouTube tutorials — that can guide you through how to create and configure a Lambda function.

In this section, I’ll focus on what matters most for this use case:

• The runtime/language you should select for your Lambda. - Python:3.9 and above

• The script you’ll use to update the Target Group.

• The environment variables that need to be configured for the Lambda to work effectively.

  • OPENSEARCH_HOST:

    • You can get this value from, Login in to console » Search Opensearch » Click Domains » Select your Domain » You can find this value under VPC EndPoint

    • (Remove “https://” when adding it in env variable)

  • TARGET_GROUP_ARN

    • You can get this value from Login in to console »Search Target Group » Select your Target Group » You can find this value under ARN

• Script to use

  Note: I’ve hard coded the port to 443 replace it with your port number

import boto3
import os
import socket

elb = boto3.client('elbv2')

TARGET_GROUP_ARN = os.environ['TARGET_GROUP_ARN']
OPENSEARCH_HOST = os.environ['OPENSEARCH_HOST']

def lambda_handler(event, context):
    try:
        # Resolve new IP of OpenSearch
        new_ip = socket.gethostbyname(OPENSEARCH_HOST)
        print(f"Resolved OpenSearch IP: {new_ip}")

        # Describe current targets
        current_targets = elb.describe_target_health(TargetGroupArn=TARGET_GROUP_ARN)
        registered_ips = [t['Target']['Id'] for t in current_targets['TargetHealthDescriptions']]
        print(f"Currently registered IPs: {registered_ips}")

        #Deregister old IPs
        for ip in registered_ips:
            if ip != new_ip:
                print(f"Deregistering IP: {ip}")
                elb.deregister_targets(
                    TargetGroupArn=TARGET_GROUP_ARN,
                    Targets=[{"Id": ip, "Port": 443}]
                )

        # Register new IP if not already registered
        if new_ip not in registered_ips:
            print(f"Registering new IP: {new_ip}")
            elb.register_targets(
                TargetGroupArn=TARGET_GROUP_ARN,
                Targets=[{"Id": new_ip, "Port": 443}]
            )

        return {"status": "Success", "new_ip": new_ip}

    except Exception as e:
        print(f"Error: {str(e)}")
        raise e

• Make sure you have necessary permission for this Lambda function to, if you don’t have create an IAM role with below access.

  • Read the VPC Endpoint configuration settings from Opensearch

  • Ability to add Targets to the Target group

  • Ability to Selete/Drain targets from the target group

• Once all are set test it manually invoking the lambda function

🔹 Step 3: Configuring Event Bridge to trigger Lambda

1. Log in to the AWS Management Console.

2. In the search bar, type and select Amazon EventBridge » From left menu Click Rules » Select Create rule

3. 

   Give preferred rule name and description and Click Next

4. 

   Click Edit Pattern

5. 

   Replace the content what ever needed as mentioned below and paste the content in the code snippet.

   - YOUR ALARM NAME » Replace this with the Alarm name which you created in Step 1

{
  "source": ["aws.cloudwatch"],
  "detail-type": ["CloudWatch Alarm State Change"],
  "detail": {
    "alarmName": ["<YOUR ALARM NAME>"],
    "state": {
      "value": ["ALARM", "OK"]
    },
    "previousState": {
      "value": ["OK", "ALARM"]
    }
  }
}

6. Explanation of the condition

   1. "source": ["aws.cloudwatch"]
      Ensures the event is coming from Amazon CloudWatch.

   2. "detail-type": ["CloudWatch Alarm State Change"]
      Captures only events related to alarm state changes.

   3. "alarmName": ["<YOUR ALARM NAME>"]
      Filters the event to trigger only for the specific alarm you created to monitor OpenSearch node count.

   4. "state": { "value": ["ALARM", "OK"] }
      Triggers the rule when the alarm enters from the ALARM to OK state.

   5. "previousState": { "value": ["OK", "ALARM"] }
      Ensures the transition is meaningful — for example, from OK ➝ ALARM and then ALARM ➝ OK — avoiding unnecessary triggers.

7. Click Next to save your pattern

8. Then For the target

   1. Target Type : AWS Service

   2. Select a target » Lambda Function

   3. Function » Select the function you created on Step 2 from drop down

   4. Leave the other box as it is and Click Next

9. In the next step add the necessary tag if required and click Next and Finally review the configuration and click on Create Rule

Voila! This concludes the overall configuration for automating the update of OpenSearch Node IPs in the Target Group.

With this setup in place, your Target Group will always stay up to date with the active and healthy IP address of your OpenSearch cluster — ensuring seamless connectivity even during restarts or node changes.