But if you are someone like me, you might have paused for a second and wondered:

“Wait… what exactly are all these tools?”
“Where are they actually used?”
“Why do we even need so many networking components in Kubernetes?”

Trust me, I had the exact same questions.

At first, terms like Ingress, Gateway API, CNI, Service Mesh, Cilium, Istio, Envoy, MetalLB, and Load Balancer can feel like a giant networking soup 🍜.

But here’s the interesting part: Most of these tools are not competing with each other. Instead, they solve different networking problems at different layers of Kubernetes.

So in this blog, we are going to take a fun and practical deep dive into:

✅ Kubernetes networking basics
✅ What happens when traffic enters a Kubernetes cluster
✅ Which networking tools are used where
✅ When to use what (without getting confused!)

By the end of this blog, whenever someone mentions a Kubernetes networking tool, you should be able to say:

“Ah okay, I know exactly where this fits in.”

So grab your coffee ☕ (or tea 😄), and let’s get started.

The Basics: How traffic flows in Kubernetes ?

Before we start throwing around fancy names like Cilium, Istio, Gateway API, Envoy, MetalLB, or Service Mesh, let’s first understand something important:

How does a normal request travel inside Kubernetes? Because once we understand the journey of a request, understanding networking tools becomes much easier.

When a request comes from the internet to your application running inside Kubernetes, it goes through multiple layers before reaching the actual Pod.

A request flowing inside Kubernetes typically goes through the following stages:

Internet → Ingress / Gateway Layer → Service Layer → Finds the Correct Backend Pod → Pod Networking Layer → Delivers Traffic to the Pod → Application Pod → Serves the Request

The request originates from an external user or application. For example: https://myapp.com

At this point, Kubernetes needs a way to expose your application to the outside world. So the traffic enters the cluster through the:

Ingress / Gateway Layer

• Ingress / Gateway Layer → Controls External Traffic.

• This layer acts as the entry point of the cluster.

Here Kubernetes decides:

• Which application/service the request should go to URL/path-based routing (/payment, /user, /orders)

• Authentication and authorization

• TLS/HTTPS termination

• Rate limiting and traffic filtering

Once the request rules are evaluated, the traffic is forwarded to the:

Service Layer

• Service Layer → Finds the Correct Backend Pod

• Services act as a stable endpoint for Pods. Since Pods are temporary and their IPs can change, Services help Kubernetes reliably find the correct application Pods.

Here Kubernetes:

• Finds matching Pods using labels

• Performs load balancing Routes traffic to healthy Pods

• Enables service-to-service communication

Once the correct Pod endpoint is identified, traffic is forwarded through:

Pod Networking Layer

• Pod Networking Layer → Delivers Traffic to the Pod

• This is where the actual packet routing happens. The networking layer ensures Pods can communicate: Within the same node, Across different nodes, Across namespaces, Across the cluster.

Here Kubernetes networking components:

• Assign IP addresses to Pods,

• Route traffic between Pods

• Handle inter-node communication

• Apply networking rules

Finally, the request reaches:

Application Pod

• Application Pod → Serves the Request

• This is where your actual application runs. The Pod processes the request and sends the response back through the same networking path.

Security & Observability → Monitors the Entire Journey

While the request travels through Kubernetes, security and monitoring tools continuously observe the traffic.

This layer helps with:

• Restricting communication between services

• Monitoring traffic flow Tracking latency and failures

• Applying network policies

• Observability and troubleshooting

This happens across all layers, not just at one specific point.

Now that we understand how traffic flows inside Kubernetes, the next obvious question is:

What tools are used at each layer, and what problem do they solve?

Deep Dive into Kubernetes Networking Tools

As we saw earlier, a request travels through different networking layers before reaching the application Pod.

Each layer has its own responsibility, and naturally, different tools are built to solve problems at those specific layers. So let’s deep dive into the tools layer by layer and understand:

• What does this layer handle?

• Popular Tools Available

1. Ingress/Gateway Layer:

As discussed earlier, the Ingress/Gateway Layer acts as the entry point into the Kubernetes cluster. This layer is mainly responsible for handling incoming external traffic and deciding how requests should enter and move inside the cluster.

What does this layer handle?

This layer controls:

• Which application/service the request should go to

• URL / Path-based routing

• Authentication & Authorization

• TLS / HTTPS Termination - Handling HTTPS Encryption

• Rate Limiting & Traffic Filtering

Popular Tools Used in the Ingress / Gateway Layer

• Ingress Controllers: NGINX Ingress Controller, HAProxy Ingress, Traefik Ingress, AWS Load Balancer Controller, Azure Application Gateway Ingress Controller

• NGINX Gateway / Gateway API: Modern implementation for managing external traffic using the Kubernetes Gateway API.

• Traefik: A lightweight and cloud-native ingress/gateway solution.

• Cilium Gateway: Part of the Cilium ecosystem. Uses eBPF-powered networking and implements Gateway API capabilities.

• Istio Gateway: Part of the Istio Service Mesh ecosystem.

2. Service Layer Tools

Once the traffic enters the cluster through the Ingress/Gateway Layer, it is forwarded to a Kubernetes Service.

But here’s the challenge: Pods in Kubernetes are temporary (ephemeral). So how does Kubernetes make sure traffic always reaches the correct Pod? This is where the Service Layer comes into play.

The Service Layer acts as the traffic manager inside the cluster and ensures requests are routed reliably to healthy application Pods.

What Does the Service Layer Handle?

• Service Discovery: Services help applications find each other without hardcoding Pod IPs.

• Load Balancing Across Pods: Traffic is distributed across Pods.

• Service-to-Service Communication: Inside Kubernetes, services constantly communicate with each other.

• Internal Traffic Encryption (mTLS): Normally, traffic inside the cluster is Plain Text. Because TLS is often terminated at the Ingress layer. But industries like Banking, Healthcare etc may require Encrypted communication between services inside the cluster as well. This is where mTLS (Mutual TLS) becomes important.

Popular Tools Used in the Service Layer

• kube-proxy: This is the default Kubernetes networking component responsible for service routing.

• Istio: One of the most popular Service Mesh tools. Istio adds advanced traffic management between services.

• Linkerd: A lightweight Service Mesh alternative.

• Consul Service Mesh: Another Service Mesh offering. It works beyond Kubernetes as well.

• Cilium Service Mesh: If you are already using Cilium, it can also provide Service Mesh capabilities without traditional sidecars.

Note: One important thing to understand is that not all tools provide the same capabilities.

For example, kube-proxy handles service routing and load balancing, but it does not provide features like mTLS (Mutual TLS), advanced traffic management, or observability.

If your requirement includes secure service-to-service communication (mTLS), traffic control, retries, or advanced visibility, you would typically need a Service Mesh like Istio, Linkerd, Consul, or solutions like Cilium Service Mesh.

The tools listed above are some of the popular options, and the right choice depends on your use case, complexity, security requirements, and operational needs.

3. Pod Networking Tools

Now the request knows which Pod it should go to. But another important question comes up:

How does Kubernetes know:

• Where the Pod is?

• What its IP address is?

• How to route traffic across nodes?

• How Pods communicate reliably?

This is where Pod Networking comes into play. Honestly, this is one of the most important layers in Kubernetes networking because:

Without Pod Networking, Pods simply cannot talk to each other.

What Does Pod Networking Handle?

• Assigning IP Addresses to Pods: Every Pod gets its own IP address

• Pod-to-Pod Communication: Within the same node, Across different nodes, Across namespaces without requiring NAT or complicated translations.

• Cross-Node Networking: A CNI((Container Network Interface)) plugin handles the actual networking implementation.

• Traffic Routing Between Nodes: This decide, how traffic moves, which node owns which pod IP range, how traffic gets routed

• Network Policies: This layer can also enforce rules. Just because Pods can communicate doesn’t mean they always should. Security restrictions matter.

Popular Tools Used in Pod Networking

• Flannel: One of the simplest Kubernetes networking tools. Flannel focuses mainly on Pod-to-Pod communication. This doesn't have all the above capabilities.

• Calico: One of the most widely used Kubernetes networking solutions. This has Strong Network Policy support. Preferred choice in Enterprise, Prod Cluster and Security-focused environments

• Cilium: Probably one of the hottest Kubernetes networking tools right now. It uses eBPF for traffic communication instead of traditional Linux Networking. Cilium is a complete ecosystem which span across all layers

• AWS VPC CNI / Cloud Provider CNIs: Cloud providers also offer native networking plugins. AWS VPC CNI, Azure CNI, Google Kubernetes Engine CNI

Note: In real world Kubernetes environments, it is common to use multiple networking tools together.

For example, in cloud-managed Kubernetes services like EKS, companies often use the cloud-native CNI plugin (such as AWS VPC CNI) for Pod communication and add tools like Calico or Cilium on top for additional features such as Network Policies, security, and observability.

4. Security & Observability Tools

At this point, traffic is successfully flowing inside the cluster. Requests are reaching the correct Pods. But here comes an important question:

How do we control who can talk to whom?
How do we monitor what is happening inside the cluster?

This is where Security & Observability tools come into the picture.

This layer helps us:

• Restrict communication between Pods and services

• Monitor traffic flow

• Debug networking problems

• Understand service dependencies

• Improve visibility into cluster communication

What Does This Layer Handle?

• Network Policies: Who can communicate with whom

• Traffic Observability: Adding detailed visibility to the network flows

• Security Monitoring: Identifying unusual traffic, Policy violations etc

Popular Tools Used in This Layer

• Kubernetes Network Policies: This is the native Kubernetes way to restrict Pod communication. Kubernetes Network Policies alone do not enforce anything you need compatible CNI like cilium or calico

• Calico: Apart from Pod networking, Calico provides strong Network Policy enforcement, Traffic Filtering, and Security Isolation

• Cilium + Hubble: Cilium takes security and observability to another level, along with many of the features offered by Calico. It provides advanced networking, security, and traffic visibility capabilities. Hubble is Cilium’s visualization tool

• Istio Observability: Similar to Cilium, Istio also provides security and observability features. If you are already using Istio as your Service Mesh, you can make use of its built-in capabilities.

Conclusion

By now, you should have a better understanding of the different Kubernetes networking tools, where they fit in the networking flow, and what problem each of them is trying to solve.

Some tools help with:

• Exposing applications to the outside world (Ingress / Gateway)

• Service-to-service communication (Service Layer / Service Mesh)

• Pod communication across nodes (CNI / Pod Networking)

• Security, visibility, and traffic monitoring (Observability & Network Policies)

So the next time you hear terms like Cilium, Calico, Istio, NGINX Gateway, or Traefik, hopefully you will have a clearer idea of:

Where they fit and what their purpose is.

For a quick recap and better understanding, feel free to refer to the Flow Diagram.

I hope this blog helped simplify Kubernetes networking a little 😄

Happy Learning! 🚀
See you in the next one.

In this blog, we’ll explore some important Terraform insights and nitty-gritties that are extremely useful for day to day work. These are the kind of concepts that not only make you better at Terraform but also give you an edge in technical discussions and interviews.

The blog is structured in a simple and practical way. Each topic will cover:

• What it is – A clear explanation of the concept

• Why and Where it is used – Real-world scenarios where it applies

• How to use it – Practical usage with Terraform

• Example – So you can relate and apply it immediately

This approach ensures that you don’t just understand the concept, you know exactly how to use it.

Let’s get started 🚀

How do we structure the terraform code for large project

One of the most common questions we encounter is: “How do we structure Terraform code for a large project?”

This becomes even more critical when working on a greenfield setup, where everything is being built from scratch. The decisions you make at this stage will directly impact how well your infrastructure scales, how easy it is to maintain, and how effectively teams can collaborate.

So, how do we design a Terraform project that not only works today but also scales seamlessly in the future? Let’s dive deeper into this.

What it is ?

Before diving into how to structure a Terraform project, let’s first get the basics clear: What exactly are we structuring? What are we really talking about here?

Every Terraform project is made up of a set of core files and components. The real challenge is not creating these files, it’s about organizing them in a way that scales with your infrastructure.

Core Components in a Terraform Project

• Modules: Modules are the building blocks of any Terraform project. It helps you to Reuse Code, Maintain Consistency and Avoid Duplication. A typical module consists of:

  • main.tf → Defines the resources

  • variables.tf → Input variables.

  • outputs.tf → Exposes values

  • provider.tf → Provider configuration

  • versions.tf → Terraform & provider version constraints

  • data.tf → Data sources

  • backend.tf → Remote state configuration (sometimes kept separate mostly added with versions.tf)

• State File Management: Terraform maintains a state file to track the current state of your infrastructure. Proper state management is critical for Preventing Conflicts, Ensuring Consistency and Enabling team collaboration.

  • Key considerations:

    • Where is the state stored? (local vs remote),

    • How is state locking handled?

    • How do multiple users safely interact with it?

    • ⚠️ Pain Point: Large State File Slowing Down Terraform: When all resources are managed in a single Terraform state file, it grows significantly in size and slows down deployments because Terraform must evaluate the entire state during every run; to avoid this, it’s a best practice to split state files logically—either by resource type or architectural design to improve performance, scalability, and maintainability.

• Multi-Environment & Multi-Region Deployment: Real-world infrastructure is rarely single-environment. This involves Structuring Folders properly, Passing Environment specific variables, Managing isolated State files

  • You need to manage deployments across Environments (dev, stag and prod etc..) and across region (us-east-1, eu-west-1 etc..)

• Input Variables & Defaults Management: Variables make your Terraform code dynamic and reusable. Doing a right setup here will allow our code base to work across different region and enviornment

  • Things to Consider

    • Defining clear input variables

    • Setting sensible defaults

    • Overriding values per environment

So, when we talk about structuring Terraform, we are essentially talking about how we organize these building blocks effectively.

In Simple Terms

Structuring Terraform is about:

Organizing modules, state, variables, and environments in a way that your code scales along with your infrastructure without breaking performance, security, or maintainability.

Now we understand the what part let's discuss the why part.

Why?

Let's talk about why this is needed, As your infrastructure grows, your Terraform code should be able to:

• Handle complexity without becoming messy: As more components (infra resource) are added, the codebase should remain clean, well organized, and easy to debug or navigate.

• Support multiple environments and regions: The same codebase should support deployments across different environments (dev, staging, prod) and regions with minimal or no changes to the core logic.

• Maintain performance and security: Execution performance and security standards should remain consistent or improve as the infrastructure scales, without degradation.

• Enable easy collaboration across teams: As infrastructure grows, more team members will contribute. The code should be structured in a way that enables seamless collaboration and reduces conflicts.

By arranging them in this way helps you to grow your IaC seamlessly as your infrastructure grows.

How ?

When starting a greenfield Infrastructure as Code (IaC) project, there are multiple ways to structure and manage your infrastructure. Choosing the right approach early helps improve scalability, maintainability, collaboration, and operational efficiency as the infrastructure grows.

In this section, we will discuss two commonly used approaches for organizing Terraform based IaC projects:

1. Terraform Workspace Approach

2. Modular Folder-Based Approach

Terraform Workspace Approach

What is Terraform Workspace?

Terraform Workspaces allow you to manage multiple instances of the same infrastructure configuration using a single Terraform codebase.

Each workspace maintains its own separate state file, enabling you to deploy the same infrastructure into different environments such as:

• Development (dev), Testing (test), Staging (stage), Production (prod)

Instead of maintaining separate folders or separate state files manually, Terraform automatically isolates the infrastructure state per workspace.

Why Use Terraform Workspaces?

As infrastructure grows, managing multiple environments becomes challenging. Teams often need identical infrastructure across environments with only minor differences such as:

• Instance size, Number of replicas, Database sizing

Terraform Workspaces help reduce duplication by allowing the same code to be reused across environments while keeping the infrastructure states isolated.

This approach improves:

• Code reusability

• Consistency across environments

• Simpler maintenance

• Faster onboarding

Where Can Terraform Workspaces Be Used?

1. Multiple Similar Environments Exist - Dev, Qa, Prod where you will use same infra setup while only minor changes are required

2. Platform or Shared Services Teams: Teams managing reusable infrastructure templates can deploy the same stack for different customers, regions, or business units.

3. Temporary or Feature Environments: Useful fof Feature branch testing, Sandbox testing

How Terraform Workspace Works

Consider a hotel where:

• The kitchen prepares a common base meal for all guests.

• Every guest stays in a separate room.

• Guests can customize their food in their own room by adding available ingredients such as:

  • Pepper

  • Salt

  • Sauces

  • Spices

The chef does not cook completely different meals for every room. Instead, one common meal is prepared and each room customizes it based on individual preferences.

Relating This to Terraform Workspaces

In Terraform:

• The main Terraform code acts like the hotel kitchen.

• Each workspace acts like a separate room/environment.

• The common infrastructure code remains the same for all environments.

• Each workspace can customize values such as:

  • Instance size

  • Resource count

  • Naming conventions

  • Scaling configuration

  • Environment variables

without changing the core infrastructure code.

How to create workspace and work with that?

• To list the available workspace you use: terraform workspace list

• To Create new workspace : terraform workspace new <workspace name>

  • terraform workspace new dev

  • terraform workspace new prod

• Switch between workspace: terraform workspace select dev

In a real-world setup, Terraform Workspaces are commonly used by creating a separate workspace for each environment such as dev, test, and prod. The typical workflow looks like this:

• Create and maintain separate workspaces for each environment

• Select the required workspace before performing any operation terraform workspace select dev

• Verify that you are in the correct workspace terraform workspace show

• Run Terraform commands, which will apply the infrastructure using the variables and state associated with that workspace terraform apply

Pros:

• Reduced Code Duplication

• Easier Maintenance

• Faster Environment Creation

• Consistency Across Environments

• Simplified State Isolation

Cons:

• Shared Backend Complexity: All workspaces typically share the same backend configuration. So all the backed files are in same location so you need to create a complex rules to restrict user to access the backend statefiles.

• Risk of Human Error: Accidentally applying changes in the wrong workspace can impact production.

• Difficult CI/CD Integration at Scale: As the number of environment increases Pipeline management becomes harder, Environment-specific approvals become complex, State access management becomes difficult

• Limited Environment Customization: Workspaces work best when environments are nearly identical.

Use Workspaces When

✅ Infrastructure is mostly identical
✅ Environment differences are minimal
✅ Small-to-medium scale projects
✅ Rapid environment provisioning is needed

If that is the case how the other modular folder based approach solve the problem let's discuss that.

Modular Folder-Based Approach

The Modular Folder-Based Approach is a Terraform project structure where infrastructure is organized into:

• Reusable Modules

• Environment-specific folders

Instead of using a single Terraform configuration for all environments, each environment has its own dedicated folder and configuration while sharing reusable infrastructure modules. This approach separates:

• Reusable infrastructure logic

• Environment-specific configurations

making the infrastructure easier to scale, maintain, and manage. This is the industry wide popular approach used to manage large projects

Why Use the Modular Folder-Based Approach?

As infrastructure grows, environments usually become different from each other. For example:

• Production may require: Multi-region deployment, High Availability, Autoscaling etc which is not needed for Dev

• For Dev we can use low cost instances and minimum setup to test the code.

Managing these differences using only Terraform Workspaces can become complicated. The Modular Folder-Based Approach solves this by:

• Separating environments completely

• Reusing infrastructure components through modules

• Allowing each environment to evolve independently

Where Can This Approach Be Used?

As mentioned this is the industry wide popular approach,

• Enterprise Infrastructure

• Production-Grade Platforms

• Multi-Region Deployments

• Microservices Platforms

How we Design this approach ?

We will create a folder structure as mentioned below :

terraform/
├── modules/
│   ├── vpc/
│   ├── ec2/
│   ├── rds/
│   └── eks/
│
└── environments/
    ├── dev/
    │   ├── main.tf
    │   ├── variables.tf
    │   └── terraform.tfvars
    │
    ├── test/
    │   ├── main.tf
    │   ├── variables.tf
    │   └── terraform.tfvars
    │
    └── prod/
        ├── main.tf
        ├── variables.tf
        └── terraform.tfvars

How It Works

• Modules Directory: Contains reusable infrastructure components. Each module is written once and reused across environments.

• Environment Directory: Each environment

  • Maintains its own Terraform state

  • Has its own backend configuration

  • Uses its own variable values

  • Can independently deploy infrastructure

Pros

• One of the major advantages of the Modular Folder-Based Approach is that it helps reduce human error compared to Terraform Workspaces. In the workspace approach, users must manually switch between environments, and in real-world scenarios engineers may accidentally apply changes to the wrong workspace assuming they are already in the correct one. With the folder-based approach, each environment has its own dedicated directory such as dev, test, and prod, and users must explicitly navigate to the corresponding folder before running Terraform commands. This clear separation between environments greatly reduces the chances of accidental deployments and improves operational safety, especially for production infrastructure.

• Better Scalability: Works well with Large infrastructure, Multi-team organizations

• Reusable Infrastructure Components: Modules reduce duplication while keeping flexibility.

• Easier Customization: Each environment can use different modules, different architecture and can scale as per the need

• Better Security and Access Control

• Easier CI/CD Integration

Cons:

• More Folder Management

• Slightly More Initial Setup

• Risk of Module Over-Engineering

• Version Management Complexity

Best Practices

• Keep Modules Small and Focused

• Maintain Independent State Per Environment

• Use Versioned Modules

source = "git::https://github.com/org/vpc-module.git?ref=v1.0.0"

When to Use This Approach

Use the Modular Folder-Based Approach when:

✅ Infrastructure differs across environments
✅ Large-scale systems are involved
✅ Strong isolation is required
✅ Multiple teams manage infrastructure
✅ Enterprise-grade CI/CD pipelines exist
✅ Security boundaries are important

Conclusion

The Modular Folder-Based Approach is one of the most scalable and production-friendly ways to organize Terraform infrastructure.

It provides:

• Reusable infrastructure through modules

• Strong environment isolation

• Better scalability

• Easier customization

• Safer production deployments

While it introduces slightly more structure and setup effort initially, it becomes significantly easier to manage as infrastructure and teams grow.

At first, I was just looking for discount Like, is there some hidden setting or plan AWS offers that I’m missing?. But as I dug deeper, I got introduced to a whole new world FinOps. And honestly, this is something many of us (especially in small companies) either:

• don’t know about, or

• know, but haven’t really implemented properly.

Here’s the surprising part, when I checked many organisation are still not aware of this. When I started looking into it seriously, I realized, We could potentially save up to ~15% of our AWS bill just by enabling few things. That’s not small.

So in this blog, I’ll walk through:

• what options AWS gives us to reduce cost

• where most people miss out

• and some practical ways you can actually start saving

Let’s get started 👇

Even though there are many ways to optimize AWS costs, in this blog I’m going to focus on one of those: The Savings Plan

💡 Savings Plans

AWS really wants you to use Savings Plans. At a high level, Savings Plans are AWS’s way of saying:

👉 “Commit to using a certain amount of compute, and I’ll give you a discount.”

Simple idea. But where most people get confused is what exactly are we committing to?

🧠 What are you actually committing? You’re not committing to:

• a specific Instance

• a specific region

• or even a specific service

Instead you are committing to:

👉 A fixed hourly spend (in $/hour) for 1 or 3 years.

Think of it like telling AWS: “Hey, I’m committing to spend around $10/hour for the next year what kind of discount can you give me?”. AWS will offer you discount and as long as your usage matches that commitment you save money.

🔍 Types of Savings Plans

At the time of writing this blog, AWS offers four types of Savings Plans. But honestly, you only need to care about two of them right now because that’s where you’ll see the biggest savings (highlighted below):

• Compute Savings ✅

• EC2 Instance Savings Plan ✅

• Database Savings plan

• SageMaker Savings plan

Compute Savings

This is the most flexible option you have and honestly, this is where most teams start. Why? Because there are very few restrictions. The trade off is simple: you get slightly lower discounts compared to other plans, but a lot more freedom.

You don’t need to worry about

• instance family,

• instance type,

• or even region.

Just commit to a certain hourly spend, and AWS automatically applies the discount across your usage.

Another big advantage is coverage. It applies to a wide range of services like EC2, Fargate, Lambda, and more so you’re not locked into a single service. This makes it a great fit when your workload is unpredictable or when you're using a mix of different services across your architecture. In this options the discounts will be in the range of around 10% to 15% approx at the time of writing this blog.

Example:

Let’s say:

• Your average usage = $10/hour --> This is what you commit to AWS

• AWS Says I will give you 15% discount

So instead of paying \(10/hour, you effectively pay: 👉 \)8.5/hour

Over a month (~730 hours):

• Without Savings Plan → \(10 × 730 = \)7,300

• With Compute Plan → \(8.5 × 730 = \)6,205

💡 You save ~$1,095/month

And the best part? This applies across services like EC2, Fargate, Lambda—no restrictions.

EC2 Instance Savings Plans

As the name suggests, this plan is specifically for EC2 instances. It comes with a few more restrictions compared to Compute Savings Plans but in return, you get better discounts.

When I say “restrictions,” here’s what I mean:

• You’re tied to a specific instance family (e.g., t3, m5)

• You’re tied to a specific region

That said, AWS still gives you some flexibility where it matters. You can freely change:

• Instance size (e.g., t3.small → t3.large)

• Operating system (Linux, Windows, etc.)

• Tenancy (shared, dedicated)

So you’re not completely locked in just scoped within a boundary. In simple terms, you’re telling AWS:

“Hey, I commit to using this EC2 instance family in this region for the next year (or three) for X amount of usage—what discount can you give me?”

Because of this tighter commitment, the discounts here are higher compared to Compute Savings Plans.

Example:

Now let’s take the same scenario:

• You commit to $10/hour

• But this time for a specific EC2 family + region

• AWS gives you 40% discount (Discount rate is an example may vary in real time)

So you effectively pay: 👉 $6/hour

Over a month:

• Without Savings Plan → $7,300

• With EC2 Plan → \(6 × 730 = \)4,380

💡 You save ~$2,920/month

What happens if your usage goes above $10/hour?

So what happens when you use beyond your commitment? AWS keeps track of your usage every single hour. As long as your total eligible usage in that hour is within $10/hour, you’ll get the applicable discount.

But the moment you go beyond that: The extra usage (above $10/hour) is charged at On-Demand pricing(no discount).

👉 Example:

• You committed → $10/hour

• Actual usage → $12/hour

Then:

• First $10/hour → discounted (15%)

• Remaining $2/hour → normal On-Demand pricing

But AWS does give you flexibility in one direction: You can increase your total $/hour commitment anytime by purchasing additional Savings Plans even in the middle of your current plan. You cannot decrease or edit your existing commitment. This is applicable for both the savings plans. This gives us flexiblity to increase our commitment cost if we find out that our average usage is more than what we commit.

Now that we understand what Savings Plans are and the value they bring, the next obvious question is:

👉 How much should I commit?

Committing the right number gives you the maximum advantage.

• If you overcommit → you’ll end up paying for unused capacity

• If you undercommit → you won’t get the full discount benefit ❌

So getting this number right is key.

How to decide the right number?

AWS already gives you a built in helper for this.

👉 It’s called “Recommendations” under Savings Plans.

This tool analyzes your historical usage and suggests an optimal $/hour commitment based on your actual workload.

How to use it

• Login to your AWS account >> Go to "Billing and Cost Management" >> From Left Side panel Expand Saving Plan >> Select Recommendations

• To Run recommendations

  • Choose Compute Savings Plan (recommended for most cases) or Or EC2 Instance Savings Plan

  • Select your preferences:

    • Term → 1 year or 3 years

    • Payment option → No upfront / Partial / All upfront (Upfront will give you more discount but you can also opt for No upfront which falls on your normal billing cycle)

    • Lookback period → 7, 30, or 60 days (This is the sample peroid)

  • AWS will automatically generate a recommendation for you. And it looks something like below.

Note: The menu may vary when you check them this is taken at the time of writing the blog

• 👉 Start with 30 days lookback + 1-year term + No upfront. This tells us that if we commit US\(0.750/hour for a 1-year term you could save an average of US\)0.15/hour that is 19% savings.

Now once you conclude with the plan you can click on Add Savings plan to cart and purchase it. It will get reflected on your next billing cycle

You can also use purchase analyzer to see what is the right amount for your current environment and many other things.

Purchase Analyzer:

The Purchase Analyzer gives you a deeper view before you actually buy a Savings Plan. It helps you understand things like:

• Cost comparison → How your current spend compares with the Savings Plan

• Coverage % → How much of your current usage will be covered by the plan

• Projected savings → What you actually save with this commitment

👉 It helps you validate your decision before committing. To use this tool

• Login to your AWS account >> Go to "Billing and Cost Management" >> From Left Side panel Expand Saving Plan >> Select Purchase Analyzer

• Choose Compute Savings Plan (recommended for most cases) or Or EC2 Instance Savings Plan

• Select your preferences:

  • Term → 1 year or 3 years

  • Payment option → No upfront / Partial / All upfront (Upfront will give you more discount but you can also opt for No upfront which falls on your normal billing cycle)

  • Lookback period → 7, 30, or 60 days (This is the sample peroid)

• Click on Run Analysis

Making the final decision

With these insights, you can make a much more informed decision on choosing the right commitment.

And if you’re still unsure, you can always reach out to the AWS Support team they can help you arrive at a number that fits your usage pattern.

Conclusion

At the end of the day, just by purchasing a Savings Plan and committing to the right $/hour value, you can easily save around 10–15% on your overall bill (or even more, depending on the plan).

The best part?

👉 You don’t need to pay anything upfront
👉 You continue with your normal billing cycle
👉 You’re simply committing: “I’ll use X $/hour for the next 1 year”

This makes it a great starting point for many teams especially since Compute Savings Plans come with minimal restrictions, so you don’t have to worry about how you provision your resources.

What’s next?

If you haven’t tried this yet in your organization, it’s definitely worth exploring you might be leaving easy savings on the table 💰

Since this blog is already getting long, in the next one we’ll dive into Reserved Instances, where you can get even better savings—but with a few more restrictions.

But now comes the real question:

• What about sensitive data?

• Passwords? Tokens? SSH keys?

That’s where Kubernetes Secrets come in. In this blog, we’ll cover:

• What are Secrets and why we need them

• How to create Secrets

• How Pods consume them

• Lifecycle & behavior

• Security best practices (this is where most people mess up 😅)

🤔 What are Kubernetes Secrets?

A Secret is a Kubernetes object used to store sensitive data like: Passwords , API tokens, SSH keys , Certificates.

Instead of hardcoding these inside your app or YAML, you decouple them using Secrets.

⚙️ How are they different from ConfigMaps?

They look similar to ConfigMaps however they have some distinct behaviour

• Stored in etcd (cluster DB)

• Mounted as tmpfs (RAM) → never written to disk 🧠

• Base64 encoded (⚠️ not encryption!)

💡 Important: Base64 ≠ Security. Anyone with access can decode it easily.

😌 Why do we need Secrets?

You already know the answer, but let’s reinforce:

• Keeps sensitive data out of code

• Helps follow 12-factor app principles

• Enables secure configuration per environment

You can think of it like this

• ConfigMap = app config

• Secret = app credentials

🛠️ How to Create Secrets

You can create Secrets in two ways:

• Inline Command

• Using YAML

Let's explore them in detail

⚡ 1. Inline Command (Quick & Dirty)

Example:

# Basic syntax
kubectl create secret <type> <name> --from-literal=<key>=<value>

# Example
kubectl create secret generic api-credentials \
  --from-literal=token=super-secret-123

Multi-value example:

kubectl create secret docker-registry private-reg-auth \
  --docker-username=my-user \
  --docker-password=my-password \
  --docker-server=my-registry.io \
  --docker-email=my-email@example.com

When to Choose this approach ?

Use this for:

• Quick testing

• PoCs

Note: Here you can directly pass the values without encoding, Kubernetes encode the value and store it in Etcd

📄 2. Using YAML (Production Way)

apiVersion: v1
kind: Secret
metadata:
  name: my-db-secret
type: Opaque
data:
  username: YWRtaW4=
  password: MWYyZDNoNGo1aw==

Here you must encode values manually using echo -n "admin" | base64

🧩 Secret Types (Quick Understanding)

💡 Tip: Use specific types when possible — Kubernetes validates structure.

🔌 How Pods Consume Secrets

Two main ways (same as ConfigMaps 👀):

• Environment Variables

• Volume Mounts

🛡️ Option 1: Environment Variables

This option is suitbale for for API keys or database passwords that the application expects to find in its environment. This is simple and clean approach

Example

apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
spec:
  containers:
    - name: my-app
      image: nginx
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-creds
              key: password

⚠️ Cons:

• Values are loaded only at startup

• If Secret changes → Pod restart required 🔄

📁 Option 2: Volume Mounts (Recommended)

This option is best for certificates, SSH keys, or large configuration files. Kubernetes mounts the secret as a directory where each key in the secret becomes a file.

Example:

apiVersion: v1
kind: Pod
metadata:
  name: secret-volume-pod
spec:
    containers:
    - name: ssh-container
      image: nginx 
      volumeMounts:
        - name: ssh-key-vol 
          mountPath: "/etc/ssh-keys" 
          readOnly: true 
volumes: 
    - name: ssh-key-vol 
      secret: 
        secretName: ssh-key-secret # The name of the Secret object

👉 Each key becomes a file:

/etc/ssh-keys/
  ├── private_key
  ├── public_key

💡 Why use Volumes over Environment Variables?

• Security: Secret volumes are stored in tmpfs (RAM). If the node is powered down, the data vanishes from the node's memory. 🧠

• Updates: If you update a Secret, Kubernetes automatically updates the files in the volume (though this takes a few seconds). Environment variables, however, are static—you would have to restart the Pod to see a change. 🔄

• Structure: Volumes are perfect for things that naturally exist as files, like SSL certificates or SSH id_rsa keys.

📁 Standard Mount vs. subPath

🔹 Standard Mount (Default Behavior)

When you mount a Secret as a volume:

• The entire target directory gets replaced

• Any existing files become invisible

• Only the mounted Secret content is available for your pod to use.

👉 If you mount another Secret to the same path, it will overwrite the previous one

This behavior is intentional in Kubernetes to Ensure data isolation, Improve security, Avoid unintended file conflicts

Now consider a scenario where you want to mount multiple Secrets into the same Pod path.

Using the standard approach:

• Mounting Secret A → works fine

• Mounting Secret B to the same path → overwrites Secret A

👉 Result: The first Secret becomes inaccessible.

To solve this, Kubernetes provides a feature called subPath.

💡 What is subPath?

subPath allows you to:

• Mount individual files or subdirectories

• Place them inside an existing directory

• Avoid overriding the entire volume path

To know in detail about subpath check here

✅ Using subPath for Multiple Secrets

With subPath, you can mount multiple Secrets into the same directory without conflicts.

Example:

volumeMounts:
  - name: secret-a
    mountPath: /etc/config/secret-a.txt
    subPath: secret-a.txt
  - name: secret-b
    mountPath: /etc/config/secret-b.txt
    subPath: secret-b.txt
volumes:
  - name: secret-a
    secret:
      secretName: secret-a
  - name: secret-b
    secret:
      secretName: secret-b

🔍 What happens here?

• secret-a is mounted as /etc/config/secret-a.txt

• secret-b is mounted as /etc/config/secret-b.txt

• No overwriting occurs

• Both Secrets coexist in the same directory 🎉

One thing to note here is that when using subPath we have to define path with filename and extension whereas in standardMount the mount path is sufficient enough.

LifeCycle for subPath
One more important thing is that when using subPath when the secret value is updated it will not update the file directly pod restart is required because here the file is a static bind-mount

🔄 Lifecycle of Secrets

In this section we will explore how changing a secret value will be handled in different Consumption option

Even though VolumeMount Auto Updates your secrets to the Mount file you app should be configured to pick the updated value.

Security Best Practices & RBAC

Now that we know how to create and consume Secrets, Let's talk about how to actually keep them "secret."

Even though Secrets are better than plain-text ConfigMaps, they have some vulnerabilities by default:

• Base64 is not encryption: It's just a way to store binary data as text. Anyone who can run kubectl get secret -o yaml can see your passwords. 🔓

• Etcd Storage: By default, Secrets are stored in plain text in the cluster's database (etcd).

Some of the best practices

Encryption at Rest (The Native Way)

You can configure the Kubernetes API server with an EncryptionConfiguration file. When this is enabled, the API server will mathematically encrypt the Secret data before it saves it to etcd. Even if someone steals the etcd hard drive, the data is unreadable garbage without the encryption key.

Strict RBAC (Role-Based Access Control)

As we discussed briefly before, you must restrict who can run kubectl get secrets. If a developer has the get or list permission for Secrets, they can just ask the API server to show them the Base64 data, bypassing all etcd encryption. You should only grant Secret access to the specific ServiceAccounts that the Pods use, not to human users.

To truly secure a Secret while still letting it be used:

• For Users (Developers): You generally want to deny get, list, and watch. If they can't get it, they can't see the password.

• For Pods: The Pod doesn't actually need the user to have permissions. The Pod uses a ServiceAccount. The Kubernetes controller-manager handles the mounting process, so as long as the Pod's YAML is allowed to be created, the system handles the rest.

External Secret Stores (The Enterprise Way)

In large companies, they don't even store the source-of-truth secrets in Kubernetes at all! They use tools like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault. They use plugins (like the Secrets Store CSI Driver or External Secrets Operator) to fetch the password from the cloud vault and inject it directly into the Pod at runtime.

Conclusion

Kubernetes Secrets might look simple at first, but how you use them makes all the difference. They help you keep sensitive data out of your code and configs—but they are not secure by default.

If you use them the right way prefer volumes over env vars, restrict access with RBAC, and integrate with external secret managers—you move from just “working setup” to a production-ready secure system 🔐

Thanks for reading. Keep learning, keep building, and keep breaking things (that’s how we grow) . See you in the next one ✌️

But let’s pause and think for a moment 🤔

• What if we just want to pass configuration values to a Pod? For example:

  • Database URL

  • Application mode (dev / prod)

  • Feature flags

  • External service endpoints

We don’t want to hardcode these values inside the container image. Because configuration changes frequently, but images should remain stable. So how do we handle this in Kubernetes? 👉 This is exactly where ConfigMaps come into the picture.

In this blog we will explore

• What is ConfigMaps

• How and Where to use it

• Creation Methods: Understanding the various ways to build ConfigMaps using kubectl literals, files, or YAML manifests.

• Consumption Strategies: Discovering how Pods actually "read" this data, whether through environment variables or by mounting them as files.

• Lifecycle & Updates: Exploring what happens when configuration changes and how to handle updates without breaking your services.

What is a ConfigMap? 📦

A ConfigMap is a Kubernetes Object is used to store non-sensitive configuration data as key-value pairs. It allows you to:

• Decouple configuration from container images. This means you can use the exact same image in development, staging, and production, simply by swapping out the ConfigMap associated with it.

• Pass environment variables to pods

• Provide configuration files to containers

• Update configs without rebuilding images

In simple words:

ConfigMap = A way to inject configuration into your Pod dynamically.

How ConfigMap Works 🔍

A ConfigMap stores configuration as:

• Key-value pairs

• Entire configuration files (like an nginx.conf or settings.py)

• Environment Variable

Then Kubernetes allows you to: Inject them as environment variables or Mount them as files inside a container.

Remember that Configmaps are not intended for sensitive information like passwords or API keys—for those, we use Secrets

Complete Internal Flow Summary:

• kubectl sends ConfigMap definition to API Server

• API Server validates and stores ConfigMap in etcd

• ConfigMap now exists as a cluster object

• Pod referencing ConfigMap is scheduled to a node

• kubelet detects Pod and fetches ConfigMap

• kubelet injects ConfigMap into container (env or volume)

• Container starts

• Application consumes configuration

ConfigMap - Creation Methods

Method 1: Literal Key-Value Pairs 🔑

The first method is using kubectl command and creating a config map using literals directly from terminal

# Syntax
kubectl create configmap <map-name> --from-literal=<key>=<value> 

# Example
kubectl create configmap app-settings --from-literal=ui_color=blue --from-literal=debug_mode=true

When to use this method:

You should use Literal Key-Value Pairs when quick testing/debugging is needed. If you want to:

• Quickly toggle a feature flag

• Change a log level

• Test a temporary configuration

• Small, Simple Configuration

While literals are great for quick flags, real-world applications often have dozens of settings stored in configuration files (like .properties, .env, or .yaml). Kubernetes allows you to take an entire file and shove it into a ConfigMap in one go.

Method 2: From a File 📄

In this method instead of passing individual Key Value pairs we can pass the whole file as an input. This is very useful when we have a need to pass a config file to the Container image.

When you use the --from-file flag, Kubernetes uses the filename as the key and the file contents as the value.

Imagine you have a file named connection.conf with this content:

# Filename: connection.conf

db_host=prod-db.example.com
db_port=5432

To configure this file in Config map we will use below command

kubectl create configmap db-config --from-file=connection.conf

Inside the ConfigMap, it looks like this:

• Key: connection.conf

• Value: (The entire multiline string of the file)

When to use this Method:

This is similar to the above literals the only difference is that here we can pass the whole file. This method is also useful for short term testing or for temporary change

Method 3: Declarative YAML 📝

This is the widely used industry standard method for declaring ConfigMaps. All the ConfigMaps are written in a YAML file in declarative way and then we apply it. This gives us an option to maintain desired state and also help us to follow GitOps. This allows us to track changes in Git (Infrastructure as Code).

A ConfigMap in YAML looks like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: test-config
data:
  # Simple key-value pair
  log_level: "INFO"
  DB_TYPE: "postgres"
  # Multiline configuration file
  nginx.conf: |
    server {
      listen 80;
      server_name localhost;
    }

To apply this to our K8 cluster we use below default apply command

kubectl apply -f <filename>

When to use this Method:

This is the most commonly used best practice method. Apart from short term testing it is recommended to use Declarative YAML method while creationg configmaps.

We understood the methods of creating config maps now lets explore how we can consume it.

ConfigMap - Consumption Strategies

Whenever you create a configmap using any of the above method the configmaps are stored in the cluster now your application needs to read it. There are two methods to consume the configmaps

• Environment Variables: Good for individual settings (like DB_TYPE).

• Volume Mounts: Good for configuration files (like nginx.conf).

Method 1: Environment Variables:

In this method we will fecth the ConfigMap and use it as an Environment variable in the pod.

Let's take the below Pod Definition example

apiVersion: v1
kind: Pod
metadata:
  name: sample-app-pod
spec:
  containers:
    - name: sample-container
      image: nginx:latest
      env:
        - name: DATABASE_KIND   # Environment variable inside container
          valueFrom:
            configMapKeyRef:
              name: test-config   # ConfigMap name
              key: DB_TYPE         # Key inside ConfigMap

When to use this Method:

When your config maps is just are just Key Value pair and your applications can consume it via environment variable you can use this method.

Method 2: Volume Mounts:

In this method we will mount the ConfigMaps as a volume. So the keys in the configMap will be file name in the Volume and value will be content inside the file.

Example:

# weather-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: weather-config
data:
  city: "Chennai"
  temperature-unit: "Celsius"
  app.properties: |
    forecast.days=5
    refresh.interval=30

# weather-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: weather-app-pod
spec:
  containers:
    - name: my-container
      image: nginx:latest
      volumeMounts:
        - name: config-dir              # Must match volume name below
          mountPath: /etc/config        # Files appear here inside container
          readOnly: true
  volumes:
    - name: config-dir
      configMap:
        name: weather-config            # ConfigMap name

How the config maps are stored in this method ?

Kubernetes will read the weather-config ConfigMap, convert each key into a file, mount those files inside /etc/config

Inside your container you will be seing the Config map as file like this : /etc/config/city , /etc/config/temperature-unit

Everything is a "File" in a Volume:

[CREATION]: Whether you create a ConfigMap from a literal (--from-literal) or a file (--from-file) or using manifest files, Kubernetes stores them the exact same way key-value pairs

[CONSUMPTION]: But, when you mount a ConfigMap as a volume, Kubernetes treats every key as a filename and every value as the content of that file. It doesn't care if the original source was a literal string or a 500-line config file.

Remember the difference between creating a config maps(which we discussed in previous section) and consuming the configmaps(which we are discussing now). This above point is when you consume configMaps as volume

There is a small catch here 👀. Whenever you mount a ConfigMap as a volume, Kubernetes mounts it on the specified path inside the container.

👉 But here is the important part:

When a volume is mounted to a path, any existing content in that path will be hidden. Only the data from the latest mounted volume will be visible.

🔎 Let’s Understand With Example

Assume that you have two configMaps named xyz-config and weather-config . If you want to mount both in the same path mountPath: /etc/config, then

• The second mount weather-config will override the first one

• Only the latest mounted config map will be visible

• Other files in that path become unavailable to the pod

🤔 Why Is It Like This?

This is Kubernetes’ default behavior. When you mount a volume to a path,

• Kubernetes overlays that volume on that directory. The original content is hidden. This ensures clean and predictable file mapping. Also improves isolation and avoids accidental file mixing (security + consistency reasons).

So what is the solution if I want to mount all configMaps to the specific Path Or map my ConfigMap to a specific path and also use other contents from the path. This is where a new parameter comes in called subPath

subPath

If you want to mount multiple ConfigMaps to the same directory without hiding each other, you can use subPath

👉 what subPath tells Kubernetes:

Mount this specific file inside the directory, not override the entire directory.

Example - without subPath

volumeMounts:
  - name: xyz-volume
    mountPath: /etc/config   
  - name: weather-volume
    mountPath: /etc/config # This will override the previous one ❌
  

Example - with subPath

apiVersion: v1
kind: Pod
metadata:
  name: multi-config-pod
spec:
  containers:
    - name: my-container
      image: nginx:latest
      volumeMounts:
        - name: weather-volume
          mountPath: /etc/config/weather.properties
          subPath: weather.properties
          readOnly: true

        - name: xyz-volume
          mountPath: /etc/config/xyz.properties
          subPath: xyz.properties
          readOnly: true

  volumes:
    - name: weather-volume
      configMap:
        name: weather-config

    - name: xyz-volume
      configMap:
        name: xyz-config

🧠 Important Note

• When using normal mount you only specify directory path. mountPath: /etc/config . This is because by default, when you mount a volume to a path, Kubernetes creates a directory at that path. If you mount it exactly at /app/static/theme.css, that path will become a directory, and your file would actually live at /app/static/theme.css/theme.css. This is why when you are not using subpath you have to define only the path without filename

• But when using subPath, You must specify the complete file path including filename. mountPath: /etc/config/weather.properties. This is because now you are mounting a specific file, not the whole directory.

ConfigMap - Lifecycle & Updates 🔄

In this section, let’s understand something very practical:

👉 What happens when you update a ConfigMap?
👉 How does Kubernetes handle those updates?
👉 Does your application automatically see the changes?

The behavior actually depends on how you consume the ConfigMap. There are two main consumption strategies which we discussed, as Environment Variables and as Volume Mounts. Let's see what happens on both clearly

🧪 Lifecycle of ConfigMap as Environment Variable

When you consume a ConfigMap as an environment variable, and later you update the ConfigMap, The running Pods will NOT see the changes.

🤔 Why is this? - Because, ConfigMap values are injected into the container only at startup. Once the container starts Environment variables are set. They become part of the container’s runtime. Kubernetes does NOT re-inject updated values.

So even you edit your config map using kubectl edit configmap my-config the values of ConfigMaps will be updated but your pods will not use the updated values.

✅ So What Should You Do? You must restart the Pod. There are multiple ways to do it

• Delete the Pod (ReplicaSet will recreate it)

• Trigger a Deployment rollout

Once the container restarts, the updated ConfigMap values will be injected.

📂 Lifecycle of ConfigMap as Volume Mount

Now this is interesting 👀

When you consume a ConfigMap as a Volume Mount, updates behave differently. When you update the ConfigMap, Kubernetes automatically updates the files inside the mounted volume. Yes — without restarting the Pod.

🤔 But there is a Catch? - The catch is in your application design.

Let's take a example, Your app reads /etc/config/app.properties , It reads it only once during startup. Now you update the ConfigMap. The file inside the container gets updated. But your app still uses old values. Because your application is not re-reading the file.

🧠 What Should Be Done? - If you consume ConfigMap via Volume, Your application should Watch the file for changes or periodically reload configuration or support dynamic configuration refresh Otherwise, the update exists at the file level — but not at the application level.

🛡️ Immutable ConfigMaps

Now let’s talk about something advanced and powerful.

Imagine, You have 300 Pods all are consuming the same ConfigMap via Volume Mount. They are watching for changes.

Whenever you update that ConfigMap, Kubernetes API server must notify all watchers, the control plane processes multiple update events, this increases API server load. It may cause unexpected behavior if files update mid-process

Now think practically, What if this configuration NEVER changes or changes very rarely ?

In such cases, Kubernetes gives you something called Immutable ConfigMaps

You can mark a ConfigMap as immutable. This tells Kubernetes that the values in the configMaps will not change. So that Kubernetes will not watch these configMaps.

Example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: static-config
immutable: true # This is the magic
data:
  api_key: "12345"

✅ Benefits of Immutability

• Safety: Prevents accidental configuration changes that could break production.

• Performance: Kubernetes stops "watching" these files, reducing the load on the API server.

❓ But What If I Need to Change an Immutable ConfigMap?

Good question 😉. Once a ConfigMap is marked as immutable, you CANNOT Modify the data, Remove the immutable flag, Edit it directly, Kubernetes strictly prevents this.

You could delete and recreate it, but that is risky. If a Pod restarts during the split second where the ConfigMap doesn’t exist Pod will fail to start.

The Expert Way: Versioning (Rolling Forward Pattern):

Instead of editing the old ConfigMap, create a new one

• Create New: You create a new ConfigMap named static-config-v2 with your updated database URL.

• Update Deployment: You edit your Deployment manifest to point to this new name.

• Rollout: Kubernetes sees the Deployment change and automatically performs a rolling update, creating new Pods with the new config and terminating the old ones safely.

At this point, you might have one more doubt. If I’m using versioned ConfigMaps and I have 100 applications, do I need to manually update all 100 Deployment files every time the config changes?

Yes… technically you could. But that would be Painful 😅, Error Prone, Not Scalable. This is where tools like Kustomize come into play.

⚙️ How Kustomize Solves This Problem

Kustomize helps you manage Kubernetes configurations in a smarter way. When you use Kustomize to generate ConfigMaps:

• It automatically hashes the content.

• It appends that hash to the ConfigMap name.

Example:

• Instead of static-config it creates hashed value something like this static-config-m9t8f5c92k

We will explore Kustomize deeply in another blog (because that itself deserves a full discussion 😄).

📋 Putting It All Together: The Expert Checklist

Now we have explored ConfigMaps in detail:

• What they are

• How they are consumed

• Update behavior

• Immutable pattern

Let’s finish with a simple, practical checklist. Because in production, the real question is: When should I use what?

Common Doubts

If My Config Never Changes, Why Use ConfigMap At All why do we need Immutable?

At first glance, this seems logical as I also had the same doubt. If it never changes, why not just hardcode it inside the application?

But here’s why ConfigMaps still matter.

Separation:

Even if configuration never changes, we still separate Application Code and Configuration, why because Same image can run in Dev, QA, Prod. Only configuration changes between environments. ou don’t rebuild image for every small config variation

So Even if production config rarely changes, it is still environment-specific. That separation is the core design principle.

Image Immutability Principle:

In Kubernetes world, Container images should be immutable. If we bake config inside the image any changes to the config maps will force us to rebuild the image which increment the image version for small change.

practical list of common ConfigMap-related errors

ConfigMap Not Found Errors:

Error: configmap "<Configmap name>" not found

Why it happens: Typo in name, Wrong namespace, ConfigMap not created before Pod, Deleted accidentally

Fix: Verify namespace: kubectl get configmap -n <ns> , Ensure order: ConfigMap must exist before Pod starts

Wrong Namespace Issue:

Why it happens: Very common mistake. ConfigMap is created in default namespace as we forget to mention the namespace

Fix: Create ConfigMap in the same namespace as Pod.

Key Not Found Error

Error: couldn't find key DB_HOST in ConfigMap

Why it happens: Wrong key name, Case mismatch (db_host vs DB_HOST), Key deleted during update

Fix: Make sure you are using same KEY across the namespace. Validate it with Configmap when you are consuming it

Pod Stuck in CreateContainerConfigError

Error: CreateContainerConfigError

Why it happens: Missing ConfigMap, Missing key, Invalid reference. This usually means Kubernetes cannot inject the ConfigMap.

Fix: Validate the configmap in correct namespace and the names are matched across the deployments. Validate Configmap is created or not

Environment Variable Not Updating

Error: You update ConfigMap. But app still shows old value.

Why it happens: Env variables are injected only at container startup.

Fix: Restart Pod - kubectl rollout restart deployment app

Volume Mount Not Updating Immediately

Error: You update ConfigMap. File inside container still shows old content for some time.

Why it happens: Update propagation delay, Kubelet sync period (usually ~1 min)

Fix: This is normal behavior. No fix needed

Application Not Picking Updated File

Error: File updated but app still behaves same.

Why it happens: Application reads config only at startup. Kubernetes updated file but app never reloaded it.

Fix: This is an application design issue, not Kubernetes issue. Need to design app to check config maps periodically or take changes whenever happens

subPath Does NOT Auto-Refresh

Error: If we use subPath then configMap does not auto refresh

Why it happens: Because subPath mounts a single file copy not the live symlink.

Accidentally Overriding Directory

Error: When you mount configmap as Volume it hides other files in the volume

Why it happens: This is the default behaviour of Kubernetes to have clean mount and saftey.

Fix: If you want to mount the configMap where you also needs to use other files from volume use subPath to mount configmap.

Large ConfigMap Size Limit

Error: etcd request too large

Why it happens: ConfigMap size limit: ~1MB.

Fix: Makesure you manage size of configmap in right way. ConfigMaps are not meant for Huge Binaries or Large Json datasets

Watch Load on API Server

Error: API server getting overloaded

Why it happens: Hundreds of Pods watching a frequently updated ConfigMap: High API server load, Performance degradation

Fix: Use immutable configmap for static configs

🧠 Production-Level Issues (Advanced)

These are not syntax errors but architectural mistakes:

• Using ConfigMap for dynamic runtime feature toggles (bad design)

• Updating ConfigMap frequently in high-scale clusters

• Not versioning production config

• Editing ConfigMap directly in production (breaks GitOps)

• Using subPath when expecting dynamic updates

Conclusion

ConfigMaps are a simple, powerful Kubernetes primitive for decoupling non-sensitive configuration from container images. They let you store key-value configuration outside of your images and supply those values to Pods either as environment variables or as mounted files — giving you flexibility to use the same image across environments and change configuration independently.

Key takeaways:

• Use ConfigMaps for non-sensitive, environment-specific values (use Secrets for sensitive data).

• Create ConfigMaps via kubectl literals, files, or declarative YAML manifests depending on your workflow and need for version control.

• Consume ConfigMaps as environment variables for small, simple values or mount them as files when applications expect file-based configs.

• Be mindful of update behavior: mounted files are updated automatically (with caveats), while env-based values require Pod restart; manage updates via rollout restarts, immutable ConfigMaps, or reloader sidecars/controllers when appropriate.

• Follow best practices: keep configs declarative (GitOps), name and label ConfigMaps clearly, avoid embedding large binaries, and consider using immutable ConfigMaps for stable deployments.

If you’ve worked with Kubernetes volumes, you’ve probably seen this statement:

“EBS (or any block storage) only supports RWO (ReadWriteOnce).
For RWX (ReadWriteMany), you need something like EFS.”

If you’ve ever wondered why that is, you’re not alone. I had the same question when I first learned it. Let’s break it down properly.

Before the Crux — Let’s Get the Basics Right

What Is Block Storage?

Block storage is essentially a raw disk. When you create an EBS volume, Kubernetes (or the operating system) sees it as:

“Here’s an empty hard drive. You decide how to use it.”

Since it’s raw, it must be formatted with a filesystem (like ext4 or xfs) before use.

Formatting what is it ?

Formatting means creating a filesystem structure on the disk. Think of it like this:

• You have a blank notebook (raw disk)

• You draw structured grids inside it. Think of it like your Microsoft Excel

• Now you can store and retrieve values in specific locations by mapping the Row and Column.

In the same way the disk gets divided into blocks, and the filesystem keeps track of:

• Which block belongs to which file

• Where data is stored

• How to retrieve it quickly

Do you know the Block Storage have Fast I/O. You know why ?'

Because block storage allows direct access to data blocks. When an application wants data, the operating system can say: “Give me block #504.” it retrieve it directly from the Block storage Disk nothing between the storage and the Operating System.

That’s why block storage is:

• Low latency

• High performance

• Ideal for databases

Then Why Can’t Block Storage Support RWX?

A traditional filesystem (like ext4 or xfs) is not cluster-aware. If you attach the same block disk to multiple nodes and both try to write:

• Filesystem corruption can occur

• There’s no built-in distributed locking (So that when one node is using it it locks other node)

• Metadata can conflict (If some node tries to write and other tries to modify the metadata conflict occurs.)

  • Metadata is nothing but the information about the data stored such as when it is created what is the size etc.

Block storage assumes: “One disk → one machine managing the filesystem.”

That’s why:

• EBS supports ReadWriteOnce (RWO)

• It cannot safely support ReadWriteMany (RWX)

What Is File Storage?

File storage (like EFS) is different. It is

• Already formatted

• Managed by a file server

• Exposed over the network (NFS in EFS case)

Instead of giving you raw blocks, it gives you A shared file system accessible over the network.

You may ask How Is File Storage Different ?

Even file storage uses blocks internally but the key difference is:

• A central file server manages metadata

• It handles locking

• It coordinates concurrent access

• It ensures consistency

Every file contains metadata like:

• Permissions

• Ownership

• Size

• Timestamps

• Access rules

What happens when multiple nodes try to connect ?

• They don’t manage blocks directly

• They talk to the file server

• The file server manages read/write coordination

That’s what enables RWX.

Is File Storage Slower?

Generally, yes — compared to block storage. Why?

Because:

• It involves network communication

• Metadata validation happens

• There’s coordination overhead

But the trade-off is: You get safe, shared, multi-node access.

The Real Crux

Block storage:

• Attached to one node

• Filesystem managed locally

• No distributed locking

• High performance

• Supports RWO

File storage:

• Managed by centralized file servers

• Supports distributed locking

• Safe concurrent access

• Supports RWX

• Slightly higher latency

The Mental Model

Think of it like this:

Block Storage (EBS)

Like a USB drive plugged into one laptop. Only that laptop controls it.

File Storage (EFS)

Like a shared Google Drive folder. Multiple machines can access it at the same time.

That "node-lock" is the biggest weakness of hostPath. If the node goes down or the Pod moves, the data is essentially "trapped" on the old node.

To solve this, Kubernetes uses a system that decouples the storage from the node entirely, often using network storage (like a cloud disk). This brings us to the "Gold Standard" of Kubernetes storage: PersistentVolumes (PV) and PersistentVolumeClaims (PVC).

Persistent Volumes(PV) and Persistent Volumes Claim (PVC)

The What ?

Persistent Volumes (PV):

A Persistent Volume (PV) represents a piece of storage provisioned in the cluster often backed by network or cloud storage that continues to exist even if Pods are deleted or moved.

You can think of a PV as block storage or file storage attached to the Kubernetes cluster, where the cluster has full control to allocate this storage to incoming requests.

Since this storage lives outside the Kubernetes cluster environment (for example, cloud disks or network-attached storage), it is not affected by Pod restarts, node failures, or most Kubernetes-level failures.

Persistent Volumes Claim (PVC):

A Persistent Volume Claim (PVC) is a request for storage made by an application running inside the Kubernetes cluster.

Instead of directly attaching a storage disk, a Pod asks for storage by specifying:

• How much space it needs

• The type of access required (read/write)

• The storage characteristics it expects

Kubernetes then looks for a suitable Persistent Volume (PV) that matches this request and binds it to the PVC.

You can think of a PVC as an interface between your application and the actual storage, allowing Pods to use persistent storage without needing to know where or how it is physically provisioned.

❓ Why PVC request is needed ?

PVCs introduce a clean separation between applications and storage.

Different applications have very different storage needs:

• Databases need high I/O and low latency

• Logs or backups can work with lower-performance storage

Instead of hard-coding storage details into applications, a PVC allows the app to simply say: “I need this much storage, with this access pattern.” Kubernetes then fulfills that request by binding the PVC to an appropriate PV.

This approach keeps applications portable, flexible, and storage-agnostic, while allowing storage performance to be controlled through PVC requests.

Here is a quick look at how they connect:

• Pod points to → PVC

• PVC binds to → PV

• PV points to → Actual Disk

Each layer has a clear responsibility, and none of them need to know the internal details of the others.

🔐 Access Modes

Both Persistent Volumes (PV) and Persistent Volume Claims (PVC) define something called access modes. Access modes describe how a volume can be accessed by Pods for example, whether it can be mounted as read-only, read-write, or shared across multiple nodes. The available access modes are explained below

• ReadWriteOnce (RWO): The volume can be mounted as read-write by a single node. (Like a USB drive plugged into one laptop).

• ReadOnlyMany (ROX): The volume can be mounted read-only by many nodes. (Like a shared CD-ROM).

• ReadWriteMany (RWX): The volume can be mounted as read-write by many nodes. (Like a shared network folder/Dropbox).

When you provision PV and PVC for a highly available application that runs across multiple nodes, choosing the right access mode becomes critical. For example, If the volume is read-only and shared across Pods across nodes, you can use ReadOnlyMany (ROX), If the volume needs to be read and written by multiple Pods across nodes, you must use ReadWriteMany (RWX)

What does “Many” mean here?

“Many” refers to the number of nodes, not Pods.

• RWO → Only one node can read/write to the volume at a time. i.e all the pods in the node can access this volume

• RWX → Multiple nodes can read/write to the same volume simultaneously. i.e pods across nodes can access the volume

An important detail to note here is a Persistent Volume (PV) can be configured with multiple access modes, indicating what it supports. A Persistent Volume Claim (PVC), on the other hand, requests a specific access mode based on the application’s requirement. If the requested access mode in the PVC is supported by the PV, Kubernetes approves the claim and binds them together.

🧠 Key Takeaway

PV advertises what it can support, and PVC asks for what it needs. Binding happens only when both agree on the access mode.

Example Kubernetes Resource Definition File

Persistent Volume (PV)

apiVersion: v1
kind: PersistentVolume
metadata:
  name: app-pv               # Name of the PV
spec:
  capacity:
    storage: 10Gi            # Size of the volume
  accessModes:
    - ReadWriteOnce          # Supported access mode
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  # Retain data even if PVC is deleted
  storageClassName: ebs-sc   # Must match PVC
  csi:
    driver: ebs.csi.aws.com  # AWS EBS CSI driver
    volumeHandle: vol-0abcd1234efgh5678
    # Actual EBS volume ID from AWS
  volumeMode: Filesystem    # Mount as a filesystem

Persistent Volume (PV)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-pvc              # Name of the PVC
spec:
  accessModes:
    - ReadWriteOnce          # Only one node can read/write
  resources:
    requests:
      storage: 10Gi          # Amount of storage requested
  storageClassName: ebs-sc   # Use this StorageClass

Consuming a PVC Inside a Pod

apiVersion: v1
kind: Pod
metadata:
  name: app-pod                 # Name of the Pod
spec:
  containers:
    - name: app-container
      image: nginx              # Sample container image
      volumeMounts:
        - name: app-storage
          mountPath: /data      # Path inside the container
          # The PVC will be mounted here
  volumes:
    - name: app-storage
      persistentVolumeClaim:
        claimName: app-pvc      # Name of the PVC to use

StorageClasses: Leveling Up to Dynamic Provisioning

In the real world the admin don’t want to worry about creating 100’s of PV’s(Persistent Volumes) with different types and sizes. Here is where StorageClasses (SC) comes in

How it works

• The administrator creates a StorageClass in the cluster

• The StorageClass is granted permission to create cloud provider disks (such as AWS EBS or Azure Disk) by installing the appropriate CSI driver in the cluster.

  • For example, in an Amazon EKS cluster, you install the EBS CSI driver, which enables Kubernetes to dynamically provision EBS volumes on demand whenever a PVC is created.

• The PVC specifies which StorageClass to use

• When the PVC is created, Kubernetes checks:

  • Is there an existing PV that matches this request?

• If no matching PV exists, Kubernetes uses the StorageClass to dynamically create a new PV.

• The newly created PV is bound to the PVC.

• When a Pod references the PVC, the volume is attached and mounted into the Pod.

For example:

• A Pod requests 50Gi

• No existing PV matches

• The StorageClass dynamically creates a new disk (EBS etc.)

• The disk is attached to the Pod as a volume

No manual PV creation needed 🎉

Reclaim Policies

Now comes an important question. Your application stores data in a PV via a PVC and that data can live forever.

But what if:

• You delete the application?

• You no longer need the data?

• You recreate the app with fresh data?

Continuously creating new volumes is not sustainable, especially when storage costs add up. This is where Reclaim Policies come into play.

Reclaim policies define what happens to a Persistent Volume after the PVC is deleted whether the data should be:

• Retained: The PV is not deleted. It stays in "Released” state and will not be assigned to any new PVC request. An admin must manually clean up the data. This is the safest for production databases.

• Deleted: The PV and the actual physical disk (AWS EBS, GCE PD) are deleted immediately. Should be choosen with extra caution as the data will be wiped once PVC is deleted

• Recycle(Deprecated): It performs a basic rm -rf /thevolume/* and makes the PV available again. (Rarely used now).

You control this with the persistentVolumeReclaimPolicy

The Binding Process

Kubernetes matches a PVC to a PV using these criteria:

1. Capacity: Does the PV have at least the amount requested?

2. Access Mode: Does the PV support the mode (RWO, RWX, etc.)?

3. StorageClass: Do they have the same StorageClass name?

4. Selectors: (Optional) Do the labels match?

Example Kubernetes Resource Definition File

StorageClass (Dynamic Provisioning)

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc               # Name of the StorageClass
provisioner: ebs.csi.aws.com # CSI driver that talks to AWS EBS
parameters:
  type: gp3                  # EBS volume type (gp2 / gp3 / io1, etc.)
reclaimPolicy: Delete        # What happens to the disk when PVC is deleted
volumeBindingMode: WaitForFirstConsumer
# Volume is created only when a Pod actually uses the PVC

Common Scenarios

The “Same Name” Scenario (ReclaimPolicy: Retain)

Assume you have an application using a Persistent Volume (PV) named pv-x, and the reclaim policy is set to Retain. You deploy the app, it creates a PVC, and everything works fine. After some experimentation, you delete the application (which deletes the PVC).

Later, you:

• Create a new application with a different PVC name, but want to use the same data

• Or simply rename the PVC for the same application

So… what happens?

When the PVC is deleted and the reclaim policy is Retain:

• The PV is NOT deleted. The underlying disk is preserved

• The PV moves into a Released state. The PV still remembers the old PVC it was bound to

This is the important part 👇

A PV in the Released state cannot be automatically reused by a new PVC—even if:

• The storage size matches

• The access modes match

• The StorageClass matches

Kubernetes intentionally blocks this to prevent accidental data leaks between workloads.

❓ Why Can’t a New PVC Reuse the Same PV Automatically?

Because the PV still contains Old application data and Metadata pointing to the previous PVC. Kubernetes assumes this data belongs to someone else. I won’t reattach it unless a human explicitly says so. That’s a safety feature, not a limitation.

So How Can You Recover the Data?

There are two correct and safe ways to recover data from a retained PV.

✅ Option 1: Manually Rebind the Existing PV (Most Common)

If you want the same data to be used by a new or renamed PVC:

• Manually edit the PV and remove the old claimRef

• Set the PV back to Available

  • Create a new PVC that matches: StorageSize, AccessMode, and Storage class

• Once this is done, Kubernetes will bind the new PVC to the old PV—and your data is recovered intact.

This is the intended recovery flow for Retain.

✅ Option 2: Create a New PV Pointing to the Same Disk

This is useful when you don’t want to touch the old PV object and you want more control over the new binding.

• Identify the underlying disk (EBS volume, Azure Disk, etc.)

• Create a new PV definition

• Point it to the same physical disk

• Create a new PVC that binds to this new PV

The data remains untouched because the disk never changed—only the Kubernetes objects did.

Do you know AWS EBS Does NOT Support RWX

AWS Elastic Block Storage (EBS) does not support ReadWriteMany (RWX) access mode. why ?

Because an EBS volume behaves like a physical hard drive. Once you attach a hard drive to one machine, you cannot plug that same drive into multiple machines at the same time and expect it to work safely. This is exactly how EBS works:

• One EBS volume → one node

• Read/write access → single node only

That’s why EBS supports ReadWriteOnce (RWO).This limitation is not a Kubernetes issue — it’s a block storage limitation.

So What If You Need RWX Across Multiple Nodes?

If your application needs to run on multiple nodes and read and write the same data simultaneously, then block storage is the wrong tool. This is where file-based network storage comes in.

EFS: The RWX-Friendly Storage

Amazon Elastic File System (EFS) is designed to solve exactly this problem. EFS behaves like a shared network drive. Multiple nodes can mount it and all nodes see the same file and Read/Write operations happens concurrently

This wraps up our deep dive into PV, PVC, and StorageClasses — what they are, why they exist, and how Pods actually consume them.

From here, you can start mapping these concepts to real-world scenarios and production use cases, not just lab setups.

If you ever wondered “Where my data go when a pod restarts“ you are in a right place. Lets dive in 🚀

🤔 Why Do We Even Need Volumes?

Let’s start with a simple question: why does Kubernetes need volumes at all?

We know that a Pod is made up of one or more containers. And we also know an important (and slightly scary) fact about containers: Containers are ephemeral. This means when a container crashes, restarts, or gets recreated, its filesystem is wiped clean. Any files written inside the container? 💥 Gone.

This is where Volumes come to the rescue. A Kubernetes Volume provides a way to decouple storage from the container’s lifecycle. So to answer the question :

🎯 Why We Use Volumes

• Protect application data from container crashes

• Share data between containers in a Pod

• Make applications more reliable and production-ready

Kubernetes offers multiple volume types, each designed for different use cases—temporary storage, shared storage, cloud disks, network storage, and more.

In the next sections, we’ll explore these volume options one by one, understand when to use what, and avoid common mistakes along the way.

📦 emptyDir — The Simplest Kubernetes Volume

The most basic volume type in Kubernetes is emptyDir. An emptyDir volume is created at the Pod level, not at the container level.

The What ?

👉 But what does Pod level actually mean?

When a Pod is created: Kubernetes creates the emptyDir volume once. Every container inside that Pod can mount and access the same volume.

So, the volume does not belong to a single container, it belongs to the pod itself. As long as the pod exists the volume exists

🤝 How Do Containers Share an emptyDir?

Let’s make this real with a common and very practical scenario. Imagine a Pod with two containers:

• App Container: Runs your main application and generate logs

• Sidecar Container: Collect those logs and ship them to a central logging system. You don’t want to add extra load or logging logic inside your main app, so you offload that responsibility to a sidecar container.

Both containers mount the same emptyDir volume as a result the app writes the logs to the volume and sidecar reads logs from the same volume. They’re isolated containers but sharing data seamlessly.

The important catch is data in the emptyDir volume will be lost when the pod restarts or crashed. But the data will be preserved even if any of the container in the pod crashes or restarted. Because emptyDir lives only as long as the Pod lives.

So in simple terms emptyDir:

• Container lifecycle ❌ does NOT affect data

• Pod lifecycle ✅ DOES affect data

The When ?

emptyDir is ideal for:

• Temporary files

• Cache data

• Shared workspace between containers

• Log sharing (like our sidecar example)

🚫 It is not meant for long-term or critical data storage.

The How ?

Here’s a minimal working example of exactly the scenario we discussed.

apiVersion: v1
kind: Pod
metadata:
  name: shared-workspace
spec:
  containers:
    - name: producer
      image: busybox
      # Writes data to the volume
      volumeMounts:          # <--- VolumeMount block (Producer)
        - name: shared-data
          mountPath: /app/data
    - name: consumer
      image: busybox
      # Reads data from the volume
      volumeMounts:          # <--- VolumeMount block (Consumer)
        - name: shared-data
          mountPath: /app/input
  volumes:                  # <--- Volume definition
    - name: shared-data
      emptyDir: {}           # <--- The magic keyword

🔍 Let’s Dissect the Example Step by Step

• Let’s begin at the bottom of the Pod spec—the volumes block. This is where we define

  • The volume name - shared-data

  • The volume type - emptyDir

• This tells Kubernetes: Create an empty directory when the Pod starts and attach it to this Pod. At this point, the volume exists—but no container is using it yet.

• Now look at the two containers inside the Pod: Each container has its own volumeMounts block. Both containers mount the same volume (shared-data), but at different paths. This is intentional—and this is where things get interesting.

🤔 The Common Doubt (Very Natural One!)

You might be thinking: If the producer writes logs to /app/data, how does the consumer read them from /app/input?

Think of the volume as a room. And think of mountPath as a door. A room can have multiple doors, no matter which door you enter, you end up in the same room

With this analogy in our example /app/data is one door and /app/input is another door, both doors lead to the same emptyDir volume, so The producer writes logs into the room through the /app/data door and the consumer reads those same logs from the room through the /app/input door

With emptyDir, data survives container crashes and restarts. However, once the Pod is restarted or recreated, all the data in the volume is lost.

But what if we want our data to survive even Pod restarts or crashes? That’s where hostPath comes in. Let’s take a look.

📦 hostPath

The main limitation of emptyDir is that the data is lost when the Pod restarts. This problem is addressed by hostPath.

The What ?

With hostPath, the data is preserved no matter what happens to the Pod, as long as the Pod is scheduled on the same node.

🤔 How does this work?

The trick is simple: hostPath mounts a specific file or directory from the node’s filesystem directly into your Pod. So even if the Pod dies and a new Pod starts on the same node, the data is still there—because it never left the node in the first place.

The When?

Use hostPath when:

• You want data to survive Pod restarts

• You are okay with the Pod running on the same node

• You are working in local development, single-node clusters, or testing environments

• You need access to node-level files (logs, sockets, configs)

⚠️ Not recommended for multi-node production workloads due to portability and security concerns.

⚠️ During node maintenance or a node crash, extra care is required. Any Pod using a hostPath volume will lose its data if it gets drained and rescheduled onto a different node. The data is preserved only as long as the Pod remains on the same node and the node is up and running.

The How ?

apiVersion: v1
kind: Pod
metadata:
  name: shared-workspace
spec:
  containers:
    - name: producer
      image: busybox
      volumeMounts:
        - name: shared-data
          mountPath: /app/data
    - name: consumer
      image: busybox
      volumeMounts:
        - name: shared-data
          mountPath: /app/input
  volumes:
    - name: shared-data
      hostPath: # <--- The magic keyword hostPath
        path: /tmp/shared-data
        type: DirectoryOrCreate

🔍 Let’s Dissect the Example Step by Step

• Instead of emptyDir, we now use hostPath

• The data is stored on the node at /tmp/shared-data

• The type field tells Kubernetes what it should expect at the given path on the node and what to do if it doesn’t exist. If it does not exist create the directory automatically

• 📦 Common hostPath Types

🤔 Common Doubts

You might have this question in mind: We know that hostPath is tied to a specific node. What happens if I restart the Pod or apply a rollout that recreates Pods? Is there any guarantee that the Pod will be scheduled on the same node where my data exists? And if it gets scheduled on a different node, will the data be lost?

Yes—if the Pod is scheduled on a different node, the data will be lost. In most cases, Kubernetes tries to reschedule the Pod onto the same node it was previously running on. This is why, during normal Pod restarts or rollouts, you often see the Pod coming back on the same node and your data appearing to be “safe.”

❓ Why Does This Happen?

When the scheduler evaluates where to place a Pod, it considers: Existing node assignments, Node availability, Resource constraints. If the node is Healthy, Not Drained, and has sufficient resources Kubernetes will typically place the pod back on the same node. ‘

However There is no strict guarantee. If the node is crashed, drained, under maintenance, or out of resources the pod will be rescheduled to different node.

So far, we’ve seen how hostPath and emptyDir volumes work—great for temporary data, experiments, caching, and understanding how Kubernetes handles storage inside a Pod or a node. But what if you want your data to live beyond Pod restarts, survive node failures, and stay safe from all the usual Kubernetes chaos? 🤯

That’s exactly where Persistent Volumes (PV) come into the picture. They solve the problem of long-lived, reliable storage in Kubernetes. I’ve covered Persistent Volumes in detail in the next blog, breaking down how they work, why they matter, and when you should use them in real-world setups.

If you’ve made it this far — great job 👏 You now have a solid understanding of Kubernetes’ ephemeral storage story.

👉 Continue the journey here: Persistent Volumes in Kubernetes — and let’s level up your storage game 🚀

If you're just joining the journey, you can follow the entire series from here:

🧠 Setting the Context: The Core Logic of Our App

The core functionality of our ToDo application includes:

• ✅ Creating a ToDo item

• ✏️ Updating a ToDo item

• ❌ Deleting a ToDo item

• 📄 Reading (fetching) ToDo items

In this section, we’ll implement the application logic that powers these features.

We'll be writing code that defines what the app should do when a user makes a specific request — and what kind of response the app should return.

🛠️ How Are We Going to Do This?

Just like in the previous blogs of this series, we’ll follow a two-step approach:

1. Start with the basic (native) approach to understand how things work at the core level.

2. Evaluate and adopt a better alternative library that simplifies and enhances the process.

3. Use the selected method to implement the entire logic and test it end-to-end.

Let’s get started and build the Application Layer of our ToDo app! 🚀

🌐 Understanding HTTP in Our Web Application

Since we’re building a web application, all user interactions will happen through HTTP requests. We should handle this request to serve the user.

💡 Wait… what do you mean by handling HTTP requests?

Great question! In a web app, users interact with the backend using HTTP — the protocol behind the web. For example, if a user wants to create a ToDo item, they would make a request like:

curl -X POST http://mytodoapp/create \
  -H "Content-Type: application/json" \
  -d '{
    "task_name": "Write blog post",
    "description": "Complete the blog post for the Golang ToDo app",
    "status": "NotStarted",
    "end_date": "2024-06-30T23:59:00Z"
  }'

You might be wondering: “What exactly is that?” Don’t worry — we’ll cover it in more detail soon. For now, just understand that users will interact with our app using HTTP URLs and endpoints (like /create, /update, etc.) to trigger certain actions. And when we say "handling HTTP requests", we mean:

The application should be able to receive the request, extract any information from it (like request body or URL parameters), and perform the appropriate action in response.

How Do We Handle HTTP in Golang? To handle HTTP in Go, there are several libraries available. One of the most basic and native options is: net/http

🛠 Let’s Start with Creating a ToDo Item

To begin, we’ll focus on one core operation: creating a new ToDo item.

We’ll implement an HTTP handler using the net/http package that listens for a POST request, extracts data from the request body, and passes it to our application logic (which we already connected to the database layer).

In the next section, we’ll walk through how to:

1. Define an HTTP handler function

2. Read and parse the JSON body from the client

3. Validate and use the data to create a new ToDo

4. Return a meaningful response back to the user

Let’s dive in and build the POST /todo/create endpoint using net/http.

🚦 Recollecting what we will be doing

From the previous step, we now understand that:

• ✅ We need to import an HTTP handler to allow our Go application to serve web requests

• ✅ To read and parse the incoming request body (usually in JSON), we need a package for decoding JSON. A quick Google search shows that Go provides a standard library for this:"encoding/json"

• ✅ We need to write the logic to:

  1. Receive incoming HTTP requests

  2. Read the data (like a new ToDo item) from the request body

  3. Pass it to our business logic (e.g., CreateToDo)

  4. Return a proper response to the user

▶️ Let’s Code from Where We Left

If you're following along, continue from the code snippet in the last blog. We'll now build the ToDoHanlderCreate using the net/http and encoding/json packages.

Let’s plug that into our main.go file from our last blog and get started 🚀

package main

import (
    "fmt"
    "gorm.io/driver/mysql"
    "gorm.io/gorm"
)

var (
    // Declare a global variable to hold the DB connection
    db *gorm.DB // Type: pointer to gorm.DB
)

type ToDo struct{
    gorm.Model
    TaskName    string     `gorm:"not null" json:"title"`
    Description string     `gorm:"type text" json:"description"`
    Status      string     `gorm:"default:NotStarted" json:"status" validate:"required,oneof=NotStarted InProgress Pending Completed"`
    EndDate     *time.Time `json:"end_date"`
}

// ConnectDB establishes the connection to the MySQL database
func ConnectDB() {
    // Data Source Name (DSN): contains database connection info
    dsn := "root:password@tcp(127.0.0.1:3306)/todo?charset=utf8&parseTime=True&loc=Local"

    // Connect to the DB using GORM and MySQL driver
    db_conn, err := gorm.Open(mysql.Open(dsn), &gorm.Config{})
    if err != nil {
        // If connection fails, panic and log the error
        panic("Failed to connect to the database: " + err.Error())
    }

    /*Why I'm passing value here why can't I pass it like db,err. 
    If I do that I have to either add err variable globally or remove db global var as I'm using := so I used a var to exchnage value
    */
    db = db_conn 
}

func InitDB(){
    ConnectDB() // This will create new connection and the value is stored in DB variable

    // Here we are saying that Use the Schema &ToDo and create table using ou DB connection 'db'
    //'{}' defines that create empty table with that ToDo Struct Schema
    db.AutoMigrate(&ToDo{})
}

// CreateToDo inserts a new ToDo item into the database.
// It takes a pointer to a ToDo struct as input and returns the same pointer after insertion.
func CreateToDo(todo *ToDo) *ToDo {
    // Use GORM's Create method to insert the new record into the database.
    // The &todo tells GORM to insert the data from the memory location of the passed ToDo.
    db.Create(&todo)
    // Return the same pointer, now updated with DB-generated fields (like ID, CreatedAt, etc.)
    return todo
}

// .... Truncating other code you can refer from the link provided above
...
...
...

Let’s import and modify the code

package main

import (
    "fmt"
    "gorm.io/driver/mysql"
    "gorm.io/gorm"
    "time"
    "net/http"
    "encoding/json"
)

var (
    // Declare a global variable to hold the DB connection
    db *gorm.DB // Type: pointer to gorm.DB
)

type ToDo struct{
    gorm.Model
    TaskName    string     `gorm:"not null" json:"title"`
    Description string     `gorm:"type text" json:"description"`
    Status      string     `gorm:"default:NotStarted" json:"status" validate:"required,oneof=NotStarted InProgress Pending Completed"`
    EndDate     *time.Time `json:"end_date"`
}

func ConnectDB() {...}
func InitDB(){...}

func ToDoHanlderCreate(w http.ResponseWriter, r *http.Request){
    // step1: Ensure it's a Post request
    if r.Method != http.MethodPost {
     http.Error(w, "Only Post method is allowed", http.StatusMethodNotAllowed)
    return
    }
    // Step 2: Parse the incoming JSON body into a ToDo struct
    var todo ToDo
    err := json.NewDecoder(r.Body).Decode(&todo)
    if err != nil {
        http.Error(w, "Invalid Json Input", http.StatusBadRequest)
        return
    }
    // Step 3: Save the new ToDo item to the database
    db.Create(&todo) // Using Gorm Native create object option
    // Step 4: Set the content type and return the created object as JSON
    w.Header().Set("Content-Type", "application/json") // Set response type as JSON
    w.WriteHeader(http.StatusCreated) // Set HTTP status code to 201 (Created)
    json.NewEncoder(w).Encode(todo) // Write the todo object as JSON in response


}

func main() {
    InitDB() // Initialize DB

    // Register the route and handler
    http.HandleFunc("/todo/create", ToDoHanlderCreate)

    fmt.Println("🚀 Server running at http://localhost:8080")
    err := http.ListenAndServe(":8080", nil)
    if err != nil {
        fmt.Println("Error starting server:", err)
}

Lets breakdown the ToDoHanlderCreate function and how we change main.

Step : 1 - ToDoHandlerFunction

• func ToDoHanlderCreate(w http.ResponseWriter, r *http.Request){…}

  📥 r *http.Request

  • This parameter is used to receive and read the incoming HTTP request from the user.

  • It contains:

    • Request method (e.g., GET, POST)

    • Headers

    • Query parameters

    • Request body (like form data or JSON payload)

  • We're using a pointer (*http.Request) so we’re accessing the actual request data without copying it.

  • We store it in a variable named r so we can access its fields inside the function.

Example: To read the JSON request body, we'll use r.Body.

📤 w http.ResponseWriter

• This parameter is used to send the response back to the user.

• It lets us:

  • Write text or JSON data as the response

  • Set response status codes (like 200 OK or 400 Bad Request)

  • Set response headers (like Content-Type)

• Anything you write to w is what the client receives.

Example: To send back a success message, we can use w.Write([]byte("ToDo created")).

Step : 2 - Json Decoding

    // Step 2: Parse the incoming JSON body into a ToDo struct
    var todo ToDo
    err := json.NewDecoder(r.Body).Decode(&todo)
    if err != nil {
        http.Error(w, "Invalid Json Input", http.StatusBadRequest)
        return
    }

• Here, we create a variable called todo, which is of type ToDo struct. We need this variable because once we extract the content from the HTTP request body, we store it in this variable so we can use it later—for example, to insert it into the database.

• err := json.NewDecoder(r.Body).Decode(&todo) - This line uses Go’s encoding/json package to decode the JSON data from the request body and assign it to the todo variable.

• After decoding, we handle any potential error—this helps us ensure that the request body is in the correct format. If it's not, we can return a proper error message to the client.

Step : 3 - Create ToDo

db.Create(&todo) - We use GORM’s built-in Create method to add a new entry to our database. The data we’re inserting comes from the todo variable, which already holds the content we extracted and decoded from the incoming HTTP request. By passing &todo, we're telling GORM to insert the data from that memory location into the corresponding table.

Step : 4 - Response to user

w.Header().Set("Content-Type", "application/json") // Set response type as JSON
w.WriteHeader(http.StatusCreated)                 // Set HTTP status code to 201 (Created)
json.NewEncoder(w).Encode(todo)                   // Write the todo object as JSON in response

• w.Header().Set("Content-Type", "application/json") - Tells the client that you're sending back JSON data.

• w.WriteHeader(http.StatusCreated)- Sends an HTTP status code 201 Created, which is the correct status code for successful resource creation.

• json.NewEncoder(w).Encode(todo) - Converts the todo struct to JSON and writes it to the response body.

main() Function:

InitDB() // Initialize DB

    // Register the route and handler
    http.HandleFunc("/todo/create", ToDoHanlderCreate)

    fmt.Println("🚀 Server running at http://localhost:8080")
    err := http.ListenAndServe(":8080", nil)
    if err != nil {
        fmt.Println("Error starting server:", err)

• InitDB() This function initializes the database connection using GORM and MySQL. It also runs the AutoMigrate() function, which ensures that our ToDo model is translated into a table in the database.

• http.HandleFunc("/todo/create", ToDoHandlerCreate) Registering the Route. This means that any POST request to /todo/create will be handled by the ToDoHandlerCreate function. This is the entry point for users to create new ToDo items.

• http.ListenAndServe(":8080", nil) Starting the HTTP Server. This tells Go to start an HTTP server on port 8080. As long as the server is running, it will listen for incoming HTTP requests on that port.

✅ Summary

• Database: Initialized and migrated with InitDB()

• Route: /todo/create handles new ToDo creation

• Server: Listens on http://localhost:8080

This covers the full operation of create ToDo item on our ToDo app using native http request. Here is the complete code snippet.

package main

import (
     "fmt"
    "gorm.io/driver/mysql"
    "gorm.io/gorm"
    "time"
    "net/http"
    "encoding/json"


)

var db *gorm.DB // Global DB connection

type ToDo struct {
    gorm.Model
    TaskName    string     `gorm:"not null" json:"title"`
    Description string     `gorm:"type:text" json:"description"`
    Status      string     `gorm:"default:NotStarted" json:"status"`
    EndDate     *time.Time `json:"end_date"`
}

// ConnectDB sets up the MySQL connection and assigns it to the global db variable
func ConnectDB() {
    dsn := "root:password@tcp(127.0.0.1:3306)/todo?charset=utf8&parseTime=True&loc=Local"
    dbConn, err := gorm.Open(mysql.Open(dsn), &gorm.Config{})
    if err != nil {
        panic("Failed to connect to the database: " + err.Error())
    }
    db = dbConn
}

// InitDB initializes the DB and migrates the schema
func InitDB() {
    ConnectDB()
    db.AutoMigrate(&ToDo{}) // Creates the 'todos' table based on our struct
}

// CreateToDoHandler handles HTTP POST requests to create a new ToDo item
func ToDoHanlderCreate(w http.ResponseWriter, r *http.Request){
    // step1: Ensure it's a Post request
    if r.Method != http.MethodPost {
     http.Error(w, "Only Post method is allowed", http.StatusMethodNotAllowed)
    return
    }
    // Step 2: Parse the incoming JSON body into a ToDo struct
    var todo ToDo
    err := json.NewDecoder(r.Body).Decode(&todo)
    if err != nil {
        http.Error(w, "Invalid Json Input", http.StatusBadRequest)
        return
    }
    // Step 3: Save the new ToDo item to the database
    db.Create(&todo) // Using Gorm Native create object option
    // Step 4: Set the content type and return the created object as JSON
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(http.StatusCreated)
    json.NewEncoder(w).Encode(todo)

}

func main() {
    InitDB() // Initialize DB

    // Register the route and handler
    http.HandleFunc("/todo/create", ToDoHanlderCreate)

    fmt.Println("🚀 Server running at http://localhost:8080")
    err := http.ListenAndServe(":8080", nil)
    if err != nil {
        fmt.Println("Error starting server:", err)
    }  
}

▶️ How to test the code

• Create Database if you don’t have one by following the steps mentioned here

• Create a file called main.go and copy the above content and paste it.

• Initialise your go app by running go mod init example.com/todo

• Run go mod tidy to download all the dependencies

• Finally run go run main.go start your application. You will see the below response on your terminal

  

• Now open new terminal and use below curl command to create an entry in your ToDo app

curl -X POST http://localhost:8080/todo/create \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Write Go server",
    "description": "Build a simple ToDo API with Go and GORM",
    "status": "InProgress",
    "end_date": "2025-06-30T17:00:00Z"
  }'

• You will get the response like this

• To check it on database run the below commands

# Open CMD and run
docker exec -it mysql /bin/bash
mysql -u root -p
# <Enter your password when it asks i.e "password">
# Now you will be in Mysql db
USE todo;
SELECT * FROM to_dos

• This command will show you the newly created Entry

📝 Conclusion

That wraps up our implementation of a simple ToDo API using Go’s net/http package, GORM, and MySQL. In this post, we created the model, set up the database connection, handled a POST request, and tested it with curl.

In the next blog post, we’ll explore alternatives to the net/http standard library—specifically, why developers often choose web frameworks like Gin or Echo for larger or more maintainable applications. We'll also complete the Application Layer, continuing our journey toward building a clean and modular Go backend.

Stay tuned!

Before diving into the actual problem and its solution, let’s take a moment to understand what OpenSearch is and where it's commonly used.

📌 What is OpenSearch? Where is it Used?

OpenSearch is an AWS-managed service based on the open-source fork of Elasticsearch. It offers powerful capabilities for indexing, searching, and analyzing large volumes of data in near real time.

OpenSearch is widely used in infrastructure for a variety of use cases, including:

• Log analytics

• Full-text search

• Application performance monitoring

• Security information and event management (SIEM)

In recent years, many companies managing their infrastructure on AWS have been shifting towards AWS-managed services. This transition helps reduce the operational burden of managing infrastructure, allowing IT teams to focus on solving business-critical problems rather than maintaining and scaling services themselves.

AWS OpenSearch has become one of the most widely adopted services in this category, providing a fully managed, scalable, and secure alternative to running Elasticsearch on self-managed instances.

⚙️ How is OpenSearch Configured?

When using AWS OpenSearch Service, you typically configure the following key components:

🌐 Exposing OpenSearch to External Networks Using ALB

When you create an AWS-managed OpenSearch cluster, AWS provides a domain endpoint that is accessible only from within the same VPC, depending on your VPC settings and security group rules. However, resources or users outside the VPC—such as developers, third-party services, or monitoring tools—cannot access this endpoint directly.

In real-world setups, it's quite common for some users or systems to live outside the VPC. For example, accessing the OpenSearch Dashboard from an external network becomes difficult since direct connectivity isn’t allowed.

To solve this problem, we can expose OpenSearch securely through an AWS Application Load Balancer (ALB).

1. Create an ALB
   An internet-facing ALB is created to act as a public entry point. This ALB provides a public DNS name that external users and resources can access.

2. Register OpenSearch Node in a Target Group
   We then create a Target Group and register the IP address of the OpenSearch node behind the domain endpoint.

   🔹 Note: Even if your OpenSearch domain has multiple nodes, AWS internally exposes only one node behind the endpoint. Other nodes are used for redundancy and failover.

3. Connect Target Group to ALB
   The Target Group is attached to the ALB. This setup ensures that whenever a request hits the ALB, it is forwarded to the registered OpenSearch node in the Target Group.

4. Route 53 for Domain Management
   AWS Route 53 manages the public DNS name of the ALB. If you're using HTTPS, SSL certificates issued via ACM (AWS Certificate Manager) are also managed and linked here.

This setup enables external access to OpenSearch in a secure and controlled way. The ALB acts as a bridge between the external network and the private OpenSearch cluster inside the VPC.

⚠️ The Problem Statement: Why the ALB Connection Breaks

Here comes the real challenge.

In most setups, we do not maintain static (sticky) IP addresses for OpenSearch cluster nodes. When an OpenSearch cluster restarts—either due to scaling, upgrades, or internal AWS maintenance—the IP address of the node exposed via the endpoint can change.

Now, since the ALB target group is manually registered with the IP of the previously exposed node, it still tries to forward traffic to that old, now-invalid IP. But OpenSearch is now responding from a new node with a different IP. And here's the problem:

• The ALB doesn’t have any native integration with OpenSearch to update its target group dynamically.

• As a result, the target group points to a stale IP, and the target becomes unhealthy, breaking the connection between the ALB and OpenSearch.

This leads to downtime or failed requests from all external clients relying on the ALB for access.

🔁 The Need for Automation

To prevent this, we need a robust, automated mechanism to keep the ALB target group in sync with the currently exposed IP of the OpenSearch endpoint.

There are several ways to implement this, and in the next section, I’ll walk you through one of the approaches I recently implemented in a production environment.

✅ Quick Summary: How I Solved the ALB & OpenSearch Restart Issue

Before we dive into the detailed steps, here’s a quick overview of how this issue was resolved.

Assumption: You already have an OpenSearch cluster running and exposed via an ALB to allow access from external networks.

The core idea is to detect when the OpenSearch cluster restarts and automatically update the ALB target group with the currently active OpenSearch node IP. This ensures that the ALB always routes traffic to a healthy target, avoiding downtime or broken access for external users.

To achieve this:

• I set up a CloudWatch alarm that monitors the number of nodes in the OpenSearch cluster.

• Using EventBridge, I track when the alarm transitions from an ALARM state back to OK, indicating that the cluster has recovered after a restart.

• This triggers a Lambda function, which:

  • Fetches the active OpenSearch node IP.

  • Updates the ALB target group with the new IP.

  • Removes any stale IPs no longer part of the cluster.

This automation ensures that your ALB always points to a healthy OpenSearch node, even after a restart — maintaining uninterrupted access for external systems.

In the next section, I’ll walk you through each step of this setup with screenshots and configuration details.

🛠️ Detailed Procedure:

The steps outlined below can be implemented using any Infrastructure-as-Code (IaC) or configuration management tool of your choice. In this guide, the focus is on what needs to be implemented rather than how it's implemented with a specific tool. You’re free to use the tool or framework that best fits your environment — this guide should help you understand the logic and flow regardless of the platform.

🔹 Step 1: Get the Total Number of Nodes in Your OpenSearch ClusterYou can find the details in Cloud watch

The first step is to identify the total number of nodes (both master and data nodes) in your OpenSearch cluster. This information is essential for setting up a reliable CloudWatch alarm.

You can find this detail using Amazon CloudWatch:

1. Log in to the AWS Management Console.

2. In the search bar, type and select CloudWatch.

3. From the left-hand menu, click on "All metrics."

4. 

   In the search bar, type “ES” to filter OpenSearch-related metrics, From the results, select “ES → Per-Domain, Per-Client Metrics.”

   

5. Choose your OpenSearch domain and look for the Nodes metric. This will show the number of nodes currently active in your cluster.

6. 

   Check the checkbox for Nodes corresponding to your specific OpenSearch domain. This will display a graph showing the number of active nodes over time.

7. While there are certainly easier ways to retrieve the total number of nodes in an OpenSearch cluster, this approach has a key advantage — it allows us to directly create a CloudWatch alarm based on this metric.

8. So, as you might have guessed — the next step is to click on “Create alarm.”

9. 

   Configure the alarm setting as follow

10. 

    In the next step, click “Remove” under the notification section to delete any default SNS notification settings — since we’ll be using EventBridge to trigger the action instead of relying on SNS alerts.

11. 

    Add your preferred name to the Alarm

    

12. Finally Click on Create Alarm

13. Now, follow Steps 3 to 5 in the alarm creation flow. Once completed, you should see 1 alarm listed under your OpenSearch domain name in the CloudWatch metrics dashboard.

14. Select the Node Checkbox, and then select the Graphed Metrics

15. 

    Select Add Math » Conditional » Equals

16. 

    From the expression click Edit icon

17. 

    Change the value to m1 == <your total number of nodes>, where m1 refers to the metric ID for Nodes in your alarm, which you can find in the same menu under ID, and <your total number of nodes> is the value you noted in Step 8. Click on Apply

18. 

    Voila! The alarm is now set. It will return a value of 1 when the current node count is exactly equal to the expected total (in our case, 6), and 0 when it’s either greater or less than that.

19. This is exactly what we need — because any change in node count usually indicates that something has happened with the OpenSearch cluster (such as a restart or scaling event). At this point, we need to verify the ALB Target Group IP and update it if it no longer points to a healthy OpenSearch node.

20. Now, let’s move on to setting up EventBridge to trigger a Lambda function whenever this happens.

🔹 Step 2: Configure a Lambda Function to Update the Target Group with the OpenSearch Endpoint Node IP

There are plenty of resources — including official documentation and YouTube tutorials — that can guide you through how to create and configure a Lambda function.

In this section, I’ll focus on what matters most for this use case:

• The runtime/language you should select for your Lambda. - Python:3.9 and above

• The script you’ll use to update the Target Group.

• The environment variables that need to be configured for the Lambda to work effectively.

  • OPENSEARCH_HOST:

    • You can get this value from, Login in to console » Search Opensearch » Click Domains » Select your Domain » You can find this value under VPC EndPoint

    • (Remove “https://” when adding it in env variable)

  • TARGET_GROUP_ARN

    • You can get this value from Login in to console »Search Target Group » Select your Target Group » You can find this value under ARN

• Script to use

  Note: I’ve hard coded the port to 443 replace it with your port number

import boto3
import os
import socket

elb = boto3.client('elbv2')

TARGET_GROUP_ARN = os.environ['TARGET_GROUP_ARN']
OPENSEARCH_HOST = os.environ['OPENSEARCH_HOST']

def lambda_handler(event, context):
    try:
        # Resolve new IP of OpenSearch
        new_ip = socket.gethostbyname(OPENSEARCH_HOST)
        print(f"Resolved OpenSearch IP: {new_ip}")

        # Describe current targets
        current_targets = elb.describe_target_health(TargetGroupArn=TARGET_GROUP_ARN)
        registered_ips = [t['Target']['Id'] for t in current_targets['TargetHealthDescriptions']]
        print(f"Currently registered IPs: {registered_ips}")

        #Deregister old IPs
        for ip in registered_ips:
            if ip != new_ip:
                print(f"Deregistering IP: {ip}")
                elb.deregister_targets(
                    TargetGroupArn=TARGET_GROUP_ARN,
                    Targets=[{"Id": ip, "Port": 443}]
                )

        # Register new IP if not already registered
        if new_ip not in registered_ips:
            print(f"Registering new IP: {new_ip}")
            elb.register_targets(
                TargetGroupArn=TARGET_GROUP_ARN,
                Targets=[{"Id": new_ip, "Port": 443}]
            )

        return {"status": "Success", "new_ip": new_ip}

    except Exception as e:
        print(f"Error: {str(e)}")
        raise e

• Make sure you have necessary permission for this Lambda function to, if you don’t have create an IAM role with below access.

  • Read the VPC Endpoint configuration settings from Opensearch

  • Ability to add Targets to the Target group

  • Ability to Selete/Drain targets from the target group

• Once all are set test it manually invoking the lambda function

🔹 Step 3: Configuring Event Bridge to trigger Lambda

1. Log in to the AWS Management Console.

2. In the search bar, type and select Amazon EventBridge » From left menu Click Rules » Select Create rule

3. 

   Give preferred rule name and description and Click Next

4. 

   Click Edit Pattern

5. 

   Replace the content what ever needed as mentioned below and paste the content in the code snippet.

   - YOUR ALARM NAME » Replace this with the Alarm name which you created in Step 1

{
  "source": ["aws.cloudwatch"],
  "detail-type": ["CloudWatch Alarm State Change"],
  "detail": {
    "alarmName": ["<YOUR ALARM NAME>"],
    "state": {
      "value": ["ALARM", "OK"]
    },
    "previousState": {
      "value": ["OK", "ALARM"]
    }
  }
}

6. Explanation of the condition

   1. "source": ["aws.cloudwatch"]
      Ensures the event is coming from Amazon CloudWatch.

   2. "detail-type": ["CloudWatch Alarm State Change"]
      Captures only events related to alarm state changes.

   3. "alarmName": ["<YOUR ALARM NAME>"]
      Filters the event to trigger only for the specific alarm you created to monitor OpenSearch node count.

   4. "state": { "value": ["ALARM", "OK"] }
      Triggers the rule when the alarm enters from the ALARM to OK state.

   5. "previousState": { "value": ["OK", "ALARM"] }
      Ensures the transition is meaningful — for example, from OK ➝ ALARM and then ALARM ➝ OK — avoiding unnecessary triggers.

7. Click Next to save your pattern

8. Then For the target

   1. Target Type : AWS Service

   2. Select a target » Lambda Function

   3. Function » Select the function you created on Step 2 from drop down

   4. Leave the other box as it is and Click Next

9. In the next step add the necessary tag if required and click Next and Finally review the configuration and click on Create Rule

Voila! This concludes the overall configuration for automating the update of OpenSearch Node IPs in the Target Group.

With this setup in place, your Target Group will always stay up to date with the active and healthy IP address of your OpenSearch cluster — ensuring seamless connectivity even during restarts or node changes.

🧱 Building the Database Layer for Our ToDo App

With everything we’ve learned so far, let’s go ahead and build the database layer for our ToDo app.

Before we dive into the implementation, let’s outline the operations we’ll be handling through our database layer. This will help us clearly identify what we need to build:

🧩 What Will the Database Layer Do?

1. We need to establish connection with the Database.

2. Define the schema (i.e., table structure) to store our ToDo items

3. Implement CRUD operations (Create, Read, Update, Delete) such as:

   • Create a new ToDo

   • Update an existing ToDo

   • Get all ToDos

   • Get a specific ToDo by ID

   • Delete a ToDo

🔁 Recap of Key Decisions

Before we begin, here’s a quick recap of what we’ve decided so far. If you missed any part, feel free to check the previous articles:

1. We’re using MySQL as our database of choice

2. We’ll use Golang with the GORM library, along with the MySQL driver, to interact with the database

✅ With all that clear, let’s start putting things together and build the database layer!

Establishing Connection with our DB:

Let’s start with establishing connection with our DB

package main

import (
    "fmt"
    "gorm.io/driver/mysql"
    "gorm.io/gorm"
)

var (
    // Declare a global variable to hold the DB connection
    db *gorm.DB // Type: pointer to gorm.DB
)

// ConnectDB establishes the connection to the MySQL database
func ConnectDB() {
    // Data Source Name (DSN): contains database connection info
    dsn := "root:password@tcp(127.0.0.1:3306)/todo?charset=utf8&parseTime=True&loc=Local"

    // Connect to the DB using GORM and MySQL driver
    db_conn, err := gorm.Open(mysql.Open(dsn), &gorm.Config{})
    if err != nil {
        // If connection fails, panic and log the error
        panic("Failed to connect to the database: " + err.Error())
    }

    /*Why I'm passing value here why can't I pass it like db,err. 
    If I do that I have to either add err variable globally or remove db global var as I'm using := so I used a var to exchnage value
    */
    db = db_conn 
}

Lets Define Schema:

package main

import (
    "fmt"
    "gorm.io/driver/mysql"
    "gorm.io/gorm"
    "time"
)

var (
    // Declare a global variable to hold the DB connection
    db *gorm.DB // Type: pointer to gorm.DB
)

type ToDo struct{
    gorm.Model
    TaskName    string     `gorm:"not null" json:"title"`
    Description string     `gorm:"type text" json:"description"`
    Status      string     `gorm:"default:NotStarted" json:"status" validate:"required,oneof=NotStarted InProgress Pending Completed"`
    EndDate     *time.Time `json:"end_date"`
}

// ConnectDB establishes the connection to the MySQL database
func ConnectDB() {
    ...
}

// This will initialize DB and create Schema (Our Table structure)
func InitDB(){
    ConnectDB() // This will create new connection and the value is stored in DB variable

    // Here we are saying that Use the Schema &ToDo and create table using ou DB connection 'db'
    //'{}' defines that create empty table with that ToDo Struct Schema
    db.AutoMigrate(&ToDo{})
}

We are defining schema i.e (Table structure) for our database in the form of Struct. Let deep dive what we have entered.

• gorm.Model

  This is a built-in struct in GORM that automatically includes four common fields:

  • • ID – a unique identifier for each record

      * CreatedAt – timestamp of when the record was created

      * UpdatedAt – timestamp of the last update

      * DeletedAt – soft delete timestamp

By embedding gorm.Model at the start of your struct, GORM auto-generates and manages these fields for each record in the table.

• TaskName string `gorm:"not null" json:"title"`

  • TaskName: This is a field in your struct that will map to a column in your database table. It stores the task name for each ToDo item.

  • string: The data type of the field.

  • gorm:"not null": A GORM tag that enforces the column to never be null in the database.

  • json:"title": This tag tells Go how to map this field when converting JSON data, such as when handling API requests and responses. So, if you send or receive JSON with the key title, it will map to TaskName in your Go struct. We will see how this is used when we are testing the app.

• Status string gorm:"default:NotStarted" json:"status" validate:"required,oneof=NotStarted InProgress Pending Completed"

  • Status: is a column in your database table used to track the current state of a ToDo item.

  • It’s of type string.

  • json:"status" : This field is same as json:title used when passing JSON using API

  • validate:"required,oneof=NotStarted InProgress Pending Completed" : This field defines that this variable is required and its value should be one of “NotStarted InProgress Pending Completed"

  • default:NotStarted: If no value is defined we are setting default value as NotStarted

• Since EndDate variable type should be handled in time we mentioned it as *time.Time type

Implement CURD Operation

As we defined the Schema in the above step lets complete all the operation required for our ToDo App

Operation Required

• Create a new ToDo

• Update an existing ToDo

• Get all ToDos

• Get a specific ToDo by ID

• Delete a ToDo

package main

import (
    "fmt"
    "gorm.io/driver/mysql"
    "gorm.io/gorm"
    "time"
)

var (
    // Declare a global variable to hold the DB connection
    db *gorm.DB // Type: pointer to gorm.DB
)

type ToDo struct{
    gorm.Model
    TaskName    string     `gorm:"not null" json:"title"`
    Description string     `gorm:"type text" json:"description"`
    Status      string     `gorm:"default:NotStarted" json:"status" validate:"required,oneof=NotStarted InProgress Pending Completed"`
    EndDate     *time.Time `json:"end_date"`
}

// ConnectDB establishes the connection to the MySQL database
func ConnectDB() {
    ...
}

// This will initialize DB and create Schema (Our Table structure)
func InitDB(){
  ...
}

// CreateToDo inserts a new ToDo item into the database.
// It takes a pointer to a ToDo struct as input and returns the same pointer after insertion.
func CreateToDo(todo *ToDo) *ToDo {
    // Use GORM's Create method to insert the new record into the database.
    // The &todo tells GORM to insert the data from the memory location of the passed ToDo.
    db.Create(&todo)
    // Return the same pointer, now updated with DB-generated fields (like ID, CreatedAt, etc.)
    return todo
}

// GetToDo returns a list of all ToDo items from the database.
// This function does not take any input and returns a slice of ToDo structs ([]ToDo).
func GetToDo() []ToDo {
    var todos []ToDo // Declare a variable to hold the fetched ToDo items (slice of ToDo structs)
    // Use GORM's Find method to retrieve all records from the ToDo table and store them in variable 'todos'
    db.Find(&todos)
    // Return the fetched ToDo items
    return todos
}

// GetToDoByID fetches a specific ToDo item from the database based on the provided ID.
// It takes an integer ID as input and returns a pointer to a ToDo struct.
func GetToDoByID(ID int64) *ToDo {
    // We declare a single instance of the ToDo struct (not a slice),
    // because we're expecting only one record, not a list of ToDos.
    var todo ToDo
    // Use GORM's First method to retrieve the first matching record by ID.
    // "id = ?" is a query condition where the placeholder (?) gets replaced by the ID value.
    db.First(&todo, "id = ?", ID)
    // Return a pointer to the fetched ToDo item
    return &todo
}

// DeleteToDo removes a ToDo item from the database based on the provided ID.
// It takes an integer ID as input and returns the deleted ToDo struct.
func DeleteToDo(ID int64) ToDo {
    // We declare a single instance of the ToDo struct (not a slice),
    // because we're expecting only one record, not a list of ToDos.
    var todo ToDo // Declare a variable to hold the ToDo item we want to delete

    // Use GORM's Delete method to remove the record from the database using the given ID.
    // The first argument is the model type, the second is the ID used for deletion.
    db.Delete(&todo, ID)

    // Return the todo struct (note: this will only contain ID as GORM does not fetch the record before deletion)
    return todo
}

// UpdateToDo updates an existing ToDo item in the database.
// It takes a pointer to a ToDo struct (with updated fields) as input and returns the updated ToDo struct.
func UpdateToDo(todo *ToDo) *ToDo {

    // Use GORM's Save method to update the existing record.
    // Save will update the record if it exists (based on the primary key in the struct).
    // When using db.Save(todo), you must pass all the field values — both updated and unchanged.
    // This is because Save performs a full update, replacing the entire row in the database.
    db.Save(todo)

    // Return the updated ToDo item
    return todo
}

✅ Database Layer Completed

We’ve now completed all the essential database operations (CRUD) for our ToDo app. This wraps up our Database Layer Setup! 🎉

Your code so far should include:

• Database connection setup using GORM

• ToDo model definition

• Functions for Create, Read (all & by ID), Update, and Delete operations

package main

import (
    "fmt"
    "gorm.io/driver/mysql"
    "gorm.io/gorm"
    "time"
)

var (
    // Declare a global variable to hold the DB connection
    db *gorm.DB // Type: pointer to gorm.DB
)

type ToDo struct{
    gorm.Model
    TaskName    string     `gorm:"not null" json:"title"`
    Description string     `gorm:"type text" json:"description"`
    Status      string     `gorm:"default:NotStarted" json:"status" validate:"required,oneof=NotStarted InProgress Pending Completed"`
    EndDate     *time.Time `json:"end_date"`
}

// ConnectDB establishes the connection to the MySQL database
func ConnectDB() {
    // Data Source Name (DSN): contains database connection info
    dsn := "root:password@tcp(127.0.0.1:3306)/todo?charset=utf8&parseTime=True&loc=Local"

    // Connect to the DB using GORM and MySQL driver
    db_conn, err := gorm.Open(mysql.Open(dsn), &gorm.Config{})
    if err != nil {
        // If connection fails, panic and log the error
        panic("Failed to connect to the database: " + err.Error())
    }

    /*Why I'm passing value here why can't I pass it like db,err. 
    If I do that I have to either add err variable globally or remove db global var as I'm using := so I used a var to exchnage value
    */
    db = db_conn 
}

func InitDB(){
    ConnectDB() // This will create new connection and the value is stored in DB variable

    // Here we are saying that Use the Schema &ToDo and create table using ou DB connection 'db'
    //'{}' defines that create empty table with that ToDo Struct Schema
    db.AutoMigrate(&ToDo{})
}

// CreateToDo inserts a new ToDo item into the database.
// It takes a pointer to a ToDo struct as input and returns the same pointer after insertion.
func CreateToDo(todo *ToDo) *ToDo {
    // Use GORM's Create method to insert the new record into the database.
    // The &todo tells GORM to insert the data from the memory location of the passed ToDo.
    db.Create(&todo)
    // Return the same pointer, now updated with DB-generated fields (like ID, CreatedAt, etc.)
    return todo
}

// GetToDo returns a list of all ToDo items from the database.
// This function does not take any input and returns a slice of ToDo structs ([]ToDo).
func GetToDo() []ToDo {
    var todos []ToDo // Declare a variable to hold the fetched ToDo items (slice of ToDo structs)
    // Use GORM's Find method to retrieve all records from the ToDo table and store them in variable 'todos'
    db.Find(&todos)
    // Return the fetched ToDo items
    return todos
}

// GetToDoByID fetches a specific ToDo item from the database based on the provided ID.
// It takes an integer ID as input and returns a pointer to a ToDo struct.
func GetToDoByID(ID int64) *ToDo {
    // We declare a single instance of the ToDo struct (not a slice),
    // because we're expecting only one record, not a list of ToDos.
    var todo ToDo
    // Use GORM's First method to retrieve the first matching record by ID.
    // "id = ?" is a query condition where the placeholder (?) gets replaced by the ID value.
    db.First(&todo, "id = ?", ID)
    // Return a pointer to the fetched ToDo item
    return &todo
}

// DeleteToDo removes a ToDo item from the database based on the provided ID.
// It takes an integer ID as input and returns the deleted ToDo struct.
func DeleteToDo(ID int64) ToDo {
    // We declare a single instance of the ToDo struct (not a slice),
    // because we're expecting only one record, not a list of ToDos.
    var todo ToDo // Declare a variable to hold the ToDo item we want to delete

    // Use GORM's Delete method to remove the record from the database using the given ID.
    // The first argument is the model type, the second is the ID used for deletion.
    db.Delete(&todo, ID)

    // Return the todo struct (note: this will only contain ID as GORM does not fetch the record before deletion)
    return todo
}

// UpdateToDo updates an existing ToDo item in the database.
// It takes a pointer to a ToDo struct (with updated fields) as input and returns the updated ToDo struct.
func UpdateToDo(todo *ToDo) *ToDo {

    // Use GORM's Save method to update the existing record.
    // Save will update the record if it exists (based on the primary key in the struct).
    // When using db.Save(todo), you must pass all the field values — both updated and unchanged.
    // This is because Save performs a full update, replacing the entire row in the database.
    db.Save(todo)

    // Return the updated ToDo item
    return todo
}

🚀 What’s Next?

In the next section, we’ll focus on building the Application Layer (Middle Layer) — the core part of our app logic. We’ll learn how to:

• Connect the DB layer with our application logic

• Handle HTTP requests

• Build out our API endpoints

Stay tuned as we start connecting all the dots and bring our ToDo app to life! 🔗✨

We’ve already discussed why a database is essential for our app and why we chose a SQL database for this project. If you’re just joining us, you can catch up on that here.

⚙️ Prerequisites

Before we start building, let’s set up the basic infrastructure for this topic.

✅ What You’ll Need:

• Docker installed on your machine:

  • Use regular Docker on Linux

  • Use Docker Desktop on Windows or macOS

  • Make sure Docker is installed and running on your system.

• Once it’s up, run the following command to spin up a MySQL container that we’ll use in our project.

•   docker run --name mysql -d \
      -e MYSQL_ROOT_PASSWORD=password \
      -e MYSQL_DATABASE=todo \
      -p 3306:3306 \
      mysql:latest
  

• This will start a MySQL container with:

  • Database Name: todo

  • Root Password: password

  • Port: 3306 (exposed to your local machine)

    Make sure port 3306 is not already in use. If it is, update the port accordingly.

🧭 Let’s Start Exploring the Database

Before we dive into code, let’s clarify what exactly we’ll be doing when working with the database from our Golang application.

Here are the key operations we’ll perform:

1. Establish a connection between our Go app and the MySQL database

2. Create the database schema — defining the table structure (rows and columns) for our ToDo items

3. Perform CRUD operations using Go:

   • Create new ToDos

   • Read existing ToDos

   • Update ToDos

   • Delete ToDos

🔗 Establishing Connection to the Database

To connect our Go application to a database, we need to use a library that enables communication between the two. Go offers multiple options for this—both native libraries and third-party libraries built by the developer community.

we’ll start by exploring how to connect to a database using Go’s native database/sql package. This standard library provides a generic interface for interacting with SQL databases. We will look at its advantages and limitations and then we will move towards more suitable options.

🧩 Database Connection with Go Native Library

To establish a connection between our Go application and a MySQL database, we need three essential components. Let’s walk through each one:

• Importing required library. We start by importing Go’s standard SQL library: database/sql. This package provides a generic interface for working with SQL databases. However, since it's database-agnostic, we need to tell Go which specific database we're using.

• Add the MySQL Driver. To work with MySQL specifically, we import the driver: _ "github.com/go-sql-driver/mysql"

  💡 How do you know to use _?
  Simple—just Google! Look up golang mysql database/sql and read the official docs or examples. This approach works not just here, but for any library or tool you use. You can try it for this as well. Give a try!!!

• Third things is DSN(DataSourceName). The DSN tells Go about connection details of database. It includes:

  • Database username

  • Database Password

  • What protocol it is using to connect

  • What is the database server url and Port

  • Database Name

  • Example :- dsn := "root:password@tcp(127.0.0.1:3306)/mysql?parseTime=true"

    • ?parseTime=true Defines Go to parse the date time in time.Time format rather byte or string format.

package main
import (
    "database/sql"
    "fmt"
    "time"
    _ "github.com/go-sql-driver/mysql"
)

func main() {
    // Setting DataSourceName(DSN) Details. "?parseTime=true" Defines Go to parse the date time in time.Time format rather byte or string format.
    // By default it parse in string, by using the "go-sql-driver/mysql" in build capability we are telling go to format the time in time.Time format so that we can use it anywhere in the program
    dsn := "root:password@tcp(127.0.0.1:3306)/mysql?parseTime=true"
    db, err := sql.Open("mysql", dsn)
    // Error Handling
    if err != nil {
        panic(err) // Stop the program immediately with give the error message
    }
    defer db.Close() // This make sure that all the db connection is closed once main exit no matter how it exit
    err = db.Ping() // We are trying to ping the DB to check DB is reachable. Not required in real world, use it for our demo
    // Error Handling    
    if err != nil {
        panic(err)
    }
    fmt.Println("Connected Successfully")

    // Declaring the variable to store the result
    var Id int
    var Zone string
    // Here QueryRow is part of our "go-sql-driver/mysql" library.
    // what we are saying here is Select 1 row(which is SQL Command) and Scan will capture the output then we ask go to store result in Memory Location(&) of testValue variable. So that the actual variable get updated instead of a copy of variable
    err = db.QueryRow("SELECT * FROM time_zone_name LIMIT 1").Scan(&Zone, &Id)
    if err != nil {
        panic(err)
    }
    fmt.Println("Test Query Result:", Id, Zone)
}

You can read through the code above, where I’ve added detailed comments explaining what each line does and why it’s used. This will help you understand the flow and purpose of each part of the connection process.

Once you're comfortable with the code, you can run it to test the database connection.

• Create a file called main.go

• Type the complete code into main.go. Make sure all imports and logic are included as discussed earlier.

• Open your terminal in the project directory and run the following commands:

  • go mod init example.com/test-db - This initializes your Go module.

  • go mod tidy - This downloads and adds all the required libraries and dependencies (like the MySQL driver).

  • go run main.go - This compiles and runs your code.

Limitations of go native library

• Manual SQL Queries: You must write full SQL queries yourself, even for simple operations.

• Error Handling Overhead: Requires explicit error checking after every DB operation.

• Manual Resource Management: You have to close rows and connections manually to avoid leaks.

• Limited Productivity Features: Offers no tools for validation, relationships, or query building.

• Repetitive boilerplate : We are forced to repeat the query multiple times at multiple location.

• And More

To overcome these issues, developers often use third-party libraries that provide a more developer-friendly experience—like GORM, sqlx, or ent. Lets explore what it is

🚀 Introduction to ORM (Object Relational Mapping):

ORM stands for Object Relational Mapping—a technique that allows developers to interact with the database using regular Go code instead of raw SQL. The ORM internally converts that code into SQL queries and executes them on your behalf.

One of the most popular and widely used ORMs in the Go ecosystem is GORM. It simplifies database operations, reduces boilerplate code, and makes your codebase cleaner and easier to maintain—especially as your project grows.

🧩 Database Connection using GORM:

Establishing connection to DB with GORM also follow the same three steps which we discussed in previous section. Only thing which changes is we import GORM library here

• Importing gorm library gorm.io/gorm

• Importing gorm sql diriver as we are using sql database gorm.io/driver/mysql

• DSN Connection details same as native database/sql connection. dsn := "root:password@tcp(127.0.0.1:3306)/mysql?charset=utf8&parseTime=True&loc=Local"

Let see how the code transforms here using GORM

package main

import (
    "fmt"
    "gorm.io/driver/mysql"
    "gorm.io/gorm"
)

// Step 1: Define a struct to map to the table 'time_zone_name'
type TimeZoneName struct {
    ID   int    `gorm:"column:ID"`   // Map to column "ID"
    Name string `gorm:"column:Name"` // Map to column "Name"
}

func main() {
    // Step 2: Define the DSN (Data Source Name) for MySQL with parseTime=true to parse datetime properly
    dsn := "root:password@tcp(127.0.0.1:3306)/mysql?parseTime=true"

    // Step 3: Use GORM to open a connection with MySQL using the DSN
    db, err := gorm.Open(mysql.Open(dsn), &gorm.Config{})
    if err != nil {
        // If connection fails, print the error and stop execution
        panic("failed to connect database: " + err.Error())
    }

    fmt.Println("Connected successfully using GORM")

    // Step 4: Declare a variable to hold the query result
    var tz TimeZoneName

    // Step 5: Use GORM's `First` function to retrieve the first record from the 'time_zone_name' table
    // This performs: SELECT * FROM time_zone_name LIMIT 1
    result := db.Table("time_zone_name").First(&tz)

    // Step 6: Handle query error (if no rows found or any other issue)
    if result.Error != nil {
        panic("Query failed: " + result.Error.Error())
    }

    // Step 7: Print the result
    fmt.Println("Test Query Result:", tz.ID, tz.Name)
}

From the above code, we can see how fetching values from the timezone field transforms into clean, user-friendly Go code. GORM handles all the behind-the-scenes conversions automatically. While this might seem simple for a basic query, GORM’s benefits become even more evident as query complexity grows, making database interactions much easier and more maintainable.

Once you're comfortable with the code, you can run it to test the database connection.

• Create a file called main.go

• Type the complete code into main.go. Make sure all imports and logic are included as discussed earlier.

• Open your terminal in the project directory and run the following commands:

  • Let’s download the Required package for our app using below command

    • go get -u gorm.io/gorm

    • go get -u gorm.io/driver/mysql

  • go mod init example.com/test-db - This initializes your Go module.

  • go mod tidy - This downloads and adds all the required libraries and dependencies (like the MySQL driver).

  • go run main.go - This compiles and runs your code.

Wrapping Up

This concludes our article on working with databases in Go. While there is much more to explore, I hope this overview helps you get started with building your ToDo app.

Quick Summary of What We Covered So Far:

• Why a database is essential for our ToDo app project — to store, retrieve, and manage persistent data efficiently.

• The various options available in Go to connect with databases, including the native database/sql package and third-party ORMs.

• How to use Go’s native database/sql package to establish connections and run queries against the database.

• The limitations of the native package, such as verbose boilerplate code and manual data scanning.

• How Object-Relational Mapping (ORM) tools like GORM solve these limitations by automating SQL generation and data mapping.

• A hands-on example showing how to use GORM to connect to a database, define models, and perform queries more conveniently.

With these foundations in place, you are now equipped to build the backend for your ToDo app with confidence, leveraging Go’s ecosystem for effective database interaction.

Next Step

With the foundational knowledge we've built so far, it's time to put it into practice!

In the upcoming sessions, we will start working on our ToDo project by:

• Establishing a robust connection to the database using GORM, ensuring efficient and safe access.

• Configuring Go to perform CRUD operations — Create, Read, Update, and Delete — to manage our ToDo tasks seamlessly.

• Implementing proper error handling, validations, and data modeling tailored to our project’s needs.

This practical approach will bring our ToDo app to life, bridging the gap between theory and real-world application.

🧠 Pre-requisites

Before jumping into this project, we expect you to have a basic understanding of Golang, including:

• Basic syntax and structure

• Variables and data types

• Arithmetic and logical operators

• Conditional statements (if, switch)

• Loops (for)

• Functions and how to define/use them

If you're just starting out with Go and haven't covered these fundamentals yet, we recommend learning them first. You can find great beginner-friendly resources here , and once you're comfortable, come back and join us on this hands-on journey.

Note: We’re intentionally keeping this app simple to focus on learning. As we progress, you might notice some discrepancies or missing best practices—that’s by design. We’ll note these issues along the way and revisit them after we've developed a deeper understanding of Go. For now, the goal is to get started without getting overwhelmed.

📝 What Are We Building?

We’re developing a basic ToDo list application that allows users to manage their tasks efficiently. The app will support the standard set of CRUD operations—Create, Read, Update, and Delete ToDos.

What is ToDo app ?

A ToDo app is a simple task management application that helps users keep track of things they need to do.

🔗 Note: We are not building a frontend or UI for this app. All interactions will happen through RESTful APIs. This will help us focus purely on backend development using Golang, and better understand how to build, structure, and manage APIs effectively

🧩 Design and Requirements

Before diving into the implementation, let’s understand the design requirements of our ToDo application and the reasoning behind each decision.

🔍 What Do We Need?

• A place to store ToDo data

• A Way to Access the content Database

• A Way for Users to Interact with the App

Let’s see how we are planning that

🗃️ 1. Data Storage — Using MySQL

The first thing our app needs is a place to store the ToDo items. For this, we’ll use a database, as it's the most common and reliable way to store structured data in applications. Specifically, we’ll use a MySQL database

💡 Why SQL?

There are two main types of databases:

• SQL (Structured Query Language) – stores data in a tabular format

• NoSQL (Not only SQL) – stores data in document or key-value format

We're choosing SQL (MySQL) for the following reasons:

• It’s faster and more efficient for structured data.

• Our data is simple and relational (each ToDo has fields like title, description, status).

• We don’t require the flexibility of a document-based (NoSQL) structure for this use case.

• It fits the purpose of a basic CRUD app with limited and predictable data.

🔌 2. Database Access — Using Golang’s Standard Library

To interact with our database and access the stored data, we need a way to connect to the database and execute SQL queries.

We’ll use one of Go’s standard libraries together with a MySQL driver. This setup allows us to:

• Open and manage database connections

• Execute SQL commands (INSERT, SELECT, UPDATE, DELETE)

• Handle errors and manage resources efficiently

We'll explore the exact setup and usage of these libraries in the later section.

🌐 3. Application Access — Through HTTP Endpoints

Now, as an end-user, how do we interact with our application?

We’ll expose a set of HTTP endpoints that clients (like browsers, Postman, or frontend apps) can call using URLs.

For example:

GET    https://<your-app>/todos         → Get all ToDos  
POST   https://<your-app>/todos         → Create a new ToDo  
GET    https://<your-app>/todos/{id}    → Get a specific ToDo  
PUT    https://<your-app>/todos/{id}    → Update a ToDo  
DELETE https://<your-app>/todos/{id}    → Delete a ToDo

These endpoints will act as entry points to different operations within our app. Internally, each endpoint will trigger logic that interacts with the database and returns appropriate responses.

🧠 Wrapping Up

Now, we have a high-level understanding of what we need to build our ToDo app—and more importantly, why we need each component.

Don’t worry if everything doesn’t fully click yet. As we build more apps and go deeper into each concept, things will start to make sense. The key is to keep going.

🚀 What’s Next?

In the next article, we’ll focus entirely on the Database Layer of our application. Here's what we’ll cover:

• How to set up a MySQL database locally

• How to connect to the database using Golang

• Different approaches to database interaction in Go—and why we’re choosing a specific one

• Step-by-step guide to implement the database layer in our ToDo app

We’ll apply everything we learn directly into our project, laying a strong foundation for how our app stores and retrieves data.

I've been learning Golang for a while now—but despite covering the basics multiple times, I still didn't feel confident enough to build a full application. At first, I thought I just wasn’t grasping the fundamentals properly. So I went back and revised the basics again. But even that didn’t give me the confidence I was looking for.

That’s when I realized the issue wasn’t with what I knew, but with how I was learning.

What worked for me in the past with other technologies was this: learning by doing. So I decided to take that same approach with Go. Instead of only reading or watching tutorials, I’ll build real applications, face real problems, and learn the concepts as I solve them. If you’re in the same boat—feeling stuck after learning the syntax but unsure how to apply it—this series is for you. Let’s walk this path together, learn practically, and gain confidence by building.

What to Expect ?

In this series, I’ll be building multiple small-to-medium apps using Golang. For each project, I’ll break down:

• What components it needs and why

• Why each component or library is chosen

• Component syntax, pros, and limitations

• How each part is implemented

I will discuss a detailed walkthrough of each decision—why a certain piece of code is written the way it is, what alternatives were considered, and how everything fits together. My goal is to provide a deeper understanding of not just how to build with Go, but why certain practices or patterns are followed. So when you encounter similar problems in the future, you’ll have the insight to adapt and solve them.

How to Follow the Series ?

Each topic/project in this series will be divided into three parts:

1. Understanding the Layer We’ll start by identifying the different layers in the app—like data, service, handler, etc.—and answer:

• What is the purpose of this layer?

• What role does it play in the app?

• Why do we need it?

2. Learning the Concepts Next, we’ll focus on the specific components or modules in Go:

• How do we implement them in Go?

• What tools or libraries are available?

• Why did we choose a specific tool or method?

3. Putting It All Together Finally, we’ll implement the component inside our project:

• Connecting the dots from concept to code

• Solving real problems using the ideas we’ve discussed

• Making sure we understand how and why it works

If you’re someone who learns better by building, not just reading, you’ll feel right at home. Let’s code, break things, fix them, and grow—together.