We are covering the fundamentals of AI so that we can build a strong foundation before moving on to more advanced AI topics. In the previous blog, we explored one of the most important concepts in AI Tokens. We discussed what tokens are, the difference between input tokens and output tokens, how tokens are calculated, and why writing efficient prompts can help reduce token usage while getting better results.

If you're just joining this series, I highly recommend reading the previous blog first, as the concepts we'll discuss here build directly on top of it.

With that said, let's move on to the next building blocks of AI Context Window and Attention Horizon. These two concepts determine how much information an AI model can remember during a conversation and how effectively it can use that information to generate meaningful responses.

Let's dive in.

Context Window

Now that we know how data enters an AI model in the form of tokens let's explore where those tokens live while the model is processing your request.

Think of the Context Window as the model's working memory. It contains everything the model can "see" at a given moment before generating a response.

Every AI model has a hard limit on the number of tokens it can process in a single interaction. This limit is called the Context Window.

For example:

• Older models could process around 8,000 tokens.

• Modern frontier models can process 128K, 200K, or even over 1 million tokens in a single context window.

When you type a prompt, it is first split into tokens. These tokens are then assigned token IDs and converted into numerical vectors that the AI model can understand and process. In simple terms, tokens are just small pieces of text that are transformed into numbers before the model begins generating a response.

Almost every AI model specifies its context window in its official documentation. For example, here's the documentation for Gemma, where you can see its supported context window:

Why the Context Window Matters

The quality of your input has a direct impact on the quality of the model's output.

A common misconception is that the Context Window refers only to the input prompt. It doesn't.

The context window includes:

• Your input tokens

• Previous conversation (if applicable)

• The model's generated output

For example, if a model supports a 128K token context window and generates a 4K token response, then approximately 124K tokens remain available for your input and conversation history.

This is why context management becomes increasingly important when building AI applications.

Now that you understand what a context window is, the next question you're probably asking is:

Does a Larger Context Window Mean Better Memory?

Not necessarily. Just because a model can accept a massive log file or an entire codebase doesn't mean it understands or remembers every part of it equally well.

This brings us to an interesting phenomenon known as Lost in the Middle.

Lost in the Middle

Research has shown that many Large Language Models exhibit a behavior called Lost in the Middle.

They are generally very good at remembering information placed:

• At the beginning of a prompt

• At the end of a prompt

However, information buried somewhere in the middle of a very long prompt is more likely to receive less attention, making it easier for the model to overlook.

Let's take a real world example.

Imagine one of your production workflows fails. Instead of extracting only the relevant error logs, you send the entire log file to the LLM. The log contains tens of thousands of lines, and the actual error occurred somewhere in the middle.

There's a good chance the model may miss the most important error or fail to connect it with the surrounding events. As a result, it might provide an incomplete diagnosis or even suggest the wrong root cause.

This behavior is commonly referred to as Lost in the Middle.

Note: This doesn't happen every time, and newer models are continuously improving. However, it is still a well-observed limitation when working with very large contexts.

How the Context Window Impacts Performance

When working with large inputs, there are two important limits to understand.

Context Window (Input + Output Limit)

This is the total number of tokens the model can process during a single interaction, including both the prompt and the generated response.

Maximum Output Tokens

This is the maximum number of tokens the model is allowed to generate in its response.

For example, a model might accept 128,000 tokens of context but be configured to generate only 4,000 output tokens. These are two different limits and are often confused.

If you exceed the available context window, the model cannot process the extra information. Depending on the model, it may reject the request, truncate older content, or lose important context. Even before reaching the hard limit, extremely large prompts can reduce response quality because of the Lost in the Middle effect.

Interestingly, you don't always need to reach the hard limit to notice a drop in quality. With very large prompts, models can suffer from the "Lost in the Middle" effect, where information placed in the middle of a long context receives less attention than content at the beginning or the end. This can lead to missed details or less accurate responses.

Have you ever been in a long conversation with an AI where everything starts off perfectly, but after dozens of prompts it begins forgetting details, asks questions you've already answered, or even stops accepting more input? That's usually because the conversation is approaching the model's context window limit.

Every prompt you send and every response the AI generates consumes part of this limited space. Once it's full, the application must either remove older messages, replace them with a summary, or reject additional input altogether. So, the AI isn't getting "confused" it's simply running out of room to keep the entire conversation in memory.

So, How Do We Avoid Lost in the Middle?

Fortunately, there are several techniques that significantly improve both accuracy and cost.

Trimming and Slicing

Instead of sending a raw 50,000 line log file, extract only the relevant sections.

For example, search for keywords such as: ERROR CRITICAL FAIL EXCEPTION

Then include a few lines before and after the matching entries to preserve context.

This gives the model exactly what it needs without overwhelming it.

Place Important Information Strategically

If you must include a large amount of reference data, place your most important instructions or questions at the beginning or end of the prompt rather than burying them in the middle.

This increases the likelihood that the model will pay attention to the information that matters most.

Use Retrieval Instead of Sending Everything

In production AI applications, it's usually better to retrieve only the relevant documents, log snippets, or code sections before sending them to the LLM.

This approach reduces token usage, lowers cost, improves accuracy, and minimizes the chances of the model getting "lost in the middle."

There are many more advanced techniques used to manage context efficiently, which we'll explore in upcoming blogs.

Attention Horizon

At this point, you might be wondering:

If my file is only 100K tokens and the model supports a 128K token context window, shouldn't it be able to read the entire file? If so, why does it sometimes miss information that's right in the middle?

The answer lies in something called the Attention Horizon.

While the Context Window defines how much information the model can see, the Attention Horizon describes how effectively the model can focus on different parts of that information

Think of it this way:

• Context Window = Everything the model can read.

• Attention Horizon = What the model can effectively focus on while generating the next token.

Imagine someone pours 10,000 balls onto a football field. Only 30 of them are red, and the remaining 9,970 are white and blue. The red balls are there you haven't lost them but spotting all 30 quickly is difficult because they're scattered among thousands of others. Now imagine there are only 100 balls with the same 30 red balls. Suddenly, finding the red ones becomes much easier because there are far fewer distractions.

Large Language Models work in a similar way. Even if all the information fits inside the context window, the model has to distribute its attention across thousands of tokens. As the context becomes larger, important pieces of information especially those buried in the middle have to compete with everything else for attention. They aren't missing; they're simply harder for the model to focus on. This is why models can overlook details that are technically within the context window, this is one of the reasons why models sometimes exhibit the Lost in the Middle behavior.

Attention Horizon is not an official metric like the Context Window. Instead, it's a conceptual way of describing how effectively a model can focus on and make use of information distributed across its available context.

Since it's a conceptual term rather than a published specification, you won't find it listed in a model's documentation so don't bother searching for it. 🙂

A Real-World Example

Suppose you provide an LLM with:

• 40,000 lines of application logs

• Kubernetes manifests

• Terraform code

• CI/CD pipeline output

• Monitoring alerts

In this file, the actual root cause is hidden around line 22,000: java.lang.OutOfMemoryError: Java heap space

Although the model can technically read all of this information, its attention is spread across thousands of tokens. The important error may not receive enough focus, causing the model to miss it or provide a less accurate diagnosis.

Now imagine instead that you extract only:

• The error message

• 20 lines before it

• 20 lines after it

Now, almost every token in the context is relevant. The model can focus its attention much more effectively, resulting in a faster, cheaper, and more accurate response.

This is why a larger Context Window doesn't automatically guarantee better performance. The model still needs to decide where to focus its attention, and that's where the concept of Attention Horizon becomes important.

Summary

Let's quickly recap what we've learned:

• Context Window is the maximum amount of information (measured in tokens) that a model can process in a single interaction. It includes both your input and the model's generated output.

• A larger context window allows you to provide more information, but it doesn't guarantee that the model will understand or utilize every part of it equally well.

• Models can exhibit a behavior known as Lost in the Middle, where important information buried deep within a long prompt may receive less attention than information placed at the beginning or the end.

• Attention Horizon is a conceptual way to think about how effectively a model can focus on different parts of the context. While it's not an official model specification, it helps explain why simply increasing the context window doesn't always improve response quality.

• To get better results, send only the most relevant information. Trim unnecessary content, place critical instructions strategically, and retrieve only the sections the model actually needs instead of overwhelming it with massive inputs.

The key takeaway is simple:

A bigger Context Window lets the model see more, but a well-structured prompt helps the model understand more.