When developers first start working with Large Language Models, they often think in terms of words, sentences, and paragraphs.

Models do not.

Before text reaches an LLM, it is transformed into smaller units called tokens.

That sounds like an implementation detail.

It is not.

Tokens affect:

• how much information a model can process
• how much a request costs
• how much latency a request introduces
• how conversation history grows
• how retrieval systems are designed
• how much memory can be preserved
• how prompts should be structured
• how agent loops consume context over time

If you do not understand tokens and context windows, many later agent-engineering problems will seem arbitrary.

This lesson builds the mental model you will need for context engineering, memory, retrieval, cost control, and long-running agents.

---

1. Models Do Not Read Text the Way Humans Do

Suppose we write:

Agent engineering is powerful.

A human naturally sees four words:

Agent
engineering
is
powerful

An LLM typically does not receive those words directly.

Before the model processes the text, a tokenizer converts it into tokens.

Conceptually:

Text
  │
  ▼
Tokenizer
  │
  ▼
Token IDs
  │
  ▼
Model

A tokenizer may split the text into pieces that do not correspond exactly to words.

For example, a tokenizer could conceptually produce something like:

"Agent engineering is powerful."

→

"Agent"
" engineering"
" is"
" powerful"
"."

Another tokenizer might split it differently.

The exact tokenization depends on the model and tokenizer.

The important point is:

Tokens are the units the model actually processes.

---

2. What Is a Token?

A token is a unit produced by a tokenizer.

Depending on the tokenizer and input, a token may represent:

• an entire word
• part of a word
• punctuation
• whitespace combined with text
• a number
• a symbol
• part of a code identifier
• part of a non-English word

For example:

unbelievable

might be tokenized conceptually as:

un
believ
able

or:

unbelievable

depending on the tokenizer.

Similarly:

getCustomerById

might become several tokens:

get
Customer
By
Id

or something entirely different.

Do not assume:

1 word = 1 token

That relationship does not hold reliably.

---

3. Why Not Just Use Words?

Using whole words sounds simpler.

But natural language contains an enormous number of possible words and variations.

Consider:

run
runs
running
runner
runners
rerun

If every possible word required its own independent representation, the vocabulary would become extremely large and poor at handling new words.

Subword tokenization provides a useful compromise.

Instead of requiring every possible word to exist in the vocabulary, the tokenizer can construct unfamiliar words from smaller pieces.

Conceptually:

observability

might be represented as:

observ
ability

This allows the model to work with:

• rare words
• names
• technical terms
• new product names
• code
• multiple languages

without requiring every possible string to exist as one vocabulary item.

---

4. Tokenization Produces Token IDs

Models do not operate directly on strings.

A tokenizer maps pieces of text to numerical identifiers.

Conceptually:

"Hello Sakib"

might become:

[15496, 8273]

These numbers are illustrative.

The actual token IDs depend on the tokenizer.

The pipeline looks roughly like:

"Hello Sakib"
      │
      ▼
   Tokenizer
      │
      ▼
["Hello", " Sakib"]
      │
      ▼
[15496, 8273]
      │
      ▼
     Model

The model then converts these token IDs into internal numerical representations and processes them.

We do not need to study those internal representations mathematically for agent engineering.

What matters is understanding that the model's input is ultimately a sequence of tokens.

---

5. Tokenization Can Be Surprising

Consider:

hello

and:

 hello

The leading space may affect tokenization.

Likewise:

Java

and:

JavaScript

may not share token boundaries in the way you expect.

Numbers can also behave unexpectedly.

For example:

123456789

may become several tokens rather than one.

Code is especially interesting:

customerRepository.findById(customerId)

A tokenizer might break this into many pieces.

This means two strings with similar character lengths can consume very different numbers of tokens.

---

6. Characters, Words, and Tokens Are Different Measurements

Suppose we have:

The user wants to cancel the subscription.

We can measure it as:

Characters: ...
Words: 7
Tokens: model-dependent

These measurements are related but not interchangeable.

For rough English estimates, developers sometimes use approximate ratios such as:

1 token ≈ 3–4 English characters

or:

1 token ≈ 0.75 English words

These are only approximations.

They can become inaccurate for:

• code
• numbers
• URLs
• JSON
• uncommon names
• non-English languages
• heavily formatted text

Never use these approximations when an exact token count matters.

Use the tokenizer associated with the target model.

---

7. Different Models Can Use Different Tokenizers

Tokenization is not universally standardized.

Two models may receive the exact same text but tokenize it differently.

Conceptually:

Input:
"Agent engineering"

Model A:

["Agent", " engineering"]

Model B:

["Ag", "ent", " engineering"]

That means token counts are model-dependent.

This matters when:

• comparing providers
• calculating cost
• changing models
• enforcing context limits
• building model-routing systems

A system designed around one model's token behavior should not blindly assume another behaves identically.

---

8. Input Tokens and Output Tokens

LLM usage usually involves two major token categories:

Input Tokens
Output Tokens

Input tokens are everything sent to the model.

Output tokens are everything generated by the model.

Suppose the application sends:

System:
You are a helpful support assistant.

User:
My package arrived damaged. What should I do?

Those messages consume input tokens.

If the model responds:

I'm sorry your package arrived damaged. Please provide your order number...

that response consumes output tokens.

Conceptually:

Instructions
Conversation History
Retrieved Context
Tool Results
Current User Input
        │
        ▼
    Input Tokens
        │
        ▼
       LLM
        │
        ▼
    Output Tokens

Both sides matter.

---

9. The Context Window

A model cannot process an unlimited number of tokens in a single request.

It has a finite context window.

A context window represents the maximum amount of tokenized information the model can consider during one inference request.

Suppose a hypothetical model supports:

128,000 tokens

That does not necessarily mean you can send 128,000 input tokens and then generate unlimited output.

Typically, input and generated output must fit within the model's supported limits.

Conceptually:

┌──────────────────────────────────────┐
│            Context Window            │
│                                      │
│ System Instructions                  │
│ Conversation History                 │
│ Retrieved Documents                  │
│ Tool Results                         │
│ Current User Message                 │
│ Generated Output                     │
│                                      │
└──────────────────────────────────────┘

Everything competes for finite space.

---

10. Context Is More Than the User Message

A common mistake is to think:

context = current prompt

In real applications, the model may receive much more.

For example:

System Instructions
        +
Developer Instructions
        +
Conversation History
        +
Retrieved Knowledge
        +
Tool Definitions
        +
Tool Results
        +
Current User Message

All of this contributes to the context.

A model does not care whether something feels like "metadata" to you.

If it is represented as tokens in the request, it consumes context.

---

11. Tool Definitions Consume Context

This becomes especially important in agent systems.

Suppose an agent has access to these tools:

getCustomer
getOrder
updateAddress
cancelOrder
issueRefund
searchKnowledgeBase
sendEmail
createTicket

Each tool typically has:

• a name
• a description
• argument names
• argument types
• schema information

That information must usually be communicated to the model.

Conceptually:

{
  "name": "getOrder",
  "description": "Retrieve an order by its identifier",
  "parameters": {
    "type": "object",
    "properties": {
      "orderId": {
        "type": "string"
      }
    }
  }
}

That schema costs tokens.

Now imagine giving the model:

5 tools

versus:

150 tools

The larger tool set can consume significant context before the user has even said anything meaningful.

This is one reason tool design and tool selection matter in production agents.

---

12. Conversation History Grows Over Time

Suppose a conversation begins:

User:
My order hasn't arrived.

Assistant:
Can you provide your order number?

User:
8192.

Assistant:
I found order 8192. It is currently in transit.

User:
When should it arrive?

If the application wants the model to understand the conversation, it typically resends relevant previous messages.

Conceptually:

Request 1:

User message

Then:

Request 2:

User message 1
Assistant response 1
User message 2

Then:

Request 3:

User message 1
Assistant response 1
User message 2
Assistant response 2
User message 3

The history grows.

So token consumption can grow with conversation length.

---

13. The Model Does Not Automatically Remember Previous Requests

This is one of the most important ideas in this module.

Suppose we make this request:

User:
My name is Sakib.

The model replies:

Nice to meet you, Sakib.

Then your application makes a completely separate request:

User:
What is my name?

If the previous interaction is not included or represented through some state mechanism, the model does not inherently know the answer.

A stateless API interaction conceptually looks like:

Request A
   │
   ▼
Model
   │
   ▼
Response A

Request B
   │
   ▼
Model
   │
   ▼
Response B

There is no magical bridge between them.

To preserve conversational continuity, the application may send:

User:
My name is Sakib.

Assistant:
Nice to meet you, Sakib.

User:
What is my name?

Now the model has the relevant information in its current context.

This distinction becomes fundamental when we study memory.

---

14. Context Is Not the Same as Memory

Developers often use the word "memory" loosely.

If the application simply sends previous messages again:

Message 1
Message 2
Message 3
Message 4

the model is not remembering them internally between requests.

The application is reconstructing the context.

A useful distinction is:

Context
=
information currently available to the model

Whereas:

Memory
=
a mechanism for preserving information across interactions
and making relevant information available later

Memory may eventually involve:

• conversation history
• databases
• summaries
• user profiles
• vector search
• checkpoints
• external state stores

We will explore those later.

For now:

If information is not present in the model's current context, the model cannot directly reason over it.

---

15. What Happens When Context Becomes Too Large?

Suppose your application accumulates:

System prompt           4,000 tokens
Conversation history   60,000 tokens
Retrieved documents    45,000 tokens
Tool definitions       10,000 tokens
Current request         2,000 tokens

Total:

121,000 tokens

If your model supports a 128,000-token context window, you are already close to the limit.

You still need room for output.

If additional content pushes the request past the model's supported limit, several things can happen depending on the provider and application:

• the request may fail
• the SDK may reject it
• older messages may be truncated
• your application may intentionally remove content
• the system may summarize earlier information
• retrieval may be reduced

This is not merely a UI concern.

It is an architectural constraint.

---

16. Truncation Can Change Behavior

Suppose an early system message contains:

Never issue refunds without approval.

Later the context grows extremely large.

If context management is implemented badly and early information is removed, the model may no longer see that instruction.

Similarly, imagine an earlier user message contains:

Order ID: 8192

If that message disappears from context, a later question:

Can you check its status again?

may become ambiguous.

Context loss can lead to behavioral changes.

Therefore context management cannot simply mean:

drop random old messages

We need strategies.

That becomes a major topic later in the course.

---

17. Larger Context Windows Do Not Eliminate Context Engineering

It is tempting to think:

If models support huge context windows, just send everything.

That is usually poor engineering.

Even when the model technically accepts the data, sending unnecessary context can create problems.

Cost

More input tokens can increase cost.

Latency

Larger requests may require more processing.

Relevance

Irrelevant information can distract from the actual task.

Conflicting Information

Old and new data may contradict each other.

Security

More context can mean exposing more sensitive information.

Quality

More information is not automatically better information.

The correct goal is not:

maximum context

The goal is:

minimum sufficient context

That principle will become central in Context Engineering.

---

18. Context Engineering Begins with Selection

Imagine a support agent has access to:

customer profile
every order ever placed
10 years of support tickets
entire company documentation
all internal policies
all product manuals
current conversation

Should we send all of it on every request?

No.

Instead we should ask:

What information is relevant to this decision?

For example, if the user asks:

Where is order 8192?

the model may need:

current user
order 8192
shipment status
relevant conversation context

It probably does not need:

all other customer orders
entire refund policy
product catalog
10 years of support history

Context selection is an engineering problem.

---

19. Tokens Have Economic Consequences

Suppose a provider charges separately for:

input tokens
output tokens

Then a request costing:

1,000 input tokens
200 output tokens

is economically different from:

100,000 input tokens
2,000 output tokens

Now imagine:

1 request per user action

versus an agent that performs:

8 model calls

for one user action.

The cost multiplies.

Agentic systems can produce variable execution paths:

User Request
    │
    ▼
Model Call
    │
    ▼
Tool Call
    │
    ▼
Model Call
    │
    ▼
Tool Call
    │
    ▼
Model Call
    │
    ▼
Final Response

Each model call may include some or all of the accumulated context.

This means token economics become part of system design.

---

20. Repeated Context Can Become Expensive

Consider a system prompt of:

5,000 tokens

If one agent execution makes:

10 model calls

and the same system prompt is included every time, that could conceptually mean:

50,000 input tokens

from the system prompt alone.

The same concern applies to:

• tool definitions
• conversation history
• retrieved documents
• policies
• examples

This is why agent cost is not simply:

number of user requests

A better model is:

user requests
×
model calls per request
×
tokens per model call
×
model price

Production systems need visibility into all four.

---

21. Output Length Matters Too

Developers sometimes focus only on input size.

But output also consumes tokens.

Suppose a model is asked:

Explain the entire architecture in exhaustive detail.

It may generate thousands of tokens.

If the application only needs:

{
  "category": "BILLING"
}

allowing a long natural-language response is wasteful and harder to consume.

We should often constrain output according to the application's needs.

For example:

Classification task
→ short structured output

rather than:

Classification task
→ 2,000-word explanation

This improves:

• cost
• latency
• parsing
• reliability
• usability

---

22. Context Windows Create Architectural Trade-offs

Suppose we are building an agent that analyzes a large codebase.

The repository contains millions of tokens.

Even a very large context window cannot necessarily hold the entire repository.

So we need strategies such as:

search
retrieval
indexing
summarization
chunking
hierarchical analysis

Likewise, an agent handling years of customer history cannot simply carry the entire history in every model call.

This is why systems such as RAG exist.

RAG is not merely a way to "give the model more knowledge."

It is partly a solution to this problem:

Select relevant information from a much larger information space and place only useful information into the current context.

We will study this properly later.

---

23. Chunking Exists Because Context Is Finite

Suppose we have a document containing:

500,000 tokens

but the model's usable context for the task is much smaller.

We may divide the document into smaller pieces:

Document

   │
   ├── Chunk 1
   ├── Chunk 2
   ├── Chunk 3
   ├── Chunk 4
   └── ...

Then retrieve relevant chunks.

The quality of chunking can affect what information the model receives.

If we split carelessly:

"The refund is allowed only if the product..."

--- chunk boundary ---

"...was purchased within 30 days."

retrieving only one half could remove critical meaning.

So even a seemingly simple operation such as text splitting becomes an engineering decision.

---

24. Code Can Be Token-Heavy

Agent systems increasingly work with code.

Consider:

public CustomerResponse getCustomerByOrganizationAndExternalId(
        UUID organizationId,
        String externalCustomerId
) {
    return customerRepository
            .findByOrganizationIdAndExternalCustomerId(
                    organizationId,
                    externalCustomerId
            )
            .map(customerMapper::toResponse)
            .orElseThrow(CustomerNotFoundException::new);
}

This may consume significantly more tokens than a short natural-language description.

Why?

Because code contains:

• identifiers
• punctuation
• operators
• whitespace
• symbols
• repeated syntax

Tokenization behavior varies, but code can be expensive in context.

For coding agents, repository context must therefore be managed deliberately.

---

25. JSON and Tool Results Can Also Become Large

Imagine a tool returns:

{
  "orders": [
    {
      "id": "...",
      "customer": "...",
      "items": [...]
    }
  ]
}

Now imagine it returns:

10,000 orders

Do we really want to put the complete tool response into the model context?

Usually not.

Instead the application might:

• filter the data
• aggregate it
• paginate it
• summarize it
• expose more specific tools
• return only necessary fields

Tool design and context design are closely related.

A badly designed tool can flood the context with irrelevant data.

---

26. Good APIs for Humans Are Not Automatically Good Tools for Models

Suppose an internal API returns:

{
  "id": 8192,
  "customer": {...},
  "order": {...},
  "payments": [...],
  "auditEvents": [...],
  "inventory": [...],
  "shipping": {...},
  "marketing": {...}
}

A frontend might find this convenient.

But an agent asking:

What is the current shipment status?

probably needs only:

{
  "orderId": 8192,
  "shipmentStatus": "IN_TRANSIT",
  "estimatedDelivery": "..."
}

Smaller tool outputs can improve:

• token usage
• relevance
• privacy
• model accuracy
• latency

Therefore tool interfaces should often be designed specifically for agent use.

We will return to this in the Tool Calling module.

---

27. Token Budgets

A useful design technique is to think in terms of a token budget.

Suppose a model supports a context window of:

128,000 tokens

We should not necessarily design around consuming all of it.

We might conceptually allocate:

System instructions       5,000
Tool definitions          8,000
Conversation history     20,000
Retrieved context        40,000
Current request           2,000
Reserved output           8,000
Safety margin            remaining

These are illustrative numbers.

The point is not the exact distribution.

The point is to intentionally reason about how much context each category is allowed to consume.

Without budgets, context tends to grow until something breaks.

---

28. Reserve Space for Output

Suppose your model supports:

128,000 tokens

and your input already consumes nearly the entire limit.

What happens when the model needs to generate a response?

You may have left insufficient room.

Therefore systems often need to reserve capacity for output.

Conceptually:

Total Context Capacity
        │
        ├── Input Budget
        │
        └── Output Budget

If you expect:

up to 4,000 output tokens

do not blindly fill every available token with input.

---

29. Long Context Can Hide Important Information

Even when information technically fits in the context window, that does not guarantee the model will use every piece equally well.

Imagine a 100,000-token prompt where one critical sentence says:

Never automatically refund orders above €500.

and that sentence is buried among thousands of unrelated details.

A huge context window solves the hard capacity limit.

It does not eliminate the problem of attention and relevance.

Good context engineering tries to make important information:

• relevant
• clear
• discoverable
• non-conflicting
• appropriately positioned

Again:

more context ≠ better context

---

30. Tokenization and Multilingual Applications

Token efficiency can vary significantly across languages.

Some languages may require more tokens than English to represent equivalent semantic content, depending on the tokenizer.

This has practical consequences for multilingual systems.

Suppose:

English conversation

uses:

10,000 tokens

while an equivalent conversation in another language uses substantially more.

Then:

• cost can differ
• context capacity can differ
• truncation can happen sooner
• latency may differ

If your application serves multiple languages, measure actual token usage rather than assuming equivalent behavior.

---

31. Tokenization and User-Controlled Input

Users can send extremely large inputs.

For example:

Summarize this:
<2 million characters>

A production application should not simply forward arbitrary input to the model.

We may need:

request size limits
token limits
document preprocessing
chunking
upload restrictions
cost controls

Otherwise a single request can become unexpectedly expensive or fail entirely.

LLM inputs need resource limits just like other production inputs.

---

32. Token Usage Is an Observability Signal

Imagine an agent normally uses:

8,000 tokens per execution

Then after a deployment, average usage becomes:

60,000 tokens per execution

Even if responses still appear correct, something may have gone wrong.

Perhaps:

• conversation history is being duplicated
• tool responses are too large
• retrieval returns too many documents
• prompts became bloated
• an agent loop is repeating

Token usage can therefore function as an operational metric.

Eventually we may observe:

input tokens / execution
output tokens / execution
model calls / execution
tokens / tool step
cost / execution

Those metrics help us understand both economics and behavior.

---

33. Token Count Is Not Quality

It is easy to assume:

more tokens → more reasoning → better answer

That is not a reliable rule.

A concise prompt containing exactly the relevant information can outperform a huge prompt filled with irrelevant material.

Likewise, a longer response is not necessarily a better response.

Good agent engineering optimizes for:

useful information

not:

maximum token consumption

---

34. Context Is a Working Set

A useful analogy for backend engineers is to think of context as a working set.

A database may contain terabytes.

But a request usually operates on a small subset.

Similarly:

All Available Knowledge
          │
          ▼
Context Selection
          │
          ▼
Current Working Context
          │
          ▼
         LLM

The model does not need every piece of information available to your organization.

It needs the right information for the current task.

This mental model becomes extremely useful when designing:

• RAG
• memory
• tools
• conversation history
• agent workflows

---

35. A Practical Java Mental Model

We are not yet building the full application, but conceptually a Java service might construct something like:

List<Message> messages = List.of(
        new SystemMessage("""
                You are a customer support assistant.
                Answer only using the information provided.
                """),
        new UserMessage("""
                Where is my order?
                """)
);

Before the model sees those strings:

Java Strings
    │
    ▼
Request Serialization
    │
    ▼
Provider API
    │
    ▼
Tokenizer
    │
    ▼
Tokens
    │
    ▼
Model

When libraries eventually expose:

model.chat(messages);

remember that a tokenized request exists underneath that abstraction.

---

36. The Context Lifecycle

A useful model for future agent systems is:

Available Information
        │
        ▼
Select
        │
        ▼
Transform
        │
        ▼
Tokenize
        │
        ▼
Fit Within Budget
        │
        ▼
Send to Model
        │
        ▼
Generate Output
        │
        ▼
Store / Discard / Summarize
        │
        ▼
Build Next Context

Notice that context is not static.

It evolves over the lifetime of an interaction.

Long-running agent systems therefore need an explicit context-management strategy.

---

37. Common Mistake: Sending Everything

A beginner implementation often looks conceptually like:

String context =
        allCustomers
        + allOrders
        + allDocumentation
        + entireConversation
        + everyToolResult;

Then:

model.generate(context);

This can work in a demo.

It does not scale well.

Good systems answer:

What is necessary for this step?

and provide that.

This distinction becomes increasingly important as your agent gains more capabilities.

---

38. Common Mistake: Keeping the Entire Conversation Forever

Another common implementation is:

messages.add(userMessage);
messages.add(assistantMessage);

forever.

Eventually:

100 messages
500 messages
5,000 messages

This creates several problems:

• token growth
• rising cost
• rising latency
• stale instructions
• conflicting information
• irrelevant history
• context overflow

A mature system might instead combine:

recent messages
+
summary of earlier conversation
+
stored user facts
+
retrieved relevant memories

We will explore those strategies in later modules.

---

39. Common Mistake: Treating Context Size as a Provider Problem

If a request exceeds the context window, it is easy to say:

We need a model with a bigger context window.

Sometimes that is correct.

But often the deeper issue is poor architecture.

For example:

Why are we sending 50,000 irrelevant tokens?

Why does this tool return 20 MB of JSON?

Why are we duplicating conversation history?

Why are all 100 tools available for every step?

Why are we retrieving 80 documents?

A larger context window can hide inefficient design.

It does not fix it.

---

40. Context Windows Affect Agent Design

Consider two agent architectures.

Agent A

Every step receives:

entire conversation
all tool definitions
all tool results
all retrieved documents

After several steps, context grows rapidly.

Agent B

Every step receives:

current objective
relevant recent history
necessary state
only appropriate tools
selected tool results

Agent B usually has a much healthier context profile.

This is why context engineering is not merely prompt writing.

It is architecture.

---

Practical Exercise — Estimate What Consumes Context

Consider this agent request:

System instructions:
2,000 tokens

Tool definitions:
4,000 tokens

Conversation history:
12,000 tokens

Retrieved documentation:
20,000 tokens

Current user message:
500 tokens

Answer:

1. Approximately how many input tokens are being sent?
2. Which category consumes the most context?
3. If the request needs a maximum of 5,000 output tokens, how would you think about the total context requirement?
4. Which parts might you optimize first if cost becomes a problem?

Do not search for a framework solution.

Reason about the architecture.

---

Practical Exercise — Choose Relevant Context

A customer asks:

When will order 8192 arrive?

Available data includes:

Customer profile
Order 8192
Customer's other 47 orders
Shipment tracking for order 8192
Company refund policy
Complete product catalog
Customer's last 200 support messages
Shipping provider documentation
Current conversation

Choose what you would include in the model context.

Then explain why you excluded the rest.

Your goal is:

minimum sufficient context

not maximum information.

---

Practical Exercise — Find the Context Bug

Imagine a support assistant behaves correctly at the beginning of a conversation.

After 100 messages, it begins forgetting important policies.

The system currently works like this:

List<Message> messages = new ArrayList<>();

messages.add(systemMessage);

for (ConversationMessage message : entireConversation) {
    messages.add(convert(message));
}

When the request becomes too large, the application does:

while (estimateTokens(messages) > maxTokens) {
    messages.remove(0);
}

What is dangerous about this strategy?

Think carefully about what exists at index 0.

Design a safer high-level approach.

---

Practical Exercise — Tool Output Design

Suppose an agent needs to answer:

What is the status of order 8192?

You have two possible tools.

Tool A

getCompleteCustomerAccount(customerId)

Returns:

customer profile
all addresses
all orders
all payments
all subscriptions
all support tickets
all audit events

Tool B

getOrderStatus(orderId)

Returns:

{
  "orderId": 8192,
  "status": "SHIPPED",
  "estimatedDelivery": "2026-08-12"
}

Which tool is better for this specific problem?

Explain your answer in terms of:

• context size
• relevance
• security
• model behavior
• interface design

---

Design Exercise — Long Conversation Architecture

You are building a customer-support assistant.

Some customers may have conversations lasting several months.

Design a high-level context strategy that supports long-running conversations without sending every historical message on every request.

You may use concepts such as:

recent history
summaries
database state
retrieval
user profile

You do not need to implement them yet.

Draw the architecture and explain what would enter the model's context for a normal request.

---

Questions You Should Be Able to Answer

Before moving to the next lesson, make sure you can answer these clearly.

1. What is a token?

A token is a unit produced by a tokenizer and used as part of the model's input and output representation.

2. Is one word always one token?

No.

Words may consist of one or several tokens, and token boundaries depend on the tokenizer.

3. What is a context window?

The context window is the finite amount of tokenized information a model can process within a request, subject to the model's input/output limits.

4. What consumes context?

Potentially all model-visible content, including:

• instructions
• messages
• conversation history
• tool definitions
• tool results
• retrieved documents
• user input
• generated output

5. Does a model automatically remember previous API calls?

No.

Relevant information must be provided again or restored through an application-level state or memory mechanism.

6. Why shouldn't we simply send every available piece of information?

Because unnecessary context can increase cost, latency, security exposure, confusion, and the risk of exceeding model limits.

7. What is a better goal than maximizing context?

Provide the minimum sufficient context for the current task.

8. Why do tokens matter economically?

Many model providers charge based partly on input and output token usage.

9. Why can agent systems consume many more tokens than simple chat applications?

One user request may trigger multiple model calls, each of which may carry accumulated instructions, tools, history, and tool results.

10. Why is context management an architectural concern?

Because the application must decide what information is selected, preserved, summarized, retrieved, removed, or exposed to the model over time.

---

Key Takeaways

Large Language Models do not directly process words or paragraphs.

They operate on tokens produced by a tokenizer.

Every model has finite limits on how much tokenized information it can process.

That information may include much more than the user's current message:

Instructions
History
Tools
Retrieved Knowledge
Tool Results
Current Input
Generated Output

Together, these form the model's working context.

This creates a fundamental engineering constraint:

The model cannot reason over information that is not available in its current context, and putting information into that context has a cost.

Good LLM applications therefore do not ask:

How much information can we fit?

They ask:

What information does the model need for this step?

As systems become agentic, this question becomes even more important because context can grow after every model call and tool interaction.

The core principle to remember is:

Context is a finite working set. Engineer it deliberately.

In the next lesson, we will examine how modern LLM applications organize that context through messages and conversational roles.

We will study system messages, user messages, assistant messages, tool messages, conversation history, instruction hierarchy, and what the model actually receives when an application appears to be having a continuous conversation.