Agent Engineering: From LLM Foundations to Production SystemsLesson 4 / 10

LLM Foundations

How LLM Generation Works

ReadingPreview

You are viewing a free preview lesson.

Large Language Models can produce answers that feel deliberate.

They can explain architecture, summarize documents, classify support requests, write code, compare options, and participate in multi-step problem solving.

Because the output is coherent, it is tempting to imagine that the model first forms a complete answer internally and then writes it down.

That is not the right mental model.

At generation time, an LLM typically produces output one token at a time.

For each new token, the model computes a probability distribution over possible continuations based on the context it currently has.

Then one token is selected.

That token is appended to the context.

The model repeats the process.

This mechanism is called autoregressive generation.

Understanding this process is essential because it explains:

  • why outputs can vary
  • why temperature changes behavior
  • why models can generate plausible but incorrect answers
  • why long outputs can drift
  • why deterministic testing is difficult
  • why stopping conditions matter
  • why agent decisions are probabilistic
  • why reliable systems need deterministic boundaries around model behavior

This lesson builds the generation model you will use throughout the rest of the course.

1. Generation Happens One Token at a Time

Suppose the model receives:

The capital of France is

It does not necessarily create the entire sentence:

The capital of France is Paris.

in a single operation.

Instead, conceptually, it evaluates possible next tokens.

For example:

Input:

"The capital of France is"

            │
            ▼
           LLM
            │
            ▼

Possible next tokens:

" Paris"      0.96
" Lyon"       0.01
" London"     0.003
" Berlin"     0.001
...

These probabilities are illustrative.

The model selects one token.

Suppose it selects:

 Paris

The context becomes:

The capital of France is Paris

The model runs again:

"The capital of France is Paris"

            │
            ▼
           LLM
            │
            ▼

Possible next tokens:

"."           0.82
","           0.07
" and"        0.04
...

Then another token is selected.

This continues until generation stops.

Conceptually:

Context
   │
   ▼
Predict Next Token
   │
   ▼
Select Token
   │
   ▼
Append Token
   │
   ▼
Updated Context
   │
   └───────────────┐
                   │
                   ▼
          Predict Again

This repeated process is the foundation of LLM text generation.

2. What Does "Predict the Next Token" Mean?

Suppose the current context is:

Java is a

The model may assign different probabilities to possible continuations:

" programming"    0.44
" popular"        0.22
" statically"     0.09
" language"       0.07
" powerful"       0.05
...

The model does not simply choose a continuation because it has a database record saying:

Java → programming language

Instead, the model has learned patterns from training that influence the probability distribution.

This distribution is conditional on the full available context.

Change the context:

Java is a type of

and the distribution changes.

Change it again:

In this course, Java is a

and the probabilities may change again.

The next-token prediction depends on everything the model is currently considering.

3. The Model Produces a Probability Distribution

Suppose there are five possible candidate tokens:

A
B
C
D
E

The model might assign:

A → 0.60
B → 0.20
C → 0.10
D → 0.07
E → 0.03

The values sum approximately to:

1.0

Conceptually:

Current Context
      │
      ▼
     Model
      │
      ▼
Probability Distribution
      │
      ├── A  60%
      ├── B  20%
      ├── C  10%
      ├── D   7%
      └── E   3%

The generation strategy determines how a token is selected from this distribution.

That selection step is where parameters such as temperature and sampling strategies become important.

4. The Highest-Probability Token Is Not Always Selected

A simple generation strategy could always select the token with the highest probability.

This is often called greedy decoding.

Given:

A → 0.60
B → 0.20
C → 0.10
D → 0.07
E → 0.03

greedy decoding selects:

A

every time.

Conceptually:

Probability Distribution
        │
        ▼
Choose Maximum
        │
        ▼
        A

This can produce more predictable output.

However, always selecting the highest-probability token can also lead to:

  • repetitive text
  • overly conservative generation
  • less diverse wording
  • locally optimal but globally awkward sequences

Many systems therefore use some form of sampling.

5. Sampling

With sampling, the model can select from multiple possible tokens based on their probabilities.

Given:

A → 0.60
B → 0.20
C → 0.10
D → 0.07
E → 0.03

A will still be selected most often.

But B, C, D, or E may occasionally be selected.

Conceptually:

Probability Distribution
        │
        ▼
Randomized Selection
weighted by probabilities
        │
        ▼
Selected Token

This introduces variation.

Two generations from the same context may therefore begin differently.

Once a different token is selected, every subsequent probability distribution may also change.

That means a small early difference can produce a significantly different final response.

6. Small Differences Can Compound

Imagine the model needs to continue:

A good approach would be to

Generation A selects:

 first

Generation B selects:

 begin

Now the contexts differ:

A good approach would be to first

versus:

A good approach would be to begin

The model calculates the next-token distribution separately for each context.

After several steps:

Path A:

first → validate → the → request → ...

while:

Path B:

begin → by → identifying → the → ...

Both may be reasonable.

But the paths can diverge substantially.

This is one reason LLM outputs can vary even when the original prompt appears unchanged.

7. Temperature

Temperature is a generation parameter commonly used to control how sharply the model favors high-probability tokens.

Conceptually, lower temperature makes the probability distribution more concentrated.

Higher temperature makes it flatter.

Suppose the original distribution is:

Paris      0.70
Lyon       0.15
London     0.10
Berlin     0.05

At a lower temperature, it might conceptually behave more like:

Paris      0.94
Lyon       0.04
London     0.015
Berlin     0.005

At a higher temperature:

Paris      0.45
Lyon       0.25
London     0.18
Berlin     0.12

Again, these values are illustrative.

The important idea is:

Lower Temperature
        │
        ▼
More probability concentrated
on likely tokens

while:

Higher Temperature
        │
        ▼
More probability available
to less likely alternatives

8. Lower Temperature Does Not Mean "Correct"

This is a very important distinction.

Developers sometimes think:

temperature = 0

means:

correct answer

It does not.

Lower temperature typically means the model behaves more consistently around its highest-probability continuations.

But if the model's highest-probability answer is wrong, lower temperature can simply make it consistently wrong.

For example:

Question
    │
    ▼
Model strongly prefers an incorrect answer
    │
    ▼
Low Temperature
    │
    ▼
Same incorrect answer repeatedly

Temperature controls generation variability.

It does not validate truth.

9. Higher Temperature Does Not Mean "Smarter"

Another common misconception is:

higher temperature
=
more intelligent reasoning

Not necessarily.

Higher temperature generally increases diversity and unpredictability.

That may be useful for tasks such as:

  • brainstorming
  • creative writing
  • generating alternatives
  • ideation

But it can be undesirable for:

  • classification
  • extraction
  • tool arguments
  • policy decisions
  • structured application outputs

For many agent-engineering tasks, we prefer controlled behavior over creativity.

10. Temperature Should Match the Task

Consider two tasks.

Task A

Write five creative names for a coffee shop.

Variation may be desirable.

A somewhat higher temperature may help produce diverse ideas.

Task B

Classify this support request as:

BILLING
ACCOUNT
DELIVERY
OTHER

We usually want consistency.

Lower randomness may be preferable.

A useful principle is:

Generation parameters should reflect the application's requirements, not personal preference.

11. Sampling Is More Than Temperature

Temperature is not the only possible generation control.

Providers may expose parameters such as:

top_p
top_k
frequency_penalty
presence_penalty
seed

Exact support varies by model and provider.

You do not need to memorize every parameter.

You do need to understand the architectural principle:

Generation behavior is partly controlled by decoding configuration, and provider APIs may expose different controls.

Let's look briefly at the most common concepts.

12. Top-K Sampling

With top-k sampling, only the k most probable candidate tokens are considered.

Suppose:

A → 0.40
B → 0.25
C → 0.15
D → 0.10
E → 0.06
F → 0.04

If:

k = 3

only:

A
B
C

remain eligible for sampling.

Conceptually:

Full Distribution
      │
      ▼
Keep Top K
      │
      ▼
Sample

Not every provider exposes top-k directly.

The concept is more important than the API parameter.

13. Top-P Sampling

Top-p, also known as nucleus sampling, works differently.

Instead of selecting a fixed number of tokens, it keeps the smallest set of likely tokens whose cumulative probability reaches a threshold.

Suppose:

A → 0.40
B → 0.25
C → 0.15
D → 0.10
E → 0.06
F → 0.04

With:

top_p = 0.80

we might keep:

A → 0.40
B → 0.25
C → 0.15

because:

0.40 + 0.25 + 0.15 = 0.80

Then sampling occurs within that candidate set.

Conceptually:

Probability Distribution
        │
        ▼
Keep likely tokens until
cumulative probability >= p
        │
        ▼
Sample

Again, exact implementation details vary.

14. Do Not Tune Parameters Blindly

A common beginner workflow is:

Response feels bad
      │
      ▼
Change temperature
      │
      ▼
Still bad
      │
      ▼
Change top_p
      │
      ▼
Change random parameter

This is not disciplined engineering.

Poor output may come from:

  • bad instructions
  • missing context
  • irrelevant context
  • weak model capability
  • incorrect tool data
  • ambiguous task definition
  • poor output schema
  • model limitations

Generation parameters are only one part of the system.

We should diagnose the actual cause.

15. What Does Temperature Zero Mean?

Many APIs allow something like:

temperature = 0

This is commonly used to request highly deterministic behavior.

But you should be careful with the word deterministic.

Even with temperature zero, exact reproducibility may not always be guaranteed.

Possible reasons include:

  • provider infrastructure changes
  • model updates
  • numerical differences
  • implementation details
  • nondeterministic serving infrastructure
  • routing between model versions
  • provider-specific decoding behavior

Therefore:

temperature = 0

should usually be understood as:

minimize sampling variability

not:

mathematically guarantee identical output forever

This distinction matters when designing tests.

16. Seeds and Reproducibility

Some providers expose a seed parameter.

Conceptually:

same input
+
same model
+
same configuration
+
same seed

may increase reproducibility.

But even then, provider documentation may not guarantee perfect long-term equivalence.

Why?

Because the underlying model or serving environment can change.

So if your application requires:

same input → exactly same result

an LLM may not be the right component for that responsibility.

Deterministic guarantees belong in deterministic code.

17. Autoregressive Generation

The repeated next-token process is called autoregressive generation.

Conceptually:

Token 1
  │
  ▼
Token 2
  │
  ▼
Token 3
  │
  ▼
Token 4
  │
  ▼
...

But each token depends on everything generated before it.

More accurately:

P(token_1 | context)

P(token_2 | context, token_1)

P(token_3 | context, token_1, token_2)

P(token_4 | context, token_1, token_2, token_3)

You do not need the mathematics for this course.

The conceptual lesson is enough:

Every generated token becomes part of the context used to generate the next token.

This explains why earlier mistakes can affect later output.

18. Generation Can Drift

Suppose the model starts with an incorrect assumption:

Order 8192 was delivered yesterday.

Then continues:

Since the order was delivered yesterday,
the customer is within the return window...

Then:

Therefore the refund should...

An incorrect early generation can become context for later reasoning.

Conceptually:

Incorrect Token / Statement
          │
          ▼
Becomes New Context
          │
          ▼
Influences Next Generation
          │
          ▼
Further Conclusions

This is one reason long reasoning chains should not automatically be trusted.

Fluent internal consistency can emerge from a false premise.

19. Probability Is Not Confidence

Suppose a model strongly prefers a particular next token.

That does not mean the model has measured the factual correctness of the entire answer.

The model's probability distribution tells us something about:

how likely a continuation is according to the model

not directly:

how likely the claim is to be objectively true

These are different concepts.

A sentence can be statistically plausible and factually wrong.

That distinction is at the heart of hallucination.

20. Why Hallucination Is Not Just a Bug

Suppose we ask:

Who invented the fictional XYZ-900 protocol in 1993?

There may be no such protocol.

But the structure of the question strongly suggests an answer format:

<Person> invented it in <organization>.

The model may generate something plausible because generation itself requires selecting continuations.

Conceptually:

Question assumes a fact
        │
        ▼
Model predicts plausible continuation
        │
        ▼
Confident-sounding answer

The model does not automatically stop and perform:

if (!factExists()) {
    return UNKNOWN;
}

unless its training, instructions, context, and generation behavior lead it there.

This is why systems that require factual correctness often need:

  • retrieval
  • tools
  • authoritative data
  • validation
  • abstention behavior
  • evaluation

21. Generated Reasoning Is Still Generated

Suppose a model produces:

Step 1: The order was purchased 20 days ago.

Step 2: The refund policy allows returns within 30 days.

Step 3: Therefore the order is eligible.

The reasoning looks structured.

But each part is still model-generated unless grounded in trusted information.

We must ask:

Where did "20 days ago" come from?

Where did the 30-day policy come from?

Are those facts current?

Were they retrieved from authoritative systems?

Structured reasoning does not remove the need for grounding.

22. Reasoning Models Do Not Remove the Reliability Problem

Some modern models perform more extensive internal reasoning before generating a final response.

These models can improve performance on complex tasks such as:

  • planning
  • coding
  • mathematics
  • multi-step analysis

But the system-design principle remains the same:

Better reasoning capability does not turn a probabilistic model into a deterministic source of truth.

You still need to consider:

  • grounding
  • authority
  • validation
  • tool permissions
  • evaluation
  • cost
  • latency

A stronger model changes capability.

It does not eliminate engineering responsibility.

23. Output Tokens Are Generated Incrementally

Because generation happens token by token, a provider can often stream output before the complete response exists.

Conceptually:

Model generates:

"The"
       ↓ send

"The order"
       ↓ send

"The order is"
       ↓ send

"The order is currently"
       ↓ send

"The order is currently in transit."

The client receives partial output incrementally.

This can improve perceived latency.

Instead of waiting:

5 seconds
      │
      ▼
full answer appears

the user may see:

0.5 sec → first token
0.6 sec → more text
0.7 sec → more text
...

We will discuss streaming properly when building applications.

24. Time to First Token vs Total Latency

Streaming introduces two useful latency concepts.

Time to First Token

How long until the model begins producing output?

Conceptually:

Request
   │
   ├──── processing ────┐
   │                    ▼
   │              First Token

Total Generation Time

How long until the full output is complete?

Request
   │
   ▼
First Token
   │
   ▼
More Tokens
   │
   ▼
Final Token

For interactive applications, both matter.

A response can have:

fast first token

but:

slow total generation

if the output is very long.

25. Output Length Is a Latency Concern

Suppose a model generates:

20 tokens

versus:

5,000 tokens

The second response generally takes longer to produce.

Therefore unnecessary verbosity affects:

  • latency
  • cost
  • user experience
  • context growth

For an agent deciding which tool to call, we usually do not want a 2,000-word essay.

We want a concise structured decision.

This is another reason structured outputs are important.

26. Maximum Output Tokens

Model APIs commonly allow an output limit.

Conceptually:

maxOutputTokens = 500

This prevents generation from exceeding a configured size.

This is useful for:

  • controlling cost
  • controlling latency
  • preventing runaway responses
  • reserving context capacity
  • matching application expectations

However, an important consequence exists.

If the model reaches the output limit before completing its response, the output may be truncated.

For example:

{
  "category": "BILLING",
  "reason":

Generation stops.

Now the response is invalid JSON.

Output limits must therefore be designed with the task in mind.

27. Stop Conditions

Generation must eventually end.

Common stopping conditions may include:

  • the model generates a special end token
  • the configured maximum output length is reached
  • a provider-defined stop condition occurs
  • a user-defined stop sequence is encountered
  • content policy or safety systems interrupt generation
  • the client cancels the request

Conceptually:

Generate Token
      │
      ▼
Should Stop?
   /      \
 No        Yes
 │          │
 ▼          ▼
Generate    Finish
Again

Understanding stop reasons becomes important when debugging incomplete model responses.

28. Finish Reasons Matter

Many APIs return metadata indicating why generation stopped.

Conceptually:

finishReason = STOP

or:

finishReason = LENGTH

or another provider-specific reason.

Suppose you expect valid structured output but receive:

{
  "customerId": "123",
  "classification":

Before blaming the model's reasoning, inspect the finish reason.

If:

finishReason = LENGTH

the issue may simply be output truncation.

Operational metadata can explain semantic-looking failures.

29. Model Generation Is a Sequence of Decisions

This becomes especially important for agents.

Consider:

User:
Cancel my latest order.

An agent-capable model may conceptually need to choose between:

respond directly

getLatestOrder()

cancelOrder(...)

ask user for clarification

refuse action

Those choices are ultimately represented through generated outputs or structured tool-call decisions.

So agent execution inherits the same probabilistic nature.

Conceptually:

Current Context
       │
       ▼
      Model
       │
       ▼
Probabilistic Decision
       │
       ├── Tool A
       ├── Tool B
       ├── Ask User
       └── Final Response

This is why giving a model access to tools changes the risk profile.

Generated text can be wrong.

Generated actions can have consequences.

30. A Tool Call Is Still Model Output

Suppose the model produces:

{
  "tool": "cancelOrder",
  "arguments": {
    "orderId": "8192"
  }
}

It is easy to mentally treat this as:

the agent decided correctly

But from the application's perspective, this is still model-generated output.

Therefore we should apply the same principle:

Model Output
    │
    ▼
Parse
    │
    ▼
Validate
    │
    ▼
Authorize
    │
    ▼
Execute

Tool calling does not make generated decisions inherently trustworthy.

31. Why Agent Loops Can Compound Uncertainty

Consider:

Model Decision 1
      │
      ▼
Tool Call
      │
      ▼
Tool Result
      │
      ▼
Model Decision 2
      │
      ▼
Tool Call
      │
      ▼
Model Decision 3

Each model decision is probabilistic.

A mistake at step 1 can affect step 2.

A mistake at step 2 can affect step 3.

The longer the uncontrolled decision chain, the more opportunities there are for the execution to diverge.

This does not mean long-running agents are inherently bad.

It means they need:

  • clear state
  • bounded actions
  • validation
  • stopping conditions
  • observability
  • evaluation

32. Nondeterminism Changes Testing

Consider a deterministic Java method:

int add(int a, int b) {
    return a + b;
}

A test can assert:

assertEquals(4, add(2, 2));

If this occasionally returns 5, the implementation is broken.

Now consider:

Summarize this customer conversation.

There may be many acceptable outputs.

For example:

The customer wants a refund for a damaged product.

and:

The customer received a damaged item and requested a refund.

Both can be correct.

Testing therefore cannot always rely on exact-string equality.

Instead we may need to evaluate properties such as:

Did it preserve the important facts?

Did it avoid inventing information?

Did it follow the required format?

Was the classification correct?

Did it select an allowed tool?

Did it reach the expected outcome?

This is why evaluation becomes a major discipline in agent engineering.

33. Exact-Match Tests Still Have a Place

Nondeterminism does not mean:

LLM applications cannot be tested

Some outputs can still be tested deterministically.

For example, if your application parses a classification into:

enum Category {
    BILLING,
    ACCOUNT,
    DELIVERY,
    OTHER
}

you can test:

assertEquals(
        Category.BILLING,
        result.category()
);

Likewise:

schema validation
authorization
tool argument constraints
business rules
database operations

should remain normal deterministic tests.

A mature AI system combines:

Deterministic Testing
        +
Behavioral Evaluation

not one or the other.

34. Separate Model Quality from System Correctness

Suppose an application produces a wrong answer.

Possible causes include:

Wrong model output
Bad prompt
Missing context
Incorrect retrieval
Stale database data
Tool bug
Parsing bug
Authorization bug
Application bug
Provider failure

Do not collapse all AI-system failures into:

the model hallucinated

That phrase can hide architectural problems.

Production debugging requires tracing the entire pipeline.

35. A Model Request Can Succeed Technically and Fail Semantically

Traditional HTTP monitoring often considers:

HTTP 200

a successful request.

For LLM applications:

HTTP 200

only tells you something about transport and provider execution.

It does not tell you whether the answer is good.

For example:

HTTP 200

"Order 8192 has been delivered."

while the real state is:

IN_TRANSIT

Operational success:

YES

Semantic success:

NO

This gives us two different reliability dimensions.

36. Operational Reliability vs Behavioral Reliability

We can think of:

Operational Reliability

Did the system function technically?

Examples:

request succeeded
provider responded
tool executed
database was reachable
latency stayed within limits

Behavioral Reliability

Did the AI system behave correctly?

Examples:

answer was grounded
classification was correct
correct tool was selected
unsafe action was avoided
instructions were followed

A production agent needs both.

Conceptually:

             Agent Reliability

          /                     \
         ▼                       ▼

Operational Reliability    Behavioral Reliability

Traditional monitoring mostly addresses the first.

Agent evaluation is necessary for the second.

37. Why Better Models Do Not Eliminate Evaluation

Suppose a new model performs substantially better.

It may still behave differently.

For example:

Model A:
calls getOrderStatus()

Model B:
asks the user for more information

Model C:
calls searchOrders()

All may eventually answer correctly.

Or one may introduce a regression.

Changing models is therefore similar to changing a major application dependency.

You should evaluate behavior before production rollout.

Do not assume:

newer model = safe drop-in replacement

38. Model Versions Matter

Providers may expose model identifiers such as:

model-x
model-x-2026-01-15
model-y

The exact naming convention varies.

A moving alias might eventually point to updated behavior.

A versioned model may provide more stability.

For production applications, you should understand:

  • which model version is being used
  • whether it can change automatically
  • how model updates are announced
  • what evaluation is required before migration

Model versioning is part of dependency management.

39. Generation Parameters Are Part of Application Behavior

Suppose production currently uses:

temperature = 0.1

and someone changes it to:

temperature = 1.5

The Java code may still compile.

Every unit test around your deterministic components may still pass.

But model behavior could change significantly.

Therefore generation configuration should be treated as versioned application configuration.

Changes deserve:

  • review
  • evaluation
  • controlled deployment
  • observability

40. Prompts, Models, and Parameters Form a Behavioral Unit

A useful way to think about model behavior is:

Behavior =
    Model
    +
    Instructions
    +
    Context
    +
    Generation Configuration

Change any one of these and behavior may change.

For example:

Same Model
Different Prompt
→ Different Behavior

or:

Same Prompt
Different Model
→ Different Behavior

or:

Same Model + Prompt
Different Temperature
→ Potentially Different Behavior

This is why AI-system configuration must be managed deliberately.

41. Model Output Is a Distribution, Not a Contract

Traditional APIs often have contracts.

For example:

Customer getCustomer(UUID id);

We expect a defined response shape or error behavior.

An unconstrained language model behaves differently.

A request like:

Give me the customer's priority.

might return:

HIGH

or:

The customer appears to be high priority.

or:

Priority: HIGH

or something unexpected.

The model produces likely continuations.

It does not automatically obey your Java type system.

This leads directly to our later lesson on structured outputs.

42. Natural Language Is a Weak Machine Interface

Humans are good at interpreting:

The request should probably be treated as high priority.

Applications usually prefer:

{
  "priority": "HIGH"
}

Why?

Because natural-language output introduces interpretation.

For machine-to-machine interaction, we prefer explicit structure.

This is especially important when model output controls:

  • routing
  • tools
  • state transitions
  • workflow decisions

The more consequential the output, the stronger the interface should be.

43. Do Not Parse Model Prose with Fragile String Logic

A poor implementation might look like:

if (response.contains("HIGH")) {
    priority = Priority.HIGH;
}

Imagine the model says:

This request is not HIGH priority.

Your parser still finds:

HIGH

and produces the wrong result.

Or:

The priority is probably medium, rather than high.

Again:

contains("HIGH")

is a poor contract.

Instead, later we will use structured outputs and schema validation.

44. Generation Length Can Affect Reasoning Quality

Longer outputs are not automatically better.

But overly restrictive output budgets can also hurt.

Suppose a complex analysis needs space to reason and you allow only:

20 output tokens

The model may be unable to complete the task properly.

Conversely, allowing:

20,000 output tokens

for a simple classification is wasteful.

The correct budget depends on the task.

This is another example of engineering trade-offs rather than universal settings.

45. Reasoning Has Cost and Latency

More capable reasoning modes can improve complex-task performance.

But they may also increase:

  • latency
  • token consumption
  • cost

This creates a model-selection question.

Do you need your strongest reasoning model to answer:

What is the delivery status?

Probably not.

For:

Analyze this distributed-system failure across 40 traces
and propose likely root causes.

a stronger reasoning model may be valuable.

Production systems may eventually route tasks to different models based on complexity.

46. Model Selection Is an Engineering Trade-off

You can think of model selection across several dimensions:

Capability
Cost
Latency
Context Window
Structured Output Support
Tool Calling Quality
Reasoning Ability
Multimodal Capability
Availability
Provider Reliability

There is rarely one universally best model.

Instead:

Choose a model appropriate for the task and system constraints.

We will explore model selection more directly in the next lesson.

47. A Useful Reliability Pattern

Suppose a model is responsible for interpreting user intent.

Instead of:

User
 │
 ▼
Model
 │
 ▼
Execute Action

use:

User
 │
 ▼
Model
 │
 ▼
Structured Proposal
 │
 ▼
Deterministic Validation
 │
 ▼
Policy Check
 │
 ▼
Authorization
 │
 ▼
Execute Action

Why?

Because the model's proposal emerged from probabilistic generation.

The action itself may require stronger guarantees.

This is the same principle introduced in Lesson 1:

Probabilistic core, deterministic boundaries.

Now we understand more precisely why that principle is necessary.

48. A Java Mental Model

We are still not depending on a particular AI framework.

Conceptually, imagine:

public record GenerationConfig(
        double temperature,
        int maxOutputTokens
) {
}

Then:

GenerationConfig config =
        new GenerationConfig(
                0.2,
                500
        );

And:

ModelResponse response =
        model.generate(messages, config);

That one method hides a large process:

Messages
   │
   ▼
Tokenization
   │
   ▼
Model Inference
   │
   ▼
Probability Distribution
   │
   ▼
Sampling / Decoding
   │
   ▼
Token
   │
   ▼
Repeat
   │
   ▼
Stop Condition
   │
   ▼
Response

When frameworks later give us a convenient API, this is the mechanism you should still imagine underneath.

49. Thinking Like an Agent Engineer

When a model produces an unexpected output, do not immediately ask:

How do I force the model to behave?

First ask:

Was the required information in context?

Were the instructions clear?

Was the model appropriate for the task?

Was generation configured appropriately?

Was the output constrained?

Was the result validated?

Was the model asked to decide something deterministic code should own?

Was the behavior actually evaluated?

This leads to better systems than endlessly rewriting prompts.

50. The Core Mental Model

At generation time, think:

Current Context
      │
      ▼
Model computes next-token probabilities
      │
      ▼
Decoding strategy selects token
      │
      ▼
Token joins context
      │
      ▼
Repeat
      │
      ▼
Stop

From this mechanism emerge many of the behaviors we observe:

variation
creativity
hallucination
drift
nondeterminism
streaming
variable output length

The model's intelligence is far more sophisticated than this simplified diagram suggests.

But for application engineering, the next-token model is an extremely useful foundation.

Practical Exercise — Reason About Temperature

Consider these tasks:

A. Generate 20 creative startup names.

B. Classify a transaction as:
   FRAUD_SUSPECTED
   NORMAL

C. Extract:
   name
   email
   phone number

D. Write three alternative marketing headlines.

E. Select which database migration should execute.

For each one, decide whether you would generally prefer:

Lower Variability

or:

Higher Variability

Explain why.

Do not choose an exact temperature value.

Focus on the application's behavioral requirement.

Practical Exercise — Find the Reliability Problem

An application asks:

Should this €25,000 transfer be approved?

The model responds:

Yes, it appears safe to approve.

The backend executes the transfer directly.

Identify the architectural mistake.

Redesign the flow so the model may still contribute useful analysis without becoming the security boundary.

Practical Exercise — Diagnose the Failure

Your application requests structured JSON.

Expected:

{
  "category": "DELIVERY",
  "priority": "HIGH"
}

Actual response:

{
  "category": "DELIVERY",
  "priority":

The API metadata contains:

finishReason = LENGTH

What likely happened?

What would you investigate before changing the prompt?

Practical Exercise — Semantic vs Operational Success

For each case, classify whether the system experienced:

Operational Failure
Behavioral Failure
Both
Neither

Case A

The model provider returns HTTP 503.

Case B

The provider returns HTTP 200, but the model says Tallinn is the capital of Finland.

Case C

The model returns the correct answer after 250 ms.

Case D

The model generates the correct tool call, but the database is unavailable.

Case E

The tool executes successfully, but the model chose the wrong customer ID.

Explain your reasoning.

Practical Exercise — Testing Strategy

You are building a support classifier.

Input:

I was charged twice for the same subscription.

Expected classification:

BILLING

Design two kinds of tests:

  1. deterministic application tests
  2. model behavior evaluation

Think about what belongs in each category.

For example, schema parsing is different from whether the model classified the request correctly.

Design Exercise — Bound a Model Decision

Design an architecture for:

An AI assistant can suggest whether a refund should be approved, but refunds above €500 require human approval and refunds must never exceed the amount actually paid.

Your design should show:

User Request
Model
Order Data
Model Recommendation
Validation
Policy
Human Approval
Payment Service

Clearly identify which responsibilities are:

Probabilistic

and which are:

Deterministic

Questions You Should Be Able to Answer

Before moving to the next lesson, make sure you can explain these clearly.

1. How does an LLM generate text?

It repeatedly predicts a probability distribution over possible next tokens, selects a token according to its decoding strategy, adds that token to the context, and repeats.

2. What is autoregressive generation?

Generation where each new token is produced based on the original context plus previously generated tokens.

3. What is temperature?

A generation parameter that influences how concentrated or spread out the token probability distribution is during selection.

4. Does lower temperature guarantee factual correctness?

No.

It generally reduces output variability but does not validate whether the model's preferred answer is true.

5. Why can identical prompts produce different responses?

Sampling and other nondeterministic factors can select different token paths.

6. Can temperature zero guarantee permanently identical output?

Not necessarily.

Provider implementation, model versions, infrastructure, and other factors may still affect reproducibility.

7. Why can an early generation mistake affect the rest of the response?

Because generated tokens become part of the context used to generate subsequent tokens.

8. Why is model probability not the same as factual confidence?

The probability represents how likely a continuation is according to the model, not a verified probability that the claim is objectively correct.

9. Why does generation affect agent safety?

Tool selections and action arguments can also be generated probabilistically, so they must be validated and constrained.

10. What is the difference between operational and behavioral reliability?

Operational reliability concerns whether the infrastructure and execution worked technically.

Behavioral reliability concerns whether the AI system behaved correctly.

11. Why are exact-string tests insufficient for many LLM tasks?

Multiple different outputs can be equally correct.

Behavior often needs to be evaluated by properties, semantics, or task outcomes.

12. Why should generation configuration be version-controlled?

Because changing the model, prompt, context, or decoding configuration can change application behavior.

Key Takeaways

Large Language Models typically generate output one token at a time.

At every step:

Context
   │
   ▼
Probability Distribution
   │
   ▼
Token Selection
   │
   ▼
Updated Context

and the process repeats.

This mechanism explains why LLM behavior is inherently different from ordinary deterministic application logic.

Parameters such as temperature can influence variability, but they do not transform generated output into verified truth.

A technically successful model request can still produce a semantically incorrect result.

And once models are allowed to choose tools or influence actions, this probabilistic behavior becomes an application-safety concern rather than merely a text-quality concern.

The engineering response is not to expect perfect determinism from the model.

It is to design systems that can safely tolerate model uncertainty.

That means:

constrained interfaces
structured outputs
validation
authorization
policy enforcement
bounded actions
evaluation
observability

The central principle remains:

Use probabilistic models for capabilities that benefit from flexible interpretation and reasoning. Use deterministic systems for guarantees that must always hold.

In the next lesson, we will move from model behavior to the infrastructure around it.

We will study models, providers, and LLM APIs: what a model provider actually provides, how an application communicates with a hosted model, model identifiers, authentication, latency, rate limits, capabilities, model selection, and why production applications should understand the provider boundary before hiding it behind a framework.