So far, we have deliberately avoided depending on a particular AI provider or Java framework.

That was intentional.

Before using libraries such as Spring AI or LangChain4j, you should understand the infrastructure boundary underneath them.

When your Java application calls an LLM, several different concepts are involved:

• the model
• the model provider
• the API
• the request
• the response
• model configuration
• authentication
• rate limits
• timeouts
• retries
• latency
• availability
• cost
• capability differences

Developers often collapse all of these into one sentence:

"We call ChatGPT."

That mental model is too vague for production engineering.

A better model is:

Your Application
       │
       ▼
Provider API
       │
       ▼
Selected Model
       │
       ▼
Inference
       │
       ▼
Provider Response
       │
       ▼
Your Application

Once an LLM becomes a dependency of your backend, you need to think about it like any other remote service—with additional probabilistic behavior on top.

---

1. Model and Provider Are Different Things

Let's begin with the distinction.

A model is the machine-learning system performing inference.

Conceptually:

Model
│
├── understands input
├── generates output
├── may reason
├── may support tools
├── may process images
└── has specific capabilities and limits

A provider operates infrastructure that gives your application access to models.

Conceptually:

Provider
│
├── hosts models
├── exposes APIs
├── authenticates requests
├── handles billing
├── enforces rate limits
├── manages infrastructure
└── returns responses

These concepts are related, but they are not interchangeable.

---

2. One Provider May Offer Multiple Models

A provider rarely exposes only one model.

Conceptually:

Provider
   │
   ├── Small Model
   │
   ├── General Model
   │
   ├── Reasoning Model
   │
   ├── Multimodal Model
   │
   └── Embedding Model

These models may differ in:

• capability
• speed
• price
• context window
• reasoning ability
• tool calling
• structured output support
• multimodal support

Your application therefore does not simply choose:

Which provider?

It also chooses:

Which model for this task?

Those are separate architectural decisions.

---

3. Different Providers May Offer Similar Capabilities

Imagine:

Provider A
   └── Model A

Provider B
   └── Model B

Provider C
   └── Model C

All three may support:

text generation

but their APIs and behavior may differ.

For example:

Provider A:
POST /responses

Provider B:
POST /messages

Provider C:
POST /generate

These paths are illustrative.

Even when the high-level operation is the same, differences may exist in:

• authentication
• message formats
• tool schemas
• streaming protocols
• error responses
• model identifiers
• generation parameters
• usage metadata

Frameworks help normalize some of these differences.

But the differences still exist underneath.

---

4. Hosted LLMs Are Remote Dependencies

Suppose your Java service executes:

model.chat(messages);

That method may look like a local function call.

But underneath, the real architecture may be:

Java Application
       │
       │ HTTPS
       ▼
Internet / Network
       │
       ▼
LLM Provider
       │
       ▼
Inference Infrastructure
       │
       ▼
Model

This immediately introduces distributed-system concerns.

The request can:

• time out
• fail
• be rate-limited
• take several seconds
• return a server error
• return malformed output
• succeed but produce a poor answer

LLM integration is therefore both:

Distributed Systems Engineering

and:

Probabilistic AI Engineering

That combination is what makes production agent systems particularly interesting.

---

5. A Typical LLM Request

A simplified request might conceptually contain:

{
  "model": "some-model",
  "messages": [
    {
      "role": "system",
      "content": "You are a software engineering instructor."
    },
    {
      "role": "user",
      "content": "Explain eventual consistency."
    }
  ]
}

Depending on the API, it may also contain configuration such as:

{
  "temperature": 0.2,
  "max_output_tokens": 1000
}

The provider receives this request and performs inference.

The response might conceptually look like:

{
  "output": "Eventual consistency is...",
  "usage": {
    "input_tokens": 42,
    "output_tokens": 311
  }
}

Real provider formats differ.

The important thing is understanding the contract.

---

6. The Model Identifier Is Part of the Request

A request typically specifies which model should handle it.

Conceptually:

{
  "model": "model-x"
}

This means your application behavior depends partly on external configuration.

For example:

MODEL_NAME=model-x

Changing that configuration may change:

• response quality
• latency
• cost
• token limits
• tool behavior
• structured output behavior

Changing the model can therefore be a meaningful production change even if no Java source code changes.

---

7. Do Not Hard-Code Model Configuration Everywhere

A poor implementation might contain:

String model = "model-x";

inside many services.

Then migrating models requires modifying application code throughout the codebase.

A better approach is centralized configuration.

Conceptually:

ai:
  provider: provider-a
  model: model-x
  temperature: 0.2
  max-output-tokens: 1000

Then:

@ConfigurationProperties(prefix = "ai")
public record AiProperties(
        String provider,
        String model,
        double temperature,
        int maxOutputTokens
) {
}

The exact design depends on your application.

The principle is:

Treat model configuration as deliberate application configuration.

---

8. Authentication

Hosted providers normally require credentials.

Conceptually:

Java Application
       │
       │ API Key / Credential
       ▼
Provider

For example, a request might use an HTTP header conceptually similar to:

Authorization: Bearer <credential>

Credentials must be treated like any production secret.

Do not:

String apiKey = "secret-key-here";

inside source code.

Do not commit credentials to Git.

Instead use appropriate secret management.

For local development:

environment variables

may be sufficient.

For production:

secret manager
container secret
orchestration platform secret
cloud secret service

may be appropriate.

---

9. API Keys Belong on the Backend

Suppose you build a browser application.

A dangerous architecture is:

Browser
   │
   │ Provider API Key
   ▼
LLM Provider

If the credential is shipped to the browser, users may be able to extract it.

A safer architecture is usually:

Browser
   │
   ▼
Your Backend
   │
   ├── authentication
   ├── authorization
   ├── rate limits
   ├── cost controls
   └── provider credentials
   │
   ▼
LLM Provider

The backend controls access to the model.

This also gives you a place to enforce:

• application policies
• quotas
• logging
• model routing
• validation
• abuse prevention

---

10. The LLM Provider Should Be Treated as an External Service

Backend engineers already understand external service calls.

For example:

Application
    │
    ▼
Payment Provider

or:

Application
    │
    ▼
Shipping Provider

We normally consider:

timeouts
retries
rate limits
errors
latency
availability
observability

The same applies to LLM providers.

A naive implementation:

String answer = model.generate(prompt);

can hide these concerns.

A production mental model should instead be:

Application
     │
     ▼
Timeout
     │
     ▼
Provider Request
     │
     ├── Success
     ├── Rate Limited
     ├── Timeout
     ├── Server Error
     └── Invalid Request

The model call is a network dependency.

---

11. Timeouts Are Essential

Never assume an LLM request will eventually return.

Suppose:

User Request
     │
     ▼
Backend
     │
     ▼
LLM Provider

and the provider takes:

45 seconds

What happens to:

• your HTTP connection?
• thread usage?
• user experience?
• upstream gateway timeout?
• retry behavior?

Every production integration should have deliberate timeout behavior.

Conceptually:

Provider Request
      │
      ├── completes within budget
      │
      ▼
    success

or

Provider Request
      │
      ├── exceeds timeout
      │
      ▼
    cancel / fail

Do not leave timeout policy undefined.

---

12. One Timeout Is Often Not Enough

A network client may expose several timeout concepts.

For example:

connection timeout
read timeout
overall request deadline

Agent systems may also need a higher-level execution deadline.

Suppose:

Model Call       8 sec
Tool Call        3 sec
Model Call       9 sec
Tool Call        4 sec
Model Call      10 sec

Each operation individually succeeds within its timeout.

But the complete user request takes:

34 seconds

You may need:

per-operation timeout

and:

overall agent execution budget

These solve different problems.

---

13. Retries Require Care

Backend engineers often see a transient network failure and think:

retry

That can be reasonable.

But AI applications introduce additional considerations.

Suppose:

LLM request
    │
    ▼
timeout

Did the provider actually process the request?

For a pure generation request, retrying may mostly mean additional cost.

But suppose the model call exists inside an agent loop:

Model
  │
  ▼
Tool
  │
  ▼
Payment Action

Now blindly replaying the entire workflow could have side effects.

Retry policies must distinguish between:

safe inference retry

and:

replaying side-effecting operations

This is a standard distributed-systems problem appearing inside an AI workflow.

---

14. Idempotency Still Matters

Suppose an agent executes:

issueRefund(orderId=8192, amount=100)

The network connection fails before your application receives confirmation.

Did the refund happen?

If you simply retry:

issueRefund(...)

you could issue it twice.

The solution is not:

Tell the LLM not to retry refunds.

The solution belongs at the system boundary.

For example:

idempotency key

Conceptually:

Refund Request
      │
      ├── idempotencyKey = operation-123
      ▼
Payment Service
      │
      ▼
Same operation executes at most once

Agents do not eliminate distributed-system principles.

They make them more important.

---

15. Rate Limits

Providers cannot accept unlimited traffic from every customer.

They may impose rate limits based on things such as:

requests per minute
tokens per minute
requests per day
account tier
model
organization

Exact policies vary.

A provider may respond with something conceptually equivalent to:

429 Too Many Requests

Your application needs a strategy.

Possible responses include:

backoff
retry
queue
reject
degrade gracefully
route elsewhere

The correct choice depends on the application.

---

16. Token-Based Rate Limits Are Important

Traditional APIs often make us think primarily in:

requests per second

LLM providers may also care about token volume.

Consider two requests.

Request A:

500 input tokens
100 output tokens

Request B:

80,000 input tokens
5,000 output tokens

Both are:

1 request

but they consume dramatically different inference resources.

Therefore capacity planning may need to consider:

requests
+
tokens

not requests alone.

---

17. Concurrency Matters

Suppose your application receives:

1,000 simultaneous user requests

and every request immediately invokes a model.

Your backend may survive.

The provider may not accept your full burst.

You may need:

concurrency limits
queues
backpressure
rate limiting

Conceptually:

Users
  │
  ▼
Application
  │
  ▼
Concurrency Limiter
  │
  ▼
LLM Provider

Without control, traffic spikes can produce cascading failures.

---

18. Backpressure Is Better Than Collapse

Imagine:

Provider capacity temporarily reduced

Your application responds by aggressively retrying every failure.

Now:

Failures
   │
   ▼
Retries
   │
   ▼
More Load
   │
   ▼
More Failures
   │
   ▼
More Retries

This is a retry storm.

A healthier design may use:

bounded retries
exponential backoff
jitter
concurrency limits
queue limits
deadlines

These are ordinary resilience patterns.

LLM applications still need them.

---

19. Not Every Error Should Be Retried

Suppose the provider returns:

401 Unauthorized

Retrying five times probably will not help.

Likewise:

invalid model
invalid request schema
context too large

may require application changes rather than retries.

Conceptually classify failures:

Provider Error
     │
     ├── Transient
     │      └── maybe retry
     │
     └── Permanent / Request Error
            └── fail appropriately

Retry policy should understand failure categories.

---

20. Cost Is Part of the API Boundary

Many model APIs charge according to usage.

Conceptually:

Cost =
Input Token Cost
+
Output Token Cost
+
Possible Additional Feature Costs

Prices differ by provider and model and can change over time.

For architecture, the important point is:

Every model invocation consumes a measurable resource.

That means an endpoint such as:

POST /chat

can effectively trigger an external metered computation.

This should influence:

• quotas
• abuse protection
• caching
• model selection
• agent loop limits
• observability

---

21. One User Request May Mean Many Provider Requests

Suppose the user asks:

Find my latest order and tell me whether I can return it.

An agent might execute:

Model Call 1
    │
    ▼
getLatestOrder()
    │
    ▼
Model Call 2
    │
    ▼
getReturnPolicy()
    │
    ▼
Model Call 3
    │
    ▼
Final Response

From your API gateway:

1 user request

From your LLM provider:

3 model requests

This distinction matters for:

• cost
• latency
• rate limits
• observability

Agentic workloads amplify external dependency usage.

---

22. Set Agent Budgets

An uncontrolled agent loop is dangerous.

Imagine:

Model
 │
 ▼
Tool
 │
 ▼
Model
 │
 ▼
Tool
 │
 ▼
Model
 │
 ▼
Tool
 │
 ▼
...

A bug or confused model could continue unnecessarily.

Production systems should consider limits such as:

maximum model calls
maximum tool calls
maximum total tokens
maximum execution time
maximum cost

Conceptually:

Agent Execution Budget

Time       ≤ limit
Tokens     ≤ limit
Model Calls≤ limit
Tool Calls ≤ limit

A bounded agent is much easier to operate than an unlimited one.

---

23. Latency Is Often Much Higher Than Ordinary APIs

A normal backend API might respond in:

20 ms
50 ms
200 ms

An LLM request may take:

hundreds of milliseconds
several seconds
tens of seconds

depending on:

• model
• input size
• output size
• reasoning effort
• provider load
• network latency

You should not design an LLM dependency as though it were a Redis lookup.

The latency profile is fundamentally different.

---

24. Agent Latency Accumulates

Suppose:

Model Call 1 = 3 seconds
Tool Call 1  = 500 ms
Model Call 2 = 4 seconds
Tool Call 2  = 300 ms
Model Call 3 = 3 seconds

Total sequential time is roughly:

10.8 seconds

Even though no individual operation appears catastrophic.

Agent architecture can therefore have significant latency amplification.

This should influence how many sequential reasoning steps you allow.

---

25. Parallelism Can Help—When Operations Are Independent

Suppose the model determines it needs:

customer profile
order history
subscription status

If these are independent:

              ┌── getCustomer()
              │
Agent ─────────┼── getOrders()
              │
              └── getSubscription()

they may execute concurrently.

Instead of:

500 ms
+
700 ms
+
400 ms
=
1.6 sec

the total may be closer to the slowest operation.

But do not parallelize blindly.

If operation B depends on A:

getOrder()
    │
    ▼
getShipment(order.shipmentId)

the dependency is sequential.

This is normal workflow engineering.

---

26. Streaming Improves Perceived Latency

As we learned in the previous lesson, tokens can often be streamed.

Architecture:

LLM Provider
     │
     │ token chunks
     ▼
Backend
     │
     │ streamed response
     ▼
Frontend

This allows users to see output before generation completes.

However, streaming creates additional engineering concerns:

• client disconnects
• partial responses
• cancellation
• error handling after output begins
• proxy buffering
• observability

Streaming is useful, but it is not free complexity.

---

27. Cancellation Matters

Suppose a user starts a long generation and then closes the page.

Without cancellation:

User gone
   │
   ▼
Backend still waiting
   │
   ▼
Provider still generating
   │
   ▼
Tokens still consumed

If your stack supports cancellation propagation, you may be able to stop unnecessary work.

Conceptually:

Client Disconnect
      │
      ▼
Cancel Backend Operation
      │
      ▼
Cancel Provider Request

This can reduce wasted:

• compute
• tokens
• money
• concurrency

---

28. Capability Differences Matter

Not every model supports every feature.

A model may support:

text

but not:

images

Another may support:

tool calling

but have weaker structured-output guarantees.

Another may have:

large context

but higher cost.

Your application should understand required capabilities.

For example:

Task:
Extract invoice fields from an image

requires something different from:

Task:
Classify a short text message

Model selection should start from task requirements.

---

29. Choose Models Per Workload, Not Per Company

A common early architecture is:

Our company uses Model X for everything.

That is simple.

But perhaps:

Simple classification
→ small inexpensive model

Complex architecture analysis
→ stronger reasoning model

Image understanding
→ multimodal model

Embedding generation
→ embedding model

This can improve:

• cost
• latency
• scalability

without sacrificing capability where it matters.

You do not need this complexity on day one.

But you should understand that:

model selection can happen at the workload level.

---

30. Model Routing

A model router conceptually does:

Task
 │
 ▼
Determine Requirements
 │
 ├── simple extraction
 │       ▼
 │   Small Model
 │
 ├── complex reasoning
 │       ▼
 │   Strong Model
 │
 └── image task
         ▼
     Multimodal Model

Routing can be:

• static
• rule-based
• configuration-driven
• dynamic

For example:

return switch (taskType) {
    case CLASSIFICATION -> fastModel;
    case COMPLEX_ANALYSIS -> reasoningModel;
    case IMAGE_ANALYSIS -> multimodalModel;
};

You do not need another LLM just to route everything.

Deterministic routing is often simpler.

---

31. Avoid Premature Multi-Provider Complexity

After learning that providers can fail, it is tempting to immediately build:

Provider A
Provider B
Provider C
Provider D

with automatic failover.

That sounds resilient.

But it introduces major complexity because models may behave differently.

Suppose Provider A returns:

{
  "category": "BILLING"
}

while Provider B tends to return:

This appears to be a billing request.

Or their tool-calling behavior differs.

Failover is not equivalent to switching identical database replicas.

Models are behavioral dependencies.

A second provider should be evaluated as a separate implementation.

---

32. Provider Portability Has Limits

Frameworks can give us abstractions such as:

ChatModel model;

Then we may configure different implementations.

This is useful.

But do not assume:

Provider A Model
=
Provider B Model

even if both satisfy the same Java interface.

A Java interface can normalize the method signature.

It cannot normalize intelligence.

Conceptually:

              ChatModel
              /       \
             ▼         ▼
        Provider A   Provider B

Both implement:

generate(messages)

but outputs may differ significantly.

Abstraction provides code portability.

Evaluation provides behavioral confidence.

---

33. Vendor Lock-In Is Not Only an API Problem

Developers often define vendor lock-in as:

our code imports provider-specific classes

That is one form.

But deeper lock-in can come from:

• provider-specific tool behavior
• prompt tuning for one model
• model-specific context sizes
• proprietary features
• response schemas
• pricing assumptions
• latency assumptions
• evaluation tuned to one model's behavior

Therefore:

We wrapped the SDK in an interface.

does not automatically mean:

We are provider-independent.

Portability is both technical and behavioral.

---

34. A Thin Internal Boundary Is Often Useful

Suppose your business service directly depends on provider-specific classes everywhere:

public class RefundService {

    private final SomeProviderSpecificClient client;

}

Provider details now leak into business logic.

A cleaner architecture might expose an application-level capability:

public interface SupportReasoner {

    SupportDecision analyze(
            SupportContext context
    );
}

Then:

Business Logic
      │
      ▼
SupportReasoner
      │
      ▼
Provider Adapter
      │
      ▼
LLM Provider

This does not mean creating dozens of meaningless abstraction layers.

It means keeping infrastructure concerns at sensible boundaries.

---

35. Do Not Build a Generic "AIService" for Everything

This is the opposite mistake.

Developers sometimes create:

public interface AIService {

    String ask(String prompt);
}

Then every feature calls it.

Soon:

Refunds
Support
Search
Classification
Recommendations
Summaries

all depend on:

String → String

You have created an abstraction so generic that it communicates almost nothing.

Prefer interfaces around meaningful application capabilities.

For example:

public interface TicketClassifier {

    TicketClassification classify(
            SupportTicket ticket
    );
}

or:

public interface RefundRecommendationService {

    RefundRecommendation evaluate(
            RefundContext context
    );
}

The business contract should describe the business capability.

---

36. Provider Responses Contain More Than Text

A useful response may include metadata such as:

generated content
model identifier
input token count
output token count
finish reason
tool calls
request identifier

Exact fields vary.

Do not immediately throw all metadata away.

Some of it is valuable for:

• debugging
• cost analysis
• observability
• tracing
• detecting truncation

Your abstraction should preserve the information your system actually needs.

---

37. Provider Request IDs Are Valuable

Suppose a production request fails unexpectedly.

Your logs say:

LLM call failed.

That is not very useful.

If the provider returns a request identifier:

providerRequestId = abc123

storing it in your trace or logs can make debugging much easier.

A useful correlation chain might look like:

HTTP Request ID
      │
      ▼
Agent Execution ID
      │
      ▼
Model Call ID
      │
      ▼
Provider Request ID

This creates traceability across system boundaries.

---

38. Observe Every Model Call

At minimum, production systems should consider recording metadata such as:

model
provider
latency
input tokens
output tokens
finish reason
success/failure
retry count

For agent systems, also consider:

agent execution ID
step number
tool selected
total execution time
total model calls

Be careful with logging full prompts and responses.

They may contain sensitive information.

Observability and privacy must be designed together.

---

39. Do Not Log Secrets

Suppose a user accidentally writes:

My API key is sk-...

If you log the full conversation:

application logs

now contain the credential too.

Similarly, tool results may contain:

• personal data
• tokens
• account information
• internal documents

A production logging strategy should consider:

redaction
structured metadata
sampling
access controls
retention

"Log everything" is not a safe default.

---

40. Measure Cost Per Business Operation

Token metrics are useful.

But business-level metrics are even more useful.

Instead of only:

input tokens = 18,291

you eventually want to understand:

Average cost per support conversation

Average cost per resolved ticket

Average cost per document processed

Average model calls per agent execution

This connects infrastructure spending to product value.

For example:

Agent A:
€0.02 average cost
70% successful completion

Agent B:
€0.08 average cost
95% successful completion

Now you can make meaningful trade-offs.

---

41. Availability Is Part of Product Reliability

Suppose your application has:

99.99% availability

but every important request depends synchronously on an external AI provider with lower availability.

Your effective user experience inherits that dependency.

Conceptually:

User
 │
 ▼
Your API
 │
 ▼
LLM Provider

If the provider is unavailable:

Your AI Feature
=
Unavailable

unless you have a fallback or degraded mode.

External AI dependencies belong in reliability planning.

---

42. Graceful Degradation

Not every AI feature needs hard failure.

Suppose AI generates:

product-description suggestions

If the provider is unavailable, perhaps the UI can say:

Suggestions are temporarily unavailable.

The rest of the application still works.

But if the AI feature is directly in the critical path:

User Request
   │
   ▼
LLM
   │
   ▼
Required Business Operation

provider failure may block the operation.

Ask:

Does this feature really need synchronous AI dependency?

Sometimes the answer is no.

---

43. Asynchronous Processing

Some AI workloads do not need an immediate answer.

For example:

Generate summaries for 50,000 historical support tickets.

Do not necessarily perform:

HTTP Request
    │
    ▼
wait for all 50,000

A better architecture may be:

Request
   │
   ▼
Job Queue
   │
   ▼
Worker
   │
   ▼
LLM Provider
   │
   ▼
Persist Result

This enables:

• retries
• rate control
• batching strategies
• progress tracking
• failure recovery

Use synchronous AI when the user actually needs synchronous behavior.

---

44. Caching Can Reduce Cost—but Requires Semantics

Suppose users repeatedly ask:

Summarize our public return policy.

If the source document and instructions are identical, caching might save model calls.

Conceptually:

Input
 │
 ▼
Cache Key
 │
 ├── hit ──► Cached Result
 │
 └── miss ─► Model ──► Cache

But caching LLM output requires care.

Questions include:

Did the source data change?

Does the answer depend on the user?

Does the prompt include private context?

Is variation desirable?

How long should the result remain valid?

Do not cache based only on superficial string equality without understanding the semantics.

---

45. Semantic Caching Is More Complex

Imagine:

How do I return an item?

and:

What's the process for sending my purchase back?

Semantically, these may ask the same thing.

A semantic cache might attempt to reuse answers across similar questions.

This can reduce cost.

But it introduces risk:

similar question
≠
same required answer

Especially when user-specific or time-sensitive data is involved.

Semantic caching is useful in some workloads but should not be added casually.

---

46. Provider Limits Should Influence Application Design

Suppose a provider supports a large context window.

Your application should still enforce its own limits.

Why?

Because your business may decide:

Maximum uploaded document: 10 MB

Maximum conversation context: 30,000 tokens

Maximum agent calls: 8

Maximum execution time: 30 seconds

Provider maximums answer:

What can the provider technically accept?

Application limits answer:

What are we willing to allow?

Those are different questions.

---

47. Provider Capability Is Not Product Policy

Suppose a provider allows:

100,000 output tokens

That does not mean your product should.

Suppose a provider allows:

500 concurrent requests

That does not mean one user should consume all 500.

Your application needs its own:

quotas
budgets
authorization
rate limits

Never outsource product policy to provider limits.

---

48. User-Level Quotas

Because model calls cost money, public AI applications need abuse controls.

Imagine:

POST /api/chat

has no authentication and no rate limit.

An attacker sends:

100,000 expensive requests

Your infrastructure may continue working perfectly.

Your provider bill may not.

Possible controls include:

authentication
per-user quotas
per-organization quotas
request-size limits
token budgets
concurrency limits
billing limits

Cost abuse is a security concern.

---

49. Multi-Tenant Systems Need Cost Attribution

Suppose LiveKlass eventually provides AI features to educators.

You might have:

Organization A
Organization B
Organization C

All use the same backend provider account.

Without attribution:

Provider bill = €10,000

but you do not know who consumed it.

A better architecture records:

organizationId
userId
feature
model
inputTokens
outputTokens
cost estimate

Then you can understand usage per tenant and feature.

This becomes important for:

• pricing
• quotas
• abuse prevention
• product analytics

---

50. Model Selection Should Be Measured

Suppose:

Model A

costs significantly more than:

Model B

but Model A achieves:

96% task success

while Model B achieves:

94%

Is Model A worth the additional cost?

Maybe.

Maybe not.

Now suppose:

Model A:
96% success
3.5 sec latency
€0.04 / request

Model B:
94% success
0.8 sec latency
€0.006 / request

This is a product and engineering trade-off.

Model selection should eventually be driven by:

evaluation
+
latency
+
cost
+
operational requirements

not hype.

---

51. Build a Model Evaluation Matrix

For an important workload, you might compare models using something conceptually like:

Model	Task Success	Avg Latency	Avg Cost	Tool Accuracy
Model A	96%	3.5 s	€0.040	98%
Model B	94%	0.8 s	€0.006	93%
Model C	89%	0.5 s	€0.003	88%

These values are illustrative.

Now the decision becomes evidence-based.

Perhaps:

Model B

is the best default.

And:

Model A

is used only for difficult cases.

This is much stronger than:

Model A feels smarter.

---

52. Model Fallback Requires Evaluation

Suppose your primary model fails.

You might configure:

Primary Model
     │
     ├── success
     │
     ▼
   result

     └── failure
           │
           ▼
      Fallback Model

Technically, this is easy.

Behaviorally, it is not.

The fallback must be tested against the same requirements.

Otherwise:

Infrastructure reliability ↑

while:

Behavioral reliability ↓

You may have made the system more available but less correct.

Both dimensions matter.

---

53. Circuit Breakers Can Be Useful

Suppose a provider is failing repeatedly.

Continuing to send every request may:

• waste time
• create retries
• increase resource usage
• worsen user latency

A circuit breaker conceptually does:

Provider healthy
      │
      ▼
Requests allowed

Repeated failures
      │
      ▼
Circuit opens
      │
      ▼
Fail fast / fallback

After delay
      │
      ▼
Probe provider

Whether you need this depends on the application and infrastructure stack.

But the principle is familiar from microservice architecture.

---

54. The Provider Boundary Should Be Observable

A model call should not be a mysterious black box.

Conceptually:

Agent Step
   │
   ▼
LLM Request
   │
   ├── provider
   ├── model
   ├── input tokens
   ├── start time
   ▼
LLM Response
   │
   ├── output tokens
   ├── finish reason
   ├── latency
   ├── tool call
   └── status

This information becomes invaluable when debugging:

Why is the agent slow?

Why did cost increase?

Why are outputs truncated?

Why are requests failing?

Why did this workflow use 12 model calls?

Observability begins at the provider boundary.

---

55. A Production Java Architecture

A reasonable high-level architecture might look like:

┌─────────────────────────────┐
│       REST Controller       │
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│       Application Use Case  │
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│   Domain-Specific AI Port   │
│                             │
│ e.g. TicketClassifier       │
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│       LLM Adapter           │
│                             │
│ model configuration         │
│ provider client             │
│ response mapping            │
│ observability               │
└──────────────┬──────────────┘
               │ HTTPS
               ▼
┌─────────────────────────────┐
│       LLM Provider          │
└─────────────────────────────┘

This keeps:

business capability

separate from:

provider infrastructure

without creating unnecessary abstraction.

---

56. What Frameworks Will Eventually Do for Us

Later, when we use a Java AI framework, it may handle:

HTTP client integration
message serialization
authentication configuration
provider-specific request formats
streaming
tool schemas
structured outputs

This is useful.

But because you now understand the provider boundary, you will know what those abstractions are hiding.

That makes debugging much easier.

Instead of thinking:

Spring AI is broken.

you can ask:

Did our application construct the request correctly?

Did the framework serialize it correctly?

Did the provider reject it?

Did the model return an unexpected result?

Did our mapper fail?

Did we hit a rate limit?

Did generation reach the token limit?

That is the difference between using a framework and understanding the system.

---

57. Do Not Couple the Entire Application to Prompt Strings

A poor architecture:

public String handleRefundRequest(String message) {

    String prompt = """
            User says: %s

            Decide whether to refund.
            """.formatted(message);

    return provider.generate(prompt);
}

Everything is mixed together:

business logic
prompt construction
provider integration
response parsing

A stronger design separates responsibilities.

Conceptually:

Refund Use Case
      │
      ▼
Build Refund Context
      │
      ▼
Refund Recommendation Port
      │
      ▼
LLM Adapter
      │
      ▼
Provider

This makes the system easier to:

• test
• evaluate
• migrate
• observe
• secure

---

58. The LLM Is Not Your Domain Layer

Suppose your domain contains:

public record RefundPolicy(
        Money maximumAutomaticRefund,
        Duration returnWindow
) {
}

Do not replace that with:

Prompt:
"Please remember our refund limit is €500."

The model can help interpret ambiguous information.

Your domain still owns deterministic business rules.

Conceptually:

Domain
│
├── policies
├── invariants
├── permissions
└── state transitions

LLM
│
├── language understanding
├── extraction
├── classification
├── reasoning assistance
└── generation

Use each component for what it is good at.

---

59. A Useful Failure Taxonomy

When an AI feature fails, classify the failure.

Network Failure

timeout
DNS failure
connection reset

Provider Failure

5xx
service unavailable
rate limit

Request Failure

invalid model
context too large
invalid schema

Model Behavioral Failure

wrong answer
wrong tool
hallucination
instruction failure

Application Failure

bad parsing
wrong context
authorization bug
incorrect tool implementation

This classification prevents every problem from being called:

AI issue

That phrase is too vague to operate a production system.

---

60. The Provider Is a Dependency, Not the Product

It is easy to build an application whose architecture becomes:

Frontend
   │
   ▼
LLM

But a useful product usually contains much more:

Users
Authentication
Authorization
Business State
Database
Workflows
Policies
Observability
Billing
Tools
Search
Knowledge
        │
        ▼
       LLM

The LLM is an important capability.

It is not the entire application.

This becomes especially important when we move from demos to products.

---

Practical Exercise — Design the Provider Boundary

You are building:

POST /api/support/classify

The endpoint receives:

{
  "message": "I was charged twice."
}

The system uses an LLM to classify the message.

Design the high-level Java architecture.

Include:

Controller
Application Service / Use Case
Classifier Interface
LLM Adapter
Provider Client

Decide where these responsibilities belong:

• prompt construction
• model configuration
• response parsing
• schema validation
• provider error handling
• business-level classification result

Do not use a framework yet.

Focus on boundaries.

---

Practical Exercise — Retry or Not?

For each provider response, decide whether you would generally consider retrying.

A. HTTP 429

B. HTTP 401

C. HTTP 503

D. Context window exceeded

E. Connection timeout

F. Unknown model identifier

G. Temporary network failure

For cases where you would retry, explain:

• whether the retry should be immediate
• whether backoff should be used
• whether there should be a maximum attempt count

---

Practical Exercise — Calculate Agent Amplification

Suppose:

10,000 user requests / day

Each user request causes on average:

3 model calls

Each model call averages:

4,000 input tokens
500 output tokens

Calculate:

1. model calls per day
2. input tokens per day
3. output tokens per day
4. total tokens per day

Then imagine an agent bug increases average model calls from:

3

to:

9

Calculate the new usage.

The goal is to understand why agent-loop observability matters.

---

Practical Exercise — Design an Execution Budget

You are building an order-support agent.

A single user request may invoke multiple models and tools.

Define reasonable categories of limits for one execution.

Do not worry about exact numbers yet.

Consider:

maximum model calls
maximum tool calls
maximum input tokens
maximum output tokens
maximum wall-clock duration
maximum retries

Then answer:

What should happen when the agent reaches one of these limits?

---

Practical Exercise — Model Routing

Your system has three model classes:

Fast Model
Cheap, low latency, moderate capability

Reasoning Model
Expensive, slower, strong reasoning

Multimodal Model
Can process images

Choose a model for:

A. Classify a support ticket.

B. Analyze an uploaded architecture diagram.

C. Generate a short title.

D. Investigate a complex production incident.

E. Extract a category from a simple message.

F. Understand a screenshot of an error.

Explain why.

---

Practical Exercise — Failure Classification

Classify each incident as primarily:

Network
Provider
Request
Model Behavior
Application

Incident A

The model returns a correct JSON response, but your parser throws an exception.

Incident B

The provider returns HTTP 503.

Incident C

Your application accidentally sends 300,000 tokens to a model that supports less.

Incident D

The request succeeds but the model selects the wrong tool.

Incident E

Your application gives Customer A's order data to Customer B's request.

Incident F

The connection to the provider times out.

Then explain which team/component should likely own the fix.

---

Design Exercise — Production LLM Gateway

Imagine your organization eventually has:

Support Agent
Course Content Assistant
Document Summarizer
Code Assistant
Search Assistant

All use LLMs.

Design a high-level shared infrastructure layer that could provide:

provider configuration
authentication
timeouts
retries
rate limiting
usage metrics
cost attribution
tracing

But avoid creating one giant business-level:

AIService.ask(String prompt)

Explain what should be centralized and what should remain inside individual product capabilities.

---

Questions You Should Be Able to Answer

Before moving to the next lesson, make sure you can explain these clearly.

1. What is the difference between a model and a provider?

A model performs inference.

A provider operates infrastructure and APIs through which applications access models.

2. Why should an LLM provider be treated as an external dependency?

Because model calls involve remote infrastructure and can experience latency, timeouts, rate limits, failures, and availability problems.

3. Why should provider credentials normally remain on the backend?

To prevent exposing secrets and to allow the backend to enforce authentication, authorization, quotas, policies, and cost controls.

4. Why do LLM systems need timeouts?

Because model requests can be slow or fail to return, and undefined waiting can consume application resources and create poor user experience.

5. Why must retries be designed carefully?

Because retries increase cost and may interact with side-effecting agent workflows.

6. Why are idempotency keys important for agent tools?

They can prevent repeated execution of the same side-effecting operation when network failures or retries occur.

7. Why are token limits relevant to rate limiting?

Two requests can consume dramatically different inference resources depending on their input and output token counts.

8. Why should agent executions have budgets?

Because multi-step loops can otherwise consume unbounded time, tokens, model calls, tools, and money.

9. Why isn't switching providers equivalent to switching database replicas?

Different models can exhibit different behavior even if they expose similar APIs.

10. What does a Java abstraction protect you from?

It can reduce provider-specific code coupling.

It does not guarantee behavioral equivalence between models.

11. Why should model configuration be versioned and evaluated?

Because changing models or generation settings can change production behavior.

12. Why should provider metadata be observed?

Token usage, latency, finish reasons, request identifiers, and errors are important for debugging, reliability, and cost management.

13. Why is AIService.ask(String prompt) often a weak domain abstraction?

Because it exposes a generic text interface rather than expressing meaningful business capabilities and contracts.

14. What should remain deterministic even when an LLM is involved?

Authorization, business invariants, policy enforcement, side-effect validation, financial constraints, and other guarantees that must always hold.

---

Key Takeaways

An LLM does not live inside your Java method simply because a framework gives you a convenient method call.

The real architecture is closer to:

Application
     │
     ▼
Network
     │
     ▼
Provider API
     │
     ▼
Model
     │
     ▼
Inference
     │
     ▼
Provider Response
     │
     ▼
Application

That boundary introduces familiar distributed-system concerns:

timeouts
retries
rate limits
backpressure
availability
idempotency
observability
cost

while the model itself introduces another category:

probabilistic behavior

A strong agent engineer understands both.

The provider should therefore be treated as a metered, remote, fallible dependency—not as a magical function that converts strings into intelligence.

And the application should maintain clear ownership of:

authentication
authorization
business rules
state
budgets
validation
reliability

The model contributes reasoning and language capabilities inside those boundaries.

The principle to remember is:

Abstract the provider where it helps your architecture, but never abstract away your understanding of what is actually happening.

In the next lesson, we will make model selection more systematic.

We will study model capabilities and model selection: reasoning models, fast models, multimodal models, context size, latency, cost, tool-use quality, structured outputs, benchmarks, evaluations, and how to choose the right model for a production workload instead of simply choosing the most powerful model available.