LLM Foundations
Embeddings and Semantic Representation
আপনি একটি free preview lesson দেখছেন।
Large Language Models generate text.
But many AI systems also need to answer a different kind of question:
Which pieces of information are semantically similar?
For example:
"Java backend engineer"
and:
"Spring developer"
use different words, but their meanings are related.
Traditional keyword search may struggle when exact words differ.
This is where embeddings become useful.
An embedding converts text, images, or other content into a numerical representation called a vector.
Conceptually:
"Java backend engineer"
│
▼
Embedding Model
│
▼
[0.18, -0.42, 0.73, ..., 0.11]
That vector captures aspects of the semantic meaning of the input.
Once information is represented numerically, we can compare vectors and estimate how similar their meanings are.
Embeddings become foundational for:
- semantic search
- retrieval
- RAG
- long-term memory
- recommendation systems
- clustering
- duplicate detection
- similarity matching
This lesson focuses on the underlying concept.
We are not building a RAG system yet.
First, we need to understand the primitive that makes many retrieval systems possible.
1. Keyword Matching and Semantic Matching Are Different
Suppose a user searches for:
distributed database failures
A keyword-based system may look for documents containing terms such as:
distributed
database
failures
Now consider a document containing:
How replicated storage systems behave during node outages
Semantically, this may be highly relevant.
But it shares few exact keywords with the query.
A purely lexical search system may miss it.
Semantic search aims to understand that:
distributed database
is related to:
replicated storage system
and:
failure
is related to:
node outage
Embeddings provide one way to represent that relationship numerically.
2. What Is an Embedding?
An embedding is a numerical representation of an input.
For text:
Text
│
▼
Embedding Model
│
▼
Vector
For example:
"Agent engineering"
might become something conceptually like:
[
0.12,
-0.47,
0.83,
0.05,
...
]
The real vector may contain hundreds or thousands of dimensions depending on the embedding model.
The individual numbers are usually not meaningful to us directly.
We generally do not interpret:
dimension 437 = 0.721
as:
"this represents backend engineering"
Instead, meaning emerges from the vector as a whole.
3. Think of Embeddings as Coordinates in Semantic Space
A useful mental model is to imagine every piece of text being placed somewhere in a high-dimensional semantic space.
In a simplified two-dimensional illustration:
Backend Engineering
●
●
Java Developer
●
Spring Boot
Chocolate Cake
●
Football
●
Related concepts tend to appear closer together.
Unrelated concepts tend to appear farther apart.
Real embeddings use many more than two dimensions, so we cannot visualize them directly.
But the conceptual idea remains:
Similar meaning tends to produce nearby vectors.
4. Embedding Models Are Different from Generative Models
A generative model is typically used like this:
Input
│
▼
Generative Model
│
▼
Generated Text
For example:
Explain eventual consistency.
might produce:
Eventual consistency is a consistency model where...
An embedding model behaves differently:
Input
│
▼
Embedding Model
│
▼
Vector
It does not primarily generate a natural-language answer.
It produces a representation suitable for mathematical comparison.
Conceptually:
Generative Model
→ language generation
Embedding Model
→ semantic representation
These are different capabilities.
Some providers may offer both, but you should not treat them as interchangeable.
5. Why Vectors?
Computers are very good at comparing numbers.
Once two pieces of text become vectors:
Vector A
Vector B
we can calculate how similar they are.
For example:
"Java backend developer"
might map to:
Vector A
and:
"Spring Boot engineer"
to:
Vector B
Then a similarity function gives us a score.
Conceptually:
similarity(A, B) = 0.89
Now compare:
"Java backend developer"
with:
"Chocolate cake recipe"
perhaps:
similarity(A, C) = 0.11
The values are illustrative.
The important point is that semantic similarity becomes something we can rank numerically.
6. Vector Dimensions
Suppose an embedding model produces:
1,536-dimensional vectors
Then each input becomes something like:
[
d1,
d2,
d3,
...
d1536
]
Another model might use:
768 dimensions
or:
3,072 dimensions
The exact dimensionality depends on the model.
More dimensions do not automatically mean:
better embeddings
Embedding quality depends on how well the model represents relationships relevant to your task.
Do not choose an embedding model based only on vector size.
7. Similarity Metrics
Once we have vectors, we need a way to compare them.
Common similarity or distance measures include:
Cosine Similarity
Dot Product
Euclidean Distance
You do not need deep mathematics to use embeddings effectively, but you should understand what they represent.
8. Cosine Similarity
Cosine similarity measures how similar the direction of two vectors is.
Conceptually:
Vector A ↗
Vector B ↗
If they point in similar directions, cosine similarity is high.
If they point in very different directions, similarity is lower.
For many embedding systems, cosine similarity is commonly used for semantic search.
Conceptually:
cosineSimilarity(query, document)
produces a ranking signal.
You do not need to manually implement the formula in most production systems.
But understand that semantic search is not magic.
It ultimately involves comparing numerical representations.
9. Distance vs Similarity
Some systems expose:
similarity
where larger values mean:
more similar
Others expose:
distance
where smaller values mean:
more similar
For example:
Similarity:
0.95 → very similar
0.20 → less similar
versus:
Distance:
0.05 → very close
0.80 → far apart
Do not blindly compare scores across databases or models without understanding their metric.
10. Embedding Scores Are Model-Dependent
Suppose one embedding model returns:
0.82
for two related sentences.
Another model may return:
0.67
for the same pair.
That does not automatically mean the first model is better.
Similarity scores depend on:
- embedding model
- normalization
- similarity metric
- content type
- task
Therefore avoid universal rules such as:
similarity > 0.8 means relevant
unless that threshold has been evaluated for your workload.
11. Semantic Similarity Is Not Exact Equality
Consider:
User cancelled the subscription.
and:
The subscription was terminated by the customer.
Their exact strings are different.
Their meaning is close.
Embeddings may place them near one another.
Now consider:
The user wants to cancel the subscription.
This is also similar.
But there is an important semantic difference:
cancelled
versus:
wants to cancel
One describes a completed action.
The other describes intent.
A similarity model may still consider them very close.
This leads to an important principle:
Semantic similarity is useful for retrieval, but similarity is not the same as logical equivalence.
Do not use embeddings to enforce precise business rules.
12. Embeddings Are Not a Replacement for Structured Data
Suppose your database stores:
customerId
orderId
createdAt
status
amount
Do not convert everything into embeddings and abandon structured queries.
If you need:
all orders where status = SHIPPED
use the database:
SELECT *
FROM orders
WHERE status = 'SHIPPED';
If you need:
orders created after August 1
use a range query.
Embeddings are valuable when the query is semantic:
Find support tickets discussing problems with duplicate payments.
A strong system often combines:
Structured Filtering
+
Semantic Search
rather than choosing one or the other.
13. Semantic Search
Suppose we have these documents:
Document A:
How to reset your password
Document B:
Troubleshooting duplicate card charges
Document C:
Tracking delayed shipments
Document D:
Configuring Spring Boot database connections
The user asks:
Why did my card get billed twice?
A semantic search system may:
1. Embed the query
2. Compare the query vector to document vectors
3. Rank documents by similarity
Conceptually:
Query
│
▼
Embedding Model
│
▼
Query Vector
│
▼
Compare With Stored Vectors
│
▼
Ranked Results
Likely:
1. Document B
2. ...
even though the query says:
billed twice
while the document says:
duplicate card charges
That is the power of semantic representation.
14. Precompute Embeddings for Stored Content
Suppose you have:
100,000 knowledge-base documents
You do not want to re-embed every document every time a user asks a question.
Instead:
Document
│
▼
Embedding Model
│
▼
Vector
│
▼
Store
This is done ahead of time.
Then at query time:
User Query
│
▼
Embedding Model
│
▼
Query Vector
│
▼
Search Stored Vectors
This is much more efficient.
15. Vector Stores
A vector store is a system capable of storing vectors and searching for nearby vectors efficiently.
Conceptually:
Document ID
Document Metadata
Embedding Vector
For example:
id: doc-8192
title: Refund Policy
embedding:
[0.13, -0.82, ...]
At query time:
Query Vector
│
▼
Vector Store
│
▼
Nearest Neighbors
Possible technologies include:
- vector databases
- search engines with vector support
- relational databases with vector extensions
- in-memory indexes
The technology is secondary.
The architecture is what matters.
16. Nearest-Neighbor Search
Suppose we have millions of vectors.
Comparing the query against every single vector could become expensive.
Vector-search systems often use nearest-neighbor indexes to efficiently locate vectors that are likely to be close to the query.
Conceptually:
Query Vector
│
▼
Vector Index
│
▼
Top K Nearest Results
For example:
topK = 5
might return the five most semantically similar documents.
You do not need to understand every indexing algorithm yet.
The important distinction is:
Embedding
=
representation
while:
Vector Index
=
efficient search mechanism
They are not the same thing.
17. Top-K Retrieval
A semantic search often asks for the nearest K items.
For example:
topK = 3
might return:
1. Duplicate Payment Troubleshooting
2. Refund Policy
3. Payment Failure Guide
Choosing K is a trade-off.
Too small:
relevant information may be missed
Too large:
irrelevant information may enter context
Later, when we build RAG systems, retrieval size becomes a context-engineering decision.
18. Similarity Thresholds
Instead of always returning exactly five results, you might require:
similarity >= threshold
Conceptually:
Candidate A → 0.91 → include
Candidate B → 0.84 → include
Candidate C → 0.56 → maybe exclude
Candidate D → 0.31 → exclude
But threshold selection must be evaluated.
A threshold that works for:
support tickets
may not work for:
source code
or:
legal documents
Do not copy arbitrary values from tutorials.
19. Keyword Search Still Matters
Semantic search is powerful.
But keyword search can outperform it for certain queries.
Suppose the user searches:
ERR_PAYMENT_8192
This is an exact identifier.
Keyword search is likely better.
Similarly:
java.lang.OutOfMemoryError
exact lexical matching may be extremely useful.
Semantic search is valuable for:
conceptual similarity
Keyword search is valuable for:
exact terminology
identifiers
codes
names
Many strong retrieval systems combine both.
20. Hybrid Search
A hybrid search system combines lexical and semantic signals.
Conceptually:
User Query
│
├──────────────┐
▼ ▼
Keyword Search Vector Search
│ │
└──────┬───────┘
▼
Combine Ranks
│
▼
Results
This can provide better retrieval when queries include both:
semantic meaning
and:
important exact terms
For example:
Kafka ISR shrinking after broker restart
contains:
ISR
as an important exact technical term, while the rest of the query has semantic meaning.
Hybrid search can capture both.
21. Metadata Filtering
Suppose your knowledge base contains documents from:
Product A
Product B
Product C
The user is asking about:
Product B
Instead of searching everything semantically, filter first:
product = Product B
then run vector search.
Conceptually:
Query
│
▼
Metadata Filter
│
▼
Relevant Subset
│
▼
Semantic Search
Metadata may include:
organizationId
language
documentType
product
createdAt
visibility
This improves both relevance and security.
22. Authorization Must Happen Before Retrieval Results Reach the Model
Suppose multiple organizations use the same vector store.
Bad architecture:
Search all organizations
│
▼
Return closest documents
│
▼
Tell model:
"Only use documents belonging to the current organization."
This is unsafe.
A stronger design:
Authenticated User
│
▼
Organization / Permission Filter
│
▼
Authorized Search Space
│
▼
Vector Search
│
▼
Model Context
The model should not receive unauthorized documents in the first place.
Vector search does not replace authorization.
23. Embedding Sensitive Data Has Security Implications
An embedding is numerical.
That does not mean:
safe to expose publicly
Embeddings are derived from source information.
They should still be treated as data assets.
Consider:
- access control
- tenant isolation
- retention
- deletion
- provider data handling
- privacy requirements
Do not assume vectors are automatically anonymized.
24. Embedding Models Must Match Between Indexing and Querying
Suppose you index documents using:
Embedding Model A
Later, you generate query vectors using:
Embedding Model B
These vectors may live in incompatible semantic spaces.
Conceptually:
Documents
│
▼
Model A
│
▼
Vector Space A
while:
Query
│
▼
Model B
│
▼
Vector Space B
Comparing them may be meaningless.
Therefore:
Use compatible embeddings for documents and queries within a given index.
Changing embedding models usually requires re-embedding indexed content.
25. Embedding Model Changes Are Data Migrations
Suppose you have:
10 million stored embeddings
and decide to switch models.
You may need to:
1. Generate embeddings with the new model
2. Build a new index
3. Evaluate retrieval quality
4. Migrate traffic
5. Retire the old index
This is not merely changing:
EMBEDDING_MODEL=model-b
in configuration.
The stored vectors themselves depend on the old model.
Treat embedding-model migration as a data migration.
26. Embedding Dimensions Must Match the Index
Suppose Model A generates:
1,536 dimensions
and Model B generates:
3,072 dimensions
A database column or vector index configured for:
1,536
cannot necessarily accept the new vectors.
Your storage design needs to know the dimensionality.
This is another reason model changes can require migration work.
27. Chunking and Embeddings
Imagine embedding an entire 200-page manual as one vector.
The resulting representation must summarize many unrelated topics into a single point.
That can make retrieval weak.
Instead, systems often split documents into smaller chunks:
Document
│
├── Chunk 1
├── Chunk 2
├── Chunk 3
└── ...
Each chunk gets its own embedding.
Now a query can retrieve the specific section containing relevant information.
Conceptually:
Document
│
▼
Chunk
│
▼
Embed Each Chunk
│
▼
Vector Store
Chunking becomes one of the most important design decisions in retrieval systems.
28. Chunk Size Is a Trade-off
Suppose chunks are extremely small:
one sentence each
Retrieval may be precise.
But context may be lost.
Suppose chunks are extremely large:
20 pages each
The relevant sentence may be buried inside unrelated material.
A useful chunk should usually contain enough context to preserve meaning while remaining focused.
There is no universal ideal chunk size.
It depends on:
- document structure
- task
- embedding model
- retrieval strategy
Evaluate it.
29. Chunk Boundaries Matter
Consider:
Refunds are allowed only when the request is submitted...
and the next chunk:
...within 30 days of delivery.
If retrieval returns only the first chunk, the policy is incomplete.
Naive fixed-character splitting can damage meaning.
Possible strategies include splitting by:
paragraph
section
heading
sentence boundaries
semantic boundaries
sometimes with overlap.
The best approach depends on the content.
30. Chunk Overlap
One common technique is to overlap chunks.
Conceptually:
Chunk 1:
A B C D
Chunk 2:
C D E F
Chunk 3:
E F G H
This reduces the chance that important context is lost exactly at a boundary.
But overlap has costs:
- more embeddings
- more storage
- duplicated retrieval results
- more indexing work
Again, this is an engineering trade-off.
31. Store Metadata with Every Chunk
A useful vector record should usually contain more than:
embedding
For example:
chunkId
documentId
section
title
content
organizationId
permissions
createdAt
embedding
Why?
Because after retrieving a vector, you need to know:
What document did this come from?
Can the current user access it?
What content should be sent to the model?
Where can we cite it from?
Embedding storage is part of a larger information architecture.
32. Retrieval Is Not Yet RAG
Semantic search:
Query
│
▼
Retrieve Relevant Documents
RAG adds another step:
Query
│
▼
Retrieve Relevant Documents
│
▼
Add Them to Model Context
│
▼
LLM Generates Answer
RAG stands for:
Retrieval-Augmented Generation
The retrieval system augments the model's context with external information before generation.
We will study RAG deeply in a later module.
For now, understand:
Embeddings
→ can enable semantic retrieval
Retrieval
→ can provide external context
Generation
→ can use that context
These are separate steps.
33. RAG Does Not "Teach" the Model Permanently
Suppose you retrieve:
Company refund policy
and include it in one model request.
The model can use it for that request.
This does not normally mean:
the model has permanently learned the policy
The information exists in the current context.
Future requests need to retrieve or provide it again if needed.
Conceptually:
External Knowledge
│
▼
Retrieval
│
▼
Current Context
│
▼
Model
This is another application of the principle:
Context is not permanent model memory.
34. Embeddings Can Support Long-Term Memory
Suppose an assistant stores useful facts from past conversations:
User prefers concise architecture explanations.
User is working on a Java agent system.
User previously chose PostgreSQL for persistence.
Later, a query can be embedded and matched against stored memory embeddings.
Conceptually:
Current Conversation
│
▼
Query Embedding
│
▼
Memory Vector Store
│
▼
Relevant Past Facts
│
▼
Current Context
This is one way to implement semantic memory.
But memory introduces difficult questions:
- what should be stored?
- what should expire?
- what if stored information becomes outdated?
- what if memories conflict?
- what information is sensitive?
We will address these later.
35. Similarity Does Not Mean Relevance
Suppose the query is:
Can I get a refund for a damaged item?
A vector store might retrieve:
General Refund Policy
with high semantic similarity.
But maybe the user purchased:
a non-refundable custom product
The generic policy is semantically similar but not relevant to this specific transaction.
Retrieval relevance depends on more than semantic similarity.
You may need:
metadata
business state
filters
reranking
current context
Do not assume:
nearest vector = correct answer
36. Similarity Search Can Return Plausible but Wrong Context
This is a subtle failure mode.
Suppose a database contains:
Policy 2024
Policy 2025
Policy 2026
All discuss refunds.
The user asks:
What is our current refund policy?
Semantic similarity may rank all three highly.
If metadata does not prioritize:
current version
the model may receive an outdated policy.
The LLM may then answer fluently and incorrectly.
The failure began in retrieval, not generation.
This is why production debugging should trace:
query
retrieved documents
model context
model response
37. Retrieval Quality and Generation Quality Are Different
Suppose the final answer is wrong.
Two major possibilities:
The wrong information was retrieved.
or:
The correct information was retrieved,
but the model used it incorrectly.
These require different fixes.
Conceptually:
Final Answer Wrong
│
├── Retrieval Failure
│
└── Generation Failure
This distinction becomes central when evaluating RAG systems.
38. Reranking
Initial vector search may retrieve:
20 candidate documents
A second stage can rank them more precisely.
Conceptually:
Query
│
▼
Vector Search
│
▼
20 Candidates
│
▼
Reranker
│
▼
Best 5
A reranker may use a model optimized for comparing a query with candidate documents.
This can improve precision.
But it also adds:
- latency
- infrastructure
- cost
Use it when evaluation shows value.
39. Semantic Search Works Best When the Search Unit Makes Sense
Suppose your vector index contains one record per:
entire book
The search unit may be too broad.
Suppose it contains one record per:
single word
The unit may be too narrow.
The search unit should align with the information users need.
For documentation, this may be:
section
paragraph group
small page segment
For support tickets:
whole ticket
may sometimes be appropriate.
For source code:
method
class
module
might make sense depending on the task.
Embedding design begins with the information unit.
40. Code Embeddings
Embeddings are not limited to natural language.
Some embedding models can represent:
source code
semantically.
For example, these may be related:
public Customer findCustomer(UUID id) {
return repository.findById(id).orElseThrow();
}
and:
Retrieve a customer by identifier or fail if the customer does not exist.
Code embeddings can support:
- semantic code search
- repository retrieval
- coding agents
- documentation matching
But code retrieval also benefits heavily from exact identifiers and lexical search.
Hybrid strategies are often valuable.
41. Multilingual Embeddings
Some embedding models can place semantically equivalent text from different languages near each other.
Conceptually:
"How do I reset my password?"
"Comment réinitialiser mon mot de passe ?"
"Wie setze ich mein Passwort zurück?"
may map to similar regions.
This can enable cross-language search.
But multilingual quality varies by model and language.
Do not assume equal retrieval quality across every language.
Evaluate with the languages your application actually supports.
42. Embeddings Can Support Clustering
Suppose you have:
100,000 customer support tickets
You can embed them and cluster similar vectors.
Possible groups might emerge around:
payment failures
login problems
shipment delays
refund requests
This can help with:
- topic discovery
- support analytics
- dataset exploration
Unlike classification, clustering may discover patterns without predefined labels.
But interpreting the clusters still requires analysis.
43. Embeddings Can Support Duplicate Detection
Suppose users create many support tickets saying:
My payment failed.
Card payment is not working.
Checkout says my card was declined.
Semantic similarity can help identify that these tickets may describe related problems.
This can support:
- deduplication
- incident correlation
- grouping
But be cautious.
Two semantically similar tickets may still describe separate incidents.
Similarity provides a signal, not certainty.
44. Embeddings Can Support Recommendations
Suppose a learner completes:
Java Collections
and the platform wants to recommend related lessons.
Lesson descriptions can be embedded and compared.
Conceptually:
Current Lesson
│
▼
Embedding
│
▼
Find Similar Lessons
Potential recommendations:
Generics
Streams
Collection Performance
But production recommendation systems often combine embeddings with other signals such as:
user behavior
course sequence
difficulty
popularity
prerequisites
Embeddings are one useful signal among many.
45. Embedding Cost Exists Too
Generating embeddings is usually cheaper than large generative responses, but it is not free.
If you need to embed:
10 million document chunks
you should think about:
- provider cost
- processing time
- batch limits
- retries
- rate limits
- storage
- re-indexing
Embedding pipelines are production workloads.
For large datasets, use asynchronous processing rather than blocking user requests.
46. Indexing Pipeline
A typical document-ingestion pipeline may look like:
Document Uploaded
│
▼
Parse Document
│
▼
Clean / Normalize
│
▼
Split into Chunks
│
▼
Generate Embeddings
│
▼
Store Chunk + Metadata + Vector
│
▼
Ready for Search
Each step can fail independently.
For example:
parsing failure
embedding API timeout
database failure
invalid document
A mature indexing pipeline should be retryable and observable.
47. Idempotent Indexing
Suppose your worker crashes after storing half the chunks.
When processing restarts, you do not want duplicate records.
A useful design might assign stable identifiers:
documentId
chunkIndex
embeddingVersion
Then an operation can safely upsert:
(documentId, chunkIndex, embeddingVersion)
This is another example of ordinary backend engineering applying to AI infrastructure.
48. Version Your Embeddings
Suppose your chunking strategy changes.
Old index:
chunkSize = 2,000 characters
embeddingModel = model-a
New index:
semantic sections
embeddingModel = model-b
You may want metadata such as:
embeddingModel
embeddingVersion
chunkingVersion
Why?
Because retrieval behavior depends on these choices.
Without version information, debugging quality regressions becomes much harder.
49. Evaluate Retrieval with Real Queries
A retrieval system may look impressive in a demo.
That is not enough.
Build a dataset:
Query
Expected Relevant Document(s)
For example:
Query:
"Why was my card charged twice?"
Expected:
Duplicate Payment Guide
Then measure whether relevant documents appear in:
Top 1
Top 3
Top 5
Possible metrics include:
Recall@K
Precision@K
Mean Reciprocal Rank
You do not need to master these metrics yet.
The key principle is:
Retrieval quality should be measured against representative queries.
50. Do Not Evaluate Only the Final Answer
Suppose a RAG system gives a correct answer.
That does not necessarily mean retrieval was good.
Perhaps the model already knew the answer from training and ignored the retrieved documents.
Likewise, a wrong answer does not prove the vector search was bad.
Evaluate layers separately.
Conceptually:
Retrieval Evaluation
│
▼
Did we retrieve the right information?
Generation Evaluation
│
▼
Did the model use the information correctly?
End-to-End Evaluation
│
▼
Did the user get the correct result?
This layered evaluation will become important later.
51. Semantic Search Is Probabilistic Too
Embedding systems are not deterministic truth engines.
Given a query, the vector search returns:
most similar according to the embedding representation
not:
objectively correct documents
That means retrieval systems also need:
- evaluation
- thresholds
- filtering
- monitoring
- fallback behavior
Agent engineering contains multiple probabilistic layers.
Not just the final LLM.
52. Do Not Embed Everything Automatically
Before using embeddings, ask:
What problem are we solving?
If the requirement is:
Find order 8192.
use a direct lookup.
If the requirement is:
Find documents discussing duplicate-payment problems.
semantic search may help.
If the requirement is:
Find all orders above €500.
use structured filtering.
Embeddings are useful when semantic similarity is actually part of the problem.
53. A Java Mental Model
We are still focusing on fundamentals, not a specific framework.
Conceptually, we might have:
public interface EmbeddingModel {
Embedding embed(String text);
}
and:
public record Embedding(
float[] vector
) {
}
Then:
Embedding query =
embeddingModel.embed(
"duplicate payment problem"
);
And:
List<SearchResult> results =
vectorStore.search(
query,
5
);
A search result might contain:
public record SearchResult(
String documentId,
String content,
double score
) {
}
This simple interface hides:
tokenization
embedding inference
vector indexing
nearest-neighbor search
But now you know what those abstractions represent.
54. A Typical Semantic Retrieval Flow
Consider a documentation assistant.
User:
How do I fix duplicate payments?
│
▼
Embed Query
│
▼
Search Vector Store
│
▼
Retrieve:
1. Duplicate Payment Troubleshooting
2. Payment Retry Guide
3. Refund Policy
│
▼
Use Results
At this stage, we could simply show search results to the user.
No generative model is required.
If we later add generation:
Retrieved Content
│
▼
LLM
│
▼
Generated Answer
we have moved toward RAG.
55. Embedding Search Can Exist Without an LLM Chatbot
This distinction matters.
A system can use embeddings for:
search
recommendations
classification support
duplicate detection
clustering
without using a generative LLM at all.
Embeddings are an independent machine-learning primitive.
Do not mentally tie them exclusively to chatbots or agents.
56. Common Mistake: Treating Vector Search as a Database Replacement
Bad architecture:
Everything
│
▼
Vector Database
Then try to answer:
What is the current balance?
Which users signed up yesterday?
Is order 8192 shipped?
using semantic search.
These are structured-data problems.
Use systems according to their strengths.
A mature AI application may combine:
PostgreSQL
+
Search Engine
+
Vector Search
+
LLM
Each solves a different problem.
57. Common Mistake: Retrieving Too Much
Suppose a query retrieves:
50 chunks
and sends all of them to the LLM.
The context may become:
- expensive
- noisy
- contradictory
More retrieval is not automatically better.
A useful architecture may:
retrieve 20 candidates
rerank
select best 5
The right numbers depend on evaluation.
58. Common Mistake: Ignoring Document Freshness
Suppose your vector index contains an old refund policy.
The source document was updated, but the index was not.
The retrieval system continues returning outdated text.
This means ingestion systems need synchronization.
Possible strategies include:
update embedding when source changes
delete embeddings when source is removed
store source version
track indexedAt timestamp
RAG quality depends on data freshness.
59. Common Mistake: Ignoring Deletion
Suppose a user requests deletion of a document.
You delete the original file.
But its chunks and embeddings remain in the vector store.
The system may still retrieve the deleted content.
Deletion must propagate through:
source
chunks
embeddings
indexes
caches
AI data pipelines must respect data lifecycle requirements.
60. Common Mistake: Comparing Raw Scores Across Models
Suppose Model A returns:
0.74
and Model B:
0.85
for the same query-document pair.
You cannot automatically conclude:
Model B is better.
Scores exist within the geometry created by each model.
Evaluate retrieval outcomes, not isolated similarity values.
61. Common Mistake: Assuming Semantic Similarity Means Factual Relationship
Consider:
The customer cancelled the order.
and:
The customer wants to cancel the order.
These are semantically very similar.
But one describes:
completed state
and the other:
requested action
If your business logic needs the difference, do not rely on vector similarity alone.
Semantic similarity is useful for finding information.
It is not a substitute for precise state representation.
62. A Strong Mental Model
Think of embeddings as a transformation:
Meaningful Content
│
▼
Embedding Model
│
▼
Numerical Position in Semantic Space
Then:
Query Vector
│
▼
Compare With Stored Vectors
│
▼
Retrieve Semantically Related Information
This gives applications a way to search by meaning, not only by exact wording.
Practical Exercise — Keyword or Semantic Search?
Choose whether you would primarily use:
Keyword Search
Semantic Search
Structured Query
Hybrid Search
for each case.
Case A
Find:
ERR_PAYMENT_8192
in application logs.
Case B
Find documentation related to:
Why are my replicas temporarily inconsistent?
when the documentation uses the term:
eventual consistency
Case C
Find all orders where:
status = CANCELLED
Case D
Search source code for:
CustomerRepository
Case E
Find support tickets describing duplicate charges even when users use different wording.
Case F
Search technical documentation where exact terms such as Kafka, ISR, and broker matter, but semantic meaning matters too.
Explain your choices.
Practical Exercise — Similarity Is Not Equality
Consider:
Sentence A:
The user cancelled the subscription.
Sentence B:
The user wants to cancel the subscription.
Answer:
- Why might embeddings consider these sentences highly similar?
- Why would treating them as logically equivalent be dangerous?
- What kind of application state should represent whether cancellation actually occurred?
Practical Exercise — Design a Vector Record
You are indexing LiveKlass lesson content.
Design a record containing:
chunkId
courseId
moduleId
lessonId
title
content
language
embedding
Add any metadata you think would be useful for:
- filtering
- authorization
- source tracing
- re-indexing
Then explain which fields should participate in vector similarity and which should remain metadata.
Practical Exercise — Design an Indexing Pipeline
A course lesson is updated.
Design the processing flow:
Lesson Updated
│
▼
?
Your pipeline should consider:
- loading current content
- chunking
- deleting or replacing old chunks
- generating new embeddings
- storing metadata
- handling partial failure
- idempotency
Do not use a framework-specific solution.
Design the backend workflow.
Practical Exercise — Choose the Search Space
Suppose LiveKlass contains:
10,000 courses
A learner is currently inside:
Agent Engineering
and asks:
Where did we explain deterministic boundaries?
Would you search:
all platform content
or first filter by:
current course
Explain the trade-offs.
Then consider:
What if the learner explicitly asks for related material from other courses?
How should the search scope change?
Practical Exercise — Retrieval Security
A multi-tenant application stores documents from:
Organization A
Organization B
in the same vector database.
A user from Organization A performs semantic search.
Design the flow so Organization B's documents can never appear in the model context.
Do not rely on the prompt to filter results.
Show where authorization or metadata filtering belongs.
Practical Exercise — Embedding Migration
Your current index contains:
5 million vectors
generated by:
Embedding Model A
You want to migrate to:
Embedding Model B
Design a high-level rollout.
Consider:
- generating new embeddings
- running both indexes temporarily
- evaluating retrieval quality
- switching traffic
- rollback
- retiring the old index
Explain why simply changing the query-side model would be incorrect.
Design Exercise — Semantic Search for Course Content
Design a semantic search feature for LiveKlass.
A learner asks:
How does an LLM remember previous messages?
The system should find the most relevant lessons or lesson sections.
Design:
Course Content
│
▼
Chunking
│
▼
Embedding
│
▼
Vector Store
User Query
│
▼
Embedding
│
▼
Search
│
▼
Ranked Lesson Sections
Consider:
- chunk boundaries
- lesson metadata
- course filtering
- top-K
- links back to the original lesson
- content updates
Do not add generation yet.
This exercise is only about semantic retrieval.
Questions You Should Be Able to Answer
Before moving to the next lesson, make sure you can explain these clearly.
1. What is an embedding?
A numerical vector representation of content that captures aspects of its semantic meaning.
2. What is an embedding model?
A model that transforms input such as text into a vector representation rather than primarily generating text.
3. What does semantic similarity mean?
That two pieces of content have related meaning even if they do not use the same exact words.
4. Why are embeddings useful for search?
Because queries and documents can be compared numerically based on semantic similarity.
5. Is similarity the same as logical equivalence?
No.
Semantically similar statements can still differ in important factual or logical details.
6. What is a vector store?
A storage and indexing system capable of storing vectors and efficiently retrieving nearby vectors.
7. What is nearest-neighbor search?
Finding stored vectors that are closest to a query vector according to a similarity or distance metric.
8. What is top-K retrieval?
Returning the K most similar results for a query.
9. Should similarity thresholds be copied blindly from tutorials?
No.
Thresholds should be evaluated for the specific model, metric, and workload.
10. Why does keyword search still matter?
Exact identifiers, names, error codes, and technical terms may be better handled lexically.
11. What is hybrid search?
Combining lexical and semantic retrieval signals.
12. Why is metadata filtering important?
It improves relevance and can enforce boundaries such as tenant, product, language, or permission scope.
13. Can vector search replace authorization?
No.
Authorization must restrict which information can be searched and returned.
14. Why must documents and queries use compatible embedding models?
Because embeddings from different models may exist in incompatible vector spaces.
15. Why can changing embedding models require re-indexing?
Stored vectors were created by the previous model and cannot necessarily be compared meaningfully with vectors from the new one.
16. What is chunking?
Splitting larger content into smaller units that can be embedded and retrieved independently.
17. Why do chunk boundaries matter?
Poor boundaries can separate information that needs to remain together for meaning.
18. What is the relationship between embeddings and RAG?
Embeddings can support retrieval. RAG uses retrieved information to augment the context of a generative model.
19. Does RAG permanently teach new information to the model?
No.
Retrieved information normally becomes part of the current request's context.
20. Why should retrieval and generation be evaluated separately?
Because the final answer can fail either because the wrong information was retrieved or because the model incorrectly used good retrieved information.
Key Takeaways
Embeddings allow applications to represent meaning numerically.
Conceptually:
Text
│
▼
Embedding Model
│
▼
Vector
Semantically related content tends to produce vectors that are close according to an appropriate similarity metric.
This enables:
semantic search
retrieval
memory
recommendations
clustering
duplicate detection
But embeddings are not a replacement for normal databases or deterministic queries.
Use:
Structured Queries
for structured facts.
Use:
Keyword Search
for exact terms.
Use:
Semantic Search
when meaning matters more than exact wording.
And combine them when the problem benefits from multiple signals.
The most important principle is:
Embeddings are a retrieval representation, not a source of truth.
The nearest document is not automatically the correct document.
Similarity is not equality.
Retrieval does not replace authorization.
And embedding quality must be evaluated against real application queries.
Later, when we study RAG, we will build on this foundation:
Query
│
▼
Retrieve Relevant Information
│
▼
Place It in Model Context
│
▼
Generate an Answer
Before that, we have one more important foundation to establish.
In the next lesson, we will study LLM Limitations and Failure Modes.
We will bring together hallucination, stale knowledge, context limitations, nondeterminism, reasoning failures, instruction failures, prompt injection, unreliable confidence, and other behaviors that every agent engineer must design around.