Project 02 of 09
Thousands of documents, one honest agent.

Ground it

The Warranty Desk agent again, but answering from three thousand documents through a Foundry IQ knowledge base on Azure AI Search, with a hit rate you measured instead of a feeling.

build time: 5 to 7 hours
cost: Under $12 if torn down the same day³
services: Foundry Agent Service, Foundry IQ, Azure AI Search, Blob Storage, Entra ID, Application Insights
assumes: Project 01. Keep its agent and its instructions file; you replace one tool.
exams: Overlaps AI-103, AI-200
vocabulary: Knowledge base, knowledge source, agentic retrieval, chunk, groundedness, hit rate, citation

The problem

Three policy PDFs was the demo. Campux Retail actually has 3,000 documents: manuals, recall notices, regional policies, supplier warranties, most of them PDFs nobody has opened since they were uploaded.

File search was right for three files and wrong for three thousand. The head of support has two new requirements. First, when the agent quotes a policy it has to say which document and which page, so a person can check it. Second, she wants a number: out of a hundred real customer questions, how many did the agent answer from the right document. Not "it seems good." A number, and a way to get it again next month after the documents change.

definition: Agentic retrieval. Instead of one search for the user's words, the knowledge base plans several searches, runs them, checks whether the results are good enough, and searches again if not. Foundry IQ is that engine, running inside Azure AI Search, exposed to your agent as one tool.

The architecture

One tool swapped. The agent does not read 3,000 documents; it asks a knowledge base that does.

livepress play to follow one question through the knowledge base

Figure 5 One tool changed from Project 01. The knowledge base does the reading; the agent cites what it was given, or refuses.

Sign in. The customer opens the same Streamlit app and signs in with Microsoft Entra ID.
Call the agent. The app calls the Warranty Desk agent as the customer.
Reason. The model decides the question needs the knowledge tool.
Ask the knowledge base. The Foundry IQ tool passes the customer's question, not a search query.
Agentic retrieval. Azure AI Search plans several searches, runs them, checks relevance, reranks.
Read the documents. Chunks come back from the indexed Blob source with document and page references.
Score it. The cited document is compared with the expected one; groundedness is scored.
Trace it. Hit rate and groundedness are recorded per run in Application Insights.

Service map, for readers coming from AWS
Job	AWS version	This build	Same idea?
Knowledge base	Bedrock Knowledge Base	Foundry IQ knowledge base on Azure AI Search	Yes. Foundry IQ plans and re-runs queries itself
Documents	S3 data source	Blob Storage knowledge source	Yes
Index and ranking	OpenSearch Serverless	Azure AI Search, hybrid plus semantic ranking	Yes
Retrieval tool	Agent action group	Foundry IQ tool on the agent¹	Yes
Quality number	Bedrock evaluations	Foundry evaluators: groundedness, retrieval	Yes. Project 03 goes deep
Identity	IAM	Managed identity, Search Index Data Reader	Different

The build

In this order. Step 2 is where most of the time goes; step 7 is the number she asked for.

Start from Project 01
Same Foundry project, same agent, same instructions file. Create an Azure AI Search service (Basic tier is enough) and a Blob Storage container called policies.
The Project 01 agent still answers from its three PDFs. Nothing is broken yet.
The documents
Generate 3,000 synthetic policy documents from a script: 40 product lines, 5 regions, 3 years of revisions, and a handful of deliberate contradictions where a newer document supersedes an older one. Upload them to the container.
3,000 blobs. You can name three that contradict each other and say which one is current.
Knowledge source and knowledge base
In Azure AI Search, create a knowledge source pointing at the container (indexed, with the indexer's execution environment left default for now) and a knowledge base over it. Give the Search service a system-assigned identity with Storage Blob Data Reader on the container.
The index reports about 3,000 documents and the indexer run shows zero failures.
Swap the tool
Remove file search from the agent. Add the Foundry IQ knowledge tool pointing at the knowledge base.¹ Give the agent's identity Search Index Data Reader on the service. Change one line of the instructions: "cite the document title and page for every policy statement."
"Is water damage covered on an AquaGuard washer sold in Ohio in 2025?" returns an answer with a document title and page, and the document is the current one.
Make it refuse
Ask about a product that does not exist. Then ask about a real product with a question none of the 3,000 documents answers. Add to the instructions: "if the knowledge base returns nothing relevant, say so and offer a person; never answer from general knowledge."
Both questions get "I could not find that in our policies" and a handoff offer, not a guess.
Evaluation set
Write 100 questions with the expected document for each, including 15 that should get a refusal. Half from the support team's real inbox, rewritten; half adversarial: superseded policies, near-duplicate product names, a region that does not exist.
A CSV with 100 rows: question, expected document, expected behaviour.
Measure the hit rate
Run all 100 through the agent with tracing on. For each, record whether the cited document matched the expected one. Run the Foundry groundedness evaluator over the answers. Write the two numbers down.
You have a hit rate and a groundedness score, both under 100, and a list of the misses.
Fix the worst miss, measure again
The misses will cluster: usually superseded documents outranking current ones. Add a "supersededBy" field to the index, filter it out at query time, rerun. Keep the first number; that is your before.
Hit rate went up. You can say by how much, and why.
Tear it down
```
azd down
```
Then delete the Search service and the storage account by hand; azd down only removes what azd made.
The resource group is empty. Today's cost is under $12.

Where it breaks

Cause each one on purpose. Retrieval fails quietly; these make it loud.

The confident wrong document A superseded 2023 policy outranks the 2025 one because it is longer and mentions the product more. The answer cites a real document and is still wrong. Fix: version metadata in the index and a filter, not a prompt.
The empty index The indexer ran, reported success, and indexed nothing, because the Search service's identity had no Blob reader role. Fix: check the document count, not the run status.
General-knowledge leakage Remove the "never answer from general knowledge" line and ask about a competitor's warranty. Watch it answer helpfully and wrongly. Put the line back.
The citation that is not there The agent cites "page 4" of a two-page document. The model invented the page. Fix: pass the retrieved chunk's real reference through, and instruct it to cite only what the tool returned.
The role you forgot The agent's identity lacks Search Index Data Reader; the tool returns a 403 the agent reports as "I could not find that." Which is the wrong kind of honest. Fix: make tool errors distinguishable from empty results.
Cost of curiosity Agentic retrieval runs several searches per question. Read the Search and token cost of your 100-question run, then multiply by a real week's volume.

The trade-offs

Foundry IQ, or plain Azure AI Search as a tool?: A plain search tool runs the query you give it. Foundry IQ plans several, checks the results, and tries again, which is what makes it survive vague customer questions. The cost is more searches per turn and a preview label. This build takes the preview because the hit rate is the point.
Indexed, or remote knowledge sources?: Indexing Blob content gives you ranking, filters and version metadata. Remote sources such as SharePoint are queried live and stay where they are. Policies are documents you own; index them.
Refuse, or answer with a warning?: "I could not find that" loses a customer who could have been helped. "Here is my best guess, please confirm" invents warranty terms. For warranty, refuse and hand off; the cost of a wrong answer is a refund.
Measure with a hundred questions, or a thousand?: A hundred gives you a number with a wide margin and a list of misses you can read in an afternoon. A thousand gives a tighter number and a list you will not read. Start at a hundred; grow it when the misses stop clustering.

In the interview

"How do you stop a RAG agent making things up?"

decisionA knowledge base that returns real chunks with references, an instruction to cite only those, and a refusal path when nothing relevant returns.
reasonThe model cannot hallucinate a citation it was never given, if it is only allowed to cite what it was given.
watchedA 100-question set with expected documents, scored before and after each change.

"What was your hit rate?"

decisionMeasured, not estimated: the percentage of questions where the cited document matched the expected one.
reasonA support lead cannot act on "seems good"; she can act on "82, and the misses are superseded documents."
watchedThe before and after of one index change.

"Why Foundry IQ over a search call?"

decisionAgentic retrieval: it rewrites and re-runs queries and ranks across them.
reasonCustomers do not ask in the words the policy uses.
watchedCost per question, since it searches more than once.

Evidence

The number, twice: Hit rate and groundedness before and after the version filter, with the date.
One miss, explained: A question, the wrong document it cited, the right one, and the index change that fixed it.
The repo: The document generator, the evaluation CSV, the index definition, and the scoring script.
One sentence for the résumé: "Grounded a support agent in 3,000 documents using a Foundry IQ knowledge base on Azure AI Search, with citation enforcement, a refusal path, and a measured retrieval hit rate tracked across index changes."

Project 03, Evaluate and red-team. Turn the hundred-question CSV into a scored run, then attack the agent on purpose.

Notes

Foundry IQ and its agent tool were in preview when this was written (September 2026), and the tool reaches the knowledge base over MCP. Names and the exact portal steps were checked against Microsoft Learn on 2 September 2026; if they have moved, the Learn page wins.
Azure AI Search knowledge bases can span several knowledge sources, including Blob Storage, SharePoint and OneLake. This project uses one indexed Blob source so the version filter is yours to control.
"Under $12" assumes a Basic Search tier, 3,000 small synthetic documents, one 100-question run, and teardown the same day. Search is billed by the hour while it exists; that is the line item that grows if you forget step 9.