Project 06 of 09
Latency is the feature.

Voice desk

The Warranty Desk agent you can talk to. One WebSocket to the Voice Live API, the same tools underneath, and a number per turn that tells you whether a customer would hang up.

build time: 5 to 7 hours
cost: Under $10 if torn down the same day³
services: Voice Live API, Azure Speech, Foundry models, Azure Functions, Container Apps, Application Insights
assumes: Project 01. The order and warranty Functions are reused as they are.
exams: Overlaps AI-103
vocabulary: Speech-to-speech, end-of-turn detection, barge-in, time to first audio, function calling, VoiceRAG

The problem

Half of Campux Retail's warranty questions still arrive by phone, from people holding a broken appliance in the other hand.

The support lead's requirements are about feel, not features. The caller should be able to interrupt. Silence after a question should be short enough that nobody says "hello?" The agent must look up the real order, the same way the chat version does, and never read a warranty date it did not fetch. And she wants the one number that predicts hang-ups: how long from the caller finishing a sentence to the agent starting to speak, per turn, on a chart.

definition: Time to first audio. The gap between the caller's end of turn and the first audio frame back. Everything in this project either shortens it or measures it. Above about a second, callers start talking over the agent.

The architecture

One WebSocket, speech in and speech out. Everything else is the same agent.

livepress play to follow one spoken turn

Figure 9 One WebSocket replaces recognition, turn handling, the model and synthesis as separate services. The tools and the no-guessing rule are Project 01's, untouched.

Speak. The caller asks a question into a phone or a browser microphone.
Stream. The voice app streams audio to the Voice Live API over a single WebSocket session.
Reply. The realtime model starts speaking, with a filler phrase if a tool call is coming.
Handle the turn. Voice Live detects the end of turn, suppresses noise, cancels echo, and stops on interruption.
Call a tool. The model calls the warranty tool; without a result it will not say a date.
Look it up. The Project 01 Azure Function answers, in the same region to keep the round trip short.
Measure. Time to first audio for the turn is sent to Application Insights as a custom metric.

Service map, for readers coming from AWS
Job	AWS version	This build	Same idea?
Speech to speech	Nova Sonic, or Transcribe plus Polly	Voice Live API, one managed endpoint¹	Yes. Here the pipeline is one service
Turn handling	Your own VAD	Built in: end-of-turn detection, interruption, noise suppression, echo cancellation	Different. Included
Model	Nova Sonic	gpt-realtime-mini, or a text model plus Azure speech	Yes
Tools	Lambda via tool use	Function calling to the Project 01 Azure Functions	Yes
Telephony	Amazon Connect	Out of scope; a browser microphone stands in²	Same idea

The build

In this order. Measure latency from step 3 on; every later step is judged by that chart.

Hello, out loud
A small server on Container Apps opens a WebSocket to the Voice Live API with gpt-realtime-mini, and a browser page streams the microphone to it and plays the audio back. Say hello.
It answers out loud within a couple of seconds, and you can interrupt it mid-sentence and it stops.
The instructions, spoken
Reuse the Project 01 instructions with one paragraph added: short sentences, no lists, say numbers as words, confirm the order number back before looking it up.
Ask a policy question. The answer is two sentences, not a paragraph read aloud.
Measure every turn
On the server, timestamp the end-of-turn event and the first audio frame back, and send the difference to Application Insights as a custom metric with the turn number.
A chart of time to first audio per turn. Write down your baseline.
Tools, spoken
Register the two Project 01 Functions as function-calling tools on the session. When the model calls one, the server invokes the Function with a managed identity and returns the result.
"Is order one zero four two still covered?" produces a tool call and a spoken date that matches the chat agent's answer.
Fill the silence
Tool calls add hundreds of milliseconds. Instruct the model to say "let me check that" before calling a tool, and confirm the Functions are in the same region as the Voice Live resource.
Time to first audio on tool turns drops below the baseline, because the first audio is now the filler, and the total turn time did not grow.
Noise and barge-in
Turn on noise suppression and echo cancellation on the session. Test from a laptop speaker with a fan running. Then interrupt the agent three times in one answer.
The agent does not answer its own voice, and each interruption stops it within a word.
Never guess a date, out loud
Stop the Function App. Ask for a warranty date. The rule from Project 01 applies with no exceptions: no tool result, no date.
It says it cannot check right now and offers a person. It does not say a date.
Pick the model with numbers
Run the same ten-turn script against gpt-realtime-mini and against gpt-5-mini with Azure speech in and out. Compare the latency chart and the tool-call correctness.
Two charts, one decision, and the reason written next to it.
Tear it down
```
azd down
```
Voice Live is billed per audio minute by tier; nothing idles expensively, but delete the resource group anyway.
The resource group is empty. Today's cost is under $10.

Where it breaks

Cause each one on purpose. Voice failures are heard before they are logged.

The agent that talks to itself Echo cancellation off, laptop speakers on. The agent hears its own answer and replies to it. Fix: the session setting, and a headset while you test everything else.
The three-second silence A tool call with no filler phrase. The caller says "hello?" and the barge-in cancels the answer that was about to start. Fix: step 5.
Numbers read wrong "o-1042" spoken as "oh, one thousand forty-two" and looked up as a different order. Fix: confirm the order number back, digit by digit, before the tool call.
The wrong region Voice Live in one region, Functions in another. Every tool turn pays a cross-region round trip. Fix: co-locate, and read the region list before creating the Speech resource.
The chat prompt read aloud The Project 01 instructions unchanged produce a bulleted list spoken as "bullet, bullet, bullet." Fix: the spoken-style paragraph in step 2.
Cost per minute, not per token Read the tier your model lands in and multiply by minutes of a real call. The basic tier model may be the right one for a support line even if the pro one sounds nicer.

The trade-offs

Native audio model, or text model plus speech?: A realtime model hears and speaks directly and is fastest. A text model with Azure speech in front and behind lets you keep exactly the Project 01 model and reasoning, at the cost of latency. Measure both; the chart decides.
Voice Live, or build the pipeline yourself?: Recognition, turn detection, the model and synthesis as separate services is more control and more latency and a lot of engineering. Voice Live is one endpoint that includes the conversational features. For a support line, take the endpoint.
Filler phrases, or honest silence?: "Let me check that" buys the tool call time and sounds human. Too many fillers sound like stalling. One, before tool calls only.
Browser microphone, or a phone number?: Telephony integration is a project of its own. This one proves the agent and the latency; the phone line is the next one.

In the interview

"Have you built a voice agent?"

decisionThe support agent over the Voice Live API: one WebSocket, speech in and out, the same tools underneath.
reasonThe conversational features, interruption and turn detection, are the hard part and they are included.
watchedTime to first audio per turn, on a chart, from the first day.

"How did you handle latency?"

decisionCo-located services, a filler phrase before tool calls, and a model choice made from the chart.
reasonAbove about a second of silence, callers talk over the agent.
watchedThe tool-call turns, which are the slow ones.

"What is different from the chat version?"

decisionSpoken-style instructions, digit-by-digit confirmation of order numbers, nothing else.
reasonThe tools and the no-guessing rule are the same; only the surface changed.
watchedThat it still refuses to say a date when the Function is down.

Evidence

The latency chart: Time to first audio per turn, baseline and after the filler and co-location changes.
A thirty-second recording: One question, one interruption, one tool call, one honest "I cannot check right now."
The repo: The WebSocket server, the spoken instructions, and the two-model comparison with the decision.
One sentence for the résumé: "Built a speech-to-speech support agent on the Voice Live API with function calling to existing Azure Functions, interruption handling, and per-turn latency instrumentation used to choose the model."

Project 07, Care Gap Report. A different client, a regulated one, and the discipline that PHI never reaches the model.

Notes

Checked against Microsoft Learn on 2 September 2026. The Voice Live API is a managed speech-to-speech endpoint over WebSocket, compatible with the Azure OpenAI Realtime API events, with noise suppression, echo cancellation, interruption detection and end-of-turn detection built in, and function calling for tools. Models include gpt-realtime, gpt-realtime-mini and text models such as gpt-5-mini with Azure speech in and out; pricing is tiered pro, basic and lite by model.
Real telephony, a phone number and a carrier, is out of scope here. The browser microphone exercises everything the agent does; the phone line is plumbing you add once the latency chart is where you want it.
"Under $10" assumes short test calls on a basic-tier model. Voice is billed by audio minutes and by tier, so a long test session on a pro-tier model costs more than an afternoon of chat.