Inside GPT-Live: full-duplex voice, one-round-trip startup

For voice AI, knowing when to speak is harder than it sounds. Human speakers can pass the conversational baton naturally in under a second, but earlier voice AI systems couldn't keep up. Their turn-based architecture relied on a small model called a turn detector, which faced a tricky task: judge too early and you interrupt the user; judge too late and responses feel sluggish. Only after the detector made its call could the much larger LLM get to work.

Our third-generation voice system, GPT-Live, removes the turn detector from the audio path. Its voice model is full-duplex, meaning it can listen and speak at the same time. That eliminates the need for a separate detector and makes conversations more immediate and natural. When deeper reasoning or tool use is needed, GPT-Live can also call frontier models like GPT-5.5 without interrupting the conversational flow. Together, these capabilities give GPT-Live an unprecedented combination of conversational responsiveness and intelligence.

Delivering this experience at scale required a new system architecture optimized for low latency. Unlike typical request-response inference, our system streams incoming audio into the voice model, streams the generated speech back to the user, and handles delegated tasks on a separate asynchronous path. Over the past six months, we redesigned model inference, context management, and media transport so that speech flows end to end without stutter.

The architecture also draws a clean boundary between the core voice path and application logic, making it easy to customize application behavior without affecting response speed. This foundation powers ChatGPT voice's expanding feature set, including the newly launched ability to control your computer and coordinate agents from the ChatGPT desktop app.

This article explains why our earlier turn-based system could not meet our needs, and how we designed a new system to stay responsive at every layer. We'll cover stateful inference, dynamic context management, asynchronous delegation, and protocol-level optimizations, and how they work together to make GPT-Live truly realtime.

Early voice architectures inherited the turn-based pattern of text LLMs, except each turn was a discrete chunk of audio instead of text. In cascaded systems, speech-to-text, the LLM, and text-to-speech ran serially, one after the other. That serial processing added latency and ignored cues like tone and speaking rate.

Speech-to-speech models improved on this by working with audio directly. Trained to understand and generate speech natively, they preserve details lost in transcription and respond faster. But the system still relied on a turn detector to decide when inference could begin. The model did more of the interactive work, yet interaction remained turn-based.

GPT-Live lets the voice model drive the conversation: audio flows continuously into and out of the model, while deeper reasoning and tool use happen asynchronously. The system's first priority is maintaining an uninterrupted media loop. Other work, like calling frontier models or persisting conversation state, happens off the realtime path.

Keeping that media loop flowing smoothly isn't always simple. Any delay in transport, processing, or inference can become an audible pause or glitch. Turn-based systems could tolerate some jitter in when audio chunks arrived, but a realtime media system must deliver every frame on time.

Earlier work on ChatGPT voice and the Realtime API laid an important foundation. We had already rebuilt our voice infrastructure so audio and video can flow into and out of the system with lower, more stable latency. GPT-Live pushes this design further, streaming media all the way into the model through a new stateful inference system built for continuous conversation.

Still, streaming inference is only part of the solution. To run reliably in production, we also had to ensure audio could travel dependably from the client to the inference stack and to cope with the challenges of stateful systems.

One early decision was to cleanly separate the media stream from application and business logic. Audio travels between the client and the voice model over a dedicated fast path. Delegation, tool use, and other application work sit behind an asynchronous RPC boundary. A slow tool call or backend service may delay its own result, but it won't block the media stream.

That separation also creates a clean boundary for customization. Applications can change tools, policies, and backend behavior without affecting the media front end responsible for keeping audio flowing. The realtime path stays lean, predictable, and focused on the work that must happen in real time.

We rewrote the media front end and inference logic in Go, replacing the previous Python asyncio implementation. This noticeably smoothed frame delivery—the new system's p95 matched the old system's p50.

WebRTC provides the transport foundation. It was designed for low-latency media and continues working even with packet loss, clock drift, or changing client connections. If a packet arrives late, WebRTC can slightly stretch the audio to avoid a gap, then briefly speed up playback to catch back up to real time.

By minimizing buffering and blocking throughout the system, we can achieve the sub-second responsiveness people expect from conversation.

Stateful inference brings its own operational trade-offs. A voice session can remain active for a long time, but its context keeps growing, and model instances spin up and down based on demand.

To address this, we built a seamless handoff mechanism across model instances. When a switch is needed, we warm up a replacement instance alongside the current one, prefill it with the current session context, run inference on both instances in parallel, and switch over once the new instance is fully ready.

The same basic mechanism also supports dynamic context compaction. As a conversation continues, accumulated context can eventually exceed the model's context limit. Compaction shrinks the context to fit, but it takes time. And because compaction changes past context, it also invalidates the model's key-value (KV) cache—the stored attention keys and values for previously processed tokens. Rebuilding that state requires another prefill, adding extra latency.

So we treat compaction as another controlled switch. The original model instance keeps the conversation going while the system compacts the context and prepares a replacement instance with the new context. Once that instance is ready, we switch over without interrupting the media stream. This lets the system support long calls and compact when necessary.

Heavy work never enters the realtime path, so even during handoffs, the conversation stays smooth and uninterrupted.

GPT-Live is powerful because it can call existing frontier models; in effect, it decouples 'speaking' from deeper 'thinking.' But making this dual-model architecture feel like one unified system required solving two related engineering problems.

First, results have to come back fast enough to be useful in an ongoing conversation. So we had to minimize latency across the entire delegation path, from routing and prompt processing to inference and tool calls. At the same time, other systems in the product still expect discrete messages, so we had to represent the continuous conversation in a form they can understand.

GPT-Live responds quickly and naturally while GPT-5.5 handles a search task in the background:

Audio 2

User — GPT-Live-1 — GPT-5.5 — Search + reason

Transcript: example conversation with GPT-Live-1 using GPT-5.5 Instant

Once a delegation is issued, we focus on minimizing the time the frontier model needs to produce something useful for the current conversation. While the frontier model is reasoning or using tools, the voice model can keep the conversation going briefly, but it can't mask arbitrarily long response latency. So we put the entire delegation loop—routing, prompt processing, inference, and tool calls—under a response-time budget.

The first optimization is to prepare the frontier model and all its required tools before a delegation is requested. When a voice session starts, the application server creates an inference session for the frontier model and prefills it with the initial conversation context, making sure the prompt is fully processed before the first delegation request is sent.

We then keep that inference session available for the entire voice conversation and use stable session affinity for consecutive requests. Combined with prompt caching, these techniques improve latency and make it easy to recover from worker failures.

Inference strength, output limits, tool architecture, and round trips between model and tools also affect how quickly the conversation gets a useful result, so we tuned these to speed up responses. By minimizing what the delegation path has to do, we let the voice model quickly incorporate what the frontier model returns.

Even though the voice model works with a continuous audio stream, many surrounding systems still operate in user and assistant turns, including ChatGPT's conversation UI and parts of the analytics and safety infrastructure. So the application server sifts through overlapping and sometimes ambiguous conversations and splits them into discrete messages.

When audio arrives, the server uses partial transcripts and timing signals to infer who is speaking and builds a message queue. The latest message is still tentative: as more speech arrives, its text, timing, and speaker attribution can change. Once a speaker has been talking long enough for the attribution to be reliable, the server finalizes the message.

Simultaneous speech makes this more complicated. If the user is talking and the assistant briefly interjects with something like “mm-hmm” or “okay,” that doesn't need to become its own message. But substantive assistant insertions usually should. Similarly, even if the user starts speaking mid-response, we prioritize keeping the displayed assistant reply coherent.

Any segmentation strategy has to balance timeliness against certainty. Finalizing too early produces fragmented history and unstable ordering; waiting too long delays transcripts and the features that depend on them. So the system keeps two linked views of the conversation: a speculative view that reflects the current state, and an authoritative version that captures what was actually said. The conversation view in the app can handle updates, so it uses the speculative view. But when writing logs to analytics pipelines, the final transcript is required.

That way, the rest of ChatGPT can have a stable picture of the conversation without forcing the realtime voice path to be constrained by turn-taking mechanics.

Fast response starts the moment a user hits the button. With GPT-Live, the system has to establish a media path and start sending audio into the model before the conversation can begin, putting every part of the startup flow on the critical path.

As noted above, WebRTC provides a strong realtime foundation, but starting a standard WebRTC session requires a surprising number of protocol handshakes and network round trips. WebRTC predates the “minimize round trips” ethos that shaped protocols like QUIC, so its underlying protocols sometimes end up doing the same work twice when combined. For example, each protocol has its own anti-DoS mechanism, even when that protection isn't needed in the full WebRTC stack.

We analyzed the protocol stack and developed a streamlined round-trip protocol for WebRTC called WARP, reducing the startup process for media and data from six network round trips to one. WARP does this through a set of backward-compatible protocol improvements: piggybacking the DTLS handshake on ICE (SPED), using the faster DTLS 1.3 handshake, pre-negotiating the SCTP handshake (SNAP), and pre-negotiating data channels instead of using DCEP.

We designed WARP with WebRTC community collaborators as a set of open specifications, so the broader ecosystem can benefit too. The proposals are moving through the IETF's TSVWG working group; libwebrtc and Pion have both added WARP support, and other WebRTC implementations are following.

After optimizing the media handshake, one latency stood out: the signaling exchange used to share SDP parameters before WebRTC establishes a connection. To move this exchange off the critical path, we built a mechanism called Instant Connect. It negotiates these parameters in advance, without reserving server capacity or changing existing WebRTC implementations.

Instant Connect runs in parallel with the standard signaling flow. If the pre-negotiated parameters are valid, the server can create the session when it receives the first media packet. If the parameters are expired or invalid, signaling is already underway, so the client can fall back without adding latency.

Together, Instant Connect and WARP dramatically shorten the time from user intent to realtime media flowing. With the SDP exchange off the critical path and WARP's streamlined transport handshake, the client now needs to send just a single UDP packet to start a session. The server can respond immediately, letting the rest of the system get to work on what the user actually cares about: listening and responding.

A system can look fast on paper and still stumble under real voice traffic. Before letting GPT-Live talk to users, we ran a silent test, routing a small percentage of production ChatGPT voice sessions to both the existing Advanced Voice Mode experience and the new system, then gradually increased the percentage. Advanced Voice Mode continued to serve users normally, while the shadow path ran inference in read-only mode. This let the system face real clients, networks, session durations, and geographic distributions without changing what users heard.

One of the first lessons was that GPU throughput alone can't measure capacity. Voice sessions stay connected and keep sending frames, so the CPU-side stream handlers, queues, and network paths have to scale along with inference capacity. Under real load, a supporting component saturated earlier than load testing predicted, causing inference requests to back up and latency to compound. We reframed the capacity question from “how many requests can a GPU handle?” to “how many concurrent sessions can the system sustain while ensuring every frame is processed on time?”

The test also made geography a top concern. Routing sessions to capacity resources in far-away regions can add latency at several points in startup and streaming. We began validating model releases against regional capacity and traffic-routing configurations, and broke down latency by traffic origin. Moving inference closer to users helped, but it also reinforced a broader lesson: end-to-end responsiveness depends on every service along the path, not just the model server.

There were also failures that only appear over real session lifecycles. Long-running sessions exposed memory and persistence pressure. Reconnects stressed compaction and state-recovery mechanisms. Ordinary client disconnects exposed race conditions in the closing handshake. These problems rarely showed up in short load tests, because they depend on time, accumulated state, and behavior across service boundaries.

Finally, production testing pushed us to improve observability and release controls. We found that some metrics conflated different sources of latency, some dashboard rollups hid individual bad engines, and there were configuration drifts between the test system and the deployed system. So we added finer-grained telemetry, validation based on verified configurations, staged rollouts, and the ability to quickly isolate or disable a single path. The silent test became a dress rehearsal for launch: not just how much traffic the system could carry, but how quickly we could find, contain, and recover from failures.

Scaling GPT-Live to ChatGPT's size required building an entirely new system around one core principle: keep voice flowing. Streaming inference continuously feeds audio into a full-duplex model. A dedicated media path ensures audio frames arrive reliably. Asynchronous delegation lets deeper thinking happen in parallel. Optimized transport keeps the experience fast from start to finish.

The architecture behind GPT-Live is already evolving into a broader realtime interaction platform. As ChatGPT voice expands from conversation to agent collaboration, this architecture supports it, and it will also be the foundation for the upcoming GPT-Live API. Over time, it will bring voice experiences to more devices, applications, and modalities, while preserving the immediacy that makes voice conversations feel as natural as talking in real time.

If you'd like to work on engineering problems like these, we'd love to have you join us.