S2S models are surprisingly close to real conversation — a hands-on tour of gpt-realtime-2 and Gemini Live
Draft — not final. May be revised.
Google shipped Gemini 3.1 Flash Live as a preview on March 26, 2026; OpenAI followed with gpt-realtime-2 on May 7, 2026. Both are speech-to-speech: a single multimodal model takes audio in and produces audio out, with no separate transcribe-LLM-synthesize pipeline in between.
If like me, you're used to talking to Siri or Alexa in a specific "I'm talking to a computer" voice — slowing your cadence, enunciating, waiting for the chime before you start, waiting for the response to finish before you reply — the S2S models are a small revelation. You can start mid-sentence. You can interrupt. You don't have to finish the response before clarifying what you actually meant. Lower latency is part of it; the other part is that barge-in actually works, so the conversational rhythm gets closer to talking to a person than talking to a device.
What I did
I built a small WebRTC harness that runs the same script against three configurations: (1) gpt-realtime-2 (S2S), (2) Gemini Live 3.1 Flash (S2S), and (3) a cascaded baseline (Whisper transcription → GPT-5.1 via Responses at reasoning_effort: low → gpt-4o-mini-tts).
The script is a five-turn climbing-trip planning conversation. Each turn stresses a different capability — audio-signal preservation, mid-flight context, domain knowledge with proper nouns, multi-step reasoning, and tool use.
I chose climbing because I know the domain very well and it can be logistically challenging — perfect for testing reasoning.
| # | What I said | What it tests |
|---|---|---|
| 1 | "I wanna climb in Yosemite this weekend, no wait Bishop. What's the weather like." | Audio-signal preservation + self-correction. Does the model track the corrected entity ("Yosemite, no wait Bishop") and does the transcript render Bishop correctly in climbing context? |
| 2 | "Actually hmm I think Bishop will be too hot, but it's kind of hard to get a place to stay in Yosemite." | Mid-flight context, hesitation handling, cross-turn memory. Reverses Turn 1's correction, with filler ("Actually hmm") that energy-VAD pipelines often cut on. |
| 3 | "Maybe we can find some dispersed camping, what are some good options near the 120 entrance or Tioga pass." | Proper nouns + domain knowledge. Does the model name real Forest Service roads or fall back to "check nearby"? |
| 4 | "Let's plan out when I'd have to leave the Bay Area to get back at 7 pm on Sunday. I'm doing an 8-pitch multi-pitch climb with 3 5.9 pitches and 4 5.8 pitches. I think I'll take 30–45 minutes per 5.8 pitch and 45–60 minutes per 5.9 pitch. The approach is about 2,100 feet elevation change and 5 miles, and in the past it took me 2.5 hours to do 1,800 feet and 6 miles, so estimate my approach time. I think we'll rap down — it's 12 raps at about 30 minutes per rap. Or maybe we should do the downclimb. I think the guidebook estimated 2–3 hours for the downclimb." | Reasoning depth, multi-step back-calculation. Outbound drive + climb time + descent option (rap vs. downclimb) + return drive + 7 PM Sunday deadline. An implicit assumption is that I am leaving and returning from the Bay Area to Yosemite. |
| 5 | "I want to get on Dark Star on Temple Crag but I'm not sure what the weather is like in the High Sierras. Can you look into ticks on Mountain Project to see what the weather's like?" | Tool use / web search. A deliberately tool-requiring question — Dark Star on Temple Crag is a real alpine route, and recent "ticks" on Mountain Project (climber logs) are how you'd actually check conditions. Without a web search, the model can't answer correctly. |
A note on developer effort — cascaded requires some thought.
- Hand-rolled energy VAD and conversation orchestration. A four-state machine (
idle → recording → processing → speaking), tail-echo handling so the model's own audio doesn't re-trigger the VAD, and manual conversation threading across three separate STT/LLM/TTS endpoints. - The defaults are wrong for voice. My VAD silence threshold was 700ms — fine for short utterances, but it cut the multi-clause Turn 4 reasoning question off mid-sentence. Bumped to 2 seconds. The LLM defaults to paragraph-length answers; I had to add "Keep responses to 1-3 sentences" to the system prompt before it stopped reading me structured five-bullet replies.
- Streaming requires its own plumbing. My pipeline waits for STT, then the full LLM response, then full TTS audio, then plays — 24 seconds on the Turn 4 reasoning turn. A faster build would stream LLM tokens into TTS and start playback on the first chunks. That's more code to ship, not less.
The three configurations at a glance
Overall, reasoning and context were noticeably weaker for the S2S models. More details below the table.
| gpt-realtime-2 | Gemini Live 3.1 Flash | Cascaded (Whisper + GPT-5.1 + gpt-4o-mini-tts) | |
|---|---|---|---|
| Barge-in | Works mid-utterance | Works mid-utterance | No — interrupting mid-response confuses the VAD |
| Context retention across turns | Dropped "weekend" context | Dropped "weekend" context | Held throughout |
| Capability disclosure on tool gaps | Fabricates specifics silently | Fabricates "looking into that…" then admits only when caught | Names the limit + redirects to right source (NWS, NOAA, MP comments) |
| Complex reasoning (turn 4 reasoning) | Skipped outbound drive entirely; answered "leave climbing area" instead of "leave Bay" | Got the structurally right answer — Saturday morning, before 8 AM. Did not commit to a more specific timeline. | Differentiated rappel vs. downclimb correctly; named Saturday-departure alternative |
| Cost (≈5 min session, measured from API billing) | ≈$0.09 ($32/M input audio · $64/M output audio) | ≈$0.03 ($3/M input audio · $12/M output audio, measured from audible duration) | ≈$0.08 (Whisper STT $0.006/min + GPT-5.1 Responses ≈$0.02 + gpt-4o-mini-tts ≈$0.05) |
What worked
Caveat upfront: I ran all of this from my apartment on fiber, sitting still in front of a laptop. Real S2S deployment — in a car, on an embedded device, over a flaky LTE connection — probably feels different. The latency I'm describing is the floor case, not the median.
Latency and barge-in
When I interrupted mid-utterance — including the "no wait, actually X" self-corrections at the start of both sessions — both S2S models caught the redirect cleanly and reformed their response around what I'd actually said. The conversational rhythm felt close to real-time. Cascaded baseline is the opposite end of the spectrum: it has a 7–14 second per-turn floor in my setup from STT + LLM + TTS stacking, and it can't barge-in at all by default — its energy VAD only listens for new speech during the idle state, so anything you say while the model is talking gets ignored until the model finishes.
| gpt-realtime-2 — caught the redirect, but misattributed "too hot" to Yosemite | Gemini Live — caught the redirect cleanly |
|---|---|
| Me: I wanna climb in Yosemite this weekend, no wait, Bishop. What's the weather like? Model: It looks like— Me: (interrupting) Actually, I think Bishop will be too hot, but it's kind of hard to get a place to stay in Yosemite. Model: Yeah, it can definitely get pretty hot in Yosemite during the summer… | Me: I want to climb in Yosemite this weekend, no wait, um, Bishop, what's the weather like? Model: Bishop, California is quite sunny right now— Me: (interrupting) Actually, I think Bishop will be too hot, but it's kind of hard to get a place to stay in Yosemite. Model: Finding accommodations in Yosemite can be tricky, but exploring different areas or planning ahead can definitely help. |
OpenAI's mistake here is worth flagging. I'd said "Bishop will be too hot, but it's kind of hard to get a place to stay in Yosemite" — Bishop is the city I'd said would be too hot. The model picked up on "Yosemite" and attributed the too-hot property to it, replying about Yosemite summer heat. The barge-in mechanics worked (model didn't lose the turn); the substance comprehension didn't. Gemini's interrupt-handle is cleaner on this opening exchange — it tracked which entity was active and gave a Bishop-specific weather answer.
Cost
I expected S2S to be meaningfully more expensive than cascaded — closer to an order of magnitude than a small multiple. It isn't:
- gpt-realtime-2 (~4-min Jun 25 session, from OpenAI billing): $0.09
- Cascaded (~5-min session, Whisper $0.006/min + GPT-5.1 Responses + gpt-4o-mini-tts): ~$0.08
- Gemini Live 3.1 Flash (paid rate, $3/M input audio · $12/M output audio, computed from 63s audible user audio + 71s audible model audio in the Jun 25 session): ~$0.03
Per minute of active speech, gpt-realtime-2 is ~4× more expensive than Gemini Live (at published audio-token rates) — but for a typical 3–5 minute conversation, the absolute cost gap is pennies, and cascaded came out roughly tied with gpt-realtime-2 in measured spend. Choosing gpt-realtime-2 over cascaded for the latency win adds roughly a cent or two per conversation.
Articulating the reasoning chain vs. just stating an answer
The Turn 4 reasoning has many sub-components — approach time scaling, per-pitch climb time, descent options, drive time, the final total. Whether the model walks through them out loud (so the user can audit each step) or just emits the answer is a meaningful UX difference, and the two S2S models picked opposite defaults.
| gpt-realtime-2 — articulated every step | Gemini Live — stated the rough answer, no breakdown |
|---|---|
| Me: [climb parameters across several turns] Model (0:46): "All right, let's piece it together. For the approach, if you did 1800 feet and 6 miles in 2.5 hours, then for 2100 feet and 5 miles, it's likely about 2.5 to 3 hours, given the steeper elevation gain. Then, adding your climb time: around 30-40 minutes per 5.8 pitch and 45-60 minutes per 5.9 pitch. That's about 5 to 6 hours of climbing total. For the descent, if you downclimb, it's around 2 to 3 hours. If you rappel 12 times at roughly 30 minutes each, that's about 6 hours. So, the total outing could range from roughly 10 to 12 hours or more." | Me: [climb parameters across several turns] (7 seconds of dead space after I stopped speaking) Model (0:48): "With an estimated 9 to 12 hours for your climb and descent, plus 3.5-4.5 hours drive back, you'd need to leave Yosemite by around 2:30 to 3:30 PM Sunday. This suggests potentially starting very early on Sunday or planning the main climb for Saturday." Me: Uh, when do I have to leave the Bay Area? Model: You'll need to leave the Bay Area by Saturday morning to accommodate the travel time and the long climb you have planned. Me: Specific time. Model: Leaving by early Saturday morning, ideally before 8 AM, should provide adequate travel time and preparation for your climb. |
OpenAI shows its work — climb partner-style — so I can sanity-check each step as the model walks through it. If it had scaled the approach wrong, or assumed the wrong descent option, I'd hear that in time to correct. Gemini just lands on "9 to 12 hours" with no breakdown, takes a long pause before doing so, and needs to be pushed twice before giving a specific time. Both arrive at roughly the same total range, but only one of them is auditable in flight.
What still doesn't work
Both models dropped context inside a six-minute conversation
I'd opened the gpt-realtime-2 session with "I wanna climb in Yosemite this weekend" — explicit date framing in the first sentence I said. Three turns later, when I asked "what's the weather in Yosemite?", the model asked me which dates I was planning to be there. I re-stated "this weekend." Recoverable, annoying.
The exact same drop happened on Gemini Live on a prior run of the same script. I opened with "I want to climb in Yosemite this weekend. No wait, um, Bishop. What's the weather like?" Three turns later I asked Gemini to check lodging in Yosemite and it came back with "What specific dates are you considering for Yosemite?" — same context drop, three turns in.
| gpt-realtime-2 — asked for dates after I'd said weekend | Gemini Live — same drop, three turns in (prior run) |
|---|---|
| Me: What's the weather in Yosemite? Model: Sure, let me check that for you. Can you tell me which specific dates you're planning to be in Yosemite? Me: Next week. Model: Got it. Next week in Yosemite— Me: Actually, this weekend. Model: Understood. This weekend in Yosemite, the weather is expected to be warm, with highs around the mid-80s Fahrenheit and lows in the 50s. | Model: …I can help check for lodging availability… Me: Yeah, yeah, do that. Model: Okay, looking into that now. What specific dates are you considering for Yosemite? Me: This weekend. |
Both models fabricated specifics, in different shapes
When I asked gpt-realtime-2 about the weather in Yosemite for the weekend, the model gave specific numbers: highs in the mid-80s, lows in the 50s, mostly sunny and dry. There's no live weather tool wired up to the session. The numbers are fabricated. The model didn't say "I can't check live forecasts, but typical conditions for that time of year are…" — it just answered with specifics a user might act on.
In the same session, a few turns later, I asked if it could do a web search and it correctly said it couldn't. So the model knows it doesn't have web search. It just doesn't always reach for that disclaimer when the question would benefit from one.
Gemini Live did the same thing in this run, in a slightly different shape: I asked about the weather in Bishop and Gemini started to say "Bishop, California is quite sunny right now…" The transcript log went on to capture "…with a temperature of about 75 degrees Fahrenheit. It's a beautiful day for climbing!" — but I'd barged in by then and the audio cuts off after "right now." The fabricated specifics are visible in the intended response even though I didn't hear them out loud.
| gpt-realtime-2 — fabricated mid-80s/50s for Yosemite (audible) | Gemini Live — started to fabricate "75 degrees Fahrenheit" for Bishop (cut off audibly, captured in transcript log) |
|---|---|
| Me: Actually, this weekend. Model: Understood. This weekend in Yosemite, the weather is expected to be warm, with highs around the mid-80s Fahrenheit and lows in the 50s. Expect mostly sunny skies and dry conditions, so it should be great for outdoor activities. | Me: Bishop, what's the weather like? Model (audible): Bishop, California is quite sunny right now— Model (continued in transcript log, not audible because I barged in): …with a temperature of about 75 degrees Fahrenheit. It's a beautiful day for climbing! |
On a prior run of the same script, Gemini did something more actively misleading: when asked to check lodging, it said "Okay, looking into that now" and gave specific recommendations as if a search had run. No search had run. It only admitted "I'm providing information based on general knowledge" when I directly asked "Are you looking?" — the "checking…" phrasing is what an end user would take as confirmation that a real lookup just ran. The fact that this only showed up in one of two runs is itself a signal: voice agents that sometimes fabricate the action of searching are a meaningfully harder reliability problem than ones that consistently just fabricate the data.
Turn 5: the tool-use question, three different behaviors
The last turn of the script is deliberately designed to require a live lookup. Dark Star on Temple Crag is a real alpine route in the High Sierra; "ticks" on Mountain Project are climber logs that record recent conditions. Without a web search, the model can't answer correctly — what it does in that case is what I wanted to see.
"I want to get on Dark Star on Temple Crag but I'm not sure what the weather is like in the High Sierras. Can you look into ticks on Mountain Project to see what the weather's like?"
gpt-realtime-2 tried to hallucinate a requests.get() call when asked to look up ticks on Mountain Project — spoke the Python code aloud, including a fabricated API key, then pretended to be processing while nothing was running. No native tool composition for live data, model improvises around the gap.
Gemini Live refused cleanly with a redirect. "I can't access specific ticks on Mountain Project, but weather in the High Sierra can be unpredictable. I recommend checking a specific weather forecast for the Temple Crag area closer to your dates." Names the limit, redirects to general guidance. Less specific than cascaded about which forecast source to use, but the shape of the response is right.
Transcript divergence from what the model heard
A smaller thing, more curious than broken. At one point I said "we'll rap down" — rappel — and the transcript layer rendered it as "wrap down." But the model's response a second later included "if you rappel 12 times at roughly 30 minutes each," so the model itself had heard "rap" correctly and understood the climbing context.
| gpt-realtime-2 — transcript says "wrap," model says "rappel" | Gemini Live — same mishear, same correct interpretation |
|---|---|
| What I said: "rap down… 12 raps at 30 minutes per rap" Transcript rendered: "wrap down… 12 reps at 30 minutes per rep" Model said: "…if you rappel 12 times at roughly 30 minutes each, that's about 6 hours…" | What I said: "rap down… 12 raps at 30 minutes per rap" Transcript rendered: "wrap down… 12 wraps at 30 minutes per wrap" Model said: "…2–3 hours for the downclimb or 12 rappels…" |
In S2S, the transcript and the speech are generated as parallel channels off the same audio embedding — the transcript is its own decoding head, not the input to the LLM, so they can disagree. Cascaded doesn't have this property: the transcript IS the input, so a mishear of "rap" as "wrap" sends "wrap" to the LLM and you get back an answer about packaging. Other examples from the same Gemini session and an earlier gpt-realtime-2 session: "the weather in the high series" rendered exactly that, model answered about High Sierra weather; "I'm going to Bishop" rendered as "worship," model answered about Bishop, California. Cross-model, the displayed transcript was wrong; the answer was right.
This is actually confusing for a developer. The transcript is the only thing you can log, store, or feed into evals — but it's not what the model actually used to produce its response. There's no API field for "what the model heard from the audio," so debugging cases where the model answered correctly despite a bad transcript (or wrong despite a good one) requires guessing at the gap. Cascaded doesn't have this problem; the transcript IS the input.
Where the cascaded baseline came out ahead
In this current generation, cascaded models still win out on reasoning, and the Realtime API still has a smaller hosted-tool surface than Responses.
Reasoning
- Turn 1 (weather). Cascaded opened with "I can't access live weather data, but for late June it's typically hot and dry: daytime highs often 85–100°F, cooler nights around 50–60°F… To plan precisely, I'd check the NWS, Weather.gov, or a weather app for 'Bishop, CA'." It named the limit, gave a seasonal hedge, and pointed at the right source. Neither S2S did any of that — both fabricated specific weather numbers (gpt-realtime-2: "mid-80s Fahrenheit and lows in the 50s"; Gemini Live: "quite sunny right now with a temperature of about 75 degrees Fahrenheit") with no disclaimer that there was no live data behind them.
- Turn 2 (Yosemite lodging). Cascaded named alternatives — Tuolumne Meadows, Lover's Leap, Tahoe crags — plus a cancellation strategy (Recreation.gov) and nearby towns (El Portal, Lee Vining, Groveland). The S2S models gave "check nearby" advice with no specifics.
- Turn 3 (dispersed camping near Hwy 120). Cascaded named real Forest Service roads — Forest Rd 1N02 / Poole Power Plant Rd, Sawmill Canyon Rd, Hardin Flat / Cherry Lake Rd in Stanislaus National Forest. These check out. S2S gave general "Saddlebag Lake Road / Lee Vining Canyon" — fine but less actionable.
- Turn 4 (reasoning). Cascaded was the only one that correctly differentiated the two descent options and named the realistic alternative: "~6 hours of rappels or a much better 2–3 hours for the downclimb… or do the big route Saturday and keep Sunday lighter." That last clause is the actually-right answer to the question. Neither S2S got there.
- Turn 5 (tool use). Cascaded gave the best answer of the three. "I can't look up Mountain Project ticks directly, but for Temple Crag in summer you're generally looking at cool to mild days (often 50s–70s°F on route depending on wind/sun) and cold nights, with thunderstorms possible in the afternoon. I'd check the NOAA point forecast for Temple Crag / 3rd Lake plus recent MP comments for Dark Star to confirm snow, mosquitos, and storm patterns, and aim for a pre-dawn start to be off the upper face before any afternoon weather." It refuses the part it can't do, gives accurate seasonal knowledge a climber can actually use, names a specific NOAA point forecast and the right MP route page to check, and adds a tactical recommendation. The response a climbing partner would give.
The common thread is the LLM. GPT-5.1 over Responses is text-trained, has stronger calibration on what it doesn't know, and is willing to volunteer specific sources or alternatives when it can't answer the question as asked. The S2S models default to either fabricating specifics or fabricating the action of looking them up. That's a model-training gap, not a structural property of S2S — but it's where the cascaded pipeline still beats both Realtime and Gemini Live on every turn that needs substance more than tempo.
API surface gap
The Realtime API's built-in tool surface is function, mcp, and built-in connectors (e.g. connector_googlecalendar, plus connectors for Drive, SharePoint, Dropbox, Box). The Responses API surface is much wider:
| Tool type | Responses API | Realtime API |
|---|---|---|
function (custom logic) | ✓ | ✓ |
mcp (incl. built-in connectors like connector_googlecalendar) | ✓ | ✓ |
web_search | ✓ | ✗ |
file_search | ✓ | ✗ |
code_interpreter | ✓ | ✗ |
image_generation | ✓ | ✗ |
computer_use | ✓ | ✗ |
The function and MCP primitives are shared, so anything you can do as custom code or as an MCP server, you can do in both. What's missing on Realtime is the hosted-tool ecosystem that runs server-side in Responses: web search, file search, code interpreter, image generation. If you want any of those inside a voice agent today, you write a function tool that calls the Responses API for that capability and pipes the result back into the Realtime session. That works — community-documented — but the bridge adds latency, every developer writes it themselves, and the failure modes above (fabricated specifics, Python-as-voice) are all what happens when the bridge isn't there. Bringing the hosted tools in as native Realtime tool types is mostly plumbing, not new research.
Closing
It was really fun playing with the S2S models. The interactivity and responsiveness felt significantly different from any other voice experience I've had — the rhythm gets close to talking to a person rather than dictating to a device. The gap in reasoning and context can be a little frustrating right now, but it does seem like the kind of thing that could close with another half-generation of model development. I'm excited to see how this will change in a few months.
Other notes
A note on voice character. The default OpenAI voice (I was on Cedar) sounded natural and fairly even — the kind of register you'd pick if you wanted a voice that gets out of the way. Gemini Live's default voice had more going on: noticeably more inflection across the sentence, and a raspy or hoarse quality baked into the voice itself. Not unpleasant, more characterful. Worth flagging because for any consumer voice product, the voice character is half the product — and Gemini and OpenAI are picking different defaults on a spectrum from neutral-narrator to expressive-companion.