Purpose

In this document, we examine what full duplex is, what half duplex is, and common features associated with full duplex. For each feature, we analyze whether it is actually related to full duplex and how it can be imitated by a half-duplex system.

What is Full Duplex

IEEE 802.3 defines full duplex for networking as:

  • Half duplex: DTEs contend for access to a shared medium. Multiple, simultaneous transmissions in a half duplex mode CSMA/CD LAN result in interference, requiring resolution by the CSMA/CD access control protocol.
  • Full duplex: allows for simultaneous communication between a pair of stations, provided that the Physical Layer is capable of supporting simultaneous transmission and reception without interference.

For speech, we use the definitions of:

  • Half duplex speech: the system alternates between listening and speaking. An end-of-turn or "over" event arbitrates the conversational floor before the model speaks.
    user talks[s]system talks[s]user talks[s]system talks[s]

    [s] = signaling for end-of-turn (push-to-talk, "over" on walkie talkie, VAD, ...)

  • Full duplex speech: the model can keep receiving live user audio while transmitting model speech. Listening and speaking are simultaneous paths in the normal operating state.
    user talking & listening .........................................................................system talking & listening .......................................................................

Examples

  • Half Duplex (ASR-LLM-TTS pipeline)
    User Talks[s]User Talks[s]User Talks[s]User Talks[s]Streaming ASR[s]Streaming ASR[s]Streaming ASR[s]Streaming ASR[s]LLM PrefillLLM Gen[s][s]LLM PrefillLLM Gen[s][s]LLM PrefillLLM Gen[s][s]LLM PrefillLLM Gen[s][s]Streaming TTS[s]Streaming TTS[s]Streaming TTS[s]Streaming TTS[s]System Talks[s]System Talks[s]System Talks[s]System Talks[s]
    • Streaming ASR generally faster than real time. It can start slightly after user starts speaking and end slightly after end-of-turn.
    • LLM Prefill can start at the first ASR token, but probably cheaper to do all of prefill at once.
    • LLM Gen can start as soon as prefill is done.
    • Streaming TTS can start at first LLM token, though often better results if done sentence at a time.
    • Often (but not always) there is some type of [s]-ignaling from one stage to the next of: 'this turn is done'
    • In our experiments: time-to-first-audio token is typically ~500ms for ASR-LLM-TTS alone, and often an additional ~500ms for the detection of '[s]'. Our typical breakdown is:
      • ~150ms ASR
      • ~200ms LLM
      • ~150ms TTS
      In theory, streaming helps, but in practice:
      • TTS: They 'why' in '*Why* is the sky blue?' sounds different from '*Why* did you egg my house?'. Therefore, sentence at a time is often better.
      • LLM: Running prefill per-token is memory-wasteful. Better to do all of prefill at once.
      • ASR: '*Where* is X' and '*Wear*' what tomorrow' sounds similar on first word, but future words disambiguate.
  • Full Duplex (Kyutai/Moshi)
    input externalB_mimi[t]B_mimi[t+1]B_mimi[t+2]B_mimi[t+3]input auto regressiveA_text[t]A_mimi[t]A_text[t+1]A_mimi[t+1]A_text[t+2]A_mimi[t+2]A_text[t+3]A_mimi[t+3]computeMoshi stepjoint stream embeddingHelium temporal32-layer 7B LMDepFormer audio6-layer local codebooksMoshi stepjoint stream embeddingHelium temporal32-layer 7B LMDepFormer audio6-layer local codebooksMoshi stepjoint stream embeddingHelium temporal32-layer 7B LMDepFormer audio6-layer local codebooksMoshi stepjoint stream embeddingHelium temporal32-layer 7B LMDepFormer audio6-layer local codebooksoutputA_text[t+1]A_mimi[t+1]A_text[t+2]A_mimi[t+2]A_text[t+3]A_mimi[t+3]A_text[t+4]A_mimi[t+4]time stept+1t+2t+3t+4

    Moshi achieves a theoretical 160ms (often 200ms practical) TTFAT. We have an in-progress blog post on Kyutai/Moshi.

Overview

Below, we split common speech system features into: half/full-duplex related and those independent of half/full-duplex. For duplex related features, we categorize on: (a) who initiates the overlap and (b) who has the floor after the overlap.

Half-duplex vs full-duplex features

Candidate featureWhere it belongs
Generating system backchannel
System says "mm-hm", "yeah", or "right" while the user continues.Generating system backchannel
System gives short floor-holding filler while the user still owns the floor.Generating system backchannel
Generating system barge-in
System interrupts the user for safety, privacy, or urgent correction.Generating system barge-in
System asks for early clarification before the user finishes.Generating system barge-in
Handling human backchannel
User says "mm-hm", "yeah", or laughs while the system continues.Handling human backchannel
Handling human barge-in
User says "stop", "no", "wait", or a correction during system speech.Handling human barge-in
User asks a follow-up during system speech.Handling human barge-in
User changes topic during system speech.Handling human barge-in
User requests repetition during system speech.Handling human barge-in
User expresses dissatisfaction during system speech.Handling human barge-in
Stop latency after user interruption.Handling human barge-in
Shared plumbing: overlap classification
System keeps listening while its own TTS is playing.Overlap classification
Echo or self-speech cancellation.Overlap classification
Overlap detection.Overlap classification
Overlap classification into backchannel, barge-in, side speech, echo, or noise.Overlap classification
Speech separation when user and system audio overlap.Overlap classification
Smooth floor transfer after overlap.Overlap classification

Features independent of half-duplex vs full-duplex

Future sectionExamples
Latency plumbingresponse latency, incremental ASR, LLM prefill, streaming TTS, speculative rollback
Endpointing and repairpause handling, false-start recovery
Speaker selectionmulti-party floor selection, ambient speech rejection, addressed-speaker detection
Prosody and affectaffect timing, prosody-aware timing, style adaptation
Candidate featureWhere it belongs
Issues independent of half-duplex vs full-duplex
Response latency after overlap.Latency plumbing
Incremental ASR.Latency plumbing
LLM prefill while the user is still speaking.Latency plumbing
Streaming TTS.Latency plumbing
Speculative response rollback.Latency / planning
Pause handling.Endpointing
False-start recovery.Intent repair
Multi-party floor selection.Speaker selection
Ambient speech rejection.Target-speaker robustness
Affect or prosody timing.Independent unless it changes floor control

Generating system backchannel

    1. Example Interactions

      • short system continuers: mm-hm, okay, right, I see
      • floor-holding fillers while tools, ASR, LLM, or TTS catch up
      • prosody-timed acknowledgements shaped by uncertainty or emphasis

    2. Canonical full duplex example

      Full-duplex systems support system-backchanneling, where the system can affirm the user.

      userinteractionsystemI need to order:10 mac-mini's: 8 w/ 16 GB, 2 w/ 32 GB5 mac-studio's; 4 w/ 64 GB, 1 w/ 512 GBand 3 64GB mbp'scontinuercontinuercontinuercontinuermm-hmokaymm-hmright

    3. How it breaks down in half duplex / attempted fixes

      Half-duplex systems, by default, lack backchanneling. Resulting in ignored users:

      user thought:HELLO? ANYONE HOME? R U THERE?
      user speech:I need to order:10 mac-mini's: 8 w/ 16 GB, 2 w/ 32 GB5 mac-studio's;
      interaction:
      system:

      One can try VAD systems that auto affirm, but they can have failure cases like:

      user speech:system speech:I lost a penny somewhere in the house.I'm going to tear the house apart until I find the penny.Starting with the floor tiles now.mm-hmokayright

    4. To correctly system-backchannel, the system needs to either agree or disagree with the user. With a moshi-style full-duplex system, this can be done in 200ms. With a half-duplex system, this delay can be 500ms-1s.

Handling human backchannel

    1. Example Interactions

      • short user acknowledgements: yeah, uh-huh, right, laughter
      • classifying listener feedback as continue, not cancel
      • distinguishing backchannel from stop, correction, or topic switch

    2. Canonical full duplex example

      userinteractionsystemCan you repeat my order back?mm-hmmm-hmrightfeedbackfeedbackfeedback10 mac-mini's: 8 w/ 16 GB, 2 w/ 32 GB5 mac-studio's; 4 w/ 64 GB, 1 w/ 512 GBand 3 64GB mbp's

    3. How it breaks down in half duplex / attempted fixes

      A naive half-duplex system might lack support for human-backchanneling entirely, generating long blobs of text:

      userinteractionsystemCan you repeat my order back?10 mac-mini's: 8 w/ 16 GB, 2 w/ 32 GB; 5 mac-studio's; 4 w/ 64 GB, 1 w/ 512 GB; and 3 64GB mbp's

      or incorrectly try to parse the backchannel as interruption or a new query:

      userinteractionsystemCan you repeat my order back?mm-hm10 mac-mini's: 8 w/ 16 GB, 2 w/ 32 GB; 5 mac-studio'sprocessing query `mm-hm`

    4. Correctly handling human back channeling is easier -- we need a small TTS to detect positive/negative backchannels.

Generating system barge-in

  1. 04

    System barge-in

    1. Example Interactions

      • urgent safety corrections before the user finishes
      • early clarification before a long wrong action proceeds
      • intentional floor grabs when waiting for end-of-turn is harmful

    2. Canonical full duplex example

      userinteractionsystemIt says here my private key is 0123-ABCD-safety barge-inStop. Never share that.

    3. How it breaks down in half duplex

      Naive cascade waits for a final endpoint, so the secret can be exposed before the system objects.

      userinteractionsystemIt says here my private key is 0123-4578-ABCD-[continues reading full key]policy check after endpointThat was unsafe. Do not read private keys aloud.

    4. Correctly generating system barge-in can be expensive. We need to, at every human input token, decide: do we barge-in.

Handling human barge-in

    1. Example Interactions

      • stop, cancel, no, wait, and correction commands during system speech
      • follow-up questions asked while the system is still talking
      • topic switches, repetition requests, and dissatisfaction mid-output

    2. Canonical full duplex example

      userinteractionsystemMake all unit tests pass. Use parallel agents. Do whatever it takes.NO. DON'T DELETE ANYTHING.barge-in -> haltOkay. Firing off parallel agents to locate and delete failed unit --

    3. How it breaks down in half duplex / attempted fixes

      Naive playback can mask or discard the user's correction until the system finishes speaking.

      userinteractionsystemMake all unit tests pass. Use parallel agents. Do whatever it takes.NO. DON'T DELETE ANYTHING.ignored until playback endsOkay. I'll delete the unit tests. Running deletion. Done. Cleaning ~/ too. Save you some space

    4. A barge-in controller detects user speech over playback, stops TTS, captures the correction, and regenerates.

Conclusion

In conclusion, half-duplex can imitate some aspects of full-duplex, but at the cost of the additional complexity of external control loops. In our next blog post, we will examine how the Moshi architecture handles this within a single unified transformer.

preview of next post: How Moshi does Full Duplex

Helium -> Moshi: what is shared, replaced, and newCopied Helium weightsChanged adaptersNew Moshi weightsSeparate Mimi codec1. Regular Heliumtext token idx[t]text embeddingE_text[x[t]] -> Float[S,D]params ~122.9MHelium transformer stackshared core typeFloat[S,D] -> Float[S,D]params ~1.93Bnorm + LM headtext logitsparams ~122.9Mnext text tokenx[t+1]2. Moshi Helium TemporalA_textbot textA_mimibot audioB_mimiuser audiojoint stream embedding adapterE_text + E_A_mimi + E_B_mimireplaces text-only input adapterparams ~290.7Msumone Float[1,D] per frameTemporal TransformerHelium transformer weights copiedthen trained in Moshiparams ~1.93Btemporal hidden statez_s = Float[1,D]3. DepFormer Audioz_s fanoutFloat[1,2560]same contextA_text_sseeds q=0linear_in[0]z_sprev[0]A_text_slinear_in[1]z_sprev[1]A_mimi[0]linear_in[2]z_sprev[2]A_mimi[1]......linear_in[15]z_sprev[15]A_mimi[14]Depth Transformer6 layersD_dep=1024q=0,1,2,...,15no temporal cacheout[0]mimi0out[1]mimi1out[2]mimi2out[15]mimi15A_mimi_s[0..15]sampled audio tokensMimi decodertokens -> waveformspeaker audio24 kHz samplestoken_ids: Int[S]xs: Float[S,D]h: Float[S,D]logits: Float[S,V_text]Int[1]Q x Int[1]Q x Int[1]embs: Float[1,D]xs: Float[1,D]transformer_out: Float[1,D]pcm: Float[C,T]shared initializationweight tensors: transformer.*text emb copiedweight tensor: text_embz_s to audio stackz_s: Float[1,D]
AuthorTongKe Xue