Helping small devices speak mixed-language sentences smoothly: plan ahead from the full sentence text, keep active language adapters ready in RAM, and blend switches gently without a heavy neural vocoder.
~4.5M-parameter vocoder-free source-filter backbone + tiny ~0.3–0.6 MB language LoRA experts.
Deterministic sentence lookahead pins needed adapters before frame 0 and preloads upcoming ones early.
Single-word loanword debouncing avoids rapid ping-pong; 20–40 ms cross-fade smooths real switches.
Bilingual speakers switch languages mid-sentence—both for multi-word phrases and single borrowed words.
Switching language modules abruptly resets pitch (F₀), speaking energy, and vocal tract timbre—making one speaker sound like two different people.
Fetching a missing language adapter at the exact moment token B₁ arrives stalls the synthesis thread and causes audible pauses or buffer underruns.
Keeping a large multilingual model plus a heavy neural vocoder (e.g., HiFi-GAN) in RAM requires 100–300+ MB—far above tight embedded or mobile limits.
Existing systems either overload device memory or stall and jump when swapping language parts reactively.
| Architecture Family | Active RAM Footprint | Switch-Boundary Latency (p99) | Cross-Lingual Speaker Consistency | 1-Word Loanword & Boundary Behavior |
|---|---|---|---|---|
| 1. Monolithic Multilingual TTS + Neural Vocoder Single big network + HiFi-GAN |
High (120–300+ MB) Too heavy for strict edge budgets |
Low (< 5 ms) All weights stay in memory |
Moderate–High Can still drift on language ID flip |
Over-switches accent No explicit loanword debounce |
| 2. Reactive On-Demand Adapter Swapping Loads adapter when token arrives |
Low (~15–35 MB) Only 1 active adapter kept |
High Spikes (25–120 ms) Blocks on cold flash/cache miss |
Drops at Cold Swap Hard swap resets prosody state |
Ping-Pongs on 1 Word Harsh step jump at boundaries |
| 3. Ours: Lookahead Cache-Resident Adapter-MoE Vocoder-Free Source-Filter + LoRA-MoE |
Ultra-Low (~8–21 MB) ~4.5M core + tiny ~0.4 MB LoRAs |
Near-Zero (< 5 ms p99) Pinned L1/L2 + preload + fallback |
High (Steady Voice) Always-on Shared Bridge anchor |
Smooth & Debounced 1-word filter + 20–40 ms blend |
Prior adapter-based TTS treats language changes as unexpected runtime interrupts, and still relies on a heavy neural vocoder for waveform generation.
By pairing pre-synthesis sentence lookahead with a vocoder-free source-filter LoRA-MoE, we eliminate both the vocoder RAM bottleneck and mid-sentence swap stalls.
Side-by-side runtime comparison of how each architecture handles a mixed-language sentence at the switch boundary.
Vocoder-free source-filter + tiny ~0.4 MB LoRAs replace bulky neural vocoders.
Sentence lookahead pins L1/L2 at t = 0 and overlaps L3 flash I/O early.
1-word debouncer stops ping-pong; 20–40 ms blend smooths pitch & formants.
Always-on Shared Bridge (ΔWshared) prevents speaker drift across switches.
Unlike live speech recognition, sentence-level TTS sees every word and language change before making a single sound.
Text frontend runs in < 2 ms for a full sentence, revealing exact language spans and switch locations before synthesis starts.
>98% of code-switched sentences mix at most two languages for one speaker—requiring under 1.2 MB of pinned LoRA weights.
If a third language appears later in a long sentence, the cache controller preloads it asynchronously while the first clause is synthesized.
system-architecture-overview.svg and embedded above
Slide 06 / 15
Instead of a massive neural vocoder, our ~4.5M backbone predicts interpretable pitch (F₀) and filter envelopes directly.
| Module | Params | FP16 / INT8 |
|---|---|---|
| Core Acoustic Trunk (W₀) Encoder + Duration/Pitch + Decoder |
~4.1M | ~8.2 / 4.1 MB |
| Source-Filter Head Harmonic-noise excitation + filter |
~0.4M | ~0.8 / 0.4 MB |
| Always-On Shared Bridge Cross-lingual speaker anchor |
~0.25M | ~0.5 MB |
| Pinned Language LoRA (×2) Rank r = 8–16 per active language |
~0.2M each | ~0.4 MB each |
Because pitch (F₀) and spectral envelope are explicit physical signals, we can directly smooth and interpolate them across language switch boundaries!
Three parallel weight paths keep speaker timbre fixed while injecting language-specific pronunciation and tone.
Without a shared bridge, switching from English LoRA to Mandarin LoRA shifts vocal tract formants. ΔWshared carries cross-lingual prosody and speaker identity across the boundary.
Inside a pure language span, only 1 language expert runs (g1 = 1, g2 = 0). During a 20–40 ms switch window, both pinned experts blend smoothly (g1(t) + g2(t) = 1).
Sentence-scoped pinning locks active languages in RAM, overlaps flash I/O with early audio frames, and never blocks synthesis.
Unlike generic LRU caches, active adapters for the current sentence are pinned at t = 0 and cannot be evicted mid-utterance.
Loading a ~400 KB LoRA from flash takes ~2–8 ms; starting the fetch at t = 0 finishes hundreds of milliseconds before the switch boundary.
Even under extreme cold-start stress, synthesis never blocks on I/O—it falls back to ΔWshared to keep the speaker’s voice flowing.
Prevents rapid expert ping-pong on borrowed words and smooths pitch (F₀) and spectral transitions at real phrase switches.
Every branch is bounded in memory and latency—with zero blocking storage calls on the audio thread.
Each architectural pillar maps directly to a measurable acoustic or systems hypothesis.
| ID | Research Question | Testable Hypothesis (Falsifiable Target) | Key Ablation Experiment |
|---|---|---|---|
| RQ1 / H1 Backbone & Voice |
Can a ~4.5M vocoder-free source-filter backbone with low-rank language adapters preserve one speaker’s voice across languages? | H1: Shared-Bridge + Routed LoRA keeps cross-lingual speaker cosine similarity within ≤ 0.03 of a monolithic multilingual model while using < 20% active RAM (~8–21 MB). | With vs. without Always-On Shared Bridge (ΔWshared); LoRA rank r ∈ {4, 8, 16} sweep. |
| RQ2 / H2 Lookahead Cache |
Does sentence-lookahead pinning + async preload eliminate mid-sentence adapter loading stalls on edge hardware? | H2: Deterministic sentence pinning + async preload keeps p99 switch-boundary stall latency < 5 ms and yields 0 audio buffer underruns under cold-cache stress. | Reactive LRU on-demand loading vs. Sentence-Pinned + Async Preload + Bridge Fallback. |
| RQ3 / H3 Switch Control |
Do single-word loanword debouncing and 20–40 ms boundary blending reduce audible switch jumps? | H3: Reduces switch-boundary pitch step error (ΔF0) by ≥ 30% and improves code-switched listener preference (CMOS ≥ +0.25) over hard token-level switching. | Hard frame switch vs. 1-word Debounce only vs. Debounce + 20–40 ms Cosine Blend. |
Proves that an explicit source-filter backbone + shared low-rank bridge decouples speaker vocal identity from language routing without a heavy vocoder.
Exploits the deterministic lookahead property of TTS sentences to turn unpredictable MoE expert paging into a pre-scheduled cache pipeline.
Addresses both linguistic granularity (1-word loanwords vs. phrase switches) and physical signal continuity (F0 & envelope blending).
We evaluate both global utterance quality and local +/-100 ms switch-boundary windows on edge CPU targets.
| Evaluation Bucket | Metrics | What It Measures & How | Target Threshold |
|---|---|---|---|
| 1. Intelligibility & Naturalness | Mixed CER / WER (ASR), 5-scale MOS & side-by-side CMOS |
Pronunciation accuracy across both languages and listener preference on phrase & loanword sentences. | CMOS ≥ +0.25 vs. hard switch; no CER degradation |
| 2. Switch-Boundary Smoothness (±100 ms) | Boundary ΔF0 step (Hz), Energy step (dB), ECAPA Speaker Cosine Sim |
Local discontinuity across switch borders B1, B2 and voice consistency between L1 and L2 segments. | ≥ 30% lower ΔF₀ jump; Speaker sim drop ≤ 0.03 |
| 3. Edge Runtime & Tail Latency | CPU RTF, TTFA (ms), p99 Boundary Stall (ms), Adapter Cache Hit Rate (%) |
Single-core ARM/x86 synthesis speed and worst-case frame wait at language switches under cold/warm cache. | RTF < 0.25 on mobile CPU; p99 stall < 5 ms; Hit > 99% |
| 4. Hardware Resource Footprint | Peak Resident RAM (MB), Per-Language Adapter Size (MB) |
Total working memory (backbone + bridge + 2 pinned slots) and storage cost to add a new language. | ~8–21 MB Peak RAM; < 0.6 MB per language LoRA |
Evaluated on three stratified code-switched sets: (A) Multi-word intra-sentential phrase switches, (B) Single-word borrowed loanwords, and (C) Rapid 3-language stress utterances.
Utterance-wide averages hide brief clicks or pitch jumps at switch points. Measuring ΔF0 and energy inside a ±100 ms boundary window directly captures switch artifacts.
Structured with clear Go/No-Go verification gates so a focused workshop paper expands cleanly into a full thesis.
| Scope Dimension | Short Conference / Workshop Paper (ICASSP / Interspeech) | Full Thesis Proposal Scope |
|---|---|---|
| Language Pair & Data | 1 primary bilingual code-switched pair (e.g., Mandarin–English) + single-word loanword test set. | 3+ languages (e.g., ZH–EN–MS), multi-speaker adaptation, and rapid 3-language intra-sentential switches. |
| Core Technical Focus | Shared+Routed LoRA-MoE on ~4.5M source-filter core + 1-word debounce + 20–40 ms blend + lookahead cache trace. | Full on-device C++/ONNX ARM runtime, FP16/INT8 quantization, hardware flash-to-RAM DMA profiling. |
| Key Ablations | Shared Bridge on/off; Hard vs. Blended switch; Reactive vs. Lookahead cache hit/stall rate. | Full LoRA rank & blend-window sweeps, perceptual threshold of bridge fallback under synthetic flash throttling. |
A practical, low-memory architecture for natural code-switched speech on everyday edge devices.
Direct source-filter synthesis (F0 excitation → spectral filter H(z)) + always-on Shared Bridge + ~0.4 MB language LoRAs fits in ~8–21 MB RAM.
Pre-synthesis lookahead pins active L1, L2 adapters before frame 0, preloads L3 in the background, and uses ΔWshared on misses (p99 stall < 5 ms).
Skips unnecessary expert flips on single borrowed words and cosine cross-fades expert gates, pitch (F0), and spectral envelope across phrase switches.
Press N (or click Notes below) to toggle speaker notes on any slide · Press E to edit any text inline · Press Ctrl+S to save your edits as a standalone HTML file.