titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity

WIP: Multilingual Piper TTS: en=amy, ru=irina, es=claude (supersedes #24) #26

Closed
hermes-automation wants to merge 1 commits from hermes/tts-voice-multilingual into main
pull from: hermes/tts-voice-multilingual
merge into: titan:main
Conversation 0 Commits 1 Files Changed 7 +367 -24
hermes-automation commented 2026-08-20 18:57:23 +00:00
Member
Copy Link

Supersedes draft PR #24

Draft PR #24 (hermes/tts-voice-hfc-female, from task t_74b76353) implemented
en_US-hfc_female-medium as the single default voice with en_US-amy-medium
baked as an unused alternate. Brad changed the decision after that task
landed: the approved policy is now multilingual, not a single feminine
default. This PR starts fresh from origin/main (does not build on
hermes/tts-voice-hfc-female) and implements the new policy end to end.

Recommend closing #24 without merging — it does not reflect the current
decision. I have not closed or merged anything myself.

Approved voice mapping (Brad, 2026-08-20)

Language Voice Notes
English en_US-amy-medium
Russian ru_RU-irina-medium
Spanish es_MX-claude-high Mexican Spanish, high quality. Piper only publishes claude under es/es_MX/claude/high/ — there is no es_ES-claude. I did not invent one. If Brad wants a Castilian option later, that would need a different upstream voice (e.g. es_ES-davefx or similar) added as a fourth mapping.

All three model identifiers and checksums were verified against the same
pinned rhasspy/piper-voices revision already used in this Dockerfile
(ea046e8458f6acd997706d6e6066a022b42f6fb1) by downloading each file fresh
and computing sha256 myself immediately before committing:

  • en_US-amy-medium.onnx — b3a6e47b57b8c7fbe6a0ce2518161a50f59a9cdd8a50835c02cb02bdd6206c18
  • en_US-amy-medium.onnx.json — 95a23eb4d42909d38df73bb9ac7f45f597dbfcde2d1bf9526fdeaf5466977d77
  • ru_RU-irina-medium.onnx — 8ff38212d23da300bbe3705c645e6e5b9475f0bfde01558eb17813e22acaaaaa
  • ru_RU-irina-medium.onnx.json — c2ec28bb38e2b59e93b959b3e40348c1afebbd272f30fed5d41205d08e98a9d7
  • es_MX-claude-high.onnx — 3ef40a71ea63852cd8ab7e6fa7d2ecdcfa67a0b47c9c48e3f10e02ee02083ea0
  • es_MX-claude-high.onnx.json — 1afc81f703c0e4cb3b4d7c0dca096b8b54a98806807f0170cf5eb5557723c12d

I also actually loaded all three with onnxruntime + piper-tts==1.5.0 in a
throwaway venv on this box and synthesized real audio for each (Cyrillic
text through Irina, Spanish text through claude, English through amy) —
this is not just "the checksum matches", the models genuinely work.

Server-side language routing (the hard part)

dockerfiles/hermes-jetson-tts-server.py previously read only input/text
and speed and loaded exactly one voice at startup from HERMES_TTS_VOICE.
The "voice" field the WebUI sent was already ignored. Baking three models
in without a selection mechanism would not have been sufficient.

Implemented: a fixed LANGUAGE_VOICE_MAP dict is the only path from a
client string to a model name:

  • en, en-US (case-insensitive, _/- both accepted) → amy
  • ru, ru-RU → irina
  • es, es-MX, es-ES → claude
  • anything missing, unrecognized, non-string, or malformed → amy (never an
    error, never an unloaded model)
  • a client-supplied "voice" field is never read by the handler — it
    cannot override policy, and no client string is ever used to build a
    Path. resolve_voice_name() always returns a member of the fixed
    three-name set, verified by property-based tests against a battery of
    hostile inputs (path traversal, NUL bytes, wrong types, etc).

Verified live end-to-end with the real downloaded models (not mocked):
curl with {"language":"ru-RU","input":"Привет, как дела?"} returns 200
with X-TTS-Voice: ru_RU-irina-medium; {"language":"en","voice":"ru_RU-irina-medium"}
still returns amy, proving the voice field cannot override the policy.

Preload vs. lazy-load (measured, not guessed)

Chose eager preload of all three voices at process start. Measured RSS
on this hardware with the real models: ~88MB for one voice loaded, ~243MB
for all three (so roughly +155MB versus the previous single-voice
baseline). That comfortably fits the pod's memory budget. Preload also keeps
the fail-closed missing-model check at process start (crash-loop is visible
immediately in kubectl get pods) rather than deferring a possible failure
to a live user's first request in a given language, and avoids
request-serializing a cold model load behind VOICE_LOCK on first use.
Bumped services/hermes/voice-deployment.yaml memory request from
512Mi → 768Mi to cover the measured increase with headroom; the 2Gi limit
already had ample slack and is unchanged.

Is the WebUI actually going to speak Russian in Irina automatically? No — and here's why

I traced dockerfiles/hermes-webui-atlas-patch.py and
dockerfiles/hermes-webui-atlas-voice.js end to end. The WebUI has no
signal for the language of the assistant reply it is about to speak.
hermes-webui-atlas-voice.js's fetchSpeech() posts only
{text: chunk, engine: 'atlas'} to /api/tts; the patched routes.py
Atlas branch previously hardcoded a stale "voice": "en_US-lessac-high"
value that was already dead (ignored server-side) — I removed it rather than
leave a misleading value, since it never did anything. There is no
i18n/locale detection anywhere in these files or in Dockerfile.hermes-webui.

Net effect today: every reply, regardless of its actual language, speaks
in the safe English default (amy). Non-English replies will not
automatically get the right voice yet. The server-side plumbing is fully
functional and tested (see above) for whenever a caller does send
language — it's specifically the signal that's missing, not the
routing.

To close this gap: something upstream of the TTS call needs to know/declare
the reply's language and pass it through
hermes-webui-atlas-voice.js → api/routes.py's atlas branch → the
language field in the JSON POST to hermes-tts. That could be: tagging
each assistant turn server-side with a detected/declared language (e.g. from
the model's own response metadata or a lightweight langid pass) and
threading it through, or a user-facing language preference. I did not
implement any of that — the task was explicit that inventing detection here
would be worse than stating the gap plainly. Documented in
services/hermes/NOTES.md under "Private Jetson voice: multilingual TTS
policy" as well.

Every site naming the voice (grepped lessac, hfc_female, HERMES_TTS_VOICE)

  1. dockerfiles/Dockerfile.hermes-jetson-tts — added 6 new checksummed
    ADD instructions (3 voices × onnx+json) from the same pinned revision;
    extended the build-time PiperVoice.load assertion to load all 6 lessac
    • 3 new models; ENV HERMES_TTS_VOICE=en_US-amy-medium. Lessac models
      kept       (not removed) — see image size delta below.
  2. dockerfiles/hermes-jetson-tts-server.py — DEFAULT_VOICE_NAME default
    now en_US-amy-medium, aligned with the Dockerfile ENV.
  3. services/hermes/voice-deployment.yaml — HERMES_TTS_VOICE →
    en_US-amy-medium; ai.bstein.dev/model annotation changed from naming
    only the single lessac voice to a piper-multilingual- prefix listing all
    three baked voices (en_US-amy-medium, ru_RU-irina-medium,
    es_MX-claude-high), so it is now truthful for a three-voice deployment;
    memory request bumped 512Mi → 768Mi.
  4. dockerfiles/hermes-webui-atlas-patch.py — removed the dead hardcoded
    "voice" field from the Atlas TTS request body (see above); its
    fail-closed replace_exact patching property is untouched everywhere
    else in the file.
  5. services/hermes/NOTES.md — did not previously mention TTS/Piper/voice
    at all; added a new "Private Jetson voice: multilingual TTS policy"
    section documenting the mapping and the WebUI language-signal gap.
  6. services/comms/knowledge/catalog/atlas.{json,yaml} — generated files
    (do not edit by hand); they reference the hermes-tts image only by
    digest, which this PR does not change, so nothing to regenerate here.
  7. hfc_female — zero hits anywhere in the repo; this branch was built
    fresh from origin/main, never from hermes/tts-voice-hfc-female.

Image size delta

Kept the lessac models (high/medium/low, 6 files) per the task instructions
rather than removing them. Added exactly the 3 new voices (6 files):
+189,539,507 bytes (~180.8 MiB) measured from the real downloaded files
(amy 63,201,294B + json 4,882B; irina 63,201,294B + json 4,765B; claude-high
63,122,309B + json 4,963B). If Brad wants to trim image size later, the
lessac set is the obvious removal candidate since nothing in the language
policy routes to it anymore.

Rollout-coupling hazard — read before merging

hermes-jetson-tts-server.py fails closed at startup if HERMES_TTS_VOICE
doesn't name one of the three baked voices, and every baked voice must
exist on disk or load_voices() raises immediately. voice-deployment.yaml:126
pins the hermes-tts image by digest with no Flux image automation
for hermes. The env/manifest change in this PR and the new image must
roll out together, or image-first — if the manifest's env/annotation
changes land before a new image exists with the new models baked in, the
pod crash-loops (old image, new HERMES_TTS_VOICE=en_US-amy-medium, model
not present because the old image only has lessac).

Required post-merge sequence:

  1. Merge this PR (or apply after review — draft, not merged by me).
  2. Trigger the hermes-agent-image-equivalent Jenkins build for
    Dockerfile.hermes-jetson-tts off the merged main commit.
  3. Manually bump the image digest at voice-deployment.yaml:126 to the
    newly built image (no Flux image automation will do this for you).
  4. Let Flux reconcile the digest-pinned manifest.
  5. Confirm the hermes-tts pod restarts cleanly on titan-21 (watch
    startupProbe — 60 × 5s = 300s budget, comfortably covers the ~15-20s
    three-voice load I measured locally) and /health reports all three
    voice names.

Tests (written before the implementation)

New testing/tests/test_hermes_tts_language_routing.py (42 cases): every
allow-listed language variant (case/separator combinations) → correct
voice; unknown/missing/malformed language (wrong types, empty, path
traversal strings, NUL bytes) → amy fallback; client voice field proven
not to override language-driven policy; resolve_voice_name() proven to
always return one of the three baked names for a battery of hostile inputs;
load_voices() fails closed when a baked model file is missing.

Updated testing/tests/test_hermes_chat_quality.py: new checksums, voice
env, annotation, and Dockerfile assertions.

Full suite: pytest testing/ → 2575 passed, 4 skipped (no cluster
access / no visudo in this sandbox), 8 failed — all 8 confirmed to
fail identically on a clean, unmodified checkout of origin/main in a
throwaway worktree before I touched anything (4 of these match the task's
documented pre-existing list; the other 4 — test_atlas_ai_dashboard.py x2,
test_hermes_ai_usage_exporter_coverage.py,
test_hermes_execution_pool_dispatch_v2.py — are pre-existing but weren't
in that list, likely drift since it was written; none touch TTS/voice code
and none are introduced by this diff).

kustomize build services/hermes renders cleanly with the new env/
annotation values present in the output.

Files changed

  • dockerfiles/Dockerfile.hermes-jetson-tts
  • dockerfiles/hermes-jetson-tts-server.py
  • dockerfiles/hermes-webui-atlas-patch.py
  • services/hermes/NOTES.md
  • services/hermes/voice-deployment.yaml
  • testing/tests/test_hermes_chat_quality.py
  • testing/tests/test_hermes_tts_language_routing.py (new)
## Supersedes draft PR #24 Draft PR #24 (`hermes/tts-voice-hfc-female`, from task t_74b76353) implemented `en_US-hfc_female-medium` as the single default voice with `en_US-amy-medium` baked as an unused alternate. Brad changed the decision **after** that task landed: the approved policy is now multilingual, not a single feminine default. This PR starts fresh from `origin/main` (does **not** build on `hermes/tts-voice-hfc-female`) and implements the new policy end to end. **Recommend closing #24 without merging** — it does not reflect the current decision. I have not closed or merged anything myself. ## Approved voice mapping (Brad, 2026-08-20) | Language | Voice | Notes | |---|---|---| | English | `en_US-amy-medium` | | | Russian | `ru_RU-irina-medium` | | | Spanish | `es_MX-claude-high` | **Mexican Spanish, `high` quality.** Piper only publishes `claude` under `es/es_MX/claude/high/` — there is no `es_ES-claude`. I did not invent one. If Brad wants a Castilian option later, that would need a different upstream voice (e.g. `es_ES-davefx` or similar) added as a fourth mapping. | All three model identifiers and checksums were verified against the **same pinned `rhasspy/piper-voices` revision** already used in this Dockerfile (`ea046e8458f6acd997706d6e6066a022b42f6fb1`) by downloading each file fresh and computing sha256 myself immediately before committing: - `en_US-amy-medium.onnx` — `b3a6e47b57b8c7fbe6a0ce2518161a50f59a9cdd8a50835c02cb02bdd6206c18` - `en_US-amy-medium.onnx.json` — `95a23eb4d42909d38df73bb9ac7f45f597dbfcde2d1bf9526fdeaf5466977d77` - `ru_RU-irina-medium.onnx` — `8ff38212d23da300bbe3705c645e6e5b9475f0bfde01558eb17813e22acaaaaa` - `ru_RU-irina-medium.onnx.json` — `c2ec28bb38e2b59e93b959b3e40348c1afebbd272f30fed5d41205d08e98a9d7` - `es_MX-claude-high.onnx` — `3ef40a71ea63852cd8ab7e6fa7d2ecdcfa67a0b47c9c48e3f10e02ee02083ea0` - `es_MX-claude-high.onnx.json` — `1afc81f703c0e4cb3b4d7c0dca096b8b54a98806807f0170cf5eb5557723c12d` I also actually loaded all three with `onnxruntime` + `piper-tts==1.5.0` in a throwaway venv on this box and synthesized real audio for each (Cyrillic text through Irina, Spanish text through claude, English through amy) — this is not just "the checksum matches", the models genuinely work. ## Server-side language routing (the hard part) `dockerfiles/hermes-jetson-tts-server.py` previously read only `input`/`text` and `speed` and loaded exactly one voice at startup from `HERMES_TTS_VOICE`. The `"voice"` field the WebUI sent was already ignored. Baking three models in without a selection mechanism would not have been sufficient. Implemented: a fixed `LANGUAGE_VOICE_MAP` dict is the **only** path from a client string to a model name: - `en`, `en-US` (case-insensitive, `_`/`-` both accepted) → amy - `ru`, `ru-RU` → irina - `es`, `es-MX`, `es-ES` → claude - anything missing, unrecognized, non-string, or malformed → amy (never an error, never an unloaded model) - a client-supplied `"voice"` field is **never read** by the handler — it cannot override policy, and no client string is ever used to build a `Path`. `resolve_voice_name()` always returns a member of the fixed three-name set, verified by property-based tests against a battery of hostile inputs (path traversal, NUL bytes, wrong types, etc). Verified live end-to-end with the real downloaded models (not mocked): `curl` with `{"language":"ru-RU","input":"Привет, как дела?"}` returns 200 with `X-TTS-Voice: ru_RU-irina-medium`; `{"language":"en","voice":"ru_RU-irina-medium"}` still returns amy, proving the `voice` field cannot override the policy. ### Preload vs. lazy-load (measured, not guessed) Chose **eager preload of all three voices at process start**. Measured RSS on this hardware with the real models: **~88MB for one voice loaded, ~243MB for all three** (so roughly +155MB versus the previous single-voice baseline). That comfortably fits the pod's memory budget. Preload also keeps the fail-closed missing-model check at process start (crash-loop is visible immediately in `kubectl get pods`) rather than deferring a possible failure to a live user's first request in a given language, and avoids request-serializing a cold model load behind `VOICE_LOCK` on first use. Bumped `services/hermes/voice-deployment.yaml` memory **request** from 512Mi → 768Mi to cover the measured increase with headroom; the 2Gi limit already had ample slack and is unchanged. ## Is the WebUI actually going to speak Russian in Irina automatically? No — and here's why I traced `dockerfiles/hermes-webui-atlas-patch.py` and `dockerfiles/hermes-webui-atlas-voice.js` end to end. **The WebUI has no signal for the language of the assistant reply it is about to speak.** `hermes-webui-atlas-voice.js`'s `fetchSpeech()` posts only `{text: chunk, engine: 'atlas'}` to `/api/tts`; the patched `routes.py` Atlas branch previously hardcoded a stale `"voice": "en_US-lessac-high"` value that was already dead (ignored server-side) — I removed it rather than leave a misleading value, since it never did anything. There is no i18n/locale detection anywhere in these files or in `Dockerfile.hermes-webui`. **Net effect today:** every reply, regardless of its actual language, speaks in the safe English default (amy). Non-English replies will **not** automatically get the right voice yet. The server-side plumbing is fully functional and tested (see above) for whenever a caller does send `language` — it's specifically the *signal* that's missing, not the *routing*. To close this gap: something upstream of the TTS call needs to know/declare the reply's language and pass it through `hermes-webui-atlas-voice.js` → `api/routes.py`'s atlas branch → the `language` field in the JSON POST to `hermes-tts`. That could be: tagging each assistant turn server-side with a detected/declared language (e.g. from the model's own response metadata or a lightweight langid pass) and threading it through, or a user-facing language preference. I did **not** implement any of that — the task was explicit that inventing detection here would be worse than stating the gap plainly. Documented in `services/hermes/NOTES.md` under "Private Jetson voice: multilingual TTS policy" as well. ## Every site naming the voice (grepped `lessac`, `hfc_female`, `HERMES_TTS_VOICE`) 1. `dockerfiles/Dockerfile.hermes-jetson-tts` — added 6 new checksummed `ADD` instructions (3 voices × onnx+json) from the same pinned revision; extended the build-time `PiperVoice.load` assertion to load all 6 lessac + 3 new models; `ENV HERMES_TTS_VOICE=en_US-amy-medium`. **Lessac models kept** (not removed) — see image size delta below. 2. `dockerfiles/hermes-jetson-tts-server.py` — `DEFAULT_VOICE_NAME` default now `en_US-amy-medium`, aligned with the Dockerfile ENV. 3. `services/hermes/voice-deployment.yaml` — `HERMES_TTS_VOICE` → `en_US-amy-medium`; `ai.bstein.dev/model` annotation changed from naming only the single lessac voice to a `piper-multilingual-` prefix listing all three baked voices (`en_US-amy-medium`, `ru_RU-irina-medium`, `es_MX-claude-high`), so it is now truthful for a three-voice deployment; memory request bumped 512Mi → 768Mi. 4. `dockerfiles/hermes-webui-atlas-patch.py` — removed the dead hardcoded `"voice"` field from the Atlas TTS request body (see above); its fail-closed `replace_exact` patching property is untouched everywhere else in the file. 5. `services/hermes/NOTES.md` — did not previously mention TTS/Piper/voice at all; added a new "Private Jetson voice: multilingual TTS policy" section documenting the mapping and the WebUI language-signal gap. 6. `services/comms/knowledge/catalog/atlas.{json,yaml}` — generated files (do not edit by hand); they reference the `hermes-tts` image only by digest, which this PR does not change, so nothing to regenerate here. 7. `hfc_female` — zero hits anywhere in the repo; this branch was built fresh from `origin/main`, never from `hermes/tts-voice-hfc-female`. ### Image size delta Kept the lessac models (high/medium/low, 6 files) per the task instructions rather than removing them. Added exactly the 3 new voices (6 files): **+189,539,507 bytes (~180.8 MiB)** measured from the real downloaded files (amy 63,201,294B + json 4,882B; irina 63,201,294B + json 4,765B; claude-high 63,122,309B + json 4,963B). If Brad wants to trim image size later, the lessac set is the obvious removal candidate since nothing in the language policy routes to it anymore. ## Rollout-coupling hazard — read before merging `hermes-jetson-tts-server.py` fails closed at startup if `HERMES_TTS_VOICE` doesn't name one of the three baked voices, and every baked voice must exist on disk or `load_voices()` raises immediately. `voice-deployment.yaml:126` pins the `hermes-tts` image **by digest** with **no Flux image automation** for hermes. The env/manifest change in this PR and the new image **must roll out together, or image-first** — if the manifest's env/annotation changes land before a new image exists with the new models baked in, the pod crash-loops (old image, new `HERMES_TTS_VOICE=en_US-amy-medium`, model not present because the old image only has lessac). **Required post-merge sequence:** 1. Merge this PR (or apply after review — draft, not merged by me). 2. Trigger the `hermes-agent-image`-equivalent Jenkins build for `Dockerfile.hermes-jetson-tts` off the merged `main` commit. 3. Manually bump the image digest at `voice-deployment.yaml:126` to the newly built image (no Flux image automation will do this for you). 4. Let Flux reconcile the digest-pinned manifest. 5. Confirm the `hermes-tts` pod restarts cleanly on `titan-21` (watch `startupProbe` — 60 × 5s = 300s budget, comfortably covers the ~15-20s three-voice load I measured locally) and `/health` reports all three voice names. ## Tests (written before the implementation) New `testing/tests/test_hermes_tts_language_routing.py` (42 cases): every allow-listed language variant (case/separator combinations) → correct voice; unknown/missing/malformed language (wrong types, empty, path traversal strings, NUL bytes) → amy fallback; client `voice` field proven not to override `language`-driven policy; `resolve_voice_name()` proven to always return one of the three baked names for a battery of hostile inputs; `load_voices()` fails closed when a baked model file is missing. Updated `testing/tests/test_hermes_chat_quality.py`: new checksums, voice env, annotation, and Dockerfile assertions. **Full suite:** `pytest testing/` → 2575 passed, 4 skipped (no cluster access / no `visudo` in this sandbox), **8 failed** — all 8 confirmed to fail identically on a clean, unmodified checkout of `origin/main` in a throwaway worktree before I touched anything (4 of these match the task's documented pre-existing list; the other 4 — `test_atlas_ai_dashboard.py` x2, `test_hermes_ai_usage_exporter_coverage.py`, `test_hermes_execution_pool_dispatch_v2.py` — are pre-existing but weren't in that list, likely drift since it was written; none touch TTS/voice code and none are introduced by this diff). `kustomize build services/hermes` renders cleanly with the new env/ annotation values present in the output. ## Files changed - `dockerfiles/Dockerfile.hermes-jetson-tts` - `dockerfiles/hermes-jetson-tts-server.py` - `dockerfiles/hermes-webui-atlas-patch.py` - `services/hermes/NOTES.md` - `services/hermes/voice-deployment.yaml` - `testing/tests/test_hermes_chat_quality.py` - `testing/tests/test_hermes_tts_language_routing.py` (new)
hermes-automation added 1 commit 2026-08-20 18:57:24 +00:00 
Supersedes draft PR #24 (hermes/tts-voice-hfc-female): Brad changed the
decision after that task landed, so this starts fresh from origin/main
instead of building on it.

Bakes three checksum-pinned Piper voices (en_US-amy-medium,
ru_RU-irina-medium, es_MX-claude-high) alongside the existing lessac set,
and adds deterministic, allow-listed language routing to
hermes-jetson-tts-server.py: an explicit request "language" field maps
through a fixed dict to one of the three baked voices, with unknown,
missing, or malformed input always falling back to English amy. A
client-supplied "voice" field is never read, so no client input can reach
a filesystem path. All three voices are eagerly preloaded at process
start (measured ~243MB RSS for three vs. ~88MB for one).

The WebUI has no signal for the language of the text it is about to
speak (verified: hermes-webui-atlas-voice.js sends only text and engine),
so no client- or server-side language detection is added; this gap is
documented in NOTES.md and the PR description rather than papered over.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bstein closed this pull request 2026-08-22 20:21:25 +00:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
atlas-reconciler bstein hermes-automation
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: titan/atlas-iac#26
No description provided.
Delete Branch "hermes/tts-voice-multilingual"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?