titan-24 is an accelerator node (co-hosts the out-of-cluster Sui validator), not
a general worker. The amd64 build leg was requiring node-role worker=true, which
forced labeling titan-24 as a worker and opened it to unrelated cluster
scheduling. It already pins by hostname+arch, so drop the worker requirement and
remove the titan-24 worker-join from the node-prefer CronJob entirely. The build
targets titan-24 specifically (hostname) and tolerates its taint; nothing else
in the cluster gets scheduled there.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Operator steps in order: prepare/uncordon titan-24, merge, run one validation
build (both arch leaves + promoted index), apply the titan-22 affinity flip,
verify placement. Documents the correction that worker membership is reconciled
by the Flux node-prefer-noschedule CronJob, not Ansible, and lists what was
validated locally vs what only a real Jenkins build can prove.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The native amd64 hermes-agent image leg builds on titan-24. Worker membership
in this cluster is reconciled by the node-prefer-noschedule CronJob (kubectl
label), not Ansible, so add titan-24 there:
- clear_worker titan-24 amd64 -> node-role.kubernetes.io/worker=true + hardware=amd64
- a soft PreferNoSchedule guard taint (atlas.bstein.dev/sui-validator=true)
mirroring titan-22's media guard, so routine pods do not crowd the
out-of-cluster Sui validator that co-hosts titan-24. GPU workloads pinned to
titan-24 by hostname are unaffected (PreferNoSchedule never blocks a pinned
pod), and the amd64 build pod tolerates this taint explicitly.
Operator note: this reconciler does not manage cordons (owned by Ananke
recovery). titan-24 is on the recovery uncordon denylist, so the operator must
ensure titan-24 is uncordoned/schedulable before the first amd64 build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Repoint the hermes-agent base FROM at the upstream multi-arch OCI INDEX
digest (tag v2026.7.7.2, revision 9de9c25f) whose arm64 leaf is byte-for-byte
the previously pinned single-arch base, so the arm64 build is unchanged while
the same reviewed version now also resolves an amd64 leaf. Kaniko selects the
matching leaf per build platform.
Rework the release pipeline to build both arches natively and promote a
multi-arch image without switching off kaniko or weakening any existing
security assertion:
- Keep the arm64 kaniko leg on the unchanged rpi5 coordinating pod; it now
pushes an arch-suffixed candidate tag (...-build-<N>-arm64).
- Add a second native amd64 kaniko leg on a titan-24-pinned, tolerating,
resource-capped pod (ceiling strictly below the arm64 leg) that
independently re-verifies the reviewed revision and stashes its leaf
evidence (...-build-<N>-amd64).
- Add ci/scripts/hermes_multiarch_combine.py: a pure-python, fail-closed
combiner that re-reads each per-arch leaf from the registry, proves its
digest AND its config architecture, assembles a Docker manifest LIST
(already inside the promote allow-list), refuses to overwrite an existing
final tag, publishes the arch-less ...-build-<N> tag, and re-verifies the
registry resolved the exact index referencing exactly the two leaves. It
emits the index digest in the SAME digest-file/image-file format the
single-arch step produced, so render/verify-evidence/hermes_oci_promote.py
promote the INDEX with no change to those scripts.
Tests: add test_hermes_multiarch_combine.py (full hash/verification chain);
strengthen the image-builder suites for the two-arch topology (both kaniko
legs carry the reviewed heredoc-compat build-arg; amd64 leg pinned+capped+
boundary-checked; combine stage wiring; expanded evidence archive) without
weakening the arm64-leg assertions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Requiring rpi5 while the historical hostname exclusion list removes the very rpi5
nodes that currently have headroom (titan-04/06) left only loaded rpi5s
(titan-05/07/11), so the agent could not schedule and worker went down. Revert to
rpi5-PREFERRED (soft) so it schedules again; proper rpi5 placement needs the
exclusion list refreshed against current node health/capacity, tracked separately.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Un-pinning let the scheduler land worker on titan-12 (rpi4). The agent is heavy
enough that an rpi4 risks the /api/status slowness that trips its liveness probe
- the exact flap we are avoiding. Make hardware=rpi5 a hard requirement so it
runs only on rpi5 storage workers (excluding the saturated/known-flaky ones);
the scheduler places it on a roomy rpi5 (titan-05).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Worker (hermes-agent) was hard-pinned to titan-08 (a workaround after an earlier
attempt to place it on the amd64 titan-22 failed on architecture). That single-
node pin is exactly what makes it fragile: a titan-08 blip (as just happened when
the node's Longhorn CSI went down) strands worker, and the Recreate strategy then
deadlocks because the replacement can't schedule on the one tight node.
Restore the intended multi-node design: run on any arm64 storage worker except the
known-bad/weak ones (matching the repo's own affinity test, which was red). Its
Longhorn volumes have data-locality disabled with replicas on titan-15/17/19, so
there is no locality penalty to running on another node; the scheduler now places
it on a roomier Pi (e.g. titan-05) and a node blip simply reschedules it.
Also relax the gateway /api/status liveness probe (timeout 10s->15s,
failureThreshold 3->5) so a transient slowness (e.g. a brief storage hiccup) no
longer trips a kill-and-restart cascade.
Follow-up (separate): multi-arch agent image to enable the amd64 titan-22 target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The overlay status caption already carries a static ellipsis (e.g. 'Thinking…'),
and after ~2.5s of silent thinking the 'working' affordance added an animated
dots ::after on top of it — so the label intermittently rendered as 'Thinking……'
only on longer turns. Drop the animated text dots and keep the orb-halo shimmer
as the liveness cue, so the ellipsis stays a single, consistent static one.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Conversation mode was only reachable via a small icon by the composer, which is
easy to miss on a fresh session. Inject a prominent, centred 'Start
conversation' button into the empty new-chat state (#emptyState), below the
subtitle and above the suggestions, so it sits in the vertical centre of the
screen. It is created only when local voice is available, honours the same
show/hide preference as the composer toggle, and enters conversation mode
through the same activate() path. Fully guarded so it degrades to nothing if the
empty state is absent. New probe scenario verifies it is created, visible, and
opens the conversation overlay on press.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
On an Android standalone PWA the composer's bottom control row was rendering
about one line below the visible viewport, behind the system gesture bar, so
those controls were unreachable. The theme already pads the titlebar with
env(safe-area-inset-top/left/right), but the viewport meta never opted into
viewport-fit=cover, so every safe-area inset collapsed to 0 and the bottom edge
had no reservation.
Add viewport-fit=cover to the viewport meta (base patch) so the insets carry
real values, and reserve env(safe-area-inset-bottom) at the bottom of the
composer (brand.css), additive with the app's existing --keyboard-bottom-inset
and absorbed by the flex-1 message scroller so total height stays within the
viewport. Inert on desktop (env() resolves to 0). Dockerfile verifies the meta
patch landed; brand/dockerfile contracts covered by tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Diagnosed from a live voice session (e1b9fccb90ef): three turns failed with
raw '**Error:** HTTP 502 ... hermes-{claude,codex}-broker' because the agent
pod hosting the model brokers rolled mid-conversation. The voice client
correctly refused to speak the error envelope, but it then dropped the user's
utterance and forced them to repeat it three times.
Voice: on a TRANSIENT provider error (5xx/502/'error sending request'/timeout),
conversation mode now re-runs the errored turn in place through the app's own
regenerate action (which truncates the errored turn — no duplicate user
message) and stays in Thinking so its cues cover the reconnect gap. Bounded to
MAX_TRANSIENT_RETRIES (2); a non-transient error or an exhausted budget still
drops cleanly to 'let's try that again — listening'. The raw error is never
spoken. New probe scenarios cover retry-then-recover and the bounded-then-drop
path; source contract updated.
STT: the same session mis-transcribed 'CUI' as 'cue'. Prime the default
initial_prompt with the domain acronyms the user uses (CUI, FOUO, DoD, NIST,
CMMC, FIPS, RMF, POA&M, ATO, SBU) so they bias to uppercase forms.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The orb watermark was showing the character art as a low-opacity square
box (visible edges) — the earlier screen-blend pass washed it into a glow
that lost the artwork. Reprocess the source into a feathered circle so the
orb crops it (no box) while keeping the character's facial features, paint
it with a normal blend at 0.5 opacity, and size it to fill the orb (inset
7%). Verified by compositing over a simulated orb before shipping.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
- REGRESSION FIX: the full-screen conversation overlay stopped opening
(fell back to the inline bar) because building the language/output
selectors inside the overlay's single try/catch could throw on a
phone (navigator.mediaDevices/setSinkId). Overlay build is now two
phases: the essential orb+captions+controls attach first; the
selectors attach after, each guarded, so a selector failure omits only
that control and never the visualization. Control builders can no
longer throw (inert hidden fallback); device enumeration is async
after attach. Regression test covers mediaDevices-undefined and
enumerate-rejects.
- Orb watermark is the processed Hermes character glyph (feathered,
circle-cropped, screen-blended so the face glows on the dark orb), no
more white/grey box.
- Output selector defaults to the loudspeaker (excludes the
communications/earpiece endpoint that the open mic otherwise forces);
explicit choice sticks; hidden gracefully where setSinkId is
unsupported. 291 voice tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Beam-2 with a temperature-fallback ladder made the final-model warmup
run all three temperature retries under beam search before the server
bound its port, so /health was refused for ~8 min and STT was down that
whole time on every roll (and hinted at slow per-utterance decodes).
Production now runs the accurate large-v3-turbo model greedily at a
single temperature, keeping the proper-noun priming prompt that fixes
names like Amy/Córdoba - fast startup, fast decodes, accuracy intact.
Beam stays env-tunable (HERMES_STT_FINAL_BEAM_SIZE) for a future pass;
serve-before-warmup is a recommended follow-up so cold start never
blocks readiness.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
A transcription that landed while the private Whisper service was
restarting (an image roll) crashed hermes_stt_client.py with a raw
urllib ConnectionRefused traceback that got dumped into the
conversation. The client now retries the request with backoff (up to 5
attempts, ~10s - long enough to ride an STT pod restart) and, on a
persistent outage, exits with one concise line ('speech transcription
unavailable...') instead of a stack trace. Delivered via the coordinator
ConfigMap; the next reconcile picks it up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Final conversation-mode polish from mobile testing:
- The Workspace toggle now sits with the chat/Telegram nav at every
width: nav.rail on desktop, the top app titlebar on mobile. The
floating pill that pushed the mobile composer's control row (and the
conversation-mode button) off screen is gone - a fallback exists only
for headless DOMs and is pinned to a top corner, never over the
composer.
- The conversation orb watermark is now the Hermes character avatar
(static/hermes-agent-192.png) instead of the caduceus staff.
- User-facing 'hands-free' copy renamed to 'Conversation mode'.
- Wrong-voice fix: strongReplyLanguage flagged Spanish on a single
accented char, so an English reply naming European cities (Zürich,
Málaga) overrode the correct English STT detection and was spoken by
the Spanish voice. Detection now requires density (Cyrillic >=4 at
>=50%, or inverted punctuation / >=2 accents corroborated by Spanish
stopwords); plain English always speaks English, forced language wins,
accent-free Spanish still routes via trusted STT. 294 voice tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Three conversation-mode fixes in one pass:
- Interim-acknowledgement truncation: when an interim ack folds into the
hidden worklog segment mid-speech, the retained unspoken tail is now
flushed and spoken in full before the Thinking transition, and a
distinct follow-up message is chunked from its own start and queued
after the interim drains (no more 'stops after the first clause, rest
resurfaces with the next message').
- Natural thinking fillers: brief per-language interjections (Umm/Hmm/
One sec; Mmm/A ver; Хм/Секунду) on genuine >1.9s thinking gaps only,
non-repeating, answer-preempting, mute-aware.
- One unified audio sink for every spoken output (reply, cues, fillers,
WAV fallback) - fixes cues playing the loudspeaker while the reply
used a different output - plus a tidy corner output-device selector
(enumerateDevices + setSinkId, feature-detected, session-only) styled
like the language selector. Also realigns two STT-server decode-param
assertions to the dict form from the STT tuning commit. 284 voice tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Default final beam 5 -> 2: the accuracy/speed knee - most of beam
search's benefit at ~2x greedy instead of ~5x, protecting commit
latency on the Jetson (still env-overridable via
HERMES_STT_FINAL_BEAM_SIZE). Prompt now also primes common names (Amy,
Claude, Hermes) so 'Amy' stops transcribing as 'aiming'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Proper nouns (Córdoba, Cancún) and dropped words came from decoding the
committed transcript with the small model. The image already ships
large-v3-turbo, so the final decode now uses it with beam_size=5, a
temperature fallback ladder, and a proper-noun/accents initial_prompt
that fixes first-pass capitalization and diacritics across EN/ES/RU;
the rolling previews stay on tiny at greedy so the on-the-fly feel is
unchanged. The accurate decode runs in the speculative predecode during
the end-of-speech silence and is cache-reused at commit, so perceived
latency stays low. All decode knobs are env-overridable for on-device
tuning (beam/temperature/prompt), with small as the guaranteed-present
rollback if turbo underperforms on the Jetson. 206 STT tests pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The size cap was applied to the raw wire body, so a nodes list (huge
because of status.images) truncated before the image-stripping ran and
came back as a truncation notice. Accept up to 6 MiB on the wire to
parse and clean, then enforce the 384 KiB model-facing cap on the
stripped result - nodes now returns real data. Delivered via the
cluster-read ConfigMap; picked up on the next pod roll.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The 'Something went wrong - listening' state with no spoken answer was a
false positive: readAssistantTurn flagged the whole turn as an error if
ANY segment was error-stamped - including a recovered/transient tool
error or a cancellation notice from an earlier interim - and threw away
the real answer that the same turn produced. Error now surfaces only
when the turn yielded no spoken answer at all; a turn with real content
is spoken normally. Softened the genuine-error label to the friendlier
'Let's try that again - listening'. New probe scenarios lock both: an
error segment alongside an answer speaks the answer (error=false), and
an error-only turn still reports the error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Real root cause (confirmed against the live build-24 DOM): the caption
and TTS extraction fell back to turn.textContent whenever a settle-frame
race left no readable answer segment, scraping the avatar letter,
author name and 'Processed 13s' chip - and that truncated reply made
TTS speak only the first segment then drop to Listening even with the
mic muted (the muted-mic first-sentence-stop). Extraction now prefers
each answer segment's data-raw-text, else the answer .msg-body only
(excluding thinking/tool/worklog/role chrome), and the textContent
fallback is gone; a genuinely mid-flight reply retries briefly so the
whole thing is read before the overlay drains. Also: the app's own
caduceus mark embedded in the conversation orb as a subtle watermark; a
corner language selector (Auto + en/es/ru) that forces both the STT
hint and the reply voice; and a thinking affordance after 2.5s of dead
time. New response probe + extraction test prove caption==body-only and
that every sentence reaches the TTS queue. 278 voice-lane tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The chat prompt still forbade all cluster access, so Hermes told users
it had no visibility even though cluster_read is live - it now knows it
has a read-only cluster tool (no Secrets/Vault) and should use it rather
than deny. Adds tool/research-conditioned acknowledgement guidance: when
a turn needs a lookup, plan or calculation, open with one short 'on it,
~ETA' line then deliver the full answer; simple questions get no
preamble. In spoken mode that first line is read aloud. Fixes two
pre-existing exact-match test pins the HUX/cluster rollout had grown
(plugins.enabled list; a fieldRef env comprehension).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Node .status.images (every cached image on the node) overflowed the
size cap and left json.loads parsing a truncated blob, so a nodes query
came back as a non-JSON error. The de-noise pass now summarizes that
list, and genuine truncation returns the readable prefix with a
narrow-your-query hint instead of an error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The read-only cluster_read tool (and the HUX-12 producer) reach the
Kubernetes API through the kubernetes Service, which kube-proxy DNATs
from the 10.43.0.1 ClusterIP to a control-plane node on 192.168.22.11-13
:6443 - addresses the tenant egress except-block was dropping, so calls
failed with connection-refused. Egress now allows the ClusterIP and
those three apiserver endpoints on 443/6443. RBAC still bounds what is
readable (no Secrets); nothing else about the isolation changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
- Captions read message bodies only (the scraper was concatenating
avatar, author and worklog chips); multi-segment interim turns are
now speakable and drive clean speak-to-thinking-to-speak cycles when
playback drains mid-turn.
- Dynamic endpointing: complete-looking partials (3+ words or terminal
punctuation) endpoint at the base window; the long hold remains only
for one-two-word fragments. A stale-busy 10s settle wait on every
post-error send is gone.
- Both overlay captions are bounded, touch-scrollable regions with
follow-tail; caps raised for long turns.
- Error envelopes are never spoken or captioned; errored turns run
resyncCapture (fresh STT session on the hot mic).
- Workspace toggle now lives in the sidebar rail (floating button only
below the rail breakpoint).
- False 'session unavailable' toast root-caused: the router continuity
guard shows it on a 409 that fired when a transient profile-listing
failure failed closed into a fake cross-profile mismatch; the patcher
now answers from the alias cache and never claims a default-vs-named
mismatch while aliases are unconfirmed.
- Barge-in sends carry a one-line cut-point marker with the last spoken
sentence; visible-history truncation judged infeasible client-side.
- Language switching works end-to-end: sticky per-session STT language
hint (restarting an unused next session on switch), reply voice from
script evidence, STT detection, then stopword heuristic; cues and WAV
fallback share the turn language.
- Legacy CI guards: node skip for the DOM probe, ffmpeg/codec skips for
the container-fallback test. 272 voice-lane tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
worker.bstein.dev (the hermes-agent Deployment) gains the same HUX
shape as chat, staged and inert: a foundation-only hux sidecar on the
reviewed WebUI image line (Flux setters bound, 5s probe budgets), an
init that provisions the HMAC identity as slot-100 on the durable home
subtree (create-once context key, O_EXCL subject binding, per-pod
worker key; no relay/router/evidence keys so those trusts fail closed),
and observe-only hook env in the agent container with the runtime
plugin mounted but deliberately NOT enabled - activation is a reviewed
one-line flip per docs/hux/WORKER-PLAN.md, which carries the rollout,
verification gates, canary/rollback ladder and open questions.
Cross-surface continuity remains unclaimed until the live gates pass.
7 new topology-adaptive delivery gates green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
