Requiring rpi5 while the historical hostname exclusion list removes the very rpi5
nodes that currently have headroom (titan-04/06) left only loaded rpi5s
(titan-05/07/11), so the agent could not schedule and worker went down. Revert to
rpi5-PREFERRED (soft) so it schedules again; proper rpi5 placement needs the
exclusion list refreshed against current node health/capacity, tracked separately.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Un-pinning let the scheduler land worker on titan-12 (rpi4). The agent is heavy
enough that an rpi4 risks the /api/status slowness that trips its liveness probe
- the exact flap we are avoiding. Make hardware=rpi5 a hard requirement so it
runs only on rpi5 storage workers (excluding the saturated/known-flaky ones);
the scheduler places it on a roomy rpi5 (titan-05).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Worker (hermes-agent) was hard-pinned to titan-08 (a workaround after an earlier
attempt to place it on the amd64 titan-22 failed on architecture). That single-
node pin is exactly what makes it fragile: a titan-08 blip (as just happened when
the node's Longhorn CSI went down) strands worker, and the Recreate strategy then
deadlocks because the replacement can't schedule on the one tight node.
Restore the intended multi-node design: run on any arm64 storage worker except the
known-bad/weak ones (matching the repo's own affinity test, which was red). Its
Longhorn volumes have data-locality disabled with replicas on titan-15/17/19, so
there is no locality penalty to running on another node; the scheduler now places
it on a roomier Pi (e.g. titan-05) and a node blip simply reschedules it.
Also relax the gateway /api/status liveness probe (timeout 10s->15s,
failureThreshold 3->5) so a transient slowness (e.g. a brief storage hiccup) no
longer trips a kill-and-restart cascade.
Follow-up (separate): multi-arch agent image to enable the amd64 titan-22 target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Beam-2 with a temperature-fallback ladder made the final-model warmup
run all three temperature retries under beam search before the server
bound its port, so /health was refused for ~8 min and STT was down that
whole time on every roll (and hinted at slow per-utterance decodes).
Production now runs the accurate large-v3-turbo model greedily at a
single temperature, keeping the proper-noun priming prompt that fixes
names like Amy/Córdoba - fast startup, fast decodes, accuracy intact.
Beam stays env-tunable (HERMES_STT_FINAL_BEAM_SIZE) for a future pass;
serve-before-warmup is a recommended follow-up so cold start never
blocks readiness.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
A transcription that landed while the private Whisper service was
restarting (an image roll) crashed hermes_stt_client.py with a raw
urllib ConnectionRefused traceback that got dumped into the
conversation. The client now retries the request with backoff (up to 5
attempts, ~10s - long enough to ride an STT pod restart) and, on a
persistent outage, exits with one concise line ('speech transcription
unavailable...') instead of a stack trace. Delivered via the coordinator
ConfigMap; the next reconcile picks it up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Proper nouns (Córdoba, Cancún) and dropped words came from decoding the
committed transcript with the small model. The image already ships
large-v3-turbo, so the final decode now uses it with beam_size=5, a
temperature fallback ladder, and a proper-noun/accents initial_prompt
that fixes first-pass capitalization and diacritics across EN/ES/RU;
the rolling previews stay on tiny at greedy so the on-the-fly feel is
unchanged. The accurate decode runs in the speculative predecode during
the end-of-speech silence and is cache-reused at commit, so perceived
latency stays low. All decode knobs are env-overridable for on-device
tuning (beam/temperature/prompt), with small as the guaranteed-present
rollback if turbo underperforms on the Jetson. 206 STT tests pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The size cap was applied to the raw wire body, so a nodes list (huge
because of status.images) truncated before the image-stripping ran and
came back as a truncation notice. Accept up to 6 MiB on the wire to
parse and clean, then enforce the 384 KiB model-facing cap on the
stripped result - nodes now returns real data. Delivered via the
cluster-read ConfigMap; picked up on the next pod roll.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The chat prompt still forbade all cluster access, so Hermes told users
it had no visibility even though cluster_read is live - it now knows it
has a read-only cluster tool (no Secrets/Vault) and should use it rather
than deny. Adds tool/research-conditioned acknowledgement guidance: when
a turn needs a lookup, plan or calculation, open with one short 'on it,
~ETA' line then deliver the full answer; simple questions get no
preamble. In spoken mode that first line is read aloud. Fixes two
pre-existing exact-match test pins the HUX/cluster rollout had grown
(plugins.enabled list; a fieldRef env comprehension).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Node .status.images (every cached image on the node) overflowed the
size cap and left json.loads parsing a truncated blob, so a nodes query
came back as a non-JSON error. The de-noise pass now summarizes that
list, and genuine truncation returns the readable prefix with a
narrow-your-query hint instead of an error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The read-only cluster_read tool (and the HUX-12 producer) reach the
Kubernetes API through the kubernetes Service, which kube-proxy DNATs
from the 10.43.0.1 ClusterIP to a control-plane node on 192.168.22.11-13
:6443 - addresses the tenant egress except-block was dropping, so calls
failed with connection-refused. Egress now allows the ClusterIP and
those three apiserver endpoints on 443/6443. RBAC still bounds what is
readable (no Secrets); nothing else about the isolation changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
worker.bstein.dev (the hermes-agent Deployment) gains the same HUX
shape as chat, staged and inert: a foundation-only hux sidecar on the
reviewed WebUI image line (Flux setters bound, 5s probe budgets), an
init that provisions the HMAC identity as slot-100 on the durable home
subtree (create-once context key, O_EXCL subject binding, per-pod
worker key; no relay/router/evidence keys so those trusts fail closed),
and observe-only hook env in the agent container with the runtime
plugin mounted but deliberately NOT enabled - activation is a reviewed
one-line flip per docs/hux/WORKER-PLAN.md, which carries the rollout,
verification gates, canary/rollback ladder and open questions.
Cross-surface continuity remains unclaimed until the live gates pass.
7 new topology-adaptive delivery gates green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
RBAC: the built-in view ClusterRole (which never includes Secrets, so
Vault-managed material stays structurally invisible) plus a read-only
extra for nodes, namespaces, PVs, storage classes, CRDs, Flux resources
and metrics, bound to the chat service account. Tooling: a cluster-read
plugin registers a GET-only cluster_read tool against the in-cluster
API using the pod's projected token - secrets paths refused in the
handler as well, malformed segments rejected, responses bounded and
stripped of managedFields noise. Classified read_files/low in the HUX
capability map. RBAC applies on push; the tool activates when the pods
next roll (bundled with the round-3 voice build).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Real traffic showed the first enforcement pass blocking core assistant
faculties: skills listing and the sandboxed Python classified as
unknown external side effects, browsing denied by default, with no
approval surface in the chat flow. Enforcement returns to observe-only
fleet-wide while the approvals UX and default grants are reworked, and
the capability map now tells the truth about the real toolset: the
Python sandbox is internal shell work, skills/todo/clarify/vision are
reads, browsing is network (medium), image generation writes an
artifact through the trusted broker. Unknown tools remain fail-closed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Removes the canary partition so all four tenants receive the reviewed
build-22 image with the HUX topology and enforcement (every lifecycle
gate passed on ordinal 3). Bumps the hux probe timeouts to 5s: the 2s
readiness exec budget timed out 405 times in 135 minutes on the loaded
ARM node — interpreter spawn cost, not service health — and the
resulting flapping stalled the StatefulSet roll at the canary. Converts
the HUX_IMAGE_TAG/DIGEST setters to block style with the marker on the
value scalar (Flux cannot attach setters inside flow mappings, which
left the metadata stale at build-21) and corrects them to build-22.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Park/resume and gateway-owned stop receipts passed the live gates, so
HUX_TOOL_ENFORCEMENT=1 ships to ordinal 3 (partition unchanged). Every
canary tool call now requires a HUX release; the fleet stays observe-off
until the approvals UX proves out under real traffic.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
hux.autonomy joins the canary flags (which also unlocks multimodal via
its dependency chain). Enforcement remains 0: approvals, gates, budgets
and stop receipts run observe-only until park/resume passes live.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Artifacts, research, friendly modes, multimodal and onboarding join the
canary flag set on ordinal 3 (partition unchanged, enforcement still 0).
Autonomy and release follow-through remain off pending their gates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Ordinal 3 only (partition unchanged): activity timeline, projects,
privacy and memory control join the foundation flag, observe-only.
The roll also serves as the HUX-11 restart-persistence gate via the
recorded identity hashes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Re-applies the staged HUX topology pinned to the reviewed build-21
image (git-2f535d3a...-build-21-release@sha256:e5b9b2fa...), with the
first-activation posture: HUX_FLAGS=hux.foundation only,
HUX_TOOL_ENFORCEMENT=0, and a RollingUpdate partition of 3 so only
hermes-chat-tenant-3 rolls. Adds the /healthz auth bypass on the chat
proxy so HUX-12 health receipts can observe a real 200, points the
evidence policy at it, and makes the delivery flag gate progressive
(foundation first, cards enabled per lifecycle acceptance; unknown
flags still never ship).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Restores the five services/hermes manifests to the live state and
removes the evidence policy/RBAC resources, so pushing this source
chain applies nothing to the cluster beyond the inert hux_mode.py key
in the auto-router ConfigMap (old pods are protected by the plugin's
degrade-to-noop import shim). The activation topology returns as a
dedicated commit pinned to the newly built WebUI digest, per the staged
release sequence in docs/hux/HANDOFF.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The generator now ships the HUX-06 adoption module alongside the
plugin, and the plugin degrades to its previous behaviour (no mode
adoption) if it starts against a ConfigMap rendered before the module
existed - the plugin can never fail to load mid-transition.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Activation-layer staging, fail-closed until enablement: a per-tenant
hux-evidence-producer sidecar on the exact reviewed WebUI image runs
hux_producer.run_once on a 60s loop, inert until the Vault-staged
evidence key (tolerant init, tmpfs, 0400, staged only for the hux
service and producer containers - never hermes or webui), the policy
ConfigMap, and the scope ConfigMap exist. Adds least-privilege
read-only RBAC (pods+statefulset in hermes, the single named Flux
Kustomization), tenant egress to the Kubernetes API ClusterIP and the
traefik edge, the policy allowlist, hux_producer packaging in the WebUI
image, and a third expected WebUI consumer in the Flux release
renderer. Delivery and image-automation gates enforce the boundary.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
HUX-04/HUX-08: after a successful tool execution the hux-runtime plugin
now registers freshly written files as typed artifacts (create or
immutable version by conversation-scoped title, bounded to 1 MiB,
deterministic idempotency keys) and records up to three deduplicated
web sources from real network/web_search output. Emitters are fail-open
and can never break the tool result. Delivered through the plugin
ConfigMap; 95% branch coverage; quality contract updated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
HUX-06: the auto-router now consults the loopback HUX service (worker
trust, HMAC-derived conversation id from the persisted session) for the
conversation's selected friendly mode and maps it onto the existing
provider-neutral pools: fast->auto/fast, thoughtful/research->auto/deep,
create->auto/balanced; private pins the local route and refuses hosted
overrides. Explicit UI picks keep precedence; every HUX absence or
failure falls through to the previous behaviour unchanged. 95% branch
coverage; quality contract registers the new module.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Defense in depth: the tenant router deletes every browser-supplied
X-Hux-* header before asserting its own HUX identity headers, so no
client can forge subject, trust class, or relay key. Regression covers
X-Hux-Subject, X-Hux-Trust and X-Hux-Relay-Key forgeries.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
