titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity

hermes: record successful LAN recovery verification

Browse Source
This commit is contained in:
jenkins 2026-09-30 11:11:04 -05:00
parent 0574eb590c
commit 3a1c5ea2dc
1 changed files with 33 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

33
docs/hermes_suite_node_recovery_20260930.md
View File

@ -42,3 +42,36 @@ be collected. Stop new submissions, reconcile the two metadata ledgers with
owner/idempotency conflicts rejected, then restore the node selector and volume owner/idempotency conflicts rejected, then restore the node selector and volume
reference through Git. Never switch back to an older ledger while accepting jobs, reference through Git. Never switch back to an older ledger while accepting jobs,
and retain both PVCs during recovery. Do not restart an active inference job. and retain both PVCs during recovery. Do not restart an active inference job.
## Verification
Recovery manifest commit: `0574eb590cd2a1ddae8a12e3535353be0c3699f5`.
Flux applied this revision; Hermes and its required dependencies reported Ready.
The planner reported one ready/available replica on titan-24, with zero restarts.
Both the original and recovery PVCs remained Bound.
Kustomize rendering and client dry runs passed for Hermes and the Flux root.
The changed planner resources also passed a server-side dry run. This kubectl
version rejects combining `--server-side` with `--dry-run=client`; those modes
were checked separately. Flux diff showed only the expected placement and
volume changes in Hermes.
From the existing LAN jump host, HTTPS used `worker.bstein.dev` with an explicit
connection to `192.168.22.50:443`, certificate verification enabled, proxies
disabled, and redirects rejected. Operational-token capabilities and health
returned 200; missing and invalid credentials returned 401; `/api/pull` returned
404. Tokens were read privately from the injected credentials and not printed.
One synthetic 14-case job completed through the same HTTPS path:
- Job: `8766db2f378f408491361b04c9e3cf13`.
- Serialized request: 9847 bytes.
- Server duration: 55.431 seconds; client duration: 56.708 seconds.
- Runtime model: `claude-opus-5-5`.
- Three validated model passes; zero retries.
- Nine natural families and nine final tasks; maximum group size two.
- Exact alias coverage passed. Repeated submission with the same key returned
the same job rather than launching duplicate inference.
No real suite was rerun. Laptop connectivity itself was not tested from here.
Titan-22's fault and its inaccessible historical job outcome remain unresolved.