45 lines
2.6 KiB
Markdown
45 lines
2.6 KiB
Markdown
# Suite planner node recovery, 2026-09-30
|
|
|
|
Titan-22 stopped reporting at 15:40:50 UTC and became unreachable at
|
|
15:46:05 UTC. The suite planner was evicted and could not reschedule: both
|
|
its node selector and its local-path metadata volume required titan-22.
|
|
The node was also unreachable over SSH. Its physical fault remains unknown.
|
|
|
|
The recovery deployment uses titan-24 with the same pinned amd64 Claude CLI,
|
|
Opus 5.5 model, credentials, routing rules, prompts, and execution policy.
|
|
It requests 100m CPU and 512Mi memory, with limits of 2 CPU and 2Gi memory.
|
|
It requests no GPU and does not move another workload. Titan-24 is the
|
|
available compatible host; the simulation host remains reserved.
|
|
|
|
The original `hermes-suite-metadata` PVC remains bound to titan-22 and is
|
|
protected from Flux pruning. Do not delete it. The recovery deployment uses
|
|
the separate `hermes-suite-metadata-recovery` local-path PVC on titan-24.
|
|
Historical job status and idempotency records are unavailable until the old
|
|
disk can be accessed. Previous results and active checkpoints were in memory
|
|
and cannot be recovered by mounting a metadata database alone.
|
|
|
|
Use a fresh client job revision and Idempotency-Key for an intentionally new
|
|
attempt after the recovery endpoint is verified. The old key cannot provide
|
|
deduplication against the inaccessible ledger. No real suite is automatically
|
|
resubmitted by this deployment. New jobs retain normal durable idempotency on
|
|
the recovery volume. The new volume is still node-local, not replicated storage.
|
|
|
|
The HTTP URLs, token retrieval, request/response schema, 7200-second maximum,
|
|
disabled estimated-cost guard, configuration revision `suite-v6-20260929`,
|
|
prompt revision `implementation-proximity-adaptive-v6-20260930`, execution
|
|
revision `suite-adaptive-v11-20260930`, and policy revision
|
|
`implementation-five-v1-20260929` are unchanged.
|
|
|
|
Deployment annotation: `suite-v6-adaptive-v11-node-recovery-20260930`.
|
|
Metadata-store annotation: `titan24-recovery-20260930`.
|
|
|
|
Hermes no longer depends on Jenkins readiness for Flux reconciliation. Jenkins
|
|
is not needed by the inference request path; its outage had blocked recovery.
|
|
The suite planner is now an explicit Hermes Flux health check.
|
|
|
|
Rollback must wait for titan-22 to be healthy and all recovery jobs/results to
|
|
be collected. Stop new submissions, reconcile the two metadata ledgers with
|
|
owner/idempotency conflicts rejected, then restore the node selector and volume
|
|
reference through Git. Never switch back to an older ledger while accepting jobs,
|
|
and retain both PVCs during recovery. Do not restart an active inference job.
|