titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity

WIP: hermes: reap orphaned CLI lane processes #20

Draft
hermes-automation wants to merge 1 commits from feature/hermes-cli-process-reaping into main
pull from: feature/hermes-cli-process-reaping
merge into: titan:main
Conversation 0 Commits 1 Files Changed 4 +779 -1
hermes-automation commented 2026-08-17 10:48:08 +00:00
Member
Copy Link

Draft; human review is required. This repair is intentionally stacked on PR #15. Review repair commit a242dcc786 after PR #15.

Problem and reproduced path

  • cli_lane_runner.py was PID 1 and started each provider with start_new_session=True.
  • Provider terminal commands can create further process groups and Git/SSH helpers. When an intermediate parent exits, Linux reparents those helpers to PID 1.
  • The threaded runner correctly waits for provider Popen owners, but it had no safe owner for adopted children. Exited adopted Git helpers therefore remained defunct.
  • Focused Linux tests reproduce that exact provider-session path with real Git and SSH helpers.

Repair

  • Install a single-threaded outer Linux child-subreaper as container PID 1 before forking the lane runner.
  • Reap only the supervisor's direct runner or Linux-adopted children. There is no SIGCHLD handler and no competing waiter for active provider children.
  • Preserve inherited provider streams and exact shell-compatible runner status.
  • Forward termination across captured process groups, allow a bounded grace period, then SIGKILL and drain on cancellation or runner exit.
  • Overwrite a mode-0600 process-supervisor.json document with fixed-cardinality active, adopted, orphaned, and reaped counts. It contains no argv, environment, credentials, or provider output. In-memory orphan identity tracking is bounded by current adoptions.

Compatibility topology

  • Current main: d8f2d818b9
  • Exact current PR #15 head: 1ba9f13959
  • This repair is a direct child of that exact PR #15 head, which is itself based on current main and merged PR #13.

Evidence

  • 16 supervisor lifecycle tests passed with 100% line and branch coverage. Cases cover normal and nonzero exit, stream integrity, timeout/cancellation SIGTERM-to-SIGKILL, provider cancellation, real Git/SSH orphan adoption, four concurrent provider lanes, runner exit/restart cleanup, exec failure, state-write failure, and bounded counters.
  • 213 Hermes CLI, blocked-task, finalization/recovery, agent layout/security, and worker restart tests passed on the exact stacked topology.
  • Local quality gate passed: 506 tests, 98.839% tracked coverage, zero managed files above 500 lines, and all docs/smell/hygiene/unit/coverage checks green. Both new files are below 500 lines.
  • Hermes kustomize render and kubectl apply --dry-run=client passed.
  • Semgrep 1.171.0 p/python: 151 rules, zero findings on the new module and tests.
  • Trivy 0.70.0 secret scan: zero secrets. Baseline/current agent manifest comparison introduced zero high/critical findings.

No image was published; no merge, Flux reconciliation, deployment, rollout, pod restart, credential inspection, or live runtime mutation was performed.

Draft; human review is required. This repair is intentionally stacked on PR #15. Review repair commit a242dcc786576ae1a18000cb4941836ff610dff2 after PR #15. Problem and reproduced path - cli_lane_runner.py was PID 1 and started each provider with start_new_session=True. - Provider terminal commands can create further process groups and Git/SSH helpers. When an intermediate parent exits, Linux reparents those helpers to PID 1. - The threaded runner correctly waits for provider Popen owners, but it had no safe owner for adopted children. Exited adopted Git helpers therefore remained defunct. - Focused Linux tests reproduce that exact provider-session path with real Git and SSH helpers. Repair - Install a single-threaded outer Linux child-subreaper as container PID 1 before forking the lane runner. - Reap only the supervisor's direct runner or Linux-adopted children. There is no SIGCHLD handler and no competing waiter for active provider children. - Preserve inherited provider streams and exact shell-compatible runner status. - Forward termination across captured process groups, allow a bounded grace period, then SIGKILL and drain on cancellation or runner exit. - Overwrite a mode-0600 process-supervisor.json document with fixed-cardinality active, adopted, orphaned, and reaped counts. It contains no argv, environment, credentials, or provider output. In-memory orphan identity tracking is bounded by current adoptions. Compatibility topology - Current main: d8f2d818b9a552ea6c2d7fe86554be829bd5ffff - Exact current PR #15 head: 1ba9f139594e2b24fd9080a13c34b2198124faae - This repair is a direct child of that exact PR #15 head, which is itself based on current main and merged PR #13. Evidence - 16 supervisor lifecycle tests passed with 100% line and branch coverage. Cases cover normal and nonzero exit, stream integrity, timeout/cancellation SIGTERM-to-SIGKILL, provider cancellation, real Git/SSH orphan adoption, four concurrent provider lanes, runner exit/restart cleanup, exec failure, state-write failure, and bounded counters. - 213 Hermes CLI, blocked-task, finalization/recovery, agent layout/security, and worker restart tests passed on the exact stacked topology. - Local quality gate passed: 506 tests, 98.839% tracked coverage, zero managed files above 500 lines, and all docs/smell/hygiene/unit/coverage checks green. Both new files are below 500 lines. - Hermes kustomize render and kubectl apply --dry-run=client passed. - Semgrep 1.171.0 p/python: 151 rules, zero findings on the new module and tests. - Trivy 0.70.0 secret scan: zero secrets. Baseline/current agent manifest comparison introduced zero high/critical findings. No image was published; no merge, Flux reconciliation, deployment, rollout, pod restart, credential inspection, or live runtime mutation was performed.
hermes-automation added 6 commits 2026-08-17 10:48:08 +00:00
hermes-automation changed title from hermes: reap orphaned CLI lane processes to WIP: hermes: reap orphaned CLI lane processes 2026-08-17 10:48:21 +00:00
This pull request has changes conflicting with the target branch.
  • services/hermes/agent-deployment.yaml
View command line instructions.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feature/hermes-cli-process-reaping:feature/hermes-cli-process-reaping
git checkout feature/hermes-cli-process-reaping
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
atlas-reconciler bstein hermes-automation
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: titan/atlas-iac#20
No description provided.
Delete Branch "feature/hermes-cli-process-reaping"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?