titan/soteria
3
0
Fork 0
Code Issues 87 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

[hermes] soteria #1250: transient_infra_failure #87

New Issue
Open
opened 2026-10-02 10:31:27 +00:00 by hermes-automation · 0 comments
hermes-automation commented 2026-10-02 10:31:27 +00:00
Member
Copy Link

Hermes auto-triage classified incident soteria/1250 as transient_infra_failure (confidence 0.92); first failed gate: none (failure occurred during SCM checkout, before any pipeline stage or quality gate ran).

Why a human is needed

Build 1250 never reached any of Soteria's actual build or test stages. Jenkins failed while checking out the soteria.git repository from the Gitea SCM server (scm.bstein.dev) in order to read the Jenkinsfile: the git fetch against https://scm.bstein.dev/titan/soteria.git came back with HTTP 500 from the server, and after Jenkins retried the checkout a couple of times it gave up ('Maximum checkout retry attempts reached, aborting'). The whole build only lasted about 1.3 seconds, which fits a failure right at checkout rather than during any build or test logic. Nothing in this evidence points at a problem with Soteria's code, tests, or the commit being built - the fetch itself was rejected by the server with a 500 before Jenkins could even load the pipeline definition. A fix here isn't a change to the soteria repository; it would mean checking the Gitea server (scm.bstein.dev) for whatever caused it to return a 500 to this fetch around 10:27 UTC on 2026-10-02 (overload, a backend error, a restart in progress, etc.), confirming the service is healthy, and then simply re-running the build against the same commit. If this kind of 500 keeps showing up across other builds or repos rather than being a one-off, that would point to a more persistent problem with the Gitea service itself worth looking into directly, separate from anything in Soteria's own code.

Ariadne did not authorize automated remediation: evidence_signature_missing.

Facts

  • jenkins — Jenkins build soteria/1250 failed with no pipeline stage recorded (jenkins.first_failed_stage is null), meaning the failure happened before any declared stage ran. (jenkins.first_failed_stage)
  • jenkins — During SCM checkout (used to read the Jenkinsfile), Jenkins ran 'git fetch --tags --force --progress -- https://scm.bstein.dev/titan/soteria.git +refs/heads/:refs/remotes/origin/', which exited with status 128. (console_failures[0], line_number 21)
  • jenkins — Git's reported cause was: "fatal: unable to access 'https://scm.bstein.dev/titan/soteria.git/': The requested URL returned error: 500". (console_failures[0]; console_tail)
  • jenkins — Jenkins retried the checkout and then aborted with "ERROR: Maximum checkout retry attempts reached, aborting". (console_tail)
  • jenkins — jenkins.failed_tests is an empty array; no structured test results were published for this build. (jenkins.failed_tests)
  • jenkins — The build ran for only 1.312 seconds (10:27:24.053Z to 10:27:25.365Z), consistent with failing immediately at checkout before any build or test work began. (jenkins.duration_seconds, jenkins.timestamps)
  • opensearch — Supplementary log correlation via OpenSearch was unavailable for this incident window due to a connection error on the query side. (log_evidence.error)

Evidence

The earliest failure region of the build console:

	at PluginClassLoader for git//hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1239)
	at PluginClassLoader for git//hudson.plugins.git.GitSCM._checkout(GitSCM.java:1310)
	at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1277)
	at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:136)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:168)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:70)
	at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:317)
	at hudson.model.ResourceController.execute(ResourceController.java:101)
	at hudson.model.Executor.run(Executor.java:460)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- https://scm.bstein.dev/titan/soteria.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: fatal: unable to access 'https://scm.bstein.dev/titan/soteria.git/': The requested URL returned error: 5
... (truncated; the full log is in the linked build)

Inferences

  • The HTTP 500 was returned by the Gitea server handling the fetch request, not by Jenkins or by anything in the soteria repository itself, since the failure occurred while fetching refs in order to read the Jenkinsfile, before any of the repository's own build or test steps had a chance to run.
  • A 500 response is a server-side error rather than an authentication or git-protocol problem, so this looks like a transient issue on the Gitea host at the moment of the fetch (overload, restart, or an internal error) rather than anything wrong with the commit or branch being built.

Links

  • Failed build: https://ci.bstein.dev/job/soteria/1250/
  • Full evidence bundle and audit trail live in Ariadne at /api/admin/audit/events, event types hermes_autotriage_incident and hermes_autotriage_diagnosis.

Filed automatically by Ariadne from a Hermes Agent diagnosis (run run_50b6dd4523ed413d8ae1a2e7a8f26742). Hermes has no write access to this repository; no files or infrastructure were changed.

Hermes auto-triage classified incident `soteria/1250` as **transient_infra_failure** (confidence 0.92); first failed gate: `none (failure occurred during SCM checkout, before any pipeline stage or quality gate ran)`. ## Why a human is needed Build 1250 never reached any of Soteria's actual build or test stages. Jenkins failed while checking out the soteria.git repository from the Gitea SCM server (scm.bstein.dev) in order to read the Jenkinsfile: the git fetch against https://scm.bstein.dev/titan/soteria.git came back with HTTP 500 from the server, and after Jenkins retried the checkout a couple of times it gave up ('Maximum checkout retry attempts reached, aborting'). The whole build only lasted about 1.3 seconds, which fits a failure right at checkout rather than during any build or test logic. Nothing in this evidence points at a problem with Soteria's code, tests, or the commit being built - the fetch itself was rejected by the server with a 500 before Jenkins could even load the pipeline definition. A fix here isn't a change to the soteria repository; it would mean checking the Gitea server (scm.bstein.dev) for whatever caused it to return a 500 to this fetch around 10:27 UTC on 2026-10-02 (overload, a backend error, a restart in progress, etc.), confirming the service is healthy, and then simply re-running the build against the same commit. If this kind of 500 keeps showing up across other builds or repos rather than being a one-off, that would point to a more persistent problem with the Gitea service itself worth looking into directly, separate from anything in Soteria's own code. Ariadne did not authorize automated remediation: `evidence_signature_missing`. ## Facts - **jenkins** — Jenkins build soteria/1250 failed with no pipeline stage recorded (jenkins.first_failed_stage is null), meaning the failure happened before any declared stage ran. (`jenkins.first_failed_stage`) - **jenkins** — During SCM checkout (used to read the Jenkinsfile), Jenkins ran 'git fetch --tags --force --progress -- https://scm.bstein.dev/titan/soteria.git +refs/heads/*:refs/remotes/origin/*', which exited with status 128. (`console_failures[0], line_number 21`) - **jenkins** — Git's reported cause was: "fatal: unable to access 'https://scm.bstein.dev/titan/soteria.git/': The requested URL returned error: 500". (`console_failures[0]; console_tail`) - **jenkins** — Jenkins retried the checkout and then aborted with "ERROR: Maximum checkout retry attempts reached, aborting". (`console_tail`) - **jenkins** — jenkins.failed_tests is an empty array; no structured test results were published for this build. (`jenkins.failed_tests`) - **jenkins** — The build ran for only 1.312 seconds (10:27:24.053Z to 10:27:25.365Z), consistent with failing immediately at checkout before any build or test work began. (`jenkins.duration_seconds, jenkins.timestamps`) - **opensearch** — Supplementary log correlation via OpenSearch was unavailable for this incident window due to a connection error on the query side. (`log_evidence.error`) ## Evidence The earliest failure region of the build console: ``` at PluginClassLoader for git//hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1239) at PluginClassLoader for git//hudson.plugins.git.GitSCM._checkout(GitSCM.java:1310) at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1277) at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:136) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:168) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:70) at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:317) at hudson.model.ResourceController.execute(ResourceController.java:101) at hudson.model.Executor.run(Executor.java:460) Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- https://scm.bstein.dev/titan/soteria.git +refs/heads/*:refs/remotes/origin/*" returned status code 128: stdout: stderr: fatal: unable to access 'https://scm.bstein.dev/titan/soteria.git/': The requested URL returned error: 5 ... (truncated; the full log is in the linked build) ``` ## Inferences - The HTTP 500 was returned by the Gitea server handling the fetch request, not by Jenkins or by anything in the soteria repository itself, since the failure occurred while fetching refs in order to read the Jenkinsfile, before any of the repository's own build or test steps had a chance to run. - A 500 response is a server-side error rather than an authentication or git-protocol problem, so this looks like a transient issue on the Gitea host at the moment of the fetch (overload, restart, or an internal error) rather than anything wrong with the commit or branch being built. ## Links - Failed build: https://ci.bstein.dev/job/soteria/1250/ - Full evidence bundle and audit trail live in Ariadne at `/api/admin/audit/events`, event types `hermes_autotriage_incident` and `hermes_autotriage_diagnosis`. Filed automatically by Ariadne from a Hermes Agent diagnosis (run [run_50b6dd4523ed413d8ae1a2e7a8f26742](https://triage.bstein.dev/chat?resume=run_50b6dd4523ed413d8ae1a2e7a8f26742)). Hermes has no write access to this repository; no files or infrastructure were changed. <!-- hermes-triage job=soteria classification=transient_infra_failure incident=soteria/1250 -->
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
atlas-reconciler bstein hermes-automation
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: titan/soteria#87
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?