Capture CI Failure Evidence Before You Classify It

A CI pipeline that fails on unclassified noise trains engineers to ignore it. The failure message says a job died, gives a log tail, and leaves the reader to work out whether they broke the build or the runner ran out of disk. Most teams respond by bolting on a classifier, which is the wrong order. Classification of unlabeled failures is guesswork dressed as automation.

Evidence first, labels second

Before any model decides whether a failure is a product defect or environment noise, someone has to capture what the failure actually looked like. Exit codes, the last successful step, whether the workspace survived, whether the network was reachable, and the diff that triggered the run are facts. A classifier reading only a truncated log is inferring from the least reliable artifact in the chain.

A classifier without structured evidence is a guess with a confidence score attached. Collect the facts at failure time, because they are gone once the runner is torn down.

The four buckets that matter in practice are product defects, environment noise, agent errors, and test defects. They have different owners and different response times, which is why merging them costs so much. A flaky test needs a quarantine label and a ticket for whoever owns the suite. A dead runner needs infrastructure attention. Collapsing both into “build failed” means both wait for the on-call engineer to triage by hand.

Capturing the evidence bundle

The minimum viable evidence set is small enough to gather on every run failure without meaningful overhead:

# Export a failure evidence bundle before the runner is destroyed
{
  "exit_code": $?,
  "failed_step": "$CI_JOB_STAGE",
  "commit_sha": "$CI_COMMIT_SHA",
  "runner_id": "$CI_RUNNER_ID",
  "workspace_preserved": false,
  "disk_free_mb": $(df -m / | awk 'NR==2{print $4}'),
  "network_reachable": $(curl -s -o /dev/null -w '%{http_code}' --max-time 5 https://registry.npmjs.org || echo "timeout"),
  "log_tail_lines": 200
}

Note the disk and network probes. Those two fields alone reclassify a large share of “mystery” failures as environment problems, without any model involved. A deterministic rule that says disk below 200 MB means environment failure is cheaper, faster, and more accurate than an LLM reading logs to reach the same conclusion.

Reserve the model for the residual case: the failure has clean environment evidence, a non-zero exit, and a diff that plausibly caused it. At that point you are asking a judgment question rather than a lookup question, and a model earns its cost. Building the pipeline in this order keeps triage honest and keeps the classifier from becoming a confidence-scored excuse for skipping diagnostics. The architecture of separating deterministic evidence collection from model-based classification is laid out in this HackerNoon piece on CI failure triage.