How We Built an AI Triage Agent That Fixes CI/CD Failures and Opens Fix PRs — No External API Keys Required
A practical guide for AI engineers, DevOps, and SecOps teams building autonomous remediation into their pipelines.
The Problem Every Platform Team Knows
A build fails at 2 AM. An engineer is paged. They log in, scan 800 lines of CI output, identify a missing package or a misconfigured environment variable, push a one-line fix, and go back to sleep. Total elapsed time: 45 minutes. Value added: one line of YAML.
This cycle repeats dozens of times a week across any active engineering organization. The failures are repetitive, the diagnosis is mechanical, and the fix is often obvious — yet it still demands a human in the loop at every step.
An AI triage agent is our answer: a repository-embedded DevSecOps agent that fetches the failed logs, classifies the failure, assesses security posture, and — where the fix is unambiguous and safe — opens a draft remediation PR. All inside GitHub Actions. All without an external API key.
The cost case is straightforward: GitHub-hosted inference is included in standard GitHub plans — no SaaS contract, no per-seat LLM subscription, no dedicated infrastructure. The build-once investment is a few days of engineering. The return is every future 2 AM page that becomes a morning review of a pre-diagnosed draft PR.
Three Design Principles
Before diving into the architecture, three constraints shaped every decision:
- Zero external dependencies. No third-party LLM API keys stored in secrets. the AI triage agent uses GitHub-hosted inference, authenticating via the ephemeral GITHUB_TOKEN every Actions run already has.
- Safety rails in code, not prompts. The LLM recommends a fix. It never applies one unilaterally. Every guardrail — forbidden file paths, patch size limits, dry-run checks — is enforced by shell scripts before git push. The model cannot override them.
- Humans always review before merge. Every PR the AI triage agent opens is a –draft with a needs-human-review label. Branch protection prevents auto-merge.
Architecture
CI/CD workflow fails
│
▼
[1] Download failed step logs ← last ~400 lines of failed output
[2] Sanitise logs ← strip tokens, keys, credentials
[3] Assemble failure context ← run metadata + logs + changed files
[4] Call the LLM ← system_prompt + SOP + context → verdict.json
│
├── block_deploy: true → Fail the job. Alert on-call.
└── auto_fixable: true → Apply patch (with safety checks) → Draft PR
The agent runs as a composite GitHub Action — a portable, self-contained unit you call from any on: failure job.
What Gets Sent to the Model
Three things go into the context:
- Sanitised log tail — the last ~400 lines of the failed step. The error is almost always in the tail; truncating controls token cost.
- Run metadata — branch name, commit SHA, job names, conclusion status.
- Changed files — the list of files modified in the triggering commit, from the GitHub API.
Alongside this, the model receives a system prompt defining its output schema and governance rules, plus a Standard Operating Procedure (SOP) document — a repo-specific runbook that grounds recommendations in your team’s actual conventions, not generic best practices. The SOP is capped at 32 KB per call.
The Verdict
The model returns structured JSON only — no prose:
{
"root_cause": "pip install failed: package not found at pinned version",
"category": "dependency-error",
"security_risk": { "level": "NONE", "description": "No credentials exposed" },
"block_deploy": false,
"retry_safe": true,
"auto_fixable": true,
"patch": "--- a/requirements.txtn+++ b/requirements.txtn...",
"recommended_fix": {
"description": "Unpin the extras or update to the available version",
"code_snippet": "fastapi[all]>=0.110.0"
}
}The block_deploy flag is what the workflow acts on. If true, the CI job fails hard — the deployment gate closes regardless of what else passed. The model is instructed to default to block_deploy: true when uncertain. A false positive is recoverable. A false negative is not.
Log Sanitization: The Non-Negotiable First Step
Before any log content leaves the runner, it passes through a sanitization script — before the context is assembled. The sanitiser strips:
- GitHub tokens (ghs_, ghp_, github_pat_)
- AWS access key patterns
- Bearer tokens in HTTP headers
- PEM-encoded private key blocks
This step is not optional. CI logs are a common source of accidental credential exposure. Encoding the sanitizer as a standalone, versionable script means it can be audited and tested independently.
Building it also revealed that several of our CI workflows were inadvertently echoing environment variables into logs. The sanitizer catches them in transit — but the right fix was cleaning up the source. The agent exposed a hygiene gap we didn’t know we had.
Safety Rails: What the Model Cannot Do
CheckEnforced by No patches to CI workflow files, secrets dirs, prod Helm values, private keysForbidden path regex in shell scriptPatch ≤ 60 changed lineswc -l on the diffPatch must apply cleanly to the failing commitgit apply –check dry runOnly patch-touched files stagedgit add scoped to paths in diff headersPR always opens as DRAFTgh pr create –draft — hardcodedAgent cannot merge its own PRBranch protection + no auto-merge
If any check fails, the patch is discarded. The verdict JSON is still uploaded as a build artefact for human review. The agent never proceeds to git push on a failed validation.
Governance Controls
The safety rails above are implementation details. From a governance perspective, what they add up to is a set of auditable, policy-enforced controls over an AI system that influences deployment decisions. Here is how they map:
Governance concernHow the AI triage agent addresses itChange managementEvery AI-suggested fix goes through a draft PR with needs-human-review — same approval workflow as any other changeDeployment gateblock_deploy: true halts the pipeline; the model defaults to blocking when uncertainLeast privilegeThe agent stages only files touched by the patch; workflow files, secrets, and prod config are unconditionally off-limitsAudit trailEvery verdict is a versioned build artefact; every model call records the SHA-256 of the active system promptAccountabilityThe system prompt is version-controlled and peer-reviewed like code; a changelog tracks who changed it, why, and whenITSM alignmentThe verdict category field uses ITIL taxonomy — maps directly into your ticketing workflow
The system prompt itself is treated as governed config, not an ephemeral string. A separate changelog records every change with author and rationale. This means a change to the prompt goes through the same PR review process as a change to application code — and leaves the same audit trail.
An auditor asking “what AI systems touch your deployment pipeline?” can be handed a git history and a folder of build artefacts. That is a stronger governance posture than most manual triage processes, which leave no trail at all.
Two Failures, Two Paths
The verdict schema means different failure types take completely different paths through the system. Here are two representative examples.
Example 1: Vulnerable Dependency — Deploy Blocked
A scheduled pipeline runs a dependency audit. It flags a transitive dependency with a known CVE rated CVSS 9.1 — remote code execution, actively exploited in the wild.
Verdict:
{
"root_cause": "Transitive dependency '[email protected]' has CVE-2021-23337 (CVSS 9.1) — prototype pollution leading to RCE",
"category": "security",
"security_risk": {
"level": "CRITICAL",
"description": "Actively exploited vulnerability in a widely-used utility library; exploitation does not require authentication"
},
"governance_impact": {
"severity": "HIGH",
"itil_category": "Incident Management"
},
"block_deploy": true,
"retry_safe": false,
"auto_fixable": true,
"recommended_fix": {
"description": "Pin lodash to >=4.17.21 via a resolution override in package.json",
"code_snippet": ""resolutions": { "lodash": ">=4.17.21" }"
}
}What happens: block_deploy: true fails the CI job immediately. The pipeline stops. Because auto_fixable: true, the AI triage agent opens a draft PR with the resolution override applied — but the deployment gate stays closed until a human reviews, approves, and merges. The security team is notified via the needs-human-review label triggering their GitHub notification rules.
The fix is not deployed automatically. The vulnerability is surfaced, classified, and a patch is ready — all before the on-call engineer opens their laptop.
Example 2: Docker Registry Rate Limit — No Block, Retry Recommended
A feature branch build fails mid-pipeline. The error: toomanyrequests: You have reached your pull rate limit from Docker Hub.
Verdict:
{
"root_cause": "Docker Hub anonymous pull rate limit reached on runner IP; image pull for 'python:3.11-slim' failed with HTTP 429",
"category": "infra-transient",
"security_risk": {
"level": "NONE",
"description": "No security exposure; transient infrastructure throttling"
},
"governance_impact": {
"severity": "LOW",
"itil_category": "Availability Management"
},
"block_deploy": false,
"retry_safe": true,
"auto_fixable": false,
"recommended_fix": {
"description": "Re-run the workflow — rate limits reset on a rolling window. For a permanent fix, authenticate Docker Hub pulls or mirror the base image to your own registry."
}
}What happens: block_deploy: false — the deployment gate stays open. retry_safe: true — the workflow posts a comment recommending a re-run rather than opening a PR. No patch is produced because there is nothing to patch; the code is correct. The failure is classified, recorded as an artefact, and the engineer sees a one-sentence diagnosis instead of hunting through 400 lines of runner output.
This is the case that matters most at scale. Infra-transient failures are the highest-volume category in most pipelines — rate limits, network timeouts, ephemeral DNS failures. An agent that correctly identifies these as non-blocking and retry-safe saves more engineering time than one that only handles dependency errors.
Lessons Learned
The SOP document made the biggest difference. Generic prompts produce generic recommendations. Once we injected the repo’s actual runbook — our branching conventions, pinning strategy, approved registries — recommendation quality improved dramatically. Grounding the model in your context matters more than tuning the model itself.
Draft PRs are psychologically important. Engineers were more comfortable with an agent that “opens suggestions for review” than one that “pushes fixes.” The –draft flag is not just a safety mechanism — it sets the right expectation about who is in control.
Prompt versioning is harder than code versioning. A change to the system prompt can silently shift behavior in ways that only surface on edge-case failures weeks later. Treating the prompt as governed config — with a changelog, SHA tracking, and PR review — is the minimum viable governance for a system that makes deployment decisions.
What This Is Not
Not autonomous. The agent never merges. It opens a draft PR. A human reviews and merges.
Not a replacement for monitoring. the AI triage agent triages failures after they happen. It does not detect anomalies or predict failures. Think of it as a first-responder, not a prevention layer.
Not zero-risk for security-critical patches. The 60-line limit and forbidden path guards reduce blast radius significantly, but any auto-generated patch touching application logic should have a senior engineer in the review chain.
Summary
This pattern shows that meaningful AI automation in CI/CD does not require a new SaaS product, a dedicated ML platform team, or a separate governance framework. The ingredients are already in GitHub Actions: a hosted inference endpoint, an ephemeral token, shell scripts enforcing safety invariants the model cannot override, and version control providing the audit trail.
On cost: GitHub-hosted inference is included in standard GitHub plans. No LLM subscription, no third-party API contract, no additional infrastructure. The pattern costs engineering time to build once; after that it scales to every repository in your organization.
On governance: every control — deployment gates, forbidden paths, draft-only PRs, prompt SHA tracking — is code. It lives in version control, it can be peer-reviewed, and it produces artefacts an auditor can inspect. That is a stronger governance posture than most manual triage processes, which leave no trail at all.
The pattern — sanitise → assemble → infer → validate → act — is reusable across any failure triage problem. Start with triage-only (no auto-apply), measure accuracy against your historical failures, then layer in auto-apply for the categories where you trust the outputs.
The goal is not to remove engineers from the loop — it is to make sure that when an engineer is in the loop, they are looking at a pre-diagnosed, pre-classified failure with a draft fix in front of them. Not a raw 800-line log at 2 AM.
Questions about the safety rail design or prompt governance approach? Happy to discuss in the comments.
Stop Waking Up at 2 AM: Your Pipeline Broke. Your AI Agent Already Knows Why. was originally published in Javarevisited on Medium, where people are continuing the conversation by highlighting and responding to this story.
This post first appeared on Read More