Deterministic First, LLM Second: Rebuilding Automated Code Review

Three months ago, a team member wired an LLM agent directly into our pull request pipeline. Every time someone pushed code, the bot swept in, parsed the git diff, and left line-level feedback. At first, it seemed convenient. Then came the noise. On a single 40-line patch, the agent generated nine comments. Seven were stylistic nitpicks that violated our local linter rules, one was a hallucinated warning about a non-existent parameter, and it completely missed an unhandled null pointer in an edge case.

The bot consumed model tokens, clogged up PR notifications, and trained developers to ignore automated comments entirely. That failed experiment forced us to reconsider how automated review pipelines should actually be built.

The Failure Mode of Pure Agentic Review

Large language models excel at reasoning over context, explaining legacy logic, and suggesting refactors. However, relying on them as the primary filter for code quality introduces structural flaws. LLM inference is stochastic, slow, and expensive compared to deterministic parsers. When an agent evaluates raw diffs without prior constraints, it spends compute on syntax checks, indentation, and imports, tasks that static analysis tools solved decades ago.

More critically, unconstrained agents suffer from high false-positive rates on trivial style points while failing on strict structural invariants. If a static analyzer can verify type safety in 12 milliseconds, passing that same check to a language model wastes resources and reduces feedback reliability.

Automated code review fails when language models perform jobs that deterministic AST parsers handle faster, cheaper, and without hallucination.

Deterministic Gatekeeping in Practice

The open-source community has shifted toward a hybrid pattern. Recent projects like Alibaba’s open-code-review demonstrate this shift by decoupling code inspection into two distinct layers: deterministic analysis first, followed by targeted agentic evaluation.

In this architecture, static linters, formatters, and Abstract Syntax Tree (AST) analyzers run inside a local runner before any LLM API call occurs. If the code breaks compilation rules, fails security linters, or violates formatting rules, the job stops immediately. The language model only receives clean, syntactically valid diffs along with enriched context, such as call graphs and dependency trees extracted during the deterministic pass.

You can run a local deterministic pipeline pass using Go tools and static analysis binaries before passing diffs to an agent review script:

# Run deterministic checks first
golangci-lint run --out-format json > lint-results.json
git diff origin/main...HEAD > pr-changes.diff

# Fail fast if static analysis finds errors
if [ -s lint-results.json ] && grep -q '"Severity":"error"' lint-results.json; then
  echo "Static analysis failed. Halting agent review."
  exit 1
fi

# Pass pre-filtered diffs to the review agent
python3 review_agent.py --diff pr-changes.diff --lint-context lint-results.json

Building a Balanced Code Review Pipeline

To keep automated PR reviews useful without annoying your engineering team, apply three practical rules:

• Filter style early: Configure linters like golangci-lint, ruff, or ESLint to block PRs before any agent triggers. LLMs should never comment on whitespace, missing semicolons, or variable naming conventions.
• Provide explicit AST context: Pass function signatures and caller relationships to the agent rather than raw string diffs. This grounds the model in actual code structure and prevents invented parameter errors.
• Cap comment output: Limit the agent to at most three high-priority inline comments per pull request. Forcing strict output limits stops prompt spam and keeps reviews readable.

Deterministic checks provide the foundation of automated software quality. By placing static linters ahead of language models, you save API costs, cut review latency, and ensure that automated comments actually help developers ship solid code.