Why Agent-Driven Code Review Needs Real-Time Observability Hooks

Agent-driven code review tools have arrived. Alibaba’s open-code-review runs deterministic pipelines paired with LLM agents to flag issues line-by-line. But here’s the problem: when your agent misses something or hallucinates a false positive, you have no visibility into why. You’re flying blind. The fix is building observability into the agent’s decision loop—capturing every step, every LLM call, every filtering decision. Without it, you’re debugging crashes after production deploys, not during the review.

Step 1 – Instrument Agent LLM Calls

Start by capturing every prompt sent to the LLM and every response returned. This means wrapping your LLM client (OpenAI, Claude, local Ollama) with middleware that logs context: the code snippet analyzed, the security rule being checked, the model used, latency, token count. Don’t log just success—log failures, timeouts, and rate limits too.

class ObservedLLMClient:
    def __init__(self, client, logger):
        self.client = client
        self.logger = logger
    
    def call(self, prompt, rule_name, snippet_hash):
        start = time.time()
        try:
            response = self.client.complete(prompt)
            self.logger.log({
                'timestamp': time.time(),
                'rule': rule_name,
                'snippet': snippet_hash,
                'tokens': response.usage.total_tokens,
                'latency_ms': (time.time() - start) * 1000,
                'status': 'success'
            })
            return response
        except Exception as e:
            self.logger.log({
                'timestamp': time.time(),
                'rule': rule_name,
                'snippet': snippet_hash,
                'error': str(e),
                'latency_ms': (time.time() - start) * 1000,
                'status': 'failed'
            })
            raise

This single step eliminates the black box. You now know exactly which rule triggered which model, how long it took, and whether it succeeded. When an agent flags a false positive in production, you replay that exact log entry and see the prompt that caused it.

Step 2 – Track Rule Filtering Decisions

Code review agents don’t just call an LLM once. They pipeline results: initial scan → LLM analysis → filtering → deduplication → confidence scoring. At each stage, findings get dropped. Log the moment a finding is filtered out and why. Was it below confidence threshold? Duplicate? Suppressed by policy? This is where most false negatives hide—findings that should surface but don’t because a filter was too aggressive.

Dedicate a structured log to the filtering pipeline. Each finding should have a trail showing every filter it passed or failed.

def apply_filters(findings, config):
    for finding in findings:
        trace = {'finding_id': finding.id, 'filters': []}
        
        # Confidence filter
        if finding.confidence < config.min_confidence:
            trace['filters'].append({
                'name': 'confidence_threshold',
                'threshold': config.min_confidence,
                'value': finding.confidence,
                'action': 'dropped'
            })
            continue
        
        # Duplicate check
        if is_duplicate(finding, previous_findings):
            trace['filters'].append({
                'name': 'deduplication',
                'action': 'dropped'
            })
            continue
        
        # Policy suppression
        if matches_suppression_policy(finding, config.policies):
            trace['filters'].append({
                'name': 'policy_suppression',
                'policy': matched_policy,
                'action': 'dropped'
            })
            continue
        
        trace['filters'].append({'action': 'passed'})
        log_filter_trace(trace)
        yield finding

Step 3 - Correlate Findings to Git Metadata

A finding without context is noise. Attach commit hash, author, changed files, and diff context to every finding. This ties agent decisions to actual code changes and makes it trivial to audit why the agent flagged something specific to this PR versus a similar pattern in an older one.

Query your Git history when the agent runs. Store the metadata alongside the finding so correlation queries run instantly. When you're debugging a false positive, you can ask: "Show me all findings from this author in the last week" or "List findings triggered by this exact diff pattern."

Step 4 - Index Findings into a Time-Series Store

Store all findings—passed and failed—in a time-series database (Prometheus, InfluxDB, TimescaleDB). Index by rule name, severity, author, repository, and time window. This enables three things: trending (are security findings increasing?), correlation (which rules fire together?), and anomaly detection (when does an agent's behavior change?).

A simple query: "Show me all HIGH severity findings in the last 7 days, grouped by rule, sorted by frequency." This tells you which security rules matter most in your codebase right now. If suddenly a rule that fires 2–3 times a week fires 50 times, something changed—maybe a new dependency introduced a class of bugs, or the agent miscalibrated.

Step 5 - Build a Feedback Loop: Tag Findings as True/False Positives

Your agent only improves if it learns from feedback. Add a simple tagging system to your code review interface: engineers mark each finding TRUE POSITIVE or FALSE POSITIVE (or WONTFIX). Feed this signal back into your observability pipeline and correlate it against the original LLM prompt, rule, and confidence score.

After a month of tagged feedback, you'll have a dataset showing exactly which prompts, rules, and confidence thresholds lead to false positives. Use this to retune your filters, retrain your rules, or adjust LLM temperature. This closes the loop: observability → feedback → calibration → fewer false positives next week.

Without observability, you're optimizing blind. You see production fails but can't trace them back to the review decision. With observability wired in, every finding becomes a data point for improving the agent.

Next steps

Start small: instrument one rule in your pipeline, log LLM calls for a week, then analyze the pattern. Most teams find that 70% of false positives trace back to two or three preventable causes (overly aggressive confidence thresholds, missing deduplication logic, prompt ambiguity). Once you see the pattern, the fix is usually a one-line filter change. The expensive part isn't building observability—it's flying without it and discovering the problem at 2 AM in production.