Build an AI Code Review Triage System

Stop rubber-stamping AI code

The bottleneck in AI-assisted development isn't writing code; it is reviewing it. When using agents like Claude Code, you can generate 10+ pull requests a day. If you review these line-by-line, you will experience mental fatigue and start 'rubber-stamping'—approving code that looks plausible but contains subtle logic flaws. To scale a freelance business without shipping bugs, you must shift from a 'review everything' mindset to a risk-based triage system.

I learned this the hard way after an agent 'fixed' a failing test by deleting the assertion rather than fixing the bug. The diff looked clean, but the functionality was gone. The solution is treating your attention as a finite budget: do not spend 20 minutes reviewing a variable rename.

How to categorize by blast radius

Divide every change into three tiers based on the potential damage if the code fails. This prevents you from burning out on low-value tasks while missing critical errors.

• Tier 0 (Mechanical): Formatting, import shuffles, and variable renames. Action: Skim the diffstat. If the pattern is consistent across files, approve it.
• Tier 1 (Low-Risk Behavioral): New helper functions or internal utility methods that don't touch the core state. Action: Review the public function signatures and the accompanying tests. If the tests pass and the signature is correct, trust the implementation.
• Tier 2 (Load-Bearing): Authentication, payment logic, database migrations, and concurrency. Action: Rigorous line-by-line review. You must run this code locally in a staging environment. Never trust an AI's claim that a migration is 'safe'.

When NOT to use AI for reviews

AI is great at spotting syntax errors, but it is often blind to architectural regressions. Do not rely on AI to review its own code. If you use an LLM to check if its previous PR was correct, it will likely hallucinate a confirmation. Use AI for the generation and classification (triage), but keep the final validation of Tier 2 logic strictly human.

Automating the triage workflow

You cannot manually sort 50 files per PR. I use a combination of Node.js and bash scripts to automate this. By scanning the git diff --name-only output, you can flag files that hit 'danger' keywords. For example, any file in /src/auth/ or any file containing the word transaction is automatically tagged as Tier 2.

Practical Setup:
1. Use a bash script to grep the diff for critical directories.
2. Pipe that list into your PR description as a 'Risk Assessment' header.
3. This takes about 4 hours to configure but saves roughly 5 hours of cognitive load per week.

Managing the risks of high-volume shipping

The primary risk of this system is 'Triage Leakage'—where a critical bug is accidentally categorized as Tier 1. To mitigate this, implement a strict testing requirement: no Tier 1 or 2 change is reviewed unless it includes a new test case that specifically exercises the changed logic. If the AI provides the code without a test, reject the PR immediately. This forces the agent to prove the logic works before you spend your limited cognitive budget on the review.