AI Code That Passes Tests Can Still Fail Review

CI green does not mean the patch is mergeable. When a coding agent has produced a change that compiles, passes the target tests, and closes the ticket, they have passed an important gate. The review still has questions to answer that tests do not answer: did the diff stay within the task, did it fit the abstractions already in the repo, and will the next developer find the code easier or harder to change?

Tests can behave myopically. In review, the team decides if they want to own the change.

Review is not merely a check

A test suite can demonstrate the patch builds and the expected assertions are true. It can't know if the implementation is safe to keep. FrontierCode tries to quantify this gap.

It's not just about raw pass rates. FrontierCode evaluates patches for behavioral correctness, regression safety, mechanical cleanliness, test correctness, scope, and code quality. If tests pass but something fails at blocker level, then score is zero.

This is what a real review looks like. One big problem can stop a merge.

For example, FrontierCode uses a LOG_WARNING() helper. The model did the right thing, but intermingled calls to LOG_WARNING() with direct std::cerr output. That works now maybe. But then again, if the project expects logging to go through the helper, it's still a bad patch. If later on, LOG_WARNING() gets mapped to structured logging or suppressed, the std::cerr calls diverge silently from the rest of the code base.

The code passes the immediate test, but does not fit into the system into which it is being merged.

Correctness implies scope

A benchmark that asks does the output just work can underestimate scope. FrontierCode has a scope criterion that considers file boundaries, diff size, line growth, and files touched. Extra work might be OK in isolation, but a patch that fixes a bug and refactors surrounding code adds review burden and regression risk.

Large diffs with AI are easy to make. Review makes them expensive again. Any extra line has to be understood and claimed.

Same point made in the FrontierCode Hacker News discussion. Developers liked the emphasis on short, mergeable patches, rather than chasing pass rates. Others also noted that code quality is a matter of taste. True. But review need not be all or nothing. Mergeability is specific to repository, conventions and maintainers.

A maintainer can reject a patch if it breaks the scope or testing standards of the project. This isn't bikeshedding. That's ownership.

The following change shows cost

A passing run is a moment in time. The code is still in development. Usually the price of a shortcut is paid when the next requirement appears.

SlopCodeBench is useful because it directly tests this pressure. It evaluates coding agents over 20 problems and 93 checkpoints requiring agents to adapt previous solutions to changing requirements. The benchmark is not structurally fixed, meaning the agent must make design decisions on the fly.

That's a common mode of failure. An agent can pass checkpoint N with a structure that makes checkpoint N+1 more difficult. The tests still pass, but the code is filled with awkward assumptions and shortcuts. People do that too. If review doesn't stop those shortcuts, AI just piles them higher.

Review today's shortcut before it becomes tomorrow's maintenance headache.

Generated tests require review

Generated tests are loved as much as generated code. FrontierCode follows a reverse-classical approach: the tests are executed on the original broken code and they should fail. If they reproduce the broken version, they have not captured the behaviour change. There is no test that passes before and after the fix to verify the fix.

This matters because agents often write code and tests at the same time. That can result in circular validation. Tests describe what the implementation does, not what the system does. The worst case is when the agent edits the test to match the code, instead of editing the code to match the intended behaviour.

The practical standard is straightforward. If possible, ask for a failing test on the code before the change. If that cannot be done, treat the test as the patch. Is this checking behavior, or just a reflection of the implementation?

Review requires sign off

SWE-PRBench evaluates eight frontier models on 350 pull requests annotated with human ground truth. Models found only a few of the issues that human reviewers found in a diff-only environment.

AI review can help with triage. It does not supplant the judgment which accepts a change.

Make sure tasks are small enough that a reviewer can understand the whole diff. Clearly state the intent and what has changed. Wherever possible, run new tests against code before change. Require the maintainer to make the merge decision explicit.

The generated code reduces the cost of patching. It doesn't make them cheaper to own.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com