A finding is not real until a second model agrees

AI code review has a noise problem. A reviewer model will flag things that are not bugs, and a team that learns to ignore the bot has lost the point of it. MythosReview treats noise as the main defect to design out.

The verification pass

Nothing is posted to a pull request until a second model confirms it. The second model must come from a different family than the first. Using one family for both roles is a configuration error and the run posts nothing, because two copies of the same blind spot do not check each other.

Each candidate finding is either confirmed or dropped, and the reason is kept in the run report. A finding can also carry a prediction: the configured check command should fail, or pass, if the finding is real. When a check command is set, it runs once in the pull request’s working directory, and a finding whose prediction does not match the result is dropped.

Measured, not claimed

When a run carries labels saying which findings were real, the report gives the false-positive rate before and after verification. On the labelled fixture in the test suite, half of the model’s findings were phantom before verification and none were after.

That sentence is deliberately small. The fixture is a handful of findings built to exercise the mechanism. It proves the mechanism works and that the number can be computed. It is not a production false-positive rate. That figure will come from labelled history on real pull requests once the live product has some, and we will publish it with the method.

What a person still does

The agent summarises and comments. It does not merge, and chat on a pull request never changes code. A check run fails only when a verified P0 or P1 exists.

What to hold us to

If MythosReview ever posts a finding that was not verified, that is a bug in the product and we want the report.