AI-Generated Code vs. Handwritten Code: What Code Reviews Reveal

Opus 5 finds more bugs in handwritten code than in AI-generated code.

Over the past few months, I’ve spent a lot of time thinking about the impact of generative AI on speed in software development. For the past few weeks, I’ve watched my focus shift from “speed” to “quality.”

What triggered this was intensive experimentation with AI-assisted code review. We tried CodeRabbit AI and found it wasn’t good enough, so we agreed to have Opus 5 review our code on our own machines for the time being. We noticed that Opus 5 reviews extremely thoroughly, always finds something, and only starts nitpicking once the bigger issues have been fixed. (It can’t just leave no comment at all. How would that look? That mindset existed before AI, too.)

The first insight was this: A local review with Opus 5 improves code quality. The second insight has been on my mind for weeks now: Opus 5 often finds more bugs in handwritten code than in code generated with Fable 5, Opus 5, or Sol 5.6. It doesn’t matter whether I wrote it, my peers did, or I did two years ago, when I still had genuine deep-work sessions more often.

Now you could argue: “AI favors the code it wrote.” I no longer think that’s true. In my setup, the AI doesn’t know that it wrote the code, and switching the generating AI (for example, from an Anthropic model to one from OpenAI) has no effect on the number or quality of the comments.

You could also argue: “Maybe you’re just not good, and neither are your colleagues.” As obvious as that burn is, it doesn’t hold up as an argument: Anyone who has read the news over the past few months has found plenty of evidence that AI may simply be good enough by now (or better): Take Claude Mythos, which found 271 security vulnerabilities in Firefox, or Linus Torvalds, who said: ”[…] no, AI isn’t perfect. But Christ, anybody who points to the problems at AI had better be looking in the mirror and pointing at themselves at the same time. Because it’s not like natural intelligence is always all that great either.”

Interestingly, this doesn’t mean that an engineer automatically gets better by having AI generate all their code. I had to witness a sad example of that just last week. What it means instead, I still don’t know even after thinking about it for a long time. All I know is that in the near future, we will no longer write code by hand, and teams that still reject AI-generated code have completely missed the memo.