An Independent Field Report on CodeRabbit AI: Is It Worth It for Your Team?

For the past few weeks, we have been using CodeRabbit AI at a client to support our code review. My interim verdict after extensive testing across dozens of pull requests: the tool’s value depends on how cleanly you work, and the alternative to CodeRabbit AI is, as is so often the case these days, your own workflow.

On paper, CodeRabbit’s offering looks very tempting. That is exactly why I had, until now, recommended it in my workshop on introducing AI into development teams: pay about €20 per developer per month and automatically get a generous number of pull request reviews in GitLab, GitHub, and the like. Configure what the AI should focus on and define your own rules. Let it learn from feedback and give it knowledge about your product so it can better judge the logic expressed in the code. The benchmarks look quite good, too: in one competitor’s report, CodeRabbit comes in second, right behind that competitor, with a bug detection rate that trails first place by only a few percentage points. So my concrete recommendation was: “The tool’s value surely depends on many factors, but you can definitely give it a try.”

Now I can say: I have tried it extensively myself, and through my long-term work with a client, I have been able to observe and shape the experiments. Here are the results:

First, the Good

1. The Reviews Are Fast

As soon as you open a pull request, you get a comment from the CodeRabbit bot letting you know that it is now reviewing the code. A few minutes later (depending on the size of the changes), the finished review is there. That speed lets the author stay in the context of the pull request for a few more minutes instead of jumping straight to the next topic. Since in software development we generally try to minimize context switches, that is a good thing. The time cost is low.

2. The Platform Integration Is Solid

CodeRabbit AI can comment, “resolve” comment threads, open separate issues, and, of course, respond to mentions. The rabbit even notices when one of its comments sparks a debate between two engineers, and then lets everyone involved know that it is “retreating into its burrow” while the humans talk it out. That is funny, and it earns a plus point for that. (Our daily work is generally drier than it needs to be.)

3. The AI Finds Real Bugs

An engineer on the team said: “It’s nitpicky, sure, but it generally makes good points and finds edge cases I missed.” (The nitpicking is a setting; more on that later.) I can confirm that impression. In my tests, what it found most reliably was messy state handling in frontend code, like a loading state that was not cleanly reset in an error case. So far, it has not been able to find any larger bugs.

As I see it, the bug detection rate depends mainly on two factors:

1. Knowledge about the product, and
2. how thorough the engineer is in the steps leading up to creating the pull request.

More on that later, too.

At this point, though, I want to say that I am convinced it would also catch glaring mistakes if they made it into the pull request.

4. A Few Quality-of-Life Features Improved My Daily Work

One example is the ability to copy a finding from the code review directly as a prompt for an AI agent. The prompt always includes a note to verify the finding against the code once more, rather than blindly applying the fix, which makes working through the findings with your local agent smooth and fairly worry-free.

Here Is Where the Problems Start

1. The Noise Is Sometimes Hard to Bear

CodeRabbit AI has two modes: relaxed and nitpicky. For various reasons, throughout the entire test phase we only tried the nitpicky mode, and in a large pull request that one will happily produce 20 comments or more. This is true not only when there really are a lot of problems in the new code, but also when the work was done cleanly and the code has perhaps already been through a review agent locally.

In that case, the AI suddenly starts hallucinating and invents rules that do not actually exist in the code. For example, it flags violations of linters that are not even defined in the project or—and this is worse—the bot finds critical problems in the business logic that, due to a lack of context, it simply cannot know about and therefore cannot judge. The reason for the many comments on clean PRs probably lies in the instructions the AI receives from CodeRabbit (the tool’s system prompt); after all, how would it look if a nitpicky reviewer found nothing in a larger PR? (The same mechanism, by the way, often applies to humans as well.)

2. The Quality of Life Is Limited

What I noted above as a plus point unfortunately has its limits: if the AI takes issue with something, and that “something” appears five times in the code, you get five disconnected comments. In that case, there is also no prompt you can simply copy that bundles those five cases together. CodeRabbit AI does mention when its finding also appears elsewhere in the same file, but not when the same finding was found in a different file. Instead, you get a separate comment in that other file.

Another example: sometimes a comment is perfectly valid but is not meant to be addressed within the same pull request anymore. In that case, as the author you of course say so; not least to be transparent toward human reviewers and teammates. CodeRabbit then kindly offers to create a separate issue for the problem—even when the platform it runs on is not used by the team for issue tracking at all. As far as I know, there is no setting to suppress this behavior.

3. The Learnings Feature Has Predictable Problems

A few months ago, I called Claude Code’s memory feature a burden. When the AI decides what it should learn and why, errors creep in. Just as between two people, information gets lost in the communication between human and machine. When I explain A to a person, what reaches them is A' or, worse still, B. The same goes for AI, and for CodeRabbit in particular: the AI model behind the bot is obviously not a GPT 5.6 Sol or Fable 5, and so CodeRabbit stores learnings that are too broadly generalized or simply wrong. Managing these within the tool is manual work and, on larger teams, can wipe out the efficiency gains from using the tool in the first place.

Now you could argue: “Then just turn off the learnings feature.” You can do that, but then you also no longer have a good way to cleanly store the few learnings that do matter. In our case, smaller rules and guidelines went into the AGENTS.md of the respective repository. That file should be kept as lean as possible. When my coding agents do not need the information, but the review AI does—because otherwise it does not produce good results—that defeats the purpose.

4. CodeRabbit’s UI Is Too Broken

Staying on the topic of learnings and configuration: in addition to the problems just mentioned, and alongside the learnings feature, CodeRabbit also has a “Coding Guidelines” feature. Here, too, the AI stores rules that are pulled, for example, from the AGENTS.md and saved separately within the tool. That is a design flaw, because the tool could just as easily read this information directly from the repository. That would have two key advantages:

1. As a maintainer, I could maintain the instructions in one place and would not have to make changes to them in an external tool on top of the repository.
2. The basis for the review would be exactly the same as for the agents that wrote the code.

As a user, I am supposed to be able to edit, extend, and delete these rules within CodeRabbit as well. At the time of writing this report, unfortunately, I simply cannot, because the UI for it has been broken for weeks. I cannot edit or delete anything, and the tool does not update the rules based on changes in the repository’s AGENTS.md either. The effect: in its reviews, CodeRabbit cheerfully references the outdated rules and favors them over newer rules in the current AGENTS.md that directly contradict the old ones. It is a sorry state of affairs and destroys any productivity gains.

At this point in my report, the question naturally comes up: is CodeRabbit AI even worth it at all? In my client’s case, and in the vast majority of other cases I am aware of, the answer is: no, not really. And the main reason follows now.

5. The Model Is Simply Not Good Enough

CodeRabbit’s usage limits are high and the price is astonishingly low: in the simplest plan, per developer per month you pay roughly as much as for a single “big” Claude Code Review, which Anthropic released as a separate flow a few months ago and priced at $15–25 per review (official approximate value; billing is based on API prices). That is exactly what makes trying it so appealing: if the tool is good enough, the investment is trivial. As someone who works on software that integrates LLMs myself, I know: you can only offer this low a price if your product is optimized for cost and not for quality. Long thinking processes and large context windows are simply expensive.

Reviewing pull requests is one of the most cognitively demanding tasks in our profession. It requires a lot of thinking, playing through different scenarios, and, in particular, reading the code beyond the boundaries of the changed spots: side effects can occur in places that are not visible in the diff as such, which requires a heightened degree of exploration from the reviewer—regardless of whether that reviewer is a human or an AI. After many weeks of intensive testing across different technologies, I can say with certainty: CodeRabbit AI simply cannot do this; at least not to a degree that would live up to the quality standards of the client and of many engineers I know.

A Better Alternative

At the start, I said: a better alternative to CodeRabbit is your own workflow. Specifically, I use a lean, self-written review skill that I have kept refining over the months and into which a quality-of-life feature has also flowed—one I saw for the first time in CodeRabbit. As you will see, the skill is not really sophisticated or complex; it simply relies on the good models that use it and gives me the output I need. That is all.

The review skill is the main reason CodeRabbit could barely find anything in the case of my own pull requests: by the time CodeRabbit got to see my code, a smarter AI had already thoroughly reviewed it and I had already applied a first round of fixes. On top of that, I had already gone over the diff myself and tested the changes locally.

At that point, an AI like CodeRabbit can hardly add any value anymore. But it does not have to, because the review skill is simple and can also run autonomously.

My old recommendation was to give CodeRabbit (or a comparable tool) a try. My new recommendation is: Use a review skill that fits your workflow, and use your existing AI subscription for it, for example from Claude Code or Codex. If you want to get fancy, you turn it into your own CI action for GitHub or GitLab; or you simply let it run locally in a new context window and grab a fresh cup of coffee in the meantime.

From there, the rule is the same as always: experiment, iterate, refine, until it fits.

What Does My Review Skill Look Like?

Here is the skill to copy out and try for yourself:

---
name: basic-review
description: Review the changes of this branch.
---

Review the changes of this branch. Assume `main` as the base branch unless specified otherwise.

## Steps

1. Explore the changes and surrounding codebase.
2. Explain the feature or bugfix to me in 3-4 sentences.
3. Review the changes:
    **Look for:**
    a. Programming bugs
    b. Completeness issues
    c. Code smells
    d. Bad practices
    e. Newly added generic utility functions that must **either** be moved to a shared location **or** replaced with an existing utility function in the codebase
4. Categorize your findings into the following severities:
    - **Blocker (B)**: Must be fixed before merge
    - **Major (M)**: Should be fixed before merge
    - **Minor (m)**: Can be fixed before merge or in a follow-up
    - **Trivial (t)**: Nitpicks
5. Return findings sorted by severity (blockers first), each with a referenceable label numbered within its severity (e.g. `B1`, `B2`, `M1`, `M2`, `M3`, `m1`, `m2`, `t1`).

## Output

Complete the full review before composing the output. Do not change, remove, or limit findings to fit the output format.

Start the response with the feature or bugfix explanation from step 2. Then output the findings in the two forms below.

### Individual findings

Put each finding in its own fenced `markdown` block:

```markdown
Before making changes, you MUST verify this finding against the current state of the code.

## M1: <finding title>

**Severity:** Major (M)

<finding text>

**References:**

- `<file and line reference>`

You MUST ignore work-in-progress changes on this branch that are unrelated to this finding. Other agents are likely fixing other findings on the same branch.
```

Use the finding's real label and full severity name. Keep the findings sorted as required by step 5.

### All findings

Below the individual finding blocks, put all findings in one fenced `markdown` block:

```markdown
Before making changes, you MUST verify each finding against the current state of the code.

## B1: <finding title>

**Severity:** Blocker (B)

<finding text>

**References:**

- `<file and line reference>`

## M1: <finding title>

**Severity:** Major (M)

<finding text>

**References:**

- `<file and line reference>`
```

For every finding, copy the label, title, severity line, finding text, and code references from its individual block. They must be exactly the same in both places. Do not rewrite or shorten them.

Two important notes:

1. The categorization of findings does not follow any official standard; here I simply use what I was taught in my first job. The categories are not rigidly defined, but they are good enough. They work.
2. The review agent’s output delivers each finding as its own Markdown block, as well as all findings bundled together once more as a separate Markdown block. That lets me decide whether I want to fix individual problems or all problems in one step—for example, when there are only a few or they are all valid.

How Do I Use a Skill Like This Effectively?

1. Different models carry out the instructions differently. Opus 5 and Fable 5 are considerably more thorough than Codex 5.6 Sol xhigh. Since the release of Opus 5, my review agent has been almost exclusively Opus 5, and I am genuinely thrilled with the output. If you only want to catch the most glaring blunders, you can certainly go with Codex 5.6 Sol xhigh as well. For this important step in our daily work, I explicitly do not recommend a smaller model.
2. If I only want to fix individual findings, I copy each one into a new session. This has the advantage that another smart model checks the finding again before it is fixed. If you fix the findings in the same session in which the review took place, the fix is subject to confirmation bias. I want to prevent that. A nice touch: the prompt for each finding also includes a note that work happening in parallel should be ignored; this makes it easy to run several sessions with individual fixes in parallel without any trouble.
3. If I want to fix all findings, I take the bundled prompt at the end of the output and drop that into a new session as well. Here, too, the same rule applies: each finding is checked again by the agent before it fixes it.

One objection I want to clear up: in my experience, it is no longer true that the review AI has to be a different one from the AI that produced the code. My Opus 5 sometimes finds blockers even in code that Opus 5 generated. The models surely have traits they cannot critically question in themselves, but the smarter they get, the less that matters. The AI has no knowledge that the same model produced the code. Nor does it know the prompt it was generated with. Claude Code could search its session logs for the session that produced the code, but it simply has no good reason to.

In my view, then, this perspective is “external enough” that in August 2026 you do not have to juggle two different AI agents to get effective local code reviews.

Closing Words

CodeRabbit AI (and similar code review tools) are tempting. Even after my extensive test, I would not actively stop anyone from trying it out for their own team. As always: my experiences are just that, my experiences, and your mileage may vary. But if the quality bar on your team is already high and you are all equipped with good AI models, it is worth sitting down together on a Friday afternoon and thinking through:

• “What problems do we have in our code review process, and how can an AI help us there?”
• “What would our optimal workflow look like, and can we build it ourselves?”

I am certain that a room full of smart minds will produce a solution that fits your team and your mission better than a generic tool ever could. If you need an outside perspective, I am happy to help. Just send me an email at hi@martinzoeller.com or connect with me on LinkedIn.