Catching bugs before they hit production is the dream. What if AI could help?
We've always believed that the best deployment is one where you're confident in what you're shipping. That means solid code reviews. But let's be honest – thorough code reviews take time, and when you're a small team or a solo developer, it's easy for things to slip through.
That's why we've been keeping an eye on AI-assisted code review tools. We've been using CodeRabbit for over a year now, and recently came across an excellent write-up from the Lychee project about their experience with it. We wanted to share both their findings and our own take – because catching issues before you deploy is just as important as having a reliable deployment pipeline.
What is CodeRabbit?
CodeRabbit is an AI-powered code review tool that automatically analyses your pull requests and flags potential issues. It integrates directly with GitHub (and other Git providers), so every time you open a PR, it gets to work.
The setup is straightforward: log in with your GitHub account, authorise the bot on your repositories, and you're done. From that point on, new pull requests are automatically scanned and reviewed.
For open source projects, there's a free plan that works on public repositories – perfect for side projects or community-maintained tools.
How It Works in Practice
When you open a pull request, CodeRabbit does two things:
Summarises the PR – It edits your PR description to add a clear summary of what the changes do. This is particularly handy for large PRs where context helps reviewers understand the scope.
Reviews the code – After a few minutes of analysis, it adds comments flagging potential issues. You can then fix them, explain why they're not relevant, or dismiss them if they don't apply.
Once all comments are resolved, CodeRabbit automatically approves the PR. It's designed to complement human review, not replace it.
The Numbers: Is It Actually Useful?
The Lychee team did something we love – they tracked every single CodeRabbit comment over a month and categorised them. Here's what they found across 28 pull requests (32,784 lines added, 4,768 removed, 693 files changed):
Category Percentage
─────────────────────────────────────────────
Quality Improvements 35%
Nitpicking 21%
Useless/Noise 15%
Wrong Assumptions 13%
Thoughtful (made you reconsider) 13%
Security/Critical 3%
The takeaway? 72% of findings were relevant, and about half of those brought genuine value – either fixing real issues or making developers reconsider their approach.
That 3% security/critical finding rate might sound small, but when you consider those could be the bugs that cause real damage in production, it's significant.
Real Security Issues Caught
The Lychee team shared some examples of critical findings CodeRabbit caught:
- Zip Slip vulnerability – Extracting a zip file could have led to malicious file injection
- Cross-user data access – A user could delete another user's basket
- IDOR vulnerability – Users could access baskets belonging to other users
- Validation bypass – A user could exploit pricing logic to get unintended discounts
These aren't hypothetical issues – they're the kind of bugs that make it through manual review because reviewers are focused on functionality, not edge cases.
Our Experience at DeployHQ
We've been CodeRabbit customers for over a year now, so we can add our own perspective to Lychee's findings.
When we first started using it, the checks were more basic. Over time, the tool has matured significantly – the analysis is sharper, and the suggestions are more context-aware. It's been interesting to watch it evolve.
That said, the security checks can sometimes be overzealous. We've had cases where CodeRabbit flags something as a potential vulnerability when it's actually fine in our context. It's not a dealbreaker, but it does mean you need to stay engaged during PR reviews rather than blindly accepting or dismissing suggestions.
One feature we really like is the "Prompt for AI Agents" output. When CodeRabbit finishes its review, it generates a prompt you can feed directly into Claude Code or Codex. This makes it easy to action the suggestions automatically – you get the AI review, then hand it off to an AI coding assistant to implement the fixes. It's a nice workflow.
We've also started combining CodeRabbit with Claude Code and Codex for code reviews more broadly. Different tools catch different things, and having multiple perspectives (even if they're all AI) helps surface issues that a single tool might miss.
Where CodeRabbit really shines is when you're working with public APIs or specifications. It can actually check live documentation and validate your implementation against it. This was particularly useful when we were adding support for Strong Passwords as specified in the PCI DSS requirements – CodeRabbit could reference the actual specification and flag where our implementation didn't fully comply.
Limitations to Know About
The free plan has rate limits:
- 200 files reviewed per hour
- 3 back-to-back reviews, then 2 per hour
- 25 back-to-back messages, then 50 per hour
The practical advice? Commit frequently, but batch your pushes. Push once you have a few commits ready rather than pushing after every small change.
Where This Fits in Your Deployment Workflow
Here's how we see AI code review fitting into a solid deployment workflow:
Code → PR → AI Review → Human Review → Merge → DeployHQ → Production
↑ ↑
CodeRabbit catches DeployHQ handles
issues early reliable deployment
The earlier you catch issues, the cheaper they are to fix. By the time code reaches DeployHQ, you want to be confident it's ready for production. AI code review adds another layer of defence between your IDE and your live servers.
Should You Try It?
If you're:
- A solo developer who doesn't have teammates to review your PRs
- Part of a small team where review bandwidth is limited
- Maintaining open source projects with community contributions
- Simply looking to catch more issues before deployment
...then it's worth giving CodeRabbit a try. The free tier for open source projects removes the barrier to entry, and the integration is genuinely quick to set up.
The Bottom Line
AI code review isn't about replacing human judgement – it's about augmenting it. The tool catches the things humans miss when they're focused on the bigger picture: edge cases, security oversights, logic errors that only surface under specific conditions.
Combined with a reliable deployment pipeline, you've got a workflow where issues are caught early and deployments happen smoothly. That's the goal, isn't it?
The statistics and categorisation in this article come from the Lychee project's excellent month-long analysis of CodeRabbit. Check out their full write-up for more details and screenshots.
Related reading: