Automation in code reviews: dos and don’ts
Automation is key to increase the quality of code reviews. However, for some, there is a tendency to overestimate the power of automation and try to replace the human factor in code reviews. In this post, we explore what machines can do for us, and why they will never completely replace human interactions in code reviews.
Automation is a catalyst of progress and widespread adoption. Taking GitHub’s pull requests as an example, in March 2021, over 50% of the pull requests on public repositories were opened by bots like dependabot.
Used correctly in code reviews, automation saves you time, alerts you for integration issues, finds bugs, and shields you from some of the more tiresome parts of doing code reviews.
There are three aspects of code reviewing where automation can make the difference:
- Enforcing code style with linters;
- Find bugs with bug finders;
- Catch regressions with testing.
Linters
Linters have been around for a very long time. The first was developed in 1978 at Bell Labs by Stephen C. Johnson. He was debugging the Yacc grammar for C, and he constantly found himself hindered by portability issues. The basic idea remains the same to this day.
Linters perform static analysis, looking for bugs and discrepancies typically as patterns over abstract syntax trees. This is especially useful if you’re using a dynamically-typed language, like Python. The less strict the compiler’s rules, the more you need linting. Modern IDEs run linters in the background as you type, and will immediately flag the most common errors live, whether they’re logical or stylistic.
Bug finders
Code reviews are not (just) for finding bugs. There is absolutely nothing wrong with automating the search for them. Bug finders flag the most common kinds of bugs, and that will certainly avoid mistakes.
Having developed bug finders for more than 10 years, you should understand how they work to understand their results. The efficiency of bug finders is a topic of much debate - mostly because users of these analyzers don’t understand their limitations and capabilities. There's no harm in using them, but to trust them entirely is not something you should do.
Running tests
Automatically running tests as a part of code reviews can save you a lot of time. There are multiple benefits of automatically running tests: first you guarantee that the build is passing and second you can catch regressions. Good testing practices can prevent unnecessary work for code reviewers.
It might also be useful for you and your team to keep track of test coverage metrics during the review to discuss the overall code quality of the codebase.
Code reviews are about the code, but also about the people
A good code review is not about ticking boxes on a checklist. It's about understanding the code and perfecting it in a collaborative and continuous way. As we said in previous articles, the earlier you can start doing code reviews the more you will increase its quality.
A key part of the review is allowing a second party to be part of the creative process, understanding the code and what you are trying to do with it. This will allow for a greater understanding of the code from everyone involved, and it will allow for proper discussion and documentation of design choices.
Of course, this doesn’t mean you need a human to validate every little thing. Sometimes a green continuous integration pipeline is enough to merge a pull request.
Why automation should be done with care
It’s about culture.
Automation is a tool. It doesn’t have a connotation, it is as good or as bad as the use we make of it. As far as we’re concerned, automation is great for code reviews. It lowers the number of errors, saves time, and makes everyone more efficient and productive.
Issues will start arising if and when developers only have automatic tools in their toolbox. When all you have is a hammer, everything is a nail.
If you apply the Continuous Improvement model to software development, then you will likely want code reviews to be integrated into your team’s processes. By thinking of code review as a boring chore that should be passed on to a machine, we risk creating a stigma around code reviews. They can be perceived as tedious. They can be perceived as an overzealous demand that doesn’t truly improve the code.
It is very important that teams understand why we do code reviews, and the best way to demonstrate it is by doing it right, because then the results speak for themselves.
Tools are not perfect
A linter is just fine. You can trust it to do what it does because it does a basic thing. Security analyzers, for instance, aren’t. The number of false positives and false negatives they produce can kill your trust in the tool, and it would be tremendous irresponsibility to simply leave things the way the machine presents them to us with a shrug.
Codacy posted an article telling a story that perfectly encapsulates why testing and automated processes aren't sufficient and will never replace proper code reviews.
A large political party was developing a comprehensive solution that would be used for their campaign. They had a dedicated testing team that ran tests every day on everything they made.
When the time came to launch the platform… it failed.
Why?
Because it was a Sunday.
The tests only considered the weekdays. Due to some issues with 0-indexed arrays, the program didn’t run.
This is the type of error that humans during code reviews would catch.
And, in our opinion, a human should always get the chance to do so.
At Reviewpad we are constantly developing new features to mitigate these issues:
Reviewpad Protect - Protect pull request merges by applying rules to your repositories to block merges that break your code review policies, code quality, compliance, and security standards.
Reviewpad Check - Gain insights into the organization's pull requests, improve developer productivity, and prevent collaboration bottlenecks with insightful dashboards and alerts automatically generated for all your pull requests in all your repositories.
Reviewpad Automate - Automate tedious tasks with tailored workflows that streamline the Pull Request process, enabling faster scalability and hassle-free management.
Reviewpad AI Summary - Get a pull request description automatically generated and instant feedback on the code you are reviewing using our AI agent.
We are building a better future for pull requests and code reviews - share with us your experience and pains.
