Lessons from maintaining a GitHub Action and a GitHub App

For the last few months, it was possible to use Reviewpad as a GitHub Action or a GitHub App. In this post, we share our main lessons from maintaining both products and why we decided to fully focus on Reviewpad as a service through a GitHub app.

At the moment, it is possible to run Reviewpad as a GitHub action and a GitHub app.

Today, we are announced that the Reviewpad core team will stop updating the Reviewpad action.

If you are wondering why we are doing this and our solution, read on!

The problems with Reviewpad GitHub action

Running Reviewpad as a GitHub action has some advantages for users:

1. Control exactly on which events Reviewpad is triggered;
2. Control the execution environment; Reviewpad is deployed as a docker image.

However, it has some serious disadvantages:

1. For many Reviewpad users latency is super important. By running as a GitHub action with Github's runners, you can't really control when the action will be executed. You can't control the timing of the execution: it could be <10 seconds or >1 minute - this means that Reviewpad can't be used reliably to merge PRs on certain events or react to completion of other checks. Perhaps more seriously, we also have found that the stability of GitHub actions in the past year to be really shaky - check the number of incidents with GitHub actions in the past year.
2. Reviewpad's features require it to react to multiple events. This creates a high burden on users to understand how to configure the GitHub action to be able react to the appropriate events.
3. Billing is starting to be a concern to some users - most users are running Reviewpad ~10x per pull request. On a team with 50+ developers, the GitHub action minutes are eaten up fast.
4. On the subject of money, GitHub hasn't yet figure out a way to partner with providers of GitHub actions in terms of monetization - we believe it is not fair to work to increase GitHub's revenue without receiving any monetary benefit.
5. We are actively adding new features to Reviewpad that require permanent state. At the moment, the GitHub action is stateless - we can provide ways for users to connect to their DB or through our API but upon analyzing the burden to the user in terms of configuration found it to be unjustifiable.

Our solution: Reviewpad service as a GitHub app

For the past few months, we have found that moving to a service via the GitHub app solves all these problems:

1. The GitHub app reacts to webhooks and with parallelism at the PR level, Reviewpad typically starts within 250ms.
2. Reviewpad's service only requires the Reviewpad configuration file and users do not have to worry about the interplay between events and Reviewpad executions.
3. As a GitHub app, you will not have to pay anything else to GitHub.
4. We have a flexible pricing model and are willing to discuss particular conditions on a per case basis.
5. From an engineering perspective, we can focus on providing high-value features to users without major constraints.

For a while now, the core team has been both supporting the GitHub action and the app but that was not sustainable:

1. To do an excellent job in releasing both the action and the app, we need to test for each feature the behaviour in the action and the app. That takes time and focus away from providing value to users.
2. The documentation was becoming confusing because some built-ins can't properly work in the action.

Information to current users of the GitHub action

At the core of our mission is an ecosystem to provide the best developer experience when it comes pull requests.

For current users of the action, we will keep providing support on the action. The action is pointing to a Docker image which we will continue to host on Docker Hub for the unforseeable future. However, at this stage we do not plan to release bug fixes and actively recommend that you migrate to the Reviewpad GitHub App.

This migration only involves two steps:

1. Install the GitHub app;
2. Remove the Reviewpad action workflow file.

The GitHub app is compatible with the action and will pick up the Reviewpad configuration file at the root of the repository.

We will work with current users of the GitHub action to ensure that any gaps in functionality between the action and the app will be closed so that they can continue to benefit from Reviewpad.

To address any security concerns about the service, Reviewpad's core will not only continue to be open-source but we have started the process to obtain SOC 2 attestation for the entire service.

Feel free to drop your comment in this dedicated GitHub discussion.

The Reviewpad Team