What is stopping us from merging 30 pull requests per day?The hard parts about collaborating on code

Kiril VidelovMay 03, 2022

Reading time: 12 minutes

Most software projects are done by teams rather than individuals, so how we integrate work is critical. Collaborating on code is surprisingly challenging because any change made in isolation creates distance between developers. Shipping small and often is probably the most effective strategy for mitigating this. But just how frequently can we ship?

Programming is as close as we get to magic

I have been interested in computers since I was a kid. I remember playing Prince of Persia and wondering “What drives the pixels on the screen?” as well as “How is pressing → connected to forward movement in the game?”. At first, I was using my imagination and guessing how it worked. I was 11 and had a few wildly inaccurate ideas of what happens “inside” the computer.

Prince of Persia classic

Later on, learning about how computers actually work was one of the most exciting experiences of my childhood. I would compare the feeling to that of when somebody shows you how a magic trick works. A huge “Aha!” moment, followed by a feeling of power in actually understanding the trick.

I wanted to learn how to program these silicon rocks that we have tricked into “thinking”

Probably my top “Aha” moment was reading about how a CPU works, described in simple terms, starting with how it adds numbers together. From that moment, I knew that I wanted to learn how to program these silicon rocks that we have tricked into “thinking”. I still believe, even today, that this is as close as one gets to performing magic. For completeness, here is a really cool video explaining how a CPU works.

Years later, I had the opportunity to study software engineering at uni. I learned about data structures, the network stacks and protocols and even graphics pipelines. While I was getting more and more familiar with how software was “made”, there was one meta-problem which sitting unanswered for me…

Collaborating on code mystery

If large systems require teams of developers to create, how do developers effectively add their code to the same system?

I am not necessarily talking about version control, but rather the process of integrating logic and assumptions written by different programmers — making a system work as one whole.

I never got a satisfactory explanation to this mystery. At my first job as a programmer in a team of developers, I was introduced to a number of processes and buzzwords. Being fresh out of uni, I was eager to learn how it was done for real. Before long, however, I was convinced this only skirted around the real issue of collaborating on code. Let me illustrate with an example.

The company was building a financial system which consisted of a “Base product” and “Customization” layers, developed by different teams. Think of the “Base product” code as an upstream which gets forked and modified by different customization teams. Perhaps many of you can already spot the challenge.

A huge issue we experienced was that code in the “Base product” and the “Customizations” always diverged significantly over time, no matter how much people coordinated development. Keeping any of the customization codebases up to date with the “Base product” always meant dealing with not just merge conflicts syntactically, but also semantically. Some weeks, this overhead was more than 50% of the work effort.

Feature customization via forking

It is obvious, in retrospect, that the separation between “Base product” and “Customization” was an inappropriate one. We mitigated this challenge by increasing the frequency of integration. This was an extreme example of how code written by any given developer diverges over time from that of other developers. Furthermore, an example of the costs associated with it.

Writing code independently and in parallel creates distance between programmers — this was precisely the core of my unanswered question about collaborating on code.

Minimizing the distance between developers

Ship small and often. PS: Push your branch before lunch.

The best strategy for effective collaboration on software I have seen is one of minimizing the code distance between developers. There are two popular techniques for achieving this — trunk-based Development and Continuous Deployment. This is usually what people are referring to when they speak of “shipping small and often”.

Trunk-based development

Trunk-based development simply means avoiding prolonged development of features on branches in favor of merging small code contributions more frequently into the code “trunk”, often referred to as the “main”.

Instead of completing a feature before merging it a week (or several) later, it is hidden behind a feature flag but merged incrementally multiple times a day. This way, the code context is available to everybody on the team.

Trunk-based development

This is quite different from strategies like “Git Flow” which promotes working in relative isolation from the team and dealing with the complexity it entails later. I bet you know that feeling when you are asked to review a pull request with 500+ lines of changed code. Trunk-based development is practiced by small and large teams alike precisely because it encourages sharing small chunks of code that are easier to reason around.

Continuous Deployment

While the purpose of trunk-based development is to minimize the distance between developers, Continuous Deployment is about minimizing the distance between developers and their users.

Continuous Deployment is the practice of always having the latest version of the “trunk” code (main / master branch) deployed in the production environment. Knowing that the latest code is what the users experience dramatically simplifies the mental model of code and its operations.

(Safely) testing in production.

The reality of software development is that you don't really know anything until you test your code in a real environment. Embracing this fact means that you can plan for it. Deploying code continuously may sound scary, but it is known to minimize risks. This is because shipping smaller changes spreads out the large risk of large Big Bang deploys over time.

With smaller and more predictable deploys, teams get to invest in observability and monitoring of the software and really understand how changes they make in code affect users. By avoiding long-lived feature branches and working in isolation we are prompted to validate each other's assumptions early rather than after a week of development.

Here's something I've come to believe: Creators need an immediate connection to what they're creating. That's my principle.

This is a quote from a great talk by Bret Victor called "Inventing on Principle (video)". His point is largely about minimizing feedback loops, and this is also exactly what trunk-based development and Continuous Delivery are about.

Okay, if we know that increasing the integration frequency of our code is a good thing, how far can we take it?

How often can we ship?

Nowadays, both trunk-based development and Continuous Deployment are very popular with modern software teams. Most of us strive to “Keep pull requests small” and “ship small & often”. Over the past 4 years, the number of teams deploying multiple times per day has almost quadrupled.

What is stopping us from merging 30 pull request per day?

I have been wondering — given how successful the strategy of integrating code more frequently has been, can we take this even further? I mean, what is stopping us from merging 30 pull requests per day?

There are two parts to this. Firstly, there is a fixed overhead per code contribution. In addition to that, the traditional code review process is fairly heavy, as it was designed for a different workflow.

Fixed overhead per code contribution

Each code change, no matter how small, requires some logistics to be integrated. Let's consider what it takes to fix a typo in a documentation file:

  • Fetching the latest code from a remote git pull
  • Creating a branch for the code change git checkout -b feature/fix-typo
  • Fixing the typo
  • Staging the file with the relevant change git add myfile.py
  • Committing the code change git commit -m "Fix typo in documentation"
  • Pushing the code change to the remote git push --set-upstream origin feature/fix-typo
  • Creating a pull (merge) request in your git provider

Of course, there is a technical reason for the existence of all these steps. But consider the disconnect between the intent "I want to fix a typo in a documentation file" and the actual steps. Defaults matter, and this default discourages shipping small changes. This effort is negligible for features that span days or weeks, but it is quite significant if we strive to "ship small and often".

This friction means that the path of least resistance is to grow the scope of code contributions and delaying integration.

Traditional code reviews

Fixing a typo in the documentation does not carry the same risks as replacing an authentication middleware. So, why do we apply the same review process in both cases? Maintaining code quality and knowledge sharing are the two main reasons the software industry has adopted strict code reviews as a “best practice”. But as everything in engineering, there is an associated cost.

If I have to wait for review anyway, I might as well batch my typo fix together with this other thing I am building

The way formal code reviews are implemented today represents a hard blocking step, it contradicts the “shipping small and often” principle. This further skews the default behavior towards expanding the scope of code contributions.

It takes a deliberate effort to keep contributions small. Some of the best engineers I have met create stacks of pull requests but this certainly takes some skill and focus to pull off.

The question "How do developers contribute effectively their code to the same system?" has been on my mind since my days at uni. At this point, I have come to believe that the unit of code contribution is flawed, but never gets challenged because it is so ingrained.

The unit of contribution is flawed

Today, the term "pull request" is almost synonymous with the unit of contribution which is being reviewed and integrated. It's worth highlighting that this workflow originates from open-source, where more often than not, contributors don't know one another and there is no implicit trust. Teams coding at work, though, are quite different.

Consider how different the work dynamics are — the developers are colleagues who know each other, plan together and likely have a daily sync and common chat channels. Development at work also happens at a much higher pace than open-source projects — with the median lead time to merge a pull request being about half as long.

Median time until pull request merge
Source: GitHub

A friend of mine was recently telling me about the steps he takes to maintain a friendly collaboration vibe in the team. It boiled down to always having a discussion on Slack before submitting a pull request or leaving code feedback on another developer's PR. Somehow, in the pull request itself, the conversation was always a bit more tense and formal.

Too often submitting pull requests feels like submitting homework and reviewing them feels like being a judge in a district court. These are the defaults that we work with, and it is up to individual teams to work out a process that makes collaboration feel more friendly.

Perhaps this is why the community is starting fundamentally to question code reviews, with passionate  discussions  on Twitter. It appears that many teams chose to avoid this process altogether in favor of pair programming. After all, it fulfills the same objectives of quality gating and knowledge sharing.

Twitter discussions on code reviews

Does pair programming answer my question about effective collaboration on code?

Well, it feels like cheating the semantics of the question. It is a form of vertical scaling, where the pair of developers acts as an extra-smart-developer. If my code collaboration puzzle was a question about a race condition, this solution is the equivalent of making the application single threaded.

I would like to conclude with some thoughts and a hypothesis I have about making multithreaded code collaboration efficient and intuitive.

Code collaboration in teams via working in the open

Building on the strategy of integrating code contributions in small, incremental chunks is the idea of working in the open. This idea extends past the scope of what current generation development tools do (think Git platforms). There are three main elements to working in the open:

  • Work-in-progress code is discoverable — the team can see what everyone is working on.
  • Work-in-progress code is interactive — developers can easily try each other's code.
  • Feedback is early, proactive and asynchronous.

Practically speaking, this allows feedback and review steps to happen early in the process, rather than later on, when a lot of time and effort has been put in. This approach is similar to shift-left testing. Because good code discussions so timing and context dependent, an environment that provides discoverability of work-in-progress can allow the compounding of ideas.

Think of this as asynchronous pair programming.

Consider for a moment how tools like Figma and Google Docs have changed collaboration around design assets and text documents, respectively. During the same time period, the fundamentals of code collaborations have changed very little. Even if we view software development tools from a purely utilitarian point of view, it is clear that there is a mismatch between what they were designed to do and how they are used by teams today.

Tools are changing

These days, there are a number of hypotheses for the future of developer tools. From environments in the cloud [1][2][3] to pair programming tools [4][5] and browser IDEs [6].

New tools should augment the existing ecosystem

It is my belief that tools should be built and adapted after how people already work, and not vice versa.
With trunk-based development and Continuous deployment are at the core of modern software development, what would a tool specifically designed for this workflow look like? If you would like to read our take on this, check out the Sturdy Docs.

So, how does a team effectively contribute code to the same system? I have come to believe that it boils down to continuous, high-quality communication around the code, and tools play an important role in this.

Thanks for reading!
- Kiril