After a year of trunk-based development, I’m giving up on pull requests and feature branching.
A year ago, my team needed to add a nontrivial functionality to production as fast as possible. Business as usual, nothing wrong with that. One thing that bothered me, though, and slowed us down was long pull request reviews (I’m talking about days).
You are “nearly” done with a functionality but you have to wait for the PR review. So you start on the next one and switch focus, and then you start receiving notifications from the code repository with comments on the work you’ve just moved away from. In reality, it gets even more complicated because you are also reviewing other team members’ PRs and have at least two more functionalities to finish.
Another thing that rubbed me the wrong way was that nontrivial refactoring, which would improve code and simplify further development, was hard or “impossible” to do. Since other people were working on the same code, refactoring would break their code badly.
So we decided to try trunk-based development. I worked that way most of my career, but we didn’t have code reviews unless you asked for one. The reviews were then done live without using any tools. And that worked well – but this time around, we decided to adopt a somewhat different approach.
Feature branching
In the words of Martin Fowler:
A feature branch is a source code branching pattern where a developer opens a branch when she starts working on a new feature. They do all the work on the feature on this branch and integrate the changes with the rest of the team when the feature is done.
Feature branching works best in zero-trust environments, like open-source, where merging is done after pull request reviews are done asynchronously and during long periods (days and weeks, or even longer).
When you have a team, or teams, however, working on a project together, feature branching creates problems. Team members should trust each other and be able to collaborate without using tolls for pull request reviews. Otherwise, that team has a bigger problem than their choice of a branching pattern.
When a developer uses feature branching, their work is isolated from the changes other team members are doing, and it may look like it is more productive that way because there are no distractions. But there is also no feedback or collaboration until the developer creates a pull request and needs to merge changes to the trunk (also called master and mainline).
Enter the merge hell
The feedback through pull request reviews is too often not valuable enough or not as good as it should be, especially if a change is significant. Reviewing hundreds or more changed lines is hard, and a review can mean the developer needs to rewrite everything after they had spent days or even a whole sprint on that feature.
Merge hell is a well-known problem, and with feature branching, you will certainly encounter it. It’s what happens when the code in your branch and the trunk diverges so much after other developers push their feature branches to the trunk that it becomes too complex to merge. Sometimes the simplest way to mitigate merge hell is to start over with development (without any guarantees you won’t end up in a new merge hell).
Refactoring also contributes to the creation of merge hell with branch development, since developers on the team make changes based on the code from the trunk, not the refactored code, which still needs to be merged into the trunk.
A better alternative: Trunk-based development
Trunk-based development, on the other hand, practices continuously merging code into the trunk and avoiding long-lived feature branches. (State of DevOps 2022 Report):
This means that developers push changes to the trunk at least daily, avoiding PRs, PR reviews, and merge hell problems. A 2018 book, Accelerate, explored and documented that trunk-based development as a better alternative to feature branching:
Our research also found that developing off trunk/master rather than on long-lived feature branches was correlated with higher delivery performance. Teams that did well had fewer than three active branches at any time, their branches had very short lifetimes (less than a day) before being merged into trunk and never had “code freeze” or stabilization periods. It’s worth re-emphasizing that these results are independent of team size, organization size, or industry.
That conclusion has been repeated every year since in the State of DevOps Reports.
It’s worth mentioning that trunk-based development is also a core practice in continuous integration. If developers don’t push changes to the trunk at least daily, then they are not doing continuous integration and are missing all the benefits of it.
Common reservations
Developers often think that pushing unfinished features to the trunk will cause problems in production since nobody wants incomplete features pushed to clients. Almost always, the solution is simple since deployment and feature release must be independent, using branch by abstraction or feature toggles, etc.
When I first used this practice we didn’t do a lot of code reviews, but that doesn’t mean it’s not an option. The best solution is pair or mob programming, but you can also do reviews after the changes are pushed to the trunk.
Refactoring can still cause problems for other developers, but this is mitigated if developers push changes often, multiple times a day (dozens times per day is OK).
To be successful with trunk-based development, and continuous integration, an application must have automated tests you trust. And to use branch by abstraction or feature toggles without creating more problems than fixing, the application’s code should be modular, loosely coupled, etc. (quality). But that is a requirement for all successful development, regardless of the way of work.
The biggest issue: Changing the mindset
When we started to use trunk-based development, it wasn’t without hesitation. What if my unfinished code breaks production? What does it mean to work in small batches?
It turned out the biggest obstacle was changing our mindset. Test-driven development helped us to trust our code and work in small batches (dozens of commits daily per developer is typical). Many new features start in test code and end up in production code very late, sometimes when a feature is finished. We rarely needed to use feature flags.
Besides unit tests, we had (and still do) integration tests covering most of the functionality. It’s not unusual that we develop a feature without starting the application even once. Automated tests will start the application, but there is no manual verification.
Additionaly, pair and mob programming helped us further trust our code before we pushed it to the trunk. Continuous review works great too. Post-merge reviews also work well, but I prefer pair and mob programming, since they yield better results and come with some other benefits.
Lately, we have also been exploring acceptance tests to further improve the quality of our code and enable faster development. There is some way to go yet, but we’re happy to explore it and share what we learned.
I will also be sharing our experiences with some of the topics mentioned above, like pair/mob programming, TDD, and acceptance tests, so stay tuned.
