Long-running branches — the price of fear
When developers fear they can’t go fast enough to meet a deadline, or they fear they will break the code by introducing bugs or by undesirably changing behaviour before a feature/change is fully ready, then they may opt for a long-running branch.
A long-running branch is a branch under active development that is not merged to the main trunk branch for a long time. The main trunk might be, for example, master or develop, from now on I’ll refer to this as master.
The long-running branch appears to deal with all the developers fears, they can go fast and can have partially complete features without any risk to master as they are in total isolation.
However, the problem with long-running branches is that changes on the master branch means that code on the long-running branch code begins to become incompatible.
This incompatibility manifests in one or more of three potential conflicts during/following the evental merge or rebase of the long-running branch.
- Merge conflicts. This is where the exact same area of code in a file has changed in both branches
- Test conflicts. Changes in
mastermight mean a new test written on the long-running branch against oldermastercode might no longer pass - Run time conflicts. Changes in
mastermight mean the long-running branch code exhibits a run time failure such as an exception, or unexpected behaviour. This is the most serious failure as it’s the least obvious. If you don’t manually test that area of the code you won’t catch it
Even if you intend to allocate 100% of the time 100% of the developers to the long-running branch, invariably changes will be eventually required on master and you risk seeing one or more of the three conflicts listed.
If you have a long-running branch and are trying to manage these potential conflicts, the smartest choice is to rebase the long-running branch frequently and deal with any conflicts then and there. A less optimal solution is to merge master into the long running branch and deal with conflicts. And the worst thing you can do is to ignore the problem and not deal with conflicts.
Rebasing
This is where the cost begins, even the smartest move of rebasing the long-running branch to the latest master leads to resolving conflicts. At this point your long-running branch might be 1 week, or a month old or more, and you’re being tasked with resolving conflicts in code you (or someone else) wrote that long ago. You’ll also have to communicate well with your team about the moving branch.
Over time, it becomes harder to make good decisions about resolving the conflicts and either you spend enough time researching the two sources of those conflicts, or mistakes creep in, or you spend time but ultimately move to the “merging from master“ option or “ignoring the problem” solutions.
Before you give up with this approach, you might consider squashing the history of the branch as rebasing one commit is easier as it minimises the number of merge conflicts. However, you will lose the history and having a week or more of work, from potentially multiple developers in one commit is far from ideal. It’s certainly not a long-term solution.
Merging from master to the long-running branch
Merging from master in to the long-running branch may be faster than rebasing as you are not faced with potential merge conflicts with each commit in the long running branch, but it’s still a source of mistakes and a time sink, and it cements the decision not to rebase which becomes far harder after this move.
On the positive side compared to rebasing you don’t need to communicate merges with the team in the same way you do with rebases.
A negative is the ugly git history you will be left with with many more commits in your long running branch that just document merges, not actual intentional code changes as part of the long-running branch.
Tipping point
However you solve the merge problems, all of this merging takes dev time, and that delays actual development on the long-running branch. Eventually your product owners become desperate for other features. So even if you started off with 100% of the time of 100% of the developers on the major feature, you eventually lose some resources to other work.
This means even more merge conflicts as they modify master more frequently and less developer time to finish the long-running branch and keep it in sync with master. The tipping point comes when you have to choose one or the other. Total stop on long-running branch development to enable synchronizing, or total stop of syncing to enable development to finish.
Product owners won’t be happy with any sudden loss of momentum so that increases the risk of a total stop of syncing with master being the chosen option, which puts you on the disastrous “ignoring the problem” path.
End game
The end game of any branch is to get merged into master so that it can be released.
When this day comes for the long-running branch, the fear of breaking master and stress that has been avoided from the previous months is concentrated in one giant merge operation. The review is near impossible. The potential for bugs from the frequent stream of merge conflicts is huge. Fear among devs and product owners (if they are even well informed) is at all time high.
At this point one of two things happen, you bite the bullet and merge, or the branch is abandoned and all the work goes to waste.
If you merge and it doesn’t go well, you might find yourself considering how to best resolve, do you: fix the issues now in master under great pressure (“fix-forward”), or do you roll back the giant commit, or even rewrite master to remove the merge. None of these are attractive propositions, so you can see how the pressure on a successful merge is high.
In extreme cases you can lose people, piling the stress on and/or making major costly mistakes leads to burnt-out developers moving on one way or the other.
If you’re in a long-running branch now, don’t fret, read the following preventative advice and then there is a special section with advice to get you back on track afterwards.
Solutions
As with most things, prevention is the best cure, and knowing the signs is the first step to prevention.
- If you’re using a ticket tracking system, you might notice a branch being created at an epic level rather than a story. Solution here is to not attempt to program to epics like this, only attempt to program to small achievable stories or tasks.
- A branch is created for a story ticket that does not have a good DOD, Definition Of Done. This can lead to feature creep, meaning the branch will be open for longer than intended. — Solution here is to not start a ticket with a fuzzy acceptance criteria, discover what is required and size the ticket as a team. When the size is too large (for example, not completable in one sprint) break it down in to sub-stories.
- A branch was not merged because it didn’t fully complete a story, and you’ve been pulled away on to another job, or maybe a vacation. As long as it is does no harm to master to do so, merge the branch and create a new story for the remaining work. When you eventually come back, you won’t need to worry about how out of date it’s got.
Those are some tips for solving at the planning level, you’re also going to have to tools to solve at the code level. How do you include partially complete pieces of work in the master branch?
The Open Closed Principle (OCP)
A software module should be Open for Extension but Closed for Modification.
This is a theory that we should be able to extend what’s there to modify behaviour without actually modifying source code. In practice there is no good way without a functioning crystal ball to know what will need extending and what will not. So the first step is usually to create an abstraction at an appropriate point which requires modification, but we leave the existing behaviour as the default. At this point tests should all run just fine and we can merge this to master. Then, in a separate piece of work, we can code an alternative behaviour. This is known as Branch-by-abstraction.
The alternative behaviour can be injected if we follow the Dependency Inversion Principle (DIP). You have free choice of how to make the decision when to inject one behaviour over another, you can hardcode it, switch based on Release/Debug status, a local configuration file, or even pull down settings from a server for A/B testing.
That’s right, while solving a pretty big issue, we’re actually gaining ability to quickly switch behaviours across the board, or for different audiences, and depending on the injection point, potentially at runtime.
The catch
What we’re doing is creating both “versions” of the software in the same branch, more software can mean more maintenance, but we don’t have more software than two branches, which can contain two versions of the same class. And unlike a separate branch we can avoid code change, and avoiding change minimizes maintenance.
Unlike two branches, the two versions of our class will never be reconciled back into one at some future merge point via a source control mechanism, this is a manual task that may need to be performed in the future. We should make time to remove old and no longer used classes and features.
This is not a hard task, but having the confidence to remove something or get the team to agree that a class will never be used again can be tricky. When classes may or may not be used based on a end users locally stored configuration, it can be even harder to find if they indeed used or not.
Unused features can deteriorate
Features that are not used in the wild may not continue to work in future. As code varies around them they may no longer be fit for purpose and by the time you turn them on, they may not work any longer. Again, avoiding code change minimizes this issue.
However, I would rather code that compiles and is just lacking a bit of integration testing than rotted code in a branch that doesn’t even compile if rebased against master. In any case, you certainly are not in a better position with an unused feature left in a branch.
If you’re already in a long-running branch today
My advice is to try to get merged with master sooner rather than later, but you can do this piecemeal;
- Rebase frequently if you haven’t been
- Identify harmless changes (i.e. not going to break master compilation or change run time behaviour) in the long-running branch that can be put in
mastertoday, this might be whole commits or the odd file here and there, get those inmasterand rebase again - Attempt to retrospectively apply the OCP/branch-by-abstraction advice above and by making more code harmless, you can repeat step 2
These steps reduces the differences and conflicts with the long-running branch and will reduce the risk/help you spot conflicts sooner. In theory you can repeat over time until there are no differences.
Further reading
For a worked example of OCP, DIP, branch-by-abstraction, see my article: Stop changing code behaviour
