Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation

by David Farley

You should never use long-lived, infrequently merged branches as the preferred means of managing the complexity of a large project.

Mainline development does not preclude branching. Rather, it means “that all ongoing development activities end up on a single codeline at some time”

Creating a branch for release replaces the evil practice of the code freeze, in which checking in to version control is entirely switched off for days and sometimes weeks.

It is important to remember that bugfixes must ultimately be merged back into trunk; in general, it is wise to do this immediately after a bugfix is committed to a branch.

Once you achieve a certain frequency of releases, around once a week or so, it no longer makes sense to branch for release.

Any changes from mainline must be merged onto every branch on a daily basis.

Branches must be short-lived, ideally less than a few days, never more than an iteration.

Nobody should start a new branch unless the branch representing their previous story is merged back to mainline.

Part of the technical lead’s role is to be responsible for keeping the trunk releasable. The tech lead should review all merges, perhaps in patch form. The tech lead has the right to reject patches that may potentially break the trunk.

It can work in larger projects, but only where the following conditions apply: The codebase is modular and well factored; the delivery team is split into several small teams, each with experienced leaders; the whole team is committed to checking in and integrating with mainline frequently; and the delivery team is not subject to undue pressure to release which might lead to suboptimal decision making.

developers branch to create features. This branch stays isolated for a long time. Meanwhile, other developers are creating other branches. When it comes close to release time, all the branches get merged into trunk. At this point, with a couple of weeks to go, the entire testing team that has been basically twiddling their thumbs finding the odd bug on trunk suddenly has a whole release worth of integration and system-level bugs to discover, as well as all the feature-level bugs which have not yet been found because nobody bothered to have the testers check the branches properly before they got integrated.

Unit and acceptance tests are run on every check-in on the branch. 5. All tests, including integration tests, are run on trunk every time a branch is merged into it.

The actual policy considers a branch stable if it can be merged into trunk without breaking any of the automated tests, including acceptance and regression tests.

There is a fundamental tension between the desire for continuous integration and the desire to branch. Every time you make a decision to branch in a CI-based development system, you compromise to some degree.

The longer the changes are kept separate from trunk—no matter what the technology is, or how sophisticated the merge tools are—the greater the risk that, when the eventual merge takes place, there will be a problem.

it is startlingly common to begin an IT program without a business case. That will likely lead to failure, because it is impossible to know what success looks like without a business case. You might as well be the Underpants Gnomes in South Park, whose strategy is 1. Collect underpants. 2. ? 3. Profit.

It is very hard to do requirements gathering, and impossible to objectively prioritize the requirements thus gathered, without a business case

Your software is always working, as demonstrated by an automated test suite including unit, component, and end-to-end acceptance tests that run every time you check in.

You deploy working software, at every iteration, into a production-like environment to showcase it to users (this is what makes the process incremental in addition to being iterative).

categorizes all risks in terms of their impact—how much damage they would cause if they materialize—and their likelihood—how likely they are to occur. These are combined to assess each risk’s severity. It is easiest to consider the impact in financial terms: how much money would be lost if the risk materializes? Then the likelihood can be modeled as a probability between 0 (impossible) and 1 (certain). Severity is then the impact multiplied by the probability, which gives you an estimate of the severity of the risk in terms of an amount of money. This allows you to make a very simple calculation when deciding what strategies to put in place to mitigate the risk: does the mitigation strategy cost more than the severity of the risk? If so, it’s probably not worth implementing.

Have regular project miniretrospectives after every showcase and get the team to brainstorm risks during this meeting.

Make risk identification part of your daily stand up meeting.

Automated scripts are the documentation of your processes that must work.