More on this book
Community
Kindle Notes & Highlights
Read between
October 16 - November 18, 2021
effective IT delivery organizations take about an hour to get code from “committed to mainline” to “running in production,” a journey lesser organizations take months to do.
In our search for measures of delivery performance that meet these criteria, we settled on four: delivery lead time, deployment frequency, time to restore service, and change fail rate. In this section, we’ll discuss why we picked these particular measures.
Lead time is the time it takes to go from a customer making a request to the request being satisfied.
We measured product delivery lead time as the time it takes to go from code committed to code successfully running in production,
your organization’s software delivery capability can in fact provide a competitive advantage to your business.
We can answer questions like, “Do change management boards actually improve delivery performance?” (Spoiler alert: they do not; they are negatively correlated with tempo and stability.)
’whenever there is fear, you get the wrong numbers’”
Generative (performance-oriented) organizations focus on the mission. How do we accomplish our goal? Everything is subordinated to good performance, to doing what we are supposed to do.
“who is on a team matters less than how the team members interact, structure their work, and view their contributions” (Google 2015). In other words, it all comes down to team dynamics.
Thus, accident investigations that stop at “human error” are not just bad but dangerous. Human error should, instead, be the start of the investigation. Our goal should be to discover how we could improve information flow so that people have better or more timely information, or to find better tools to help prevent catastrophic failures following apparently mundane operations.
Our research shows that Lean management, along with a set of other technical practices known collectively as continuous delivery (Humble and Farley 2010), do in fact impact culture, as shown in Figure 3.3.
You can act your way to a better culture by implementing these practices in technology organizations,
In contrast to Scrum, XP prescribes a number of technical practices such as test-driven development and continuous integration. Continuous Delivery (Humble and Farley 2010) also emphasizes the importance of these technical practices (combined with comprehensive configuration management) as an enabler of more frequent, higher-quality, and lower-risk software releases.
Many Agile adoptions have treated technical practices as secondary compared to the management and team practices that some Agile frameworks emphasize. Our research shows that technical practices play a vital role in achieving these outcomes.
In order to implement continuous delivery, we must create the following foundations:
Comprehensive configuration management. It should be possible to provision our environments and build, test, and deploy our software in a fully automated fashion purely from information stored in version control. Any change to environments or the software that runs on them should be applied using an automated process from version control.
Automated unit and acceptance tests should be run against every commit to version control to give developers fast feedback on their changes. Developers should be able to run all automated tests on their workstations in order to triage and fix defects. Testers should be performing exploratory testing continuously against the latest builds to come out of CI.
By giving developers the tools to detect problems when they occur, the time and resources to invest in their development, and the authority to fix problems straight away, we create an environment where developers accept responsibility for global outcomes such as quality and stability. This has a positive influence on the group interactions and activities of team members’ organizational environment and culture.
when the automated tests pass, teams are confident that their software is releasable. Furthermore, they are confident that test failures indicate a real defect. Too many test suites are flaky and unreliable, producing false positives and negatives—it’s worth investing ongoing effort into a suite that is reliable.
when developers are involved in creating and maintaining acceptance tests, there are two important effects. First, the code becomes more testable when developers write tests. This is one of the main reasons why test-driven development (TDD) is an important practice—it forces developers to create more testable designs. Second, when developers are responsible for the automated tests, they care more about them and will invest more effort into maintaining and fixing them.
None of this means that we should be getting rid of testers. Testers serve an essential role in the software delivery lifecycle, performing manual testing such as exploratory, usability, and acceptance testing, and helping to create and evolve suites of automated tests by working alongside developers.
Every commit should trigger a build of the software and running a set of fast, automated tests. Developers should get feedback from a more comprehensive suite of acceptance and performance tests every day. Furthermore, current builds should be available to testers for exploratory testing.
Continuous delivery improves both delivery performance and quality, and also helps improve culture and reduce burnout and deployment pain. However, implementing these practices often requires rethinking everything—from how teams work, to how they interact with each other, to what tools and processes they use. It also requires substantial investment in test and deployment automation, combined with relentless work to simplify systems architecture on an ongoing basis to ensure that this automation isn’t prohibitively expensive to create and maintain.
a critical obstacle to implementing continuous delivery is enterprise and application architecture.
employing the latest whizzy microservices architecture deployed on containers is no guarantee of higher performance if you ignore these characteristics.
We can do most of our testing without requiring an integrated environment.1 We can and do deploy or release our application independently of other applications/services it depends on.
Make large-scale changes to the design of their system without the permission of somebody outside the team Make large-scale changes to the design of their system without depending on other teams to make changes in their systems or creating significant work for other teams Complete their work without communicating and coordinating with people outside their team Deploy and release their product or service on demand, regardless of other services it depends upon Do most of their testing on demand, without requiring an integrated test environment Perform deployments during normal business hours
...more
In short, approval by an external body (such as a manager or CAB) simply doesn’t work to increase the stability of production systems, measured by the time to restore service and change fail rate. However, it certainly slows things down. It is, in fact, worse than having no change approval process at all.
What are the chances that an external body, not intimately familiar with the internals of a system, can review tens of thousands of lines of code change by potentially hundreds of engineers and accurately determine the impact on a complex production system? This idea is a form of risk management theater: we check boxes so that when something goes wrong, we can say that at least we followed the process. At best, this process only introduces time delays and handoffs.
We’re not proposing that you set your developers free to work on whatever ideas they like. To be effective, experimentation should be combined with the other capabilities we measure here: working in small batches, making the flow of work through the delivery process visible to everyone, and incorporating customer feedback into the design of products.
Before implementing the technical practices and discipline of continuous delivery on the Bing team, engineers reported work/life balance satisfaction scores of just 38%. After implementing these technical practices, the scores jumped to 75%. The difference is striking. It means the technical staff were better able to manage their professional duties during work hours, they didn’t have to do deployment processes manually, and they were able to keep the stresses of work at work.
Statistical analysis also revealed a high correlation between deployment pain and key outcomes: the more painful code deployments are, the poorer the IT performance, organizational performance, and organizational culture.
Second, the probability of a failed deployment rises substantially when manual changes must be made to production environments as part of the deployment process. Manual changes can easily lead to errors caused by typing, copy/paste mistakes, or poor or out-of-date documentation.
Furthermore, environments whose configuration is managed manually often deviate substantially from each other (a problem known as “configuration drift”), leading to significant amounts of work at deploy time as operators debug to understand configuration differences, potentially making further manual changes that add to the problem.
In order to reduce deployment pain, we should: Build systems that are designed to be deployed easily into multiple environments, can detect and tolerate failures in their environments, and can have various components of the system updated independently Ensure that the state of production systems can be reproduced (with the exception of production data) in an automated fashion from information in version control Build intelligence into the application and the platform so that the deployment process can be as simple as possible
Managers who want to avert employee burnout should concentrate their attention and efforts on: Fostering a respectful, supportive work environment that emphasizes learning from failures rather than blaming Communicating a strong sense of purpose Investing in employee development Asking employees what is preventing them from achieving their objectives and then fixing those things Giving employees time, space, and resources to experiment and learn Last but not least, employees must be given the authority to make decisions that affect their work and their jobs, particularly in areas where they
...more
A classic hallmark of burnout is indifference and cynicism, as well as feelings that your work is no longer helpful or effective.
the best thing you can do for your products, your company, and your people is institute a culture of experimentation and learning, and invest in the technical and management capabilities that enable it.
developers should be primarily responsible for creation and maintenance of automated test suites.
Effective practices include having adequate data to run your test suite, the ability to acquire necessary data on demand, the ability to condition your test data in your pipeline, and the data not limiting the amount of tests you can run. We do caution, however, that teams should minimize, whenever possible, the amount of test data needed to run automated tests.
software is in a deployable state throughout its lifecycle, and the team prioritizes keeping the software in a deployable state over working on new features. Fast feedback on the quality and deployability of the system is available to all team members, and when they get reports that the system isn’t deployable, fixes are made quickly. Finally, the system can be deployed to production or end users at any time, on demand.
