The DevOps Handbook Quotes

“As has been proven time and again, the further the distance between the person doing the work (i.e., the change implementer) and the person deciding to do the work (i.e., the change authorizer), the worse the outcome.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“a small number of reliable, automated tests are almost always preferable over a large number of manual or unreliable automated tests.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“Humble and Farley define the difference between unit and acceptance testing as, “The aim of a unit test is to show that a single part of the application does what the programmer intends it to....The objective of acceptance tests is to prove that our application does what the customer meant it to, not that it works the way its programmers think it should.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“whether Ops used version control was a higher predictor for both IT performance and organizational performance than whether Dev used version control.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“The team quickly made a surprising discovery: only 50% of the source code in their development and test environments matched what was running in production.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“In order for a field or discipline to progress and mature, it needs to reach a point where it can thoughtfully reflect on its origins, seek out a diverse set of perspectives on those reflections, and place that synthesis into a context that is useful for how the community pictures the future.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“TRANSFORM LOCAL DISCOVERIES INTO GLOBAL IMPROVEMENTS When new learnings are discovered locally, there must also be some mechanism to enable the rest of the organization to use and benefit from that knowledge. In other words, when teams or individuals have experiences that create expertise, our goal is to convert that tacit knowledge (i.e., knowledge that is difficult to transfer to another person by means of writing it down or verbalizing) into explicit, codified knowledge, which becomes someone else’s expertise through practice. This ensures that when anyone else does similar work, they do so with the cumulative and collective experience of everyone in the organization who has ever done the same work. A remarkable example of turning local knowledge into global knowledge is the US Navy’s Nuclear Power Propulsion Program (also known as “NR” for “Naval Reactors”), which has over 5,700 reactor-years of operation without a single reactor-related casualty or escape of radiation. The NR is known for their intense commitment to scripted procedures and standardized work and the need for incident reports for any departure from procedure or normal operations to accumulate learnings, no matter how minor the failure signal—they constantly update procedures and system designs based on these learnings. The result is that when a new crew sets out to sea on their first deployment, they and their officers benefit from the collective knowledge of 5,700 accident-free reactor-years. Equally impressive is that their own experiences at sea will be added to this collective knowledge, helping future crews safely achieve their own missions.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“When these accidents affect our customers, we seek to understand why it happened. The root cause is often deemed to be human error, and the all too common management response is to “name, blame, and shame” the person who caused the problem.† And, either subtly or explicitly, management hints that the person guilty of committing the error will be punished. They then create more processes and approvals to prevent the error from happening again. Dr. Sidney Dekker, who codified some of the key elements of safety culture and coined the term just culture, wrote, “Responses to incidents and accidents that are seen as unjust can impede safety investigations, promote fear rather than mindfulness in people who do safety-critical work, make organizations more bureaucratic rather than more careful, and cultivate professional secrecy, evasion, and self-protection.” These issues are especially problematic in the technology value stream—our work is almost always performed within a complex system, and how management chooses to react to failures and accidents leads to a culture of fear, which then makes it unlikely that problems and failure signals are ever reported. The result is that problems remain hidden until a catastrophe occurs.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“Patrick Debois was not there, but was so excited by Allspaw and Hammond’s idea that he created the first DevOpsDays in Ghent, Belgium, (where he lived) in 2009. There the term “DevOps” was coined.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“Dan Milstein, one of the principal engineers at Hubspot, writes that he begins all blameless post-mortem meetings by saying, “We’re trying to prepare for a future where we’re as stupid as we are today.” In other words, it is not acceptable to have a countermeasure to merely “be more careful” or “be less stupid”— instead, we must design real countermeasures to prevent these errors from happening again.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“we value improvement of our daily work more than daily work itself.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“20% on detailed planning (Their poor throughput and high lead times were misattributed to faulty estimation, and so, hoping to get a better answer, they were asked to estimate the work in greater detail.)”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“Integration problems result in a significant amount of rework to get back into a deployable state, including conflicting changes that must be manually merged or merges that break our automated or manual tests, usually requiring multiple developers to successfully resolve.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“In other words, when we conflate deployment and release, it makes it difficult to create accountability for successful outcomes—decoupling these two activities allows us to empower Development and Operations to be responsible for the success of fast and frequent deployments, while enabling product owners to be responsible for the successful business outcomes of the release (i.e., was building and launching the feature worth our time).”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“To achieve market orientation, we won’t do a large, top-down reorganization, which often creates large amounts of disruption, fear, and paralysis. Instead, we will embed the functional engineers and skills (e.g., Ops, QA, Infosec) into each service team, or provide their capabilities to teams through automated self-service platforms that provide production-like environments, initiate automated tests, or perform deployments.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“Just prior to the production push, all developers with changes going out must be present and check in on their IRC chat channel—any developers not present have their changes automatically removed from the deployment package. Rossi continued, “If everything looks good and our test dashboards and canary tests† are green, we push the big red button and the entire Facebook.com server fleet gets the new code delivered. Within twenty minutes, thousands and thousands of machines are up on new code with no visible impact to the people using the site.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“The deal [between product owners and] engineering goes like this: Product management takes 20% of the team’s capacity right off the top and gives this to engineering to spend as they see fit. They might use it to rewrite, re-architect, or re-factor problematic parts of the code base...whatever they believe is necessary to avoid ever having to come to the team and say, ‘we need to stop and rewrite [all our code].’ If you’re in really bad shape today, you might need to make this 30% or even more of the resources. However, I get nervous when I find teams”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“Greenfield DevOps projects are often pilots to demonstrate feasibility of public or private clouds, piloting deployment automation, and similar tools. An example of a greenfield DevOps project is the Hosted LabVIEW product in 2009 at National Instruments, a thirty-year-old organization with five thousand employees and $1 billion in annual revenue. To bring this product to market quickly, a new team was created and allowed to operate outside of the existing IT processes and explore the use of public clouds. The initial team included an applications architect, a systems architect, two developers, a system automation developer, an operations lead, and two offshore operations staff. By using DevOps practices, they were able to deliver Hosted LabVIEW to market in half the time of their normal product introductions.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“Myth—DevOps Means Eliminating IT Operations, or “NoOps”: Many misinterpret DevOps as the complete elimination of the IT Operations function. However, this is rarely the case. While the nature of IT Operations work may change, it remains as important as ever. IT Operations collaborates far earlier in the software life cycle with Development, who continues to work with IT Operations long after the code has been deployed into production. Instead of IT Operations doing manual work that comes from work tickets, it enables developer productivity through APIs and self-serviced platforms that create environments, test and deploy code, monitor and display production telemetry, and so forth. By doing this, IT Operations become more like Development (as do QA and Infosec), engaged in product development, where the product is the platform that developers use to safely, quickly, and securely test, deploy, and run their IT services in production.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“Many psychologists assert that creating systems that cause feelings of powerlessness is one of the most damaging things we can do to fellow human beings—we deprive other people of their ability to control their own outcomes and even create a culture where people are afraid to do the right thing because of fear of punishment, failure, or jeopardizing their livelihood. This can create the condition of learned helplessness, where people become unwilling or unable to act in a way that avoids the same problem in the future.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“The latter pattern is what has become known as immutable infrastructure, where manual changes to the production environment are no longer allowed—the only way production changes can be made is to put the changes into version control and re-create the code and environments from scratch. By doing this, no variance is able to creep into production.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“In order to create fast and reliable flow from Dev to Ops, we must ensure that we always use production-like environments at every stage of the value stream. Furthermore, these environments must be created in an automated manner, ideally on demand from scripts and configuration information stored in version control and entirely self-serviced, without any manual work required from Operations. Our goal is to ensure that we can re-create the entire production environment based on what’s in version control.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“Without these self-service Operations platforms, the cloud is just Expensive Hosting 2.0.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“We will actively manage this technical debt by ensuring that we invest at least 20% of all Development and Operations cycles on refactoring, investing in automation work and architecture and non-functional requirements (NFRs, sometimes referred to as the “ilities”), such as maintainability, manageability, scalability, reliability, testability, deployability, and security. Figure 11: Invest 20% of cycles on those that create positive, user-invisible value (Source: “Machine Learning and Technical Debt with D. Sculley,” Software Engineering Daily podcast, November 17, 2015,”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“In the technology value stream, we optimize for downstream work centers by designing for operations, where operational non-functional requirements (e.g., architecture, performance, stability, testability, configurability, and security) are prioritized as highly as user features.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“Furthermore, integrating the objectives of QA and Operations into everyone’s daily work reduces firefighting, hardship, and toil, while making people more productive and increasing joy in the work we do. We not only improve outcomes, but our organization is better able to win in the marketplace.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“Kanban boards are an ideal tool to create visibility, and visibility is a key component in properly recognizing and integrating Ops work into all the relevant value streams. When we do this well, we achieve market-oriented outcomes, regardless of how we’ve drawn our organization charts.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“However, we must remind everyone that improvement of daily work is more important than daily work itself, and that all teams must have dedicated capacity for this (e.g., reserving 20% of all cycles for improvement work, scheduling one day per week or one week per month, etc.). Without doing this, the productivity of the team will almost certainly grind to a halt under the weight of its own technical and process debt.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“When we asked for permission, we were told no, but we did it anyway, because we knew we needed it.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

“Traditional managers will often object to hiring engineers with generalist skill sets, arguing that they are more expensive and that ‘I can hire two server administrators for every multi-skilled operations engineer.’” However, the business benefits of enabling faster flow are overwhelming. Furthermore, as Prugh notes, “[ I] nvesting in cross training is the right thing for [employees’] career growth, and makes everyone’s work more fun.”
― Gene Kim, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations