Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations

by Nicole Forsgren

teams that build security into their work also do better at continuous delivery.

preapproved, easy-to-consume libraries, packages, toolchains, and processes

When the tools provided actually make life easier for the engineers who use them, they will adopt them of their own free will.

ARCHITECTS SHOULD FOCUS ON ENGINEERS AND OUTCOMES, NOT TOOLS OR TECHNOLOGIES

microservices or serverless architectures? Should

use

INTEGRATING INFOSEC INTO THE DELIVERY LIFECYCLE

testing, product management, and information security.

create win-win solutions in the pursuit of system-level goals, rather than throwing work over the wall and pointing fingers when things went wrong.

cites a ratio of 1 infosec person per 10 infrastructure people per 100 developers in large companies (Wickett 2014)—and they are usually only involved at the end of the software delivery lifecycle when it is often painful and expensive to make changes necessary to improve security. Furthermore, many developers are ignorant of common security risks, such as the OWASP Top 10,1 and how to prevent them. Our research shows that building security into software development not only improves delivery performance but also improves security quality. Organizations with high delivery performance spend significantly less time remediating security issues. SHIFTING LEFT ON SECURITY We found that when teams “shift left” on information security— that is, when they build it into the software delivery process instead of making it a separate phase that happens downstream of the development process—this positively impacts their ability to practice continuous delivery. This, in turn, positively impacts delivery performance. What does “shifting left” entail? First, security reviews are conducted for all major features, and this review process is performed in such a way that it doesn’t slow down the development process. How can we ensure that paying attention to security doesn’t reduce development throughput? This is the focus of the second aspect of this capability: information security should be integrated into the entire software delivery lifecycle from development

information security teams doing the security reviews themselves to giving the developers the means to build security

people building the software are doing the right thing than inspect nearly completed systems and features to find significant architectural problems and

don’t have the capacity to be doing security reviews when deplo...

found that high performers were spending 50% less time remediating security issues than low performers.

One is DevSecOps (coined by a few in the industry, including Topo Pal of Capital One and Shannon Lietz of Intuit).

am rugged and, more importantly, my code is rugged. I recognize that software has become a foundation of our modern world. I recognize the awesome responsibility that comes with this foundational role. I recognize that my code will be used in ways I cannot anticipate, in ways it was not designed, and for longer than it was ever intended. I recognize that my code will be attacked by talented and persistent adversaries who threaten our physical, economic, and national security.

recognize these things—and I choose to be rugged. I am rugged because I refuse to be a source of vulnerability or weakness. I am rugged because I assure my code will support its mission. I am rugged because my code can face these challenges and persist in spite of them. I am rugged, not because it is easy, but because it is necessary and I am up for the challenge (Corman et al. 2012).

MANAGEMENT PRACTICES FOR SOFTWARE

the project and program management paradigm,

Limiting work in progress (WIP), and using these limits to drive process improvement and increase throughput

Creating and maintaining visual displays showing key quality and productivity metrics and the current status of work (including defects), making these visual displays available to both engineers and leaders, and aligning these metrics with operational goals Using data from application performance and infrastructure monitoring tools to make business decisions on a daily basis

They are used to ensure that teams don’t become overburdened

asking teams whether they are good at limiting their WIP and have processes in place to do so.

teams use tools such as kanban or storyboards to organize their work.

IMPLEMENT A LIGHTWEIGHT CHANGE MANAGEMENT PROCESS

calling over another developer to review your code before pushing a change live.

All production changes must be approved by an external body (such as a manager or CAB). Only high-risk changes, such as database changes, require approval. We rely on peer review to manage changes. We have no change approval process.

negatively correlated with lead time, deployment frequency, and restore time, and had no correlation with change fail rate. In

This person should approve the change by recording their approval in a system of record such as GitHub (by approving the pull request) or a deployment pipeline tool (by approving a manual stage immediately following commit).

changes should only be applied to production using a fully automated process that forms part of a deployment pipeline.

committed to version control, validated by the standard build and test process, and then deployed through an automated process triggered through a deployment pipeline.

Every developer has made a seemingly innocuous change that took down part of the system.

This idea is a form of risk management theater: we check boxes so that when something goes wrong, we can say that at least we followed the process.

people following some of the common practices while failing to address wider organizational culture and processes.

example, in larger companies it’s still common to see months spent on budgeting, analysis, and requirements-gathering before work starts; to see work batched into big projects with infrequent releases; and for customer feedback to be treated as an afterthought.

testing your product’s design and business model by performing user res...

beginning, working in small batches, and evolving or “pivoting” products and the business models behind them early and often.

LEAN PRODUCT DEVELOPMENT PRACTICES

The extent to which teams slice up products and features into small batches that can be completed in less than a week and released frequently, including the use of MVPs (minimum viable products). Whether teams have a good understanding of the flow of work from the business all the way through to customers, and whether they have visibility into this flow, including the status of products and features. Whether organizations actively and regularly seek customer feedback and incorporate this feedback into the design of their products.

Whether development teams have the authority to create and change specifications as part of the development process without requiring approval.

significant in predicting higher software delivery performance and organizational performance, as well as improving organizational culture and decreasing burnout.

improve your ability to work in small batches and incorporate customer feedback along the way.

MVP is a prototype of a product with just enough features to enable validated learning about the product and its business model. Working in small batches enables short lead times and faster feedback loops.

gather user feedback quickly using techniques such as A/B testing.

feedback includes multiple practices: regularly collecting customer satisfaction metrics, actively seeking customer insights on the quality of products and features, and using this feedback to inform the design

Agile are nonetheless obliged to follow requirements created by different teams.

products that don’t actually delight and engage customers and won’t deliver the expected business results.

seek input from customers throughout the development process, including

informs the next stages of development.