Lean Enterprise: How High Performance Organizations Innovate at Scale (Lean (O'Reilly))

by Jez Humble

The initial solutions we come up with are unlikely to be the best. Better solutions are discovered by creating, testing, and refining multiple options to discover what best solves the problem at hand.

Organizations can only move fast at scale when the people building the solutions have a deep understanding of both user needs and business strategy and come up with their own ideas.

A program-level backlog is not an effective way to drive these behaviors — it just reflects the almost irresistible human tendency to specify “the means of doing...

Goal-oriented requirements engineering has been in use for decades,3 but most people are still used to defining work in terms of features and benefits rather than measurable business and customer outcomes. The features-and-benefits approach plays to our natural bias towards coming up with solutions, and we have to think harder to specify the attributes that an acceptable solution will have instead.

Software development should always be a last resort, because of the cost and complexity of building and maintaining software.

The key to the experimental approach to product development is that we do no major new development work without first creating a hypothesis so we can determine if our work will deliver the expected value.

In the case of an internet-based service, we can use a powerful method called an online controlled experiment, or A/B test, to test a hypothesis. An A/B test is a randomized, controlled experiment to discover which of two possible versions of a web page produces better outcome.

new ideas before completely developing them, the chances are that about 2/3 of the work we are doing is of either zero or negative value to our customers — and certainly of negative value to our organization, since this work costs us in three ways.

Etsy is a website where people can sell handcrafted goods. Etsy uses A/B testing to validate all major new product ideas. In one example, a product owner noticed that searching for a particular type of item on somebody’s storefront comes up with zero results, and wanted to find out if a feature that shows similar items from somebody else’s storefront would increase revenue. To test the hypothesis, the team created a very simple implementation of the feature. They used a configuration file to determine what percentage of users will see the experiment.

Bezos hired West Point Academy graduate and ex-Army Ranger Rick Dalzell to enforce these rules. Bezos mandated another important change along with these rules: each service would be owned by a cross-functional team that would build and run the service throughout its lifecycle. As Werner Vogels, CTO of Amazon, says, “You build it, you run it.”4 This, along with the rule that all service interfaces are designed to be externalizable, has some important consequences. As Vogels points out, this way of organizing teams “brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. This customer feedback loop is essential for improving the quality of the service.”

Amazon did not replace their monolithic Obidos architecture in a “big bang” replacement program. Instead, they moved to a service-oriented architecture incrementally, while continuing to deliver new functionality, using a pattern known as the “strangler application.” As described by Martin Fowler, the pattern involves gradual replacement of a system by implementing new features in a new application that is loosely coupled to the existing system, porting existing functionality from the original application only where necessary.9 Over time, the old application is “strangled” — just like a tree enveloped by a tropical strangler fig (Figure 10-3).

The biggest mistake people make is porting existing features over as-is. This generally means reproducing complexity created to serve business processes as they looked years ago, which is enormously wasteful. Whenever you are asked to add a feature which represents a change to a business process, go and observe the process from scratch and look for ways to simplify it before implementing the code to support it. You will find that much accidental complexity in business processes actually comes from being forced to use the old software you are replacing!

Make the initial release of your new application small enough that you can get it deployed and providing value in a few weeks to a few months. When building the first module, it’s hard — but essential — to resist feature creep. The measure of success for the first release is how quickly you can do it, not how much functionality is in it. Typically, this is achieved by using the “vertical slice” approach in which we build small increments of functionality end-to-end across the whole technology stack.

If you find that you are expected to follow a process that compromises your ability to do a good job, it’s worth actually reaching out to the people who created the process to discuss its intent. Return to the Principle of Mission discussed in Chapter 1 and use it as an opportunity to collaborate, build relationships, and develop a shared understanding. You may be surprised to discover that you are able to have a productive conversation about how to meet their goals in a different way, or indeed to see if your work is even in scope for the law or regulation in question. If you are told that a particular process is “required” by some regulation, politely ask where you can find more information about that requirement. In many cases, onerous rules and GRC processes that are put in place are simply somebody’s interpretation of what is required, not mandated by the regulation in question.

“Trust, but verify”7 is a concept that is gaining acceptance in GRC circles. Instead of preventing teams from accessing environments and hardware so they can’t do anything bad, we trust people to do the right thing and give the team access and control on the systems and hardware they need to use daily. We then verify the team is not abusing their authority by developing good monitoring and frequent review processes to ensure the established boundaries are observed and there is complete visibility and transparency built into the team’s work.

Etsy’s most important architectural decision was to decouple the CDE environment from the rest of the system, limiting the scope of the PCI-DSS regulations to one segregated area and preventing them from “leaking” through to all their production systems. The systems that form the CDE are separated (and managed differently) from the rest of Etsy’s environments at the physical, network, source code, and logical infrastructure levels.

GRC structures and processes must be developed collaboratively by both GRC teams and the product teams that work day to day to deliver value to customers. By identifying the intent of the laws and regulations we must comply with, our GRC teams can collaborate with product teams to determine local approaches that fit best with improving value delivery. We start by exploring, with GRC teams, how we can minimize the negative effects of relying on restrictive controls through creative use of system architecture, process improvement, containment of scope, applying compensating controls, and leveraging new technologies. We can then exploit our learning to continuously improve our processes to provide both better governance and better outcomes for all stakeholders.

Having a budget is a good thing, especially when we have set some stretch financial targets for ourselves. Financial constraints can be a strong catalyst for creativity, collaboration, and innovation. Particularly in the explore domain, we can spur innovation if we purposefully reduce funding to localized areas or products and allow teams to decide how they can best utilize available funds, as we describe in Part II of this book. However, this approach will not work if we simply reduce funding and tell teams what their targets are and how to achieve them.

We still need to provide teams with clear definitions of what is off-limits, but the teams need to be trusted and given the chance to make decisions. As described in Implementing Beyond Budgeting, when European petrochemicals giant Borealis took this approach, they expected that costs would go up. Instead, they went down.4 Although Borealis was well positioned and prepared for the change with a culture that supported the move, CFO Bjarte Bogsnes attributes most of the outcome to better visibility into cost drivers through the use of activity-based accounting principles:5 those responsible for the activities that generate costs report on their finances, and teams assume responsibility for better management of costs.

This can require buy-in from executives: we know of one Fortune 500 company that gave bonuses to its VPs based on the number of services retired during the year, aiming to reduce system complexity and encourage innovation.

IT operations — a department within the IT department and perhaps the ultimate cost center — experiences the consequences of these decisions on a daily basis. In particular, the integrated systems they must keep running are incredibly complex and crufty, built up over years, and often fragile, so they tend to avoid changing them. Since stability is their first priority, IT operations has developed a reputation as the department that says “no” — an entirely rational response to the problems they face.

The practices most highly correlated with high IT performance (increasing both throughput and stability) are: Keeping systems configuration, application configuration, and application code in version control Logging and monitoring systems that produce failure alerts Developers breaking up large features into small, incremental changes that are merged into trunk daily (as discussed in Chapter 8) Developers and operations regularly achieving win/win outcomes when they interact

However, Westrum’s work on safety culture shows that no process or control can compensate for an environment in which people do not care about customer and organizational outcomes. Instead of creating controls to compensate for pathological cultures, the solution is to create a culture in which people take responsibility for the consequences of their actions — in particular, customer outcomes.

There is a simple but far-reaching prescription to enable this behavior: You build it, you run it. Teams that build new products and services must take responsibility for the operation and support of those services, at least until they are stable and the operation and support burden becomes predictable. By doing this, we also ensure that it is easy to measure the cost of running the service and the value it delivers. Turn central IT into a product development organization. The product development lifecycle and strategies described in this book should be used to deliver internal products and services as well as customer-facing ones. Invest in reducing the complexity of existing systems. Use the capacity gained from step 1 to invest in ongoing improvement work with the goal of reducing the cost and risk of making changes to existing services.

In Google, teams working on a new product must pass a “production readiness review” before they can send any services live. The product team is then responsible for its service when it initially goes live (similarly to ITIL’s concept of early life support). After a few months, when the service has stabilized, the product team can ask operations — called Google’s Site Reliability Engineers, or SREs — to take over the day-to-day running of the service, but not before it passes a “handover readiness review” to ensure the system is ready for handover. If the service encounters a serious problem after the handover, responsibility for supporting it is transferred back to the product team until they can pass another handover readiness review.6

In particular, centralized IT departments are responsible for: Providing clear and up-to-date documentation on which processes and approvals are necessary for new services to go live and on how teams can access them Monitoring lead time and other SLAs for these services, such as approving software packages, provisioning infrastructure (such as testing environments), and working to constantly reduce them For live services under active development, developers share equal responsibility with operations for:7 Responding to outages and being on call Designing and evolving monitoring and alerting systems, and the metrics they rely on Application configuration Architecture design and review

The correct way to address this problem is to allow product teams to use the tools and components they want, but to require them to take on the risks and costs of managing and operating the products and services they build — to repeat Amazon CTO Werner Vogels’ dictum, “You build it, you run it.” Recall the Lean definition of optimal performance from Chapter 7: “Delivering customer value in a way in which the organization incurs no unnecessary expense; the work flows without delays; the organization is 100 percent compliant with all local, state and federal laws; the organization meets all customer-defined requirements; and employees are safe and treated with respect. In other words, the work should be designed to eliminate delays, improve quality, and reduce unnecessary cost, effort, and frustration.”14 Processes inhibiting optimal performance should be a target for improvement.

When organizations begin customizing, it’s hard to stop — but customizations of COTS packages are extremely expensive to build and maintain over time. Once you get beyond a certain amount of customization, the original vendor will often no longer support the package. Upgrading customized packages is incredibly painful, and it’s hard to make changes quickly and safely to a customized COTS system. Instead, make changes to your business processes to match what the COTS packages can do out of the box. Any time you choose a solution based on COTS, you’ll have a list of features to implement or bugs to fix. These should always be treated as the input to a business process change management activity. Business process changes are much cheaper and more effective than changing COTS packages to match an existing process. If you have gone down the road of customization, take the next major release of your COTS package as an opportunity to migrate to a new, uncustomized version of the package, as Telstra (Australia’s biggest telco) did when it moved from a heavily customized install of Remedy to a completely vanilla one.

The next step is to understand and clarify our organization’s current situation. Participants in the strategic planning exercise should identify which problems need to be addressed and gather data to better understand each problem. Typically, even large organizations have limited capacity and can manage only a handful of initiatives at any one time; choosing what not to focus on and making sure the team sticks to its decision is critical. An economic framework such as Cost of Delay (see Chapter 7) is useful to stimulate discussion about prioritizing work.

A top-level strategy planning exercise can have a horizon of six months to a few years, depending on what is appropriate to your business. Review meetings should be held at least monthly where the team, along with the leaders of all teams that report to them, gather to monitor progress and update target conditions in response to what they discover. Teams at lower levels will typically work to a shorter horizon, with more frequent review meetings.