David Scott Bernstein's Blog, page 28

A Billion Here, a Billion There

In a 2002 National Institute of Standards and Technology report entitled “The Economic Impact as Inadequate Infrastructure for Software Testing,” software defects costs nearly $60 billion annually in the United States alone. This report is often misquoted assaying that software bugs cost $60 billion but software bugs likely cost industry and consumers much more than that but we have no way of knowing since many companies downplay their failures. What the report actually says is that *poor software development processes* cost the software development industry nearly $60 billion a year in wasted effort during the software development process.

This conclusion is supported by other studies as well. In the paper, “Software Project Failures Costs Billions: Better Estimation and Planning Can Help,” author Dan Gallerith estimated the cost of software project failures to be in the range of $50 billion to $80 billion a year. These are enormous losses and they are commonplace in the software industry. And all of these losses are preventable. If we just pay attention to building software more effectively we could significantly decrease these losses.

But as much as it costs to build software, and as much as we waste by doing it so inefficiently, the real losses for many companies are yet to come. It’s estimated that 80% of the cost of software happens after the initial release.

The next time your manager complains to you about how long it’s taking o build a feature and how much that costs the company, you can tell them that for most software, the vast majority of costs are in maintaining and not developing new code.

Fixing bugs costs about half as much as all initial development, and post-release enhancements cost almost twice as much as initial development. Why does it cost so much to fix and enhance software?

The short answer is that as an industry we do not value maintainability as much as we should and so we build products that are difficult to maintain. We operate under the assumption that just getting something out the door is good enough. Yet we’re slayed by these very assumptions when the burden of our technical debt gets in the way of us releasing new product.

But it doesn’t have to be this way. When we understand the nature of change and how to accommodate change through engineering, we can build software that costs less to construct and drastically less to maintain and enhance. This is the subject of my new book Beyond the Legacy Code: Nine Practices to Extend the Life (and Value) of Your Software, in which I present specific technical practices can help us build more maintainable software and how to apply them most effectively.

I have shown teams from the top software companies on the planet how to build more effectively using these technical practices. I can attest to the fact that they work, but they don’t work automatically. They work when they are understood and used correctly.

Very few other sources delve into why these practices ae effective. And just knowing what they are without understanding why they work isn’t enough for most people to use them effectively, grasping the motivations that underlie these practices is key, and that’s what Beyond Legacy Code focuses on.

You can find out more by going to http://BeyondLegacyCode.com.

First Principles

A first principle is a principle that is so fundamental that we can infer other principles from it. The Golden Rule is an example of a first principle, “Do unto others as you have them do unto you.” This is a first principle because many of our laws, our code of ethics, and even our culture are derived from this one simple statement.

So then what are the first principles in software development?

Software is so fundamentally different from everything else in the world we often have a hard time wrapping our brains around it.

In medicine, the first line of the Hippocratic Oath is a first principle. “First, do no harm.” Medicine requires intervention but the purpose and duty of a doctor is to heal her patient. She may have to amputate an infected leg but that’s in order to save a life. So doctors are constantly weighing the risks and potential benefits of treatment plans.

Back to software, what is its purpose? What is the goal of writing software? The answer to that question used to be to make the computer do something meaningful, but that answer isn’t good enough anymore. We have to do more than just make the computer do something. We have to build it in a way that can provide value now as well as into the future, which means software has to be built to be changeable and easy to work with in the future.

There are a whole litany of principles and practices around doing this correctly and many of these practices have only recently been discovered so they’re not all pervasive throughout the industry. But someday they will be.

Someday we will understand software development and we’ll know exactly what first principles to draw on, but that day hasn’t yet arrived.

Do This Because

Newton said we stand on the shoulders of giants and that’s certainly true for me. Many of the ideas that I have drawn on for *Beyond Legacy Code: Nine Practices to Extend the Life (and Value) of Your Software* came from a range of other sources both within the software industry and outside of it.

I’ve developed and tested these ideas over the last thirty years teaching classes in Agile software development. I have no shortage of material or ideas, but organizing them was the main challenge. I asked myself: “What if I only had a few hours to explain to the top executives of the world why these technical practices are valuable, and how to support their teams in doing work for them?” From that I came up with a set of seven and then nine practices I most wanted to share. I encourage readers to skip around and focus on what’s most important to them, but there is also a through line that makes reading the book from start to finish a satisfying experience.

I felt the practices offered great solutions, but I wanted to clearly state the problem first, which I have rarely seen done. So I wrote part one, which first covers my own personal experience of failures in the industry and then looks at how the industry as a whole is doing. I found that my experiences were pretty typical. I also introduce many of the concepts of Agile software development and talk about how it has been incorrectly or incompletely implemented in many organizations so they aren’t getting the results they hoped for.

It’s not enough to just do the practices, we also have to understand the principles that underlie them so we can use them effectively and get the most out of them. This is what I wanted to focus on in the book because I felt like other books don’t deal with these ideas in the depth that help people connect with *why* these practices are important. And without knowing the why, we can’t use them effectively.

I wrote the book in the first person, as narrative non-fiction, so that I could convey some of the stories and experiences that I’ve had in my career as well as offer some advice and analogies.

Basically, I wanted to convey the same energy and enthusiasm and information-rich conversations I have when I teach classes. We’re doing it in a different medium. It’s a very different experience and requires different considerations, but the goal is the same: Don’t just “do this,” “do this because.”

Be the SME

SME stands for Subject Matter Expert. This is a key role in the development of software. The level of understanding one must have to write software is very deep. Developers have to become SMEs when they work on a project.

In my career as a software developer I have learned tons of things in many fields from navigating boats to image processing.

Sometimes new developers ask questions that even subject matter experts don’t know the answer to, or perhaps they know intuitively but aren’t able to easily explain it. So much of the process of developing software is to take that implicit knowledge and make it explicit in the software we write. Even if it’s a situation that occurs one time in a million, we still have to be able to handle it or risk the computer being thrown into an error state. Sometimes the questions developers ask require a little bit of research on the part of the SME.

But developer expertise is different than regular subject matter experts. I have written a lot of software for financial markets over the years and was involved in stocks and mutual fund modelling for many years. The software I maintained provided value per investment advice. But personally I recognize that I am a poor investor, myself. Although I am an expert at modelling the stock market in software I am far from an expert investor.

While developers need to have a deep knowledge of the domain they’re modelling, we still need a product owner to answer questions and help us stay directed to build the most important features next. Product owners are like super SMEs. They have to have an intimate knowledge of the system being built. Although this is not necessarily a technical position, product owners have to thoroughly understand what the product is and how it will be used in order to direct the team to build the right things.

Developers must understand the domain they’re working in. Use domain language and operate from the domain model. We must think like domain experts and embody that thinking in the system we build. When we do this we can build a system that is not only useful but highly consistent. We can focus our attention around concepts in the domain model so that developers can use the same terminology and concepts that our users or subject matter experts are using. This helps increase the flow of information between developers and users and makes the system a working system.

In a sense, every system that is built is an expert system in that it embodies expert knowledge. That expert knowledge is often the core competitive edge that a company has, so making sure to be accurate and relevant is essential.

Development is informed by our knowledge of the domain. It helps us build a product that is more valuable to our users.

Crossing the Chasm

In his book Crossing the Chasm, Jeffrey A. Moore discusses what he calls the technology adoption life cycle for new products. He describes five distinct phases that innovative products go through based on the five different kinds of adoptors. The technology adoption life cycle applies not only to new products but also to anything new including new methodologies.

The first group to adopt these new approaches are called the innovators. Innovators like to discover new things. So much so that they put up with the inevitable kinks and bugs that come with innovation.

The next group are called the early adopters. These people are inspired by the success of innovators and see the potential in a new technology way before the mainstream. There are still challenges and issues to be worked out, but early adopters recognize the benefits of the new technology for themselves.

The next group to get involved are called the early majority. These people have waited until much of the kinks are worked out, so things are easier to work with. As the innovative technology penetrates the early majority, Moore refers to this as “crossing the chasm.” This is a sign that an innovation is on its way to becoming mainstream.

Now that that innovation is mainstream due to the assimilation and promotion of the early majority, the late majority gets involved. At this point the innovation is easy to use and commonplace.

Finally, the last group, called the laggards, gets involved only when there is no other alternative available.

Agile is crossing the chasm. It has gone beyond the early adopters and is penetrating into the early majority. Even after fifteen years, Agile practices have not fully crossed the chasm. In truth, only some of the easer Agile practices have actually crossed the chasm. Many of the more complex technical practices of Agile are just starting to hit early adopters.

As I say in my book, Beyond Legacy Code: Nine Practices to Extend the Life (and Value) of Your Software. “it’s not unusual for this continuous innovation to be adopted discontinuously.” However, much of the real value of doing Agile development comes from the technical practice of this.

In Beyond Legacy Code I focus on nine key Agile practices that provide the highest value for software development teams, the ones that allow us to rapidly build maintainable software. It’s not just the practices that are essential, but the purpose behind the practices is also critical for applying the practices correctly. If we understand the principals behind the practices we’ll help carry Agile fully across the chasm.

Discussing Metrics that Matter

Check out the video roundtable I was in yesterday: Continuous Discussion-Metrics that Matter: http://bit.ly/1WODPZJ (26 Jan 2016)

What is Waste?

A large part of the success of the lean manufacturing movement was its call to eliminate waste. Deming introduced the concept that waste in manufacturing came from excessive inventory. Inventory was considered waste because it tied up capital and required that it be kept in storage until it was ultimately sold.

But what is waste in software? Unlike physical goods, software has no mass so it doesn’t need to take up space, and it doesn’t wear out. So does lean manufacturing concepts really apply to the software construction process?

In software, waste is defined as any work that has begun but is not yet completed. We define work in progress as waste for two main reasons. First, like inventory, work in progress is unused while under development and therefore no value is being derived from it. It’s like unused inventory, it is not yet realizing positive cash flow.

There’s another reason that unfinished work is considered waste in software development, and that has to do with tasks switching. It can take a lot of effort to come up to speed on a particular task, and when developers are asked to work on multiple projects at the same time there is a required ramp up time where the developer is re-familiarizing herself with the task and is not yet productive. This ramp up time can be quite significant when developers are asked to juggle several projects. It’s far more efficient to take a project from start to finish as quickly as possible. Not only does this eliminate the ramp up time required on task switching but it also keeps us focused on delivering, as quickly as possible, value for the one feature we’re working on. Rapid delivery means rapid feedback, which means we’re not wasting our time working on tasks that aren’t needed.

Until a feature is coded, integrated, tested and delivered, it is not providing value to a user. The sooner a feature can provide value to a user the better.

Until a feature is tested and integrated into a system it is not considered done. All the work that sits in a queue waiting for quality assurance to test it is considered waste. Any change to a system that requires additional QA effort turns existing code into waste until it is tested.

The way to eliminate this waste is to automate the process of verifying release candidates. Test-first development allows developers to create an automated suite of tests to validate release candidates so when the system is changed it can be verified with a keystroke.

Why Nine Practices?

My new book, Beyond Legacy Code, Nine Practices to Extend the Life (and Value) of your Software is out. I guess one obvious question is: Why nine practices?

I wanted to organize the book around core practices because this would let readers jump around in the text and focus on the practices that are most important to them. So far in the feedback I’ve seen, this format works pretty well.

They say you can remember about seven things, plus or minus two, so nine is the most that most people can remember. Also, nine practices filled about as much room as I had in the book!

I always try to reduce knowledge to the smallest serving I can. There’s a lot developers have to keep in mind when building software, so I tried to focus on just the things that provide the highest value and from which we can infer other good ideas.

The nine practices that I call out in my book aren’t the most popular or well-known software development practices but they are the highest benefit practices. These are the ones that will make the biggest difference for software developers because after all, software development is a technical activity. The Scrum ceremonies like standup meetings and retrospectives are valuable, but in my opinion are not among the most essential practices for building maintainable software.

I’m happy that Scrum and Agile have become so popular, but companies are not adopting all of the Agile practices and are leaving out some very important aspects of Agile software development, so I wanted to set the record straight and discuss the key practices that are most valuable for building software.

I’ve provided an overview of these practices elsewhere so I won’t do that again here. I just want to point out that it’s the last five of the practices that are most important. Starting with CLEAN code, writing the test first, specifying behaviors with tests, implementing the design last, and refactoring legacy code are all technical practices. All of them require a great deal of discipline.

It’s true that in Agile we don’t do a lot of upfront design but that doesn’t mean that we don’t do design at all. In Agile, we do emergent design, which means we design as we go. Instead of devoting the first 20% of a project to design, we devote the entire project to design and then we implement that design little by little in code as we go.

This is not just a reorganization of priorities but a fundamental shift in how we build software. This shift engages developers to make the right decisions in the moment as we’re building code and the end result is a far better product than we could achieve any other way. It’s only through incremental construction that we discover the capabilities to make code extensible and flexible. It never comes up in a traditional Waterfall project when we build to releases, because we don’t return to our code very often, and we’re not incentivized to build code that’s maintainable. As a result, code quality stinks in traditional Waterfall projects.

If we build software with the same poor code quality in an Agile environment, we have still doomed our project to failure. People think Agile is an undisciplined shoot-from-the-hip kind of approach to building software, and that’s simply not true. In fact just the opposite is true. Agile requires a great deal of discipline and a strong focus on code quality. Instead of spending time on checks and balances, Agile developers focus on building quality into their code. But just removing the checks and balances without adding the focus on quality is a recipe for disaster.

You have been warned.

Why I Wrote Beyond Legacy Code

I wrote Beyond Legacy Code: Nine Practices to Extend the Life (and Value) of Your Software because I noticed two major disconnects in my experience as a developer. First, the more challenging technical practices such as test driven development and refactoring, were not being as quickly assimilated by teams as some of the simpler Agile practices like standup meetings and iterations.

This is understandable because many of the technical practices are more advanced and although they represent the highest value practices in Agile they are easy to get wrong. When we do get them wrong we suffer greatly. I also noticed that in many teams, management didn’t really understand or value the technical practices so developers were not really being empowered to use them.

Second, I wanted to demystify software development and make at least some of the ideas accessible to anyone. Do people outside the software industry really understand how software is created and maintained even though software affects all of us in profound ways?

While software is radically different than the world of physical things we all live in, many of the ideas we’ve been developing in the software industry have a great deal of value for everyday life, and for other disciplines like science and even art. Software is the essence of our new society and it’s at the core of the information revolution. So the ideas are important but very difficult for people without the background to access, and that’s because software is a world in and of itself.

One of the big challenges we face in our society both personally and collectively is the explosion of information that is coming at us all the time. It’s difficult to make rhyme or reason out of the barrage of data that constantly accosts us. Art, science, and the social sciences are also barraged with huge amounts of data. The reductionist aspects of science have helped us decompose ideas but at the same time it makes it very difficult for us to see the big picture.

These are not varying concepts to software development. One of the biggest challenges we face in software is managing complexity, so we’ve built many techniques and tools to help us with that. These ideas are either directly or indirectly transferrable to other areas of our lives.

I started playing with some of these concepts and discovered that I can explain them using some common terminology and some common sense.

I read a lot of books on software development, many of them are good but many of them aren’t. And even the good software books can be challenging to read, so I wanted to write a book that was more accessible and would actually be fun and engaging to read.

I also wanted to write a book that would change how people think about many of these technical practices and I know that in order to change people’s thinking and behavior, I’d need a lot more than just a series of facts. People learn through stories. They learn through anecdotes and metaphors and when we use these approaches, we often see people naturally taking up new behaviors. So instead of writing a technical treatise I decided to tell stories. The stories I tell come from my own experience working with thousands of software developers, and through these stories readers can see what works and what doesn’t.

Everything boils down to common sense, even the most technical practices are based on common sense and so I try to boil down the technical practices to things that anyone can relate to. I have a lifetime of experience in the industry, and I’ve seen ideas come and go. I spent much of the past thirty years being frustrated, but over the last fifteen years or so, I have seen great progress being made.

I have seen the practices that I discuss in my book both succeed and fail in the real world, and the key distinctions seem to be related to how well the purpose or context of these practices are understood by those who use them. So *Beyond Legacy Code* focuses on the reasoning behind the practices. There are principles that make them work, and we want to understand their purpose so we can apply them correctly.

I remember in the late 70s people talking about how thinking computers were just a few short years away. We don’t talk about that much anymore because we realized we have no idea how to program our computers to approach many of the problems we deal with as humans. If anything, the last few decades have taught us how *little* we know and if we want to take computers to the next level, to do really interesting things, then we have to build software more reliably. And that won’t happen with the “cowboy coder” mentality that pervades our industry.

Some people think that the practices of Extreme Programming (XP) are sort of a shoot from the hip approach to writing software but nothing can be farther from the truth. XP is a highly disciplined approach that requires a great deal of skill and focus. It’s not for beginners but when used correctly, it can be amazingly powerful way of building software.

These practices are not for everyone or for every project. They work especially well for challenging projects or pioneering software development where we’re building things we have never built before. The industry has a long way to go and we have been making great strides. Agile has gone from a nice idea to the standard in software development today. But Agile was started by software developers for software developers and the heart of Agile lies in technical practices that have to be well-understood.

This is why I wrote Beyond Legacy Code, to show that there are alternatives. The code we write is not inherently doomed to become legacy code that’s difficult to work with and hard to repurpose. If we pay attention to the forces involved in software development, to how things change and how we can adapt to change, then we can make our code far more adaptable. This will not only drop the cost of developing software but radically drop the cost of maintaining it.

To learn more, read excerpts, watch a video, and purchase Beyond Legacy Code, visit http://BeyondLegacyCode.com.

How to Define a Successful Software Product

People define success differently. This is not only true individually but we define the success of products or projects based on different criteria, too.

To the Standish Group, success is about predictability. They define the success of a software project as one that is delivered on time, on budget, and with all the features and function as originally specified. But this is not a meaningful definition of success. All it really measures is the project manager’s ability to guess how much a project will cost. For Standish it’s all about predictability but it excludes the possibility of coming up with a better idea along the way. Old definitions of success bind us to a particular implementation of success. If it doesn’t need that original vision in its entirety then it isn’t considered a success. But in Agile, our goal is to learn along the way so that we can come up with a better plan, a better execution than we had originally envisioned.

On the other hand, success cannot be solely defined by the success of a product in the marketplace. Of course, this is a huge part of what might be considered successful, but just having a financially successful product is by no means the only factor of true success. Consider Clipper, made by Nantucket Software, a database program that ran on desktop computers back in the day. It was wildly popular. The product made a fortune, but it was so full of technical debt and so difficult to work with, the developers couldn’t get out new releases fast enough and ultimately the company went under.

Code quality matters. And code quality is not the only thing that matters. If we don’t build the right things we’re done for. No matter how beautiful our code looks, if we don’t ultimately do what the user needs, we’ve wasted our rime. We must do the right thing!

But doing the right thing is not enough. We must also do the thing right and building the thing right means conforming to solid engineering principles and practices that support creating testable, maintainable software.

How do we know if we’re building the right things? Well, that’s easy. We can verify it with the customer. We can see how our software is used and measure the time the user had to wait until they received value from us. We can also measure if we built the thing right by looking at code metrics such as defect density and code quality, as well as technical debt.

Building the right thing and building the thing right are critical for success, but there are not equally viable in every context. In practice we often have to make compromises. How do you define successful software? Is your definition of success measurable?