David Scott Bernstein's Blog, page 2

Identify Areas of Risk

A big part of managing a software development project is monitoring and managing the risks involved. Software development is a risky activity. Remember that throughout our tumultuous history in software development, according to the Standish Group, one-third of all software projects get canceled and never see the light of day. This was true in the 1994 CHAOS Report, the 2004 CHAOS Report, and every other Chaos Report published to date. This is true regardless of the methodology used, team size, and any other factor that we can think of because it doesn’t depend on development capabilities but instead is more reflective of the business environment that we live in.

Things change. That’s a fact. When I asked developers why projects that they had worked on were canceled, they gave me a variety of reasons. They tell me that the needs of the market changed, or a competitor came out with a similar product before they did, or the team was reorganized, or any number of other things.

Change happens. Change is inevitable. And because most teams cannot keep up with change, primarily due to technical debt, they cannot respond, and the project gets canceled. One-third of all projects get canceled in software development. One third! That represents a huge amount of waste. This is waste in Waterfall that is never addressed. This is waste in agile that isn’t addressed either.

So far, we have not been able to deal with or improve the cancellation rate of projects. It’s like the movie industry. Nine out of ten films lose money. But that tenth film makes so much more that it offsets the rest. The movie industry looks for blockbusters, or at least they did before COVID.

I’ve always been somewhat envious of the movie industry. Every film seems so completely different, just like a computer program seems different from every other one, but somehow the film industry has been able to standardize what they produce. They have well-defined roles, and while they can take dozens or even hundreds of people to make a movie they all seem to work together seamlessly.

Not so in software development. We don’t have standard ways of doing things, and it seems like we are constantly reinventing the wheel. I often find managers measure the wrong things, and they often get a false sense of security from monitoring the wrong data.

We can’t think about progress in a linear fashion when we do software development. It’s not about how many keystrokes you type or how many story points you complete if the quality of your work is poor. It takes experience and skill to know what ‘good enough’ means in software development.

We are not striving for perfection. We don’t want to overbuild. We want to do just enough so that our users can benefit from the features we create, but we don’t want to put too much effort into things that are not of high value to our users. Finding the balance for what’s good enough can be challenging, but it keeps us from overbuilding.

To my mind, the first step in managing risk is to identify it and then prioritize it, and then find ways to mitigate it. In software, risks come in many forms. There are risks in misunderstandings, risks in technical debt that can make it challenging to extend software, and risks in trying something new. Some of what we build has never been built before, so there are inherent risks in researching and discovering new ways to do things.

Managing a project that tries to do something genuinely new and innovative is different than managing traditional projects. Sometimes we have to take a lot of faith, but that’s what it takes to create something truly innovative and new.

The post Identify Areas of Risk appeared first on To Be Agile.

Invest in Automated Tests

I’ve built my career on writing software to automate other industries. I’ve learned a lot about a range of disciplines and businesses, how they operate, and very often their “secret sauce” can be embodied in the software that they write to run their businesses.

I believe that the quality of our build and the quality of our software, in general, have a huge dependency on the quality of our tests. In test first development the tests are our plan for implementing a behavior.

Writing tests well is a critical skill that I don’t see enough developers possess. Anyone can write a test but writing a good test requires a great deal of skill. It’s not just that a test is unique and fails for only one reason but a test also has to validate behaviors in an implementation-independent way so that we have the freedom in the future to refactor our code and not make our tests break.

This requires conceptualizing the software that we want to build at a higher level that makes it far more extensible. All of these magical things happen when we start to do test-first development but very often we don’t talk about them and so these benefits aren’t consciously recognized.

I find that when I do test-driven development well I am able to build features and validate them while still retaining the freedom of changing my designs in the future without breaking my tests. This makes my tests “open for extension” so that they support me when I refactor my code. Refactoring is when I want my tests to support me the most and it all comes from writing good implementation-independent tests.

But how do we do this? How do we get clear on writing the correct number and kind of unit tests for any behavior that we might want to create? The answer to that question is to think of tests and the assertions we create in our unit tests as a form of specifying the behavior we want our code to have. We call this “test as spec” and it is a way of thinking about our unit tests as a form of specifying the behavior we want to create.

I don’t put any code into my system without making a failing test pass so when I want to add code to my system I have to ask myself what test I need to write in order for me to have a reason to create the behavior to make this test pass.

So good tests not only support refactoring but also support helping me think about my features at the right level of abstraction. This allows me to encapsulate functionality, forge better method signatures, and generally produce higher-quality code. These are just some of the unspoken benefits of having well-written automated unit tests.

The post Invest in Automated Tests appeared first on To Be Agile.

Avoid Branching

I’m going to go out on a limb here (pun intended) and say that the purpose and benefit of continuous integration have to do with integrating our code continuously. That makes sense but I see so many teams use their CI systems to isolate their code in branches and prevent the code from being integrated with the rest of the system until the end of the project. We have a name for that. It’s called Waterfall.

Waterfall software development had many problems but perhaps the biggest one was integration. When we put off integration, there are more and more opportunities for things to go wrong and because problems can compound each other, it can be challenging to dig your way out. This is the danger of isolating code.

Git makes it easy and cheap to create branches. It’s a technological breakthrough but that doesn’t mean we should always use it. There are definite use cases for branches so I’m not saying we should never branch but I am saying that we should avoid long-lived branches because the goal of continuous integration is to integrate our code continuously.

I like to create experimental branches where I’m trying out an idea to see if it’ll work. These are short-lived branches that I integrate back into main very quickly, usually anywhere from a few hours to a few days. When I know what I want to build but I don’t yet want to expose it because it is incomplete then I used feature flags.

Perhaps the idea of feature flags is so simple that many developers are unaware of it but it is the recommended way of building a feature while integrating the code into the rest of the system. It’s a simple idea. Build the feature with the rest of the system but make it inaccessible to users while it’s being built. We can do that with a simple flag that we set for the feature while in development so it’s removed from the user interface but it still gets built with the rest of the system and so we can integrate while we are building.

Integrating code as we are building it is vitally important because the only way to have a true measure of our progress is when we integrate a feature within the rest of the system. Only when we integrate our code do we see if it will work in the context of everything else. This is why integration is so critical for reducing risks and giving us a true measure of our progress.

Computer systems are complex and the way to simplify integration is to do it all the time, do it when you only have one tiny little thing to integrate because if something goes wrong you’ll have a good idea of what that would be. As soon as we start to integrate multiple issues at a time the permutation and combination of things that could go wrong start to grow exponentially.

Developing software is risky. Not only is software development risky but it’s opaque to many management techniques because it’s not well understood. In order for something to be managed, it has to be seen and recognized. This is essentially what continuous integration does for us. It tells us the truth about the state of our system and so it’s an essential part of virtually every software development project.

In 2020, I finished a series of 72 blog posts that expanded on the first set of “Seven Strategies…” for each practice in my book Beyond Legacy Code: Nine Practices to Extend the Life (and Value) of Your Software. I included two sets of “Seven Strategies…” in my book and so I am expanding on the second set here. The next seven blog posts are expanded versions of “Seven Strategies for Burning Down Risk”. You can find the original post here.

 

The post Avoid Branching appeared first on To Be Agile.

Integrate Continuously

This is the first post in a new series of posts called Seven Strategies for Burning Down Risk based on the section with the same name in my book, Beyond Legacy Code: Nine Practices to Extend the Life (and Value) of Your Software.

Of all the agile technical practices the first and most important one for reducing risk and giving a true measure of our progress is continuous integration. The very first principle of the Agile Manifesto says “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”

We may not want to deliver continuously. When we release is a business decision based on business factors such as the seasonality of our products, support contracts, etc. However, we almost always want to practice continuous deliverability, meaning that we always want to have a buildable system. Only when a feature is integrated into the build shall it be considered done.

Continuous integration is the only way that we can get potentially shippable software at the end of a sprint. Most teams have code that they passed over to QA at the end of a sprint. If that’s the case for your team then I have news for you, you’re doing waterfall, in my opinion. As soon as we start to spread out the phases of development then we are no longer doing agile. In practice, it turns out that integrating the phases of software development is pretty straightforward, with the exception of QA.

Agile quality assurance looks very different than traditional quality assurance. Testing has to happen in tandem with development and in some cases even before development begins. How is that even possible? Well, remember software is soft. We can take advantage of the unique features of software to build into our code self-verifiability. This is essentially what test-first development does and we end up with a suite of unit tests that validate the behaviors of a system.

I find that the value of a continuous integration system depends on the value of the tests. When we write tests that directly validate the behaviors that we want to create then those tests can serve us later when we refactor our code. But if we write tests that have dependencies on implementation details or validate the wrong things then they might break when we refactor our code, becoming a burden rather than supporting us. This is why having the right tests is so vitally important when building software.

It all really starts and ends with the unit test. We want to test the smallest verifiable behaviors that we can identify in our systems. This is really not a testing activity but an automated verification process. We are automating the validation that our features work as expected. We are not trying to assure quality because that’s impossible. One can never assure quality in a complex system, we can only create quality. What we are trying to do with our unit tests is to demonstrate that our code works as intended.

But even more important than using our unit tests to validate behaviors we use them as scaffolding to help us form the behaviors at the right level of abstraction and for those of us who practice writing the test first, we find it an invaluable activity.

When teams are just starting out with agile I recommend that they begin with continuous integration. CI is a foundational practice and creates the infrastructure and context for all the other agile technical practices. All the software needed to set up a CI server is free and you can set it up on an old PC or even on a virtual machine. It runs in the background looking for changes in your version control system and when detected it invokes a build and then runs your tests.

The power and magic of CI come from your suite of unit tests. Like all medicine, your tests can either kill you or cure you depending upon how you use them. Knowing how to write good unit tests for behaviors allows you to map out and conceptualize services so that their testable and straightforward to work with. All these benefits come from integrating code continuously. This is why I see continuous integration as the very foundation of agile software development.

In 2020, I finished a series of 72 blog posts that expanded on the first set of “Seven Strategies…” for each practice in my book Beyond Legacy Code: Nine Practices to Extend the Life (and Value) of Your Software. I included two sets of “Seven Strategies…” in my book and so I am expanding on the second set here. The next seven blog posts are expanded versions from “Seven Strategies for Buring Down Risk”. You can find the original post here.

The post Integrate Continuously appeared first on To Be Agile.

Keep Stories Testable

I want to conclude this collection of seven blog posts on strategies for story splitting with one of the most important aspects that a story should include, which is the ability to be tested.

Just like testable code, testable stories are essential. When we make stories testable, we find that many of their other qualities also improve.

We start by asking what is the main value of the story and how will we know when we’ve achieved it. This helps keep us focused on the core of the features that we’re building and what it means to the user. This often helps clarify exactly what the feature is about, so we can write the story in a way that clearly expresses who it’s for, why they want it, and exactly what it is that the story is to achieve. This also helps us clearly define our acceptance criteria.

In fact, testability really reflects our story’s ability to verify its acceptance criteria. In other words, what we need to test in our story is if we achieve the desired intention. Another way to ask this question is, has our user achieved their goal or their “why” clause of the story? Keeping stories testable also helps keep us focused on achieving the acceptance criteria of a story.

Ultimately, the only thing that a computer can really do is change the state of some memory. If we are automating a task, then often what we need to do is identify the preconditions when we start and the post-conditions when we end. The difference in the state of the system from when we start to when we end, which is caused by the triggering event, is one way to recognize or define a story.

We can formalize this into a structure that we call the “given-when-then” clause. Given some precondition, when a trigger occurs, then some post-condition should be created that we can then use to compare to an expected post-condition.

Thinking of stories in this way allows us to easily test them and also to automate the process using an acceptance test-driven framework. These frameworks may take a bit of effort to get set up but once they are set up they can give clear visual feedback that all the stories are passing their acceptance criteria and if any fail they immediately generate an alert.

Acceptance tests are great ways to track the progress of story development and then to keep track that all stories are working as expected while new features are added to the system. And this gives us a true measure of our progress as we’re building a system.

This concludes my seven blog posts on “Seven Strategies for Splitting Stories.”

In 2020, I finished a series of 72 blog posts that expanded on the first set of “Seven Strategies…” for each practice in my book Beyond Legacy Code: Nine Practices to Extend the Life (and Value) of Your Software. I included two sets of “Seven Strategies…” in my book and so I am expanding on the second set here. The next seven blog posts are expanded versions from “Seven Strategies for Splitting Stories”. You can find the original post here.

 

The post Keep Stories Testable appeared first on To Be Agile.

Keep Intentions Singular

Throughout this series of blog posts on story splitting, I’ve stressed the need to make our stories as small as possible. The way to do this is to keep a story focused on a single outcome for a single type of user with a single purpose.

Keeping the intentions of a story singular helps us build focused code around that story that is well encapsulated and straightforward to maintain. It also helps us create well-defined acceptance criteria for a story.

Cohesion is an essential quality of object-oriented software. It means that each and every piece of code is focused on a single intention. Every class has a well-defined intention, and every method also has a well-defined intention.

If we start to add additional behaviors to a class that are part of a different intention, then we dilute the meaning of that class. It can become “schizophrenic,” having one set of behaviors and a completely different set of behaviors that draw on different instance data. These are clues that tell us that there are really two classes in that single class.

One of the classes is typically hidden in situations like this and when we have hidden classes in our domain model it becomes distorted and difficult to discern. Instead, we want to make our classes about fulfilling a single intention and make each method of that class about fulfilling some functional aspect of that single intention.

This is how we use the tools of object-oriented software to communicate and build accurate domain models that are maintainable and extensible. By keeping intentions in our system clear, we remove redundancy in our code so that there is a single place for everything, and nothing is repeated.

Cohesion and all good code quality start before the code is written or even designed. It starts by building a good story, a story that’s clear and focused on a single intention.

Every story should be about achieving something, and just like we want to make the classes in our system focused on a single intention, we want to make our stories also focused on a single intention.

]When stories are focused on a single intention, we find that they end up being far more independent from other stories in a system, so that there is less coupling between stories and that they can be built independently and more easily integrated into the system. Focus stories yield better code, they are more understandable and more straightforward to verify.

The post Keep Intentions Singular appeared first on To Be Agile.

Minimize Dependencies

In 2020, I finished a series of 72 blog posts that expanded on the first set of “Seven Strategies…” for each practice in my book Beyond Legacy Code: Nine Practices to Extend the Life (and Value) of Your Software. I included two sets of “Seven Strategies…” in my book and so I am expanding on the second set here. The next seven blog posts are expanded versions from “Seven Strategies for Splitting Stories”. You can find the original post here.

What discussion of story splitting would be complete without covering Bill Wake’s INVEST acronym? INVEST stands for:

The ‘I’ stands for independent. Independent stories should be stand-alone as much as possible. If we must have dependencies between stories, then a story should depend on a previous story rather than a future story.

‘N’ stands for negotiable because it’s really about the conversation around the story rather than the story statement itself.

‘V’ stands for valuable (or sometimes referred to as a vertical slice) because we want a story to provide ultimate value to a customer and be an end-to-end solution that goes vertically across slices to produce some unit of value. This really speaks to one of the most essential differences between building to stories rather than building to architectural layers.

In the past, I’ve built software in layers meaning that I would implement all the platform APIs first and then build my services on top of them, and finally put a user interface on top of that so that we could run the system. This is a horrible way to build software because you don’t really see everything running until late in the development cycle and we were constantly running into issues around integrating our code and making it all work nicely together. Instead of building horizontally, stories encourage us to build vertical slices of value, and this does a couple of things for us. First, it gets valuable features in the hands of our customers sooner but also it has us focus on implementing the most valuable things first.

‘E’ stands for estimable, which means that it’s easy to estimate. We get the ability to accurately estimate stories by making them as smallest as possible with clearly defined elements. Estimations are always more accurate when they’re on short horizons versus on longer horizons. For this reason, it makes sense to make stories as small as possible so that they’re more straightforward to estimate.

‘S’ stands for small. All stories should fit within the length of an iteration but ideally, story should be much smaller. Small stories are easier to schedule, and they also get verified sooner than large stories. In Agile, smaller is better, especially with stories.

And finally, ‘T’ stands for testable, which means that we can clearly define acceptance criteria for the story, and we use this to drive the creation of the story.

Bill Wake’s INVEST acronym is a handy way to remember six key aspects of well-written stories.

For me, the most important and the most difficult to achieve aspect of a good story is the first one, Independence. Often, I find that one story that I write depends on the existence of another story. This can be good if it’s allowing me to start with a simple story and then add from there but when multiple stories have interdependencies, it can be difficult to figure out what to do first.

As I mentioned, I used to spend a lot of time trying to figure out how to build architectural layers that would support the features I wanted to build but I prefer not to do this anymore and find that often if I just focus on building the most important features first, that I come up with a reasonably good order of construction and the technical practices that I use to build decoupled systems also helps me minimize dependencies. I find this approach works well for me most of the time.

The post Minimize Dependencies appeared first on To Be Agile.

Split Stories on Acceptance Criteria

In 2020, I finished a series of 72 blog posts that expanded on the first set of “Seven Strategies…” for each practice in my book Beyond Legacy Code: Nine Practices to Extend the Life (and Value) of Your Software. I included two sets of “Seven Strategies…” in my book and so I am expanding on the second set here. The next seven blog posts are expanded versions from “Seven Strategies for Splitting Stories”. You can find the original post here.

Many years ago, I asked my friend and Agile coach Darian Rashid how he likes to split stories. Without hesitation, he replied that he prefers to split stories on acceptance criteria. I thought this was a brilliant response.

What are ‘acceptance criteria?’ It is how you know that you’re done implementing the story. Sometimes people ask me at what level of abstraction or detail should a story be and my response is also at the same level of abstraction as the acceptance criteria.

Our acceptance criteria are the goals of our story. They’re what we’re trying to accomplish with the story and represent the main value of the story.

If there are multiple acceptance criteria, then it usually means that the user wants something that can be composed of other little things. If this is the case, then we can split these little things into separate stories, most typically. Remember, our goal is to make our stories as small as possible while still providing some value.

And each acceptance criterion corresponds or could correspond to its own acceptance test, which embodies that criterion in the form of an assertion.

There are several acceptance testing frameworks to automate the process of creating and running acceptance tests. Some frameworks are HTML-based such as FIT. Others are based on a more generalized language called Gerkin, such as SpecFlow or Cucumber. The purpose of these higher-level acceptance test languages is to allow the creation of acceptance tests that test acceptance criteria and not the implementation details associated with the acceptance criteria.

These higher-level acceptance test languages allow acceptance tests to be more durable so that when code is refactored, the test doesn’t break. Refactoring is when we want our tests to service us the most because that’s when we’re likely to make a mistake in code, so we want to write tests in such a way that they support us when we refactor code, rather than break. The way we do this is simply to test the behaviors for the end results that we’re trying to create rather than incorporating implementation details in our tests.

I know that’s easy to say and sometimes not so easy to do. We may realize that we need to reach into implementation somewhat to feel confident that we’re actually testing a feature. This is not uncommon. I’m really doing a bit of quality assurance in the process. That’s not bad but it helps to identify that we’re writing tests in a slightly different way because those kinds of tests might break when we refactor our code, so they may require additional maintenance but what they give us is an additional level of assurance that our code is working as we expect.

However, acceptance tests should have as few implementation details as possible. In fact, by their very definition, they should be free of implementation details. That’s what an acceptance test is. We’re testing that a system does something and not testing how the system does it.

I find that it’s highly useful to think about developing software based upon acceptance criteria. It helps me be more immersed in the domain that I’m building in and create systems that are better encapsulated. I’ve come to recognize that acceptance criteria are the main driving forces that I use for implementing the behaviors that my customers request.

The post Split Stories on Acceptance Criteria appeared first on To Be Agile.

Bonus: Free Chapters from my book, Beyond Legacy Code: Nine Practices to Extend the Life (and Value) of Your Software

 

Beyond Legacy Code: Nine Practices to Extend the Life (and Value) of Your Software is filled with practical, hands-on advice and a common-sense exploration of why technical practices such as refactoring and test-first development are critical to building maintainable software. You’ll discover how to avoid the pitfalls teams encounter when adopting these practices, and how to dramatically reduce the risk associated with building software–realizing significant savings in both the short and long term. With a deeper understanding of the principles behind the practices, you’ll build software that’s easier and less costly to maintain and extend.

Beyond Legacy Code: Nine Practices to Extend the Life (and Value) of Your Software is available from Amazon, Pragmatic Bookshelf, and other fine booksellers

Want to read sample chapters from my book?  Here are links to four sample chapters from my book, Beyond Legacy Code: Nine Practices to Extend the Life (and Value) of Your Software:

Introduction
The Nine Practices
Practice 1 – Say What, Why, and For Whom Before How
Practice 9 – Refactor Legacy Code

Enjoy!
David.

The post Bonus: Free Chapters from my book, Beyond Legacy Code: Nine Practices to Extend the Life (and Value) of Your Software appeared first on To Be Agile.

Iterate on Unknowns Until They are Understood

In 2020, I finished a series of 72 blog posts that expanded on the first set of “Seven Strategies…” for each practice in my book Beyond Legacy Code: Nine Practices to Extend the Life (and Value) of Your Software. I included two sets of “Seven Strategies…” in my book and so I am expanding on the second set here. The next seven blog posts are expanded versions from “Seven Strategies for Splitting Stories”. You can find the original post here.

Rather than trying to figure out everything upfront before we begin coding, the Agile approach is to take unknowns with us into iteration so that we can resolve them there.

Many times, we’re not even clear on what we don’t know until we begin coding so in many ways that’s the ideal time to research unknowns. I know this can make some developers uneasy. They’re used to having an analysis phase and getting clear requirements that tell them exactly what to do. But when we are building something that’s never been built before, there are bound to be questions that we didn’t think of that come up during development.

When issues come up for me during development and I can’t move forward without resolving them I stop and focus my attention on resolving the issue. This is when I am most familiar with what I need, and I look for ways to answer my question as quickly as possible.

My goal is to shrink the domain of the unknown so that it gets smaller and smaller until finally it disappears and is engulfed by the known. Once I understand the missing piece, I can then implement and integrate it into the rest of the system.

There are times that we do not know what we don’t know. I just spent several hours trying to ramp an audio frequency, only to find that there is an audio library that will do this for me, so I didn’t have to implement it myself. It’s part of HTML 5 so it’s guaranteed to work correctly on all (most) browsers. Finding this API save me an enormous amount of effort and time.

One of the first things I do with any unknown is that I try to encapsulate it. In other words, I try to draw a border around the area that’s unknown so that it’s separate and distinct from the things that I know. If I can encapsulate the unknown by hiding it and making it appear to the rest of the system as a generic service with no side effects, then I know I can deal with that unknown later, and when I do it won’t affect any of the existing components that I built. When I can encapsulate an issue by creating a strong contract with a well-defined interface, it means that that issue will not leak into the rest of my system. Most issues can be encapsulated in software, so I use this approach all the time.

There are times when an issue is so big and all-pervasive that it is not possible to fully encapsulate and when I encounter issues like that I stop what I’m doing immediately and focus all my effort on resolving it. Fortunately, I don’t encounter these issues frequently.

Moving into iteration with unknowns was a scary thought for me at the beginning of my career as an Agile developer. But I’ve gotten used to it and now I find that I can figure things out more readily in the middle of an iteration than I could in the planning cycle, so I spent far less time planning and far more time coding.

The post Iterate on Unknowns Until They are Understood appeared first on To Be Agile.