David Scott Bernstein's Blog, page 16

Continuous Integration Requires the Right Tests

I see continuous integration as a “gateway practice” because it will lead to other Agile practices.

Continuous integration and automating the process of validating release candidates represents a large part of the Agile vision realized. We spend nearly half of our time and effort integrating and testing code. Many companies do this manually, which makes integration and last-minute changes exorbitantly expensive.

Automating the verification of release candidates offers a way to not only drop the costs of last minute changes but also significantly boots our confidence in the software that we build. A fast, reliable continuous integration server is at the heart of every successful Agile software development implementation that I know of.

But automation and continuous integration depend upon having the right tests. You can’t just write any tests to make continuous integration work and you can’t write the kind of tests that quality assurance writes in order to get good automated regression tests from doing test-first development.

Quality assurance is still an absolutely critical component of development that sits on top of the suite of unit tests the developers build when writing their systems using test-first techniques. When we do QA we think about what might go wrong but when we do test-first development we have a completely different mindset and focus instead on how we create tests that elicit the behavior we want to build. This is a very different way of thinking about tests and testing than the QA mindset.

Testing seems like an easy and obvious thing to do but what we do when we’re doing test-first development is not testing. We’re defining behaviors using tests and this is a very different way of thinking about using tests.

So what do I mean by the “right tests?” First of all, tests must be unique. If you think about writing specifications, you wouldn’t repeat the same sentence over and over in the specifications document, so why would you repeat the same tests over and over in your code? If our tests are to have value and allow us to more easily refactor our code safely then we surely want to make our tests unique so that if something changes in our code we’re not changing multiple tests.

Unit tests should fail for a single reason. When I see a test fail in my unit test runner I wanted it to tell me exactly where the failure happened. If my test covers too many things then when it fails I won’t know the reason that it failed without further investigation. Therefore, when something fails I want only one or a few tests to fail rather than a whole slew of tests failing because this will help me identify where the problem is more quickly.

Making only one test fail for a single issue can be challenging. Sometimes issues are interdependent. For example, if I can’t get access to data that I need in a database then my data access query would fail but all of the other tests that depend on that data would also fail. In this scenario, several tests are failing however there is only one point of failure, which is the first test failure that can’t retrieve the data from the database. The other tests that depend on that data are not telling us about other problems in the system but rather they fail because of the true dependency in the system and so when we analyze these failures we look at the first failure and recognize the other failures are due to natural dependencies on the first failure.

Most importantly unit tests are implementation-independent. By that I mean we can refactor our code and without changing the behavior, none of our unit tests should break. I’ll talk more about implementation independence in my next post.

Avoid Long-Lived Branches

If you ask a hundred consultants what Agile is, you’ll likely get 100 different answers.

So, here’s my answer. Agile software development can be summed up by the first principle of the Agile Manifesto, which states, “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”

To my mind, that statement says it all. In order to provide early and continuous delivery of valuable software, we must abandon the Waterfall process. Instead of building to a release, we must build to a feature.

This means that instead of doing all the design up front and then code and then test, we identify a high priority feature and build it from start to finish. We write the code but instead of letting it sit there as we build other features, we take that feature and integrate it into our system immediately so that we can see it interacting with the other features of the system to get a true measure of our progress. This is what continuous integration means, to continually integrate features as we build them.

I think that continuous integration is the most valuable and important process for Agile software development because it provides the context and infrastructure for all the other Agile technical practices. It is the central part of a healthy Agile software development process.

But continuous integration is a discipline that many teams are doing wrong. The purpose of continuous integration is to continually integrate code as it is written into the system. Integrating new code means that the new code is then is part of the build with the rest of the code in the system. That’s what “integration” means.

Unfortunately, I see a lot of teams use continuous integration tools but are not doing continuous integration. Instead, they create a system where every team member builds features in their own branch of the version control system and then, just before release, they take those branches and integrate them together. This is called Waterfall software development!

If developers build features in their own feature branches, then they are putting off integration just like we did in the Waterfall process. You may think I’m nitpicking or describing edge cases but many teams I see are stuck in this trap. Even GitFlow, Git’s recommended workflow, advises setting up continuous integration with long-lived feature branches, which I advise against because it defeats the whole purpose of continuous integration. I strongly recommend GitHubFlow over GitFlow for designing a true continuous integration system with Git.

But then how you integrate features that are still being worked on while they are being built? The answer is to use feature flags.

Feature flags, which are sometimes called feature toggles, are a really simple concept that can be implemented in several different ways. One easy way to implement feature flags is with conditional statements. If we’re still working on a feature that is not ready to be consumed by the customer, we simply hide it behind a conditional statement that makes the feature inaccessible to the user interface. That way, users cannot access the feature and they don’t see it but it’s still part of the build.

Having the feature still be part of the build is critical because it can still be compiled, and we can still run it and test against it, and if there are errors or issues with the feature then we can often find them quickly. All of our development tools still work, and we can see how the feature interacts with the rest of the system and when were satisfied that it works correctly, we can flip the feature flags that make the feature active in the user interface so that users can start to work with it.

Feature flags help isolate features, just like branching. But unlike branching, feature flags allow the code to be integrated into the common code base which means that we are really doing continuous integration and getting its benefits by continually integrating our code without branching.

What’s Agile Software Development?

For many complex processes, there are several ways to do them wrong and only one or a few ways to do them right. This is true with Agile.

Many organizations claim that they’re doing Agile, but they really aren’t. Many of these organizations are frustrated because they’re not getting the benefits they expected but that’s really no wonder because they’re not doing anything differently than what they were doing before.

Scrum is a useful methodology for planning and organizing work, but Scrum is a minimal framework and it says nothing about how to actually do the work. Scrum say for teams to self-organize and adopt the practices of Extreme Programming, but this often gets translated into “leave the team alone and hope for the best.”

Scrum software development does have something to do with software development. To really gain the benefits of Agile software development we have to do a bit more than just hold a good meeting. Planning sessions are valuable but the real challenge with software development is going from the plan to the product, which involves real technical work.

The Agile movement has made great strides in technical practices over the last few decades. The design patterns movement, the test first development movement, and most recently the dev-ops movements are really showing us how we can improve quality in our process that builds not only a better product but does so cost-effectively.

So, what really is Agile software development? Is it all about the daily standup meetings and estimating story points or is there something more to it? I know that everyone has their own opinion. If he were to ask the original authors of the Agile Manifesto, the guys who met at Snowbird in 2001, they would probably say that Agile software development follows lightweight processes to allow developers to apply their technical skills most effectively.

Back in the 90s, we were overwhelmed with software development processes. There was not only Waterfall but CMMI was starting to come on board and the Rational Unified Process and all of these software methodologies were heavyweight and development required a great deal of administration. Agile was a direct response to this. They wanted to come up with a lightweight process that would give developers more time to do development.

Agile was originally called the Lightweight Software Development Process but was renamed for obvious reasons. Who wants to do “lightweight development?” We don’t call it the lightweight software development process but that’s still the intention.

One benefit Agile gave us was the ability to measure our success based on working software. It doesn’t matter how pretty our plan is or how comprehensive our documentation is if our code doesn’t work. Customers pay developers to write code.

Before Agile, the time developers got to actually write code was a small part of their responsibilities. I know a lot of teams that spent less than half of their time writing code and many teams spend as little as 10% to 20% of their time actually coding.

What were they doing during the rest of their time? The answer is they were attending meetings, reviewing designed documents, writing documentation, etc. And while those things appear to be important, they actually detract from the primary value that developers create, which is writing code.

Agile simply says that the software development process should be primarily focused on developing software. We do this by removing extraneous distractions. We keep meetings to a minimum. We eliminate formal requirements and remove the overhead in creating, maintaining and interpreting formal requirements. We avoid batching up tasks into separate phases like analysis, design, code, and test because that is also highly inefficient when it comes to building software.

To the uninitiated, the Agile software development practices of Extreme Programming appear to be highly undisciplined but nothing is further from the truth. Doing emergent design takes great human skill and discipline. Developers don’t acquire the skills by osmosis, they have to study them and even more importantly they have to unlearn many of the practices that they were taught in school because schools don’t support iterative development techniques for Agile software development yet. But when teams avail themselves of the technical practices of Agile then they start to see the real benefits of Agile software development.

The WIP Whip

WIP stands for work-in-progress. It’s a fundamental concept in queuing theory, which has become a central part of Lean Software Development.

Some of us like to work on several projects at once. It feels like we’re getting a lot of things done sometimes but often that’s not really the case. Work-in-progress means that we have several things we have to attend to and so we round-robin our time and attention to address these multiple things.

The problem with having multiple pieces of work in progress is that there is a ramp-up time to get re-familiar with the task after we’ve abandoned it to work on a different task. How much time do we lose re-familiarizing ourselves with tasks? The answer varies but it could be as significant as half of all our time is spent getting back up to speed on projects that we dropped.

Of course, we can’t make perfect progress on every task all the time and when we’re blocked and waiting for something that we depend on, such as an external library or an answer to a question, we don’t just want to do nothing. Having other tasks that we can work on at these times is a good use of our effort.

Limiting work-in-progress is so critical that Lean defines waste in software as any work started but not yet finished. Based on this definition the Waterfall approach of building to a release is enormously wasteful.

Writing software is not like assembly line work. You can’t just pick up where you left off. There is an enormous amount of details that we have to keep track of when we build software, so getting back up to speed on a project can take a lot of time and effort.

Sometimes managers ask developers to work on several tasks at once. Perhaps they have critical knowledge of a certain technology or are the most senior person on the team. But I often find that as we start slicing these people’s time up they get less efficient. After all, we are only human.

In situations like these, I like to take out what I call my WIP whip. No, it’s not a physical whip but rather a mental one. It’s about imposing WIP limits so that no more than one, two or three tasks can be juggled by any one person at a time. I know managers who assign their top developers to up to eight projects within a single iteration and then they wonder why these developers aren’t getting all of their work done. Task switching is expensive.

For this very same reason, we say that you shouldn’t change directions on a team in the middle of an iteration because it pretty much invalidates everything that was done in that iteration. Instead, wait a week or two until the iteration is finished and then introduce new work. Even if what’s currently being worked on may not be used it’s still advisable to wait until the iteration ends before changing tasks because it’s very frustrating to be told that a task being worked on is no longer valuable.

I find that the very best teams limit their work-in-progress and that’s one of the main ways that they get more efficiency in what they do.

The Power of Continuous Integration

As an Agile technical consultant, people often ask me how to get the most value from their Agile technical practices. My answer is often the same, which is start with continuous integration.

Continuous integration is at the heart of all Agile software development. I see continuous integration as the defining characteristic of Agile software development, which I believe they summed up most clearly in the Agile Manifesto’s first principle, which states “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” I advocate continuous delivery and automating the validation of software because manual processes that do these things can become exorbitantly expensive.

The way quality assurance works in most organizations is fundamentally flawed, in my opinion. Don’t get me wrong, I am a huge quality advocate and I also believe that if I’m a developer I should be responsible for finding and fixing the defects that I write. Often, with a separate quality assurance organization, the developers throw code “over the wall” and ultimately become less responsible for that code. They have moved on and having to go back into code they have written previously to fix defects is majorly inefficient.

The majority of the time that developers spend responding to quality assurance issues has to do with familiarizing themselves with code that they had forgotten about. If only there was a way for the quality assurance team to constantly monitor the code and whenever there was any change whatsoever to any of the code they would be able to immediately run all of the relevant tests to prove that the new code and the existing code still work as expected. Of course, a quality assurance organization that fast and that responsive would be very expensive. Or would it?

I understand if what I’m describing is difficult to grasp but this is exactly what we get when we have a good continuous integration system that relies on a solid foundation of automated unit tests. With continuous integration, suddenly the things that were the most painful and time-consuming activities, like finding and fixing bugs, nearly goes away completely because the suite of fast regression tests in a continuous integration system can instantly find defects so the developer can fix them at the moment they’re introduced and move on quickly. This is like having a safety net under a high wire act and working with existing code is definitely like walking on a high wire at times.

Sadly, the way most software is written today, code intertwined in such a way that is very difficult to add a feature to an existing system without breaking other parts of the system. When we use continuous integration and build good automated unit tests for regression, we end up building functionally independent pieces that are not just independently verifiable but are not interwind to the rest of the system. This means our code is fundamentally more extensible and that is one of the most valuable benefits of continuous integration.

Playing Dumb isn’t Agile

I believe that Agile software development is ideal for discovering what really needs to be built and to build it in a way that it will last, but I don’t believe that Agile is required on every project. There are many kinds of projects and many reasons for not doing Agile.

I’ve written quite a few accounting systems in my day. Credits and debits. I helped to build a multi-currency accounting system for the wholesale banking industry in the 1980s. I know a lot about accounting systems. If you wanted me to build a new version of the system that I was already familiar with then I probably wouldn’t have to build it with Agile methodologies.

If you know exactly what to do, then there’s no need for discovery. There’s no need for feedback. If you can’t do anything with the feedback, then there’s no need to get feedback.

Agile is about learning as we go. It’s about taking the giant daunting task of building a software system and breaking it down into little pieces that each provide value and in and of themselves are simple to build.

I worked on many Waterfall software development projects back in the day and not all of them were failures. Several projects that I had worked on, as well as other developers that I’ve known, were quite successful in the Waterfall world. But I’ve had my share of failures as well, especially on Waterfall projects. And I’ve seen this same outcome throughout the software industry.

Agile helps but it’s no panacea. You can turn Agile into a whip and start getting the worst from people. Agile is not intended to be a tool for manipulation people.

And it’s not just a planning tool. Before Agile became the management process known as Scrum, Agile was a set of development practices that software developers used to effectively build products for clients in the wild.

To me, Agile is about the technical practices that deliver on the principles of Agile and help us “continuously deliver valuable software” to our customers.

Agile isn’t for everyone and Scrum isn’t for every team. The reason that Waterfall software development is still so prevalent is that it provides synchronization points with the business because most businesses are cyclical.

Since most businesses are cyclical there is always a component of interfacing with the rest of the business that happens in cycles. Lots of teams stay in synchronization with the rest of their company by following a Waterfall process for planning and interfacing with the rest of the business but internally they employ the practices of Extreme Programming.

I have several clients that build software using Waterfall methodologies in terms of designing to a release. However, it always makes sense to build the system feature by feature, integrating each feature into the system as it’s being built. To me, this kind of continuous integration is the very essence of Agile software development.

Build What Users Want

Our users are only human and its human nature not to always know exactly what we want. I realize that sometimes we have to build software on specifications and all the requirements are needed to be identified upfront. But statistically, it is rarely the case that all requirements are needed. It turns out that nearly half of all the features built are never actually used. We do this because we don’t have a really good sense of exactly what we want before it’s actually built, again because of human nature.

Once we build it we often learn a tremendous amount and when we get feedback from our users as to what’s working and what could be improved then we end up building something better than we originally conceived. To me, this is what Agile is all about.

One of the main tenets of Agile software development is having the opportunity for feedback because only through feedback can we really emerge a solution that fits well with the user’s needs. Feedback gives us an opportunity to learn and that’s critically important when building software.

The process of writing software is not some rote translation of a specification into code. The process of writing software is the codification of understanding. We understand the process and then embody that understanding in code. By doing this we make the code understandable to others. This is key because it doesn’t matter how well a design is if only the programmer who wrote it understands it and can extend it.

The software profession is in dire need of commonly agreed standards and practices. Those of us who are vocal in the Agile community are taking a stand for software quality because we know that in order to build a sustainable, reliable industry and one, by the way, that all other industries depend on, we have to have a common set of standards and practices that we can base our profession upon.

I’m a big advocate for code quality and Extreme Programming practices because I often end up with higher-quality software then if I were to follow a Waterfall software development process. And I tried to make Waterfall work for over a decade. It worked well on small projects, but it didn’t scale.

But it doesn’t matter how beautiful your code is or how maintainable it is if it doesn’t do the right thing in the first place. It’s far more important to “do the right thing” then it is to “do the thing right.” Fortunately, we don’t have to trade one for the other and we can decide to do both.

Feedback from our users helps us give them more of what they want. It also helps inspire us because we see how our work is used and how it helps other people. This is one of the most satisfying things about being a software developer. Seeing how people use the software that we create reminds us of the impact of our work and how it helps people.

Aggregation or Composition

I often get questions when I teach object-oriented programming about the difference between aggregation and composition. These are two relationships that are easy to confuse.

Aggregation in the UML class diagram is represented by an open diamond whereas composition is represented as a closed or filled-in diamond. Related to this is the “uses” or “depends on” symbol which is indicated by an arrow. A closed arrow represents inheritance.

So why does the UML call out these different relationships? It is easy to see the difference between inheritance and dependency because inheritance is a very special kind of relationship and we have to explicitly call this out in code. But why does the UML call out the difference between aggregation and composition? It doesn’t require any difference in coding syntax, at least not in managed code (we’ll discuss unmanaged code in a moment).

It turns out that this distinction is not nearly as important today with managed languages as it was fifteen years ago when we didn’t have automatic garbage collection in languages. In unmanaged languages like C or C++ the difference between aggregation and composition includes the notion that composed objects are memory managed by the one who composes them. In other words, if an object allocates memory for an object then it has the responsibility for de-allocating the memory when it’s done using it. This is not necessarily true for aggregate objects. However, in managed memory languages like Java and C Sharp, this is no longer an important distinction because memory is cleaned up by the garbage collector.

But there is a conceptual difference between aggregation and composition. I like to illustrate it by talking about parking lots. A parking lot has both aggregation and composition in it. A parking lot is an area that aggregates cars. A parking lot is still a parking lot when there are no cars on it. Aggregation is just a place for the aggregates to inhabit.

By contrast, composition refers to a stronger relationship where the composed object is a required part of its composer. For example, a car is composed of an engine, a steering wheel, four tires, etc. By our definition, a car ceases to be a car if you remove the engine. Composition refers to a strong relationship between the composer and the composed.

I like to think of aggregation as referring to the “has-a” relationship and I like to think of composition as referring to the “has-to-has-a” relationship.

In many situations, we simply do not care whether we are aggregating or composing objects or it may not be clear or relevant to us at the point we are creating a class diagram, so we simply use the arrow that indicates “uses”. This arrow indicates that one class uses or depends upon or calls methods upon another class, without specifying who allocates and de-allocates the memory or if the services they are using are required or optional for the other object.

There are only a handful of relationships in software development and in programming languages such as C++ or Java. It is of critical importance that we as developers use the right relationships. When we model using the right relationships it makes a better design. Virtually all of the Gang of Four’s twenty-three design patterns are some variation on combining aggregation and inheritance.

One of the most pervasive problems that I see as a software development coach coming in to various organizations or working with various developers is the overuse of inheritance. Inheritance is important and useful but it is not a cure-all for every kind of relationship in programming and a lot of times developers overuse it or use it for the wrong reasons and they end up with brittle code as a result.

Thinking with Objects

It took me more than a decade of doing object-oriented programming-and then teaching it-before I really started to understand the subtle power of using objects, and how to create behavior through the interaction of objects rather than through program logic. I struggled for many years with how to explain this different way of thinking. It’s easy to teach algorithms or the keywords of the language, but to teach a new way of thinking is a challenge in any field.

So, what is the essence of thinking with objects? It starts by having multiple perspectives is a program.

In procedural programs, we took a single perspective, which was the instruction counter the CPU. In other words, to understand what was going on we traced through our code just like the computer did, albeit much slower. That may seem reasonable and straightforward, and it worked well in many situations, but the problem came when we wanted to make changes to a program.

For example, let’s say you want to travel from point A to point B. There are at least a few ways that you can get guidance. You could get directions-series of steps to follow-and write them down, e.g. first go to the next light then turn left then go three lights down and turn right, etc. Alternatively, you could use a map, and while there are no instructions on how to get from where you are now to where you want to go, you can use the map to figure out how to get there.

Now let’s say you decide that you want to take a detour on your way and stop off for some ice cream. Yum. This may be difficult to do when you have a set of instructions but if you have to map that shows where the ice cream shop is located then you can use it to see how to detour to get there and then get back on track to your final destination. This is not the perfect metaphor for the difference between procedural programming and object-oriented programming, but it does get close.

Object-oriented programming takes many perspectives. Each object in the system represents an entity and each entity has a particular perspective in the system. By having separate entities, each with their own perspective, we can limit the amount of knowledge that each entity has, giving us the flexibility to change things without affecting everything else in the system.

It really wasn’t until I started to study design patterns that I saw how we can create complex behavior through the interaction of objects. Some patterns really taught me the essence of what OO is about, and that’s one of the main benefits I get from studying design patterns. There are many other benefits of studying design patterns. Patterns help us communicate with higher fidelity and understand the essence of a problem quickly. You can implement design patterns in virtually any language including procedural languages, although it is much more difficult in a non-OO language.

While the Gang of Four’s design patterns book is really revolutionary, even today, it is also a mess. There so many little mistakes throughout the text and it seems that there are no common definitions for some of the words they use. However, the big things are so amazingly groundbreaking that it makes up for some of the smaller problems. Also, there are some subtle assumption and certain presuppositions in the book that is also profound. For example, patterns often use polymorphic behavior and this implies for right objects are created prior to their use, and in fact, there’s a whole group of patterns called creational patterns that are involved simply in creating objects. This reflects one of the most fundamental yet misunderstood aspects of OO, to separate the use of objects from the construction of objects.

The great “ah-ha” moments for me came from the realization that this separation of using objects and creating objects has a huge effect on how we design systems that are more resilient to change. More on this in future posts.

Commit Frequently

One metric that I think is most important to look at during an Agile transformation is the frequency of commits that developers make due to the build.

This gives us a huge amount of information about several important things. First and foremost, it indicates the amount of friction in the build and how much work developers have to go through in order to commit their code. For example, without a set of automated regression tests committing our code requires a manual testing process that is often deferred to the end of the iteration, or even to the end of a release. This might make a lot of sense in building construction, but it’s highly inefficient when constructing software.

Building software requires many parallel tasks, but it’s far more efficient to take a task start to finish without interruption, and finding a balance between what to do now and what to defer on is very important.

To me, this speaks to the very essence of Agile, which is all about delivering working software. In Scrum, at the end of every sprint we’re supposed to have working software, not software we hand off to QA for testing.

When we separate development and test, we can introduce major inefficiencies to our system, and we’re also interrupting the flow of development, forcing developers to go back later to fix mistakes and incur massive time delays to reacquaint themselves with code that they wrote just a few days or weeks earlier.

This is what I call Painful Agile and I see many teams doing Painful Agile. Committing every two weeks is less painful than committing every three months, which is less painful than committing every year, but all of these commitment intervals include some degree of pain.

The pain only really goes away when we are committing all the time, whenever we have any little bit of functionality to add to the system. When we can commit frequently, we get working software more frequently as well, which lets us to validate that we’re on track more frequently and gives us a true measure of our progress.

Committing frequently also means that developers are breaking down stories into small, manageable tasks, which has innumerable benefits for Agile development. It not only shows us that our build has low friction so it’s straightforward to use, it also shows us that developers are good at breaking down stories into manageable tasks, which is another set of skills that are very important for an effective Agile development cycle.

In a healthy development environment, we commit several times a day. This may not mean that were writing several features a day but rather that we’re good at breaking out features into manageable chunks and each one is a step in the direction of overall progress.

And when we commit to version control, we are committing to trunk. The whole purpose of committing frequently is to let our newly developed code coexist with the rest of the system. This is what continuous integration is all about, but it can easily be defeated through branching.

I know lots of teams who use version control but each developer works in their own branch and it’s only much later, right before release, that the developers integrate their feature branches back into truck. There is a name for this approach and it’s called Waterfall.

Building software in feature branches really defeats the purpose of version control in the first place. Long live branches may seem like they are a good idea but they are almost always an anti-pattern because it isolates code rather than integrating it.

We want newly integrated code to be part of the whole system so we can validate that it doesn’t affect any other part of the system. Integration is traditionally when the worst bugs show up. When we put off integration we’re basically keeping ourselves from seeing the truth of our build and if we don’t know about problems we can’t fix them. When we integrate toward the end of a release, we can learn about problems and bugs when it’s too late to do anything about them.

But when we commit frequently we can see a true measure of our progress and the problem space is fresh in our heads, so if we discover a bug we can typically fix it right away. This has one of the largest boosts to productivity that we can get in software development.