David Scott Bernstein's Blog, page 26

Creativity

In an industrial society workers perform rote tasks that are interchangeable with others so the faster a worker accomplishes his task the better a worker he is. That being the case, it’s not difficult to understand why so many industrial workers have been replaced by robots.

But information workers are different. What we do requires, but does not necessarily reward, creativity.

Speeding up one task, such as writing code (a local optimization), doesn’t necessarily mean it serves the project. Rushing through coding causes mistakes that ultimately slow the project down even more. Defining the right things to do and the meaning of quality is far more subtle in the virtual realm.

Measuring developers by the number of lines of code they write in a month tends to incentivize them to write that much code, no more and no less. And the “no less” half of that equation de-incentivizes them to use code already written in a repository. So not only is that developer’s productivity capped at the high end, a valuable resource is going unused at the cost of efficiency.

But what if a project begins with the positive expectation that developers are skilled professionals who take pride in the quality and efficiency of their work, and managers lead by encouraging input into both process and product? What if we tried to understand creativity and its role in software development?

“Creativity is a central source of meaning in our lives,” wrote Mihaly Csikszentmihalyi in “The Creative Personality” (Psychology Today: www.psychologytoday.com/articles/1996...). “Most of the things that are interesting, important, and human are the result of creativity. What makes us different from apesour language, values, artistic expression, scientific understanding, and technologyis the result of individual ingenuity that was recognized, rewarded, and transmitted through learning.”

Software developers love a good challenge. We love figuring things out, and we’re drawn to this profession for just that reason. Putting unnecessary boundaries on our problem-solving ability like demanding a certain number of lines of code per day or over-incentivizing or de-incentivizing use of code libraries, has a dampening effect on the one thing we need most:

Creativity.

We need to think through a problem, find the right path through a feature, and discover not just the fastest way to deliver the basic functionality the requirements call for, but create software that exceeds our customers’ expectations.

Only a creative software developer can do that, and the only requirement necessary is respect for what a smart, committed developer can bring to the table.

Turn Pain into Gain

Integration is one of the most painful parts of a traditional software development process. After the system is designed and coded it gets integrated together in a Waterfall process, and this is when the nastiest bugs show their faces. It’s also when we really can’t afford to address fundamental issues.

When I was doing Waterfall development throughout the 80s and 90s, life was good for the first nine months of our annual development cycle, but in the last three months, which were devoted to integration and testing, I aged far more than a year. When I was in it I recognized that this was not the way to build an industry and keep it sustainable. People burned out, but then the next project would come along and things would be great for the first nine months.

Building to a release rather than to a feature is not the way to construct virtual things. Software is fundamentally different than practically everything else in our world and if we want to be able to build it effectively we have to begin to understand its unique nature.

Instead of putting off integration, what if we were to integrate all the time, as we go, wWhenever we have any little bit of functionality? If we do that then the big pain of doing a giant integration goes away. We may instead have lots of little integrations to do throughout the day, but they’re so small, and the changes we made are still fresh in our heads, that they’re relatively trivial to fix if something isn’t working.

In fact, integrating code as it’s being built is one of the most valuable forms of feedback a developer can have. If we’re told about a bug days, weeks, or even months after it’s written, it takes a lot of time and energy to re-familiarize ourselves with the code, find the bug, figure out what the right thing to do is, and fix it.

Building software incrementally means that we are going back into the code we wrote in the previous iteration to enhance and add features. If the code we wrote is difficult to work with we’ll suffer. So instead, developers doing Agile pay attention to making their code easier to work with for a really practical reason: They’ll have to work with it in the future. This is one of the main benefits of Agile that isn’t often talked about.

Feedback is critical for building anything but especially for building software, which cannot be touched or tasted or seen. Software is intangible, it’s the most opaque thing that we deal with as human beings, yet our process for constructing it contains very few checks and balances. This is not true in physical construction where there’s all sorts of testing and verification that goes into building a bridge or a shopping mall or an engine—but the way we build software is different.

We start with this specification that kind of sort of almost describes how a system should behave. That specification typically comes right out of our minds, from our imagination, what we can visualize we want. Then a developer takes those specifications and writes code around it based upon the ideas in their minds that they got from reading the specifications. And then they deliver the code.

No checks, no balances. Sometimes there is a QA process that tries to hunt for bugs or usability issues, but even that process is rarely formalized. And since it’s impossible to assure that there are no bugs in even a relatively trivial program all testing can do is find common problems that the tester thinks to look for.

We need checks and balances that make sense, that support us in moving forward with good code, and I believe that many of the methodologies in extreme programming do just that.

But the thing about extreme programming that a lot of people don’t understand is its context. The developers who came up with extreme programming are highly disciplined, and to do extreme programming successfully requires paying attention to good quality in software, following principles and practices, and being highly disciplined about it.

I hear from so many new Agile teams that they feel they shouldn’t have to do any upfront design because they’re doing Agile—and this is just a major misinterpretation of what it means to do Agile.

It’s true that in Agile, we don’t do a lot of upfront design, but that’s because we’re doing design throughout the entire development process, not just upfront like we did in a Waterfall process. We’re emerging designs as we go and that requires a great deal of discipline and paying attention to our code and the designs we’re working with. If we don’t pay attention to these things we’ll end up just making a mess.

I liken Agile practices to power tools. You can cut more woods with the chainsaw than you can with a handsaw, so chainsaws are valuable. But they’re also much more dangerous than a handsaw. Agile includes some very powerful methodologies but as Spider-man’s uncle said, “with great power comes great responsibility,” and it’s our responsibility if we’re following an Agile process to be disciplined about it especially since we all have the checks and balances that Waterfall tries to impose to assure that we are creating quality software.

Just because you’re “doing Agile” doesn’t assure that you’ll be successful. We can’t just blindly follow the practices. While many processes in life can work if you just follow them without understanding why, this isn’t true with the Agile practices and processes. We must understand them and constantly be staying alert for how it’s working. It’s not enough to just get feedback, we have to do something with that feedback. We have to learn from it, and if we do we can be successful.

Time Box to Scope Box

One key characteristic to Agile software development is time boxing. We build software in fixed intervals called time boxes.

Time boxes for iterations are usually one to four weeks, with most teams opting for a two-week iteration length. Any fixed time iteration is going to be somewhat artificial. Software development tasks don’t all lend themselves to neat little two weeks sprints. So then why do we time box?

We time box in order to get good at the scope boxing. Time boxing means to work on a task for a specific length of time. Scope boxing means to work on a small task until it’s complete. But people coming from the Waterfall world often don’t have the skills to break large tasks down into smaller tasks, so time boxing is usually easier to do than scope boxing.

By making a time box efficiently small, it means that we can only work on the core parts of a feature. This often leads to keeping development more focused around building a single thing at a time. Building this skill helps us get good at managing scope and building the smallest increment we can.

Once we get good at breaking tasks down into smaller components then we don’t necessarily need to use time boxes anymore and we can move on to more of a poll system like Kanban rather than working in fixed iterations.

This very much depends on the nature of the work we’re doing. Maintenance tasks and bug fixing might lend itself more toward a scope box approach, whereas new development might be more appropriate for a time box approach.

In the end, time boxing is artificial but it can be a valuable discipline. Like the beat of a drum in a song helps keep all the musicians in sync, time boxing helps keep everyone on the team working at the same cadence.

Ultimately, the most mature teams that I see tend to move toward scope boxing rather than time boxing.

Test Driven Damage?

Understanding the technical practices of software development is not easy for everyone and a lot of otherwise smart people are failing when they try to apply them. Some people even claim that the practices I cover in my book, Beyond Legacy Code: Nine Practices to Extend the Life and Value of Your Software, don’t work.

For example, David Heinemeier Hanson, the author of Ruby on Rails, says that Test Driven Development (TDD) is dead. He refers to what he calls “test-driven damage” where developers write too many tests, and write implementation dependent tests, and go about changing their production code to accommodate their tests.

Justin Searl wrote a blogpost about how TDD doesn’t really work very well. He says it may work well initially but as soon as we need to start to refactor our code, adding classes and methods, it requires us to write new tests and it breaks old tests.

Justin Searl and David Heinemeier Hanson are really smart people and bring a great deal of wisdom to our industry, but they’re missing some key points. I have seen “test-driven damage” before, where developers make compromises in code due to their tests, but this is not a fault of test driven development. It’s the fault of the way we’re using it.

Kent Beck said we should “write tests until bored,” because by the time a developer gets bored they probably have pretty decent test coverage for their code. But TDD is not about testing—at least not exclusively—and we shouldn’t put our tester’s hat on when doing TDD. It’s a different mindset. It’s like putting your editor’s hat on as you’re writing, and any writer will tell you that trying to edit yourself while you’re writing is a recipe for writer’s block. When writing it’s far better to just let the words flow then go back later to do the editing. A similar thing is true when we’re doing software development. The goal of writing code is to create behavior. Focusing on every little thing that could go wrong can block us, and we don’t want to do that—at least not initially.

No one ever said that if you have test coverage for a behavior and then you refactor the implementation to use more classes and methods that you have to add more tests. No one said you have to but the term “unit test” implies that you do. If you see a unit as a piece of code but that was not the intention of the phrase “unit test.” The word “unit” is referring to a unit of behavior. If the behavior that you’re testing does not change, your test doesn’t need to change either.

The purpose of doing TDD is to support us in refactoring our code but if we make up the rule that every unit of code has to have a test associated with it, then refactoring becomes nearly impossible. Suddenly, tests become a major burden instead of a support system. I sometimes refer to our unit test as a safety net that’s there to catch us if we make a mistake. It’s important that a safety net be attached to the ground so it’s stable but that doesn’t mean we want to entomb our safety net in concrete. Indeed, it would be very stable but the whole purpose of having a safety net would be defeated. The same thing is true with TDD.

Many developers buy into the notion of code quality and so they pay particular attention to the software they write to make sure it’s maintainable and easy to work with. But often they fail to see their tests as code as well. While their code is nice and clean, their tests can get awfully messy. If we’re going to do TDD correctly, we can’t treat our tests as second class citizens. Tests are code every bit as valuable as production code and should be treated with the same respect and with the same focus on quality.

If we reject TDD in particular, like some people are claiming we should, then we will be setting the software industry back at least twenty years. Nothing that we build in the world gives us so little feedback as the software development processes that most developers employ. We are dealing with something that we cannot touch or taste or smell and yet we are perfectly comfortable writing it out of our hands and then throwing it into production without any real verification. Because of the complex nature of software, where code is coupled and intertwined together, changing one line of code can affect an entire system’s behavior so the whole system often times needs to be retested even after minor changes are made.

Physical systems do not behave this way, I can knock out a support beam at my house and while the cieling may sag a little, the whole structure won’t come toppling down. Even bookkeepers use something called “double book entry accounting” where every credit has a corresponding debit to make the accounts balance. If the bookkeeper made an error entering a credit then that error will be picked up when they enter the corresponding debit. But there is no double book accounting for software development. Code comes from our heads, through our hands, and right into the computer.

This is not a very reliable approach, so having a way to triangulate or verify code as we write it is incredibly important, and that’s exactly what TDD does for us. In the future, if we aren’t using TDD as an industry then we’ll needsomething else that gives us the same benefit. We simply cannot progress as an industry without it.

People are not Resources

Whenever I hear a recruiter or a manager refer to a software developer as a “resource,” I cringe. People are not resources.

Unlike assembly line workers, software developers don’t perform wrote tasks and so therefore they are not interchangeable nor are they scalable.

In his book The Mythical Man Month, Frederick P. Brooks, Jr. writes “The bearing of the child takes nine months no matter how many women are assigned.” Clearly there are certain tasks that cannot easily be deconstructed and this is true for most tasks in software development.

Developing enterprise software is fundamentally a social activity. It requires a great deal of communication even on projects with one developer. That developer’s primary focus is to communicate their intent through the code that they’re writing.

The challenges we face and the things we do day to day are often very different from each other. Sure we wiggle our fingers and type in the same characters every day, but what we make the computer do is often very different from day to day and requires different problem solving skills that hopefully keep us constantly learning and growing.

This is very different from the kind of manufacturing mentality from the Industrial Revolution where people work in assembly lines and are just as interchangeable as the parts they’ve assembled.

This was done for economic reasons and the idea was to make every task so simple that anyone could do it. Because assembly line work didn’t require any special skills, it became a commodity that drove down the price of manufactured goods.

But software development isn’t like manufacturing. Assembling code is not nearly as clean and simple as assembling Lego blocks to build a toy structure. Software is messy and the tools, the language of code, are still not yet well understood.

But software, like literature, has profound depth in both form and content. Code is an incredibly rich form of communication when it’s used that way.

Of all the languages that humans have invented, the software languages are the most abstract and the best representation of pure information models. Software is critical to business because software often embodies their competitive advantages. So the maintainability of their software speaks directly to their bottom line but this is also interesting from a human perspective.

Physics have shown us that most of the material world, the world that we experience, is actually an illusion. The solidity of matter is in illusion because the atoms that compose it are themselves composed of positively and negatively charged waves. Waves are energy but not just energy, they have frequency and amplitude. They contain information, so the core of the universe isn’t a particle or a wave but rather information, which takes the form of particles or waves.

I think of information as stable forms of waves, but however you think of it, information has characteristics. It has organization. And these characteristics are manifestations of the fundamental principles that underlie information, which we can infer indirectly. When we do we begin to get a glimpse of the causal universe, the true nature of reality, and we begin to understand the fundamental properties of a world we can’t experience directly but from which everything we know is experienced. I find this fascinating.

Assert Behaviors Through Acceptance Tests

Acceptance Test Driven Development (ATDD) is a relatively new but essential practice in software development. This practice goes by many names, including Behavior Driven Development (BDD) and Specification by Example.

This practice is not to be confused with Test Driven Development (TDD) or other forms of testing. ATDD or BDD is something unique. And as it turns out it fills in several missing pieces in the software development process.

First and foremost ATDD sets the context for a vital conversation to happen between the Product Owner and the developers that defines, in a concrete way, precisely how a feature should behave. It establishes the acceptance criterion that defines when the feature is done. This is important for many business reasons as well as many technical reasons. It turns out to be an efficient and effective way of collecting and validating requirements, far more efficient than traditional requirements documents, which are non-executable and can easily get out of sync with the system’s behavior.

Acceptance tests also give us a way of calling out edge cases and exception cases. They can be informally jotted down on the back of the story card or they can be coded up using frameworks like Fit or Cucumber.

When acceptance tests are automated, you get a system that automatically certifies the expected behaviors of your system.

The Single Wringable Neck

Every successful software development project I’ve ever worked on has had a Product Owner. They may not have been called by that exact title, but that was essentially their job. They were responsible for defining the product, prioritizing features, and answering the myriad of questions developers had as they built those features.

The Product Owner isn’t necessarily a technical role. In fact, in some ways it’s better if it’s not done by a technical person because it’s important to see the product from the user’s perspective. I know from my own experience that sometime my technical skills can get in the way of that. But even though the Product Owner isn’t a technical role, it is a critical and vital role.

One of the benefits of having a champion for the product with a clear vision of what the product is and how it should behave is that the team can then rely far less on written requirements.

Traditional requirements can consume up to half of developers’ efforts on a project. It takes a lot of time and energy to gather requirements, write up a document that describes them, and then have developers read, interpret, and act on those requirements. As you might imagine, this is also the largest source of bugs in any part of the software development process.

When we use traditional requirements we’re really setting ourselves up to fail because we’re defining a very complex system but our definitions really have no basis in reality—they come from our heads. And if you ask craftsmen in any other field,they’ll tell you that the process of construction teaches them how to make the right choices. When we develop code in a traditional way, where there are requirements that can’t be executed, we’re depriving ourselves of this vast source of knowledge and feedback.

There are certainly times when written requirements are appropriate, and there are certainly some kinds of software projects where, on a Waterfall approach, it’s the right choice. I’m really not an absolutist though sometimes I appear that way in writing.

In most situations, on most projects—and especially projects that are particulalry complex—it really pays to have a Product Owner or a product champion to streamline the development process, cutting out 30% to 50% of the effort upfront and removing at least that many bugs on the backend from misunderstandings in interpreting requirements.

Sometimes managers say to me that they can’t afford to have a dedicated person act as a Product Owner for the development team and I say you can’t afford not to. Development is insanely expensive. Most companies don’t realize that they spend a fortune on software development. Their salaries, office space, insurance, and a myriad of other things—let alone opportunity costs—all add up.

So when development is delayed or is inefficient, it profoundly affects a company’s bottom line and very often the managers on the front lines don’t recognize this is happening. It’s a classic example of penny wise and pound foolish and I’ve see it almost universally in the industry.

We are, after all, knowledge workers, and we can leverage our knowledge to be more effective and efficient. If you assume your skills or your team skills are stagnate then they will be. But if you recognize the power of the gray matter between your ears—and everyone else’s—then you can do extraordinary things. As managers it’s really our job to do this for our team, and not just because it’s better for the company but because it’s more fulfilling for the individuals, too, and if you can address fulfillment for the individuals in your team then you are a great manager.

The main job of the Product Owner, aside from defining the product and prioritizing the features, is to help the developers any way they can by removing impediments, answering questions promptly, working with other teams to coordinate dependencies, and supporting the team to do the right thing.

Scrum defines the two roles of ScrumMaster and Product Owner to almost be complementary to each other. The ScrumMaster is concerned about the health of the team and their sustainability. The Product Owner is concerned with the product. But a wise Product Owner is also concerned with the health of the team because that’s the engine that drives the creation of the product.

They’re also concerned with the quality of the whole product, not just features, because they recognize that the product must give value not just today but well into the future. And so they encourage the team to write good code that is supportable even if it takes a little extra time up front.

But the Product Owner is also where the buck stops. They carry the major responsibility for the definition of the product so if the team builds a wrong thing or builds the thing in the wrong order, it’s the Product Owner who often takes the heat for this because that’s their responsibility.

So the Product Owner has two aspects. On one hand, they’re the superstar, the champion for the product, and they drive the team to build the right thing. On the other hand, they are the single wringable neck, the person management blames if they get the product definition wrong.

If you’re a Product Owner then you probably don’t want to figure out your job all by yourself. There are standards and practices you should be aware of. It’s an important role and it’s fundamentally different than traditional product managers. All their skills on building complex spreadsheets and charts won’t really help you very much, but there is a wealth of information online on how to be a good Product Owner. It’s vital in Agile development and it can also be very useful even in traditional Waterfall development, even with requirements documents. It’s extremely helpful to have someone who truly understands what the product should be and can answer the many questions that come up during development that are never in requirements.

Scrum says that the ScrumMaster and the Product Owner should be different people and usually that makes a lot of sense, but I have worked on projects where the ScrumMaster and Product Owner was the same person and it worked out very successfully but I consider those individuals to be particularly enlightened. They cared about their teams and they cared about their products. There is a middle ground where both can be kept in balance, and this is an ideal place for the Product Owner to be.

Smaller Tasks Have Big Benefits

A central benefit to Agile software development is breaking down tasks into the smallest component that can provide measurable feedback. When we do this we take complex tasks and simplify them by breaking them down into smaller, easier-to-handle tasks.

Smaller tasks have four essential benefits:

First, they’re easier to understand, which means they cost less to work with and are less risky.

Smaller tasks are also easier to estimate. More accurate estimates give projects a higher degree of predictability.

Smaller tasks are also easier to implement. When development is more highly focused, tasks get competed quicker.

Finally, smaller tasks are easier to test and verify. When we work with smaller tasks it means we get validation more frequently, which gives us more opportunities to adjust.

And that’s all I have to say about that.

Dev.Cast

Dag König and I spoke on Dev.Cast 145 about the state of software and its future, my book, and lots of other stuff. I was so excited that Dag could hardly say a word. But it was a fun podcast. Check it out here.

Tell Stories

A story is a single sentence description of a feature.

Stories tell three things about a feature:

• What it is

• Why it’s there

• Who it’s for.

Stories do not tell how to implement a feature or even what it looks like. Stories are very high level.

Stories are placeholders meant to remind the Product Owner and developers to discuss a particular feature. It’s that discussion that replaces requirements and turn software development from blindly fulfilling a specification into a discovery process where the Product Owner and the development team work together to find the best solutions to problems.

For example, a story for an online auction system might read as follows:

As a registered user, I want to bid on a started auction so that I may become the highest bidder.

Stories typically have this format:

As a _____ I want to _____ so that I can _____.

The first part of the story says who the story is for. The story is written in the first person: “As a registered user, I…” I have now included an initial constraint: “as a registered user,” which would have been defined in the previous story.

The next part of the story says what the story is about. “I want to _____” This is expressed as a desire from the user, again in the first person. It’s important here to focus on the goal of what we’re trying to achieve while giving us as little information as possible about how to achieve it. The goal is to state what the subject of the story wants.

The last element of the story states why the feature is wanted “…so that I can _____.” The so-that clause really speaks to why the feature is wanted and what the subject will get out of it.

These three pieces of information are key: who it’s for, what they want, and why they want it.

It’s easy to get lost in implementation details or the look and feel of a feature, and sometimes we lose sight of the bigger picture: Why we’re building the feature in the first place.

Stories keep us focused on what the feature is all about, who it’s for, and why they want it. It helps us build a better product for our customers.