Rethinking Productivity in Software Engineering

by Caitlin Sadowski

Any single productivity metric is intrinsically problematic. Productivity is too broad of a concept to be flattened into a single metric, and confounding factors will exacerbate the challenges with attempting such a flattening.

Productivity is a broad concept with many aspects. The problem is that productivity metrics are poor proxies of the underlying behavior or activity that we want to measure. As poor proxies, they are ripe for misuse.

Our approach works because we explicitly do not attempt to create a single metric to measure engineering productivity. We instead narrow down the problem into a concrete research statement and seek metrics that address precisely the question at hand. This allows us to validate each individual metric against a specific goal, rather than against the vague concept of productivity.

There is no single productivity metric for software engineers. Instead, focus on a set of custom metrics targeted to a specific question.

the ultimate bottleneck in realizing productivity improvements is behavior change.

if our productivity utopia relies on developer insight into their own productivity to identify opportunities for individuals to change, why not just focus on developers in the first place, working with them individually and in teams to identify opportunities for increased productivity, whatever the team and organizational goals?

One of the core principles of agile development is to create customer value

“efficiency is doing things right”

“effectiveness is doing the right things.”

an agreement prevails that efficiency refers to the utilization of resources and mainly influences the required input of the productivity ratio. Effectiveness mainly aims at the usefulness and appropriateness of the output as it has direct consequences for the customer.

for measuring software productivity we need a measurement of input and output of a software project. The input is the effort dedicated to its development and evolution. The output is the value of the software for its users or customers.

Hence, we suggest a purpose-based definition of software value. Given a purpose (a business goal or an application vision), we ask, how well does the software address its purpose in terms of functional and nonfunctional requirements? The answer to this question is determined by the functionality as well as the nonfunctional quality of the software.

a team can be productive only if it is effective and efficient!

Learning or skill development that may positively impact long-term quality, developer retention, or velocity may manifest as an increase in satisfaction. For developers, satisfaction may be impacted by the real or perceived effectiveness of their personal work or their team’s work.

For example, the goal-question-metric (GQM) approach for understanding and measuring the software process [1, 2] works by first generating “questions” that define goals and then specifying measures that could answer those questions. GQM suggests a systematic approach to do the following: Conceptualize goals aimed at understanding or improving software engineering tools and processes Specify research questions to operationalize those goals Define metrics for understanding or measuring tools and processes

measuring engineer satisfaction is challenging, as satisfaction is influenced by and refers to many different concepts.

Productivity should be considered along three dimensions: quality, velocity, and satisfaction. These three dimensions complement each other but often are in tension with each other. The dimensions have several possible attributes; measuring them is highly task and situation dependent. Productivity goals may be refined by considering the three dimensions through a set of perspective lenses. The main lenses we suggest include the stakeholders, the development context, the levels, and the time scale.

one study showed that in terms of task completion time, the skill of comprehending what a program does explains much of the variance in task completion time [

teaching novices expert strategies can make them match expert performance quite quickly

study found that simply using rudimentary tools for navigating code (scroll bars, text search, etc.) can account for up to a third of the time spent debugging code

Training developers on tools that increase productivity is a potentially cheap way to help developers get more work done in the same amount of time.

Research shows that team productivity is actually bounded not by how efficiently individual developers work but by communication and coordination overhead

even for individual decisions, developers often need information from teammates, which studies have shown is always one or two orders of magnitude slower to obtain than referencing documentation, logs, or other automatically retrievable content

An additional challenge to outcome-oriented metrics for software engineering is that difficult software problems may have similar-appearing output to easy problems.

measuring productivity as completion of self-determined goals has good construct validity, as research suggests that task or goal completion is the top reason that software developers report having a productive workday [5].

Studies of knowledge workers have found that a physical environment that increases productivity is one where there is adequate space for solitary work for concentration, official and unofficial meetings, and informal collaboration. A physical environment that enhances productivity also has good ergonomics with low noise and few interruptions.

The virtual environment refers to the technology that knowledge workers use. A virtual environment that enhances productivity is one where the technology is easy to use and available wherever the knowledge worker is working.

research suggests that usable programming languages and powerful tools, as well as collaboration platforms like GitHub, are important for improving software developer productivity.

The social environment refers to the attitudes, routines, policies, and habits performed by workers in an organization. Productive social environments are those where knowledge workers are given freedom to choose their work methods, work times, and work locations; information flows freely among workers; meetings are efficient; clear technology usage and communication policies exist; goals are cohesive and clearly defined; work is assessed in terms of outcomes, not just in terms of activities; and experimentation with new work methods is encouraged. A social environment for software development that enhances productivity is one where, for example, developers are given freedom to try new tools and methodologies. The importance of the social environment is underscored by Google’s finding that psychological safety—that members of a team should be able to take risks without fear—is the most important predictor of effective teams.

individual work practices measure to what extent knowledge workers will actually implement these practices.

A productive knowledge worker is one who enjoys and is enthusiastic about their work, finds meaning and purpose in their work, is not continuously stressed, is appreciated, has a work-life balance, finds the work atmosphere pleasant, and resolves conflicts with co-workers quickly. This suggests that the famous 80-hour workweek developer is not a productive developer.

it is not sufficient to look at quantity of output but to include the quality of the work as well

There are four main techniques for measuring knowledge worker productivity: outcome-, process-, people-, and multi-oriented productivity measurement techniques.

There are five categories of drivers that knowledge worker research suggests influence productivity: the physical environment, the virtual environment, the social environment, individual work practices, and well-being at work.

Additional constraints potentially slow down development.

Longer projects are more difficult to organize but benefit more from rules and custom tools. A more recent study [8] distinguished between development and integration projects. Development projects create most of the software during the project, while integration projects mostly connect and configure existing software. They found that integration projects are more productive.

The completeness of design before the start of coding impacts how much changes need to be done later.

Early prototyping can increase productivity because requirements can be clarified and risks can be resolved. Today, this is often replaced by iterative and incremental development. Such a development probably is able to better deal with volatile requirements, but the completeness of the design during initial coding is low.

A general factor is the process maturity, meaning how well-defined the development process is. In the recent years, research has focused on agile processes and found that they impact productivity.

Camaraderie means a social and friendly atmosphere where team members socialize but also help each other.

clear goals that are necessary so that all team members work toward the same objective.

Most general is the factor communication that includes the degree as well as the efficiency of information flow inside the team. In general, what is surprising in the studies is that communication effort is positive for productivity. In discussions, we often hear that communication should be reduced to decrease unnecessary work. However, the actual problems seems to be the increase of communication effort when putting mo...

Psychological safety is similar to camaraderie but more specifically refers to an atmosphere where individual developers can take risks and share personal information, but know that teammate...

If the team believes that they are the best engineers always building the highest-quality software, they are more likely to go the extra mile to actually achieve this.

Also related to psychological safety is support for innovation. This contains to some degree safety for taking risks, but it also means that the team members are open to bring in innovations and also change the way they work.

Team cohesion describes how well all team members are willing to work together. This does not necessarily include a social and friendly atmosphere but a ...

A common team identity also seems to support productivity, probably by influencing other factors such as camara...

less turnover is better for productivity, and it is one of the few factors that we can easily measure.

although experience is often brought up and is in interviews considered important, in empirical studies it is rather insignificant. By far more interesting is the capability of the developers. Hence, this suggests that being in a profession for a long time does not necessarily make one productive.

Experience does play a role but more in the sense of the experience with application domains and platforms. We have the three factors of application domain experience, manager application domain experience, and platform experience. The first two refer to how long and with what intensity the developers and managers have worked on software in a specific application domain. The latter refers to the experience of the individuals with a hardware and/or software platform such as the iOS operating system for mobile Apple devices.