Building Microservices: Designing Fine-Grained Systems

by Sam Newman

Being the main point person responsible for the technical vision of your system and ensuring that you’re executing on this vision isn’t just about making technology decisions. It’s the people you work with who will be doing the work.

I am a strong believer that great software comes from great people. If you worry only about the technology side of the equation, you’re missing way more than half of the picture.

Once you become very proficient, you may decide to skip the step of keeping the bounded context modeled as a module within a more monolithic system, and jump straight for a separate service. When starting out, however, keep a new system on the more monolithic side; getting service boundaries wrong can be costly, so waiting for things to stabilize as you get to grips with a new domain is sensible.

Prematurely decomposing a system into microservices can be costly, especially if you are new to the domain.

have seen too often that thinking about data leads to anemic, CRUD-based (create, read, update, delete) services.

I called this onion architecture, as it had lots of layers and made me cry when we had to cut through it.

The ideas presented in Eric Evans’s Domain-Driven Design are very useful to us in finding sensible boundaries for our services, and I’ve just scratched the surface here. I recommend Vaughn Vernon’s book Implementing Domain-Driven Design (Addison-Wesley)

I strongly recommend you take a look at the Richardson Maturity Model, where the different styles of REST are compared.

One pattern I saw used effectively by one of our teams was to delay the implementation of proper persistence for the microservice, until the interface had stabilized enough.

Shared code between client and server can be very dangerous, as we’ll discuss in “DRY and the Perils of Code Reuse in a Microservice World”

If you want to know more, I recommend REST in Practice (O’Reilly), which covers the topic of REST over HTTP in depth.

A bug had crept in whereby a certain type of pricing request would cause a worker to crash. We were using a transacted queue: as the worker died, its lock on the request timed out, and the pricing request was put back on the queue — only for another worker to pick it up and die. This was a classic example of what Martin Fowler calls a catastrophic failover.

I also strongly recommend Enterprise Integration Patterns (Addison-Wesley), which contains a lot more detail on the different programming patterns that you may need to consider in this space.

Structure 101

Chek this out

SchemaSpy, which can generate graphical representations of the relationships between tables.

Typically concerns around performance are now raised. I have a fairly easy answer to those: how fast does your system need to be? And how fast is it now? If you can test its current performance and know what good performance looks like, then you should feel confident in making a change. Sometimes making one thing slower in exchange for other things is the right thing to do, especially if slower is still perfectly acceptable.

Refactoring Databases by Scott J. Ambler and Pramod J. Sadalage (Addison-Wesley).

Continuous delivery (CD) builds on this concept, and then some. As outlined in Jez Humble and Dave Farley’s book

The tool I’ve used the most for this is Fabric, a Python library designed to map command-line calls to functions, along with good support for handling tasks like SSH into remote machines. Pair it with an AWS client library like Boto, and you have everything you need to fully automate very large AWS environments.

Terraform

thoroughly recommend you read Jez Humble and David Farley’s Continuous Delivery (Addison-Wesley),

Crispin and Janet Gregory’s book Agile Testing

Succeeding with Agile (Addison-Wesley), Mike Cohn

Growing Object-Oriented Software, Guided by Tests, by Steve Freeman and Nat Pryce (Addison-Wesley).

While I feel that stubs and mocks are actually fairly well differentiated, I know the distinction can be confusing to some, especially when some people throw in other terms like fakes, spies, and dummies. Martin Fowler calls all of these things, including stubs and mocks, test doubles

Brandon Bryars has potentially saved many of us a chunk of work with his stub/mock server called Mountebank.

If you have tests that sometimes fail, but everyone just re-runs them because they may pass again later, then you have flaky tests.

A test suite with flaky tests can become a victim of what Diane Vaughan calls the normalization of deviance — the idea that over time we can become so accustomed to things being wrong that we start to accept them as being normal and not a problem.

With the end-to-end test step, it is easy to start thinking, So, I know all these services at these versions work together, so why not deploy them all together? This very quickly becomes a conversation along the lines of, So why not use a version number for the whole system? To quote Brandon Bryars, “Now you have 2.1.0 problems.”

This situation worsens if we fall into the trap of adding a new end-to-end test for every piece of functionality we add. Show me a codebase where every new story results in a new end-to-end test, and I’ll show you a bloated test suite that has poor feedback cycles and huge overlaps in test coverage.

When considering canary releasing, you need to decide if you are going to divert a portion of production requests to the canary or just copy production load.

If you are interested in reading more about testing, I recommend Agile Testing by Lisa Crispin and Janet Gregory (Addison-Wesley), which among other things covers the use of the testing quadrant in more detail.

A discussion about all the nuances involved in the graphical display of quantitative information is certainly outside the scope of this book, but a great place to start is Stephen Few’s excellent book Information Dashboard Design: Displaying Data for At-a-Glance Monitoring (Analytics Press).

If you want to know more, I go into some of these ideas and more in my earlier publication Lightweight Systems for Realtime Monitoring (O’Reilly).

And the more functionality something has, the greater the attack surface.

Encrypt data when you first see it. Only decrypt on demand, and ensure that data is never stored anywhere.

AppArmor, for example, allows you to define how your application is expected to behave, with the kernel keeping an eye on

Finally, if you want a more general discussion of cryptography, check out the book Cryptography Engineering by Niels Ferguson, Bruce Schneier, and Tadayoshi Kohno

Loose and Tightly Coupled Organizations In Exploring the Duality Between Product and Organizational Architectures (Harvard Business School), the authors Alan MacCormack, John Rusnak, and Carliss Baldwin look at a number of different software systems, loosely categorized as being created either by loosely coupled organizations or tightly coupled organizations. For tightly coupled organizations, think commercial product firms that are typically colocated with strongly aligned visions and goals, while loosely coupled organizations are well represented by distributed open source communities. In their study, in which they matched similar product pairs from each type of organization, the authors found that the more loosely coupled organizations actually created more modular, less coupled systems, whereas the more tightly focused organization’s software was less modularized.

When the cost of coordinating change increases, one of two things happen. Either people find ways to reduce the coordination/communication costs, or they stop making changes.

No matter how it looks at first, it’s always a people problem. Gerry Weinberg, The Second Law of Consulting

We can also spend a bit less of our time trying to stop the inevitable, and a bit more of our time dealing with it gracefully. I’m amazed at how many organizations put processes and controls in place to try to stop failure from occurring, but put little to no thought into actually making it easier to recover from failure in the first place.

Antifragile (Random House), Nassim Taleb talks about things that actually benefit from failure and disorder.

Michael Nygard’s book Release It! (Pragmatic Programmers)

REST In Practice (O’Reilly) or read section 13 of the HTTP 1.1 specification, which describes how both clients and servers are supposed to implement these various controls.

Decentralize All the Things To maximize the autonomy that microservices make possible, we need to constantly be looking for the chance to delegate decision making and control to the teams that own the services themselves. This process starts with embracing self-service wherever possible, allowing people to deploy software on demand, making development and testing as easy as possible, and avoiding the need for separate teams to perform these activities.

Change is inevitable. Embrace it.