The REST API Design Handbook

by George Reese

Data are plain facts. When data are processed, organized, structured or presented in a given context so as to make them useful, they are called Information. The golden egg is your data, be it business, social or consumer. It doesn’t matter. What matters is that you have data, so you just need to plan on how to turn those straws of data into the gold of actionable information.

By analyzing and understanding traffic flow within your APIs, you can quickly take empirical data, understand patterns and create a comprehensive view of the usage of each API. This deep, and often previously unavailable insight into actual usage provides the basis for effective decision-making processes around continued or new investments. This is how APIs support and eventually become your business.

the growth in demand for this skill set has rapidly outpaced the actual cultivation of this knowledge among developers.

most APIs suck is because, as a community, we’re all still learning how to get them right. We know SOAP sucks. We know we should be RESTful, but it’s not common knowledge what it means to be RESTful or why being RESTful is important.

SOAP, like its predecessors, also provided mechanisms for tools to automatically generate language bindings for different languages. In over-simplified terms, an IDE can examine a SOAP WSDL (Web Services Descriptor Language) XML document that describes the remote services and subsequently generate language-specific bindings. As the developer, you never know anything about the underlying SOAP communication. You know only where the WSDL is located and use the language-specific bindings for your target language. If you are programming in Java, you see a bunch of Java objects. If you are programming in C#, you see a bunch of C# objects. And so on.

the level of SOAP complexity to which you are exposed varies from programming language to programming language. My preferred programming language, for example, is Python. But I absolutely can’t stand SOAP development in Python. The real issue with SOAP isn’t in the differences in which complexity is hidden in various languages. The issue with SOAP is simply that the complexity gets hidden.

REST isn’t even a standard and its origination had nothing to do with APIs. It is instead an architectural philosophy that prescribes how web-based client/server systems should be designed.

The particular focus was on how users through some tool like a browser would interact with distributed resources. So, REST is not about how you define distributed APIs and it has nothing to do with API development.

a REST API is nothing more than a service that supports systems consuming its resources in accordance with core REST principles. In some cases, how we interpret REST principles may need to adjust slightly in an API context, but a truly RESTful API still uses those REST principles as their foundation.

six REST constraints: • Client/server • Stateless • Caching • Uniform Interface • Layered System • Code on Demand

(the client and the server) should be able to evolve independently from one another. In other words, the system divides processing between a client and server with limited, well-structured modes of communication between them. The client doesn’t care about the many complexities of the server, and the server doesn’t care about the technologies supporting the client.

Obviously they have to agree on the schemes and endpoints of the API, but not about implementation. or even business logic?

Statelessness means that a RESTful system does not require the provider to track any state. The consumer holds all responsibility for state management.

When you have no state, you can have any number of machines handling requests from any number of consumers and not care what consumer is making what request of what machine or in what order.

So aside from session validation across a pool, can you really keep every call completely stateless? If your API does not, what issues does this reveal about coupling and dependencies within your workflow?

Caching states that a provider should indicate to a consumer which requests may be cached so that consumers are free to avoid repeating the same request to a provider within the cache lifetime.

How many of us ever design services with a focus on caching the requests? Invalidating the cache when entities change?

an API generally has a smaller population of resources that are cacheable than an interactive client/server environment.

Not everything can be cached, e.g. CRUD operations. But most static, config, and high-level objects rarely change - where do you draw the line?

The uniform interface constraint means that your service enables you to view representations of resources and interact with them through a generic, finite set of requests. In an applied context, it means you have resources that can be referenced via a URI and operated on through HTTP verbs in a manner consistent with those HTTP verbs. We’ll talk a lot about this particular constraint in this book. For now, let’s just say that if your API is re-defining the HTTP verbs or if it is assigning new meanings to HTTP status codes or making up its own status codes, it is not RESTful.

How many people really use PUT, DELETE, or anything else besides GET and POST? Collapsing the API to use these verbs reduces the # of endpoints and makes it easier to have a consistent schema across endpoints. What other problems does it solve? Or create?

a RESTful service must be capable of being implemented as a layered system in which the layers are invisible to the consumer.

So load balancers and session pool persistence should be totally invisible to the client. How feasible is this in the real world?

Code on demand means that you can extend the behavior of the consumer of the RESTful services by serving it executable code. In a browser context, a Java applet or JavaScript counts as “code on demand”. In an API context, it may or may not be advantageous to support this constraint depending on what you are trying to do.

You have one URI with a finite set of verbs you can use to manipulate it and its representations. The server doesn’t care how the client displays the resource or enables a user to interact with it and the client doesn’t care what the “real” resource is one the server or the technologies used to manipulate or store it. In case of an error, you also have a finite set of error codes that you need to be able to understand. The exact same ideas should apply in an API context. Your API should be defined in terms of POST, GET, PUT, DELETE (and maybe other) HTTP verbs using HTTP status codes. Unfortunately, most “RESTful” APIs don’t do any of this. In fact, the biggest misconception is that you can call your API RESTful if its delivering XML or JSON over HTTP. And that misconception and the resulting proliferation of horrible APIs is likely the source of RESTafarian irritation with the application of the term “RESTful” in an API context. But let’s be brutally honest. Most interactive systems don’t follow this model either. Common web interaction models include: • Performing a GET from one URI and then PUT to another URI • Or, more likely, executing a POST for that update against the second URI • Doing a GET against one URI to perform one kind of operation on a resource (e.g. change password) and then doing another GET against a second URI to perform another kind of operation (change phone number) on the same resource There are a lot of reasons why systems are designed this w...

Do you generally know when a given web server GETs a resource from a different URI than it PUT—or POSTs—to? Pretty much never. Someone consuming an API will definitely notice when the provider of that API is diverging from RESTful principles and it will seriously complicate that consumer’s life.

One of the biggest differentiators is the idea of being “service-based” versus “resource-based”. A service-based API exposes an arbitrary set of actions you may remotely trigger. SOAP is like this.

A resource-based API is one in which you manipulate resources through a generic interface. REST is the prime example of a resource-based API. You have a resource identified through a URI and a known set of actions you can take on that (or any other) resource. The API designer has much less discretion with a resource-based approach than a service-based approach. In short, REST is based on the idea that constraints on design are a good thing.

Under a RESTful approach, there would be a single instance behind a single URI, Instance. You would POST to Instance to start up a VM, you would GET against Instance to list all VMs, and you would DELETE against Instance to terminate a VM. These operations would similarly apply to all other resources supported by the API.

It’s very common to see APIs that mix the resource and services metaphor, define their own status codes, or confuse POST vs. PUT or worse. In many cases, these APIs are trying to be RESTful but failing.

Does REST necessarily include a discovery endpoint? Good idea of course, but not part of the definition? Something like Swagger / OpenAPI makes this easy

I tend to prefer SOAP in a very few narrow use cases that fit the following criteria: • I have known language homogeneity and that language has solid SOAP support • My audience is limited (in other words, not some generic, publicly consumable service) • Delivery time is absolutely critical With SOAP, you don’t have to spend too much time modeling your services. If you are writing in a single language, you can just define an interface, implement it, and move on. It hardly looks like a distributed system at all. SOAP starts to fall apart, however, when multiple languages come into play or cost of maintenance becomes a factor. While SOAP is meant to be language agnostic, the reality is that many commonly used languages do not have very well evolved tools for building SOAP-based applications. Furthermore, as time goes on, the lack of control over your SOAP infrastructure (and the lack of planning often associated with SOAP services) can slowly increase your technical debt.

As far as REST versus other non-RESTful web services strategies, the complexity argument still applies. The problem with any service-based approach like AWS is that you have to learn a whole taxonomy for interacting with the system.

Whatever the subject, all truly RESTful APIs should look basically the same. Of course, if you have any experience consuming REST APIs, you’re probably doubting my sanity because very few of them look anything alike.

Is a standard API design consistent with “screaming architecture”? Should it be?

you can discover all services and resources within those services starting from a single logical endpoint. In an ideally implemented REST API, the service endpoint is the only piece of information you need to give a developer to enable them to build a system against that API.

Does this sound realistic? Should the service index really include enough detail for a consumer to implement an app without any other knowledge of the business rules or domain objects?

The biggest mistake you can make in resource modeling is to tightly couple the model of the resources in the API to their underlying implementation. That includes the population of resources, their attributes, and the operations you can perform on them.

Modeling resources takes into account all of the following concerns: • Geographic distribution—are your resources going to be split across geographies at different service endpoints and how visible should this geographic distribution be? • Resource hierarchy and namespace—do you have a small problem domain that can fit into a flat hierarchy or do you need a more complex namespace? • Representation—what are valid representations of your resources and what attributes should be exposed in those representations?

Your internal representations need the ability to evolve independent of the external representations. Otherwise you will have a versioning nightmare on your hands. Take control over your conversion from internal to external representation.

Does this imply hand-rolling your entire set of API objects/classes and the conversions to/from business objects? Or just defining a clear interface and an auto-mapping library?

With HTTP, you have a finite set of methods to work with and your API must remain faithful to the standard meanings of those methods.

Otherwise, it’s not RESTful.

the four main methods: GET Fetches a representation of the named resource without any side-effects. Two consecutive GETs should always provide the same result. POST Creates a new resource from an external representation at the named point. A POST is not for making changes to an existing object. PUT Executes updates to a resource from an external representation of that resource. DELETE Deletes the target resource from the system. There are other methods available, but they are generally not strictly necessary to create a complete REST API. In fact, not every resource will be impacted by the four core methods.

These map exactly to CRUD operations.

I need to determine in what ways I will allow people to manipulate the resources in the system. Can people delete products from the system? If so, the API should support the DELETE HTTP method. It should not define some other mechanism for deleting products. If I don’t want to allow deletes, I will not support a DELETE operation and I will return the proper status code (405) upon a DELETE request.

Both JSON and XML have strengths for specific problem domains. They both have one extraordinary weakness: there’s always a large population of people religiously opposed to each format. XML people tend to hate JSON. JSON people tend to hate XML. Understanding the religious nature of this debate must play a critical role in your decision as to how you represent your objects. Your personal preference should generally play absolutely no role.

Pick the right tool for the job. Browser clients will generally be fit from JSON since that is their native format, but there’s no reason the same endpoints couldn’t return XML to a mobile app or another server based on a format header.

minimizing versioning headaches is one of the most important things to achieve in an API—REST or otherwise.

As the API designer, your preferences don’t really matter. What are the preferences of your intended consumers? The answer may be simple if you have a very narrow audience. In anything but the most narrow environments, however, the answer is more complex.

If you are developing a public service for a global audience, I honestly believe it’s absolutely imperative to support at least JSON and XML and a really, really good idea to also support HTML. The only scenario in which I believe in picking one and going with it is when you are developing a service purely for internal consumption.

As consultants we tend to be working on projects intended only for internal consumption, but that’s probably true of most devs and consumer products that are just providing a back end for their apps. B2B apps and SaaS offerings are far more likely to need an API as part of the business model, so that others can build their own client apps.

Best practice is to use the HTTP status code as the general error category and include a more detailed error description in the response body.

1. An API must provide full problem domain coverage 2. An API must be fully backwards compatible indefinitely 3. A REST API must adhere to the REST constraints 4. A REST API is hypertext driven If you cannot get #1 or #2 right, the whole debate over what is truly RESTful is unimportant.

Really fully backwards for all time?

In its purest form, “hypertext-driven” means a client should know only a base endpoint and supported media types and be able to discover everything else through references to URIs.

How often do we really expect an API to really be able to discover all endpoints and resources on its own? Or to implement business logic against autonomously discovered resources?

the idea that all you need is an endpoint and a set of media types is a complete myth—even if with a browser pointing to a web site. A client will need to understand the URI namespaces in use. In addition, because the media delivered through an API is generally some kind of structured XML or JSON, how to interpret that XML or JSON. That often includes understanding the relative naming within a URN/URL namespace for locating referenced objects.

Authentication is a tricky problem for REST web services. On the one hand, HTTP provides three different authentication mechanisms and thus the default thinking should be to leverage one of those three approaches. It turns out, however, that the problem of identity management and access control for system-to-system communication is very different than the user-to-system identity management around which the HTTP authentication models were developed. As a result, you generally need to define your own.

This does *not* mean roll your own!!

System-to-system communication often operates over plaintext channels or SSL connections not governed by signed SSL certificates

Says who? APIs can be locked down to TLS-only access. Is he talking about 3rd-party integrations like Mint needing plaintext credentials? That’s still user to system initially, inter-system requests should always be secured! What HTTP-based protocol is he thinking of that doesn’t support a security layer?!

However, digest authentication still must occur over an SSL channel using signed server certificates in order to avoid man-in-the-middle attacks. A man-in-the-middle can force clients into basic authentication, gain access to the MD5 hash of the authentication credentials, and alter the request/response payload.

User names and passwords are inherently weak, low entropy means of authentication. The only reason we tolerate them in an interactive context is because the memory burden on humans to remember them is so heavy. No such burden exists in system-to-system communications.

A user name/password system is difficult to manage because there’s no meaningful way to expire keys. If you feel your password has been compromised, you will have to change the password in every consuming system after you change it with the provider. During the period between the time you change it with the provider and you change it with a consumer, the consumer will be broken.

This can be mitigated by enabling both sets temporarily, but this doesn’t work for immediate scenarios like a breach.

The primary advantage that token authentication has over “credentials in request” is that, under ideal circumstances, you aren’t sending your authentication credentials over the wire very often. How well this actually works depends on how readily the consuming system can maintain state. If the consuming system is stateless, token authentication simply becomes redundant overhead since every API request becomes two API requests: one for authentication and one for the real request. On the other hand, if your consuming application is allowed to maintain state (and there are good reasons why it should not be expected to maintain state), you may end up with only minimal overhead and only the one authentication call will contain the actual authentication keys. Nevertheless, this model still preserves passing in sensitive, predictable keys in every request. The token just has the advantage of having a limited lifetime.

Don’t ever use user names and passwords for anything other than authenticating interactive users with your web applications or identity management systems. If it’s someone else’s web application, tie it into your identity management system or use OAuth. If it’s a non-interactive system or a native third-party interactive application, use API keys.