Structural Design for APIs
There are at least four different ways to structure your API designs and JSON over HTTP is not the interesting approach.
I’ve spent quite a bit of time reading, using, and designing APIs and, like so many other things in life, you do something often enough and you start to see some patterns emerge. And what I call “Structural Design” for APIs is the pattern I’ll discuss here.
The Message is the Medium
When someone wants to create an API for the web, the most common approach is to focus the work on the messages that get passed back and forth. And, as of this writing, the most popular message format for APIs is JSON. So, a simple “To-Do List” API might have a message that looks like this:
{"todo": [
{"id" : "...", "title": "...", "status": "..."},
{"id" : "...", "title": "...", "status": "..."}
...
]
}
Quite a few things are implied here:
there is an object called “todo” that has three elements
the messages passed between machines are arrays of todo objects
It’s worth pointing out that there is also some assumed set of actions (add, edit, delete, filter, set status, etc.) and that those possible actions are not listed in the message. This is not, in the parlance of the web, a self-desribing message. We can also assume there is human-reable documentation for all the missing elements.
Finally, we can expect that humans will write code for both clients (API consumers) and servers (API providers) that internalize the details of the todo object and all the actions.
To sum up, we see a message, often supported by a schema document (JSONSchema, XMLSchema, RELAX NG, etc.) — the “rest” is somewhere else (LOL).
There are lots of examples of this approach such as GEO+JSON, Atom, SDP and others. The Internet Assigned Numbers Authority (IANA) keeps a long list of registered message designs and most of them are domain-specific.
However, this focus on messages leaves a lot to be desired. When we change the shape of the object, our foundational element — our structure — also changes. That’s a shakey foundation upon which to build our house!
We do, of course, have other options…
Format Follows Function
One way to improve the stability of our foundation is to start with a more solid struture. What would that be? How about the approach that the WWW took thirty years ago? We’ll create a generalized message that everyone can use, not matter what content they want to share. We’ll use a hypertext markup language (let’s call it HTML) as our message.
Now, our “ToDo List” messages look like this:
<html>
<body>
<ul>
<li>
<span name="id">...</span>,
<span name="title">...</span>,
<span name="status">...</span>
</li>
<li>...</li>
...
</ul>
</body>
</html>
This structure certainly a bit more verbose, but clearly has some advantages. Now I can write an API Consumer application that focuses on the message format instead of just the message content. And, if/when I add new fields in the future (e.g. “dueDate”), the API Consumer is less likely to break.
Even better, since the message structure is no longer limited to todo objects, I can add more metadata in the responses. Like the action information we left out earlier.
<html>
<body>
<ul>
<li>...</li>
<li>...</li>
</ul>
<form name="filter" action="http://todo.example.org">
<input name="search" />
<input type="submit" />
</form>
</body>
</html>
There are a handful of message formats gaining popularity in the API space including HAL, SIREN, CollectionJSON and others. Using these “structured format” message models makes it easier to build stable consumers and providers that can support minor changes in content without experiencing breaking changes.
However, applying application-level semantics (“id”, “title”, “status”, etc.) to structured formats introduces an additional level of abstraction to your designs. HTML has handled this “added cost” quite well for over three decades but there are other options.
For example, instead of focusing structural design on the message, you can move it further down the stack — to the protocol that ships those messages around.
Observing Proper Protocol
Another approach to creating machine-to-machine level exchange is to design application at the protocol level. IOW, to focus the structural efforts on the commands in the protocol, not the content of the messages.
Most of my readers has probably spent the bulk of their API time using the Hypertext Transfer Protocol (HTTP). The common assumption is that you can’t/shouldn’t deviate from that. But recently there has been a renewed interest for creating APIs for other protocols.
MQTT has been around almost as long as HTTP and there are others like Thrift (created at Facebook) and CoAP. While most of these protocols are still rather generic (like HTTP), that’s just a design choice.
For example, it would not be too hard to design an application-level protocol focused in the “ToDo List” domain. It might look like this on the wire:
LIST /todos/
WRITE /todos/1 title:... status:...
FILTER /todos/ f=...
COMPLETE /todos/1
ASSIGN /todos/1 marcia
… and so on.
When you focus your structural design at the protocol level, you can greatly simplify your messages and improve the interaction experience. As you can see from the example above, you have the freedom to make the protocol itself “domain-aware”. No more POST and PATCH and GET — now you can use well-understood commands like LIST and FILTER and CREATE.
Programmers might see this as an API equivalent of a DSL (domain-specific language).
The good news is that adding new commands (like that ASSIGN command I have above) has a relatively low impact on running API consumers. The downside, however, is that any changes at the protocol level need to be adopted by API Consumers before these changes can be safely rolled out to the community at large.
One of the communities that has taken this approach and flourished for over 25 years its the IRC group. A quick scan of their protocol documents reveals the dozens of commands added over the years to enhance the ecosystem. Other examples where the application domain is embedded into the protocol include FTP, SMTP, and POP3.
Welcome to the Jungle
There is at least one more approach, that of using format-specific messages along with domain-specific annotation. This is the approach the RDF community has taken.
For example, there are a handful of general formats (JSON-LD, Turtle, n3, etc.) to choose from. And there lots and lots of application-specific vocabularies (FOAF, DOAP, vCARD, XFN, etc.) you can use to decorate a generic format with domain specifics.
Here’s an example of JSON-LD with FOAF annotations:
{
"@context": {
"name": "http://xmlns.com/foaf/0.1/name",
"homepage": {
"@id": "http://xmlns.com/foaf/0.1/workplaceHomepage",
"@type": "@id"
},
"Person": "http://xmlns.com/foaf/0.1/Person"
},
"@id": "https://me.example.com",
"@type": "Person",
"name": "John Smith",
"homepage": "https://www.example.com/"
}
The good news in this approach is that designing a vocabulary (or ontology as it is often called) is a much easier task than creating (or updating) a new protocol or generic format. The downside is that, often, in order to get something meaningful done for a user application, you need to employ more than one vocabulary. Many people who look at RDF comment that it seems you need to do a lot of initial “heavy lifting” before you can start creating API Consumers for this kind exhange approach.
And So…
So, it turns out, there are several possible ways to design APIs to solve your problem. We saw four of them here in this post:
Domain-specific protocols (like IRC and SMTP)
General formats (like HTML and HAL)
Domain-specific message schema (like GEO-JSON and Atom)
Domain-specific annotations over general formats (like FOAF and XFN)
The key takeaway here is that we don’t need to just use HTTP and JSON for every new problem that comes along. And, while protocols like CoAP and formats like UBER might not be “the popular thing to do” right now, there may come a time in your career when applying your domain structure to something other than a generic protocol (HTTP) and a generic message format (JSON) will be a good idea.
And now, if and when that day comes, you’ll know what to do<g>.