< Strange Horizons Reviews "Constellation Games"
new robotfindskitten! >

[Comments] (6) RESTful Web APIs: I guess it's time to announce my next book project. Mike Amundsen and I are working on a book for O'Reilly called RESTful Web APIs, a follow-up to 2007's RESTful Web Services. Our deadline is the end of March 2013, and the book should come out a few months later. (Here's Mike's announcement of the project.)

Here's my announcement of RESTful Web Services from about six years ago. You can see how far we've come since then.

In 2008 I came up with the now-famous "maturity heuristic" (aka the "RMM") for quickly judging the quality of an API. If you go back and look at RWS through the lens of the maturity heuristic, you'll see that it spends a lot of time trying to get the reader to grasp the concepts behind Level 1 services: that URLs identify resources, and that resources are the proper targets of HTTP requests. That's because back then, Level 1 was about the best you could expect from a "REST" design.

RWS spends a whole chapter treating Amazon S3 as though it were a great example of RESTful design, because in RMM terms it made it all the way to... Level 2. RWS spends another chapter arguing for the legitimacy of REST against a thriving part of the industry that's content at Level 0.

We don't live in that world anymore. I'd say the modal RMM level of a new "REST" API today is 2. Most of the 2007-era arguments have been rendered irrelevant (SOAP vs. REST, the usefulness of PUT) or been given generally accepted best practices (URL formats). Restful Web Services was the first book-length treatment of REST; there are now over ten books on REST from O'Reilly alone. There are lots of development tools that make it really easy to crank out a Level 2 service.

In short, a lot of gravel- and rock-sized problems have been excavated, revealing two boulder-sized problems. I mentioned these two problems in my QCon talk "How to Follow Instructions" (available as video in hour-long version and in a half-hour condensed version presented at RESTFest; see also my QCon slide deck and speaker notes). Now I'll state them explicitly as the two major problems I want to address with RESTful Web APIs.

Duplication of effort

Here's an article from May on Programmable Web, a great site that tracks the world of public APIs: "53 Microblogging APIs: Twitter, TwitPic and What The Trend". Fifty-three was really the number of organizations publishing microblogging APIs, and some of them are utility APIs rather than "microblogging APIs" per se, but after doing some spot checks I do believe there were approximately fifty-three distinct APIs.

Should there really be fifty-three distinct APIs in this field? What if there were, like, five? We're not talking about something complicated, like insurance policies or regulatory compliance. We're talking about posting a little bit of text to a user account. Can we at least get down to ten?

No, apparently not. Like I said, that number was from May. There are now fifty-seven microblogging APIs. Well, damn. What if we got together and hammered out a standard, interoperable microblogging API? Sounds great, except nobody's interested. Evan Prodromou tried that with OStatus, and nothing came of it. You could use AtomPub as a microblogging API, pretty much unmodified, but nobody seems to want to.

The original RESTful Web Services sets out a design procedure for taking your problem domain and turning it into an API. But there's a big problem with this procedure: it doesn't consider the possibility that fifty-six other people may have already done the work for you.

Back in 2007, that possibility was remote. There was one microblogging API. I had to scrape the barrel pretty thin to find enough quasi-RESTful example APIs to fill Appendix A. Now the big problem is too many APIs. When you design the fifty-eighth microblogging API you're limiting your audience and wasting your users' time.

This is a really huge problem and we won't solve it with a book. But we can point out that it's a problem and take the first step towards mitigation. In RESTful Web APIs we present a new design procedure that focuses on reusing existing designs, coming up with new stuff to fill any gaps, and then publishing the new stuff for the benefit of a) your users and b) API developers who come after you.

Hypermedia

Whatever influence RESTful Web Services has had, it stops dead at RMM level 3, the hypermedia level. In 2012, the modal new "REST" API is at level 2. It has been level 2 for a while, and will continue to be level 2 unless something changes. Most architectures don't support hypermedia very well (or at all), and a lot of developers still don't understand it.

That's understandable. To the programmer brain, hypermedia is weird. When you read RWS it's possible to just skip the parts about hypermedia, and I get the feeling a lot of people did. There's stuff in RWS that is now outdated, and stuff that was useful for dealing with 2007's problems but that I now consider just plain wrong. Most of that stuff has to do with hypermedia.

RESTful Web APIs is tightly focused on hypermedia--you will not be able to skip the "hypermedia parts"--and it deals with 2013's problems. Some specific shortcomings of RWS we'll address:

  1. We as developers have a strong desire to export our server-side object models as APIs. This is the idea behind JAX-RS. It's the idea behind lazr.restful, the framework I developed to make the Launchpad web service. But it's also the idea behind SOAP, and it's how we got fifty-seven microblogging APIs.

    This desire isn't technically at odds with the use of hypermedia, but it does tend to use hypermedia only to describe safe state transitions (i.e. to link resources together), and RWS didn't really go further than that.

  2. In 2007 there were no hypermedia formats based on JSON, but JSON-based representations were becoming overwhelmingly popular. RWS does not offer advice for this scenario. Developers either made up their own one-off hypermedia formats (as I did with the Launchpad web service), or, more commonly, ignored the hypermedia constraint. Today JSON has a number of general-purpose hypermedia formats (like Siren, Collection+JSON, and HAL), and we'll cover them in RWA.
  3. In RWS I was very dogmatic about avoiding overloaded POST. I felt strongly about this because I really needed to yank people out of the SOAP-RPC mindset. But I've come to accept that overloaded POST exists for a reason: to convey state transitions that don't map naturally onto one of the other HTTP methods.

    These state transitions are legitimate if they are described by hypermedia controls, as they are on the Web. They're not just hacks to get around the fact that HTML can't describe PUT and DELETE methods. Prohibiting overloaded POST in the name of the uniform interface forces designers into complicated workarounds to minimize the difference between their application semantics and the protocol semantics of GET/PUT/POST/DELETE. Those workarounds usually reflect the object model the designers already defined for their CRUD databases. Or, the designers give up, use overloaded POST without using hypermedia to describe the state transitions, and feel guilty about it.

    Besides which, forbidding overloaded POST means that you can't use HTML, the world's most popular hypermedia format. That's a big problem.

The title

As I said, this is a brand new book, not a second edition of RESTful Web Services. The only major things we're reusing are the appendices dealing with HTTP headers and response codes. To signify the handoff, we're changing the name.

RESTful Web Services was a great name in 2007, but it's not so good now. See, when SOAP went down, it took the term "Web Service" with it. "Web Service" is now an enterprise software term. Everywhere else it's all "API API API".

As I mentioned last year, this is backwards. The SOAP approach is to make a remote service look like a local library. (This idea is not limited to SOAP, of course.) Whereas the RESTful believe that a remote service should look like the Web. (Whatever that means.)

But I'm not gonna re-litigate that argument. "Web Service" is dead, we're stuck with "API", and the name of the book needs to change. "RESTful Web APIs" is a direct translation of "RESTful Web Services" into 2013-speak. It keeps continuity with the old book, but it's a new title for a new book.

Interested?

If you want to be a beta reader, let me know (via comment or email to leonardr@segfault.org). I'll get in touch with you around the end of March, or possibly earlier. Update: We've got a lot of beta readers and we're no longer accepting applications.

Once in a while I add a section to a chapter called "The Hypermedia Zoo", which takes a not-too-detailed look at a lot of hypermedia-aware data formats to demonstrate their diversity. I'm always on a low-level search for more hypermedia formats, and if you name your favorites in the comments I'll take a look. Don't worry about duplicating my list; I also want to see what springs to mind when you think "hypermedia format".

There's also a shorter section called "The Semantic Zoo" which talks about interesting domain-specific data standards. I'm mostly interested in reusable profiles (the IANA registry of link relations, microformats, schema.org microdata, Dublin Core), but I'm also interested in pointers to standalone data formats like Activity Streams. There's a whole lot of stuff out there and I'm not confident that I've seen even half of it.

Filed under: ,

Comments:

Posted by Glenn Block at Fri Dec 21 2012 11:50

Exciting to see this, so is this what you guys were plotting at the table of QCon NYC? :-)

Posted by Leonard at Fri Dec 21 2012 11:56

We plotted it after hours, but yeah.

Posted by Lex Slaghuis at Mon Dec 24 2012 11:47

Hi can you alwso explain that an API is not always the same as open data?
-(apikeys)
-(replicatable)
-historic info is often missing.

I get sucky API implementations that I need to 'harvest' to get the data from my gov that I need.

ThankS!

Posted by Leo Simons at Wed Dec 26 2012 05:30

Exciting news! I've been getting people into the right kind of web programming with RESTful web services ever since it came out, which I find in 2011/2012 has been getting harder as a lot of the web is now somewhat better than what's in the book. We need this update.

I imagine you'll talk about OAuth 'n stuff. From the trenches, OAuth still isn't as easy as it should be with all the slight quirks and incompatibilities and low quality and/or outdated libraries; I hope you guys can show us a reasonable way.

I do hope you also go out on the edge and talk about hypermedia forms and querying, if only so people have some new stuff to skip over. RESTful web services probably gave many people hypermedia as their stretch goal. Assuming RESTful web APIs does as well, hypermedia will happen, and we'll need some new stretch goals :-)

Posted by Markus Lanthaler at Wed Dec 26 2012 08:31

Excited to hear that.. I would definitely be willing to review the book or parts of it. You might be interested to add JSON-LD to your Hypermedia Zoo. The plan is that it goes Last Call at the end of January.

Posted by Eric Budd at Thu Dec 27 2012 01:39

Excellent! I'd definite be interested in reading through the beta. :)


[Main] [Edit]

Unless otherwise noted, all content licensed by Leonard Richardson
under a Creative Commons License.