News You Can Bruise

But ten years later, Whisper makes it cheap and easy to do basic audio transcription with a laptop. I've used Whisper to transcribe my talk and edited it into what it should have been. Some of the talk has aged poorly: the same underlying technology that transcribes the audio also makes it possible for a computer to follow some of the human-readable "instructions" I mention in the talk. But I think it was pretty prescient at describing what was happening in the world of APIs and where we've gone over the intervening decade.

First, let me make it super clear that there is no longer any need to buy Services. It's out of date and you can legitimately get it for free on the Internet. O'Reilly is taking Services out of print, but there's going to be a transition period in which copies of the old book sit beside copies of the new book in Barnes & Noble. Don't buy the old one. The bookstore will eventually send it back and it'll get deducted from my royalties. If you do buy Services by accident, return it.

If you're not specifically interested in the difference between the old book and the new one, I'd recommend looking at RESTful Web APIs's chapter-by-chapter description to see if RESTful Web APIs is a book you want. As to the differences, though, in my mind there are three big ones:

1. The old book never explicitly tackles the issue of designing hypermedia documents that are also valid JSON. That's because JSON didn't become the dominant API document format until after the book was published. If you don't know that's going to happen, JSON looks pretty pathetic. It has no hypermedia capabilities! And yet, here we are.

   In my opinion, a book that doesn't tackle this issue is propping up the broken status quo. RESTful Web APIs starts hammering this issue in Chapter 2 and doesn't let up.

2. There are a ton of new technologies designed to get us out of the JSON trap (Collection+JSON, Siren, HAL, JSON-LD, etc.) but the old book doesn't cover those technologies, because they were invented after the book was published. RESTful Web APIs covers them.
3. New ideas in development will, I hope, keep moving the field forward even after we all get on board with hypermedia. I'm talking about profiles. Or some other idea similar to profiles, whatever. These ideas are pretty cutting edge today, and they were almost inconceivable back in 2007. RESTful Web APIs covers them as best it can.

Now, for details. Services was heavily focused on the HTTP notion of a "resource." Despite the copious client-side code, this put the focus clearly on the server side, where the resource implementations live. RESTful Web APIs focuses on representations—on the documents sent back and forth between client and server, which is where REST lives.

The introductory story from the old book is still present. Web APIs work on the same principles as the Web, here's how HTTP works, here's what the Fielding constraints do, and so on. But it's been rewritten to always focus on the interaction, on the client and server manipulating each others' state by sending representations back and forth. By the time we get to Chapter 4 there's also a pervasive focus on hypermedia, which is the best way to for the server to tell the client which HTTP requests it can make next.

This up-front focus on hypermedia forces us to deal with hypermedia-in-JSON (#1), using the tools developed since 2007 (#2). The main new concept in play is the "collection pattern". This is the CRUD-like design pioneered by the Atom Publishing Protocol, in which certain resources are "items" that respond to GET/PUT/DELETE, and other resources are "collections" which contain items and respond to POST-to-append.

We covered AtomPub in Services, but over the past six years it has become a design pattern, reinvented (I think "copied" is too strong a word) thousands of times.

RESTful Web APIs focused heavily on the collection pattern, without ever naming it as a pattern. I'm not dissing this pattern; it's very useful. I'd estimate about eighty percent of "REST" APIs can be subsumed into the collection pattern. But REST is bigger than the collection pattern. By naming and defining the collection pattern, we gain the ability to look at what lies beyond.

Attempts to encapsulate the collection pattern include two new JSON-based media types: Collection+JSON and OData. The collection pattern also shows up, more subtly, in the Siren and Hydra formats. Which brings me to the second major change.

In 2007, there were two big hypermedia formats: Atom and HTML. Now there are a ton of hypermedia formats! This is great, but it's also confusing. In "The Hypermedia Zoo", Chapter 10 of RESTful Web APIs, we give an overview of about two dozen hypermedia formats. The ones we seriously recommend for general use (HAL, Siren, HTML, JSON-LD, etc.) are covered in more detail elsewhere in the book. The quirkier, more specialized media types just get an exhibit in the zoo.

Now for the third new thing, profiles. If you go through the RESTful Web APIs narrative from Chapter 1 to Chapter 7, you'll see that we introduce a problem we're not able to solve. Hypermedia is great at solving the following problem:

How is an API client supposed to understand what HTTP requests it might want to make next?

But there's a superficially similar problem that hypermedia can't solve:

How is an API client supposed to understand what will happen in real-world terms if it makes a certain HTTP request?

How do you explain the real-world semantics of an HTTP state transition? Before chapter 8, the two solutions are to do it ahead of time in one-off human-readable documentation; or to define a domain-specific media type, a la Maze+XML. Both of these approaches have big problems. Chapter 8 introduces profiles, which lets you get some of the benefits of a new media type without doing unnecessary work.

Maybe profiles will turn out not to be the right answer, but we gotta solve this problem somehow, and the old book is not equipped to even formulate the problem.

There are also a few additions to the book I consider minor. There's a whole chapter in RESTful Web APIs on Semantic Web/Linked Data stuff; in Services there was nothing but a cursory discussion of RDF/XML as a representation format. There's a chapter in RESTful Web APIs about CoAP, which didn't exist in 2007. These are good chapters that took me a long time to write, but I don't think it's worth buying the book if you only want to read the chapter on CoAP. (Or maybe it is! There's not a lot of competition right now.)

So, what hasn't changed? HTTP hasn't changed all that much. RESTful Web APIs's information about HTTP has been brought up to date but not changed significantly. So if you were using Services solely as an API-flavored HTTP reference, you don't need the new book. You can just read up on the protocol-level additions to HTTP since 2007, like the Link header and standardized patch formats for PATCH.

Hopefully this helps! RESTful Web APIs has a lot of distinguished competition that the old book didn't have, but its competition is newer books like Designing Hypermedia APIs and REST in Practice. If you compare APIs to Services I think it's no contest.

I don't think there's anything in the article you won't find in the RESTful Web APIs introduction and my discussion of my RESTFest talk, but I wanted to let you know about it and provide a forum on NYCB for asking me questions/taking issue with my assertions.

If you've bought RESTful Web APIs (and if you haven't, you should), you may have noticed that we promise that this will happen in a footnote of the Introduction. It took a while to get the contract amended, but now it's all complete.

Here's a direct link to the PDF in case you just want to grab the book instead of hear me talk about it.

Obviously I think the new book is a lot better than the old book, but the old book is still very good. The source code is long obsolete (this is why RWA contains no source code, only messages sent over the wire), but the sections on HTTP still hold up really well. A lot of RWS Chapter 8 went into RWA Chapter 11. With a few edits and additions, RWS Appendix B and C became RWA Appendix A and B. Those are the only bits of RWS that I reused in RWA.

From my vantage point here in 2013, my main critique of RWS is that it makes HTTP do too much of the work. It focuses heavily on designing the server-side behavior of resources under a subset of the HTTP protocol. I say "a subset" because RWS rules out overloaded POST ahead of time. You don't know what an overloaded POST request does. It's a cop-out. You're sweeping something under the rug. It's better to turn that mystery operation into a standalone resource, because at least you know what a resource does: it responds to HTTP requests.

In retrospect, RWS is that way because in 2007 hypermedia data formats were highly undeveloped whereas HTTP was a very mature technology. Nowadays it doesn't matter so much whether an HTTP request uses POST or PUT, so long as a) the state transition is described with a link relation or other hypermedia cue, and b) the protocol semantics of the HTTP request are consistent with the application semantics of the state transition. That's why RWA focuses on breaking down a problem into a state diagram rather than a set of static resources.

So, RWS is very much a 2007 book, but that's the meanest thing I can say about it. A lot of it is still useful, it's historically interesting, and I'm glad to give it away. I'd also like to give my thanks once again to Sam Ruby and O'Reilly, for their work on RWS.

But right now, there's some more stuff I want to say about "LCODC$SSU", and some stuff I couldn't say in the talk due to the framing device.

When I first mentioned this talk to Mike Amundsen, he told me about Bret Victor's talk from 1974, "The Future of Programming", which Victor gave in July and which had a similar conceit. Victor is also a much better actor than I am, but I went ahead with my talk because wanted to do something different with "LCODC$SSU" than happens in "The Future of Programming". I get a strong "You maniacs! You blew it up!" vibe from Victor's talk. And there's some of that at the end of "LCODC$SSU"—I really feel like we've spent thirteen years making five years worth of progress, as you can see from my frustration at the beginning of "How to Follow Instructions"—but I also wanted to do some new things in my talk.

While writing Appendix C of RESTful Web APIs I came to appreciate the Fielding dissertation as a record of the process used to solve an enormous engineering problem. Comments from RESTFest attendees confirm that seeing it this way helps folks grasp the dissertation's gem: the definition of LCODC$SSU (a.k.a. REST). Thinking about it this way doesn't require a historical-fiction framing device (Appendix C has no such framing device), but it does require you stop treating the Fielding dissertation as a prescient guide to the 21st century and see it as a historical record of the 1990s.

And once you do that, the missing stair we've been jumping over or falling through for thirteen years becomes visible. The Web works because it has four domain requirements that reinforce each other: low entry-barrier, distributed hypermedia, extensibility, and Internet scale. But there's also a fifth implicit requirement: the presence of a slow, expensive human being operating the client and making the final call on every single state transition. In the talk I identified the inverse of this implicit requirement as an explicit requirement: "machine legibility". In RESTful Web APIs we use the term "semantic gap" to describe what happens when you remove the implicit requirement.

Making the human unnecessary on a transition-by-transition basis (the goal of "Web APIs" as a field) is a really difficult problem, and it's partly because of the phenomenon I describe in the talk and in RWA Appendix C. Getting rid of the human raises the entry-barrier dramatically. Looking around for a cheap way to lower the entry-barrier, we decide to get rid of distributed hypermedia. But distributed hypermedia is the only thing that allows Internet-scale and extensibility to coexist! We must lose one or the other. We end up with an increasingly ugly system that can never be changed, or else a fascist dystopia.

And here's the bit I couldn't put in the talk because it would break the framing device. We've seen a decade-long obsession with lowering entry-barrier at any cost, and although the cost has been enormous I can't really say the obsession is misplaced. Low entry-barrier is the main reason why the Web succeeded over all other hypermedia systems. Low entry-barrier drives adoption. You get adoption first and you deal with the other problems (which will be enormous) down the road.

Well, we're down the road. The bills are coming due. If we want this to go more smoothly next time, we need to stop chasing entry-barrier local minima and come up with a better solution. We need to make change easier so we can make progress faster.

The "machine legibility" problem will still be very difficult, and frankly I can't see a way to a complete solution. But there's cause for optimism: every step forward we've taken so far has illuminated the space a little more and made the next step visible.

It's always been this way. That's how hypermedia works. That's why I called my now-infamous 2008 QCon talk "Justice Will Take Us Millions Of Intricate Moves" (after William Stafford), and that's why I take my motto from a Johnny Cash song that's probably not on most peoples' list of inspirational Johnny Cash songs.

I built it one piece at a time.

I've submitted an erratum for the wrong URL, and I'm here to give you some temporary URLs that will work for the other stuff. They're temporary because Mike controls the DNS for restfulwebapis.org and youtypeitwepostit.com, and he's out of commission at the moment.

• My lousy temporary home page. (Links to the other stuff.)
• Sample application hosted on Heroku.
• The correct GitHub repository.

In fact, there's one subfield (profiles) where you could argue this book is premature. The way RESTful Web Services was a little premature in describing an OAuth-like system before OAuth was released. But I don't think we can wait any longer.

Back in February I discussed the differences between APIs and Services. That hasn't changed much, though we have added more stuff:

• A chapter on Linked Data, the Semantic Web approach to REST.
• A chapter on CoAP, the fabled "RESTful system that doesn't use HTTP", designed to connect embedded systems over low-power networks.
• An appendix that explicates the Fielding dissertation from an API designer's perspective.

This post is mainly my way of asking you to pre-order your copy of RESTful Web APIs through my O'Reilly affiliate link. That's a hypermedia-driven change in resource state which will get you the book in a couple weeks, and get me some extra cash. (I estimate about $1.70 extra. Don't do this if the shipping charge on a physical book is prohibitive, or whatever.)

But this post is also a back-door way for me to brag about what a great book Mike and I have written. You don't have to take my word for it. Here's the blurb we got from John Musser of ProgrammableWeb.

A terrific book! Covers a lot of new ground with lots of valuable specifics.

Here's Steve Klabnick of Designing Hypermedia APIs:

The entire time I read this book, I was cursing. I was cursing because as I read each explanation, I was worried that they were so good that it would be hard to find a better one to use in my own writing. You will not find another work that explores the topic so thoroughly yet explains the topic so clearly. Please, take these tools, build something fantastic, and share it with the rest of the world, okay?

You get the picture. I've tried to recreate the relevatory experience a lot of people got from RESTful Web Services, on a higher level, in a way that gives access to more powerful tools. Time will tell if I've succeeded, but I don't think I, or anyone, could have done much better. I'm really proud of this book, and I hope it helps you.

With the beta closed I think it's a good time to go into a little detail about the structure of the book. My guiding principle was to write a book that will be as useful now as RESTful Web Services was in 2007. Like RWS, RESTful Web APIs has a main storyline that takes up most of the book. My inspiration for the main storyline were a few books that followed RWS, notably REST in Practice and Mike's Building Hypermedia APIs with HTML5 and Node.

RWS focused on the HTTP notion of a "resource", and despite the copious client-side code, this put the conceptual focus clearly on the server side, where the resource implementations live. RWA focuses on representations, and thus on hypermedia, on the interaction between client and server, which is where REST lives. The stuff you remember from RWS is still here, albeit rewritten in a pedagogically superior way. Web APIs work on the same principles as the Web, here's how HTTP works, here's what the Fielding constraints do, and so on. But the focus is always on the interaction, on the client and server manipulating each others' state by sending messages back and forth.

We've also benefited from a lot of tech work done by others. The IANA registry of link relations showed that state transitions don't have to be tied to a media type. The RFC that established that registry also showed how to define custom state transitions (extension relation types) without defining yet another media type to hold them.

Insights like these inform the new parts of RWA's main storyline. What makes your API different from every other RESTful API in existence? That's the only part you really need to buckle down and design. Everything else you can reuse, or at least copy.

In particular, you shouldn't have to design a custom media type. Your API probably isn't that different from other APIs, and a ton of hypermedia formats and protocols have been invented since 2007. We cover a few of the most promising ones in the book's main storyline. We cover even more of them afterwards, mostly in the big "Hypermedia Zoo" chapter. Here's the book-wide list:

• Maze+XML
• Collection+JSON
• AtomPub
• OpenSearch
• HTML
• HAL
• Siren
• JSON-LD
• The HTTP headers Link, Link-Template, Location, and Content-Location
• URL lists
• JSON Home Documents
• Problem detail documents
• Hydra
• OData
• WADL
• Web Intents
• XLink
• XForms
• SVG
• VoiceXML
• GeoJSON (which is only a hypermedia format on a bizarre technicality!)
• RDF
• XRD/JRD
• Web host metadata documents
• WebFinger
• CoRE Link Format

After the main storyline and the hypermedia zoo, RWA continues the RWS tradition of giving an API-centric view of the HTTP standard. We have a "crash course in advanced HTTP" chapter, some of which is an update of Chapter 8 from RWS. (Look-before-you-leap requests never caught on, but I still feel like I have to describe them in RWA because I have no other source to refer you to!) Appendix A is an updated version of Appendix B from RWS, with the addition of these exciting new status codes:

• 308 ("Permanent Redirect")
• 428 ("Precondition Required")
• 429 ("Too Many Requests")
• 431 ("Request Header Fields Too Large")
• 451 ("Unavailable For Legal Reasons")
• 511 ("Network Authentication Required")

Appendix B is an update of appendix C from RWS, with these API-licious new HTTP headers:

• Link and Link-Template
• Prefer and Preference-Applied

The amount of reused material in RWA is really small, because the main storyline is completely rewritten for 2013. And I haven't even mentioned our coverage of profiles, partly because I can't yet think of a way to talk about profiles at less length than what we say in the book.

Here's my announcement of RESTful Web Services from about six years ago. You can see how far we've come since then.

In 2008 I came up with the now-famous "maturity heuristic" (aka the "RMM") for quickly judging the quality of an API. If you go back and look at RWS through the lens of the maturity heuristic, you'll see that it spends a lot of time trying to get the reader to grasp the concepts behind Level 1 services: that URLs identify resources, and that resources are the proper targets of HTTP requests. That's because back then, Level 1 was about the best you could expect from a "REST" design.

RWS spends a whole chapter treating Amazon S3 as though it were a great example of RESTful design, because in RMM terms it made it all the way to... Level 2. RWS spends another chapter arguing for the legitimacy of REST against a thriving part of the industry that's content at Level 0.

We don't live in that world anymore. I'd say the modal RMM level of a new "REST" API today is 2. Most of the 2007-era arguments have been rendered irrelevant (SOAP vs. REST, the usefulness of PUT) or been given generally accepted best practices (URL formats). Restful Web Services was the first book-length treatment of REST; there are now over ten books on REST from O'Reilly alone. There are lots of development tools that make it really easy to crank out a Level 2 service.

In short, a lot of gravel- and rock-sized problems have been excavated, revealing two boulder-sized problems. I mentioned these two problems in my QCon talk "How to Follow Instructions" (available as video in hour-long version and in a half-hour condensed version presented at RESTFest; see also my QCon slide deck and speaker notes). Now I'll state them explicitly as the two major problems I want to address with RESTful Web APIs.

Duplication of effort

Here's an article from May on Programmable Web, a great site that tracks the world of public APIs: "53 Microblogging APIs: Twitter, TwitPic and What The Trend". Fifty-three was really the number of organizations publishing microblogging APIs, and some of them are utility APIs rather than "microblogging APIs" per se, but after doing some spot checks I do believe there were approximately fifty-three distinct APIs.

Should there really be fifty-three distinct APIs in this field? What if there were, like, five? We're not talking about something complicated, like insurance policies or regulatory compliance. We're talking about posting a little bit of text to a user account. Can we at least get down to ten?

No, apparently not. Like I said, that number was from May. There are now fifty-seven microblogging APIs. Well, damn. What if we got together and hammered out a standard, interoperable microblogging API? Sounds great, except nobody's interested. Evan Prodromou tried that with OStatus, and nothing came of it. You could use AtomPub as a microblogging API, pretty much unmodified, but nobody seems to want to.

The original RESTful Web Services sets out a design procedure for taking your problem domain and turning it into an API. But there's a big problem with this procedure: it doesn't consider the possibility that fifty-six other people may have already done the work for you.

Back in 2007, that possibility was remote. There was one microblogging API. I had to scrape the barrel pretty thin to find enough quasi-RESTful example APIs to fill Appendix A. Now the big problem is too many APIs. When you design the fifty-eighth microblogging API you're limiting your audience and wasting your users' time.

This is a really huge problem and we won't solve it with a book. But we can point out that it's a problem and take the first step towards mitigation. In RESTful Web APIs we present a new design procedure that focuses on reusing existing designs, coming up with new stuff to fill any gaps, and then publishing the new stuff for the benefit of a) your users and b) API developers who come after you.

Hypermedia

Whatever influence RESTful Web Services has had, it stops dead at RMM level 3, the hypermedia level. In 2012, the modal new "REST" API is at level 2. It has been level 2 for a while, and will continue to be level 2 unless something changes. Most architectures don't support hypermedia very well (or at all), and a lot of developers still don't understand it.

That's understandable. To the programmer brain, hypermedia is weird. When you read RWS it's possible to just skip the parts about hypermedia, and I get the feeling a lot of people did. There's stuff in RWS that is now outdated, and stuff that was useful for dealing with 2007's problems but that I now consider just plain wrong. Most of that stuff has to do with hypermedia.

RESTful Web APIs is tightly focused on hypermedia--you will not be able to skip the "hypermedia parts"--and it deals with 2013's problems. Some specific shortcomings of RWS we'll address:

1. We as developers have a strong desire to export our server-side object models as APIs. This is the idea behind JAX-RS. It's the idea behind lazr.restful, the framework I developed to make the Launchpad web service. But it's also the idea behind SOAP, and it's how we got fifty-seven microblogging APIs.

   This desire isn't technically at odds with the use of hypermedia, but it does tend to use hypermedia only to describe safe state transitions (i.e. to link resources together), and RWS didn't really go further than that.

2. In 2007 there were no hypermedia formats based on JSON, but JSON-based representations were becoming overwhelmingly popular. RWS does not offer advice for this scenario. Developers either made up their own one-off hypermedia formats (as I did with the Launchpad web service), or, more commonly, ignored the hypermedia constraint. Today JSON has a number of general-purpose hypermedia formats (like Siren, Collection+JSON, and HAL), and we'll cover them in RWA.
3. In RWS I was very dogmatic about avoiding overloaded POST. I felt strongly about this because I really needed to yank people out of the SOAP-RPC mindset. But I've come to accept that overloaded POST exists for a reason: to convey state transitions that don't map naturally onto one of the other HTTP methods.

   These state transitions are legitimate if they are described by hypermedia controls, as they are on the Web. They're not just hacks to get around the fact that HTML can't describe PUT and DELETE methods. Prohibiting overloaded POST in the name of the uniform interface forces designers into complicated workarounds to minimize the difference between their application semantics and the protocol semantics of GET/PUT/POST/DELETE. Those workarounds usually reflect the object model the designers already defined for their CRUD databases. Or, the designers give up, use overloaded POST without using hypermedia to describe the state transitions, and feel guilty about it.

   Besides which, forbidding overloaded POST means that you can't use HTML, the world's most popular hypermedia format. That's a big problem.

The title

As I said, this is a brand new book, not a second edition of RESTful Web Services. The only major things we're reusing are the appendices dealing with HTTP headers and response codes. To signify the handoff, we're changing the name.

RESTful Web Services was a great name in 2007, but it's not so good now. See, when SOAP went down, it took the term "Web Service" with it. "Web Service" is now an enterprise software term. Everywhere else it's all "API API API".

As I mentioned last year, this is backwards. The SOAP approach is to make a remote service look like a local library. (This idea is not limited to SOAP, of course.) Whereas the RESTful believe that a remote service should look like the Web. (Whatever that means.)

But I'm not gonna re-litigate that argument. "Web Service" is dead, we're stuck with "API", and the name of the book needs to change. "RESTful Web APIs" is a direct translation of "RESTful Web Services" into 2013-speak. It keeps continuity with the old book, but it's a new title for a new book.

Interested?

If you want to be a beta reader, let me know (via comment or email to leonardr@segfault.org). I'll get in touch with you around the end of March, or possibly earlier. Update: We've got a lot of beta readers and we're no longer accepting applications.

Once in a while I add a section to a chapter called "The Hypermedia Zoo", which takes a not-too-detailed look at a lot of hypermedia-aware data formats to demonstrate their diversity. I'm always on a low-level search for more hypermedia formats, and if you name your favorites in the comments I'll take a look. Don't worry about duplicating my list; I also want to see what springs to mind when you think "hypermedia format".

There's also a shorter section called "The Semantic Zoo" which talks about interesting domain-specific data standards. I'm mostly interested in reusable profiles (the IANA registry of link relations, microformats, schema.org microdata, Dublin Core), but I'm also interested in pointers to standalone data formats like Activity Streams. There's a whole lot of stuff out there and I'm not confident that I've seen even half of it.

I was very nervous about the talk, because my work on Constellation Games has taken my creative attention out of the world of REST for the past couple years. To me the talk feels more like complaints from a user than advice from an expert. But it was well-received and I may be giving an updated version of "How to Follow Instructions" at REST Fest in September.

Because of that possibility, the text of the talk is still in flux and I'm not going to immediately put in the kind of work I did to get my 2008 QCon talk online with a transcript. But I have put up a PDF of my slide deck (4.0 megabytes), so you can see what I put on screen. And you can see the rhetorical structure of the talk by getting the LibreOffice Impress file (6.3 megabytes), which includes my speaker notes. Both are licensed under the CC BY-SA 3.0 license, and anything else I produce from this talk will be put under the same license. (As oppposed to stuff like QCon's video recording of me giving the talk, which they'll probably retain copyright on.)

I know from being on the other end of this that even with speaker notes, a slide deck after the fact is more a mnemonic for people who've already heard the talk than a way of conveying the knowledge contained in the talk. So this is more a show of good faith on my part than anything else. At some point you'll be able to read a transcript of this talk and reuse whatever parts of it turn out to be interesting, as happened with the 2008 talk. But for now the slides and notes are what I have to offer.

2023 update: It took over ten years, but you can now read this talk online.

Suffice to say that I'm gradually switching from implementing new features to improving the code that's already written, and we'll launch a semi-public beta in the next couple weeks.

A couple miscellaneous things learned from the week I spent writing web service clients:

• Tumblr's web service supports OAuth, but they don't really mention this anywhere except in the post announcing it. Be aware!
• URLs are awesome. A URL is a unique identifier that has meaning to the outside world. Don't send me an object's primary key; send me its URL. I'm just soapboxing here, really.

Apropos revising RESTful Web Services, I have a question for you. When I started talking with people as part of my job search, I noticed that nobody uses the term "web services" anymore. Everyone talks about "APIs".

I don't want to overstate the magnitude of the change here; people have been referring to web services as APIs for years. I did this myself until around the time RWS came out, and even today the Launchpad web service is officially called the "API". The change I noticed is just that the terms used to be used interchangeably, and now "web service" has pretty much died out. When I say "web service" people know what I'm talking about, but I feel like I'm speaking out of a phrasebook, or doggedly saying "free software" in an "open source" world.

This is one of those little nagging things that may or may not be important. I have an inkling that it's fairly important: that there's a reason people might have stopped saying "web service" around 2007, but that using "API" as a generic term leads to products that are worse than they could be, and that it's something I'd need to address in any second edition of RWS. But I would like to get a broader spectrum of opinion on this.

Has the term "web service" really died out in the past few years (or, as I suspect, changed its meaning)? When do you think this happened, and why?

Update: This post spawned an InfoQ article, so I should probably stop playing coy and say what I think.

I agree with commentor Foone. I think that WS-* successfully claimed the term "web service" in the mid-2000s, and that when people soured on WS-* they also stopped using the term "web service". While looking for work, the only companies I saw talking about their "web services" were big companies that were using SOAP/WSDL. Everyone who wanted to go above level 0 on the maturity heuristic was talking about "APIs".

The problem is, this terminology is backwards! SOAP/WSDL services are at level 0 just because they're not "web services". They don't use any of the web technologies. Their goal is to abstract away the network boundary in order to appear like a native client library. An API, if you will. As you adopt more of the web technologies, your product becomes less like a native client library ("API") and more like the web ("web service").

Thinking of all web services as APIs buys into the assumption that a web service is like a native client library. You can do a lot with that assumption, you can even get to level 3 of the maturity heuristic, but I don't think you can use that assumption to build complex hypermedia-driven applications of the kind seen in REST in Practice. Saying "API" is saying that SOAP/WSDL had the right idea, but the technologies were too heavyweight.

evpuneqfba gurfvf erfg jro freivpr

No, but thanks for playing!

So I was kind of cheesed off at this person, but frankly the rudeness worked—I answered the question just to get them out of my hair. The question is, how do you make the creation of a new resource an idempotent operation? Imagine a factory resource that you POST to to create something new. Typically, if you send the same POST request twice, you'll get two new resources. How do you make it so that the first request creates a new resource and the second request fails?

The short answer is to use POST Once Exactly. I mention POE in the REST book, but POE never made it out of the Internet-Draft stage, and I can't even find the Internet-Draft at Mark Nottingham's site anymore. Besides which, the email message described something similar to POE as an option, but wanted something simpler. So here's something simpler, for some value of "simpler".

When you buy (eg.) computer hardware from NewEgg, they give you a space for a "purchase order" along with shipping address etc. This is for use when you're buying stuff in a commercial context. You can put whatever you want in that space. The purchase order is a key into your company's purchasing system. It's meaningless to NewEgg.

Newegg doesn't use the purchase order for anything, they just put it on the invoice for your convenience. But in theory, they could keep track of the purchase orders you've used. Since they know which orders are yours, when you place a new order the site can run a quick check to see if you've used that purchase order before. If you have, they can stop the purchase from going through, since you're probably trying to process the same purchase request twice.

So the simpler solution is. You're defining the media type or the form describing what the client is supposed to POST to the factory. Include a slot for a "purchase order". This is a random string chosen by the client whose only purpose is to uniquely identify a resource to be created. On the server side, reject a POST request that uses a purchase order already used by this client ("this client" distinguished from other clients by the authentication credentials). Now the POST request is idempotent: sending it 10 times is the same as sending it once.

I call this strategy "Purchase Order Equivalent", or POE. Thoughts?

This talk is a quick exegesis of the "Richardson Maturity Model" for web services. If you want to get into it you can see the slides and hear the (slightly edited) recording. If you just want the stuff from my talk that's not in other, similar, discussions, here it is:

First, a clarification: some people think the Richardson Maturity Model is named after me, but that would violate Stigler's Law of Eponymy. It's actually named after my father.

Second: for this talk I identified three really simple questions you can ask to determine where a web service sits on the RMM:

1. Is there more than one URI?
2. Do URIs designate specific things? (As opposed to invoking actions.)
3. Are there any links?

Third, something I mention on the main talk page but that I'd like to give more exposure here and get feedback on. After the talk I got dinged by a friend for not giving a good enough response to this question in the Q&A:

"We've been pounded with REST for the past ten years, and no one has come up with a standard besides 'HTML' or 'XML', ad nauseum. Do you see any resolution to that coming?"

What I said in response to that question was technically accurate, but I didn't provide any advice. Here's my advice. You want a good default choice that can settle arguments and save you from having to make all the design decisions yourself? There is a standard: AtomPub. Take whatever you're trying to do and make it fit the AtomPub paradigm. Not only will your design be RESTful, your service will fit into a preexisting ecosystem.

If you think AtomPub won't work with what you're trying to do, you're probably wrong. Google, for instance, publishes web services for many very different applications (spreadsheet, calendar, map, etc. — basically anything that gets pluralized and has "Google" slapped on the front), and they're all based on AtomPub. You might find a better fit with your problem space if you did a RESTful design from scratch, but that's the nature of standards.

Convincing?

Yup.

OK, that's in New York. On April 26th I'm going to be in Raleigh giving a paper at the WS-REST subconference of WWW2010. You probably already know if you're going, and I honestly don't recommend buying your own ticket--I had to, and it's damn expensive--but I thought I'd mention it. The true title of my paper is "Developers Like Hypermedia, But They Don't Like Web Browsers"; the title you see on the schedule was camouflage to make my paper look more academic when I submitted it.

And then while I'm in Raleigh, on the 27th I'm talking at the Triangle Zope and Python Users Group. Thanks to my co-workers Gary Poster and Brad Crittenden for setting this up. I'll probably give two mini-talks: I'm planning on "Six Years of Beautiful Soup", then a run through the maturity model heard 'round the world. Then, your questions.

Here's the problem, on a high level of abstraction. Consider a representation (#1):

<p id="1">Forklift</p>
<p class="read-only" id="2">Green</p>

And let's say the ETag of this representation is the string "x".

According to the protocol governing this media type, you can modify the text in any paragraph unless its class is "read-only". So maybe you can PUT a document like this (#2):

<p id="1">Hovercraft</p>
<p class="read-only" id="2">Green</p>

Or PATCH a document like this (#3):

<p id="1">Hovercraft</p>

OK, that's easy. Now suppose that the read-only text changes randomly according to conditions on the server. Let's say the read-only text suddenly changes from "Green" to "Red". If I were to GET the document again, I'd get this document (#4):

<p id="1">Forklift</p>
<p class="read-only" id="2">Red</p>

And let's say the ETag of this document is "y". If I sent a conditional GET with an If-None-Match of "x", I'd get 200 and a new representation instead of 304 ("Not Modified").

OK, but I don't send a conditional GET. I don't get the document again at all. Instead, I PUT document #2, with an If-Match of "x", and the request fails with 412 ("Precondition Failed"). Maybe it should fail anyway; maybe the server is very strict and thinks I'm trying to change a read-only paragraph from "Red" to "Green", which would probably be 400 ("Bad Request"). But we don't even get to that point because the ETags don't match.

The request also fails with 412 if I PATCH document #3 with an If-Match of "x". But there's nothing really wrong with that request. The point of If-Match in conditional writes is to avoid conflicts with other clients, and there are no other clients here. The ETag is different because a read-only paragraph changed on the server side.

One obvious solution is to calculate the ETag only from the read-write portion of the document. This fixes conditional writes, but it breaks conditional reads. A client that requests document 1 and then makes conditional requests will never get document 4. The ETag is no longer a strong validator (update: actually, it's not any kind of validator); the document might change significantly without the ETag changing. So that's no good.

The solution Björn came up with is to split the ETag into two parts. The first part is derived from the read-only portions of the document, and the second part is derived from the read-write portions. The ETag is a totally opaque string to the client, but the server knows what it means. On a conditional read, the server checks the entire ETag. On a conditional write, the server only checks the second half.

In this example, the ETag for document #1 might be "1.a" and the ETag for document #4 might be "2.a". A conditional GET of document #4 with If-None-Match="1.a" would fail, but a conditional write with If-Match="1.a" would succeed. When the write went through, the document's ETag would change to "2.b", and "1.a" would not be good for either conditional reads or writes.

From the client's perspective everything just works: your conditional read returns 200 iff the representation has changed, and your conditional write returns 412 iff someone else is messing with the resource. But is this okay from a standards perspective? Section 13.3.3 of RFC 2616 says "The only function that the HTTP/1.1 protocol defines on [ETags] is comparison." That doesn't seem to prohibit me from defining another one.

If "x" is a strong validator then so is "1.a", but the new comparison function ignores some of its information about the resource state, effectively treating it as a weak validator (update: or as something that's not a validator at all). Is that okay? Would you believe the following definition of a strong validation function? "In order to be considered equal, the second halves of both validators MUST be identical in every way, and both MUST NOT be weak." (cf 13.3.3 again)

I'm interested in your thoughts on this. Smartass comments like "you should have two resources" will not be dismissed out-of-hand but also will probably not convince me. If you're curious, here's the real-life bug that spawned this thinking.

By now it's a cliche to observe that allegedly "RESTful" web services like Amazon's SimpleDB and the Flickr web service and the del.icio.us web service aren't really RESTful. But there's something about them that makes their creators distinguish them from SOAP-based web services (by calling them "RESTful"), and there's something about those services that users love despite the possibility of, eg. deleting data by accident.

Rather than say one service is more or less "RESTful" than another, attempting to quantify an easy-to-misuse term that wasn't even intended to be used in relation to web services, I've found it useful to judge web services based on how many of the web technologies they adopt. Think of the World Wide Web as having a technology stack that looks like this:

Hypermedia (ie. HTML)

HTTP

URI

In every case I know about, people build web services from the bottom of the stack. They pick some point on the stack and take the technologies below that point seriously. The technologies above that point are not considered important. They're ignored, or used to the minimum extent you can get away with and still technically be a "web service".

XML-RPC and SOAP services are at level zero. They don't take any of the web technologies seriously. They use one URI, one HTTP method, none of the interesting features of HTTP, and they have no notion of hypermedia.

URI

The web services I mentioned at the beginning of this entry are at level one. They take URIs seriously and assign a URI to every aspect of the system. But they only use one HTTP method (GET) and don't use any of the interesting features of HTTP. Nonetheless, people love these web services, because people love URIs.

HTTP

URI

Amazon S3 is at level two. It has problems (or so I've heard) with its use of HTTP, but it takes HTTP seriously. It uses URIs to designate objects in the system that can be operated on with different HTTP methods. It takes advantage of HTTP's features like conditional requests. But there's no hypermedia. All the information about how to manipulate the resources, and how to detect the connections between them, is in English documentation that you have to read when writing your custom S3 client.

Hypermedia

HTTP

URI

Most of my current thinking is hypermedia-related. Useful here is the formal definition of hypermedia, from 4.1.3 of the Fielding dissertation: "Hypermedia is defined by the presence of application control information embedded within, or as a layer above, the presentation of information."

Services like the Web, AtomPub, the Netflix web service, and the Launchpad web service are at level three. They serve documents with embedded "application control information": links and forms that give a more or less generic client hints about how to manipulate this particular web service.

There are degrees of quality within these levels. For instance, big parts of the Web use URIs to name operations, rather than the objects those operations can act on. I think that's bad design, but that's a problem on level one. It doesn't negate the value of HTTP or hypermedia. Similarly, when we have the well-worn argument about which HTTP methods a web service should expose, we're having an argument on level two, not an argument about who's more RESTful. In fact, almost all the currently raging arguments are level one or level two arguments, which is unfortunate as there are some really great flamewars to be had on level three.

I don't intend to defend this classification technique in detail. It's a set of heuristics. In theory you could take URI and HTML seriously but not HTTP; or HTML and HTTP but not URI. But the heuristics are useful in that they 1) encapsulate a fact about how people tend to design web services, 2) make it easy to classify an argument or problem, 3) let you make a snap judgement of how much Web knowledge went into a web service, 4) make it easy to think about RESTfulness (an abstract meta-architectural concept) in terms of specific technologies you should know about already, technologies that are probably the reason you care about REST in the first place.

The Internet is scattershot with uninformed commentary on this issue, which I now supplant with a single clearinghouse of uninformed commentary. Mozilla developers know about this problem but for some reason, presumably bug-compatibility, added a sendAsBinary method rather than fixing send. Maybe in five years that will be a standard, but it doesn't help me now. (Also, in experiments I couldn't get sendAsBinary to send binary data! It crashed with NS_ERROR_DOM_INVALID_CHARACTER_ERR!)

One solution is to create a MIME multipart message with one part and a Content-Transfer-Encoding of "base64". Bandwidth-wasty, but effective. Except, thought I, why bother with the multipart message? Just encode the binary data as base64 and send a Transfer-Encoding of "base64". (Content-Transfer-Encoding is explicitly prohibited by RFC2616; it was more or less split into Content-Encoding and Transfer-Encoding.)

Now, "base64" is not an officially registered transfer coding, but using it seems to be in the spirit of the Transfer-Encoding spec, which wants 'to ensure "safe transport" through the network.' The only difference is that the unsafe thing (stupid Javascript) is on one end of the network rather than in the middle.

Here's another solution that I don't really understand. It uses Mozilla-specific hacks to create an input stream containing binary data, which input stream is passed to send() instead of a string which will inevitably be chopped or parsed as a DOM object or something.

For now I'm going to hope that the problem goes away when I start using a different Ajax toolkit, which we were going to do anyway, but as always I'm very disappointed when I try to make an HTTP request and something totally random happens that's not my HTTP request.

Update: Because of this problem it's apparently common to send binary files using a hidden IFRAME that contains an HTML form. The binary file is sent as a MIME multipart message over POST, but it's not base64 encoded--it's a regular browser file upload.

Update 2: Have fun arguing over the legitimacy of my desire to do this instead of investigating the problem, Reddit dudes. The %00 trick sounds feasible, other Reddit dudes. I'll try it. PS: switching to another toolkit didn't solve the problem because all the toolkits use XHR.

Looking around I see only a couple nerds wondering about this, about as many as in 2005 were wondering why HTML forms don't support PUT or DELETE. But it's at the intersection of two trends: growing interest in putting metadata in HTTP headers, and growing interest in not using MD5. Has anyone else wanted to send a non-MD5 checksum in HTTP headers? If nobody says anything I'm going to go ahead and make a X-Content-SHA1.

Uh, and in keeping with the peurile theme established by the previous entry, here's my current favorite wacky faqs.org comment.

"The solution... is to provide a resource that reflects all of the changes on Flickr during a given time period." --Roy Fielding

"[A] simple and compact 'ping feed' that web services can produce in order to alert the consumers of their feeds when a feed has been updated." --Simple Update Protocol

In October 2007, straight out of Viable Paradise, I started work at Canonical, the company that makes the Ubuntu Linux distribution. I work on Launchpad, a web application for coordinating open source software development. You may have noticed similarities--eldritch similarities--to my previous full-time job at CollabNet, where I worked on a web application for coordinating open source style software development.

Well, anyway, Launchpad now exposes a RESTful web service. I built a framework on top of Zope, and my manager Francis Lacoste wrote a nice set of Python decorators on top of my framework. The combination of the two means that a Launchpad programmer can do a little bit of additional work in their Zope interfaces, and the objects that implement those interfaces will be published RESTfully through Launchpad's web service. As we work behind the scenes, more of Launchpad comes online through the service.

My goal is quality without compromise. If there are problems with the design (which there are) I consider them bugs to be fixed. On the client side I want to provide a web service browser: a program that works by navigating hypermedia documents, but is as easy to program against as a bunch of statically-typed crap generated from a WSDL file. To that end I've written launchpadlib, which has some Launchpad-specific hacks in it, but is based on the generic wadllib. The Launchpad-specific ideas are similar to the emerging standards around RESTful JSON. I can give more details if people are interested.

I'm posting weekly updates about our progress on the Launchpad weblog. On Tuesday I'll be giving an IRC Q&A as part of Ubuntu Developer Week. But I'd recommend not crashing that chat if you're just interested in the architecture, because it's for people who want to write Launchpad clients to help with development. I'll be discussing the architecture and the development process as part of my talk at QCon in November.

There are a number of ways to solve the problem. One way would be for Apache to magically strip the "-gzip" off the entity-tags when they come back in, but how is Apache supposed to know that that particular -gzip was one it added? So, as usual, the magic solution fails.

An alternative that appeals to me is to stop treating compression as an aspect of the representation such that having it or not having it means you need to change the ETag. Move it out of Accept-Encoding and Content-Encoding, and into TE and Transfer-Encoding. This is related to what I was saying last year, but I didn't make the connection between "this is part of the transmission, not part of the representation" and "this should be in Transfer-Encoding, not Content-Encoding." Or to quote the RFC, "the transfer-coding is a property of the message, not of the entity."

The problem with this is that Transfer-Encoding describes the transmission from one intermediary in a chain to the next, not the transmission from server to client. Similarly for TE and the transmission from client to server. Intermediaries are my RESTful weak point so I don't fully grasp the implications of this. If the client sends TE: gzip, the proxy can decide to get an uncompressed representation, look at it, and then compress it before sending it back to me. That hurts performance but everything still works, right?

Update: "That is why transfer encoding was invented in the first place."

[0] To be 100% correct it would change the ETag to "foo-gzip", since entity-tags are supposed to be enclosed in quotes.

Is Statelessness Required? The universe will not end in an ontological segfault if you violate statelessness. Instead, you will give up a lot of scalability potential, and you will hide state where your clients can't get at it, which will annoy them. So why do it? Maybe because you're using a framework that makes it really easy to write programs that violate statelessness, and then tries complicated tricks to get scalability back, because that's the way web frameworks have been written for ten years.

We took a pretty hard anti-cookie line in RWS (pages 252-253), but we made it clear that cookies themselves are not the problem. Cookies, like links, are a way of serving application state. They have one big RESTfulness problem but it's moot because nobody follows the cookie standard that slavishly. The real problem is the session IDs that go into those cookies.

A session ID is a key into a big chunk of state kept invisibly on the server. That's what causes the problems. It's just as wrong to serve that session ID as a query variable in a link as it is to serve it in a cookie. You need to turn the hidden state into application state by incorporating it into links and forms, or you need to turn it into resource state by exposing it as part of your web service and letting clients manipulate it directly.

Is Being Message-Centric Good Enough? I say yes. I'm young enough that the Web is the first distributed programming environment I ever used. I've never felt like I was missing something that justified switching to a competitor. The more I've learned about its design the more impressed I've been. When something better comes along, I predict it will look more like the Web than it will look like DCOM or CORBA or WS-*, whether it's "better" in a global sense or in some application-limited sense.

Are PUT and DELETE Essential? The universe will not end in an ontological segfault etc. Also, you won't even stop being RESTful, assuming you have proper resource design, as per my first entry in this series. What you'll give up is the ability to make various reliability guarantees. I was going to say you also give up the ability to avoid the lost-update problem, but I guess you could do that with POST, so long as (yes) you were POSTing to the same URI you sent GET to earlier.

I don't know how good an argument this is anymore, but: designing with PUT and DELETE forces you to think in terms of resources. It's a form of discipline, like eschewing cookies. When you've got "read data" and "whatever!" it's very easy to think in terms of operations, flex your ten-year-old web application muscles, and end up with a mess like the Flickr web service. The vocabulary is negotiable. The underlying idea, (that a URI designates an object which responds to a standard vocabulary) is essential. That's the Architecture of the World Wide Web.

Is "Do Like the Web" a Good Argument? Not in and of itself, because parts of the Web are lousy. You need a tool to separate the good from the bad. I follow the admonition of Paul: "Prove all things; hold fast that which is good." Look at what people love about the Web and see if you can bring that same joy to distributed programming. Look at what people complain about and try to eliminate those things, whether you're building web applications or web services. When you find something that improves the distributed programming side of things, bring that knowledge back into the field of web application design.

Where Are The Tools? Traditionally this question has been answered by assertions that the (HTTP-level) tools already exist. As annoying as this answer is, it's technically correct. Without additional abstractions on top of HTTP/URI/*ML, there's nothing to write tools about. As we invent those abstractions (AtomPub, ROA, the ActiveResource control flow, OAuth, WADL, etc.) we write more tools. I'm writing tools now. As we get more practice the higher-level tools will get better, and as they get better they'll consolidate (as will the abstractions beneath them) and there will be fewer.

One thing I'd like to add is that it would be cool to see the existing frameworks for web applications apply some of the principles people have come up with while using the Web as a distributed programming environment. Make it easy to publish resources rather than operations, easy to support conditional GET, easy to write client interactions that respect statelessness. Rails has the right idea here.

Saying that RESTful services "work like the web" is not just saying that things have URIs or you can cache GET requests. Ideas as confusing as hypermedia-as-the-engine-of-application-state become understandable when you think about the way you use the web.

If a website is well-designed, you don't need to mess around with the URL bar to get where you want to be: you fill out forms and follow links. You don't need a custom web browser to use crummy.com. My website is distinguished from all other websites by the HTML documents I serve. If I change my weblog software, it'll serve you different documents, and you may have to change the way you use the website, but your browser doesn't crash.

You know what to do (how to drive the crummy.com "application" into the desired state) because the document I send you lays out your options. And because the document represents the options as hypermedia links and forms, you know how to build an HTTP request that will carry out your desires.

"Has REST Been Fortunate in Its Enemies?" I really hope the whole "enemies" thing goes away and that the period 2000-2005 is seen in retrospect as a big misunderstanding. Just in the period since RWS was published there have been really interesting developments (OAuth, PATCH) that have advanced the state of the HTTP art, and we lost five years of that kind of thing going down the wrong path and then having a huge argument about which path was right. That said, when your enemies turn out to be big corporations with lots of money (as always seems to happen to me), I would not describe the situation as fortunate.

Schema-driven mapping: probably hopeless if you want a really detailed mapping to plug-and-chug into your strongly-typed language. Even if possible, very boring. Not too difficult if you just want an index that points out the interesting parts of a document.

Contracts: My personal crackpot theory that I can't get anyone to believe is that hypermedia is contracts. A link or form is a promise that if you make a certain HTTP request, you'll get a certain result. HTML is a primitive contract language. The AtomPub service document format and WADL are more advanced contract languages. That said, I think contracts are less useful than often supposed, because the ELIZA effect artificially narrows the distance between the syntax of the contract document and its semantics. An AtomPub service document works like magic because a human programmer did some work beforehand understanding the AtomPub standard and programming in the semantics.

Registries: We have registries on the human web; we should be able to have them for web services. How much of their imagined benefit is due to the ELIZA effect? How much work will a human need to put in to find the one of 10000 AtomPub implementations they want to post to? I don't know. It depends on the conventions and standards we come up with.

Payload wrappers: HTTP is a sufficient payload wrapper for my needs, and if I ran into a problem with it I'd extend it (a la OAuth), not that I feel competent to do that. I can kind of see how you'd consider Atom a payload wrapper, but I'm just a simple caveman and I prefer to think of it as a representation format.

Message-level signing and security: yes! to the former. OAuth works for me. I don't know if I'd ever use message-level security. In general, experimentation is good when it happens on top of HTTP where everyone can use the new standards.

"Is Getting HTTP Right Good Enough?" No, not in the way we mean "HTTP" when we have this never-ending argument about verbs. I don't even think it's the first step. The first step is to expose resources: to give a distinct URI to every object you want to publish. That's the first of the RESTful rehabilitation suggestions on page 303 of RWS. That's the first three steps of the ROA design procedure on page 148. The first document to understand is not the Fielding thesis, not RFC 2616, not even RWS, it's Architecure of the World Wide Web, Volume One. That document is mostly about URIs. HTTP shows up as a URI scheme and as a popular protocol for dereferencing URIs.

This is why Flickr and del.icio.us put up terribly-designed web services and people love them and use them all the time: they stick URIs on everything. People love URIs! Their problem with those services is they don't know when to stop. So the first step is to get on the web, to not ignore the intuitively obvious usefulness of URIs. It's terrible that these web services self-identify as "RESTful", but at least they've taken the first step.

The second step is to know when to stop handing out URIs. Picking a vocabulary lets you move some of the information out of the URI and into the HTTP method. It lets you expose objects that have methods, instead of a big C library of del.icio.us or Flickr-style functions. This is the step where you master HTTP. This is where the arguments are happening right now. These arguments are not based on intuitively obvious things. They're arguments about scalability and fiasco avoidance and client and network capabilities. They're also arguments about whether using all of HTTP would improve the Web.

The third step is the mastery of hypermedia. (Note that there's one step for each of the Web technologies: URI, HTTP, HTML.) This is about making it possible to program a computer to do the mapping of options to actions that happens in our brains when we surf the web. The advantage here is the same as in the other steps: loose coupling and flexibility. This is the part that confuses everybody, because we don't usually write programs like this unless we're scripting or screen-scraping. And the ELIZA effect is in play, because we've moved past HTTP requests and responses to what happens inside us when we look at response N and decide to make request N+1.

To the extent that there is a "religious" aspect to RESTful thinking, I think it's on this level, in the intuition that there's something in the high-level way we use the Web that we can use when programing a client. Maybe it will turn out to be a bust like AI in the 80s. I've had some encouraging results on my current project. I think it's likely that we're only now figuring this out because we spent five years arguing over whether to use URIs and then another three arguing over whether to use GET/POST or GET/POST/PUT/DELETE.

It looks like this is going to be a multi-part entry, so here is the single most important point I'd like to make. I mentioned it here, last year, and also mentioned it in RESTful Web Services (see especially pages 220-221), though not as prominently as I probably should've. Maybe I'll work this here into the second edition.

REST says you should use a uniform interface, but it doesn't say which one. How do you choose? You pick a set of verbs that 1) gives you the semantics you need for your application, and 2) lets you tie into network effects.

On one end there's the degenerate uniform interface of XML-RPC and SOAP: [POST]. Like a language with only one word, this is pretty useless. Seeing that "POST" gives you absolutely no information. POST means: "whatever!" You're just pushing the complexity somewhere else.

The one thing I want to say about this degenerate vocabulary is that your decision to use it is orthogonal to your decisions about resource design. XML-RPC and SOAP services provide a single "endpoint" resource, a monster message-processor at a single URL that responds to a wide variety of POST requests. That's bad resource design. But you could expose a tiny "message processor" for every part of your application you want clients to manipulate, and have them serve hypermedia documents (in response to POST) that linked these resources together. (See RWS page 303.) It would just suck, because clients could only communicate with those resources through POST.

Then there's the uniform interface of the human web: [GET POST]. Now that there are two verbs, you can give them separate meanings and split the complexity. GET means "read data" and POST means "change data." Actually, POST still means "whatever!" but since it's defined in opposition to GET, it's convenient to think of it as the opposite of GET.

Choose this set of verbs and you can perform a series of nifty optimizations on GET, which probably make up the bulk of your requests anyway. The GET optimizations work because GET means something. Its meaning is constrained, and constraints can be starting points for optimizations.

But, now that the words have real meanings, they can be misused. Use of GET where you mean POST will make you a victim of the next Google Web Accelerator type fiasco. Use of POST where you mean GET will ruin addressability and annoy your users.

And again, your decision about vocabulary is orthogonal to your decisions about resource design. Whence my rule of thumb: "POST to the same place you GET." If you had well-designed resources that responded to the degenerate vocabulary of [POST], you could convert your "read data" operations to GET and get a much better web service without changing your resource design. (again, see RWS page 303.)

Moving further down the scale of complexity is the uniform interface most often seen in discussions of REST: [GET POST PUT DELETE]. This vocabulary isolates certain classes of "change data" operations, rips them out of POST, and gives them their own names. POST still means "whatever!"

Why would we do this? Splitting out PUT and DELETE means giving them a meaning distinct from "whatever!" And a meaning is a set of constraints, and you can optimize around constraints, etc. Your decision to use this vocabulary is a decision about which constraints are useful.

And for the third time this is orthogonal to resource design. If you had a RESTful web service that used GET and POST for everything, you could PUTify any POST operations that fit the PUT constraints, DELETEify any operations that fit the DELETE constraints, and then you'd have a four-verb RESTful service. You could go the other way, too: fold PUT and DELETE back into POST and you'd have a RESTful two-verb service. What you'd lose is the ability to say, "make sure the state of the resource reflects the submitted document" or "make sure the resource goes away", instead of "whatever!"

A little more complex is the uniform interface we used throughout RWS. We use the same four verbs, but we told POST to stop meaning "whatever!" When we use POST, it's only allowed to mean "create a new resource underneath this one." We did this because "whatever!" is a linguistic rug under which to sweep things. If you write a book about home organization where you say "and here's how to sweep anything inconvenient under the rug!", readers will suspect that there are systemic flaws in your organization techniques. So we wrote a book where we organized complex things like queues and transaction systems without recourse to "whatever!"

But on a technical level, the point is not that "whatever!" is evil. The point is that if you chip a piece off "whatever!" and make it mean something specific, you can optimize around the constraints and reap the benefits. It's pretty clear from the history of the web that you need to separate "read data" out from "whatever!" Because there's less history, there's active disagreement about the cost/benefit tradeoffs of PUT, DELETE, PATCH, et al., though I find them very useful. But the vocabulary you use is an interoperability shibboleth, not a RESTfulness shibboleth.

I cut the chapter because it's only about 7 pages long, and being historical in nature (and dealing with protocols most programmers haven't used in years or even heard of) it wasn't necessary. I've always thought it was interesting, though, and I turned it into a standalone essay. I tried for a while to get some money for it by publishing it on one of the O'Reilly sites, but that never went anywhere. So this morning I say "screw it" and give you the link.

It's called "The System of the World Wide Web", I guess because I thought a nerdy joke in the title would sell it; I think used to be called "How the Web Won". It's free for you to enjoy.

Since I wrote that essay I met Rohit and we became friends. (Also, that weblog entry is serious foreshadowing for this one.) He also read the essay and said "Come on!", G.O.B.-style, about a part where I implied he didn't appreciate why Gopher selectors were technically inferior to URIs, so I made a small update. Riveting stuff--did you know that I also took frozen food off his hands when he moved back to California?

• Amazon's web service team is relatively open and approachable.
• This is a design flaw that can easily lead to destroyed data, not an academic imperfection.
• Outsiders have been loudly complaining about this class of problem since Amazon's first web service was exposed in 2004.
• The second time this happened (Amazon SQS), Amazon fixed it in response to complaints. So it's not like they're committed to this bad design.
• The design of Amazon S3 is very solid. It has everything except connectedness/HaEoAS. It came out almost two years ago.

Reading the tea leaves it looks like Amazon has some internal framework that takes a set of functions, and generates code for a web service that handles calls to those functions through HTTP GET or SOAP. If people complain and an Amazon coder gets around to it, said coder can generate or write a proper resource-oriented interface to the same underlying code, but that's not what's generated by default. S3 and the web service that controls EC2 were done by different teams who didn't use or didn't know about this framework.

A new kind of (SOA) Registry. In del.icio.us, that great sandbox for rhetorical questions, Mark Baker asked "So what's the difference between a RESTful SOA registry and a Wiki?". I'd say the main difference is that someone points at the RESTful SOA registry and says "Everyone should use this as a RESTful SOA registry." Also the registry will probably have rules about providing credentials to modify a registry entry. So, I think it's like the difference between my weblog and a wiki.

That part of the book is now outdated because I went on to describe a set of equivalent ad hoc solutions to this problem, each created by a different web service provider for their services, with the implication that someone who's not me should get off their butt and come up with an open standard. This has now happened, and the result is OAuth. The people working on the standard are posting introductory guides to drum up interest. The standard is good, I've been pushing it on the companies I consult with, and the time is right to use the NYCB bully pulpit to spread it further. If I were Kenneth Turan I would rave, "The most significant HTTP authentication mechanism since Basic!"

Incidentally, today's the first day of my job, but I don't actually start until tomorrow because it's a holiday.

I guess my struggle with it is that looking at /recent doesn’t make it obvious that’s its recent bookmarks whereas something like /bookmarks/recent or /bookmarks?fromDate=2007-06-01 is more obvious that its a list of bookmarks.

That's a fair point. /bookmarks/recent (or just /bookmarks) would make it more clear to a human what "recent" things exactly are at the other end of the URI. I never considered that because Rails doesn't encourage that kind of thinking, but it's a better URI choice. But we're now in a side issue because what a URI looks like has no formal bearing on what resource it identifies. The URI could be /4350/9885677.4169 and it would still be a resource that gives you recent bookmarks when you GET it.

This is probably not something you were wondering, but I'll address it briefly: how do you make it clear to a computer that there's a list of recent bookmarks at the other end of the URI? There are a number of ways. At one end of the spectrum you can hard-code this information into every client. At the other end you can do some magic Semantic Web thing where terms like "bookmark" and "recent" are given meaning within some ontology that a general client understands. In between you have clients that are programmed to understand certain media types or microformats, and clients that are programmed to understand certain hypermedia formats that contain semantic cues.

In none of these cases does the computer "understand" the resource the way a human does. The goal is to reduce the space between the human's desire "I want to do x to recent bookmarks" and the HTTP request the client ends up making. One one end of the spectrum, the user has to hard-code their desire into the client. On the other, the client can take "x", "recent", and "bookmarks", plug them into the ontology, and figure out which request to make.

"Why is the recent list of bookmarks a resource? Why isn't "recent" a filter on the resource bookmarks?"

I couldn't figure out how to post a comment on Scott's weblog, so it goes here. The short answer is that it's both. "Resource" is the HTTP-level term, and "bookmark filter" is an end-user-level description of the resource.

It's legitimate and often useful to talk about /recent as a filter on the bookmarks, but that's an abstraction. All /recent is, is a URI that gives you something when you GET it. You might link to /recent, retrieve or cache a representation of it, etc. etc. It names a resource. Calling it a filter on the bookmark list is on the same level as calling it a list of recent bookmarks. It's accurate but not very precise: you're using high-level terms instead of well-defined terms like "resource". Resources are defined as part of HTTP and filters aren't.

Scott again:

How is "recent" bookmarks any different than "bookmarks for the last 10 days" or bookmarks since "9/10/2007"? I don't think anyone would argue that the last two aren't filters. But "recent" isn't any different then either of those two.

They're exactly the same. They're all resources. In chapter 7 when I mention features like date filters, I describe them in terms of end-user functionality so I use end-user-level terms like "filter", but on the level of HTTP analysis, /users/leonardr/bookmarks/?date=2007-06-01 or whatever points to a resource.

Hope this helps.

Everyone reading this weblog is probably sick of my explanations, but here it is: FPS, Flickr, and del.icio.us all use URIs to designate functions when URIs are supposed to designate objects. This takes you from object-oriented programming back to procedural programming. They expect clients to invoke functions through GET requests, which means caching can't work, clients need to handle URIs like sticks of dynamite because there's no telling what GET might do, etc.

There are other problems, like their two status codes and 70ish error codes delivered through the single HTTP status code 200 (OK). But you don't have to drink the entire pitcher of REST kool-aid to want to get rid of the big architectural problems.

Amazon SQS had almost exactly the same non-RESTful "REST" interface until people complained, so maybe complaining will work here.

I'm also talking to a couple companies about full-time positions. I'm not going to make any decisions for at least a couple weeks, so if you have an idea for what I should do next, please email me. I'd really like to apply the stuff I learned while writing RWS, but if you want to give me a job crashing giant robots into each other I probably won't say no.

[T]he authors' goal is stated as being one "to reunite the programmable web with the human web" ... The unstated assertion here then is that in order to meet the authors' stated goals, the majority of today's Web Applications will need to be re-architected and re-written in order to conform to a RESTful style! ... [T]he RESTful approach is essentially unproven outside of the read-only Web... doesn't this run counter to the often-voiced argument that REST builds upon what makes the web what it is today?

First off I want to say that my goal of uniting the programmable web with the human web is the lofty, inspirational kind of goal, not the kind that gets a budget and a schedule. "Dream" might have been a better word. But yes, it goes both ways: web applications should be more RESTful, and web services should be more weblike.

Now, on to the critique of web services vs. the web. My quick answer comes from Sam's and my InfoQ interview. Again, I'm just quoting the general flow:

InfoQ: Another argument often brought up is that the standard claim that REST's advantages have been proven by the Web's success is actually a wrong conclusion — the reason being that most of the Web's usage is actually non-RESTful.

LR: There's some truth to this. Earlier I said that most websites don't expose write operations RESTfully. But I think removing the non-RESTful parts of the Web would make the Web better, and wouldn't hurt its success—except in the same second-order way it would hurt the Web's success if web browsers were stricter about invalid HTML... I decided there were three important reasons [for the Web's success]: everything has an address, pages link to each other, and you can manipulate the web using only a web browser.

The first two correspond to important features of RESTful web services: addressability of resources by URI, and hypermedia-as-engine-of-application-state... Rather than promoting REST by pointing to the Web as a whole, I would point to the incredible usefulness of URIs and links, two features largely ignored by WS-*.

Longer answer. First, by number of requests the read-only web is most of the web. Even if REST only made sense for read-only web services, the success of the read-only web and the tools built around it would justify a (short) book.

Second, the read-only web is not just GET. It's also URIs, media types, links, and application forms. That's most of the web technologies, and they're used on the web as they are in RESTful web services. All that's missing are write operations and resource forms. Fittingly, there's significantly more controversy about these: see GET/POST vs. GET/POST/PUT/DELETE and do-we-need-WADL. In the interview I ask whether "architectures consciously designed with REST in mind will 'win' over architectures that are simple but only intermittently RESTful." Those other architectures are today's web application architectures.

How big is the gap between web applications and RESTful services? Well, here's a RESTful rule for web applications: POST to the same URI you GET. The URI designates an object, not an operation, and we have a special way of distinguishing between safe (GET) and unsafe (POST) operations on objects.

I don't think that all web applications should be rewritten to follow this rule, because in general I'm opposed to doing work. But when I write web applications in the future I'm certainly going to hew to that rule. It's a cleaner design. Even if you think that rule is a waste of time, you probably don't think following it will break the web—and following it makes a web application resource-oriented and RESTful.

It's true that the success of the web doesn't reflect success on every aspect of RESTful web services, but the overlap is pretty large, and the web's success certainly doesn't stem from any systematic repudiation of RESTful principles.

In other news: first there was East Germany's Polyplay. Now, Russia's arcade games are revealed in your capitalist WIRED.

Video games were not bargains in the Soviet economy. "It wasn't exactly a luxury, but you couldn't afford to play every day, either," said museum co-founder Alexander Stakhanov. "More like once a week."

This one presented a bit of an audio challenge because Leonard speaks loudly and Sam speaks very softly.

It's twue, it's twue. When I'm nervous I get loud and sound like an idiot.

In that spirit, here is my defence of WADL, called "It's Just a Hypermedia Format". Sometimes people say they'd like to do strange things with WADL, things I don't approve of and things they wouldn't think of doing with HTML. But WADL is just a hypermedia format. I like WADL because it's a better hypermedia format than HTML or plain XML.

WADL is a way of describing the infinite variety of HTTP requests a client might make in the future. WADL also helps the client figure out which of those infinitely many requests they actually want to make next. One term for this is "service description". Another is "hypermedia". When I say "hypermedia" I just mean links and forms, like you have in HTML.

I'm going to talk about several common objections to WADL, and then put a new objection into circulation which I think is better. A lot of this has been said already, back and forth, but I wanted to gather a bunch of thoughts into one place. I'm taking most of these these from entries in the weblogs of Mark Baker, Aristotle Pagaltzis, Dare Obasanjo, John Heintz, and Patrick Meuller, but also from general murmuring I've heard on the web.

Disclaimer: despite the fact that I wrote a frickin' book about this stuff, I don't think of myself as a "REST expert" and I don't have anything in particular invested in WADL. I just like it as a hypermedia format.

WADL encourages/requires/represents the RPC style.

I'm putting this one first because I think something like this underlies many of the ideas that some standard part of HTTP is what you really want instead of WADL. We all take it for granted that you should rarely if ever need to expose an RPC-style interface through HTTP. It follows that if I fall into the RPC style, I'm probably not using HTTP correctly.

So, here's some WADL for your delectation, a partial description of a service that supports the Atom Publishing Protocol. It's from RWS but it's very similar to stuff in section A.2 of the WADL standard.

  <resource id="atom_collection" path="{feedID}">
    <resource href="#member" />
    <method href="#getCollection" />
    <method href="#postNewAtomMember" />
  </resource>
...
  <method name="POST" id="postNewAtomMember">
    <request>
      <representation href="#entry" />
    </request>
  </method>

You might look at that and think "They've taken a RESTful POST and renamed it 'postNewAtomMember'! Those RPC bastards!" Except "postNewAtomMember" isn't the name; it's the unique ID. The name is still "POST". An HTML form can have an ID, and so can a WADL method. The ID is optional, and its job is to distinguish this POST method from other POST methods so you can reference it from elsewhere. Same as with HTML forms. That "RPC-style" method (once you multiply out the reference to #entry) has the same meaning as this HTML form:

<form action="post" id="postNewAtomMember" 
      enctype="application/atom.entry+xml">
</form>

OK, I'm not naive. I know that lots of people are going to want to make tools that take that WADL file and generate something ugly like an AtomService class which defines a method postNewAtomMember. When they should be doing something non-ugly like defining a "resource" object or class which responds to post.

But the solution is to educate people about REST so they don't make bad tools (and bad services). There's nothing here that's not in HTML forms.

The uniform interface is good enough.

If the uniform interface was good enough, we'd have no hypermedia. The uniform interface restricts the methods you might apply to a resource. It doesn't tell you which methods are applicable to a specific resource, what the representation to a POST or PUT request should look like, which resource the client wants, or how to get from one resource to another. We use hypermedia for that.

There is an HTTP method called OPTIONS, which is supposed to advertise the capabilities of a resource. The first problem is that nobody uses OPTIONS. The second problem is that the only defined behavior for OPTIONS is to say which methods of the uniform interface are supported (in the Allow header). I'd like to try a system where sending OPTIONS to a URI gets you a hypermedia description of that resource, and see where that goes.

(The uniform interface plus) Media types are good enough.

They're usually not good enough to describe the representation for a POST or PUT request. Think about HTML forms. Specifying the enctype as application/form-encoded doesn't save you from having to define the form fields. The client needs some way of mapping its internal state onto the media format the server expects. This can be described in English text, or it can be described in hypermedia, like WADL.

If there's a custom media type for the specific XML vocabulary you're using, the media type might be good enough. The catch is that your XML vocabulary must also be usable as a hypermedia format. Atom is a good hypermedia format because it's got the atom:link element. atom:link has attributes like "rel" and "type" that let the server add semantic meaning to a link for a client to pick up on. But if you need to specify a hypermedia form (see below) instead of just a link, you'll need to interpolate some WADL or XHTML into the Atom file.

You can't get away from hypermedia. Neither the uniform interface nor media types help you describe the levers of state or connect resources to each other.

Links are good enough.

Links certainly are good, which is why as a hypermedia format WADL describes them. If the XML vocabulary you're serving isn't a very good hypermedia format, if your "links" are just URIs stuck somewhere in an XML document, a WADL addendum can explain what the URIs are pointing to and what they mean. WADL links have useful semantic attributes like rel.

But links are not good enough. They're not good enough for the WWW. Lots of RESTful, resource-oriented websites make their clients do things other than follow links. If links were good enough, Google would look like this:

Hypermedia consists of links and forms. XHTML is one way of expressing forms, and WADL is another. AFAIK they are the only two XML vocabularies that can describe hypermedia forms. [Update: I forgot XForms, OpenSearch, RDF Forms, and Poland.]

There are two kinds of forms: forms for manipulating application state (where the action is GET) and forms for manipulating resource state (where the action is POST, PUT, or DELETE).

An application form helps the client figure out the next URI it should GET. This is how a search engine form works. There are a set of data structures that need to be filled in ("q should be a string") and a set of rules for turning that data into a URI (http://www.google.com/search?q=jellyfish). WADL does the same thing, with some additional features like URI templates.

The client is not making assumptions about the URI space. It's calling in structured promises that the server made earlier, possibly in the response to the previous GET. These promises were represented in hypermedia, in WADL as they are in HTML. If HTML 5 was out and had URI Templates, the hypermedia could always be HTML, but right now there are cases where it has to be WADL.

A resource form helps the client figure out how to format the representation it sends along with POST or PUT to a particular resource. This includes the media type (eg. application/form-encoded) as well as a description of what the document should look like, possibly with reference to some XML schema. Lots of people (inc. me) hate XSD, but that's how you describe what an XML document should look like.

RFC2616 is enough.

It's not enough for the WWW. We also have a hypermedia standard: HTML. HTML is not just another media format like PNG. Media formats convey state. Hypermedia formats convey the levers of state. They show the client how to turn its internal desires ("I want the next item", "I want to spawn a subordinate resource with properties x,y,z") into a next request whose URI and representation the server will understand.

Okay, RFC2616+HTML is enough.

Maybe. I haven't heard anyone say this, but I've thought it. If you grant the HTML5 improvements and URI Templates, HTML is good enough for a large number of real-world services.

I know this because in chapters 5 and 6 of RWS I designed a fairly complicated web service that was completely described in machine-readable ways, and I didn't need a line of WADL to do it. Somebody else could write WADL for that service, just as someone else can write their own link-filled HTML guide to crummy.com, but I didn't need any WADL because I chose a different hypermedia format: HTML.

HTML is missing some of WADL's nice features, but if people start serving HTML representations instead of XML representations, then WADL might die out.

It's better to program directly to the HTTP client library.

A properly written (ie. not sneakily RPC) WADL library is an HTTP client library. My Ruby WADL library is pretty lame, but it's an HTTP library. The methods that trigger HTTP requests are get, post, put, and delete. It doesn't look like most other HTTP libraries, because its metaphors are resources and representations and links, where most existing libraries are based on the request/response cycle. But it's not hiding HTTP under something else, any more than Restlet's client library is.

WADL files are monolithic, like WSDL files.

I haven't heard anyone say this explicitly, but it's a reasonable complaint, and it's one of the things that worries me. That's why in RWS I proposed WADL "snippets" that describe a resource's behavior under the uniform interface. These snippets would be interspersed in an XML representation, the way links and forms are interspersed in an HTML representation.

Monolithic "site map" hypermedia files are a problem because they tend to be cached a long time and treated as unchanging. When hypermedia snippets are served along with representations, a client will hear about any changes in the hypermedia description of a resource. It will also hear about any changes in the relationships between one resource and others. A client can automatically adapt to some changes, assuming the semantic cues stay constant. But when the client does all its work off of an old WADL file which contains all hypermedia for the service, it won't know if the WADL file changes.

To sum up, I'd like to respond to something Aristotle wrote:

So this is how you describe a REST service: you explain the type of response body your server will send, where to find links in it, and what sort of semantics and verbs a particular link implies about the resource it points to.

This is exactly what WADL is for. You can describe a RESTful service in English text so that a human being can write a custom (and probably procedural) client. Or you can describe it in hypermedia so that a client library can present it in terms of the resources you were thinking about when you designed it. Right now your options for rich hypermedia are HTML and WADL.

I'm perplexed why your transaction example in Chap. 8 didn't simply create a transaction resource that included both checking & savings account as well as the transfer amount w/ a PUT (defining the resource as XML in the request body). Then you could simply call /transaction/11a5/commit or even just assume that this is a request to commit the transaction by default and avoid the 2nd call altogether. Is there a specific reason why it was not done this way? I can already see the "REST-haters" rolling their eyes to this three request/response transaction pattern.

The short answer is that if I'd presented it that way, the "REST-haters" would have an even better reason to roll their eyes: it would look like I couldn't think of a resource-oriented way to do transactions and I'd had to fall back to the RPC style.

Let me explain. When I read this question my immediate vision was of a request that looked like this:

PUT [some URI] HTTP/1.1
Content-Type: application/xml

<transaction type="transfer" amount="50.00">
 <source href="/accounts/checking/11" />
 <destination href="/accounts/savings/55" />
</transaction>

I don't know if this is what Scott had in mind, but let's run with it. This actually won't work as is: PUT only works when the client knows the URI of the new resource, and the URI for a transaction resource is chosen by the server. That's a minor problem we can fix by sending POST to a "transaction factory" resource:

POST /transactions HTTP/1.1
Content-Type: application/xml

<transaction type="transfer" amount="50.00">
 <source href="/accounts/checking/11" />
 <destination href="/accounts/savings/55" />
</transaction>

My first suggestion about transactions (page 231) was: "You can expose simple transactions as batch operations..." This is a kind of batch operation (albeit not quite the kind I had in mind), one that affects two resources identified by URI.

The problem with this representation is that it makes the second HTTP request look a lot like an RPC-style overloaded POST request. It looks less like an object the client is creating, and more like a command that /transactions is supposed to execute.

I wouldn't go so far as to say it is overloaded POST, because its result is a new addressable resource (the transaction record). There's a fuzzy overlap here between RESTful and RPC-style, just as there is with a request like "GET /rest-endpoint?method=photos.search&tag=penguin". But in the book I don't design URIs that look like method calls, because it sets a bad example, and for the same reason I don't design representations that look like commands. If you looked at that second HTTP request in isolation, you'd probably think "okay, there's no fundamental difference between RESTful and RPC-style transactions". When in fact there is a more elegant way to do them that looks nothing like the RPC style.

Why do I think the way I showed in the book is more elegant? Mainly because requests on transaction views work exactly the same way as would non-transaction requests on the underlying resources. If you could change an account balance at will, you'd PUT to /accounts/savings/55. You can't, so you create a transaction and PUT to /transactions/11a5/accounts/savings/55.

Two lesser reasons. First, you don't have to do the whole transaction at once. You can build it up on the web service as real-world events happen on your end, and commit when you're ready. It works the same way as a transaction in a programming language. Second, a transaction is an addressable resource even while in progress. Multiple authenticated clients can inspect or collaborate on a transaction, and the rules for avoiding conflicts are the same as on the rest of the web (eg. If-Unmodified-Since).

That leaves the question of efficiency. You can package multiple operations into a single HTTP request for efficiency's sake. But if a single operation is an HTTP method applied to a URI, that's what you should package. If the uniform interface is good enough for one-at-a-time operations, it's good enough for descriptions of batch operations. You don't need an application-specific vocabulary like the one I came up with reflexively. A simple, uniform vocabulary will do; something like this:

<transaction>
 <request method="PUT" target="/accounts/checking/11">
  <representation>balance=150</representation>
 </request>

 <request method="PUT" target="/accounts/savings/55">
  <representation>balance=250</representation>
 </request>
</transaction>

(You could probably use an XML document that contains a series of SOAP envelopes!)

I think POSTing that representation to a transaction factory satisfies Scott's original criteria: "create a transaction resource that included both checking & savings account as well as the transfer amount... defining the resource as XML in the request body". And it would make a good addendum to the "Transactions" section, once the reader has seen which HTTP requests underly the transaction. But presenting it instead of the multi-request version would probably be confusing, and would certainly give the impression that there was no RESTful way to make those HTTP requests one at a time.

• /releases/104.en
• /releases/104.txt
• /releases/104.en.html
• /releases/104.es.txt
• etc.

When a client requests one of those URIs they can leave out one or both of the Accept- headers and still get what they want. The Content-Location response header is set to /releases/104 so that you have a URI to use when talking about the press release in general, rather than a specific version of it.

The question in the weblog entry I can't find is more or less this: what about the other two Accept- request headers, Accept-Charset and Accept-Encoding? Why don't I recommend exposing URIs like /releases/104.es.txt.gzip.UTF8?

The reason I put any special levers in the URI is because we pass around URIs, not URIs plus headers. Lots of programs and services take a URI, perform a GET on it, and expect that they got what you told them to get. One example I give in the book is the W3C HTML validator. If the only URI I exposed was /releases/104, there'd be no way to validate the Spanish version separate from the English version. If the default representation format for /releases/104 was plain text (unlikely, but this is just a thought experiment), there'd be no way to validate the HTML formats at all.

To the extent that this reason applies to some piece of information, I argue for putting a lever for it in the URI. Obviously it applies to things like which press release you want. I think it also clearly applies to the language and the data format. I think it doesn't apply to compression or character encodings. Here's a quote from page 243, in the section "Compression" where I talk about Accept-Encoding:

You probably remember that I think different representations of a resource should have distinct URIs. Why do I recommend using HTTP headers to distinguish between compressed and uncompressed versions of a representation? Because I don't think the compressed and uncompressed versions are different representations. Compression, like encryption, is something that happens to a representation in transit, and must be undone before the client can use the representation. In an ideal world, HTTP clients and servers would compress and decompress representations automatically, and programmers should not have to even think about it.

We can argue over what counts as a different representation (as you can see I take a fairly high-level view), but even if you think the compressed and uncompressed data are different representations, this isn't a difference that needs to go into the URI. A client can be automatically programmed to detect a compressed representation and uncompress it. The key is this ability. If it were possible to algorithmically translate any human language into another, or any data format into another, there'd be a much weaker case for extending the URI with levers for language and data format.

The same logic holds for character encodings, because a client can be automatically programmed to convert any character encoding into Unicode. The case is weaker because 1) the programming is difficult unless there happens to be a library for your language, and 2) you only get compression if you ask for it, but you get a character encoding whether you like it or not.

If you're exposing a resource in multiple encodings, and you have reason to believe that a URI-driven client might choke on your default encoding, then sure, put an encoding selection lever in the URI. But at the risk of sounding provincial (if being too cosmopolitan can be a kind of provinciality), your default encoding ought to be UTF-8 or UTF-16.

Aron calls this "a well considered example of a discipline-specific API, far outside the more typical realms of computing, e-commerce, and social software." I'll add that it's odd to see a page full of jargon, half of which you understand and half of which is totally opaque.

Much as I dislike the way SOAP is used these days, I'm not inclined to gloat, because the deprecation just means more work for me. I used the SOAP search service as an example in all three of my books. Now I've got to find another free, public, non-obscure SOAP service to use as an example (ideas?).

The official Google narrative is that the SOAP-based web service has been replaced by something called the "Google AJAX Search API". If you take this narrative at face value it means that Google has taken down their web service and put up an AJAX library in its place. What's AJAX? Who knows, it's AJAX. Here's a typical weblog entry on the topic.

Another victory for REST over WS-*? Nope -- Google doesn't have a REST API to replace it. Instead, something much more important is happening, and it could be that REST, WS-*, and the whole of open web data and mash-ups all end up on the losing side.

It's probably my recent proximity to Sam that's doing it, but I'm noticing a tendency in myself to draw fine distinctions. There's a sense in which this narrative is right and a sense in which it's not. I'm going to pick apart the narrative and show what exactly is disturbing about the Google AJAX Search API.

It's not true to say that "Google doesn't have a REST API to replace it." In fact, Google has two REST APIs, and one of them predates even the SOAP API. You've probably used this old API: its primary endpoint is http://www.google.com/.

Yes, the Google website is in fact a very RESTful web service. The downside of this web service is that it's a little bit difficult to use automatically, as opposed to through a web browser. It serves data in a human-oriented format (HTML), and you have to screen-scrape it into a data structure if you want to do anything for it.

There are libraries for doing this, but the other problem is Google doesn't want you to do it. It violates Google's Terms of Service ("No Automated Querying"). Lots of inconsiderate people write scripts that hammer Google's REST API day and night. Google tries to prevent this by sniffing out anything that might not be a web browser and preventing it from accessing the API. (To see this, set your browser's User-Agent to "libwww-perl" and try to use Google.)

But it can't be denied that people outside of Google have a powerful hankering for Google's dataset, so eventually someone (Nelson Minar, it seems) came up with a second web service API that was designed just for use by automated clients. The catch was that you had to sign up for a unique key to use it, and that key would only work for you 500 (later 1000) times a day.

In point of fact Nelson chose a SOAP/WSDL architecture for this web service. But there was no need to use any different architecture at all. Here's a possible different way of implementing the constraints above:

When you make an HTTP request to google.com, we try to figure out whether you're a web browser or an automated client. Ordinarily, if you're an automated client, we shut you out. But here's the deal. Now you can sign up for an "automated client key". When you make an HTTP request to google.com, stick your key into the Authorization header. Not only will we not shut you out, we'll try to make things easy for you. Instead of a human-oriented HTML document, we'll send you the appropriate data in an easy-to-parse XML format. But, we'll only do this for you 500 (later 1000) times a day. Then it's back to shutting you out.

This technique has a number of subtle benefits which I could bore you with for quite a while. But its obvious benefit is that it's got the exact same "API" as the Google website, which everyone knows how to use.

Anyway, instead of going down a route like this (which would, I think, have changed the history of web services quite a bit), Google went down the SOAP/WSDL route. Now they're deprecating the SOAP service in favor of some mysterious "AJAX API". This brings me to the second of Google's REST APIs.

There is no magical thing called an "Ajax request". An Ajax client makes normal HTTP requests, and processes the results automatically, just like a web service client. An Ajax client is a web service client.

What HTTP requests is the Google Ajax client making? I poked around a little bit and it looks like it mainly makes GET requests to URIs that look something like http://www.google.com/uds/GwebSearch?callback=GwebSearch.Raw&context=0&lstkp=0&v=1.0&key=xxxxxxxxxx&term=web+services. That's not exactly http://www.google.com/search?q=web+services, but it's not too far off either.

The Google AJAX API consists of a browser-side Javascript library and a server-side web service. The one acts as a client for the other. From what little I've seen of the web service I'd consider it quite RESTful. In fact, it's architecturally very similar to Yahoo!'s RESTful search API. They both use the same (IMO, fairly unsafe) trick to get a web browser to execute dynamically-generated Javascript code from another domain.

The main difference is that Yahoo's search API can also be made to send data (in JSON or ad-hoc XML format) instead of executable Javascript. That makes it possible for the service to be consumed by automated clients, not just by web browsers running client-side Ajax programs.

Let me just see if I can do something similar with the Google web service. The Javascript it serves is extremely close to also being a JSON document; I should be able to hack it a little and parse it as JSON.

Here's some Ruby code that gives you kind of a command-line Google search like people used to write for the old SOAP API. It requires the json gem.

You can skip the code.

#!/usr/bin/ruby
require 'rubygems'
require 'uri'
require 'open-uri'
require 'json'

KEYS = %w{GsearchResultClass unescapedUrl url visibleUrl cacheUrl
          title titleNoFormatting content results adResults
          content1 content2 impressionUrl}

def search(key, term)
  uri = "http://www.google.com/uds/GwebSearch" + 
    "?callback=GwebSearch.Raw&context=0&lstkp=0&v=1.0" + 
    "&key=#{key}&q=#{URI.escape(term)}"
  javascript = open(uri).read
  
  # Hack quotes around the hash keys to make the Javascript string
  # into JSON.
  KEYS.each do |key|
    find = Regexp.compile("\s*#{key}\s*:")
    json.gsub!(find, " \"#{key}\" : ")
  end 

  parsed = JSON.parse(json)
  return parsed["results"], parsed["adResults"]
end

# Command-line interface begins here

(puts "Usage: #{$0} [API key] [search term]"; exit) unless ARGV.size == 2
key, term = ARGV

results, ads = search(key, term)
puts "#{results.size} results for '#{term}':"
results.each do |result|
  puts result['titleNoFormatting']
  puts " #{result['url']}"
  puts " #{result['content'][0..70]}" unless result['content'].empty?
  puts 
end

unless ads.empty?
  puts "Look at some ads while you're at it:"
  puts '------------------------------------'
  ads.each do |ad|
      puts ad['title']
      puts ad['visibleUrl']
      puts " #{ad['content1']}"
      puts " #{ad['content2']}"
      puts
  end
end

Now, in old episodes of MacGyver, whenever MacGyver built a bomb out of baking soda and masking tape, the writers would change some crucial detail (like change the masking tape to Scotch tape) so that if kids copied MacGyver they wouldn't blow up the house. I've done something similar here. I've removed a crucial line of code from that program, so that people don't just go copying it and running it all over the place.

Why did I do that? Because when it works, that program violates the Google AJAX Search API Terms of Service. "The API is limited to allowing You to host and display Google Search Results on your site." I can use the old SOAP API to write a command-line search tool, but I can't use the new, RESTful API in that kind of application. My users can only access the RESTful API through a specific library (Google's Javascript library), running in a specific way (in their web browsers), for a specific purpose (displaying search results).

Wait a minute... running only in a web browser? Terms of Service? Bootleg scripts that hack the output into something a parser can understand? This REST web service is made available on exactly the same programming-unfriendly terms as the Google website "REST web service"!

Instead of screen-scraping a web page, I'm now screen-scraping a web service. I'm reverse-engineering undocumented URI formats, just like I do when I screen-scrape. So far, there's nothing on Google's end that sniffs my user-agent to make sure the web service only runs in a browser, but you can bet there will be as soon as that becomes a problem for Google.

The "blow to web services" narrative is incorrect. Google did in fact deprecate their SOAP API and expose a RESTful API. A win for REST!

Though incorrect, the "blow to web services" narrative is also correct. Google deprecated their SOAP API, exposed a RESTful API, and then erected a bunch of technological and legalese barriers around any attempt to actually use the RESTful API. You're only allowed to use it through one library in one language in one environment for one purpose. A loss for everyone!

On the level of technological choices, this move is a big improvement. They've gone from SOAP, which has a lot of overhead, to plain old HTTP, which has strictly less. Gone from an RPC style, which doesn't play well with the web, to a RESTful style, which does. This makes an enormous amount of technological sense. From its first day on the web, Google has exposed its dataset through a RESTful interface that gets orders of magnitude more traffic than any "web service" it might expose. In a sense, all they're doing now is unifying the architectures.

When it comes to getting information into the hands of people who can use it, Google has taken a big step backwards. The SOAP interface was serious overkill, but what you did with it was your business (though you could only do it 1000 times a day). The new RESTful interface is a technical improvement, but it's encumbered with restrictions that make it a museum piece. Unless you're writing an Ajax application using Google's library, its true value can only be obtained illicitly. And that's the other reason why I'm not inclined to gloat.

Pete quotes Anne Thomas Manes as saying:

Service oriented architecture (SOA) is a software design discipline in which application and infrastructure functionality are implemented as shared, reusable services.

As Pete points out this definition has a dependency on the definition of "service". Pete's definition is "a network available collection of related operations".

Okay, I'll buy that, and I'll also buy a $0.50 contract on Pete's "a resource is not a service". I'd say a resource is a service, albeit a very limited one that can only operate on one piece of data. An RSS feed is a good example of a one-resource service. But a collection of resources is definitely a service. In fact, "service" is what we call a collection of related resources.

A single resource can't have arbitrary operations, but you can expose arbitrary operations by exposing combinations of resources. It's not the same design as an RPC service, but only intertia associates RPC with "service-oriented" in the first place, and a service-oriented system could easily consume a mix of resource-oriented and RPC-style services.

The MOA field is dominated by RPC-style services, most of which use SOAP messages constrained by WSDL files. The client communicates its desire by sending a message (a box with stickers on it) to a message processor, which unpacks the box to find both the scoping information and the action information.

We've got two edge cases to explain: overloaded POST and so-called "HTTP+POX" services. It's been said (see link at end) that I shouldn't try to fit HTTP+POX services anywhere at all because they're just a mishmash, but I press on.

We've also got an unexplained coincidence. I've been describing SOAP as a way of putting a document in a box and slapping stickers on it. But you can look at HTTP the same way. The HTTP request (or response) is the box; the entity-body is the document; and the URI, headers, method name, and response code are all stickers.

HTTP+POX, as they say, is a mishmash, but I think I can identify the things it's a mishmash of. An HTTP+POX service is a message-oriented RPC-style service that uses HTTP as the box format, rather than using SOAP.

This is most obvious if you look at how you modify the dataset in HTTP+POX services (I'm using the del.icio.us API and Flickr's allegedly "REST" API as examples). You either send a GET or an overloaded POST to a URI that's not a resource: it's a description of the procedure you want to call. There's a message processor for every operation, rather than just one (as generally happens with SOAP+WSDL services), or one for every object on which you might want to operate (as with resource-oriented services). Examples: del.icio.us /v1/posts/add and /v1/posts/delete, Flickr's multitude of methodName arguments.

The confusing part is this: when the RPC method you're calling is "get some data", the HTTP method is usually GET, and the arguments to the RPC method usually go into the URI. This URI designates a resource, and you can GET a representation of that resource! This is why I thought that HTTP+POX services are hybrids of the resource-oriented and message-oriented architectures. In many cases, the message is one you'd send if you were using the uniform interface to get a representation of a resource. The service may not have been designed with resources in mind, but this part of it is functionally resource-oriented.

Examples. Consider an endpoint of the del.icio.us API: https://api.del.icio.us/v1/posts/get?tag=restbook That doesn't have the same ring as http://del.icio.us/leonardr/restbook, but it's the same kind of URI. Those are two URIs to two representations of the same resource: "my recent posts tagged with 'restbook'". There are infinitely many URIs of this form, each identifying a different resource.

Similarly for http://api.flickr.com/services/rest?method=flickr.photos.search&api_key=xxx&name=penguin. We're supposed to interpret that as a remote procedure call, but it's also the URI to a resource: "Pictures tagged 'penguin'". Another URI to a different representation of the same resource is http://flickr.com/photos/tags/penguin. Even a URI like /rest?method=flickr.people.findByEmail&find_email=leonardr@segfault.org is the URI to a resource, though a better URI for the same resource might be /people?email=leonardr@segfault.org.

We have our own recommendations about how to structure URIs in a resource-oriented service, but it's pretty small-minded to say that services aren't resource-oriented just because their URIs look funny and contain things that look like method names (like "get" and "method=flickr.photos.search"). From a functional standpoint, these services stop being resource-oriented when they stop using resources. And, in general, they stop using resources when it comes time to modify the data set.

The method names in those URIs are useless to the extent they agree with HTTP's uniform interface. The "get" and the "flickr.photos.search" basically mean HTTP GET, and the client's already using GET. When those strings become useful ("add" or "flickr.photos.comments.deleteComment") it's because the RPC interface supercedes the uniform interface. The URI can no longer be conceived as pointing to a resource.

/rest?method=flickr.photos.comments.deleteComment&comment_id=100 could be one of many URIs to the resource "comment #100". Maybe the "method" argument is just random junk. Maybe if you GET this URI you get the comment, and if you delete this URI you delete the comment. But of course if you GET this URI you delete the comment. This URI doesn't identify the comment: it identifies an operation on the comment. That's not a resource. It's just a procedure call.

You can see this by looking at web applications. A web application like a search engine is pretty resource-oriented. It exposes an infinite number of URIs that slice up the search engine data in various ways: by query, by page, by language searched. All of these URIs point to resources. You can't see any cracks between the message-oriented and resource-oriented models, because 1) the "messages" you're sending are standard HTTP requests, and 2) you can't modify the search engine data through the web interface, so you only use GET and you use it as intended.

Then again, consider NewsBruiser, the weblog program that publishes these words. I wrote NewsBruiser before the term "REST" was coined, but it's pretty typical of examples even today. I've got URIs like /nb.cgi/add, /nb.cgi/comment-add, /nb.cgi/configure, and so on. In general GETting one of these URIs gives you an HTML form, which you fill out and POST to the same URI to modify the data set.

Even putting aside the fact that you can't do PUT or DELETE with HTML forms, this is a very RPC-oriented application. It's got method names in the URIs. Yet when you're not changing the application you're using URIs like /nb.cgi/view/nycb/2006/11/03/1. That's got a method name ("view"), but it's redundant with the name of the HTTP method (GET), just like with the del.icio.us and Flickr examples. The rest of the URI describes a resource: the second News You Can Bruise entry of the third of November, 2006. Indeed, I've got a rewrite rule that cuts out all the redundant and default data so that you can use /2006/11/03/1 instead.

So that's HTTP+POX. It looks like there are two main paths to this architecture. You might come to it by designing web services on the exact same principles as web applications. This overlaps with the resource-oriented architecture when that architecture congrues with the capabilities of web browsers (GETting data from resources identified with URIs, sending data to hard-coded locations). It departs from the resource-oriented architecture when it comes to things you can't do with a web browser (using PUT and DELETE instead of overloading POST so much, sending data to URIs that the client generated dynamically).

Or you might come to it through an attempt to simplify the RPC style. You might be doing pushbutton SOAP+WSDL services and one day say "screw only having a single endpoint; I'm going to have an endpoint for every procedure and it's going to be identified in the URI and then I won't need WSDL". Or you might say "screw putting an XML document in an XML box; I'm already using HTTP, I'll just use it as my box and then I won't need SOAP".

Now a word about the name. I've decided "HTTP+POX" is inaccurate and I think I won't use it in the book except insofar as I need to translate real-world terms into my wondrous precise terminology. Here's the problem: like "AJAX" of old, "HTTP+POX" hard-codes a reference to XML where there might not be any XML.

To reiterate one of my earlier points in new language: what's "Plain Old" about Plain Old XML? What's "Plain Old" is that it's a document, not a box with a document inside it. You've already got a perfectly good box: the HTTP request. But that's a statement about the box. "POX" is a statement about the contents, and those don't have to be XML. An HTTP+POX web service might serve plain text, JSON, HTML microformats, or graphic files in an HTTP box. This holds even if you don't buy my argument that HTTP+POX is an RPC-style architecture.

I can think of two different names. One involves doing what they did to AJAX and lowercasing the "POX", making it a word instead of an acronym: HTTP+Pox (HTTP can stay an acronym because you are using HTTP and only HTTP). This has the advantage that it's an almost invisible change. But depending on how cocky I feel about my classification of these services as RPC services, I may try to push the term HTTP+RPC. Because "Pox" doesn't mean anything, and I don't like the trend towards stripping acronyms of their meaning when it becomes inconvenient (as also happened to SOAP).

The next installments will cover what to do with terms like "Service-Oriented Architecture" in the face of definitions that would a priori prevent REST from getting any of the pie, and also finally tackle the question of overloaded POST in resource-oriented architectures. I think it's going to be a while until the last one, at least, shows up. Before I try to figure out overloaded POST I want to incorporate into the book text my thinking in the series so far.

The Ajax chapter is also done, and I'd like someone who knows more than I do (ie. something) about Ajax to look at it.

• How is the request scoped? How does the client tell the server what part of the application it wants to operate on? Let's say the purpose of the request is to fetch some data. Why should the server send back data X instead of data Y?
• How does the client convey to the server what it wants the server to do? Why should the server send the requested data instead of modifying it somehow?

Now I'm going to put out a description on the same principles of the Message-Oriented Architecture. Then I think I can make a case for how to classify weird cases like HTTP+POX.

To state the obvious (though it stops being obvious if you or we call the MOA something else), the Message-Oriented Architecture is about passing messages back and forth. Client sends server a message, server takes some action and sends client a message in response. A SOAP box with stickers on it is an example of a message. An HTTP request with headers is also an example of a message.

In an ideal MOA there is no information outside of the message. So if the message is in a SOAP box being transmitted over HTTP, then all the features of HTTP are just distractions and should be used as little as possible. There should be no leakage from the message into the transport protocol.

In the ROA, the fundamental server-side object is the resource. An ROA service usually has an infinite number of resources. In the MOA, the fundamental server-side object is the message processor, sometimes called an endpoint. A MOA service exposes a small number of message processors: usually one. A message processor is the thing that has a URI (or, possibly, the thing that can be the target of the WS-Addressing SOAP sticker).

Side note: A resource is just a message processor with a very limited vocabulary. So there's a pretty easy conceptual translation between the ROA and the MOA. This makes sense because HTTP is itself a message-oriented protocol. Of course, almost all ROA services expose too many "endpoints" to list. You either generate their URIs according to a rule, or you follow links in a document the server sends you.

An MOA might have ten or 200 or a million endpoints, but it's not ROA unless it has an endpoint or "resource" for every object the client might manipulate. Any service that exposes an infinite number of endpoints has at least some of the ROA in it.

This ties back into the famous triangle of "nouns" (resources), "verbs" (HTTP methods), and content types. The ROA constrains the "verbs" and lets the "nouns" run free. The MOA constrains the "nouns" and lets the "verbs" run free.

Side side note: an ROA service can contain a finite number of endpoints, but only if there's a finite number of objects the client might manipulate. A static website (like your cat's homepage, or the site that serves Google Maps image tiles) is an ROA service with a finite number of endpoints. A web service version of Parmenides's model of the universe would only have one endpoint, because there is only one proper object of discussion.

Now let's consider how the MOA answers the two questions:

How is the request scoped? Why should the server do x to data D1 instead of data D2? Well, in the URI it's scoped to the message processor. If your MOA service is actually ROA then that's all you need, because the "message processor" is the "resource" is the part of the application you're operating on. Otherwise, you need some additional scoping, and this is done by putting information in the message itself. This might be in the box or it might be in one of the stickers; the details don't matter.

How does the client convey to the server what it wants the server to do? Why should the server do x to data D1 instead of doing y to data D1? This too is scoped by information within the message itself. Again, this might be in the box or in one of the stickers; it doesn't matter to the MOA. Unlike with the ROA, the vocabulary of this sort of information is not constrained at all. If you constrained it you'd need to create more endpoints to handle the same actions with the more limited vocabulary, and you'd move towards the ROA.

Since in the MOA both of these pieces of information go into the message, there's a tendency to conflate them, and indeed that's what I used to do.

To recap from last time: the ROA puts scoping information in the URI, and puts the action information in the HTTP method. The big exception seems to be overloaded POST, where the action information can go in the entity-body. I'm now fairly certain that overloaded POST in the ROA is best described in terms of the MOA; I'll try to cover that next time.

Now I'd like to talk about a common design style for MOA services: the RPC style. In an RPC service, the scoping information corresponds to the information that would identify an object if this were an object-oriented program. The action information corresponds to a method that should be called on that object. As is proper with the MOA, both of these go into the message.

The message also contains some additional information, which corresponds to arguments passed into the procedure. The scoping information can be found among these arguments: things like the bank account number or the user ID. This is because an RPC service isn't really object-oriented. If it was, you'd be able to work directly with the objects (as, say, resources) instead of going through message processors. But with a little work you can get the same kind of "fake" OO you see in the GNOME project's C code, where the first argument to every function is a pointer to the "object". I put "fake" in scare quotes, possibly sending the most mixed message ever, because I think you can do real OO design in a non-OO system. But it looks strange, and you have to do work to get it.

In an RPC-style service, the HTTP request-response cycle is used to simulate a method or function call in a programming language. The request is the method invocation, and the response contains the return value of the method, or the exception it throws.

We all know that method calls are not neccessarily simple stuff where I ask you for information and you return it in a data structure. This "method call" might do anything: fetch information, store information, put a job in a processing queue, set up callbacks for later, etc. The response message might contain useful information, it might describe an exceptional condition, or it might just be a formality. If you register a callback the real data might be coming in later through the callback. So saying "RPC style" is not a negative value judgement about the capabilities of the style.

Let's take an easy case: XML-RPC. It's obviously an MOA architecture, since the only URI in an XML-RPC service is the URI to the XML-RPC service. It also rejects HTTP headers, methods, and status codes almost entirely, preferring to convey all information information in its XML message document.

It's also obviously an RPC architecture, because 1) it says so right in the name, and 2) its messages are full of incriminating tag names like methodCall and params.

Now, what about SOAP? Is SOAP an RPC style? No, that's a category error. SOAP came from XML-RPC (more accurately: they have a common ancestor) but they aren't the same kind of thing. XML-RPC is a way of describing method calls; SOAP is a way of sticking a message in a box and putting stickers on the box. This makes it useful for any MOA application, but it doesn't make it RPC.

So, riddle me this, Batman: how come basically the entire installed base of services that use SOAP also use the RPC style? Why does the average programmer think SOAP and then think RPC style?

SOAP is the message format of choice for automated tools that take Java or C# code you've already written, and with one click turn it into a web service. These tools are to web services as FrontPage is to web pages. The resulting "web service" exposes only one URI and accepts methods only in a very specific form that a person can't comprehend without a detailed map (a WSDL file) and a software tool on the other side (a WSDL client) that can read the map so you don't have to. When the software tool is done reading the map it's reconstructed something like the method signatures of the original Java or C# code.

This is why REST people dislike these services so much, and why that dislike often spills over into SOAP. This is why people tend to think that anything that doesn't fit this model is "REST". This is why I say these services are not really on the web: they're looking out at the web through a little peephole. To get any information out of them, you have to construct a very specific document, and then you have to put in a box and put the right stickers on it.

These services have the same design as XML-RPC services. They use XML Schema instead of XML-RPC's idiosyncratic data serialization format, and they use WSDL instead of the listMethods extension to XML-RPC. All the work done to make SOAP nothing but a good way of putting an XML document in a box and slapping stickers on the box, has passed these services by.

I don't like this because it's ugly and it pollutes the web. SOAP fans don't like this because it conflates SOAP with RPC when what they use SOAP for isn't RPC at all. I'm not really clear on what MOA styles they are using if not RPC, but I don't think it's going to come up a lot in a book about REST and the ROA, so I can just leave them to it and have Sam act as my lookout in case anything relevant to the book happens.

Who's the culprit here? I think it's those tools. They make it easy to get a web service that has the same interface as your code (that'd be a Procedure-Call interface that works Remotely), disregarding the fact that the architecture of the web doesn't look like that. Combine this disconnect with the complexity of the interface and the idiosyncracies of the tools, and you've made it easy for someone else to work with your service if they've got the exact same setup, and very difficult otherwise. This is pretty disgraceful since the real-world power of the web comes from its ability to connect everyone together.

The secondary culprit is WSDL. WSDL doesn't enforce RPC but it makes life easy for tools that want to force SOAP into an RPC mold. Sam is ambivalent about WADL. I suspect he's afraid that WADL will make it easy to force HTTP into an RPC mold, that tools for doing this will proliferate, and that we'll never see resources again. Sure, WADL makes you specify your resources, but a smart tool could easily subvert that and create services with a single "resource" that handles a whole bunch of method calls.

I think the scenario I've ascribed to Sam is not inevitable. A lot of the mess we see today comes from historical contigencies like SOAP's heritage in XML-RPC. But to prevent a future without resources we need a basic book that's willing to go out and fight for them. It means we need to straighten out the terminology—even if only informally and we only use the terminology locally—so we can create a frame for our argument.

Next time: HTTP+POX.

Always with deliberate irony, but my infernokrusher prose on the REST book page has two other goals. First, to market the book, to make people excited about it so I can ask them for help. Second, to act as a guideline and a goal. As the book nears completion I will find myself wrestling with the question of whether the manuscript does or does not put the "web" back into "web services". If it doesn't then we should probably do some more work on it. Does it drive all opposition before it, or is it just okay? Etc.

I'm exhausted, so no continuation of the taxonomy series tonight. But I would like to give an update on the name of the book. I got about 20 emails about it, almost uniformly against "with Ruby" in the title. On a semirelated note I can report that "with Ruby" is almost certainly gone from the title. The title is not finalized; editor Michael thinks "REST Web Services" is ungrammatical. But I think we agree on what kind of name will capture the goal of the book: an introduction to REST that has contains real examples but also has some philosophical heft to it.

The best argument for "with Ruby", I think, apart from "cheap trick to sell more books", is the same argument for using Ruby as our implementation language. We're not using Ruby because its XML support or HTTP client library is the best (it's not), but because Ruby is where the open source REST work is happening. This gives us lots to talk about without having to be polyglot throughout the whole book. Right now you don't see things like ActiveResource happening in other languages. (Note: this statement is precisely calculated to bring people out of the woodwork saying "you've neglected my awesome non-Ruby server-side REST tool!" so that I can take a look at said tools.)

The counterargument is that most peoples' interest in REST doesn't start on the server side. It starts when they write clients for big-company services like Yahoo!'s and Amazon's and Google's. They hear the term "REST" used (possibly incorrectly), and want to know more about the underlying architecture. There will always be more clients than servers, and the client programmers are using all kinds of languages.

Those whose interest does come from the server side are either trying to decide what to use, or just curious about RESt. Inside the book it's fine to say "Like REST? Your best bet right now is Ruby." But the book will fly past a chunk of its audience if the title says: "Like REST and Ruby? You'll like this book."

However Sam and I did have a long IM conversation today, and cleared up a lot of taxonomy issues, as well as some of my misunderstandings about SOAP. I feel confident enough now to take another stab at my taxonomy of the programmable web. I got about halfway through before Ben came over to watch Battlestar Galactica, and that half is what I present to you tonight.

Again the disclaimer: though people will argue about these topics until the end of time, for purposes of this book we just need to get a coherent story about what it is we're cutting away as we zoom in on the programmable web to focus on the parts called "REST" and "ROA".

There are two major architectures on the programmable web. One is oriented around the passing of messages from place to place, and one is oriented around the manipulation of server-side objects called resources. We're (still tenatively) calling these the Message-Oriented and Resource-Oriented Architectures. Both architectures have real web applications and real web services to their credit.

These architectures can show up elsewhere than on the web, but we're not concerned with that. On the programmable web a service under either architecture uses the same underlying technologies: HTTP; a data format like XML, JSON, or form-encoded key-value pairs; and at least one URI.

Because they're built on HTTP, both architectures have a request cycle where the client asks the server to do something, and the server does it and sends a response. But we've found two questions which the MOA and the ROA answer differently:

1. How is the request scoped? How does the client tell the server what part of the application it wants to operate on? Let's say the purpose of the request is to fetch some data. Why should the server send back data X instead of data Y?
2. How does the client convey to the server what it wants the server to do? Why should the server send the requested data instead of modifying it somehow?

First a brief digression. When you send an HTTP request, you have two main options as to what to send in the entity-body. You can just send your data, or, if your data is an XML document, you can stick it in a box first. The name of the box is SOAP.

Just to avoid confusion: "SOAP" here is referring to the SOAP standard, not to the things people do with SOAP in the real world (that comes next). Sam says: "There isn't much to SOAP. It is three elements and a fault descriptor. Oh, that, and a world view." I'm just talking about the three elements and the fault descriptor. The worldview is the MOA.

Why would you stick your XML document in a box? Well, there are a variety of stickers that can be slapped onto the box without disturbing the contents. These stickers are the WS-* stack. That's the five-cent definition.

With that in mind, let's consider how the Resource-Oriented Architecture answers the two questions. A client for an ROA service scopes its request to a resource, by sending its request to a URI that identifies that resource. In an idealized ROA application there's one resource for every part of the application you might want to manipulate. Every resource has at least one URI, and you've got to send your request to some URI, so you might as well designate your resource by its URI. The URI goes in a special place in the HTTP request and can't be confused with anything else.

When conveying what it wants an ROA service to do to a resource, the client is restricted to a small and standardized list of methods. These methods are the famous HTTP GET, PUT, etc. Again, the HTTP method goes in a special place in the HTTP request and can't be confused with anything else.

If the client wants to convey an action that's not on that list, the problem is almost always that they've chosen the wrong resource. The service ought to expose some other resource for which the client's desired action translates naturally into one of the standard methods. The service might not actually expose this resource, but that just means the service can't do what the client wants. It's no different from the aggravation of a C library that's missing the function you need.

(There are edge cases I don't really understand, about overloading POST for atomic operations that span resources. I'm still trying to figure this part out. I think I may end up modelling this as a tiny MOA architecture inside the ROA.)

Now, depending on the service, when you need to send a document with your request, you might just send it as is, or you might stick it in a SOAP box first. The second possiblity, is, AFAIK, only theoretical. I don't know of any services I would describe as ROA where the client sticks its entity-bodies in a SOAP box before sending them.

This is partly for historical reasons. It's partly because it's simpler for the server to accept the entity-body as a document, instead of parsing it into a SOAP box covered in stickers, plus the "real" document. But it's mainly because HTTP already has stickers (HTTP headers). An HTTP request or response is already a box that can contain data and have stickers slapped on it.

HTTP stickers are more restricted in form than SOAP stickers, but they're good enough for most purposes. To the extent that ROA service programmers covet the features of WS-* stickers, they tend to devise ways of representing the same information in HTTP headers. Example: WS-Security's WSSE "ported" to the ROA as the X-WSSE header.

OK, enough about the ROA for a while. What about the MOA? And where do popular acronyms already in common usage fit in? Tune in tomorrow for the action-packed conclusion.

Good points, and highlighting an increasing source of uneasiness on my part. Which is good, despite the fact I generally don't like uneasiness, because it shows it's paying off to embarrass myself by formulating this book in public. People are interested enough in the topic to form uncoordinated revolts around ideas that are wrong or need work.

First a disclaimer about how much this matters: not a whole lot. This is a book about REST and the ROA. The name of the ROA might change yet for marketing reasons, but we've got its technical features pretty well defined. There is a chapter (#1) that covers the programmable web as a whole, a chapter (#3) whose first half covers services popularly considered RESTful that aren't really, and a chapter (#8) that compares REST/ROA to "the other guys".

I was calling "the other guys" "SOA" but that's starting to seem wrong. My current wild hare is to call them "Message-Oriented Architectures", though that has a bit of a collision with Message-Oriented Middleware. That way "Service-Oriented Architecture" is freed up to describe a higher-level architecture based on providing and consuming web services (ROA or MOA or both). Which seems closer to existing usage of "SOA", insofar as "SOA" means anything besides "your new issue of BusinessWeek is here."

Point is, our taxonomy could change completely and we'd only have to make real changes to those three chapters (and #8 isn't even started yet, partially because we're trying to sort this stuff out first). More important, much more detailed, and hopefully less contentious, will be our description of what goes on inside the "REST" part of the Venn diagram.

Anyway, our task here is the Confucian task of the rectification of names. We're trying to classify the set of techniques and technologies used on the programmable web. If it turns out to be best described with gradients instead of subsets, it might be a 2D or 3D map instead of a Venn diagram or taxonomy.

This doesn't have to be like the Linnean classification system where everyone signs onto it for all time, but it should be done with respect for the way "the other guys" see themselves. Sometimes in the text of the book we'll poke fun at what I call Big Web Services⁰. But we need to have a credible story about what Big Web Services really are, how they differ from RESTful web services, and what lives in between. Ideally someone who disagrees might say "Well, I wouldn't call it that, but I see how you draw the distinction."

So here are some unsolved mysteries of our taxonomy:

I don't think anyone will dispute that there's at least one rival web service architecture to REST/ROA: the RPC architecture that shows up in many web applications, in web services like Flickr's, and in XML-RPC. Can we consider this architecture as similar to the message processing architecture that rarely shows up in web applications, and that is commonly associated with SOAP? That is, granted there are differences, do they matter for a book about REST, or do they both differ from REST in a way we can capture in some snappy term like MOA?

Are all these architectures mutual opposites, or points on a gradient?

It it useful to distinguish between the typical robotic SOAP/WSDL interfaces autogenerated from Java or .NET code at the touch of a button, and the in potentia SOAP architecture that can be anything, even RESTful?

Consider the services popularly considered RESTful but seriously non-thesis-compliant. We currently term those things "HTTP+POX" in deference to common (kind of common) usage. Is that just another case of conflating architecture and technology? Is it worth coming up with another term? And given that there are multiple rival architectures for web services, do these not-quite-RESTful services consistently deviate from REST in the direction of some other architecture, or are they just freakish one-off mutants? Are they the missing links on a gradient between the rival architectures?

⁰ Said like in one of those political ads with lots of black-and-white closeups: "Senator Bedfellow doesn't want you to know about his millions in contributions from Big Web Services. When the FBI came knocking, he started shredding the XML documents. But he forgot to shred the envelopes. Senator Bedfellow: wrong on URIs. Wrong for the web."

I'd like to recommend open-uri for everything, including as a client for REST web services, but there are two big features it's missing. You can't use any HTTP method other than GET, and you can't send any body data (which makes sense if GET is the only supported method). Lately I've been building on my Cookbook work, coming closer and closer towards a general HTTP client library for Ruby, but really all I'm doing is reinventing open-uri with two extra features. It would be a lot better to just add those features to open-uri.

I submitted a patch for open-uri on this topic a while back but have yet to get any response, even from people saying it's stupid. I'm writing REST clients with open-uri under the assumption that that'll work by the time the book is published. For instance, here's some code from my Amazon S3 client:

  # Saves this bucket to S3.
  def save
    open(uri, :method => :put)
    return self
  end

  # Deletes this bucket and all its contents.
  def delete
    open(uri, :method => :delete)
  end

I guess you could argue that these features break the metaphor, that PUTting to a URI isn't really "opening" it, or that it's like opening a file for write access and you should have to write the body data inside a code block the way you do for a file.[0] But c'mon, let's have that argument. I don't want to have to keep telling everyone about fake reimplementations of open-uri just so they can write REST clients in Ruby. There's also precedent, like XMLHttpRequest.open in Javascript.

I'd also like to add caching support to open-uri, like the caching support in Joe Gregorio's httplib2 for Python. But I'll wait on that until the fate of this simpler patch is decided.

[0] I doubt this is really the intent of open-uri, since right now it doesn't support HEAD either. I think the intent of open-uri is to make it trivial to make an HTTP request from Ruby.

The first problem is to sort out the terminology. There are many ways of classifying services, and we're introducing two more in the book: "Resource-Oriented Architecture", which I've mentioned before, and "The programmable web", which is a way of saying "web services" without implying Big Web Services, or even saying the word "services". We got the latter term from the excellent programmableweb.com. (We also introduce "Big Web Services" itself, which is a way of saying "web services" and implying only the SOAP/WSDL/WS-* stack, but I think I'm going to use that term a lot more in these entries than in the book.)

The main work thing I did today was draw diagrams: a taxonomy and Venn diagram. Don't worry, they'll be redrawn by real graphic designers before they're printed in the book. One or both might make it in; the Venn diagram is simpler but the taxonomy has examples and chapter references.

Both diagrams show two decisions I've made for the book: first, we draw a distinction between Resource-Oriented and Service-Oriented architectures. Second, we define HTTP+POX as a hybrid of the two architectures. The former because it's not useful to pit a technology against an architecture (as in "SOAP vs. REST").

The latter because the web services described as HTTP+POX, the ones that are kind-of-but-not-really RESTful, tend to deviate from REST in the direction of RPC and the Service-Oriented Architecture (rather than having no architecture at all, which was what I expected). Examples include the Flickr API, which includes explicit method names ("flickr.photos.search"), and the del.icio.us API, which has special URIs that you GET to invoke particular methods ("/posts/delete").

Some interesting edge cases: almost all web applications can be considered HTTP+POX web services. They're usually not all that RESTful, but they take standard HTTP requests with non-fancy entity bodies, and serve a non-fancy XML dialect (HTML or XHTML). This is how libraries like WWW-Mechanize look at the web.

In fact, I suspect the combination of URI- and RPC-based architectures to form HTTP+POX comes from our experience writing apps for the human web. You can't do nontrivial REST in a web browser without Ajax. HTML forms only support GET and POST, they don't let you set request headers, and they have hard-coded destination URIs (this is why URI templates are going to be great).

I used to think that, while the lack of PUT and DELETE was lame, people who obsessed about those methods as a determiner of RESTfullness were just being pedantic. But the ROA/SOA classification shows why it's important: if you have to overload POST, you end up creating extra URIs which the client POSTs to in different circumstances. This puts you well down the road to the RPC model. Your URIs no longer point to resources; they point to procedures you can call by POSTing. It's natural to keep following that model when you go from writing a web app for browsers, to writing a web service for computer programs.

Another edge case is that a website full of static pages is technically a RESTful web service. It's composed of resources (files), each with one URI and one representation (the data in the file). Each URI correctly supports a subset (GET and HEAD) of the uniform interface. This is how libraries like Beautiful Soup look at the web.

If you look at the Venn diagram you see that the entire human-visible web (static websites and web apps) is part of the programmable web. This is quite intentional. People were programming to the web long before anyone started talking about "web services". It's dirty work because the human web wasn't designed with software clients in mind, but sometimes it's the only way to get the data.

There's one thing in the diagram I'm not clear on. It might be a minor point or it might be an important aspect of our definition of HTTP+POX. The question is whether there are web applications that are so RPC-oriented they cross the limits of HTTP+POX and move in on the architectural turf of SOAP/WSDL services. I drew the Venn diagram with web apps that were neither REST nor HTTP+POX, thinking of really monolothic web apps that only expose a single URI, and do everything by POSTing to that URI.

The "POX" in HTTP+POX basically means "no SOAP", but what specifically don't HTTP+POX people like about SOAP? Is it the complexity of the SOAP message itself? Plain old XML can get pretty complex too. Or is it the fact that the SOAP message contains information (like the method name) that they think should go in the URI?

There are also, as one emailer pointed out, problems with calling anything a "web service" if we're drawing distinctions between a "service-oriented architecture" and something else. I'm not sure what to do about that. I'm tossing around the idea of switching that label to "RPC-oriented architecture", but I don't want to introduce yet another term of art.

There are lots of books about Big Web Services: complex distributed-object systems that reproduce the mechanics of method calls over HTTP. The problem is 1) these systems are way too big for what they do, and 2) they're on the web but they aren't of the web. They don't use any of the web's features, or interact with anything else on the web. They just use HTTP as a transport protocol. They could just as easily run over TCP and get better performance.

These un-weblike systems were able to take the name "Web Services" away from the competition because the competition is... the web. Just writing programs that interact with the web. Sounds pretty sketchy! The web may be all right as a platform for serving movies, or selling books, or trading stocks, or democratizing publishing, or coordinating huge volunteer projects, or searching much of the information currently in existence, but there's no way it's up to the task of managing Accounts Payable! For that you need... Web Services!

As Big Web Services gathered steam, the pro-web forces rallied behind the banner of REST: a name for the design philosophy that made the human-visible web so successful. They preached simplicity, addressability, statelessness and the uniform interface of HTTP. They also got into lots of heated arguments about what REST really meant.

Some organizations created services that claimed to be RESTful, and others critiqued those services and said they weren't RESTful really, and is "RESTful" even a word? Meanwhile the world's programmers started finding options in their IDEs that generated code for Big Web Service clients and servers, so they wouldn't have to do so much programming.

I got the idea for a REST book when I started seriously trying to figure out what was and wasn't REST. I noticed that though certain best practices showed up repeatedly in REST folklore, they never really progressed beyond that point. I decided to write a book that would set down the folklore and hopefully create some new canon, some common ground. I got Sam Ruby to agree to do the hard parts. And when I started work on the book I discovered a basic rule of thumb, a framing device that really focuses on what's important about REST, and makes it easy to tell what's RESTful and what's not.

We call this framing device the Resource-Oriented Architecture (we're not the first, but the other uses are compatible with ours), and I'm going to be writing about it a lot more in NYCB. It's too good to keep hidden in a book along with a bunch of implementation details.

Some people might say, "Leonard, why are you guys even writing this book? There already are lots of books about the web, and REST is just the web, right? That's the whole point! We wrote those books back in the 90s."

To them I say: "Some people, think about Ajax for a bit. Ajax, as a technology, is only about two years old, but it's composed entirely of technologies that are much older, like Javascript and XMLHttpRequest and the browser DOM. So why are there all these books about Ajax? (Including Christian Gross's Ajax and REST Recipes, coming out later this month and seemingly the only other book to mention REST in the title.) Why does Ajax excite people when DHTML was a big yawn? Because Ajax is a way of framing these old technologies so that people can do awesome things.

"There's nothing on the market that explains what REST really means and how to make it work for you. Nothing about programming the web in a way that's more structured than screen-scraping without falling into the pit of Big Web Services. This is a way of framing preexisting technologies that lets people do awesome things."

Then I add, "Speaking of Ajax, did you know that an Ajax application is basically a REST web service client that runs in your web browser? That's why Christian Gross's book is called what it is, and that's just one of the amazing reality-twisting facts you'll learn from my new book, Jon." Because I'm talking to Jon Stewart! I'm on The Daily Show, plugging my new REST book! It's a dream come true! But wait--I'm not actually on The Daily Show! I'm just on a clip from The Daily Show that someone uploaded to YouTube! And now someone is sending a DELETE request to my URI! I can see the Authorization request header, its value is "DMCA Takedown Notice", and nooooooooooo

Then I wake up. Whew! It was just a regular dream, not a dream come true. I get up and go back to work on my REST book.

Ruby is awesome and the Ruby Cookbook was a lot of fun to write, but it wasn't my idea. It's part of a high-profile O'Reilly series. There was destined to be a Ruby Cookbook, written by somebody, from the moment Rails strapped an E motor to Ruby and hit the igniter. This book was my idea, and Sam and I are driving it. In the coming months I'll be writing a lot more on this topic. There's a lot of work we need to do in the real world, in addition to writing stuff down in a book.

PS: I say this on the homepage for the book, but in case you haven't clicked through and you're interested in REST, please email me or Sam. We'd love to have you help us with the folklore and best practices, and we need lots of reviewers.

Right now every REST web service is slightly different. You access them all with simple tools: your language's HTTP library, and probably its XML parser. Which is great because it keeps the services simple. What's not so great is that you have to learn each REST service separately.

What's more, it's useful to package and distribute your implementation of a particular service in a particular language. There's PyAmazon and Ruby/Amazon and a4j, PySearch and Yahoo-Ruby. That's always seemed wrong to me.

With a SOAP/WSDL service you just load up the WSDL file and call or generate the native language methods. Even though the underlying technology has more layers and is much more complex, when WSDL is one of those layers (and nothing goes wrong), everything just works. Sometimes there are language-specific wrappers on top of a SOAP/WSDL service (PyGoogle and Ruby/Google), but they don't do much: they make your code look nicer, or compensate for the shortcomings of the underlying API.

WADL is the best way I've found to get that ease of use for REST services, without creating documents that don't make sense to humans, or selling out the resource-oriented architecture of the web. WADL is very human-readable and it describes resources, dammit. I like it a lot and I think it'll make things easier for programmers if it gets widely adopted.