News You Can Bruise for 2007 June 4 (entry 0)

(12) Mon Jun 04 2007 19:25 It's Just A Hypermedia Format: There's a children's book about marijuana called "It's Just a Plant". The title always sounded disingenuous to me because if marijuana had no interesting features apart from plantness, nobody would write a book about it. Maybe the lesson of the book is along the lines of: "It's a very special plant, a plant that Mommy and Daddy smoke to get high, but it's not ontologically disjunct from other plants."

In that spirit, here is my defence of WADL, called "It's Just a Hypermedia Format". Sometimes people say they'd like to do strange things with WADL, things I don't approve of and things they wouldn't think of doing with HTML. But WADL is just a hypermedia format. I like WADL because it's a better hypermedia format than HTML or plain XML.

WADL is a way of describing the infinite variety of HTTP requests a client might make in the future. WADL also helps the client figure out which of those infinitely many requests they actually want to make next. One term for this is "service description". Another is "hypermedia". When I say "hypermedia" I just mean links and forms, like you have in HTML.

I'm going to talk about several common objections to WADL, and then put a new objection into circulation which I think is better. A lot of this has been said already, back and forth, but I wanted to gather a bunch of thoughts into one place. I'm taking most of these these from entries in the weblogs of Mark Baker, Aristotle Pagaltzis, Dare Obasanjo, John Heintz, and Patrick Meuller, but also from general murmuring I've heard on the web.

Disclaimer: despite the fact that I wrote a frickin' book about this stuff, I don't think of myself as a "REST expert" and I don't have anything in particular invested in WADL. I just like it as a hypermedia format.

WADL encourages/requires/represents the RPC style.

I'm putting this one first because I think something like this underlies many of the ideas that some standard part of HTTP is what you really want instead of WADL. We all take it for granted that you should rarely if ever need to expose an RPC-style interface through HTTP. It follows that if I fall into the RPC style, I'm probably not using HTTP correctly.

So, here's some WADL for your delectation, a partial description of a service that supports the Atom Publishing Protocol. It's from RWS but it's very similar to stuff in section A.2 of the WADL standard.

  <resource id="atom_collection" path="{feedID}">
    <resource href="#member" />
    <method href="#getCollection" />
    <method href="#postNewAtomMember" />
  </resource>
...
  <method name="POST" id="postNewAtomMember">
    <request>
      <representation href="#entry" />
    </request>
  </method>

You might look at that and think "They've taken a RESTful POST and renamed it 'postNewAtomMember'! Those RPC bastards!" Except "postNewAtomMember" isn't the name; it's the unique ID. The name is still "POST". An HTML form can have an ID, and so can a WADL method. The ID is optional, and its job is to distinguish this POST method from other POST methods so you can reference it from elsewhere. Same as with HTML forms. That "RPC-style" method (once you multiply out the reference to #entry) has the same meaning as this HTML form:

<form action="post" id="postNewAtomMember" 
      enctype="application/atom.entry+xml">
</form>

OK, I'm not naive. I know that lots of people are going to want to make tools that take that WADL file and generate something ugly like an AtomService class which defines a method postNewAtomMember. When they should be doing something non-ugly like defining a "resource" object or class which responds to post.

But the solution is to educate people about REST so they don't make bad tools (and bad services). There's nothing here that's not in HTML forms.

The uniform interface is good enough.

If the uniform interface was good enough, we'd have no hypermedia. The uniform interface restricts the methods you might apply to a resource. It doesn't tell you which methods are applicable to a specific resource, what the representation to a POST or PUT request should look like, which resource the client wants, or how to get from one resource to another. We use hypermedia for that.

There is an HTTP method called OPTIONS, which is supposed to advertise the capabilities of a resource. The first problem is that nobody uses OPTIONS. The second problem is that the only defined behavior for OPTIONS is to say which methods of the uniform interface are supported (in the Allow header). I'd like to try a system where sending OPTIONS to a URI gets you a hypermedia description of that resource, and see where that goes.

(The uniform interface plus) Media types are good enough.

They're usually not good enough to describe the representation for a POST or PUT request. Think about HTML forms. Specifying the enctype as application/form-encoded doesn't save you from having to define the form fields. The client needs some way of mapping its internal state onto the media format the server expects. This can be described in English text, or it can be described in hypermedia, like WADL.

If there's a custom media type for the specific XML vocabulary you're using, the media type might be good enough. The catch is that your XML vocabulary must also be usable as a hypermedia format. Atom is a good hypermedia format because it's got the atom:link element. atom:link has attributes like "rel" and "type" that let the server add semantic meaning to a link for a client to pick up on. But if you need to specify a hypermedia form (see below) instead of just a link, you'll need to interpolate some WADL or XHTML into the Atom file.

You can't get away from hypermedia. Neither the uniform interface nor media types help you describe the levers of state or connect resources to each other.

Links are good enough.

Links certainly are good, which is why as a hypermedia format WADL describes them. If the XML vocabulary you're serving isn't a very good hypermedia format, if your "links" are just URIs stuck somewhere in an XML document, a WADL addendum can explain what the URIs are pointing to and what they mean. WADL links have useful semantic attributes like rel.

But links are not good enough. They're not good enough for the WWW. Lots of RESTful, resource-oriented websites make their clients do things other than follow links. If links were good enough, Google would look like this:

Hypermedia consists of links and forms. XHTML is one way of expressing forms, and WADL is another. AFAIK they are the only two XML vocabularies that can describe hypermedia forms. [Update: I forgot XForms, OpenSearch, RDF Forms, and Poland.]

There are two kinds of forms: forms for manipulating application state (where the action is GET) and forms for manipulating resource state (where the action is POST, PUT, or DELETE).

An application form helps the client figure out the next URI it should GET. This is how a search engine form works. There are a set of data structures that need to be filled in ("q should be a string") and a set of rules for turning that data into a URI (http://www.google.com/search?q=jellyfish). WADL does the same thing, with some additional features like URI templates.

The client is not making assumptions about the URI space. It's calling in structured promises that the server made earlier, possibly in the response to the previous GET. These promises were represented in hypermedia, in WADL as they are in HTML. If HTML 5 was out and had URI Templates, the hypermedia could always be HTML, but right now there are cases where it has to be WADL.

A resource form helps the client figure out how to format the representation it sends along with POST or PUT to a particular resource. This includes the media type (eg. application/form-encoded) as well as a description of what the document should look like, possibly with reference to some XML schema. Lots of people (inc. me) hate XSD, but that's how you describe what an XML document should look like.

RFC2616 is enough.

It's not enough for the WWW. We also have a hypermedia standard: HTML. HTML is not just another media format like PNG. Media formats convey state. Hypermedia formats convey the levers of state. They show the client how to turn its internal desires ("I want the next item", "I want to spawn a subordinate resource with properties x,y,z") into a next request whose URI and representation the server will understand.

Okay, RFC2616+HTML is enough.

Maybe. I haven't heard anyone say this, but I've thought it. If you grant the HTML5 improvements and URI Templates, HTML is good enough for a large number of real-world services.

I know this because in chapters 5 and 6 of RWS I designed a fairly complicated web service that was completely described in machine-readable ways, and I didn't need a line of WADL to do it. Somebody else could write WADL for that service, just as someone else can write their own link-filled HTML guide to crummy.com, but I didn't need any WADL because I chose a different hypermedia format: HTML.

HTML is missing some of WADL's nice features, but if people start serving HTML representations instead of XML representations, then WADL might die out.

It's better to program directly to the HTTP client library.

A properly written (ie. not sneakily RPC) WADL library is an HTTP client library. My Ruby WADL library is pretty lame, but it's an HTTP library. The methods that trigger HTTP requests are get, post, put, and delete. It doesn't look like most other HTTP libraries, because its metaphors are resources and representations and links, where most existing libraries are based on the request/response cycle. But it's not hiding HTTP under something else, any more than Restlet's client library is.

WADL files are monolithic, like WSDL files.

I haven't heard anyone say this explicitly, but it's a reasonable complaint, and it's one of the things that worries me. That's why in RWS I proposed WADL "snippets" that describe a resource's behavior under the uniform interface. These snippets would be interspersed in an XML representation, the way links and forms are interspersed in an HTML representation.

Monolithic "site map" hypermedia files are a problem because they tend to be cached a long time and treated as unchanging. When hypermedia snippets are served along with representations, a client will hear about any changes in the hypermedia description of a resource. It will also hear about any changes in the relationships between one resource and others. A client can automatically adapt to some changes, assuming the semantic cues stay constant. But when the client does all its work off of an old WADL file which contains all hypermedia for the service, it won't know if the WADL file changes.

To sum up, I'd like to respond to something Aristotle wrote:

So this is how you describe a REST service: you explain the type of response body your server will send, where to find links in it, and what sort of semantics and verbs a particular link implies about the resource it points to.

This is exactly what WADL is for. You can describe a RESTful service in English text so that a human being can write a custom (and probably procedural) client. Or you can describe it in hypermedia so that a client library can present it in terms of the resources you were thinking about when you designed it. Right now your options for rich hypermedia are HTML and WADL.

Comments:

Posted by Dare Obasanjo at Mon Jun 04 2007 20:16

It seems you skimmed my blog post because it doesn't look like you addressed any of the issues I raised in it. As usual with any recent distributed programming technology, many people are approaching WADL from different perspectives. The SOAP/WS-* folks want another WSDL so they can keep using their codegen tools like JAXB and XSD.exe, you want a generic version of Atom's service documents.

The main problem I see is that WADL will encourage SOAP/WS-* to keep doing what they have without changing, the same way these code gen tools ended up making SOAP/WS-* development not much different from DCE/RPC or CORBA development which it intended to replace.

I don't want us to go 2 steps forward only to the take 3 steps back.

Posted by Leonard at Mon Jun 04 2007 21:34

I didn't address any of the issues you raised in your blog post because I don't feel competent to address them. I know nothing about the WS-* world. What I saw was people with knowledge of both worlds (like Mark) addressing WADL as though it were intrinsically what the WS-* folks want it to be. That's what I'm trying to address. I put you in the list because I thought your post said interesting things about XSD.

I'm trying to prevent the backwards steps by positioning WADL as a way to get the *kind* of thing WS-* people want (tools that work on a higher abstraction level than HTTP request/response) without reintroducing the status quo (in WADL the higher-level abstraction is the resource, not the procedure call). Does this make sense?

Posted by Mark Baker at Mon Jun 04 2007 22:04

Good post. It still seems a lot of wishful thinking though, just as I used to wish that people wouldn't use SOAP for RPC.

Some data points; there are other forms language, such as XForms, OpenSearch, and my RDF Forms.

Also, I have complained about WADL not being embeddable. See;

http://lists.w3.org/Archives/Public/public-web-http-desc/2006Sep/0014.html

Mark.

Posted by Leonard at Mon Jun 04 2007 22:24

Good catch on XForms etc. I'd forgotten about those in my fervor.

Most of my writing on WADL is marketing, and I always prefer optimistic marketing. The same forces that might ruin WADL might also ruin good service design for another tech generation--and are capable of ruining good service design even if there were no WADL. The only solution I see is education, and getting good tools out in front of the bad ones.

Posted by John Heintz at Tue Jun 05 2007 13:16

Most of the points you make are good as far as they go, but the biggest issue I have with WADL is that it's used to generate code: stubs and skeletons.

As soon as I want to change my service I've invalidated hard-coded clients out there.

If I understand Dare's message about XSD, it's this: XML Schema is a bad language for extensible, interoperable, evolvable systems because it make to many confining choices (especially when codegen is used, but also I think just at the XML level.)

My message about WADL is that it locks a web service into a rigid URI structure and workflow. Only some of the URI's should be stable, most should be fully server generated without any constraint.

If you intend WADL to be used more dynamically, then it's a different thing than I think it is. Codegen for the web is bad though.

Posted by John Heintz at Tue Jun 05 2007 13:33

Just wanted to re-phrase the beginning of my comment. (I realize it doesn't sound even nice, darn written communication...)

"I agree with your points individually, but I take issue with how WADL is used ..." is closer to what I actually wanted to say.

Posted by Leonard at Tue Jun 05 2007 14:03

John, you've brought up a huge topic that I haven't talked about much because I'm still devoting heavy thought to it. I think we have not yet seen well-connected RESTful services of great complexity, and I think our intuitions about such services are likely to be inaccurate. Especially when we're talking about features, like generated interface code, which come from the world of web-services-that-don't-look-like-the-web.

One of the big REST benefits is that it's possible to program flexibility into clients. But if I can write flexible code, I should be able to write a program to generate flexible code. Code generated from WSDL is always brittle because WSDL isn't a hypermedia format. Code generated from WADL doesn't have to be.

If people are using WADL as a drop-in replacement for WSDL (I haven't been paying attention, and would appreciate representative links) then they're repeating the mistakes of the past and their generated code will still be brittle. But when we start making complicated interlocking webs of resources, we're going to need something like WADL--even if it's never used for code generation. The only good enough alternative is XHTML5 + URI Templates, and that's barely good enough.

Posted by John Heintz at Tue Jun 05 2007 14:52

Leonard, we aren't saying very different things.

Regarding flexibility: A client and server both need certain degrees of flexibility, and certain constrained rules of behavior to agree on. Something must be constrainable enough for a machine client to proceed towards some goal.

Deciding what should be unconstrained and adaptive vs agreed and constrained is the hard part of course.

HTML Forms don't provide enough information, and I don't yet know all the details on XHTML5+URI Templates. XForms and RDF Forms likely provide enough to code smart and adaptive clients against.

What degrees of flexibility / constrainable entities exist on the Web:
* media type
* URI

The media type has more details that fit into flexibility:
* hypermedia is navigable with a flexible client
* HTML is sloppy and loose (and extensible)
* XSD schemas XML is not generally extensible

The URI has more details as well:
* Cool URI's don't change
* URI's should be opaque (the server has flexibility)

With this breakdown, I view WADL as (by default) using a rigid media schema (XSD) to fully constrain the server URIs. Or, removing all the flexibility/extensibility from the system.

I don't know how to build a flexible, adaptive client with that breakdown of WADL. I'm open to ideas of course, and perhaps my declarative signposts idea is just bundling a WADL fragment into HTML Forms at runtime.

Posted by Dan Diephouse at Tue Jun 05 2007 17:20

John writes: "With this breakdown, I view WADL as (by default) using a rigid media schema (XSD) to fully constrain the server URIs. Or, removing all the flexibility/extensibility from the system."

Practically speaking though:
1. An application needs an entry point into the system (i.e. a URI). This needs to be communicated somehow (verbal, email, WADL, HTML link ,etc)
2. An application does not always want to browse to the correct resource. I don't want to browse a million customers resources, I want to navigate right to a specific one. People will have to hard code this logic

Unless I'm missing something, given these constraints, this leads us directly toward defining the structure of the URI so consumers can construct them. Its only a matter if we're going to let consumers reverse engineer them - which is probably error prone and liable to cause problems in the future - or if we're going to communicate them somehow (documentation, WADL, etc)

If Cool URIs don't change then whats the problem in constraining the URIs anyway? ;-) It seems that that goal is a bit opposed to opaque URIs...

---
Re: XML schemas - I call bullshit on everyone who is saying not to use XML schema (or schema languages in general) because they can be abused (and can be written so they're non extensible). The web can be abused as well, but that doesn't mean that we shouldn't use it.

Posted by John Heintz at Tue Jun 05 2007 18:32

Dan,

The entry point and the "correct resource" should both have cool, stable URIs.

That doesn't mean the processing and search and navigation URIs should be fixed as well.

The entry to my service should expose a search capability. This should be identified _not_ by hardcoding the URI, but with some self-descriptive data format (like OpenSearch).

Opaque URIs are important because to loosely couple the client and server (client follows hyperlinks), and self-descriptive media types allow highly cohesive processing.

Regarding XSD. I'm not saying don't use them because they can be abused (and I don't think others are either). I think XSD works from a fundamental assumption of "lock everything down", which is exactly the wrong way to deal with open formats on the Web. Couple that with codegen and it's no wonder a google search on "xsd versioning" has no answers.

Posted by Leonard at Tue Jun 05 2007 19:19

At last I've arrived; the URI construction debate is on my weblog. FWIW, I think the best way to construct a URI is to fill out a hypermedia form. In chapter 8 of RWS there's some stuff about the value of links and forms vs. client-side URI construction.

On the other stuff, I think I'm now approaching the level Dare was talking about. The next step is probably to see if there's a better way to describe in hypermedia what an XML PUT/POST representation should look like. I do not relish this task and will probably put it off.

I notice that Atom's RelaxNG schema explicitly defines support for extensibility.

Posted by Torbjörn Gannholm at Thu Jun 07 2007 16:00

I have not yet formed an opinion of whether WADL is needed or not, but applaud it as a stroke of genius compared to WSDL.

If WADL should be combined with http://tobe.homelinux.net/xis instead of XMLSchemas I think we may be on to something.