<M <Y

Beautiful Soup 4.3.2, and all previous versions: Through long practice I'm able to write decent code while I'm sick, but I should not try to release code while I'm sick. While putting up the release of Beautiful Soup 4.3.2, I accidentally deleted the entire beautifulsoup4 project on PyPI and had to recreate it manually. I've given PyPI all the crummy.com tarball URLs for releases going back to 4.0.1, and I've installed each one via pip to verify that it works, so if your build process depends on installing a specific version of Beautiful Soup 4 via PyPI, it should still work. And indeed, random versions of BS4 have been downloaded about 200 times since I switched over. I'm sorry about this screwup. Let me know if there are any remaining problems.

4.3.2 itself is a pretty minor bugfix release. Still left unfixed is a bug I can't reproduce because the federal government is shut down. When you file a bug that happens with a specific web page, please provide the HTML of the web page, not the URL.

September Film Roundup: I missed a whole lot of museum movies in September because I was out of town for two weekends. And yet I still managed to see nine movies, plus wrap up a TV show, and write a huge blog post about it. Wonders, or at least me writing about them, will never cease.

What's up for October? More Howard Hawks, it looks like. See ya then.

[Comments] (1) RESTful Web Services now CC-licensed: Hey, folks, I got some pretty exciting news. Now that RESTful Web APIs has come out, there's really no reason to buy 2007's RESTful Web Services. So Sam Ruby and I and O'Reilly have gotten together and started giving the old book away. You can get a PDF from the RESTful Web APIs website or from my now-ancient RESTful Web Services site. The license is BY-NC-ND.

If you've bought RESTful Web APIs (and if you haven't, you should), you may have noticed that we promise that this will happen in a footnote of the Introduction. It took a while to get the contract amended, but now it's all complete.

Here's a direct link to the PDF in case you just want to grab the book instead of hear me talk about it.

Obviously I think the new book is a lot better than the old book, but the old book is still very good. The source code is long obsolete (this is why RWA contains no source code, only messages sent over the wire), but the sections on HTTP still hold up really well. A lot of RWS Chapter 8 went into RWA Chapter 11. With a few edits and additions, RWS Appendix B and C became RWA Appendix A and B. Those are the only bits of RWS that I reused in RWA.

From my vantage point here in 2013, my main critique of RWS is that it makes HTTP do too much of the work. It focuses heavily on designing the server-side behavior of resources under a subset of the HTTP protocol. I say "a subset" because RWS rules out overloaded POST ahead of time. You don't know what an overloaded POST request does. It's a cop-out. You're sweeping something under the rug. It's better to turn that mystery operation into a standalone resource, because at least you know what a resource does: it responds to HTTP requests.

In retrospect, RWS is that way because in 2007 hypermedia data formats were highly undeveloped whereas HTTP was a very mature technology. Nowadays it doesn't matter so much whether an HTTP request uses POST or PUT, so long as a) the state transition is described with a link relation or other hypermedia cue, and b) the protocol semantics of the HTTP request are consistent with the application semantics of the state transition. That's why RWA focuses on breaking down a problem into a state diagram rather than a set of static resources.

So, RWS is very much a 2007 book, but that's the meanest thing I can say about it. A lot of it is still useful, it's historically interesting, and I'm glad to give it away. I'd also like to give my thanks once again to Sam Ruby and O'Reilly, for their work on RWS.

API Design is Stuck in 2008: I've got a guest post up at ProgrammableWeb with the provocative title of "API Design is Stuck in 2008". Often an author can blame their editor for that kind of title, but no, that's my title. The good news is that over the past few years we have developed the tire chains necessary to get ourselves unstuck.

I don't think there's anything in the article you won't find in the RESTful Web APIs introduction and my discussion of my RESTFest talk, but I wanted to let you know about it and provide a forum on NYCB for asking me questions/taking issue with my assertions.

"Constellation Games" reading: Anne Johnson and I are doing a comedy SF reading on Wednesday at the Enigma Bookstore, a new genre bookstore in Astoria. It starts at 7 PM. The details, as you might expect, are on a Facebook page. Hope to see you there!

[Comments] (1) Reading After-Action Report: In preparation for my reading at Enigma Bookstore I asked people on Twitter which bit of Constellation Games I should read. I decided to read Tetsuo's review of Pôneis Brilhantes 5 from Chapter 18, both by popular Twitter demand and because Sumana had reported success reading that bit to people.

I practiced reading the review and also practiced another scene: Ariel's first conversation with Smoke from Chapter 2. No one suggested that scene, but it's one of the last scenes I wrote, so I personally haven't read it a million times and gotten tired of it. I abandoned this idea after a test reading because it's really hard to do a dramatic reading of a chat log, especially when most of the characters have insanely long names. So, Pôneis Brilhantes it was.

However, shortly before the reading I learned that Anne and I were each going to be reading two excerpts! Uh-oh. On the spur of the moment I chose to read a scene I had never practiced and that only one person (Adam) had suggested: the scene from Chapter 11 where Ariel meets Tetsuo and Ashley and they go visit the moon.

That scene has three good points: a) it introduces Tetsuo, increasing the chance that the Pôneis Brilhantes scene would land; b) it's full of the most gratuitous nerd wish-fulfillment I could write; c) it ends strongly with the call from Ariel's mother, which unlike a chat log is very easy to read because it's a Bob Newhart routine where you only hear one side of the phone call.

This was a really good idea. People loved the moon scene, even though my unpracticed reading stumbled and ran too quick. But when I read the Pôneis Brilhantes scene, it wasn't such a great hit! The room wasn't really with me. That's the scene I had practiced, and I think it's the funniest, most incisive thing in the whole book. Not a big hit! I think if I'd only read that scene I wouldn't have sold many books that night.

So, thank goodness for the moon scene, is all I can say. But what was going on? How had I misjudged my audience so badly? Sumana said she'd read Pôneis Brilhantes and gotten big laughs.

I think you have to be a very specific kind of computer geek to find Tetsuo's Pôneis Brilhantes review funny as a review of a video game, rather than as an expression of the personality you've just spent seven chapters with. That's the kind of geek that Sumana and I habitually hang out with, but it's not representative of the SF-reading population as a whole. I think that computer-geek population hosts a lot of the readers who wish that the second half of Constellation Games was more like the first half. Whereas someone who really digs the moon scene is more likely to stay with me the whole book.

I guess you could say the moon scene is just more commercial. And I guess I subconsciously knew this, because my current project gets more of its humor from the plot-driven character interaction found in the moon scene, and less from high concept Pôneis Brilhantes-style set pieces.

What's New in RESTful Web APIs?: I was asked on Twitter what changed between 2007's RESTful Web Services and 2013's RESTful Web APIs. I've covered this in a couple old blog posts but here's my definitive explanation.

First, let me make it super clear that there is no longer any need to buy Services. It's out of date and you can legitimately get it for free on the Internet. O'Reilly is taking Services out of print, but there's going to be a transition period in which copies of the old book sit beside copies of the new book in Barnes & Noble. Don't buy the old one. The bookstore will eventually send it back and it'll get deducted from my royalties. If you do buy Services by accident, return it.

If you're not specifically interested in the difference between the old book and the new one, I'd recommend looking at RESTful Web APIs's chapter-by-chapter description to see if RESTful Web APIs is a book you want. As to the differences, though, in my mind there are three big ones:

  1. The old book never explicitly tackles the issue of designing hypermedia documents that are also valid JSON. That's because JSON didn't become the dominant API document format until after the book was published. If you don't know that's going to happen, JSON looks pretty pathetic. It has no hypermedia capabilities! And yet, here we are.

    In my opinion, a book that doesn't tackle this issue is propping up the broken status quo. RESTful Web APIs starts hammering this issue in Chapter 2 and doesn't let up.

  2. There are a ton of new technologies designed to get us out of the JSON trap (Collection+JSON, Siren, HAL, JSON-LD, etc.) but the old book doesn't cover those technologies, because they were invented after the book was published. RESTful Web APIs covers them.
  3. New ideas in development will, I hope, keep moving the field forward even after we all get on board with hypermedia. I'm talking about profiles. Or some other idea similar to profiles, whatever. These ideas are pretty cutting edge today, and they were almost inconceivable back in 2007. RESTful Web APIs covers them as best it can.

Now, for details. Services was heavily focused on the HTTP notion of a "resource." Despite the copious client-side code, this put the focus clearly on the server side, where the resource implementations live. RESTful Web APIs focuses on representations—on the documents sent back and forth between client and server, which is where REST lives.

The introductory story from the old book is still present. Web APIs work on the same principles as the Web, here's how HTTP works, here's what the Fielding constraints do, and so on. But it's been rewritten to always focus on the interaction, on the client and server manipulating each others' state by sending representations back and forth. By the time we get to Chapter 4 there's also a pervasive focus on hypermedia, which is the best way to for the server to tell the client which HTTP requests it can make next.

This up-front focus on hypermedia forces us to deal with hypermedia-in-JSON (#1), using the tools developed since 2007 (#2). The main new concept in play is the "collection pattern". This is the CRUD-like design pioneered by the Atom Publishing Protocol, in which certain resources are "items" that respond to GET/PUT/DELETE, and other resources are "collections" which contain items and respond to POST-to-append.

We covered AtomPub in Services, but over the past six years it has become a design pattern, reinvented (I think "copied" is too strong a word) thousands of times.

RESTful Web APIs focused heavily on the collection pattern, without ever naming it as a pattern. I'm not dissing this pattern; it's very useful. I'd estimate about eighty percent of "REST" APIs can be subsumed into the collection pattern. But REST is bigger than the collection pattern. By naming and defining the collection pattern, we gain the ability to look at what lies beyond.

Attempts to encapsulate the collection pattern include two new JSON-based media types: Collection+JSON and OData. The collection pattern also shows up, more subtly, in the Siren and Hydra formats. Which brings me to the second major change.

In 2007, there were two big hypermedia formats: Atom and HTML. Now there are a ton of hypermedia formats! This is great, but it's also confusing. In "The Hypermedia Zoo", Chapter 10 of RESTful Web APIs, we give an overview of about two dozen hypermedia formats. The ones we seriously recommend for general use (HAL, Siren, HTML, JSON-LD, etc.) are covered in more detail elsewhere in the book. The quirkier, more specialized media types just get an exhibit in the zoo.

Now for the third new thing, profiles. If you go through the RESTful Web APIs narrative from Chapter 1 to Chapter 7, you'll see that we introduce a problem we're not able to solve. Hypermedia is great at solving the following problem:

How is an API client supposed to understand what HTTP requests it might want to make next?

But there's a superficially similar problem that hypermedia can't solve:

How is an API client supposed to understand what will happen in real-world terms if it makes a certain HTTP request?

How do you explain the real-world semantics of an HTTP state transition? Before chapter 8, the two solutions are to do it ahead of time in one-off human-readable documentation; or to define a domain-specific media type, a la Maze+XML. Both of these approaches have big problems. Chapter 8 introduces profiles, which lets you get some of the benefits of a new media type without doing unnecessary work.

Maybe profiles will turn out not to be the right answer, but we gotta solve this problem somehow, and the old book is not equipped to even formulate the problem.

There are also a few additions to the book I consider minor. There's a whole chapter in RESTful Web APIs on Semantic Web/Linked Data stuff; in Services there was nothing but a cursory discussion of RDF/XML as a representation format. There's a chapter in RESTful Web APIs about CoAP, which didn't exist in 2007. These are good chapters that took me a long time to write, but I don't think it's worth buying the book if you only want to read the chapter on CoAP. (Or maybe it is! There's not a lot of competition right now.)

So, what hasn't changed? HTTP hasn't changed all that much. RESTful Web APIs's information about HTTP has been brought up to date but not changed significantly. So if you were using Services solely as an API-flavored HTTP reference, you don't need the new book. You can just read up on the protocol-level additions to HTTP since 2007, like the Link header and standardized patch formats for PATCH.

Hopefully this helps! RESTful Web APIs has a lot of distinguished competition that the old book didn't have, but its competition is newer books like Designing Hypermedia APIs and REST in Practice. If you compare APIs to Services I think it's no contest.

Col. Bert Stephens: Recently Rob Dubbin made a ridiculous right-wing parody bot named Ed Taters. I thought this was funny because Rob already has a ridiculous right-wing parody bot: he's a writer for The Colbert Report. But I didn't think much about it until Rob gave Ed Taters the ability to spew nonsense at anyone who started an argument with him on Twitter.

That's when I had the idea of using Rob's own words against him! So I created my own bot, Col. Bert Stephens, who takes his vocabulary from the "memorable moments" section of a Colbert Report fan site. (Thanks to DB Ferguson for hosting the site, and to those who typed up the "memorable moments".) Col. Bert Stevens argues with Ed Taters, he argues with Ed and then reconciles, he argues with you (if you follow him and start an argument), and he occasionally says Tetsuo-like profundities all on his own.

To avoid infinite loops I've made Bert a little more discerning than Ed. He'll only respond to your messages 4/5 of the time. I'm not super happy about this solution but I think it's the safe way to go for now. Update: Hell with it. Bert will always respond to anyone except Ed. If you write a bot to argue with him, avoiding infinite loops is your responsibility.


Unless otherwise noted, all content licensed by Leonard Richardson
under a Creative Commons License.