| jakey b
| rachel 2
Crummy: The Site
We fight 'em until we can't.
Tue May 14 2013 10:04 Beautiful Soup 4.2.0:
My work on RESTful Web APIs is pretty much done, so I went through the Beautiful Soup bug tracker and fixed everything I could. The result is a new, stoner-iffic release of Beautiful Soup.
Here are the release notes. The main new features are a much more capable CSS selector engine, and a diagnostics module that should help with tech support.
Wed May 01 2013 14:55 Story Bundle:
Constellation Games is featured in the current video game-themed StoryBundle. It's a pay-what-you-want, like the Humble Indie Bundle. This means that if you're the ultimate cheapskate, you can get my book and six others for the Steam-sale-level price of three bucks. Pay ten bucks, and you also get three bonus books,
including Jordan Mechner's "The Making of Prince of Persia and a Ralph Baer memoir which--just guessing here--is probably enjoyably cranky.
And for people who discover Constellation Games based on this bundle, this is my occasional notification that there are tons of free extras: four bonus stories, in-character Twitter feeds, and an episode guide with commentary.
Side note: the bundle was assembled by Simon Carless, who is the reason I wrote Constellation Games in the first place.
(1) Wed May 01 2013 10:59 April Film Roundup:
Another month, another few movies. RESTful Web APIs is almost done, but not quite, so once again there's not a whole lot here. The theme of this month is "really loving a movie, seeing a different movie on that basis, and being very disappointed."
- The Face You Deserve (2004): This movie caused a rare Siskel and Ebert-type rift between me and Sumana. I thought the first part of the movie was boring, but that it picked up once it turned into a surreal fairy tale. Sumana thought the first part showed promise and hated the Michel Gondry-esque manchildren in the fairy tale part. I don't recommend this movie, either, but I was engaged for the fairy tale.
- The General (1926): This is one of the best movies I've ever seen. It's so well put together. The movie is basically two chase scenes, and each chase scene is made entirely of inventive Jackie Chan-style action gags. People in the theater were cheering, which I've never experienced before, and laughing to an extent not heard since The Whole Town's Talking. The General has all the good things Chaplin put into his films, but none of the treacly sentimentality. The one bit of sentimentality is deflated by its co-occurance with the one bit of corny dated-looking special effects.
No surprise, then, that this was Keaton's Ishtar: a way-too-expensive flop that cost him his creative control. You can Watch The General on the Internet Archive, but as always with silent film the problem is finding an appropriate soundtrack. We heard an amazing live soundtrack performed by Viola Dana, and they have a CD available, but the CD only has "selections". So maybe try Stravinsky's "Chamber Works", as suggested by a comment on this page? I bet some peppy chiptunes would also work.
- Die Hard (1988): Not one of the best movies I've ever seen. It deserves a lot of credit as the pinnacle of the 80s action movie, but at this point I've seen some action movies from the 70s, and it feels like movie execs saw Die Hard and said "well, we found it!" and the genre never advanced again. Not really Die Hard's fault, but it's hard not to be bitter.
For improvisational comedy-violence, The General is better. Not just my idiosyncratic opinion! The General's IMDB rating is 8.4, versus Die Hard's 8.3, and at the high end of the distribution, 0.1 IMDB star is worth a lot.
- Steamboat Bill, Jr. (1928): A big disappointment after The General. Buster Keaton has lost his creative control, and it shows. The film lacks a through-line (unlike The General, which was literally on rails), and promises a "snobs vs. slobs" rivalry that never gets going. Partially redeemed by great stunts. This, too, can be seen on the Internet Archive.
- Tai Chi Zero (2012): Zany anti-colonialist steampunk kung fu movie that annotates events with video game-style infographics and otherwise breaks the fourth wall all the time. It's kind of China's Scott Pilgrim Versus the World, and judging from online reviews it's just as divisive. We liked it a lot. (I also liked Scott Pilgrim.) It's got big problems, notably the acting, which is very stiff. But most of the actors were chosen for their martial arts ability, and martial arts are happening about seventy percent of the time. The only time the fourth-wall-breaking got out of control was a scene at the beginning of the third act, which blends together "we're planning this heist" shots and hypothetical "this is what it will look like when you carry out the heist" shots, and then starts mixing in "this is how the heist actually went down" shots! It took about thirty seconds before I realized that I was now watching the actual heist.
Sometimes the problems made me enjoy the movie more! The awkward English scenes gave me an experience similar to what I imagine a Mandarin speaker feels watching a Mandarin scene in an American movie. There's a steampunk tank with an English instruction manual, which was supposedly written by Brits but which reads exactly like the instruction manuals that come with Chinese-manufactured kitchen appliances. I thought the villain was a more complex character than he actually was, because I assumed that if everyone derides a character as a wimp, that makes him the underdog and you're supposed to have some sympathy for him. But no, apparently not in this movie.
Oh, and you know how they say "there ought to be a law?" Well, the special effects supervisor for Tai Chi Zero is credited as "A Law". So now there is A Law!
But I gotta tell you that this is not a standalone movie. It could have been, but about three minutes from the end it turns into The Good, the Bad, and the Ugly, introduces a whole bunch of new characters, sets up a sequel and leaves you hanging.
- Tai Chi Hero (2012): Fortunately, the sequel is playing in Times Square right now, so we went and saw it the next day. Aaaand... we were very disappointed. If you liked all the steampunk and fourth-wall-breaking from the first movie, then too bad, because there's no steampunk until the second act and no fourth-wall-breaking until the third. (The steampunk, when it finally happens, is still great.) On the other hand, if you hated all that nerd shit from the first movie, you'll love the by-the-numbers soap opera they replaced it with.
On top of everything else, the title of this movie retroactively makes the title of the previous movie dumb. I can't believe they got Peter Stormare to... wait, he was in Armageddon, never mind.
Sat Apr 27 2013 17:08 Board Game Dadaist Improvements:
I've finally relented to Adam's demands and made some improvements to the Board Game Dadaist RSS feed. He broke his kneecap recently and I figured this would be a good way to cheer him up. Every game that shows up in the feed now has a permalink (here's "Plue"), and that page has a very basic link for posting your find to Twitter.
Thu Apr 18 2013 13:05 In Search of the Beautiful Soup Double-Dippers:
Recently I noticed that certain IPs were using distribute or setuptools to download the Beautiful Soup tarball multiple times in a row. For one thing, I'm not sure why distribute and setuptools are downloading Beautiful Soup from crummy.com instead of using PyPI, especially since PyPI registers almost 150k downloads of the latest BS4--why are some people using PyPI and not others?
If anyone knows how to convince everyone to use PyPI, I'd appreciate the knowledge. But it's not a big deal right now, and it gives me some visibility into how people are using Beautiful Soup. Visibility which I will share with you.
Yesterday, the 17th, the Beautiful Soup 4.1.3 tarball was downloaded 2223 times. It is by far the most popular thing on crummy.com. The second most popular thing is the Beautiful Soup 3.2.1 tarball, which was downloaded 381 times. The vast majority of the downloads were from installation scripts: distribute or setuptools.
1516 distinct IP addresses were responsible for the 2223 downloads of 4.1.3. I wrote a script to find out how many IP addresses downloaded Beautiful Soup more than once. The results:
|Downloads from a single IP
||Number of times this happened
Naturally my attention was drawn to the outliers at the top of the table. I investigated them individually.
The IP address responsible for 55 downloads is a software company of the sort that might be deploying to a bunch of computers behind a proxy. The 35 is an individual on a cable modem who, judging from their other traces on the Internet, is deploying to a bunch of computers using Puppet. The 15, the 13, and the 11 are all from Travis CI, a continuous integration service.
One of the two 5s was an Amazon EC2 instance. Five of the twelve 4s were Amazon EC2 instances. Thirty-seven of the forty-three 3s were Amazon EC2 instances. And 395 of the 453 double-dippers were Amazon EC2 instances. Something's clearly going on with EC2. (There was also one download from within Amazon corporate, among other BigCo downloaders.)
I hypothesized that the overall majority of duplicate requests are from Amazon EC2 instances being wiped and redeployed. To test this hypothesis I went through all the double-dippers and calculated the time between the first request and the second. My results are in this scatter plot. Each point on the plot represents an IP address that downloaded Beautiful Soup twice yesterday.
For EC2 instances, the median time between requests is 11 hours and 45 minutes. So EC2 instances are being automatically redeployed twice a day. For non-EC2 instances, the median time between requests is 51 minutes, and the modal time is about zero. Those people set up a dev environment, discover that something doesn't work, and try it again from scratch.
(2) Mon Apr 01 2013 09:25 March Film Roundup:
Okay, look. I don't see movies just for their entertainment value. I dig film as an art form. But my permit to dig is premised on an amateur understanding of film as a narrative art form. If you want to present an endless stream of disconnected images, let's do an installation piece, because I want to decide for myself when I've had enough. I'm not going to be your captive for fifty minutes. (I'm looking at you, Andy Warhol.) And all that aside, I'm not gonna see a movie called Trash Humpers (2009), when the nicest thing the folks doing the screening can say is that it "rewards the open-minded viewer with moments of astonishing and unexpected poignancy."
Which is to say that I skipped most of the museum's highly avant-garde March offerings. I also got this book I have to work on. So not many movies in this roundup. Let's-a go:
- Fallen Angel (1945): Decent noir with a fake mystery and an interesting twist at the end (in terms of which characters got what they wanted and how, not in terms of plot). John Carradine appeared as a classic comedic noir conman, but he had to appear in twenty other movies, so he left after the first reel, much to Fallen Angel's detriment.
a noir film is my popcorn movie. I'll go see any number of them but I'm not expecting great things from them. PS: there is no popcorn allowed in the museum theater.
- Horse Feathers (1932): Still really funny, but this is the first viewing where I noticed that the Marx Brothers' general disrespect for society encompasses a lot of misogny. It's not just Harpo chasing the choir girls. In fact, Groucho's the worst. It doesn't help that there's no Margaret Dumont here to take up the flag of society and fight back. But Chico filling bottles in the speakeasy will never get old.
- Ikiru (1952): Watched on Hulu during the free Kurosawa weekend. Highly recommended. A little heavy-handed at the beginning, but it really started paying off when the main character died. (Not a spoiler.) At that point I saw a masterful display of one of the most difficult and most important things that fiction can bring to our attention: the mechanics by which we all construct narratives for our lives in which we're the good guy making good things happen.
Bonus: everyone referred to Takashi Shimura's character as "Kacho", deepening my belief that Game Center CX is a workplace satire in the vein of Ikiru.
- Wreck-it Ralph (2012): I'm not sure who gave Disney the idea that it's okay to use other people's intellectual property in their movies, but it gives good results. Wreck-it Ralph is a by-the-numbers Disney narrative, but the fact that it's a movie about arcade games and their by-the-numbers narratives leaves quite a bit of room for subversion and criticism, in service of the larger goal of feel-good entertainment. As with Who Framed Roger Rabbit?, I thought the original characters were a lot more interesting than the famous cameos, to the extent that I continually wished the famous cameos would butt out and let the original characters get on with it.
(The worst cameo was Sonic the Hedgehog's infodumpy PSA near the beginning. Awful! But! What if it was a sly reference to those dumb PSAs at the end of the old Sonic cartoons? Does an obscure reference deserve respect even when deployed as a cheesy infodump? OH THE DILEMMA)
The museum showed this in 3D, and I was apprehensive about the extra D. I can report that it neither caused me headaches nor made me want to see all movies in 3D from this point on.
- 2 or 3 Things I Know About Her (1967): The French New Wave eludes me again. There's one great scene in this, in which the conspiratorially-whispering narrator (Godard himself) deconstructs the subject-object distinction to the extent that he loses the ability to make directorial decisions, and lets the camera linger on some trees for a while. There's a few other good bits, and lots of Ballardian imagery. Makes me want to watch Alphaville even more. But... eh. Eh, I say!
Tue Mar 26 2013 08:00:
From an interview with Ken Liu, recent Hugo/Nebula/WFA winner:
I went to law school, started a new job, and kind of gave up on writing for a while due to a supreme act of stupidity. I wrote this one story that I really loved, but no one would buy it. Instead of writing more stories and subbing them, as those wiser than I was would have told me, I obsessively revised it and sent it back out, over and over, until I eventually gave up, concluding that I was never going to be published again.
And then, in 2009, Sumana Harihareswara and Leonard Richardson bought that story, "Single-Bit Error," for their anthology, Thoughtcrime Experiments. The premise of the anthology was, in the editors' words, "to find mind-breakingly good science fiction/fantasy stories that other editors had rejected, and release them into the commons for readers to enjoy."
I can't tell you how much that sale meant to me. The fact that someone liked that story after years of rejections made me realize that I just had to find the one editor, the one reader who got my story, and it was enough. Instead of trying to divine what some mythical ur-editor or "the market" wanted, I felt free, after that experience, to just try to tell stories that I wanted to see told and not worry so much about selling or not selling. I got back into writing—and amazingly, my stories began to sell.
Case closed, I'd say.
(1) Mon Mar 04 2013 14:38 February Film Roundup:
The second in the 2013 series, as promised. Note: I draw no distinction between information about a movie that's a "spoiler" and information that's not.
- Roots (1977): I'd never seen Roots. We watched it so we could reasonably go to a reunion interview at the museum featuring Levar Burton, Louis Gossett Jr., Leslie Uggams, and Ben Vereen. Here's Sumana's take. Non-surprise: Roots is really good! I have only one complaint: it was disconcerting to see Levar Burton abruptly replaced by John Amos in a "nine years later" flash-forward, since we now know that age cannot change the essential Levar Burton-ness of Levar Burton.
My other complaint was going to be that some (not all) of the villains were one-dimensional villains defined entirely by their hatred for the main characters. I'm actually okay with this in certain kinds of pieces, and Roots definitely qualifies, so my criticism was going to be somewhat muted. But then something magical happened that cancelled my criticism altogether.
You see, in a decision that probably made sense to whichever ABC exec was trying to backpedal from having greenlit Roots in the first place, it was decided to cast "television's most likeable white actors" in the roles of the villains, to sort of tone it down a little. At least that's what it said in the little mini-documentary on Roots that they showed before the reunion interview. If an actor's natural "likeability" tones down the evil of their character, that means they're not a very good actor, so it's a good thing that the whole "likeability" thing was a bust.
But then! The most despicable villain of the entire show, the final boss of Roots, the very face of evil, was played by genial, honey-voiced Burl Ives. Amazing! It was like watching Burl Ives play Saruman. I don't think that's what they were going for, but it was great. And I can confirm that Roots as a whole deserves all the praise given to it over the years. Ben Vereen, in particular, is amazing.
- Killer of Sheep (1979): A sad movie about a man so beat-down by the grim meathook future that he can't appreciate the odd moments of grace when they pop up. It's always been tough for me to stay engaged in a movie that has no through-line, but after the first half of Celine and Julie Go Boating I've learned to treat it as the dramatic equivalent of sketch comedy. And although this movie is not a comedy, its brand of existential despair gives it something in common with sketch comedy, so it wasn't as much of a mismatch as you might think.
- Metropolis (1927): The first of several movies we saw on Hulu when they made their Criterion movies free to watch over the course of a weekend. I'd seen it before; Sumana had not. My current opinion is an amplified version of my old, uninformed opinion. As a story, Metropolis is terrible, but if you treat it as an opera it's a pretty good opera, with lots of awesome stuff to look at. And the robot's wink is one of my favorite film shots ever.
- Modern Times (1936): Sumana has zero tolerance for protagonists whose incompetence is supposed to be endearing. Since endearing incompetence is the Little Tramp character's stock in trade, I probably should have anticipated her reaction to Modern Times. I wouldn't say I have zero tolerance for such protagonists, but I don't have a lot, and it really dampened the mood of the movie. Bright moments include the feeding-machine scene, the Tramp getting high on coke, and the one line of silent-film dialogue that redeems his incompetence (paraphrased): "I'll do whatever it takes to get you a home, even if I have to work for it!"
- Diabolique (1955): The cream of Criterion Weekend, an exciting thriller that Hitchcock wanted to direct but Henri-Georges Clouzot got there first. So the authors of the novel on which Diabolique was based wrote another novel with a similar twist but which was probably a lot worse, because that novel became Vertigo. I guess back then you couldn't just write a screenplay, you had to try it out as a novel first.
While watching this movie we noticed that it's effectively a Columbo episode. A little while later we watched the movie that serves as the pilot episode of Columbo, and there were quite a few similarities to Diabolique! Coincidence? I really don't know. The play that introduced the Columbo character premiered in 1960.
Bonus: Hulu kept interrupting the movies with commercials, creating the bizarre experience of watching subtitled French films on a local TV station in 1993. They've clearly got an algorithm for determining how many commercials they can cram in before people stop watching, because near the end of Diabolique the commercial breaks started coming once every six minutes. It's so suspenseful, they know no one's going to tune out.
It was the same three or four commercials the whole weekend, and one of the commercials included the perfect iambic-tetrameter line "In every segment we compete," which we mashed up with Sydney Smith's Answer to an Invitation to Dine at Fishmongers Hall.
In every segment we compete
The monsters of the deep to eat
- Shoot the Piano Player (1960): Hulu called this "Truffault's most playful film", and I misinterpreted this statement as implying that the movie would be a comedy. It is not a comedy, but it's not bad. I dunno about this whole French New Wave thing, though. I just don't know.
- I Married A Witch (1942): This was a comedy, and it was terrible. You know how sometimes a movie gets into Criterion more because it's representative of a genre than because it's good? I suspect that happened here. This is the perfect "stupid black-and-white non-musical comedy." After a decent opening it went downhill fast, and by mutual agreement Sumana and I skipped the middle 45 minutes of this 77-minute movie. Fortunately, this movie ends with a witch stealing a gubernatorial election through brainwashing and magical vote fraud! Stupendous! But please don't take that as an endorsement of the movie as a whole. I Married A Witch has an IMDB rating of 7.1, a rating that rightfully belongs to Ishtar.
- Foreign Correspondent (1940): Hitchcock's propaganda thriller. I was definitely caught up in it but I don't have a lot to say about it in the cold light of day. Some of the twists were not that surprising, others were good examples of Hitchcock thinking outside the cinematic box. E.g. most of the main characters get on a passenger plane to America and you're in the middle of some piddling drama that pits Group A against Group B, but then the real twist happens which is the plane is shot down by a German submarine!
- The Great Dictator (1940): I rewatched this after Sumana went to sleep, in an effort to get one more free Criterion movie out of Hulu. I remember really liking The Great Dictator and I wanted to bring my recently honed film-watching skills to bear on it. And it's... uneven. Chaplin's tramp-like character is as genially incompetent as ever, and his author-mouthpiece speech at the end, which expresses a lot of nice sentiments, makes no sense in terms of plot and does nothing to fulfil the incredibly tense dramatic situation that the rest of the movie has been building up. (Compare the speech at the end of Foreign Correspondent, in which Hitchcock puts away all his cinematic tricks and gimmicks and simply begs the American audience for help.) So I guess I have zero tolerance for sappy melodrama in comedies, especially comedies about horrible wars.
But when Chaplin plays Hitler it's amazing. This is the portrayal I can imagine getting under the dictator's skin. "Adenoid Hynkel" is a petty, insecure, puffed-up, blustering asshole, the opposite of Chaplin's tramp, a man whose legendary incompetence threatens to ruin the world. You can tell that Chaplin wasn't aware of how evil Hitler really was (or else you can understand why he claimed he hadn't been aware). Even though this is the most scathing satirical portrayal of Hitler I've ever seen--and well-timed to boot--you'd always feel bad for even deploying satire as a weapon instead of buying more government bonds.
- Emma Mae (1976): Kind of an unintentional pun on "MMA" there. Director Jamaa Fanaka tried to make a funny action film for a black audience, only to see it dubbed "blaxploitation" and released on DVD as "Black Sister's Revenge". Well, it's a very fun movie, but if you're expecting "Black Sister's Revenge" you're gonna be disappointed.
IMDB trivia says Fanaka was a fan of Billy Wilder, and it shows in this story of a country girl who comes to L.A. to live with her aunt and uncle. The plot, the action and the comedy are all driven by Emma Mae's tendency to take the most direct approach to any problem. A guy calls Emma Mae a hick, so she hits him. Her boyfriend gets arrested, so she starts a car wash to raise money to pay his bail. Someone else thinks a car wash is a dumb idea, so Emma Mae hits her. The Man shuts down her car wash, so Emma Mae robs a bank. And so on.
Jerri Hayes, who played Emma Mae, came to the showing and said that before Fanaka died last year, they'd been talking about doing a sequel to Emma Mae. That would have been really fun to see.
- J.S.A.: Joint Security Area (2000): March bonus! I don't have the stomach for Oldboy, so I'm skipping most of the Chan-wook Park festival, but I figured I could handle a thriller about the Korean DMZ. I mean, the worst that can happen is nuclear war. And it was pretty good! I was analyzing the movie while watching it (occupational hazard) and thinking how its use of symbolism and callbacks was corny and heavy-handed, but then the very last shot of the movie tied together two earlier scenes, which initially seemed to be nothing but comic relief, into the movie as a whole. Tied it all together in a way that made those scenes transcend comic relief to create something moving. The very last shot changed my opinion of the movie from "pretty good" to "very good". I didn't think that stuff happened in real life.
My attempt to explain this experience to Sumana revealed that this is very much a "you had to be there" thing, but if you have the chance to be there, I think you should take it. I'm not gonna say there's no disturbing violence in this movie, but it's no Oldboy.
(1) Thu Feb 28 2013 09:49 Ragtime Synchronicity:
"Bugs," said Krakowski. "In-tell-i-gence gathering devices. The
Constellation loves recording things. Now they're going to record
every conversation anyone ever has."
"I think you might be projecting a little."
Fri Feb 22 2013 10:44 100 Years of Markov Chains:
Back in January I took a little trip to Boston and hung out with Kirk. Among other things, we attended an event at Harvard celebrating the 100th anniversary of the paper that kicked off the Markov chain craze. I only wish Adam had been there. I've held off on talking about the event because I've been waiting for Harvard to put the video of the talks online. But that's a sucker's game, and now I have something better!
See, the first talk, by Brian Hayes, covered the amazing history leading up to the publication of Markov's seminal paper. He's now turned his talk into an article in American Scientist. The first few pages of that article are a basic introduction to Markov chains; the history starts on page four. Basically, Markov was a cranky old man who liked picking fights.
Markov’s pugnacity extended beyond mathematics to politics and public life. When the Russian church excommunicated Leo Tolstoy, Markov asked that he be expelled also. (The request was granted.) In 1902, the leftist writer Maxim Gorky was elected to the Academy, but the election was vetoed by Tsar Nicholas II. In protest, Markov announced that he would refuse all future honors from the tsar... In 1913, when the tsar called for celebrations of 300 years of Romanov rule, Markov responded by organizing a symposium commemorating a different anniversary: the publication of Ars Conjectandi 200 years before.
As acts of political protest go, the well-timed symposium is pretty great. At that symposium Markov revealed the Markov chain, which he'd invented as a way to smack down the dumb theological arguments of rival mathematician Pavel Nekrasov. His paper wasn't called "Markov Chains: Future Basis for Art and Scientific Discovery, Named After Me, A. A. Markov." It was called called "An Example of Statistical Investigation of the Text 'Eugene Onegin' Concerning the Connection of Samples in Chains".
Markov had manually gone through the first 20,000 characters of Pushkin's "Eugene Onegin", looking at every pair of letters, writing down whether the letters were both vowels, both consonants, vowel-consonant, or consonant-vowel. Then he'd modelled the transitions between those four states with a Markov chain. The result disproved an assumption about the law of large numbers, an assumption crucial to Nekrasov's mathematical argument for free will. There's something about this mindset that always gets me--inventing the sledgehammer so you can use it to kill a fly.
The other two talks were a lot more technical. I was mostly able to follow them, but I don't think I got much out of them. Here's a summary of all three talks from someone else who was there. But I strongly recommend Hayes's article to anyone who reads this weblog.
Wed Feb 20 2013 13:14 Welcome BoingBoing Readers:
If you're coming here from Cory Doctorow's review of Constellation Games, you might like to know about my web page for the book. The book was originally a serial, and I wrote chapter-by-chapter commentary as it was serialized. I also wrote four bonus stories set before, during, and after the novel, which I've released under CC-BY-SA. All that stuff is here.
You might also be interested to know that you get a DRM-free PDF version of the novel by buying direct from the publisher.
Now would also be a great time to mention that Constellation Games is eligible for this year's Hugo.
Thu Feb 14 2013 13:04 Fundamental Indeed:
I could spend all day just posting games that Board Game Dadaist comes up with. I forbear, for the sake of you, my readers, but Adam Parrish and I will email each other when we find an especially good one. And I think you should know about the best game BGD ever came up with (found by Adam back in December):
Fantasy Fundamental Rails (2005)
Players divide themselves into two teams.
Tue Feb 12 2013 14:43 What's New in "RESTful Web APIs":
We're ahead of schedule, which is good because we have a lot of work to do that isn't part of the book manuscript. Yesterday I sent out over forty copies of the manuscript to beta readers. That is too many beta readers, so at this point I must refuse anyone else who wants to be part of the beta, unless they have/had a hand in one of the standards we discuss, and they want to specifically critique our coverage of that standard.
With the beta closed I think it's a good time to go into a little detail about the structure of the book. My guiding principle was to write a book that will be as useful now as RESTful Web Services was in 2007. Like RWS, RESTful Web APIs has a main storyline that takes up most of the book. My inspiration for the main storyline were a few books that followed RWS, notably REST in Practice and Mike's Building Hypermedia APIs with HTML5 and Node.
RWS focused on the HTTP notion of a "resource", and despite the copious client-side code, this put the conceptual focus clearly on the server side, where the resource implementations live. RWA focuses on representations, and thus on hypermedia, on the interaction between client and server, which is where REST lives. The stuff you remember from RWS is still here, albeit rewritten in a pedagogically superior way. Web APIs work on the same principles as the Web, here's how HTTP works, here's what the Fielding constraints do, and so on. But the focus is always on the interaction, on the client and server manipulating each others' state by sending messages back and forth.
We've also benefited from a lot of tech work done by others. The IANA registry of link relations showed that state transitions don't have to be tied to a media type. The RFC that established that registry also showed how to define custom state transitions (extension relation types) without defining yet another media type to hold them.
Insights like these inform the new parts of RWA's main storyline. What makes your API different from every other RESTful API in existence? That's the only part you really need to buckle down and design. Everything else you can reuse, or at least copy.
In particular, you shouldn't have to design a custom media type. Your API probably isn't that different from other APIs, and a ton of hypermedia formats and protocols have been invented since 2007. We cover a few of the most promising ones in the book's main storyline. We cover even more of them afterwards, mostly in the big "Hypermedia Zoo" chapter. Here's the book-wide list:
- The HTTP headers
- URL lists
- JSON Home Documents
- Problem detail documents
- Web Intents
- GeoJSON (which is only a hypermedia format on a bizarre technicality!)
- Web host metadata documents
- CoRE Link Format
After the main storyline and the hypermedia zoo, RWA continues the RWS tradition of giving an API-centric view of the HTTP standard. We have a "crash course in advanced HTTP" chapter, some of which is an update of Chapter 8 from RWS. (Look-before-you-leap requests never caught on, but I still feel like I have to describe them in RWA because I have no other source to refer you to!) Appendix A is an updated version of Appendix B from RWS, with the addition of these exciting new status codes:
- 308 ("Permanent Redirect")
- 428 ("Precondition Required")
- 429 ("Too Many Requests")
- 431 ("Request Header Fields Too Large")
- 451 ("Unavailable For Legal Reasons")
- 511 ("Network Authentication Required")
Appendix B is an update of appendix C from RWS, with these API-licious new HTTP headers:
The amount of reused material in RWA is really small, because the main storyline is completely rewritten for 2013. And I haven't even mentioned our coverage of profiles, partly because I can't yet think of a way to talk about profiles at less length than what we say in the book.
Tue Feb 12 2013 08:26 Hire Aaron DeVore:
I don't often use the NYCB bully pulpit to tell you to hire someone (apart from myself), but folks, you should hire Aaron DeVore. He was effectively the maintainer of Beautiful Soup during the period when I wasn't working on it. He answered tons of questions on the mailing list and sent me bugfix patches. When I started work on Beautiful Soup 4, he gave me a lot of feedback that helped stabilize the API.
Aaron did all this while a college student in Portland, Oregon. Now he's about to graduate, and he's looking for a job. Send him an email and let him know what you've got going on.
Thu Feb 07 2013 13:14 Constellation Games Interview in Bookslut:
Hey folks, CG fan Jeanne Thornton interviewed me a couple months back, creating a text that has now been published on Bookslut. (There's also an interview with Saladin Ahmed in the same issue.) The interview ranges over the CG publication process, games as an art form, space exploration, and so on.
One thing the published interview doesn't include is a question about Tetsuo Milk, which Jeanne cut before submitting the interview because it was kind of inside-baseball. But hey, inside baseball is the whole point of News You Can Bruise, so with Jeanne's permission I've reproduced the original question and my answer here:
I would feel remiss in not asking you about Tetsuo Milk, a character whom you’ve said (in your really, really mind-blowingly extensive commentary on the novel) essentially ran away with the book. Tetsuo is a brilliant character, but also feels at times like a heterogeneous element. I like this effect a lot, but I’m curious as to where this guy came from, what you’re saying through him, and how you see him fitting into the overall mix.
Maybe this will help: Tetsuo Milk is the ET version of Ariel. His silly mistakes and misunderstandings are mirror images of the mistakes Ariel makes trying to understand the Constellation. We don't laugh because we're not the ones being misunderstood. When Tetsuo does it to us, it's funny.
Here's a spoiler-free example. One of Ariel's post-contact hobbies is posting reviews of alien computer games to his blog. There's one really important scene that reverses the roles: Tetsuo writes a review of a game Ariel worked on as a developer, Brilhantes Poneis 5.
Brilhantes is a stupid Farmville-type mobile game where you have a pet pony and do pointless tasks to earn coins to buy accessories for it. Tetsuo tackles the game from a post-scarcity Marxist perspective, putting a lot of work into understanding how a game's economy can work when the player is the employee of an animal. He gets a lot of it right (i.e. he recognizes that the game demeans both its players and its developers), but he's operating from completely the wrong framework.
That's the kind of mistake Ariel makes. He brings his human assumptions to everything, whether he realizes it or not, whether or not Tetsuo or someone else calls him on it.
(This is why there's a reference to "Tetsuo-like ideas" later in the interview; we shoulda cut that reference.)
(1) Sat Feb 02 2013 14:40 Spacewar! The Interview:
Went to the museum last night not for a movie, but to meet Peter Samson and (via poor-quality videoconferencing) Steve Russell, for a conversation about the second video game ever made, Spacewar!.
I asked Russell the question that's been burning in my mind for years: why does Spacewar! have an exclamation mark in its name? His answer: "Once I got it working, I thought it deserved an exclamation point!" I also asked Russell if he considered any other names for the game. "Nope."
No one asked the obvious final question, so I got that one in too: what games are they playing now? Both Russell and Samson are fans of solitaire card games. Russell also said he likes the Android game Tiny Village.
Some other tidbits from the conversation, which I found especially interesting and/or which I don't think are on the net already:
- I knew this before, but it didn't really sink in. Spacewar! was released (through PDP user groups) in 1962. There would be no more computer games released until 1969. When the famous Rolling Stone article about Spacewar! came out, the game was ten years old, and it was still the most popular video game in the world, almost by default.
- Samson said the name of the program Expensive Planetarium was a "double irony": it's an obvious reference to Expensive Typewriter, but "planetariums really do cost a lot of money."
- Russell was asked his opinion on Asteroids. He said it was "a great solution to the problem of only one person in the bar who wants to put quarters in the machine."
- It took too much computing power to calculate the trajectories of Spacewar! missiles under gravity, so they were deemed "photon torpedoes" which wouldn't be affected by gravity. Was this a common idea from earlier sci-fi or did Spacewar! invent the term? I should have asked.
- Samson said he later worked on a computer system for a "three-letter agency." This was a system with eight screens; it sounded a lot like IBM's SAGE computer. The hardware was late so they threw in some software for free, including a port of Spacewar! which could play four games simultaneously on all eight screens, and which had random matchmaking, so you didn't know who you were playing with. According to Samson, the intended purpose of the computer system was not too far removed from playing Spacewar!.
Fri Feb 01 2013 10:13 Video Roundup: January 2013:
Gonna put one of these up every month, so as to avoid the big bolus of reviews that happened last time. There are only three films here, all from a Paul Williams retrospective at the Museum of the Moving Image.
- Phantom of the Paradise: A flop on its initial release, Phantom has seen a resurgence as a cult classic. A very uneven movie that does over-the-top comedy very well but everything else is kind of a mishmash taken from other films. Well, a lot of the comedy is also taken from other films: there's a great scene that uses the car-bomb scene from Touch of Evil by way of killing off the Beach Boys. But that scene is great because it doesn't just blindly reference another movie, It adds a layer of complexity onto the original (using a split screen to show an "onstage" view and a "backstage" view which are united by the explosion) that creates suspense even if you've seen the referent.
But for me, the character of Beef stole the show. Beef, the flaming metalhead, progenitor of both Rocky from Rocky Horror and Vyvyan from The Young Ones. Good times. Rocky Horror has better songs, though.
- The Muppet Movie: It's The Muppet Movie. We all know how it goes. People in the theater were pre-laughing when we saw Miss Piggy and Kermit in the restaurant, because we knew Steve Martin was gonna show up, and then he showed up and life was good.
- Ishtar: This film has a really bad reputation and a horribly low IMDB rating, but I was looking forward to it because I loved Elaine May's A New Leaf. And I was right to look forward to it! This is a great movie. It's really funny and the plot comes together nicely. It's not perfect, but it doesn't deserve a 3.9 rating.
To illustrate my point, I've brought back The MST3K-IMDB Effect for a return engagement. Here, the part of "a movie being (or not being) on MST3K" will be played by "a movie being (or not being) Ishtar."
That's right, Elaine May's MST3K-IMDB effect is 22.9 standard deviations, sixty percent higher than the previous highest MSTm of 14.1, which was itself highly suspect. It's not impossible that a director would hammer out three movies that cluster very tightly around an IMDB rating of 7.1, and then deliver a movie that's 23 standard deviations worse, but I'm not seeing it. Instead, I would rate Ishtar a little lower than A New Leaf. The Onion agrees.
So what explains the low rating? The obvious reason is sabotage. The movie cost too much to make, May took way too long to edit her films, and the studio head had some kind of feud with Warren Beatty which he took out on Ishtar in public. Once the movie had become a punchline (specifically, the punchline to the Far Side "Hell's video store" cartoon), it was too late to change anyone's mind.
But I have two other explanations. First, during the Reagan years, during Iran-Contra, the moviegoing public was probably not in the mood for Ishtar's ham-fisted satire of America's foreign policy. Let alone one that climaxes with two Americans firing rockets at a CIA gunship that's trying to take them out. But now it seems pretty light by comparison with real life.
Second, Ishtar is a film about idiots who are really bad at something creative. Any time you write an infrafictional work that's supposed to be bad (and the songs in Ishtar are hilariously bad), you're inviting people to skip a step and believe that you have written something simply bad. Spinal Tap are idiots, but they have good songs.
Anyway, Elaine May was actually at this showing, and she took a couple questions, but she seemed really shy so we didn't bombard her with our fandom. Especially because other people were bombarding her with their fandom, and we saw how awkward it was.
Sat Jan 26 2013 10:10 The Crummy.com Review Of Things 2012:
I've been battered from all sides, and working all the time on RESTful Web APIs but I really feel like I need to get this out before the end of January, so I took some weekend time and finished it. First let's briefly review The Year in NYCB!
- Probably the NYCB champion of 2012 is Dada Da Dada Da Dum, aka Dada Limericks. Endlessly amusing.
- Runner up: A Time Machine And Other Poems. Age cannot wither "Was it a whale with its spout?", nor custom stale its infinite variety.
- Frances Daily. Don't worry, it's still running. My mom skipped January 1988, possibly for the same reason I'm posting my year-end review at the end of January 2013. Though now that I think of it, it's more likely that page was simply lost in the intervening 25 years.
- My explanation of what I wanted to do with Findings. This was later rendered moot when Amazon nerfed Findings by changing its Terms of Service. My puny revenge: ASINs That Spell Words.
- My experiments with IMDB data, especially Worst Best Picture.
- Software and tech stuff: Beautiful Soup 4, the new robotfindskitten, and "How to Follow Instructions", for which I need to set up a dedicated page, but I'm busy turning it into a book, give me a break.
- Fiction stuff: "Four Kinds of Cargo" and Lisa Imas's illustrated "Let Us Now Praise Awesome Dinosaurs". The majority of 2012's NYCB work went into the Constellation Games author commentary, and that probably deserves some special recognition, but in terms of individual CG posts I just want to highlight the results of the CDBOEGOACC game design contest.
- The 2012 Loaded Dice update didn't get very much attention at the time, but I think it quantified something interesting about the way board gamers think about games.
- Crazy the Scorpion, the card game that rocked a living room.
And now, our feature presentation. Of all the artifacts I experienced last year, these were my favorites.
According to my LibraryThing stats I only read fifteen books in 2012! I think it missed a couple books I read on the Kindle, but that's really low. What was I doing instead? You'll find out below.
The new novel, currently on hold until RESTful Web APIs is finished, features lots of lesbian sex, so I asked a friend (who can speak up in comments if she wants to be identified) to recommend the best lesbian porn novel. Her recommendation, Sara Waters's Tipping the Velvet, is the Crummy.com Book of the Year. Great historical fiction plus dirty sex.
Runner-up: Charles E. MacKenzie's Coded Character Sets: History and Development. (q.v.)
- Audio: Man, this is starting off kind of pathetic. The only known good album I added to my collection in 2012 was the Bit.Trip Runner soundtrack, and playing the game (which I'll mention below) generates a soundtrack that's better than the prepackaged one. I say "known good" because although I did buy a fair amount of music in 2012, listening to new music requires my attention, and if I'm sitting at the computer just listening to music, I start thinking I should get some writing done. I will say that The World Record's "Freeway Special" seems pretty good. What is my problem? I'm seeing a Ben Folds album and a TMBG album in here that I haven't listened to. Anyway, my problem, not yours.
I did a lot better with podcasts. New-to-me educational podcasts include
The Human Bible
and The History of Rome, which dramatizes ancient history in a way that has gotten it to stay in my head. (It also has an unauthorized sequel, The History of Byzantium, which I haven't gotten to yet.) Prominent among non-educational podcasts are No More Whoppers, in which two friends press each other's buttons by pressing buttons, and Flip the Table, in which the terrible design of old media tie-in board games collides with the fact that it's hard to have a really bad time playing a game with your friends. Somewhere in between is the edutational Roguelike Radio, which I've been listening to since it started in late 2011, but which has been consistently good, and which got better in 2012, to the point where I feel it deserves a mention.
Finally, music and podcasts collide with Music for Programming, a great idea which only succeeds in making me feel guilty for not listening to the albums I've bought.
- Video: This was the big one, thanks to our Museum of the Moving Image membership. I posted a pretty comprehensive review post for the second half of the year, and also this earlier post that's kinda sickly in comparison. According to my metrics I saw thirty feature films in 2012. That's probably not all of them, but it's the most I've ever seen in a year, by a wide margin, and twice the number of books I read. Damn, me!
It's really tough to pick a Crummy.com Film of the Year, but that's the whole conceit here, so I'm gonna give it to Celine and Julie Go Boating, the Bechdel-riffic surreal fantasy buddy comedy. Runners-up: more conventional comedies A New Leaf and The Whole Town's Talking. All three were big surprises.
Not a lot of non-film video going on. Sumana and I really enjoyed Breaking Bad and the House of Cards trilogy, though. In Internet video news, Joe Hills is hilarious.
- Games: First, a rundown of the games I didn't play.
You'll recall that 2011 was the year I decided I was done with non-tactical RPGs. 2012 was the year I didn't play any non-tactical RPGs, and I have no regrets. The good thing about writing off an entire genre is you can crown an all-time champion. The best non-tactical RPG ever made was Mother 3. (My in-depth coverage: 1 2)
2011 was also the year I was playing some Zelda game or other and I thought to myself, "you know what, I've played this game enough times." In 2012 I played some old Zelda with Beth on her Virtual Console, but I didn't buy Skyward Sword and I won't be buying future games in the series unless they hammer out a Majora's Mask level of weirdness. The best Zelda game ever made was Link's Awakening for the Game Boy.
So what did I play? Not a whole lot, actually. I think my revived interest in video games may have been an artifact of my desire to write Constellation Games, not a permanent feature of my personality. I checked the contents of my DS card slot to see what was the last game I played on that system, and it was... the RPG that convinced me to give up on RPGs back in 2011.
But on the Wii there was Bit.Trip Runner, a fun twitch-reflexes game that creates a soothing Phillip Glass type soundtrack as you play it, which Sumana really likes to hear. On the PC there was more Minecraft, thanks to the insane modpack collections, especially Feed the Beast, which skyrocket the complexity of the base game and fulfill all my "Programmable Minecraft" dreams.
I bought every Humble Indie Bundle in 2012, and this time I tried all the games, but the only game that came out of it that I really liked was Bastion, which I mainly liked for its clever way of narrating your actions. But I loved FTL, a game I got as a Kickstarter reward. FTL combines solid theme, real-time tactics, and roguelike procedural generation+permadeath to recreate the experience of running your very own smuggling ship, just like in "Four Kinds of Cargo"! Or Firefly, if that's what you're into. It's great fun, and I'm giving it Crummy.com Game of the Year just so it doesn't go to Minecraft two years in a row.
Board games are still going strong. The Kickstarter reward games that have really paid off are Sunrise City and Glory to Rome. I started playing Dominion online with Dan. Pat and I played three games of 1989: Dawn of Freedom, and I enjoyed them all. Pat introduced us all to the auction action of For Sale, and I can't believe I didn't mention the fatalist chaos of Galaxy Trucker last year, but it's possible we only started playing it in 2012. It's great!
As embarrassing as it seems, I think the Crummy.com Board Game of the Year is Lords of Waterdeep, a game that was actually released in 2012. I give it this coveted honor for the two times we've been thinking about learning some other, heavier Eurogame, and decided to just play some Lords of Waterdeep instead. The theme is so terrible! (And you can't role-play as your character because your character must stay hidden until the end of the game!) But it's fun. Honorable mention: Galaxy Trucker.
- Food: Not a lot new here, but it was a big year for cheese in my neighborhood. We discovered the grilled-cheese-heavy vegetarian restaurant The Queens Kickshaw, and celebrated the opening of a store that sells cheese, Astoria Bier and Cheese.
Looking forward to 2013: man, we're already 1/12 of the way through 2013! This should have gone up a month ago! If I can finish RESTful Web APIs and Situation Normal I'll call it a good year.
Sat Jan 19 2013 22:05 Crazy the Scorpion Semi-Online:
Kirk and I collaborated on an in-browser version of Crazy the Scorpion for Klik of the Month Klub. It's "online" in the sense that you download an HTML file containing the game and play the game in your browser. But everyone who plays must be gathered around the same computer.
I scraped a bunch of Wikipedia page titles to make fake Trivial Pursuit cards. It's not great, but the whole thing's not bad for two hours of work. I mainly hope this version inspires you to play Crazy the Scorpion using physical components.
Fri Jan 18 2013 08:06 429 Too Many Requests:
I don't like repeating what everyone else is saying on this weblog,
and I don't have much to add to the general outpouring following the
death of my friend Aaron, but I have to say something, because you can't say goodbye if you don't say anything. His death was awful, our loss great, his crimes (assuming any crime was committed at all) minor, and their prosecution farcical. I feel like a lot of what
we're going through is our frustrated desire to see Aaron's case
properly litigated, to see our friend vindicated, and I have no experience with that stuff, but I do
have two personal stories to share. Two points where my life intersected with Aaron's in ways I haven't talked about publicly.
- Beautiful Soup was partly inspired by xmltramp, an XML
parser Aaron wrote because he was frustrated with other XML parsers. I've been thinking a lot about this, and
this is why my initial mourning of Aaron took the form it did, because
screen-scraping—the use of an automated agent to replace
a human-driven web browser—seems to have been at the core of the
prosecutor's belief that this was a blockbuster case, more akin to a bank
heist than a defaced storefront.
- In 2005 Aaron wanted me to join his startup, Infogami. He showed
me a prototype, a NewsBruiser-like blogging site. I was looking to
quit my job at CollabNet, but I didn't take Aaron's offer because I
was comfortable in San Francisco and really didn't want to move across the country. (In a
Twilight Zone-level twist, in early 2006 I'd end up moving to
New York, which I now like a lot better than San Francisco.) Aaron tells
the next part of the story here. He
couldn't find a partner and eventually ended up merging Infogami with
Reddit, which was then sold to Conde Nast in 2006.
Back in 2005 there was enough of the college-era me left that I
would have seen this outcome as a big missed opportunity. I still had
some desire, left over from the dot-com era, to win the startup
lottery. But of course the Reddit merger happened because Aaron
couldn't get a partner for Infogami. And my life over the next couple
years, including my secondhand reading of Aaron's experience at Reddit (he was fired soon after the Conde Nast acquisition) made it clear to me
that I would not enjoy winning the startup lottery any more than Aaron
did. I count this among the most important things Aaron taught me.
I've cut a lot of what I wrote here because I don't want this entry to be a bunch of stuff about me and my opinions and what I think. But I'm the one who's still here. Aaron is gone, and all that's left of him is the parts we can share.
 I can't let go of these little technical inconsistencies between what I'm seeing now and what I remember. It looks like during the merger, Infogami stopped being a blogging
site, and its framework (which became the first Python version of
Reddit) was renamed "Infogami". Or maybe "Infogami" was the framework
all along, and the blogging site was only one application of the
product; I don't know.
Mon Jan 14 2013 09:44 For Aaron:
from bs4 import BeautifulSoup
url = 'http://www.poemhunter.com/best-poems/william-stafford/thinking-for-berky/'
soup = BeautifulSoup(urllib2.urlopen(url))
for s in soup.find(attrs="poem").strings:
(2) Fri Dec 21 2012 17:50 new robotfindskitten!:
It's a big day for announcements. There's a new "Mayan Apocalypse" version of robotfindskitten, due largely to to the work of new dev team member Eric S. Raymond.
My major contribution to the release was editorial. I consolidated the list of NKIs with lists produced for other ports (notably Dave Griffith's Frotz port). I removed some stupid fifteen-year-old jokes, and added new stupid jokes. The end result is that this version of rfk ships with 701 high-quality NKIs, versus about 400 NKIs of wildly varying quality.
(6) Fri Dec 21 2012 10:34 RESTful Web APIs:
I guess it's time to announce my next book project. Mike Amundsen and I are working on a book for O'Reilly called
RESTful Web APIs, a follow-up to 2007's RESTful Web
Services. Our deadline is the end of March 2013, and the book
should come out a few months later. (Here's Mike's announcement of the project.)
Here's my announcement
of RESTful Web Services from about six years
ago. You can see how far we've come since then.
In 2008 I came up with the now-famous "maturity
heuristic" (aka the "RMM") for quickly judging the quality of an API. If you go back and
look at RWS through the lens of the maturity heuristic, you'll
see that it spends a lot of time trying to get the reader to grasp the
concepts behind Level 1 services: that URLs identify resources,
and that resources are the proper targets of HTTP requests. That's
because back then, Level 1 was about the best you could expect from a "REST" design.
RWS spends a whole chapter treating Amazon S3 as though it
were a great example of RESTful design, because in RMM terms it made
it all the way to... Level 2. RWS spends another chapter
arguing for the legitimacy of REST against a thriving part of the industry that's
content at Level 0.
We don't live in that world anymore. I'd say the modal RMM level of
a new "REST" API today is 2. Most of the 2007-era arguments have been
rendered irrelevant (SOAP vs. REST, the usefulness of PUT) or been
given generally accepted best practices (URL formats). Restful Web
Services was the first book-length treatment of REST; there are
now over ten books on REST from O'Reilly alone. There are lots of
development tools that make it really easy to crank out a Level 2
In short, a lot of gravel- and rock-sized problems have been excavated,
revealing two boulder-sized problems. I mentioned these two problems
in my QCon talk "How to Follow Instructions" (available as video in hour-long version and in a half-hour condensed version
presented at RESTFest; see also my QCon slide deck and speaker notes). Now I'll state them explicitly as the two
major problems I want to address with RESTful Web APIs.
Duplication of effort
Here's an article from May on Programmable Web, a great site
that tracks the world of public APIs: "53
Microblogging APIs: Twitter, TwitPic and What The
Trend". Fifty-three was really the number of organizations
publishing microblogging APIs, and some of them are utility APIs rather than "microblogging APIs" per se, but after doing some spot checks I do
believe there were approximately fifty-three distinct APIs.
Should there really be fifty-three distinct APIs in this field?
What if there were, like, five? We're not talking about something
complicated, like insurance policies or regulatory compliance. We're
talking about posting a little bit of text to a user account. Can we
at least get down to ten?
No, apparently not. Like I said, that number was from May. There
are now fifty-seven
microblogging APIs. Well, damn. What if we got together and
hammered out a standard, interoperable microblogging API? Sounds
great, except nobody's interested. Evan Prodromou tried that with
OStatus, and nothing came of it. You could use AtomPub as a
microblogging API, pretty much unmodified, but nobody seems to want to.
The original RESTful Web Services sets out a design
procedure for taking your problem domain and turning it into an
API. But there's a big problem with this procedure: it doesn't
consider the possibility that fifty-six other people may have already
done the work for you.
Back in 2007, that possibility was remote. There was one
microblogging API. I had to scrape the barrel pretty thin to find
enough quasi-RESTful example APIs to fill Appendix A. Now the big problem
is too many APIs. When you design the fifty-eighth
microblogging API you're limiting your audience and wasting your
This is a really huge problem and we won't solve it with a book. But we can point out that it's a problem and take the first step towards mitigation. In RESTful Web APIs we present a new design procedure that focuses on reusing existing designs, coming up with new stuff to fill any
gaps, and then publishing the new stuff for the benefit of a) your users
and b) API developers who come after you.
Whatever influence RESTful Web Services has had, it stops dead at RMM level 3, the hypermedia level. In 2012, the modal new "REST" API is at level
2. It has been level 2 for a while, and will continue to be level 2
unless something changes. Most architectures don't support hypermedia
very well (or at all), and a lot of developers still don't understand it.
That's understandable. To the programmer brain, hypermedia is weird. When you read
RWS it's possible to just skip the parts about hypermedia, and
I get the feeling a lot of people did. There's stuff in RWS
that is now outdated, and stuff that was useful for dealing with
2007's problems but that I now consider just plain wrong. Most of that
stuff has to do with hypermedia.
RESTful Web APIs is tightly focused on hypermedia--you will
not be able to skip the "hypermedia parts"--and it deals with 2013's
problems. Some specific shortcomings of RWS we'll address:
- We as developers have a strong desire to export our server-side
object models as APIs. This is the idea behind JAX-RS. It's the
idea behind lazr.restful, the framework I developed to make the
Launchpad web service. But it's also the idea behind SOAP, and it's how we got fifty-seven microblogging
This desire isn't technically at odds with the use of hypermedia,
but it does tend to use hypermedia only to describe safe state
transitions (i.e. to link resources together), and RWS didn't
really go further than that.
- In 2007 there were no hypermedia formats based on JSON, but
JSON-based representations were becoming overwhelmingly popular. RWS does not offer advice for this scenario. Developers
either made up their own one-off hypermedia formats (as I did with the
Launchpad web service), or, more commonly, ignored the hypermedia
constraint. Today JSON has a number of general-purpose hypermedia
formats (like Siren, Collection+JSON, and HAL), and we'll cover them
- In RWS I was very dogmatic about avoiding overloaded
POST. I felt strongly about this because I really needed to yank
people out of the SOAP-RPC mindset. But I've come to accept that overloaded POST exists for a
reason: to convey state transitions that don't map naturally onto one
of the other HTTP methods.
These state transitions are legitimate if they are described by
hypermedia controls, as they are on the Web. They're not just hacks to get around the fact that HTML can't describe PUT and DELETE methods. Prohibiting
overloaded POST in the name of the uniform interface forces designers
into complicated workarounds to minimize the difference between their application semantics and the protocol semantics of
GET/PUT/POST/DELETE. Those workarounds usually reflect the object
model the designers already defined for their CRUD databases. Or, the
designers give up, use overloaded POST without using hypermedia
to describe the state transitions, and feel guilty about it.
Besides which, forbidding overloaded POST means that you can't use
HTML, the world's most popular hypermedia format. That's a big
As I said, this is a brand new book, not a second edition of
RESTful Web Services. The only major things we're reusing are the
appendices dealing with HTTP headers and response codes. To signify
the handoff, we're changing the name.
RESTful Web Services was a great name in 2007, but it's not
so good now. See, when SOAP went down, it took the term "Web Service"
with it. "Web Service" is now an enterprise software term. Everywhere
else it's all "API API API".
As I mentioned last
year, this is backwards. The SOAP approach is to make a remote
service look like a local library. (This idea is not limited to SOAP,
of course.) Whereas the RESTful believe that a remote service should
look like the Web. (Whatever that means.)
But I'm not gonna re-litigate that argument. "Web Service" is dead,
we're stuck with "API", and the name of the book needs to change.
"RESTful Web APIs" is a direct translation of "RESTful Web Services"
into 2013-speak. It keeps continuity with the old book, but it's a new
title for a new book.
If you want to be a beta reader, let me know (via comment or email to firstname.lastname@example.org). I'll get in touch with you around the end of March, or possibly earlier. Update: We've got a lot of beta readers and we're no longer accepting applications.
Once in a while I add a section to a chapter called "The Hypermedia Zoo", which takes a not-too-detailed look at a lot of hypermedia-aware data formats to demonstrate their diversity. I'm always on a low-level search for more hypermedia formats, and if you name your favorites in the comments I'll take a look. Don't worry about duplicating my list; I also want to see what springs to mind when you think "hypermedia format".
There's also a shorter section called "The Semantic Zoo" which talks about interesting domain-specific data standards. I'm mostly interested in reusable profiles (the IANA registry of link relations, microformats, schema.org microdata, Dublin Core), but I'm also interested in pointers to standalone data formats like Activity Streams. There's a whole lot of stuff out there and I'm not confident that I've seen even half of it.
Fri Dec 21 2012 09:23 Strange Horizons Reviews "Constellation Games":
I've been nervous about this review all week, but it's very positive:
Constellation Games is a very funny book, with an effective sweep into pathos that delivers a powerful message... Like the Ignobel awards, Constellation Games makes you laugh, and then it makes you think.
Definitely the first review to call the naming of the Constellation species "an effective piece of othering".
(3) Thu Dec 20 2012 15:40 Year's Best:
"Four Kinds of Cargo" will be appearing in the 2013 edition of Prime Books's "The Year’s Best Science Fiction & Fantasy". That's me in the same book as Elizabeth Bear, Ursula K. LeGuin, Kelly Link, Robert Charles Wilson, and other luminaries.