News You Can Bruise for 2006 December

"Hi. I'm a human..."
"...and I'm a Cylon."

Well, it didn't take me long to get up and search for those commercials online, where they now live. They're surprisingly similar to Douglas Adams's much more enjoyable Hyperland. In fact, I think Steven Levy was thinking about Hyperland when he described some aspects of Knowledge Navigator.

This year Backup Thanksgiving was roast chicken, orange-cranberry sauce, dressing, mashed potatoes, mashed sweet potatoes, carrots, green beans, and Parker House rolls. So delicious. But now Sumana's sick of the leftovers and I'm almost done reincorporating them. I made stock from the chicken carcasses and bread pudding from the cranberry sauce. Now I have to start thinking about what to make instead of just pulling stuff out of the fridge.

PS: The first six chapters of the REST book are now done!

The mail consisted entirely of commemorative postal covers addressed to President of the United States Dwight Eisenhower, other government officials, the Postmasters General of all members of the Universal Postal Union, and so on. They contained letters from United States Postmaster General Arthur E. Summerfield.

Reinforces my belief that the postal service makes most of its money acting as a kind of live-action MMORPG, arbitrarily creating things and declaring them to have value.

You do have to pick something in a field the other person knows fairly well, because the fun of the game is in being able to make a valid guess every time a new piece of information is revealed. I did very poorly when Sumana was going through "Hollywood Squares contestants" and "People from Cleveland". OTOH Sumana did well identifying the honorary Globetrotters (not an official Wikipedia category at the moment, though if you put the Futurama characters in "fictional honorary Globetrotters" you'd have one of the best Wikipedia category names ever). Sumana reminds me that I did well with "Fictional robots with emotion" and "Characters played by a member of the opposite sex". At the intersection of those two lies Gypsy.

Sumana says "You're really only indie and fresh if Wikipedia deletes you for not being notable."

• I got my Futurismic check today! (Plus T-shirt) I'm not yet a published fiction author but I am a paid fiction author.
• I thought of a publishing scheme that combines the street performer protocol with micropayments and the tradition of pulp magazines. I put up descriptions of stories with word counts. You send in payments at magazine rates (5-6 cents a word) and I reveal the appropriate number of words. If people think a story is boring they stop paying for it and it just cuts off in the middle.
• Recently I've noticed two comics that operate under similar constraints. If only there was some way to combine them...
• I got a copy of 1984's The Whole Earth Software Catalog, ostensibly for research for a 1980s steampunk story, but really because contemporaneous reviews of early personal computer software are SOLID GOLD. This is the tradition I like to deface in my Software and Game Roundups (been a while since the last one of those, actually).
• Diane Duane linked to my Star Trek slash pairing generator. This is vicariously exciting as Diane Duane wrote the best Trek-franchise novel among the far too many I read in grade school (that'd be Spock's World, which has the truest-to-life depictions I've seen of everyday life in the Federation).

He spoke to no one in particular. Most likely he was hawking fake or otherwise black-market Rolexen, getting plausible deniability from the crowd and his deadpan delivery. But I like to think of him as one of the new breed of mad monks, who take their mantras from the Heraclitan river of spam that flows through our lives. Strong buy strong buy one to watch about to explode strong buy...

what do the different parts of the christmas cracker mean?

What does a paperclip mean?

Here's one answer, though I can't determine the relative positions of tongue and cheek:

The Christmas cracker was invented, purely by chance, by an English baker called Tom Smith. He took the simple principal of the wrapped sweet or 'bon bon' and added first a love motto then, after much experimenting, a strip of paper impregnated with a compound which would 'crack' when opened. Over the course of time he dropped the sweet, lengthened the wrapper and introduced small novelty gifts.

I guess 2% inspiration and 98% perspiration moves an invention into "purely by chance" territory?

Much as I dislike the way SOAP is used these days, I'm not inclined to gloat, because the deprecation just means more work for me. I used the SOAP search service as an example in all three of my books. Now I've got to find another free, public, non-obscure SOAP service to use as an example (ideas?).

The official Google narrative is that the SOAP-based web service has been replaced by something called the "Google AJAX Search API". If you take this narrative at face value it means that Google has taken down their web service and put up an AJAX library in its place. What's AJAX? Who knows, it's AJAX. Here's a typical weblog entry on the topic.

Another victory for REST over WS-*? Nope -- Google doesn't have a REST API to replace it. Instead, something much more important is happening, and it could be that REST, WS-*, and the whole of open web data and mash-ups all end up on the losing side.

It's probably my recent proximity to Sam that's doing it, but I'm noticing a tendency in myself to draw fine distinctions. There's a sense in which this narrative is right and a sense in which it's not. I'm going to pick apart the narrative and show what exactly is disturbing about the Google AJAX Search API.

It's not true to say that "Google doesn't have a REST API to replace it." In fact, Google has two REST APIs, and one of them predates even the SOAP API. You've probably used this old API: its primary endpoint is http://www.google.com/.

Yes, the Google website is in fact a very RESTful web service. The downside of this web service is that it's a little bit difficult to use automatically, as opposed to through a web browser. It serves data in a human-oriented format (HTML), and you have to screen-scrape it into a data structure if you want to do anything for it.

There are libraries for doing this, but the other problem is Google doesn't want you to do it. It violates Google's Terms of Service ("No Automated Querying"). Lots of inconsiderate people write scripts that hammer Google's REST API day and night. Google tries to prevent this by sniffing out anything that might not be a web browser and preventing it from accessing the API. (To see this, set your browser's User-Agent to "libwww-perl" and try to use Google.)

But it can't be denied that people outside of Google have a powerful hankering for Google's dataset, so eventually someone (Nelson Minar, it seems) came up with a second web service API that was designed just for use by automated clients. The catch was that you had to sign up for a unique key to use it, and that key would only work for you 500 (later 1000) times a day.

In point of fact Nelson chose a SOAP/WSDL architecture for this web service. But there was no need to use any different architecture at all. Here's a possible different way of implementing the constraints above:

When you make an HTTP request to google.com, we try to figure out whether you're a web browser or an automated client. Ordinarily, if you're an automated client, we shut you out. But here's the deal. Now you can sign up for an "automated client key". When you make an HTTP request to google.com, stick your key into the Authorization header. Not only will we not shut you out, we'll try to make things easy for you. Instead of a human-oriented HTML document, we'll send you the appropriate data in an easy-to-parse XML format. But, we'll only do this for you 500 (later 1000) times a day. Then it's back to shutting you out.

This technique has a number of subtle benefits which I could bore you with for quite a while. But its obvious benefit is that it's got the exact same "API" as the Google website, which everyone knows how to use.

Anyway, instead of going down a route like this (which would, I think, have changed the history of web services quite a bit), Google went down the SOAP/WSDL route. Now they're deprecating the SOAP service in favor of some mysterious "AJAX API". This brings me to the second of Google's REST APIs.

There is no magical thing called an "Ajax request". An Ajax client makes normal HTTP requests, and processes the results automatically, just like a web service client. An Ajax client is a web service client.

What HTTP requests is the Google Ajax client making? I poked around a little bit and it looks like it mainly makes GET requests to URIs that look something like http://www.google.com/uds/GwebSearch?callback=GwebSearch.Raw&context=0&lstkp=0&v=1.0&key=xxxxxxxxxx&term=web+services. That's not exactly http://www.google.com/search?q=web+services, but it's not too far off either.

The Google AJAX API consists of a browser-side Javascript library and a server-side web service. The one acts as a client for the other. From what little I've seen of the web service I'd consider it quite RESTful. In fact, it's architecturally very similar to Yahoo!'s RESTful search API. They both use the same (IMO, fairly unsafe) trick to get a web browser to execute dynamically-generated Javascript code from another domain.

The main difference is that Yahoo's search API can also be made to send data (in JSON or ad-hoc XML format) instead of executable Javascript. That makes it possible for the service to be consumed by automated clients, not just by web browsers running client-side Ajax programs.

Let me just see if I can do something similar with the Google web service. The Javascript it serves is extremely close to also being a JSON document; I should be able to hack it a little and parse it as JSON.

Here's some Ruby code that gives you kind of a command-line Google search like people used to write for the old SOAP API. It requires the json gem.

You can skip the code.

#!/usr/bin/ruby
require 'rubygems'
require 'uri'
require 'open-uri'
require 'json'

KEYS = %w{GsearchResultClass unescapedUrl url visibleUrl cacheUrl
          title titleNoFormatting content results adResults
          content1 content2 impressionUrl}

def search(key, term)
  uri = "http://www.google.com/uds/GwebSearch" + 
    "?callback=GwebSearch.Raw&context=0&lstkp=0&v=1.0" + 
    "&key=#{key}&q=#{URI.escape(term)}"
  javascript = open(uri).read
  
  # Hack quotes around the hash keys to make the Javascript string
  # into JSON.
  KEYS.each do |key|
    find = Regexp.compile("\s*#{key}\s*:")
    json.gsub!(find, " \"#{key}\" : ")
  end 

  parsed = JSON.parse(json)
  return parsed["results"], parsed["adResults"]
end

# Command-line interface begins here

(puts "Usage: #{$0} [API key] [search term]"; exit) unless ARGV.size == 2
key, term = ARGV

results, ads = search(key, term)
puts "#{results.size} results for '#{term}':"
results.each do |result|
  puts result['titleNoFormatting']
  puts " #{result['url']}"
  puts " #{result['content'][0..70]}" unless result['content'].empty?
  puts 
end

unless ads.empty?
  puts "Look at some ads while you're at it:"
  puts '------------------------------------'
  ads.each do |ad|
      puts ad['title']
      puts ad['visibleUrl']
      puts " #{ad['content1']}"
      puts " #{ad['content2']}"
      puts
  end
end

Now, in old episodes of MacGyver, whenever MacGyver built a bomb out of baking soda and masking tape, the writers would change some crucial detail (like change the masking tape to Scotch tape) so that if kids copied MacGyver they wouldn't blow up the house. I've done something similar here. I've removed a crucial line of code from that program, so that people don't just go copying it and running it all over the place.

Why did I do that? Because when it works, that program violates the Google AJAX Search API Terms of Service. "The API is limited to allowing You to host and display Google Search Results on your site." I can use the old SOAP API to write a command-line search tool, but I can't use the new, RESTful API in that kind of application. My users can only access the RESTful API through a specific library (Google's Javascript library), running in a specific way (in their web browsers), for a specific purpose (displaying search results).

Wait a minute... running only in a web browser? Terms of Service? Bootleg scripts that hack the output into something a parser can understand? This REST web service is made available on exactly the same programming-unfriendly terms as the Google website "REST web service"!

Instead of screen-scraping a web page, I'm now screen-scraping a web service. I'm reverse-engineering undocumented URI formats, just like I do when I screen-scrape. So far, there's nothing on Google's end that sniffs my user-agent to make sure the web service only runs in a browser, but you can bet there will be as soon as that becomes a problem for Google.

The "blow to web services" narrative is incorrect. Google did in fact deprecate their SOAP API and expose a RESTful API. A win for REST!

Though incorrect, the "blow to web services" narrative is also correct. Google deprecated their SOAP API, exposed a RESTful API, and then erected a bunch of technological and legalese barriers around any attempt to actually use the RESTful API. You're only allowed to use it through one library in one language in one environment for one purpose. A loss for everyone!

On the level of technological choices, this move is a big improvement. They've gone from SOAP, which has a lot of overhead, to plain old HTTP, which has strictly less. Gone from an RPC style, which doesn't play well with the web, to a RESTful style, which does. This makes an enormous amount of technological sense. From its first day on the web, Google has exposed its dataset through a RESTful interface that gets orders of magnitude more traffic than any "web service" it might expose. In a sense, all they're doing now is unifying the architectures.

When it comes to getting information into the hands of people who can use it, Google has taken a big step backwards. The SOAP interface was serious overkill, but what you did with it was your business (though you could only do it 1000 times a day). The new RESTful interface is a technical improvement, but it's encumbered with restrictions that make it a museum piece. Unless you're writing an Ajax application using Google's library, its true value can only be obtained illicitly. And that's the other reason why I'm not inclined to gloat.

Leonard: I'll do it, but I'll be using scare quotes the whole time! "Scare quotes"!
Sumana: Yes, Leonard, scare quotes. That's what keeps you from being bourgeois.

I laughed and laughed.

Indeed, according to this random weblog entry the artificial beach is right next to a real beach. I guess that makes sense logistically but it takes us into the territory of a Vegas casino whose theme is "ripping off the other casinos on the Strip". Which undermines my point, because that would be the most awesome casino ever.

I finally got my sample web service written for the book, though it's still missing one feature. It's about 16K of code, which I don't know how many pages that translates into.

Clearly this is why my mother always signed registries as "Sigmund Freud".

• 2 chopped up zucchini
• 2 chopped up leeks
• 1 cup milk
• 4 eggs
• shredded hard cheese
• bread crumbs

Preheat oven to 375 degrees. Saute the leeks. Beat together milk, eggs, and cheese. Stack the zucchini in a 9-inch baking pan, mixed with the leeks. Pour the milk mixture on top and put bread crumbs on top of that. Bake for 50 minutes. It turned out great!

The web is full of stories of these arbitrary cutoffs, which I can now confirm in their relevant details. You get an email saying "we've cancelled your account based on classified information" and that's it. You get a pro forma chance to defend yourself, but it's not much good because you don't know what happened. It also doesn't make much difference, because the response always comes back: "we re-ran our classified algorithm on our classified dataset and it gave the same result as before, buh-bye".

Fortunately, I have a friend in the online ad business. Ryan's Project Wonderful mostly serves ads for webcomics right now, which is fine by me, as it means fewer "KEYWORD? Buy it now at creepysite.com" ads.

I bought the Waldrop book as a biography of Licklider (actually I bought it because of the BRIGHT YELLOW COVER that blocked out all other books in the bookstore), but it really strains to fill a whole book with his life and make it interesting. Fortunately the book's method of straining is to bring in all the other characters of American computer science from the 1940s through the 1970s, with a theme of Licklider as the networker between them, and that was very entertaining. Though I admit part of the entertainingness was the celebrity-spotting equivalent of those old Looney Tunes cartoons that feature charicatures of all the Warner Bros. contract stars of the 1940s. Oh look, it's {Peter Lorre,John McCarthy}.

The most interesting new information I found in the book is the history of ARPA, specifically of the Information Processing Techniques Office (Licklider was IPTO head twice). The major theme is the Vietnam-era mission anticreep that slowly pushed ARPA from advanced research projects, to the more circumscribed realm of advanced research projects that we can use on the battlefield within two years without fundamentally changing anything (somewhat parallel to the problems Xerox PARC had). Meanwhile Licklider's trying to scrape together a couple million to fund his crazy "Intergalactic Network" project, and his office is sticking boilerplate defense justifications onto incoming grant proposals so they can get ARPA money.

"All that language about military rationale wasn't in the Stanford version of the proposals," explains Ed Geigenbaum: it was slapped on at the very end by the ARPA funding officers back in Washington. "The only people who ever saw it were the students who would later dig it up under the Freedom of Information Act. Then they'd bring it on campus and say, 'See, McCarthy is working on such and such.' McCarthy would say, 'What do you mean? I never heard of that!'"

But the book also ties together things that I learned about in college with no that they were connected. For instance, ALOHA and Ethernet use the same collision resolution mechanism not because it's an obvious fact about the universe, but because Bob Metcalfe read a paper slamming ALOHA and decided to prove it could work. And multitasking came directly from time-sharing, as cheaper computers made time-sharing less of a selling point.

Fun fact: the book made it sound like ARPA was originally planned to encompass space research, and that NASA was created as a separate agency after a bureaucratic turf war. So the space program could have been part of ARPA.

Understanding the book requires no technical knowledge, but I don't think I would have enjoyed it as much if I didn't already have all these pieces in my mind ready to be connected. The book made me more interested in another item on my wishlist, John Markoff's What the Dormouse Said.

This entry got kind of long, so I'll talk about the Turing book in another entry.

One thing that jumped out of me was, reading the WWII sections, how much of the codebreaking relied not on abstract mathematical ingenuity but in ingenuity applied to exploiting the adversary's mistakes. For instance, the Germans often used a less secure version of Enigma for unclassified things like the weather report, not realizing that if Bletchley cracked the weather report they'd be 80% done cracking the secure cipher for that day. Not surprising that this got largely left out of Cryptonomicon, because fiction is usually more interesting when the adversaries are competent.

This creates a problem analagous to the problem of the anthropic principle in physics. It's not a perfect analogy because the development of the Internet took place within space-time, but I think you can see it. Why did all these unlikely contigencies happen?

I can think of three possible responses. The first is that the contigencies weren't unlikely at all. There is some hidden force of society that makes sure open solutions tend to win. This is the knee-jerk free software response. I like it okay, but I've grown more dissatisfied with it as I've read these books, because there are lots of other situations in the world where the open solutions lost big time. When does the hidden force work?

The second response is that we gravitate to whatever open solutions we can find to solve our problems. The general public can't use something unless it's open. If things had turned out differently we might be speculating on how unlikely it was that we would develop a tradition of collaborative biology or open letter-writing.

The third is to deny that there's anything special about the contigencies. We don't talk about things that don't happen, so if these contigencies hadn't happened, we wouldn't be talking about them. Saying that the Internet and the Web worked out the way they did is just a tautology. This response has its points when applied to the physics anthropic principle, but I find it unsatisfying in this case, because it looks like the same sorts of contigencies turned out the same way twice.

Anyway, just some high-concept rambling to see out the year.

I sustained an actual enthusiasm for clothes shopping for a record two stores, but by this time I was willing to settle for just trying on a topcoat and seeing how I looked in it. This resolve was strengthened by the $200 price tag on the topcoat.

The trying-on was a disaster. I'd thought I could just wear a topcoat on top of whatever and it would be warm and make me look classy, but I looked like a slob who'd stolen a topcoat from somebody. I was like the woman in Cryptonomicon who starts wearing stockings and then discovers that stockings dictate your entire wardrobe and lifestyle.

The topcoat works on Evan because Evan already has a compatible wardrobe, but it's totally out of my demographic. This was a fairly distressing realization. The coat would not transfer its class to me merely through a mercenary monetary transaction. This is the difference between class and mere wealth, and I don't even have the consolation that I became extremely wealthy and then discovered it.

I'm glad I got to spend a lot of time with my mother before she died, and I'm glad we got to see Susanna and John for Thanksgiving. My main 2006 regret is along the same lines: I wish we could have all been together for Christmas; I think we should start making plans for next year and get it set up as early as possible. Someone said that with Mom dead we would need to put in work to see each other, because we'd no longer have a natural gathering place, and I'm starting to see the truth in that.

My current plans for next year are writing-related: the REST book and lots more SF. Let me know if you want to read the noir space opera I'm trying to sell.