Sun Jan 01 2006 02:03 PST: It's 2006, year of the crazy scheme! I'm all set to go with my plan to create a utopian society built on platforms suspended by hot-air balloons. The seafaring utopian society always gets busted up by the Tongan Navy, and the Lagrance point utopian society isn't feasible yet, so I'm taking a middle ground (as it were) to feel out the investor interest in something a little more orbital.

Sun Jan 01 2006 21:46 PST Taste the Rainbow: Noise comes in all colors.

Mon Jan 02 2006 21:48 PST: Hacking Matter, online without cost. So awesome, it's impossible to download it because everyone is already downloading it.

Mon Jan 02 2006 22:10 PST: Also here's a cool-looking thing on Custom Lego parts.

(1) Tue Jan 03 2006 09:03 PST Battle of the Bertrand Russell Anecdotes: I thought I'd do something useful today, so I'm going to put a new fact on the Internet. There's an anecdote about Bertrand Russell which goes like this:

Around the time when Cold War started, Bertrand Russell was giving a lecture on politics in England. Being a leftist in a conservative women's club, he was not received well at all: the ladies came up to him and started attacking him with whatever they could get their hands on. The guard, being an English gentleman, did not want to be rough to the ladies and yet needed to save Russell from them. He said, "But he is a great mathematician!" The ladies ignored him. The guard said again, "But he is a great philosopher!" The ladies ignore him again. In desperation, finally, he said, "But his brother is an earl!" Bert was saved.

Bertrand Russell's autobiography is presumably the original source for this anecdote. Here's the event as described in the autobiography (fake paragraphing inserted for clarity). According to the biography, it actually happened some time prior to 1918.

It was decided at Leeds to attempt to form organizations in the various districts of England and Scotland with a view to promoting workers' and soldiers' councils on the Russian model. In London a meeting for this purpose was held at the Brotherhood Church in Southgate Road. Patriotic newspapers distributed leavelts in all the neighbouring public houses (the district is a very poor one) saying that we were in communication with the Germans and signalled to their aeroplanes as to where to drop bombs. This made us somewhat unpopular in the neighbouthood, and a mob presently besieged the church. ...

The mob burst in led by a few officers; all except the officers were more or less drunk. The fiercest were viragos who used wooden boards full of rusty nails. An attempt was made by the officers to induce the women among us to retire first so that they might deal as they thought fit with the pacifist men, whom they supposed to be cowards. ...

Two of the drunken viragos began to attack me with their boards full of nails. While I was wondering how one defended oneself against this type of attack, one of the ladies among us went up to the police and suggested that they should defend me. The police, however, merely shrugged their shoulders. "But he is an eminent philosopher," said the lady, and the police still shrugged. "But he is famous all over the world as a man of learning," she continued. The police remained unmoved. "But he is the brother of an earl," she finally cried. At this the police rushed to my assistance. They were, however, too late to be of any service, and I owe my life to a young woman whom I did not know, who interposed herself between me and the viragos long enough for me to make my escape.

All the elements are in the original: the chivalry, the attacks with random objects, the comical deference to hierarchy. But the anecdote has undergone some well-placed mutations:

• The mob and the women are merged. This means fewer characters to remember and simpler interactions between them. It also changes the tone of the story from sinister (mob of toughs with nail-boards, police won't do anything) to somewhat comical (mob of elderly garden club ladies pummeling Bertrand Russell with their handbags, police try to politely stop them).
• The roles of the policemen and the women are switched. This also changes the tone: the rough chivalry of the mob ("Ladies, please leave so we can beat up the menfolk!") is replaced by the more gallant chivalry of the policemen. The only loss is the final bit of the anecdote, where a woman exploits the mob's chivalry to help Russell escape.

Altogether, I must say the later version is more memetically fit. Therefore, that's what actually happened. Another blow for the old primary sources, Jack. How long can they continue to be given their current respect when mutated versions are so much better?

Wed Jan 04 2006 20:03 PST: When I was young and bored, or just young, I would while away the minutes by writing stories using only words that could be spelled using the chemical abbreviations for elements. Now that there's a comprehensive list of such words, I must sadly shake my head and pronounce the medium dead. When anybody can simply look up which words meet this arbitrary criteria, how can art survive?

(1) Thu Jan 05 2006 21:34 PST Bill the Galactic Hero: Funny, well detailed over-the-top '60s satire from Harry Harrison, who wrote the story that became Soylent Green. Has the same thermodynamically bizarre attitude towards overconsumption as Soylent Green. I was (SPOILER) fairly disappointed that Bill never actually became a Galactic Hero, but that was probably intentional. I thought it captured a Paranoia-type mood much better than the Paranoia novelization.

Fri Jan 06 2006 22:48 PST: We went to Mountain View today to say goodbye to Andrew and Claudia. Lots of saying goodbye recently. I will miss my ability to team up with Andrew to spout forth nonstop jokes, but Andy Schile lives in New York and I have the same ability with him. Andy rarely shows up in NYCB until now because I haven't seen him since 1998, but you'll probably see a lot more of him in the coming decades. Anyway, here's a cool link I stole from Kevan about dead whales as oases in the deep ocean.

Tomorrow: more saying goodbye to people. Goodbye!

Sat Jan 07 2006 18:31 PST: Today: loaded stuff up. Strained back slightly. Can't think of anything else to write. Ideas?

(3) Sat Jan 07 2006 20:22 PST: Susan McCarthy came over and we talked a bit. Her agent says that books about the sea are really hot right now, but she doesn't have any ideas for such books. I came up with a couple ideas but I guess I should write them instead of her. I bet you could do a project on whale falls (q.v.) and write a book about that. Also I think a book about the deep sea would be a real hot seller.

I don't know much about the deep sea so I was thinking of writing a book that was just full of lies. Some people might call this "fiction" but these would be real actual lies. It would be okay because they would be funny lies, like the coffinfish interview. Actually this is starting to sound like an interesting book project, though somewhat different from the book ideas I brainstormed with Michael yesterday.

(5) Sun Jan 08 2006 22:07 PST Thought Experiment: Which single Unicode character would convey the most information, say, on a sign?

Tue Jan 10 2006 07:50 PST Game Title: Manahpoly

(2) Wed Jan 11 2006 08:48 PST Durian Milkshake: We are moved out. We're in Bakersfield now. I drove us down last night rather than spend another dismal night in our empty apartment. Sumana has details if you want them. I would rather talk about the durian milkshake.

Yesterday I went with Dan, Beril, and Kevin to the Vietnamese restaurant near CollabNet. They (the restaurant) recently changed their name and rearranged their menu a little, so now it features clip art and pixellated pictures of food. It also features milkshakes that star the fruits of southeast Asia, including a durian milkshake. I ordered a durian milkshake.

The durian milkshake didn't smell bad (probably made from frozen durian) and it didn't taste good (turns out I don't like durian), so it was an all-around disappointment. But it's a cheap and harmless way to claim you've eaten durian.

Wed Jan 11 2006 09:04 PST: Bread author Peter Reinhart has a weblog, and on that weblog he has a post, looking for recipe testers for his the whole grains book he's working on.

Wed Jan 11 2006 18:53 PST: Do you crave more than 100 amazing facts about Jake Berendes? Now there's hope!

(3) Thu Jan 12 2006 17:56 PST Phrases as Slogans: I was thinking of perfectly normal English phrases that could be used as advertising slogans if said in the ad voiceover tone of voice. The only one I have thought of so far is "connect to the wireless network", but there are probably a lot of them. Unlike last year's quest, which has yet to reveal a single pair of jokes with the same punchline.

Fri Jan 13 2006 21:56 PST robot gone wild: Today I learned from a robotfindskitten user that there appears to be a bug in Python's curses interface to Mac OS X. It seems you can't hide the cursor on a Mac terminal with curs_set. Calling it returns ERR, and Python's curses.wrapper can't handle the ERR. So you crash. Has anyone else experienced this?

Also, robot finds panties

(1) Fri Jan 13 2006 22:07 PST: Here's a bunch of public domain movies you can download with BitTorrent. Speaking of which, I was looking around to see if you could download all of the Project Gutenberg files with BitTorrent. This site alleges to be the official tracker, and an 88 gigabyte download sounds within the right order of magnitude.

Sat Jan 14 2006 18:49 PST Spam Tunes: Over at Adam's house with Kris and Kim. We are about to go to the airport. We whiled away the afternoon writing music for spams. We finished two with Adam playing guitar, me singing, and everyone else doing occasionally appropriate backup vocals.

• Hard Like Steel Pk5 (Not Classy Mix)
• Viagra Pro has become available since 1998 and

I wrote one more which I'll re-record once I get set up in New York. Goodbye, California.

(1) Sun Jan 15 2006 10:02: Okay, I live in New York now. But first, sleep.

(5) Sun Jan 15 2006 18:42 California Burger: We spent most of the day in Astoria looking at apartments. We ate dinner at a diner which had on its menu a "California Burger". Now, in California a "California X" has avocado on it. But there was to be no avocado on this California Burger. So what made it Californian? Two things I could see:

• It came with lettuce and tomato, instead of that costing extra.
• It came with "raw onions" (ie. onions) instead of onion rings. Onion rings! Onion rings are a side dish, not a topping!

So clearly we have a thing like the British use of "salad" going on here. This diner's version of "California Burger" is more accurate, but less delicious than California's. Hamburgers in California don't have avocado on them by default, but they do have lettuce and tomato and onions by default. Not at this diner. This only confirms my suspicions that fresh vegetables are as rare spices here in the Northeast.

(3) Mon Jan 16 2006 20:31 A Survey of Mutant Miceology, and Other Topics Biological: Today was more looking at interesting things and less at apartments. We met up with Andy Schile! I hadn't seen him for almost ten years, but there he was, or at least a convincing animatronic simalcrum. Together we went to the Natural History Museum. We saw the special exhibit on the life of Darwin, which was cool: they had a bunch of skull casts for all the known hominids, arranged in an evolutionary tree. Actually the whole museum was arranged around evolutionary trees. One exhibit took up the entire fourth floor with the Vertebrates' Greatest Hits, organized cladistically.

We walked through Central Park to a Japanese restaurant that Andy likes. While we ate, he brought us up to speed on his research into the creation of mutant mice. He creates a mutant mouse by taking a mouse embryo and replacing the DNA inside some of its stem cells. The mouse develops with two sets of DNA (it's a "chimeric" mouse), and you check the color of its coat.

The mouse embryo would naturally have, say, a white coat, and the new DNA is for a mouse with a black coat. If the chimeric mouse has a dark coat then it's a good bet that most of its cells came from the modified stem cells. Specifically, its sex cells probably came from the modified stem cells. This means you can breed it with another chimeric mouse and get a mutant mouse.

A mutant mouse comes from a modified sperm and a modified egg, so it's not a chimera. It's what you would have gotten if all the cells of the original mouse had come from modified stem cells.

Here is the part I'm not as clear on. Andy is trying to breed mutant mice which have a certain gene truncated. The truncated gene will not be able to produce a certain protein. The protein is in charge of attaching a chemical tag to cells that need to be garbage-collected (or maybe the protein is, in fact, that chemical tag; I'm not sure). So if Andy's mad plan succeeds, these mutant mice will not be able to engage in cell death.

At this point in the pop science article I would talk about the applications of this revolutionary new technique for "the fight against aging" and leave it at that. However I have two other things to tell you. The first is that these mice will not be immortal just because they can't kill their own cells. They will probably get some sort of anti-cancer and die (I speculated as such and Andy didn't correct me). The second thing is that this is not a really new ability. You can already create a mouse whose cells don't die by giving the mouse a new gene, one copied in from a virus. I don't remember what the virus gene does, but obviously it works differently from Andy's technique; I think it might prevent cells from accepting the garbage-collection tag. Andy's technique has the same end result, but it doesn't require genes from other organisms. Once he gets some mice whose cells behave in an interesting way (or he fails in an interesting way), he can write his dissertation and he's Dr. Andy.

I wrote this mainly as a writing excercise, but it's a pretty interesting set of hacks and I never really understood how genetic engineering works, so maybe you'll find this interesting too. Rewiring mice stem cells sounds like a pretty dull job to me, but Andy likes it. I have more interesting stuff to write from our conversation, such as the operations you can perform on a gene or genome, and when you can copy a gene from one species to another. I will write this other entry if people are interested, or if I am bored.

In the museum I told Andy lots of trivia about prehistoric animals and evolution which I'd think he would know, as a biology major, but he was appreciative of the trivia. Maybe he was just humoring me. For instance, they had an Irish elk skeleton. I told him how biologists used to think the Irish elk demonstrated that evolution could get stuck in a rut, with the male Irish elk getting larger and larger antlers long after it ceased being any kind of advantage. This continued until the males couldn't lift their heads anymore, leading to extinction.

This page yields some more Irish elk trivia. For instance, the antlers of the Irish elk are not particularly large for its size. They just look abnormally large because the animal is so big. And the Irish elk also figured in a more prominent scientific argument: whether or not animals could go extinct at all.

(4) Tue Jan 17 2006 21:18 Blood Orange Juice: We have an apartment now, though we probably won't actually move in until the end of the week. It's in a nice part of Astoria which we wandered around today to get the lay of the land. There's a really crowded grocery store, which will probably be our main grocery store since it's right near the subway stop, but a couple blocks off to the side there's a nearly deserted grocery store specializing in Eastern European stuff. I took a shine to this grocery store since it wasn't crowded with people, and peeked at its goods. When I saw the Tetra-Pak containers of juice I had to scour them for the long-lost treasure of the past: blood orange juice! They have it! It's $2 a quart, much more expensive than the stuff Susanna brought back from Romania if you ignore the cost of going to Romania to get it; but it's delicious (not as good as the Romanian stuff; maybe I need to try another brand).

(2) Wed Jan 18 2006 17:36: People have been complaining that Rubyful Soup is too slow. Bah! What do they know? Except they're right; it is too slow. Today I fought back the sickness that has consumed my corporeal form, and stayed in the hotel room all day and profiled the bugger (bonus: I can write a profiling recipe for the cookbook at the same time). I sped it up by a factor of three, which I think is about as good as I can do without going down a level and optimizing the SGML parser. I don't have the wherewithall to rebuild the gem since the gemspec is on my other computer, but you'll get it soon enough.

(4) Thu Jan 19 2006 21:20: Still sick. Jake, I don't think I can make your show tomorrow because I need to unload the shipping container tomorrow and I may not survive. How long are you in town? Can we meet some other way?

(1) Sat Jan 21 2006 12:28 Hello: Still alive. We moved into our apartment yesterday, which was exhausting but it looks okay now. We won't have Internet for the next few days, so I'm just going to be quietly writing book recipes and exploring Astoria, which looks more and more like a good neighborhood for me to live in. I keep typing but nothing interesting comes to mind, so I'm going to run Crummy through the Eater of Meaning and look for fun phrases. "Hi, I'm Leopard." Okay, that cheered me up. "Rubles Souls is tool slowest."

(1) Mon Jan 23 2006 11:19: Yesterday I spent some quality time with Jake Berendes; the first time we'd met or talked on the phone in the nine years we've known each other. We ate some pitas, walked around Astoria, and because I don't have my guitars yet the closest thing we could do to a jam session was to make up drum loops with Hydrogen. When I was with Jake I felt the laws of comedy compelling me to be Oliver Hardy, because Jake is such an outlier on the Stan Laurel side of the scale.

We are Internet squatters, taking a little bit of bandwidth from our neighbor NETGEAR. I am back to work on the Cookbook, which is now available! That is, the embarassing rough draft we're still working on is now available, thanks to O'Reilly's perfidious new "Rough Cuts" program. You can read my writing this very minute and then complain about it directly to me. There's one really awful recipe which I was hoping to replace with a better one before this happened, but I didn't get to it in time. Anyway, it'll improve the final quality of the book, though I'm a little distressed by the estimated publication date of September. I was hoping it would be out around July.

(9) Wed Jan 25 2006 20:06 Chat 'n' Chew: Kevin said that when in New York I should check out the restaurant Chat 'n' Chew, for what he calls "the correct burger experience". Many hours have I suffered listening to Mr. Maples expound his crackpot hamburger theories. According to him the correct burger experience is just a plain burger with nothing fancy. Well, the Chat 'n' Chew serves trailer trash food (one of their specials is called the "TV Dinner"), and they've got a non-fancy hamburger. The problem with Kevin's theory is that a restaurant that doesn't make fancy hamburgers may also cut corners when it comes to the non-fancy hamburgers. What you really want is a basic hamburger from a place that also sells fancy hamburgers.

There was good ambience and lots of good-looking stuff in the restaurant, like macaroni and cheese and grilled sandwiches, but the hamburger was charred and, though it came with some vegetable fixings they were not very healthy-looking. There was no mustard or mayonaise to counteract the dryness of the patty. The french fries were okay but not great. Sumana's meal (green bean casserole and grilled sweet potato; yes, sweet potato cut in half and put on the grill) was far superior.

And what is the customer supposed to do? You can't complain that you got lowbrow food at a self-consciously lowbrow restaurant. You're trapped. Well played, Mr. Maples, well played. You'll be the death of me yet.

The CnC seems expensive to me (that burger cost $9) but it's probably average for Manhattan. The other stuff on the menu looked good, especially the wide variety of trashy desserts, so I'll probably go again eventually. It's near Union Square.

Sat Jan 28 2006 19:53 Tristram Shandy: We are having fun going to Manhattan for some purpose or other and getting in some wandering around time. I am discovering that some places are more fun to wander around than others; some have interesting things like Japanese pastry shops and some have nothing but expensive trattorias.

Today we came fr'over the short sea to see Tristram Shandy, which started out really well and then it got all Adaptiony which I could have done without. All extant material on this film (including the film itself) alleges that the source material is "unfilmable", but they were doing a really good job of filming it until they decided to stop. I'm going to be charitable and say they just didn't have enough money to film the whole thing as a period piece. It reminded me that I need to read the book, though. It's kind of sad how public domain books I download don't actually go into my book queue because of the physical stacks of non-public-domain books taking up space in my house and bugging me to read them.

I really like the idea of the Japanese pastry shop, but the thing I tried (a sherbet jelly made of grapes) just tastes like an Otter Pop. Here's some of the other stuff they had.

Sun Jan 29 2006 16:44 Sunday Fun: The meeting-people juggernaut rolls on as today we hung out with Camille! She showed us around Brooklyn and we ate brunch at a Southernish restaurant called Enid's, a "Williamsburg hipster HQ" according to a website I just found while Googling for an epithet to give it. There were a lot of hipster-looking folk there, including one Camille had had an unpleasant encounter with earlier. She was a little worried but not much because "it's not very hipster to exact revenge."

Then we went to another restaurant for dessert but the portions were really small. Camille vows that next time we will do things old-fashioned Brooklyn diner style so we can have huge pieces of pie.

Camille is somehow representing or works to book gigs for musicians. She was trying to expand her business and branch out, but then she decided to just go for a niche and own it. Her niche is "eclectic" musicians (eclectic as a group, or, I suppose, individually), so she'll find gigs for you if your act is just a guy with a zither and a girl with a triangle. Very vaudeville-like.

We also went over to see this exhibition of black-box analog synthesizers. Most of them did not respond well to my twiddlings, but I got a good sound out of one of them: a sort of funky metronome beat with Star Trek: TOS ship effects on top of it. People I thought about telling about this exhibit in a gloating manner: Jake, Seth, Brian Behlendorf. Then I decided that doing that would be equivalent to gloating that I lived in New York, rather than that I had gotten a chunk of culture into my otherwise philistine system, which isn't something I should gloat about.

(3) Tue Jan 31 2006 12:42: Ah, sweet Internet. So full of lies. What would I do without you?

(2) Tue Jan 31 2006 21:24 Food for Kevin: Bacon Snack Bars. Made from bacon, corn syrup, and Tang. "YOU MAY ALSO ENJOY: The Original Munch Mix." If you're insane.

Wed Feb 01 2006 21:25: At last, the not-incredibly-slow version of Rubyful Soup is released! Just gem install rubyful_soup once the list of gems refreshes on RubyForge. Also includes a new way of programming the parser for better performance, which I need to document. But first I need to make the documentation page not look horribly ugly.

Beautiful Soup needs an update, too, which will come probably this weekend.

Wed Feb 01 2006 21:39 If Only We Could Delay Starting The Work: Sumana sent me Why I Hate Frameworks

Wed Feb 01 2006 22:09 The Lack of Sunshine State: Xena: It's Bigger Than Pluto. Not a great tourism slogan.

Thu Feb 02 2006 16:53: More DS9 art from Brandon Bird: Of Bajor.

Thu Feb 02 2006 21:35: Some links found while going through a month's worth of RSS feeds:

• LEGO used to have 1x1 bricks with letters on them, so you could spell things out in bricks. I have a lot of the old bricks mentioned on that page (including a sign that says ESSO), but I never heard of those. A font of Lego! LEGO. This explains all those really short mills and really long amphitheatres you see in old sets.
• This guy mocks bad MAME cabinets. I think this one actually looks really cool, but not as a MAME cabinet.

(1) Fri Feb 03 2006 21:07 Ambiguously-Titled Project Gutenberg Texts:

• "Throwing-sticks in the National Museum"
• An Introduction to the Study of Browning

Neither as interesting as you'd think on first glance.

Also of interest: The New York Subway: Its Construction and Equipment. And from the classics section I recommend My Bondage and my Freedom, which among other things includes a novel reason for not revealing spoilers:

Keen is the scent of the slaveholder; like the fangs of the rattlesnake, his malice retains its poison long; and, although it is now nearly seventeen years since I made my escape, it is well to be careful, in dealing with the circumstances relating to it.

Fri Feb 03 2006 23:36 Shoot the Messenger: Today instead of writing the book I wrote rules for a game (don't worry, Lyle is okay. I mean, don't worry, I got my book quota for the day done eventually). It's called Shoot The Messenger and it still needs some work, but it's got an interesting mechanic (simulating a language barrier using the inaccuracies introduced by online translation services) and a good source of paranoia. A lot of the games the Dispatch people come up with have a theme of paranoia. I wonder why that is. I wonder who else wonders why that is.

Fri Feb 03 2006 23:43: I was pretty excited about Polynomial Hierarchy Collapse, but then the PDF loaded and it was just une histoire drôle like the kind I used to write for Segfault. Pretty drôle, though.

(2) Sat Feb 04 2006 21:44: Today I wrote a poem:

Shake and shake the ketchup bottle
None will come, then Quetzlcoatl

Sat Feb 04 2006 23:46 The Games They Are A'Roundup: Now with babbling about game design problems and death!

• Miller's Quest is a game you can just let run and it does the dull repetitive things for you. If you are a Windows person you can play its ancestor, Progress Quest. Written by a guy who I think used to write Segfault stories, though I can't prove this. Basically a non-bad Segfault story in game form.

  This game is written in Ruby, and it has a bizarre attitude towards generating colored text, but I'm probably the only person who will notice. I have been thinking about writing an MMORPG in which all the players are scripted robots, but the idea keeps running into problems. I've also been thinking about mentioning in the Ruby Cookbook the bizarre curses-less colored text hack used in this program, but I don't think I want to encourage this kind of behavior.

• Enemy Lines wins many points for extreme literalism (your enemies are lines) and its castlezzt.net-type presentation. It foolishly squanders them all by being kind of like Missile Command. Bah!

  The author has also put out three other games with the same name, only some of which are like Missile Command. The best-looking one is #4, which has a great cyberpunk Zaxxon look.

• You'd think a game whose file is called "todoom" would be "To-Doom", an action-packed slaughterfest organized around the principles of the Getting Things Done productivity philosophy. But it's actually a Roguelike game called Tower of Doom. Pretty good, not great. All your characters are various lowlife types (because the Tower of Doom is a punishment). Peculiarities: no enemy spawning and complete knowledge of all enemies on the level.

  Hmm, since I downloaded this game, the project seems to have undergone a freakish metamorphosis! It's now called Warp Rogue and is much different. That's what I get for writing reviews of ancient copies of games found on my hard drive while I have no Internet access. Developing...

• Zoid's Quest is potentially a really good side-scroller game like Jill of the Jungle. It has an excellent, original fighting mechanic, and good first-level music that reminds me of NES games like Faxanadu. But this game makes many of the tiny mistakes or misalignments that ruin so many NES-like open source games. I should make a big list of these and do a cookbook on how to avoid them, so here are a couple:
  • Jerky or disproportionate character movement. Zoid's Quest has pretty smooth character movement, but as in almost all of these platform games, the speed of the character is much slower than the character dimensions would indicate. A good rule of thumb is that I should not be continually jumping from place to place because it's faster than running.
  • Ladder behavior. When you jump and grab onto a ladder, your horizontal motion should stop or be absorbed by the ladder. The ladder is not a strip on the wall that lets you defy gravity; it's actively supporting you.

    The ladder is not a hole. It's bolted to its platform; it doesn't bisect it. You should be able to walk on top of a ladder square.

    If you stop moving on a ladder you should stay clinging to the ladder, rather than falling off of it.

  • Jumping and gravity behavior. First, you should be able to jump more than one square in height. This overlaps somewhat with disproportionate character movement. Maybe you think it's cliche that your character should be able to jump as high as Mario, but Mario jumps that high for a reason: jumping games aren't fun when you can't friggin' jump.

    Before dropping down below the edge of the screen, you should have some way of seeing whether or not a drop is infinite. You either need to provide some kind of "look up/down" facility, or design the level so that the player has a chance to experience the lower levels before the higher, or design the level so that it all fits on the screen. (This is actually in the Zoid's Quest TODO, as a magic spell.)

  • Enemies. You must have more than one kind of enemy, and each type of enemy must have different behavior, and in particular the first level must have more than one kind of enemy. I know your game is in development, but just make another kind of enemy for us.
  • Okay, that's enough for now. You get the idea.
• Holotz's Castle is the first game to ask, "What the hell are doing all these keys here?" But it never answers it. Instead, it disguises itself as two games with identical mechanics. In one, you must gather keys to escape the level. In the other, you must gather gems to... escape the level. After Zoid's Quest, the movement in this game felt perfect, and your characters gradually become faster and able to jump higher. Nice cut scenes use the game engine and have a somewhat juvenile Nintendo of America-approved sense of humor. The only problems: one touch of an enemy's hand kills you, and it's possible to get stuck in the corner of a block. And there's a big ol' cut scene cliffhanger. (though I should check a newer version; maybe it's done a Tower of Doom).

  "How do you know what I think?"

  "I KNOW EVERYTHING!"

• The other day I was laughing about the classic Rounded-up game Apricots. Specifically, I was recalling the limerick I'd written about the hilarious deaths which are impossible to avoid in that game. Sumana said I shouldn't talk that way online about death, because it might be read by someone who's thinking of committing suicide and it might give them the idea that suicide is somehow fun or that I approve of death in real life. Well, I'm here to tell you that suicide is not fun and that death is a bunch of crap. And there's no better game to tell you this than N.

  N is a game with a wonderful physics model and a million deaths. Your character is a greedy ninja with a lust for gold--and not being killed! Unfortunately you get killed a lot, usually in hilarious ways that top even the deaths in Apricots. In Apricots you might crash your plane into a tiny island in the middle of a lake. Explosion. In N you might do a ninja leap right into an active death ray. Your body plummets the length of the screen and splats on the ground. Then your wireframe corpse bounces around for a bit and hits a mine, which blows it into its little wireframe parts. The parts themselves will bounce around and get into all sorts of trouble: setting off more mines, being shot around the game board by trampolines, being zapped by zap-robots, and so on. It's the most morbid game with the simplest graphics I've ever seen, and I've seen at least five.

  But it's not fun to die, because that means you've got to start the whole frustrating Lode Runner-esque level over again instead of going to the next cool level. Which, under Nietzsche's theory of eternal recurrence, is basically the same thing that happens in real life. So I guess I'm saying it inculcates a healthier attitude towards your own, actual death than Apricots.

  It's also one of the best games I've ever seen on my Game Roundup scale of game-liking: "a flexible set of techniques to use towards your goals, and lots of random variation within well-defined parameters." Except it's not random here; varity here comes from a huge number of well-designed levels. There are also many user-created levels that simply use the player's (non-dead) body as a pinball in the machine of the board, bouncing you around narrowly avoiding rockets and zap-bots and all sorts of horrible fates.

  Oh, I should mention that it's composed of Flash and it's not open source. Good game though.

(4) Sun Feb 05 2006 23:40 Home Improvement Questions: For my mother to answer.

1. I know you're not supposed to ever ever cut paper with the sewing scissors. But can you cut cloth with scissors designed for cutting paper, or kitchen shears? I don't want to have to buy sewing scissors just to make the curtains.
2. What are the names of the parts of the hardware that I need to hang pots from the ceiling? O-hooks and S-hooks?

Sun Feb 05 2006 23:53: More items from my attempted RSS feed cleanup, still in progress. Enjoy this seemingly endless chain of NYCB entries full of lists.

• The creation of different types of Lego minifig heads has angered the Elder Gods, and Cthulu Attack Lego is the result. I like the fake Lego catalog storyline the creator added.
• For a long time I've wondered how to see the worst-selling items on Amazon. I don't know why I never tried simply hacking the URL. I saw a cool-sounding book somewhere in the depths of low sales: "Broadcasting From Space".
• At last, the ultimate mola mola portal. Did you know that the Spanish name for the mola is "pez luna"? Moonfish! That always slays 'em down at the comparative linguistic zoology lab. Unfortunately, the field is so small that, having slain the lab inhabitants, there's no one left who thinks it's funny.
• Fractal Images of Formal Systems has a fractal representation of tic-tac-toe which you can play.

Mon Feb 06 2006 00:12 The Thin Man: The basic cable package here includes 2 million channels, give or take a million. These include a couple that we actually like, like TCM and a channel that only shows science documentaries, whose mascot should be Professor Science. So we are Tivoing old movies which we will eventually watch. First up was The Thin Man, which was really funny. Man, the main characters were constantly drinking! Even by the standards of their time, Nick and Nora were such lushes that TCM felt the need to follow the movie with an anti-drunk-driving short as a chaser.

In said short, the Highway Patrol officer of "one of our western states" solemnly told us that, even though you may not feel anything, scienticians have proven that a mere four cocktails can affect your reflexes.

The mystery in the movie wasn't that interesting but we kept laughing and laughing at the character studies. We loved the dysfunctional Arrested Development-type family, including Lucille; a classic Buster+George Michael+Tobias; and Cesar Romero, probably my favorite movie-grade-spanning actor, as George Sr.+G.O.B. As Joel Hodgson once said, "Render unto Cesar what is Cesar's!" Also, there was a cop who gave a good impression of what Clark Kent would be like if he wasn't also Superman.

(4) Mon Feb 06 2006 18:30 Cool Charts: Sumana's got this big wall chart of human civilization ("Hammond Ultimate Civilization Chart") which we hung up in the living room and it's pretty nice but it's got a couple big problems. One is a big blob of something I don't know what it is or want to touch stuck to it. The other problem is that even though the copyright date is 1989, it was obviously made in the 1970s:

According to the chart, civilization stopped in the 1970s. Which it did, but it started back up again a bit later, and I'd like a chart that reflects that. Also, it's got some fossils of errors that died out in the 1970s. One of the first items of civilization is "Negroes move into Africa from Asia". Uh, no. It also looks like the 1970s, and in general I try to banish the 1970s from my house.

So I'm wondering what kind of cool wall charts you people know of. I've found various sites for updated charts, but I don't know which one's better, or what else is out there. Basically I want something for people to stare at if they're visiting us and they become incredibly bored, because sometimes that happens to me at other peoples' houses and things to stare at are much appreciated. The other main thing I want to get is the National Geographic map of the universe.

(1) Tue Feb 07 2006 23:32: Went to the NY Tech Meetup, which was fun. I found some reviewers for the book and met Max, who'd sent me a Beautiful Soup patch (which I haven't done anything with yet, but it's a good patch).

Reading John Henry Days, which is hilarious. "One Eye had been blinded in a tragic ironic quotes accident a few years before." I wish I could write like that. Like David Foster Wallace, but with more jokes instead of just wit.

(2) Tue Feb 07 2006 23:39 Coolest Thing Ever: Dinosaurs insta-buried and fossilized while fighting.

Wed Feb 08 2006 22:28: Andy came over today with girlfriend Trudy. I'm surprised at how much more social activity we're doing now that we live away from most of our friends. We had a good time w/Andy and Trudy. I made lasagna which was great, and chocolate souffles which were pretty good.

I discovered that if you're making lasagna noodles you should roll them out as you build the lasagna. I rolled out all my noodles real thin, and folded them up for compact storage while I got the rest of the stuff ready. Then I discovered that my noodles had fused back together into thick skinny noodles, and I had to roll them out again. Great noodles though.

(2) Thu Feb 09 2006 13:50: The best part about writing a book is coming up with totally ridiculous examples for the code samples. Today I reviewed the four elements ("Fire: Perhaps the most overrated element."), and just now I wrote the phrase "Oh, MySQL, you're the only one who really understands me.". Probably some of them will be cut, but I hope not; I think they make good mneumonic devices. You can remember "Oh yeah, that was the recipe with Brunching Shuttlecocks-style reviews of the elements," and search for "water".

(1) Thu Feb 09 2006 22:46: If NaNoWriMo is not extreme enough for you, try The 3-Day Novel Contest.

(1) Fri Feb 10 2006 00:05 Anacrusish #4: I missed the deadline for The Ultimate Genre Challenge because it was buried in my RSS feeds, but I was intrigued by its premise: to write a coherent story containing dragons, spaceships, and vampires. Here is my belated entry, which contains one of each. It scores relatively few points (six to eight, depending), but I think it's a pretty good story for 100 words. Because that's Anacrusis-length, I give it the Anacrusish title of

Malcolm

"I'll have you know I find this whole thing very distasteful," says Charmed. Slaved to the ship's fins, his wings carve microgravity.

"What part of 'thrall' don't you understand?" says Malcolm. "Ten minutes before last time. Do it n-- as soon as is safe." His voice commands; he can no longer seduce.

Charmed visualizes a familiar point in spacetime. He yawns wide and breathes neutrinos.

Just before last time, Malcolm wails as he dyes Malcolm's lips red. "Diffbuhphting," says Malcolm, but he cannot allow the plague to spread. Each time he does this, his new shape feels a little older.

Fri Feb 10 2006 08:35 Ruby Database Access Technique Fight: Does anyone here think that Ruby's DBI module is abandonware and should be shunned in favor of database-specific Ruby bindings? I have heard this opinion and it seems bizarre to me. Even if DBI is abandonware, it's stable and it seems to work fine. Plus it's got features like prepared statements, which are otherwise a big database-specific pain. So are we going to be passé if we write recipes for the Cookbook using ActiveRecord and DBI instead of ActiveRecord and database-specific Ruby bindings?

(4) Fri Feb 10 2006 23:30: I'm having a lot of fun playing with this game that uses Super Mario Bros. 3 as a game engine. It really messes with your preconceptions of what a Mario game is like. Reminds me of the first ZZT game you play that uses the weirder features of the ZZT Toolkit or whatever that was called. Highly recommended.

(1) Sun Feb 12 2006 23:13 Snow: We finally got the snow the weird TV news has been predicting since we got here (one of the TV news shows has a piece of technology called the "Doppler 2 Million", which sounds like a zombie army). It's a lot of snow, but we're fine and from peoples' reactions this is about as bad as it gets. Tonight we went out in the parking lot behind the building and played in the snow, but it was insufficiently clumpy to make good playing material.

Snow is really interesting; it's like a visualization of statistical distributions.

Mon Feb 13 2006 00:09: Buncha game clones. I dunno, I got nothing. Once again I've stayed up too late.

(1) Mon Feb 13 2006 23:31: Work work work, all day. Wrote three recipes, edited three more. 90% of the recipes are now done. This book is starting to get on my nerves. Oh well, time to sleep.

(4) Tue Feb 14 2006 10:31: All these procedural dramas; why no object-oriented dramas?

(1) Tue Feb 14 2006 22:54 Naptime: Good advice.

Wed Feb 15 2006 22:33 Cargo Cults: I read an article once that the Melenasian cargo cults had died out but -- this was the interesting part -- recently resurrected as a tourism thing. So that people were putting on reenactments of cargo cult rituals because doing so made people come to their islands and spend money.

But according to this Smithsonian article, the John Frum cult is still active and is gradually turning into a real religion (Leonard's heuristic for cults turning into real religions: the eschatalogical event predicted by the cult keeps not happening, so it gets moved off into the indefinite future or the realm of metaphor). And I can't find the other article, so I guess the Smithsonian one wins. Probably there's just a few people doing cargo cult demonstrations on the side. It doesn't seem like they could support the whole tourist economy.

Wed Feb 15 2006 23:36: I thought I had me a breakthrough but apparently the only true breakthrough is spray-on solar-power cells.

Wed Feb 15 2006 23:44 Link cleanup:

• Floating rock.
• Lego Churches #2: Church of the Flying Spaghetti Monster (see #1)
• Bacon Activity for Kevin.

Bonus quote:

"'Make money in your spare time?'"
"More like 'Make money by working in what would otherwise be your spare time.'"

Thu Feb 16 2006 21:59 Dude, The Day Was So Wasted: I intended to start work on the book intro today, but instead I wasted most of the day watching videos of speed runs for old video games. The most amazing was one for a SMB1 hack called Air, a pathologically difficult game that you can only win by ruthless exploitation of bugs in the engine. Beautiful. The author has a bunch of other Mario hacks I haven't looked at. Maybe I can use one of them instead of the book intro.

(2) Fri Feb 17 2006 22:31 Where Are My Pants? An Editorial: That's the headline of a news story in the recipe I just finished. Kind of a callback to "Microsoft Stole My Pants", which I think was my first Segfault story. I just find it really funny when somebody uses the bully pulpit to complain that their pants are missing.

That's not what I wanted to talk to you about. I wanted to talk to you about Elvis sandwiches. I'll do that in another entry.

Fri Feb 17 2006 22:51 Elvis Sandwiches: There's a restaurant near NYU that only sells different kinds of peanut butter sandwiches. Well, they sell other sandwiches, but they're a kind of sop thrown to someone who has been dragged there by peanut butter obsessed people. I went there last week for lunch with Sumana and Adam.

Generally I don't order stuff at restaurants when I can make a superior version myself, and a peanut butter sandwich is well within my reach. So I decided to order "The Elvis", which I could make at home but never actually had. This is a peanut butter sandwich with bananas, honey, and bacon, which gets fried.

It was okay. The experience, and the recent visit of the Elvis sandwich to the Minutillo household, got me thinking: why couldn't I make my own Elvis sandwich? I decided to try it.

It was good; better than the restaurant one, certainly. While making it I figured out a couple of things about the Elvis sandwich. Longtime readers (that is, since December) may remember Eggnog Transitive Closure, where it was revealed that eggnog, ice cream, and French toast have the same ingredients. Well, an Elvis sandwich is basically French toast minus eggnog.

Peanut butter, bacon, bananas, and honey are all things that could reasonably go into the middle of a piece of french toast before you dip it in the eggnog and fry it. Load up a piece of french toast with all of those things, and don't dip it in the eggnog, and you've got an Elvis sandwich. The chain of foods continues apace! Chain chain chain, chain of foods.

Fri Feb 17 2006 23:27: The Ruby Cookbook is now available for pre-order on Amazon. If I were Neal Stephenson this would get on Slashdot. But, as I think I've noted before, if I were Neal Stephenson a lot of things would be different.

(2) Sat Feb 18 2006 22:45: Kind of worked on my story tonight but I was really just typing stuff up from notebooks. Does that count?

Anyway, here's a Windows game that looks pretty cool.

Sun Feb 19 2006 22:48: We were over in Jackson Heights doing a Craigslist adventure to fill in the missing spaces in our furnishings (sofa, filing cabinet, etc.) and we ate at the Afghan Kebab House IV. You're saying to yourself "I never saw the first three Afghan Kebab House", and that's because those are somewhere else. I'd never had Afghan food before; except for the bread (which was great) I would have believed it to be Indian food. It was really good ,though that specific restaurant was kind of pricey.

(1) Mon Feb 20 2006 11:08: For a while I've been looking for a word or phrase that has a very common incorrect variant. So common an incorrect variant that you can't use the naive "Google them both and see which one gets more hits" trick to figure out which one's right. But not so common that descriptivists would argue that the "incorrect" one is actually right because that's how people use it. Anyway, I finally found one: Aye, there's the rub gets 127k hits, and Ay, there's the rub gets 150k.

Mon Feb 20 2006 21:13 Meaningless Milestone #2: Or #3, or #4, depending on just how meaningless you want 'em. Today we passed 300 Cookbook recipes in fact, we way passed it: we have 305 now.

Tue Feb 21 2006 12:12 QOTD: "That guy talks in tags!"

Tue Feb 21 2006 23:40 Occupation by more than 868 persons unlawful: Today I went to lunch with Josh Lerner, Rich Mintz, et al. at Jing Fong, a dim sum restaurant. This was the largest restauraunt I have ever been in. It was huge. There were ladies who rolled carts full of food from table to table and foisted food onto us. But it was very cheap: five people ate a huge amount of food for about $45.

I am not a huge fan of dim sum because honestly I find genuine Chinese food kind of weird. I feel that this is in some sense okay because I read an article in Gourmet about the article writer taking some visiting top Chinese chefs to the French Laundry and other fancy Bay Area restaurants, and them generally thinking everything was gross and weird (though one chef liked the oysters at a Fisherman's Wharf oyster bar). I figure if it's okay for actual chefs to be that un-cosmopolitan, I can be too.

Not that I went hungry. I like the sticky pork buns a lot, so I had two of those. There was also a vegetable of the brocolli/cauliflower/brussel sprout family which was also very good, and a dessert bun with red bean paste. But some of the other stuff I tried made me queasy. And I always felt like a friend's mother was coming around with a cart every few minutes urging me to eat more.

Wed Feb 22 2006 08:20 Unpopular Games #2: "Salivator Action"

(8) Wed Feb 22 2006 22:23 Stereotype New York: Car #1 wants to turn left. It is blocked by car #2, which is trying to go straight in the opposite direction but is blocked by traffic all the way to the next light. Car #2 can't move, and it's blocking the intersection. So of course car #1 honks at it. Cars behind car #1 honk at it because its deadlocked desire to turn left is preventing them from going straight. Honk honk.

The occupants of cars #1 and #2 now roll down their windows and begin to argue. Who knows what about.

This argument continues long after the light in front of car #2 turns green, and the cars in front of it have moved on. Now car #2 could move too, but the driver's still arguing with the driver of car #1. Everyone loses! Honk honk. Thats's New York.

(2) Wed Feb 22 2006 23:13 CONSTRAINED WRITING COMICS: I submitted a set of constrained constraints to Jack Masters of Flag on the Moon, and he wrote a comic incorporating my constraints! Awesome! Not sure why I think it's so awesome since that's basically what he said he'd do. Oh wait, it's the comic itself that's awesome. (The constraints on my constraints are listed here).

I think Crow is made out of a nickel alloy, but I can't find a source for that. All I could find was a non-canon Bill Corbett saying Crow is "some strange gold alloy". But the inclusion of Crow in the comic makes up for any inaccuracies. Anyway, clearly he should be made of chromium. Chroooomium.

(1) Wed Feb 22 2006 23:44: I'm done with my Cookbook recipes (the rest are contributors' or Lucas's), so I'm writing some additional recipes about really cool things. This because 1) such things are, by definition, really cool, and 2) I hope that their very presence will make people want to buy the book. They will leap out at you from the table of contents and say "I am awesome! Buy this book at once!". That's the plan, anyway.

Today's cool thing is Ruby's midilib, the... wait for it... Ruby interface to MIDI files. I did a recipe that shows how to visualize data as a MIDI file, and how to compose algorithmic music (sample composition).

While learning about midilib I came up with an observation about MIDI, which is that it's good for accurately representing what happens when you play a real instrument. It's kind of frustrating and unnormalized if you need to programatically compose music. I ended up taking a tip from the excellent MIDI Scripter project, and showing how to write a little wrapper on top of midilib that makes composition easier.

The experience gave me a huge number of ideas for musical programming projects. With my near-total lack of theoretical training, I will be unencumbered by traditional notions of "harmony" and "taste". For one, I plan to get into fractal music, though probably not the way the author of that web page defines it. Hard to tell, with that color scheme.

Thu Feb 23 2006 12:04 Sports in the Hive Mind: I just recycled a Honey Nut Cheerios box which has a cartoon on the back about a baseball game played by bees (probably against the bees of another hive). The Honey Nut Cheerios mascot, a worker named Buzz (I guess they'd all be named Buzz) has to save the game in a Casey at the Bat style scenario, but she's[0] been illicitly dipping into the hive's honey supply and is a little... buzzed.

Actually no. I'm just making stuff up to make the comic more interesting while keeping it in its chosen genre of dumb comedy. There's a really interesting idea here about how team sports would work when the players are all part of a hive superorganism. But the actual cartoon takes up the entire back of a cereal box and the payoff is a dumb sight gag and a couple lame puns. Plus it violates the rules of baseball. I may ask Jack Masters for another metaconstrained comic that does this idea better.

[0] The cartoon refers to Buzz as "he", but "he" has a stinger => worker bee => female.

Thu Feb 23 2006 18:49: I had an idea. Can you make ghee in the microwave? Yes you can.

(5) Thu Feb 23 2006 23:11: Today's cool things: Ruby encryption libraries! Crypt, which is pure Ruby, and EzCrypto, a wrapper around the user-unfriendly OpenSSL bindings. Dunno why I didn't think of an encryption recipe until so late in the project. Also, only today did I really put all the pieces together about how public-key encryption works. The only difference from single secret key encryption is that there are two ways of getting to the key. Is that right?

Update: I don't know anything.

Fri Feb 24 2006 22:25 We will win through bee telepathy and motivational posters!: My Honey Nut Cheerios bitching has been turned into yet another excellent metaconstrained comic. That's a sneaky-looking queen bee.

Soon I'm going to make some chapati, because how hard could it be?

(1) Fri Feb 24 2006 22:29: Incidentally, I'd like to try my hand at doing the constrained comics myself, so if someone wants to give me some constraints I'll give it a try this weekend. Also this weekend: Sumana and I are going to be extras in an episode of Uncle Morty's Dub Shack.

Sat Feb 25 2006 21:38 CONSTRAINED COMICS COMICS: Jack Masters gave me a fun set of constraints and the result is a two-page comic (click the first page to get the second page).

The constraints were:

1. The comic must be non-linear
2. The Xth panel of the comic must contain a character who's name starts with letter X of the alphabet. See below for a full list.
3. The comic must be a murder mystery, and for every character in the comic there must be an ending where they are revealed to be the murderer
4. You and I must be in the comic. I couldn't find any pictures of Jack, so I represented him by the kangaroo from the movie "Kangaroo Jack".
5. The comic must advertize WHAMMO, made by satan's minions

Sumana requested a suggested order for the panels, to deal with the non-linearity. The best I can do is 4-9-5-1-11-12, and then 7 as the PSA at the end. The rest of the panels are just possible endings.

List of characters: Adam (played by Adam Kaplan), Beelzebub, Cerberus, the Devil (in snake form), Eve (played by a character from the video game Parasite Eve; probably not actually named Eve), Furfur (look it up! I did!), Giant Anthropomorphic Pineapple, Hastur, Imp, Jack Masters, Kraken, and yours truly.

Sat Feb 25 2006 22:37 That Touch Of Mink: Not sure why we're Tivoing so many Cary Grant movies. This one painted the early 60s as a time where people were insane or sociopathic in very retro ways, in a rather creepier way than, say, The Apartment did. Pretty funny, but also disturbing. When did the Hayes Office shut down? Wikipedia says 1967. It's the kind of movie that makes you ask that question.

(6) Sun Feb 26 2006 07:53: Sumana said my comic was too crowded and complicated and therefore not funny. I can't do anything about the complicatedness at this point, but I can do something about the crowdedness. Crowdedness is my great artistic sin, especially when it comes to collage. So I went back and got rid of some unneccessary graphical elements and improved the colors for the JPEG output while I was at it. Still no guarantees as to funniness, though I did treat a couple really overused jokes as "unneccessary graphical elements".

(1) Mon Feb 27 2006 22:42 Kris: Kris was in town for ComicCon, so we hung out today. Lots of fun, but not much to report. I'm going to go back to California in March after the first draft of the book is done, but I'm mainly just going to be in Bakersfield, so I probably won't get to see anybody except family.

Tue Feb 28 2006 23:45: Argh. All my energy is going into book writing, leaving only scraps for NYCB. I don't even have link dumps to show you, unless you like looking at random parts of the Flickr and GTK API documentation.

"I'll teach you how to make cheese," he said. It was a lie. "My hand slipped," he'd say. "It's thermodynamically impossible." Always something. She'd had enough of the excuses, the anthills left by unfinished projects on the back porch.

It had to stop, so she stopped it. With biscuits. The sour milk, the sawdust. She cleaned it all up, not the way a woman usually does, but for the first and last time.

"Look," she said then. "I've invented fiberboard."

(1) Wed Mar 01 2006 18:21: Murder in the Gunroom. Perhaps a GUN was involved?

(2) Wed Mar 01 2006 21:35 Random Books: Jarno has a little application that picks random search terms to find a random book on Amazon. But dang if I didn't click that doohickey ten times without it ever coming up with a book. I decided to try and implement a random book finder that would reliably come up with books.

The main problem is the arbitrary restriction of Amazon Web Services where you can't get more than 250 pages of query results. If it weren't for this restriction, you could pick a random number from a huge range and do a Browse Node search to get the corresponding page of results. This is still a pretty good solution if you want to search for products in a restrictive set like the set of science fiction books: find the corresponding Browse Node and do a browseBestSellers.

As it is, the simplest solution is to only search for a single word, instead of Jarno's two. Here's some code that does that and works pretty well.

import amazon
import random

def random_product(keyword, max_page=250, associate="crummthesite-20"):
    result = None
    while result == None:
        page = random.randint(1, max_page)
        try:
            results = amazon.searchByKeyword(keyword, page=page, type="lite",
                                             associate=associate)
            if results:
                result = random.choice(results)
        except amazon.AmazonError:
            results = None
        if not results:
            max_page /= 2
    return result

def format_product(p):
    print '<a href="%s">%s</a>' % (p.URL, p.ProductName)

keyword = random.choice(open("american.0").readlines()).strip()
format_product(random_product(keyword))

Sample book: Reutilization of Waste Materials.

The other solution is screen-scraping. As covered earlier, there is an Amazon URL such that you can randomly vary a parameter and get a more or less random set of books. Here's some code that fetches one of those URLs and grabs an ASIN at random. Then it does an AWS lookup of that ASIN because life is too short to scrape all the appropriate information off that big ol page. This is a little less reliable than the earlier implementation, but it gets a better random distribution of books (since the other one picks a word to search on, and word distribution is not random).

from BeautifulSoup import BeautifulSoup
import amazon
import random
import urllib2
import re
import socket

URL = "http://www.amazon.com/exec/obidos/tg/new-for-you/top-sellers/-/books/all/books/0/1/%s/1/"
def random_book(max_page=150000, associate="crummthesite-20"):

    product = None

    while product == None:
        url = URL % random.randint(1, max_page)

        req = urllib2.Request(url)
        try:
            soup = BeautifulSoup(urllib2.urlopen(req).read())
            asins = soup.fetch("input", {"type" : "hidden",
                                         "name" : re.compile("asin\.*")})
            if asins:
                asin = random.choice(asins)['name'][5:]
                product = amazon.searchByASIN(asin, "lite", associate=associate)[0]
        except socket.error:
            product = None
        if not product:
            max_page /= 2
    return product

def format_product(p):
    print '<a href="%s">%s</a>' % (p.URL, p.ProductName)

print format_product(random_book())

Sample book: Choice, Welfare and Measurement

Wed Mar 01 2006 21:55 Restaurant Tradeoff: When I moved to New York I gave up a lot of burritos (as The Burrito Eater indicates) but apparently I gained a huge number of extremely specialized restaurants that only serve one dish. For instance, the arachibutyrophobic-hostile peanut butter restaurant I went to last month.

Now I've discovered Rice to Riches, which only sells rice pudding with lots of different toppings. I think if I start a restaurant it would be easier to start one of the really specialized ones. Let's say, I would sell only choux puffs with various sweet and savory fillings. Meanwhile, here's a copycat recipe for their Rocky Road rice pudding.

Wed Mar 01 2006 22:17 Flickr Rebus: I had the idea to write something that used the Flickr API to turn a sentence into a rebus. For every word it would do a tag search on that word, and use the appropriate thumbnail picture. Well, I'm not that interested in doing it now because there's a similar thing called flickrfling which takes an RSS feed and rebusizes the first part of it. It's a Flash application and I don't know why it takes an RSS feed (instead of, say, a string) except to get another buzzword in there, but it's satisfying enough to stave off my desire to do my own for a while.

(1) Thu Mar 02 2006 22:23 Link dump:

• Just when you thought Lucene couldn't get any cooler, now it has an Atom-based web service API.
• What causes double rainbows? The surprising answer... after this!
• A cool-sounding movie about the sun. No details are available about this cool-sounding movie, but you can learn fun facts about the sun.
• This PBEM game about the Singularity sounds like a great idea, but after reading the rules I think the dressing is the main cool thing, and the game mechanics are kind of generic. Let's come up with some Singularity-specific game mechanics.
• Displaying large datasets atop Google Maps

Turns out double rainbows are caused by inflation. Now here's Bob with the weather!

(1) Fri Mar 03 2006 12:21: I thought this comic verse would be pretty funny, but upon examination none of it is as good as the stanza that made me laugh in the first place:

  The Bard subsists on simpler food:
    A dinner, not severely plain,
  A pint or so of really good
        Champagne--

I don't understand anything about poetic meter or why one meter is funny and another one is serious, but that one's funny. It's a meter with a punchline.

(5) Fri Mar 03 2006 13:32 Humanocentrism: "Ever wondered what form your favorite Transformers would take if they existed in different eras?" Presumably they would look a lot like they normally do, since they're aliens from another planet and not restricted to Earth's current level of technology.

Just part of my longstanding tradition of taking the logic of Transformers at face value when nobody else does.

Fri Mar 03 2006 19:55 Amazing New Web Meme: that I just made up. Reveal interesting Wikipedia articles you looked at recently.

• General MIDI
• Wikipedia recently deleted pages.
• Black Like Me
• Production Code
• Laser harp
• Suspension bridge
• Dog Latin
• Intellectual rights to magic methods

Unfortunately, this web meme has no way of propagating, so it is sure to die out. Just like the laser harp.

Fri Mar 03 2006 20:05 Language Games: A while back I designed a game involving the unreliability of translations and the mendacity of translators. One requirement for playing the game without finding it extremely tedious was some automated tool for garbling a message by using an automatic translator to translate it back and forth. Multibabel is the kind of thing I had in mind.

For a while before I A game was designed; Non- reliability of conversion and the inclusion of mendacity of translator. Boredom without finding that very, one condition in order to do the game went the tool which in order to distort message with use to converting that of the automatic translator on front and back is automated. Multibabel There is a kind of thing which I had with heart.

Fri Mar 03 2006 20:43: For a while now I've been looking for a way of keeping track of my various lists of books and cultural artifacts. I want a website that offers a web service interface so I can munge my list of books a million ways. This quest of mine has gotten me into all kinds of trouble and I'm still using a series of parallel Amazon wish lists because they at least have a web service interface for getting the list of books.

The latest entrant into this fray is lib.rario.us, which looks just like del.icio.us except it's for keeping track of things that have an ASIN or ISBN. I like the del.icio.us interface a lot (mainly because it's easy to hide it behind a NewsBruiser interface), but lib.rario.us doesn't offer even one tiny web service.

Amazingly, though, the traditional del.icio.us rip-off name proves itself useful for once: it's given me the obvious-in-retrospect idea of just using del.icio.us to keep track of my wish list. Use it as a platform. Why didn't I think of that before? It could work, once I put up something between me and del.icio.us that looks up ISBNs from titles.

(1) Fri Mar 03 2006 22:25 Paleoworld: This is a whole documentary series (on the Science Channel... OF SCIENCE!) about dinosaurs. I guess it could be about other paleo-things but I only seem to see the ones about dinosaurs. Sample shows: "The Lost Dinosaurs of Egypt" and tonight's episode, "Troodon: Dinosaur Genius". It's really good to listen to while making dinner because it's about stuff that's extinct. All they can do is circular pans around models of the dinosaurs, and pictures of paleontologists digging in the desert. So you're not missing anything by just listening to it.

"Troodon: Dinosaur Genius" is about the Troodon, which had an opposable finger and the largest brain/body mass ratio of any dinosaur. Pretty soon the documentary will reach the traditional "unwarranted extrapolation" phase, where people will speculate that, were it not for the K/T event, Troodon could have continued its trend towards greater intelligence and become a sentient lizardoid species. Yes, just like in that episode of Star Trek: Voyager.

The main problem with Paleoworld is that there's no warning when it goes to commercial, and the voiceover announcer for the show sounds just like the voiceover announcer in the commercials. So you hear things like this. "But the story of Troodon does not end here. It launches us from the dinosaur age to the space age. The latest Mars Orbiter is about to arrive!" And "Troodon was a dinosaur with incomparable brainpower. All of us have internal plumbing. But sometimes our pipes doesn't work as well as they should."

I once worked with a guy who was going to voiceover school nights, and he said that there is a standard set of voices that you learn in voiceover school. I guess this is so the school and employers can grade you on an objective scale, but it's really creating a voiceover monoculture.

Sun Mar 05 2006 23:22: Shuffling a Stacked Deck has a good approach to a popularity feedback loop. The solution is to randomly insert certain results into the list of the most popular. More elegant than the Ultra Gleeper's solution of multiple tiers of results.

I first noticed the popularity feedback loop back in the BBS days. A common feature of BBSes was a "Most Popular Files" list, and people just kept downloading the files on the list, which to my mind defeated the purpose of advertising these files. I set up a "Least Popular Files" list that listed all the files that had never been downloaded. It didn't really work.

Mon Mar 06 2006 09:17: Mark Dominus's weblog is great! Don't worry, it contains hardly any Perl. It does contain "The Sesame Workshop people were not able to solve the problem."

(1) Mon Mar 06 2006 22:51: I have an idea for applying the Dissociated Press algorithm to a body of music. Dissociated Studio kind of does the same thing, but I have plans for a much larger-scale project. Stay tuned... as I never get it done!

(3) Tue Mar 07 2006 22:36: Sumana went to an MC Frontalot gig and ended up in Overcompensating.

(2) Tue Mar 07 2006 23:20 Search Request: Frances government - good or bad? I think it would be good.

Wed Mar 08 2006 14:24: Sumana called me up and read me items from this poorly-translated Chinese restaurant menu, and then refused to mention it in CES because "it's played". I find this attitude extremely shortsighted, so I threatened to post it to NYCB. The awe I felt this would provoke was not forthcoming. But I posted it anyway. Enjoy. Our favorite: "A west bean pays a fish a soup". I am a decadent fish! Fill my aquarium with the finest soups!

Thu Mar 09 2006 08:32 Crosspost: I posted to Spam As Folk Art about honest stock spam.

Thu Mar 09 2006 21:55 Off By One:

• List of fictional plants
• List of fictional planets

Bonus Wikipedia link: Kappa

Fri Mar 10 2006 14:08 Alternate Universe News:

The White House maintains that Gamera is a friend to all children.

Sat Mar 11 2006 10:30: It's well-known to we Game Roundupologists that a large class of games can be improved by replacing the humans with dinosaurs or cars. Now, a Flash game programmer who demonstrates that any NES platform game can be improved by replacing the protagonist with Mega Man. Found via Adam Parrish.

(5) Sat Mar 11 2006 16:33: Where are the boats? In the ocean, silly.

Sat Mar 11 2006 20:35 QOTD: "Interrupting cows? In Berk--" "Moooo!"

Sun Mar 12 2006 21:09: The first draft of the Ruby Cookbook is almost done! I still have to edit three contributed recipes, one of which won't be delivered until tomorrow. But I'm done writing recipes. Now I begin rewriting them.

I redid the outline so it shows the reviewer coverage of the various recipes. I'm also working on a script that grabs the code snippets out of the recipes, runs them, and sees the results match what it says in the book. That one's a little tricker.

Sun Mar 12 2006 21:50: If you've been wondering what it is the companies who sponsor the Starslip Crisis RSS feed actually make: it's this stuff. I especially like the "Neutron Simulator", and the "Beam". I picture the neutron simulator as being basically a big piece of radioactive chalk. As for the beam...

Mon Mar 13 2006 13:42: Z Machine Sets Unexpected Earth Temperature Record. The forthcoming Inform 7 must be hotter than I'd thought.

(3) Mon Mar 13 2006 16:48 Formal: I have to convince some people that I'm being paid for my work on the Cookbook instead of doing it for no reason. I faxed them a printout of the invoice page from my agent's website. Not good enough. I've got the right idea, but they need something that looks more formal. Could I maybe fax them a printout of a web page that shows the deposits going into my bank account? Those bank web pages have real formal-looking HTML. Of course I can black out all my account numbers and any sensitive information before faxing in the printout.

Incidentally, I recently became a Texas oil millionaire. Why you laughin', li'l buttercup? Ain't it the truth, why it says so right here on this printout of a web page about my bank account.

(1) Tue Mar 14 2006 20:33: Bootleg Thin Mint recipe.

(2) Wed Mar 15 2006 23:15 Done!: The first draft of the book is done!

Thu Mar 16 2006 10:15 I'm Partched!: Hey, Jake, you can now play with Flash simulations of Harry Partch's instruments.

Thu Mar 16 2006 23:14: Brendan and Maria are in town. Today we went to the Met and saw a bunch of stuff. More about this later as I have to sleep. But I will say that ancient Chinese office supplies remain awesome. We saw a book called "Mr. Fang's Ink Compendium". Unfortunately copies of this book were not available in the gift shop.

(7) Sun Mar 19 2006 01:09 Once Upon a Time Variant: We hung out with Brendan and Maria all day and played games. We came up with this Once Upon a Time variant which fixes what I think is the biggest problem with OUaT: the underuse of ending cards. You get a million story cards that have single concepts on them, and there's an equally large deck of ending cards, each containing an entire plot point. This other deck is cooler but you go through it at 1/100th the speed.

So, the variant: you start off with two ending cards. Whenever it's your turn, you must play one of your ending cards before you can play a story card. You can just play the ending card and start tying it into the story, or you can talk for a few sentences before playing the ending card, to connect the current story to the ending. No one can interrupt you before you play your ending card, but you can't play any of your story cards either.

Once you play the ending card, your turn continues as in vanilla: you play story cards and try to get to the point where you can play your second ending card. If someone interrupts you, you must draw a story card and an ending card (so you begin each of your turns with two ending cards).

If someone's story gets to the point where you could play an ending card, you can interrupt with the ending card and start your story from there. (This actually happened.)

Since OUaT stories jump around a lot anyway, it's not difficult to use an ending card to close one thread of the story or to spawn a new thread. The additional constraint is generally what I need to feel comfortable making up an OUaT story.

Update: See comments for feedback from OUaT co-author James Wallis.

(5) Sun Mar 19 2006 13:33 Why Do You Exist Here?: I read a philosophy paper (probably online) that hypothesized that our perception of consciousness was an artifact of the fact that time only goes in one direction. I want to reference this paper in the Ruby Cookbook, but I can't find it. All the words I can think of to describe the paper are also keywords used by the wormhole aliens who fill the web with crackpot stuff like "The Illusion of Linear Time and Solar Dynasties". In fact, I did find a paper that seems to say the same thing as the paper I'm trying to find, but it too was written by a wormhole alien and is unintelligible.

Does anyone else remember this paper? Maybe I should have developed an intense emotional hangup when I first discovered it, so that I'd always be revisiting that moment and I could remember the URL.

(2) Sun Mar 19 2006 22:48: Lego aircraft carrier. Reminds me of the plane boards used on real aircraft carriers.

Mon Mar 20 2006 22:07: Going to Bakersfield tomorrow to see everyone. I don't really have things to say right now because my brain is boggled from debugging recipes (the automated test runner helps a lot; has any other book ever done this? Is it worth writing about?).

I am reading The Dalkey Archive but so far it's a much less funny ripoff of The Third Policeman. It better get surreal and fast, or I'll... regret my purchase!

(2) Tue Mar 21 2006 07:43 Caution: Congratulations!: "Congratulations! Your seat comes with 36 channels of DIRECTV programming." Like I did something to win those 36 channels of DIRECTV programming.

(2) Wed Mar 22 2006 20:59 Pac-Man Heresy: I was looking at this list of official Pac-Man franchise games and ports (caution: page uses "gay" as a pejorative). And a couple things occured to me. First, what would be much more interesting (and longer) would be a list of unofficial Pac-Man games and ports. Like, uh, this page, except more opinionated. The other thing is the monsters/ghosts. According to Pac-Man orthodoxy, the monsters who chase Pac-Man around the maze are just that: monsters. Generic monsters. NOT ghosts. To call them ghosts is counterrevolutionary. Some fools think they are ghosts because the horrible Atari 2600 port of Pac-Man had really bad flicker, and called them ghosts to cover up the fact.

But come on. They look like ghosts. Specifically they look like Charlie Brown ghost costumes, baggy sheets over the head with holes cut for eyes. That's why people think they're ghosts, not because one port twenty-five years ago couldn't handle five moving objects on the screen at once. People don't see random things flickering and say "oh, that must be a ghost". There has to be some preexisting resemblance to a ghost. So don't blame people for thinking that the monsters are ghosts.

Instead of simply asserting that the monsters are not ghosts, it's pretty easy to prove it, a la Aristotle. A ghost is the disembodied essence of a dead person. When Pac-man, under the influence of a power pellet, eats a monster, the essence of the monster is separated from its body and goes into the penalty box to regenerate. What is that disembodied essence but the monster's ghost? How can a ghost have a ghost? It does not make sense.

Hopefully this proof can bring some civility to the enlightenment of those who think the Pac-Man monsters are ghosts. This is my Pac-Man heresy, brought on, I believe, by Pac-Man fever.

Thu Mar 23 2006 23:20: Rachel and I went to the Newport Pier thinking that it was the place where they filmed the boardwalk scenes for Arrested Development. That's basically where those scenes were set, but they were actually shot at a more photogenic pier, many miles northwest in Marina del Rey. But we ate a frozen banana anyway. (As things you buy on a pier go, frozen banana is pretty healthy.)

I originally tried to write this entry in the style of an Arrested Development episode, but though it was formally correct there were no jokes in it. This is how you repay me for how I repay you?!?!

Fri Mar 24 2006 20:29 Emergency Samosa Sauce: If you have samosas but no samosa sauce, use apricot jam, melted in the microwave. Kind of like the tamarind-based samosa sauce.

I developed a samosa song which I sing when I'm awaiting samosas. It goes: "Samosa!" Except you say it on the fourth beat of the measure. I think it could be the next "Tequila".

(1) Sat Mar 25 2006 00:57 Terror of Fox Terrier: When I was younger I read a Stephen Jay Gould essay about how textbook authors copy from each other instead of coming up with original ways of expressing ideas. The textbook (or, metatextbook, I guess) example is that Eohippus has been consistently described as "the size of a fox terrier" for over a hundred years, as the fox terrier itself has plummeted in popularity, and as the number of people who need to learn about vertebrate evolution despite not being the kind of Nick Charles WASPs who would know what a fox terrier was anyway has climbed.

I reread this essay today and I must have gone through postmodern puberty since my first reading, because this time I saw the subtext. Human culture is supposed to change and propagate quickly, by Lamarckian means. But the blind, semantics-ignorant copying seen in these textbooks shows that culture propagates by slower Darwinian processes too.

Textbooks evolve through sexual selection. Nothing selects against this bizarre and useless (and, Gould claims, wrong) comparison between Eohippus and a fox terrier, because nobody ever reads that sentence and tries to make sense of it. It is junk DNA. The whole text of the textbook only exists to make it larger, so it looks more attractive to school districts. All the resources are going into improving the textbook's plumage.

The reason I read the essay in the first place is that I've seen the fox terrier comparison three times in the past month or so. This is twenty years after big-shot paleontologist Gould publishes an essay totally slamming the comparison, with no one coming forward to explicitly defend it. The first time I saw it was next to an Eohippus in the horse evolution exhibit at the American Museum of Natural History. I guess it was about the right size, but I've seen a real Eohippus fossil and I've never seen a real fox terrier, so what do I know? Could we get a more helpful comparison in a real museum exhibit right next to the actual skeleton? (The Gould essay makes a tenuous connection between the AMNH and the original source of the comparison, so maybe they're just being stubborn.)

The second place I don't remember where I saw it. The third place was in a book I just read called The Eternal Frontier, an ecological history of North America. Apart from the fox terrier comparison it was a pretty good book, though it was written by an Australian which means it exhibits the cowboy obsession common to non-North Americans who write about North America. It was also pretty depressing since it portrays the entire history of life as organisms going in and really screwing things up for other organisms (though occasionally an non-organism like an asteroid or an ice cap really screws things up for everybody).

Sat Mar 25 2006 10:35: Just now I learned how to not have your multiple bash shells overwrite each other's history. And I realized that that was the first time in a long time that I expanded my knowledge of living in a Unix environment (as opposed to programming in one). I guess this big list of other things you can change in bash is a good start.

Sun Mar 26 2006 23:54: Last day of my California trip. Visited with Adam/Kris/Kim for a bit. We recorded a song which may or may not ever see the light of day. You're probably hoping not.

Nothing has really happened except slow spotty progress on editing the book. People are starting to ask me what I'll do next. Next I will decide what to do next.

Mon Mar 27 2006 19:54 Abominations: I saw this on the screen of the guy sitting next to me on the plane (that 36 channels of DirectTV really is a curse). The Three Robonic Stooges. Hanna-Barbara cartoon from the seventies. No, wait, it gets worse. In this Hanna-Barbara cartoon from the seventies the Three Stooges are given Inspector Gadget-style abilities so they can catch spies. But they can't because they're incompetent. But somehow they end up doing it anyway.

I could buy the robonic (were they trying to get a trademark?) modifications if the show took place in a Harrison Bergeron society, where as a matter of social policy people at the Stooges' level of competence were given special abilities to bring them up to the level of the general population. But it ain't so. There aren't even any Three Laws of Robonics. The only thing this cartoon has going for it is that according to IMDB, the guy who dispatches the Six Million Dollar Stooges is code-named Agent 000. The show predates and has the same basic plot as Inspector Gadget (minus the PSAs, but if you started doing PSAs after Stooges shorts you'd never be able to stop). You think that'd be worth something, but it's not.

Obviously the Stooges are not some wonderful cultural icon that has been demeaned by this stupid cartoon, but that doesn't excuse anything. Though by any measure the Stooges come up short when compared to the Marx Brothers, they have the same basic dynamic, and it's one you don't see enough anymore. Characters with no marketable skills, who cling together more through inertia than attraction or synergy. Coveting a position in high society, they end up going for it because they've seen that nothing really separates them from the current inhabitants of high society (who have no marketable skills either). But their plans are jeopardized by their own self-destructive behavior. <-- This paragraph doubles as my "outreach" attempt to explain to women the appeal of the Stooges, such as it is.

Tue Mar 28 2006 09:24: Did you know there was an Arctic Winter Games? They "promote culture and values". I got lottery spam trying to convince me I'd won some AWG-related cash prize (maybe victims think they won bronze in Dog Mushing and forgot to pick up their prize?), so either the games are better known in other parts of the world, or a spammer is a big fan of them.

(1) Tue Mar 28 2006 23:17 Earwax: This BoogaBooga entry reminded me of an experience I had at the Met. I was looking at ancient Chinese artifacts including an earwax pick shaped like an alligator. I was thinking what a strange kind of thing an earwax pick was. Then a family came up to the same case and the mother of the family said "Look, an earwax spoon like the one we have, only it's shaped like an alligator! Isn't that neat?" Supposedly George Washington had an earwax spoon too, so I guess that makes me the only person on earth who doesn't own one.

Would you like to learn more about earwax? Too bad. Already the knowledge burrows through your tympanum and into your brain!

Tue Mar 28 2006 23:36 And then I FREEZE-DRY the marshmallows!: When I was hanging out with Adam/Kim/Kris someone pointed out that if kids' breakfast cereal didn't already exist and have a lot of machinery dedicated to making it, no one would eat it except as an avant-garde dessert like a floating island.

(3) Wed Mar 29 2006 00:08: Famous comic Cat and Girl has a Flash-based comic creator program. Write constraints in comments and I will proactively utilize web technologies to meet clients' needs, assuming the clients are you and your needs are to be totally blown away by constrained Cat and Girl comics.

Thu Mar 30 2006 13:06: Encoding problem.

(4) Thu Mar 30 2006 21:22: Once again, only ONE MAN dares give me comic constraints. In addition to breaking the 200-recipes-edited mark today, I created my own Cat and Girl, in the spirit of the original. The constraints were:

• It starts with a joke.
• It ends with a real conversation I've had. (Conversation was fictionalized to fit the Cat and Girl characters.)

(8) Thu Mar 30 2006 23:31 Vowecowedrrr: I hate the vocoder because it's used in all the damn pop songs I hear in the supermarket, and I suspect its primary use is to cover up the fact that pop singers can't sing. But this tutorial and its obvious-in-retrospect explanation of how vocoders work is pretty neat.

(1) Fri Mar 31 2006 15:04 More Cat, More Girl: Factitious's constraints:

• The strip should start with a bad pun, and deal with the repercussions of that pun.
• At least three panels must be in languages other than English; understanding them should be helpful but not necessary for appreciating the comic.
• Cat wins.

With further ado, the comic:

Tomorrow: Kevan's constraints, if I have time.

Fri Mar 31 2006 21:48 Mini Game Roundup: BlinkenSisters: Hunt for the Lost Pixels. A 2D platform game, a huge download due to embedded OGG files and graphics. Is it worth the download? No, not really. But the mechanics aren't bad. Your character jumps (and looks) kind of like the Princess in SMB 2, which I've always felt was a vastly underused mechanic. There's also a cool-looking squid monster. Just too much a basic jump-and-squish game and not really improved over older games 1% its size.

(1) Sat Apr 01 2006 22:23 Cleaning the grotch cages: One of my recipes contains fake Hitchhiker's Guide swear words. I just went back to it and edited it to contain fake Planetfall swear words instead.

That can't be a good use of my time.

Sun Apr 02 2006 18:23 Repurposed Art: Here's one I call "The Founding of the Federation"

Sun Apr 02 2006 20:56: robotfindskitten now exists as a Google Maps add-on.

(2) Sun Apr 02 2006 21:05: Speaking of Google Maps add-ons, here's one that shows the sea level rising. Still looking for what I'd really like: something that can also show what happens when the sea level falls due to ice age.

(5) Wed Apr 05 2006 22:05: Sorry for no NYCB. Very busy editing book. Despite many peoples' help, still no progress on finding that time-and-consciousness paper. I'll probably take the reference out. Unrelatedly, has anyone seen this movie?

Thu Apr 06 2006 23:11 You Won't Believe The Cardinality Of The Set Of Twists: I'm getting pretty good at figuring out the twist in Hitchcock movies (except I don't remember North by Northwest really having a twist). The only problem is that the movie is never as long as Tivo says it is, so I'm second-guessing myself right until I see "The End", whether the twist happened for real, or whether it will be reversed by another twist before the end of the movie. So even when I guess correctly, modern technology enforces Hitchcock's idea of suspense: that you know the Titanic will sink, but you don't know when. Hmm, maybe the twist gets reversed after the end of the movie. That would be... quite the twist!

Thu Apr 06 2006 23:34: I just saw a spam weblog that takes incoming search requests, reproduces the search to get some results, and chops up one of the results to create a weblog entry. I think there must be a non-spam possibility in there. Also a pony.

Also, I think this is the first review of my music not written by someone who knows me.

There are some twisted songs out on the net, and if you search for the good stuff, you're bound to come across a couple that, bad as they are, they're really catchy, and you laugh everytime you even think about the song.

Bad but catchy: that's me.

Incidentally, my gee-tars were delivered the other day, so once the book is done and I'm in the post-book era I can resume recording my new album.

Fri Apr 07 2006 23:33: Hypnotic lists of TLAs.

Sat Apr 08 2006 18:02 QOTD: "Nagaland. Snake land. Snakes in a land!"

(2) Sun Apr 09 2006 23:32 Zardoz: Andy came over and we watched Zardoz. Sumana left after the first ten minutes. We watched the whole thing apart from some Tivo-skipping during two extremely long repetitive scenes. I think people get distracted by the visuals of that movie because the plot is not that complicated. Some of the visuals are great and some (ie. Sean Connery's costume) are terrible. I liked it better than Mad Max Beyond Thunderdome. Save your breath, Jake.

Tue Apr 11 2006 23:30 Fission Cuisine: I was thinking about Indian desserts. I've never had much in the way of Indian desserts because they're way too sweet. They all seem to have the normal sugar you'd put into an American dessert, plus a simple syrup, plus sweetened condensed milk. I haven't tried it yet but I'd like to figure out a way to do, say, galub jamun, at a sweetness level I can tolerate.

Tue Apr 11 2006 23:31 Ominous Agency Names: The Committee of Public Safety. I should start collecting these.

(2) Wed Apr 12 2006 13:39 Whee!: I've completed my editing pass through the Ruby Cookbook. Now I just have to incorporate about 60K of feedback.

(2) Wed Apr 12 2006 20:09 Limerick:

There once was a man from Peru
This limerick ends on page 2.

It's in the book. I've got all kinds of silly stuff in this book. I had an earlier limerick and cut it, but this one's in there.

Thu Apr 13 2006 00:06: Apparently my Stock Spam Effectiveness Monitor was the dataset used in some German guys' talk on the topic. Even my name was Germanized to Leonhard. Their number-crunching actually answers the question on which my graphs are ambiguous:

In the first two to three days of a stick Spam wave therefore the applied shares rise over up to two per cent. Afterwards the course consolidates, in order to sink in the consequence strongly. Apparently this is in all rule frequently the case if those coat the Spam clients their profit. This behavior is statistically significant in each case, avowed Rainer Boehme.

He avows it! It must be a real effect! But not drastic enough to see just by looking at graphs.

Actually I just gave the stock spam code to another German guy (a Ph.D student), but he's not one of the ones who did this report. And last July I gave the data to yet another researcher, but he was at U Penn. So this is a previously unknown use of my data.

Thu Apr 13 2006 20:18 Worthwhile Ruby Library: The HighLine library needs some publicity. I didn't know about this until people reviewing the Cookbook pointed me to it. It's a library that makes it really easy to write cross-platform line-based console applications, with ANSI color. It doesn't do cursor positioning, so it's not the second (or third) coming of curses, but it's really cool. It's the highline gem. It doesn't really have a web page: just the Ruby Quiz entry that gave it birth.

Fri Apr 14 2006 19:03: Okay, the Cookbook is basically done. Yay! I went through my mail looking for old feedback and cleared out about 100 messages pertaining to the book. What's left? Mostly Beautiful Soup stuff I haven't really looked at. Hopefully it's 50 people all saying how great Beautiful Soup is and reporting the same three bugs. Somehow I doubt it.

Fri Apr 14 2006 21:34: This is really cool: gerrymandering can be quantified. My new Congressional district features "moderate gerrymandering". Also it seems to include Central Park.

(1) Fri Apr 14 2006 22:20 Local Science: A decent heuristic for seeing whether something is real science is to check whether other places have it too. This is probably the flip side of Jared Diamond's idea about why China's early lead in science and exploration died out (more on this later, actually).

Everyone has relativity and genetics and evolutionary biology and quantum physics and plate tectonics, but only the US has creationism (similarly for intelligent design, or as I call it, c******nism). Only the USSR under Stalin has Lysenko-style evolutionary biology. Everyone has astrology and alchemy, but everyone's astrology and alchemy is different. And only the UK has this special newspaper-based "formula for the perfect x" branch of science.

This is where you take something vague, dissect it into a bunch of vague contributing factors, assemble them back into a vague equation, and claim that it represents perfection. This gets you in all the papers and nowadays the weblogs as well. As far as I know only one article two years ago has ever dared to point out the ridiculousness, and it's still going on.

Oddly, NYCB has previously encountered Len Fisher, unwitting founder of this particular branch of pseudoscience. He's the one who started it with his measurements of liquid absorption in biscuits, and his use of a real, preexisting equation to describe it. I think he deserves some kind of meta-Ignobel for the monster he's created.

Sat Apr 15 2006 00:00: Sumana just mentioned levels of referentiality. Among people who weren't deliberately aiming to maximize levels of referentiality, who has achieved the most levels and how?

Sat Apr 15 2006 00:05 Dangerously Low-Budget Search Requests: clorox bottle easter baskets. Alert MAKE Magazine!

(5) Sat Apr 15 2006 13:20: I realize that this is my third entry on the topic, but the Cookbook is now 100% done and submitted.

Sat Apr 15 2006 23:25: Rainer Böhme sent me a copy of his stock spam paper, which is really interesting (I'll link to it when it's put up publicly). It demonstrates two results I found interesting: on average, stock spam does increase the price of a stock, but the effect decays and after four days the stock price is lower than before. Second, the greater the volume of spam you send, the bigger the effect.

Entering the realm of speculation, there's also an indication that stock spam actually destroys wealth by reducing the capitalization of the spammed companies over the long-term (though maybe the wealth just leaves the stock market and goes into the spammers' pockets). Also includes a movie-plot scenario where spammers use the techniques of stock spam to start runs on banks. An entertaining paper. And it mentions me, which is awesome.

Sun Apr 16 2006 14:28 Teeth crock, clipped less clumps sal: The Eater of Meaning can now eat poetry as well as prose, with the new "chew rhythmically" feature. It replaces each word with another word of the same number of syllables. It rips off Danny O'Brien's haiku finder, which in turn rips off of someone else's syllable counter.

Inspired by Kevan. The stress of the replaced words is often wrong, and sometimes the result just doesn't scan, but it's sufficient for the secret also-Kevan-inspired project for which I wrote it. It works especially well on cadences that have been drummed into your head for years.

Sun Apr 16 2006 22:28 SAFA Crosspost: Not the best one ever, but some funny stuff in there.

(2) Mon Apr 17 2006 18:19 A Coney Island of the Body: Today I went to Coney Island. I saw a horseshoe crab.

Mon Apr 17 2006 18:23 PSA Search Requests: Q: What should i do with my rabbit because my rabbit cannot walk on his hind legs?

A: Leave your rabbit alone!

Mon Apr 17 2006 23:34: BibliOdyssey is awesome. Check out this old engraving of Ganesha. Or, as they used to call him, Quenavady.

Not only does it find great pictures but it links to things like this summary of really old travel books (summary contains durian), a genre that has always fascinated me. As you travel further, your reports back home get less accurate.

Tue Apr 18 2006 09:10 Beautiful Soup Poll: Would people rather that Beautiful Soup automatically converted documents to UTF-8 (including modifying the declarations and META tags)? Or would they rather have Beautiful Soup leave the encoding alone, but detect it and automatically convert specific strings to UTF-8 on command?

(1) Tue Apr 18 2006 13:33 Less Cat, Less Girl: I finally did a Cat and Girl comic using Kevan's constraints. It's not that good, but fortunately quality was not one of the constraints. The constraints were:

• The first line of dialogue should be taken from a recent entry in a weblog you've read today. This was a very difficult constraint because it was so wide-ranging. In the end I just picked a recent sentence from Kevan's weblog.
• The strip should contain a complete recipe (simplified as much as you like, but enough that someone could reasonably prepare it from those instructions).
• Neither Cat nor Girl should appear visibly in the comic.

(31) Tue Apr 18 2006 20:50:

Sumana

We see love between others thousands of times but only participate in it a few times. Can we hope to find a new kind of love? How would we know it if we did? The best we can do is to explore the deeps of one great love, always finding new aspects of it.

Five years ago I stumbled upon a vast love which my lamp and map have still not charted. This is where I want to live, Sumana, among the crystals, the faraway noise of an underground city, the glimmering lights in the ceiling, with you. Marry me, Sumana.

Wed Apr 19 2006 19:32 Well, Nobody's Going to Top That: But today O'Reilly said they were interested in my book proposal, which is awesome. I have to go back to Bakersfield for a while to take care of my mother, so I'm probably going to write that book while I'm there.

(15) Thu Apr 20 2006 23:52: I've been thinking about the best way to tell people about voting machine fraud without sounding crazy. I have had success in telling people about it, but only with people who were themselves kind of crazy.

But the success of 24 proves that you can get people to incorporate crazy political ideas into their worldview by presenting those ideas in story form. So I had an idea for a political thriller TV show. The elevator pitch is 24 meets the last season of The West Wing. It starts with a third party senator (like Jesse Ventura, but more politically savvy) who decides to run for president. He starts out wanting to act as a spoiler, but then he meets a programmer who works for the company that makes the voting machines, and he decides to win.

At the end of the pilot, you've got got two overlapping stories. The senator tries to make his spoiler campaign look like something that could really win the election, and the programmer works to form a conspiracy within the voting machine company. Not only does the conspiracy have to cover their tracks, they must form an ad hoc political organization, changing their voting machine hacks to deal with downticket races and otherwise make the voting results look realistic, without having any contact with the real campaign.

This can go for half a season or a full season. There are twists and turns but the two teams win/steal the election. Then the real fun begins. The conspiracy is discovered and publicised, but it's too late.

This will never be made into a TV show, mainly because nothing I ever say will be made into a TV show, but also because all American TV shows seem to have a premise that can extend, in theory, for five seasons. I don't see any way to make a presidential election and a constitutional crisis run more than two seasons.

(7) Sat Apr 22 2006 07:59 Bridezilla and Groomera: Yesterday Sumana and I got married! Yes, we did. Here are all the details.

We got married in the Shakespeare Garden in Central Park. Nandini, Andy, and Camille were there. The officiant was the Reverend Tony Johnson. We read our weblog proposals to each other, and also poems. I read Shakespeare's Sonnet 116, and Sumana read Yevgeny Yevtushenko's "Colors".

Then we were married! It was awesome. We devoured snacks that I brought: pears, pear-based muffins, and brownies. Camille left to go to work, but Nandini, Andy, Sumana and I went to Belvedere Castle and looked down at the turtles in the turtle pond.

So far we've gotten two gifts! (But don't send us more gifts; see below). Andy's family gave us a great new Shun knife with a caricature of Alton Brown's face on it--a knife that claims to be dishwasher safe! And Brendan and Maria sent us a bamboo steamer, in reference to when they were in New York and I went with them and a friend to Joe's Shanghai in Flushing and ate juicy dumplings.

So, instead of gifts, we would like you to send us advice about marriage. And nice letters. Sumana says that if you want to spend money, you should give to one of the charities Steve and Alice listed when they got married, starting here. Also, Sumana needs a new Brita pitcher and a bookcase that folds up--but don't tell her I said so!

Nandini and Camille took pictures, so soon you'll be able to see pictures.

(2) Sat Apr 22 2006 12:23 Wedding Pictures: From Camille. Try this one and also this one, where I'm on the phone with Sumana's mother.

(10) Sun Apr 23 2006 18:58 Beautiful Soup 3.0 Beta: For your delectability. The major new feature is that Beautiful Soup 3.0 takes XML or HTML documents in any encoding and turns them into UTF-8; in most cases you don't have to know the current encoding. I wrote this without really knowing anything about encodings: most of the code is stolen from Mark Pilgrim's Universal Feed Parser. But I am able to write tests, and the tests work.

The other major new feature is that you can now rip out a chunk from the parse tree with the extract method. You can use the chunk and abandon the rest of the tree, or vice versa. This is especially useful because the data structures you abandoned can now be garbage-collected: in current Beautiful Soup, the whole tree stays in memory forever because every Tag and NavigableText is connected to every other Tag and NavigableText through an intricate web of lies. And by "lies", I mean "instance variables".

There are some more new features, but I have to take a shower now to go and meet Pete Peterson II for dinner. Test it out; I'll be rewriting the documentation over the next month or so, and hopefully by then I'll have gotten enough feedback to release it.

Sun Apr 23 2006 23:33 Just When You Thought It Was Safe To Not Look At Cute Baby Elephant Pictures: Cute baby elephant picture!

(5) Mon Apr 24 2006 13:43: OK, the Beautiful Soup beta now stores everything internally as Unicode. You specify an encoding when you turn part of the tree into a string. Now I'm definitely going to work on my book outline.

Tue Apr 25 2006 08:26: Going to California pretty soon. Good news: according to the O'Reilly web site, the Ruby Cookbook will be released in July, not August.

(2) Wed Apr 26 2006 12:45: The stock spam paper is online.

(2) Thu Apr 27 2006 21:32 PST: Buncha page scans of old sci-fi books, with great titles like "The Cosmic Computer" and "Edison's Conquest of Mars". I always get disappointed by these books, though "Cosmic Computer" does mention "the collapsium-shield death-to-people area". Now that's quality technobabble.

(2) Fri Apr 28 2006 20:30 PST: I am in Bakersfield taking care of my mother and it is stressful. Today I got a little freelance work done though.

I heard from Michael that the Ruby cookbook is in the neighborhood of 900 pages. That's as big as the second editions of the Perl and Python Cookbooks, which makes sense because it's got about as many recipes. We just didn't know before because the wiki doesn't automatically generate the Docbook output that gets printed on pages.

900 pages! I wrote a Neal Stephenson book! (Sumana: "Does it have an ending?" Me: "No, it just kind of ends." Sumana: "Yup.")

Fri Apr 28 2006 22:11 PST: The I Ching calculator from Dirk Gently--online! I always pictured the error message as being displayed in the same font as the rest of the calculator output instead of a wacky Comic Sans-meets-Dingbats font, but Marvin in the Hitchhiker's Guide movie didn't look like I expected either, so I'm accustomed to the bitter Arcturan Mega-gin of disappointment.

Fri Apr 28 2006 22:16 PST: Most book reviews are only in passing about the books they're reviewing. Usually I find this annoying but this "review" of three books on evolutionary development is a good introduction to a topic I didn't know the first thing about. Now I know the first thing but not the second.

(4) Sat Apr 29 2006 09:54 PST A People Kill B People: Last ninja: 'Be able to kill your students'. Wouldn't that result in a strictly decreasing quality of ninja over time?

Sun Apr 30 2006 20:42 PST: Newest Beautiful Soup beta solves everyone's problems that I know of. Except my problem of having to rewrite the documentation.

(1) Mon May 01 2006 08:09 PST Vacuum Diagrams: At last, sci-fi that fills my need for creepy speculation about the creepy Great Attractor.

(1) Mon May 01 2006 10:24 PST Coconut Rice Pudding: I made this for my mother. It turns out you can cook rice in coconut milk and then it tastes like delicious coconut. Then you can make rice pudding out of the leftovers and it's good. I used the James Beard recipe.

(3) Mon May 01 2006 19:52 PST: Inform 7 looks awesome, as promised, but I can't use it because it's a Windows-and-Mac-only IDE. But this is actually good because I shouldn't be writing IF right now. Sure makes me want to start again though.

(1) Tue May 02 2006 21:19 PST Buttons: Going through stuff I've been storing at my mother's house that I'm going to have to take to New York or get rid of. Among these things is a big pile of buttons from computer trade shows in the early 80s. I got these from my dad, and I seem to remember culling them from a larger set which is now lost. Once I get a digital camera I'll take pictures of them for you. Some of them are really weird ("SAMPLE MY ATTRIBUTES", "HOW CAN IT BE?").

Wed May 03 2006 22:33 PST: The power went out two days in a row. Yesterday it was just us and a couple other houses, but today it was most of Bakersfield. This is not a good time, emergent behavior!

Thu May 04 2006 19:11 PST: The Frogs, my favorite Greek play, has an ongoing webcomic adaptation that's pretty good. Same comic also has a transparent Pogo-esque satire using Winnie-the-Pooh characters. From Sumana.

Thu May 04 2006 19:31 PST: More webcomics: Keyword Cartoons is in some sense the counterpart to Spamusement: high-grossing (but not engrossing) AdSense keywords turned into cartoons. The... character... of the main character fits this concept exactly.

Thu May 04 2006 20:19 PST: On the back of an old pay stub from 1996 I found some FTP sites I'd visited in the last days of the BBS. ftp.uu.net amd ftp.cse.unsw.edu.au are still around; ftp.digex.net and macbeth.cogsci.edu.ac.uk are gone. What do these random hostnames have in common? Well, three of them are mentioned in the Roguelike Games FAQ. Looks like I wanted Larn and the PC port of Rogue.

More miscellany coming soon; I typed up my mother's annotations to her copy of the James Beard Cookbook. I'll put that on a separate page instead of in NYCB.

Thu May 04 2006 22:36 PST: By the rings of Earth!

(3) Fri May 05 2006 19:18 PST: My mother died early this afternoon. We're writing on her weblog about her last days.

(3) Mon May 08 2006 22:00 PST Postcards from my Father: We've been sorting through my mother's archives, planning to digitize everything. Hopefully we will actually do it and not leave it to the next generation.

After a long morning we thought we had it all done. Then I opened a cupboard and there were about ten boxes: "Rachel's Schoolwork", "Leonard's Postcards", "Susanna's Certificates", etc. etc. We let out a collective groan.

But what a find. "Roy's Postcards" fill up two boxes. There are more than 500 of them, and the vast majority were never sent to anyone. My dad was a compulsive postcard buyer. He bought them for himself, and wrote on each postcard a description of how he got it, or what he'd done that day.

In our excavations we have encountered many of my dad's journals. Usually he'd only written on the first five or ten pages. But these boxes contain a journal going from 1982 to 1991. My dad could keep a journal going; he just needed the right medium. He needed a LiveJournal.

The vacations, museum visits, and other memorable moments of my youth are especially well-represented in the postcard box, since that's where he usually got the postcards. So on balance, I'm glad we found all those extra boxes of stuff that needs to be scanned.

I haven't even mentioned "Leonard's Postcards", which include many examples of me-learning-to-write hilarity such as this one from 11/23/1984. In this piece, I start off following the rules of a mailed postcard, but as space runs out I get more and more desperate, finally piling all the letters atop each other in the corner like a typewriter that won't line-feed:

TO DAY WE WENT
TO DISNE-
YlAND
ANDWE WENTOBIG THUN-
DER MOUNTAI
NANDIDIDNOTL[big jumble of letters]

(1) Wed May 10 2006 22:47 PST: Today my mother's funeral service. She planned the service, which means we sang all the crazy Mormon songs no one ever sings, and then there were delicious ribs.

I think that Sisyphus, Brendan's Anacrusis from May 4, is apropos.

(2) Tue May 16 2006 20:18 PST: Back in Bakersfield from Utah burial. Scanning and packing stuff night and day. Am currently working on a binder of Whitney letters from the 1940s and 1950s. They had a really wide variety of paper quality back then.

Continuing the tradition unwittingly started by my postcard panic entry, I give you an e.e. cummings-esque letter sent by my mother to her aunt LeJeune:

                        oct ober 12  1956
                           o                d
                                             ear jeuny

                thankyou for the new c clothes
An ne likes the shoes and hat


            very much
                      so we hunng them up high.


                            love frances

(1) Thu May 18 2006 23:29 PST: "Even time travel's more expensive than it used to be."

Mon May 22 2006 20:39 PST: Back in New York. Over the course of my three weeks in Bakersfield I mentally moved back to California, so now I need to readjust to New York.

Sorry, nothing else going on.

Wed May 24 2006 19:39 PST: Kevin Marshall, author of some Ruby Cookbook recipes, has an O'Reilly PDF booklet out about web services in (on) Rails.

Wed May 24 2006 19:48 PST Then, you bait him!: Today I did a dancing bear dance which Sumana found very entertaining. You just dance the way you imagine a bear would if someone was making him stand on his hind legs and dance. Making bears dance: mean. Pretending to be a bear who dances: fun.

(2) Fri May 26 2006 08:05 PST The Big U: Neal Stephenson says people shouldn't buy The Big U. I bought it anyway (it was at the used bookstore for $3). Sumana said I shouldn't read it. I read it anyway. I can see why Stephenson doesn't want people to buy it, but it was pretty fun. It reminded me a lot of a Gordon Korman novel (cue more anger from Sumana), only much more violent. I liked the brain reprogramming mechanism a lot better than the one in Snow Crash. The main problem was the extreme heavy-handedness. I think it would work well as a "young adult" novel a la Korman; I remember heavy-handed satire working better on me when I was a "young adult". Now, I can only find solace in the opiate of scare quotes.

Sat May 27 2006 10:11 PST: "These books had such long titles! If you wanted to buy a book in the eighteenth century, how did you ask for it?"

"I want the book about 'Being A'... and 'Most Excellent'... and it has woodcuts."

Sat May 27 2006 23:02 PST: Beautiful Soup 3.0 release coming soon. Big ole book of documentation. Get ready!

Sun May 28 2006 16:02 PST: I declare Beautiful Soup 3.0 released. Here's the changelog. Go to town.

Sun May 28 2006 21:52 PST Gnath: Check out the beak on Gnathosaurus.

(4) Mon May 29 2006 13:50 PST Beautiful Soup Trickery: Here's a nice bit of trickery I discovered, based on code sent me by Staffan Malmgren. A common task among the target audience for Beautiful Soup is to strip HTML tags from a document. It turns out this is a one-liner:

''.join([e for e in soup.recursiveChildGenerator() if isinstance(e,unicode)])

Where soup is your soup object. I really like the if conditional in list comprehensions; I don't think Ruby has anything like it. I can think of ways to simulate it but they kill its simplicity, which is what I like about it.

Tue May 30 2006 11:47 PST: I fixed a bug, reintroduced an old shortcut, and the result is Beautiful Soup 3.0.1.

Wed May 31 2006 22:58 Disturbing Spam Subjects: BOB examined ANDY in DANIEL

Short entries these days, because I'm spending all my time writing a story.

(5) Wed May 31 2006 23:08 I Have Arrived: Sam Ruby and Mark Pilgrim endorse Beautiful Soup.

Thu Jun 01 2006 23:11: Excellent report on what's available at the Union Square Greenmarket. Needs an RSS feed.

Fri Jun 02 2006 12:50 Beautiful Soup 3.0.2: No real user-visible changes; it should do what you expect, more reliably. I did add back aliases for the old method names, so your 2.x scripts are pretty likely to work again now.

Fri Jun 02 2006 18:30 Mushy Applesauce: I make up random pet names for Sumana. Sometimes there are namespace collisions.

"You're the apple-finder!"
"Wait, I'm the Apple Finder?"

(1) Sat Jun 03 2006 09:52 Weird Fact: Emacs has no built-in word count function.

(1) Sun Jun 04 2006 11:41 Brick: Sumana and I went and saw Brick last night. Actually we went to do some unspecified thing in the East Village, and while perusing the Voice (<--aaaaargh) we saw that close by was the only theater in Manhattan playing Brick, so that's what we did.

This is a good movie but, pace Brendan, not the best ever made. The problem is that while it's stylistically excellent, it follows the noir mystery formula too faithfully. The best noir mysteries have a non-stylistic twist on the formula (The Big Lebowski, which for my money remains the king of stylistic twists as well). The best noir movies abandon the mystery element altogether and just go after the depravity of man (Double Indemnity). Or you can just be an extremely confusing mystery like The Big Sleep, which gives you some points in my book. In between you have movies like The Maltese Falcon and Brick, which are very stylish but formulaic.

I think the other major noir subgenre is the underworld film, but I haven't seen any of those yet, so I'm even less able than usual to pass ill-informed judgement. Those movies seem kind of sensationalistic and Manichean, but I could be wrong.

(5) Mon Jun 05 2006 09:50: Frickin post office lost eight chapters of my Ruby Cookbook edits.

Tue Jun 06 2006 23:12: Bonus: Beautiful Soup 3.0.3

Tue Jun 06 2006 23:57 Anacrusish #5: [This is the entry to which the previous entry was supposed to be a bonus.] While I wait for feedback on my story, here's another Anacrusis-type story for you.

Phyla

"We are the whales," says Phyla. "the ocean's voice. Keepers of the deep eternal secrets." She strokes the fishbowl of warm seawater.

"We want krill," she says, surprising herself. "You have boats. Go north. Bring us krill." Even this crowd hoots and leaves. A few stay.

"Prepare to be boarded!" says Phyla, still translating. "Drop nets, me hearties!"

"Good job," say the whales. "Now..."

Then, Phyla gets another radio in her head: one eye's made of crystal. "Fishing craft, change course ten degrees."

"I know who you are," she says, "and why you're here."

To her crew: "Load the cannons."

Wed Jun 07 2006 20:38: Great paper on detecting spam web pages. Factoid: about 70% of the sampled .biz sites were spam.

Fri Jun 09 2006 20:49 Parrot: Sumana was talking about parrots and I made a penguin noise, which is really the only kind of bird noise I know how to make.

"Parrots don't make that noise!" said Sumana.

"They do if they're around penguins."

Sat Jun 10 2006 18:13 MOCCA: We went to the MOCCA indie comics show today with Sarah. We met many people including Ryan North, and spent way too much money on comic books. On Monday something even more awesome will happen.

We met Dorothy of Cat and Girl, who was as shy as I would be if I had written an amazing webcomic and had to stand around while people fawned over me all day. WHAT I AM SAYING IS THAT I SYMPATHIZE.

We wandered for a while (usually but not always a good strategy) and found a good vegan pan-Asian restaurant and a chocolate-obsessed store. Now we are admiring our huge stacks of comic books.

Sat Jun 10 2006 19:01 Uncle Leonard's Dub Shack: Hey, Uncle Morty's Dub Shack is showing new episodes again. I meant to warn you earlier, but I was too busy buying comic books. Today they showed the episode in which Sumana and I were extras, and you missed it. Unless you've been Tivoing it, you'll have to wait until it gets rerun, which I'm sure it will be indefinitely.

Here's a still Jimbo sent us, though not from footage that was actually used. I wonder if this qualifies us for IMDB entries. I'm only credited as "Leonard".

I recommend you watch it if only to use our gaudy dress as markers for the amazing continuity errors. We warp around the bar, appearing in almost every scene because we were the only extras who showed up on time for the shoot.

The episode itself is best summarized by a Trevor line: "That's the second-best song I've ever heard about big butts!" Later I'll tell you about a funny scene that they cut.

Sat Jun 10 2006 19:28: Earlier I speculated that you could improve any game by replacing the human characters with cars. The movie Cars demonstrates that this does not work for movies.

Sun Jun 11 2006 21:53 One Night Only: On Tuesday I go up to Worcester to start a band with Jake Berendes. Thursday night the band plays a set in Nicholas Reville's basement. Nobody will be there--will you? This amazing musical event will probably NEVER BE REPEATED in the near future!

PS: our band is called "early internet".

(2) Mon Jun 12 2006 18:44: Today I hung out with Ryan North! Sumana and I had lunch with him, his SO Allene, and friend Steve. Sumana went back to work but we wandered around and ran into more web comics fans and creators, including David Malki!. Verdict: awesome. Here are a couple pictures.

Thu Jun 15 2006 02:21 Never Spent A Single Day In Retail: Until yesterday, when I worked with Jake at Happy Birthday Mike Leslie. Not a whole lot happened, but I made some buttons and then decided to improve the sale value of Jake's junk. I turned a Parcheesi board into a crackpot diagram of the universe (which is a rotating twelve-dimensional hypercube). Today I hope to improve the Ouija board.

Then we went to karaoke, which was fun. Jake and I did a duet on "A Boy Named Sue". Karaoke tip: if you're going to sing "Piano Man", save it for the karaoke contest.

(3) Mon Jun 19 2006 23:47: Back in New York. More later.

(2) Tue Jun 20 2006 10:56: J. Michael Straczynski's rejected proposal for rebooting the Star Trek franchise. Gorgeous.

(4) Tue Jun 20 2006 23:16 Extra Spatial Dimensions: I have a question about these little buggers. Kris couldn't answer it, so you know it's tough. Whenever I heard an explanation of string theory or any other theory that predicts more spatial dimensions, they always said the extra dimensions were "rolled up very small". That doesn't make sense. Rolled up very small through what dimensions?

Then I think I figured it out with a Flatland analogy. Suppose Flatland exists in a 3D universe, but one in which nothing is more than a Planck distance tall. The Flatlanders extend in three dimensions, but they only perceive two. They would say that the third dimension was "rolled up very small", even though that doesn't make sense.

Wikipedia seems to back me up on this one, ladies, so the first question is, am I right? Does the analogy hold? Second question is, what is with the "rolled up" nonsense? Are they trying to convey that the structure of space makes it impossible for something to be large in that dimension ("the universe measured along these additional dimensions is subatomic in size")?

(4) Wed Jun 21 2006 19:08: Nightmarish tips on flag design. Nightmarish because they present terrifying flag design faux pas I'd never even considered, like putting different things on the front and back of your flag, or including other flags on your flag so that you're always changing it. Gah!

Thu Jun 22 2006 17:02 Here Goes Nothing: I submitted my short story to Futurismic. Who knows if they'll like it, but at least I can work on other stuff now instead of always trying to polish it.

Fri Jun 23 2006 09:12:

"I got a map of the Pacific Ocean."

"I don't like maps of oceans. They're too big. I thought you knew that."

"Well, what if it's like a MAD fold-in? See? Now you can walk from Idaho to Japan."

"That's better."

(2) Fri Jun 23 2006 11:46 Strapping Young Men: Jake Berendes has two guitars but only one guitar strap. This posed problems when we were practicing for our one and only gig. He ingeniously made a new guitar strap by cutting up a bicycle inner tube. It felt great on the shoulders, really light. Take that, MAKE Magazine!

Then we practiced and played the gig (which went well), and the next day my left hand hurt. The elasticity of the inner tube that made it so easy on the shoulders made it very hard on my left hand, which had to support the neck of the guitar as well as do the fingering. Take that, people who reflexively build anything they see described on a weblog! Fortunately, the pain is now mostly gone.

If we can figure out a way to do that kind of strap that doesn't stress the left hand, we'll have it made. It was really comfortable until I stopped using it.

(4) Fri Jun 23 2006 16:47 Comic Books: When I was a kid I never read comic books. My uncle Don (I think) once gave me a box full of comic books for Christmas. Now that I think about it, that was a really creative gift, but that box just stayed untouched in my room for about six years until I was about to move out, when I gave it to a friend.

Sumana likes the highbrow comic books they put out nowadays. I've read some of hers and I didn't like those a whole lot either. So what does picky me like? Well, I like ACKs. I like books of comic strips. Though I don't like the comic books themselves, I love the insane crowdedness of the Marvel and DC universes, caused by the respective companies having to crank out twenty books a month for decades.

Jake has a bunch of comic books, and I trust Jake's taste, so when I was in Worcester I decided to give the traditional comic book a try. Verdict: not great. The plots are pretty dull and the dialogue is either stilted or fake-casual. The drawing's good but I don't really notice that kind of thing.

Highlights:

• The community of comic book creators, as revealed in the books, is interesting. They dish dirt on each other, make up ridiculous retcons and silly nicknames. It seems pretty tight-knit and full of love-hate relationships.
• The Green Lantern comics are a lot like really old science fiction, especially the Lensman novels. I think those stories work a lot better as comic books. But I wasn't crazy about the Lensman novels and I don't want to read a lot of Green Lantern either.
• Silver Surfer has a ludicrous story and some of the most stilted dialogue I've ever seen, but it's really compelling. I think this is what comic books feel like to big fans.
• The Nethack phrase "undergo a freakish metamorphosis" shows up a lot in comic books. I don't know what the original source is.

But the big highlight was Jake's set of Ambush Bug comics, which were everything I want comic books to be. Unfortunately, Ambush Bug can only exist in the context of everything I find boring about comic books.

Ambush Bug is if Checkerboard Nightmare lived in the DC universe and thought he was Superman's pal. He had bit parts in Superman comics (including one where he kind of casually figures out Superman's secret identity), and then two mini-series of his own. Which actually ended the same way Checkerboard Nightmare ended. But which mainly focused on the weirdness of obscure comic book characters and the nature of continuity. It's really funny stuff and I recommend it. The miniseries also has parodies of those comic book ads that sell junk that doesn't work, which aren't that funny, but you can never have enough parodies of those ads.

Sat Jun 24 2006 23:52 A Prairie Home Companion: A fun movie, not mind-blowing, about what you'd expect. Garrison Keillor is self-deprecating and deadpan funny and writes the worst parts of his personal life into the script. The singers are good, the plot happens, there are dirty jokes. Oh, the penguin joke is in this movie, too. It's good to see in the theater, but with prices the way they are these days you might want to wait.

Sun Jun 25 2006 00:29 Lucky Wander Boy: I've had this book for a long time, but I held off reading it because I knew it had thematic overlap with the story I just finished. I didn't want to accidentally plagiarize it, or always be second-guessing myself about similarities.

Now I've read it and I'm happy to report that there's not much overlap. It's true that these are the only two pieces of fiction I know of that use MAME as a dramatic device, but the purpose is totally different. I would actually say this book is closer to Frog/Antifrog.

Okay, about the book itself. The middle is a Douglas Coupland-esque slog, but the beginning is fun and the end has a twist that I should have seen coming but didn't. Good job--the best kind of twist. Plus, there's a tiny piece of exposition hidden in the book that makes the central reveal semi-plausible sci-fi instead of squishy novel mysticism. I think many people who didn't like the ending missed that piece of exposition. But I say, thanks for including it, D. B. Weiss.

Mon Jun 26 2006 07:42 Fascinating! Educational!: Reproduce trademarked characters and sell them! via Kevan.

Mon Jun 26 2006 23:24 Nobody Expects The Game Roundup: But here it is.

• Flatland goes to war in Dangen, the geometric shoot-em-up. Minimalist Galaga-like sharpshooter action. Fun and elegant.
• Less elegant but more three-dimensional-looking is Triplex Invaders, featuring hot triplex action. Everything is rendered in real-time as big Dactyl Nightmare-style polygonal shapes. Cool Ghostbusters-like beam weapons add to the fun. Lousy projectile weapons subtract from the fun. Can this game achieve a fun surplus?
• Lämmerlinge is a fun Javascript game where you guide sheep around a board, feeding them flowers and preventing them from wandering off or drowning. Documentation is both in German and totally superfluous. Levels are password-protected: also superfluous since the passwords are right there in a supplimentary Javascript file.
• Deadly Rooms of Death is a fun shareware puzzle game with a video game sense of humor. This is not a sophisticated sense of humor, not an interactive fiction or Ealing Studios humor. Maybe a ZZT game humor. If you saw it in a sitcom it would be undistinguished, but so few video games these days have any sense of humor that DROD's juvenile antics are charming. DROD has lots of puzzle elements and scriptability that make it more enjoyable than Rocks 'n' Diamonds. Fun voice acting separates this game from free software games with no voice acting.
• Trackballs (Debian: trackballs) captures the sleek futuristic look of Marble Madness but seems really obsessed with narrow passageways for your marble to navigate, instead of the sweeping ski-slope wireframe mountains of the original. On the plus side, your marble can jump, and the game levels are scriptable with Guile. So make some levels that don't rely on split-second marble manipulation cause I don't do that. Oh, this game also has enemies that you can squash with your marble. Unfortunately this does not make your marble bigger a la Katamari Damacy.
• How's that crackpot scheme of yours to mine Mars for mercy and sell it back on Earth? Not going too well? Maybe you should have played the cautionary game Mars: Land of No Mercy. You'll find out that anything that lands on the surface of Mars can only move once and then has to wait for the opponent's move, which never comes. Basically the game is still in a very early stage. Nice realistic Mars graphics though.

Tue Jun 27 2006 06:56 ActiveResource: Excellent.

(1) Tue Jun 27 2006 08:14 Game Roundup Addendum: I didn't mention this yesterday because I wanted to go to sleep instead of playtesting, but Battle for Wesnoth is a really good turn-based strategy game. Simple with good presentation. The only downside is the cheesy fantasy art in the cut scenes.

Thu Jun 29 2006 23:49: Sorry for no NYCB. I'm in the early stages of working on another book, as you can probably tell if you watch my del.icio.us feed. Also writing an article and fiction. Why should I waste my time with this puny weblog? My atomic grapefruit will split it asunder, revealing the iron-rich core within!

(1) Sat Jul 01 2006 08:59 Recursive Love: I got one of those "I ♡ NY" plastic bags when I bought some cheese. But inside the heart was the word "Love", for those who speak English but don't know the connotation of a heart. But the word "Love" was itself spelled with a heart: "L♡VE". So why stop there?

Sat Jul 01 2006 10:28: Went with Sumana and Adam to a party last night which was way too loud. But we did hear a story about a theme restaurant in Manhattan called Ninja New York. It sounded like a very expensive Chuck E Cheese for adults. I say "adults" only because it serves this weird Asian fusion food that I suspect kids would hate. The New York Times hated it.

Confusing the point of a restaurant with the mission of a "Saturday Night Live" skit...

In the name of "new style sushi" Ninja employs rice cakes as beds - or sometimes graves - for a rectangle of truffle-flecked omelet (it tasted like soggy French toast), a sliver of sautéed foie gras (pleasant, but how could it not be?) and a finger of seaweed-crowned mackerel (fishy in the extreme).

Adam thought this was orientalism at its finest, but this restaurant is an offshoot of one in Tokyo, so there's clearly some self-orientalism going on as well.

The only cool part: the restaurant is camoflauged. It looks like an apartment building and you go down in the elevator and the door opens to reveal a ninja standing in front of you. They should cut the dining experience short right there and call it "Be Scared by a Ninja For $5".

Sat Jul 01 2006 12:14 Blurb Brag: Oh, I don't think I mentioned that the Ruby Cookbook has back-cover blurbs from why and Matz! You can see them both at the Ruby Cookbook Official Unofficial Homepage.

(1) Sun Jul 02 2006 22:39: Pictures from my trip to Worcester. Includes photos of the game boards I "modded" for Jake's store. Includes picture of me and Jake together, which will not convince Kevin that Jake is a real person.

Mon Jul 03 2006 22:03: Check out these musical notations. Cornelius Cardew's notation looks like the Ferengi language.

Tue Jul 04 2006 20:53 And what did I see?: why has devised hpricot, a permissive HTML parser with tree-crawling functions. It's written in C, so it's fast. Just another great piece of software named after apricots.

(2) Wed Jul 05 2006 22:50: Today I realized that we've lived in New York for six months and I haven't put up any pictures of our neighborhood in Astoria. Then I realized that this was because I didn't have a camera. But now I do, though it's falling apart and doesn't take pictures half the time. So yesterday when I was walking around I took some pictures. Since it was the Fourth I paid special attention to flags. Here are the pictures. I tried to get one of the statue of Athena, but that was one of the pictures the camera didn't really take.

(1) Thu Jul 06 2006 22:35 Early Birthday: This is my first birthday, I think, where I've really become obsessed with the mortality symbolism of the birthday. So bah to that! Let's concentrate on PRESENTS. We hung out with Andy today and he surprised me with a Not For Tourists guide to Queens (not that Queens is really a big tourist destination anyway). And Sumana presented me with an awesome print of The Death of Jennifer Sisko! Susanna sent me something I don't know what. Yes, physical objects will keep my mind off the eventual termination of my physical existence. Wait, how is that supposed to work?

Fri Jul 07 2006 23:40: Here's a great lecture series on the Byzantine Empire.

Sat Jul 08 2006 12:13: I improvised a song for Sumana today, which was pretty good (cf.), so I recorded it. It's called I am a Cowboy.

Sat Jul 08 2006 12:27: How can we stimulate production of awesome deep-sea photography? Hold a contest! Pictures must be of the high end of the ocean depth gradient, which as is well known is correlated with awesomeness.

(2) Sun Jul 09 2006 22:16: Common IF author mistakes. Alert: alotta alliteration allocated afore all asterisked areas.

(1) Sun Jul 09 2006 23:00: I wasn't planning on doing this until later, but I made myself a birthday cake today.

Mon Jul 10 2006 17:10 The Answer Is Yes: "I pity the fool who has to ask me if I still pity the fool." From Joe Mahoney.

(1) Mon Jul 10 2006 23:25 A Game Divided Cannot Roundup:

• Overgod looks like one of the games that comes built into your Hitchhiker's Guide. It's a visually appealing 2D space shooter where you're free to move about the arena shooting at other ships. It's a lot of fun, but would be more fun if your ship weren't so slow to maneuver.
• Linux Shuttle Lander is a frustrating take on those frustrating "lunar lander" games (in fact, LSL author Thoma Raes also wrote one of those), where you have to land the fuel-less Space Shuttle. The viewfield is cleverly centered on the piece of the sky you just went through, so that you don't see the ground until a split second before you hit it. It doesn't end well. Let's hope the real Space Shuttle fares better. At least the game code is reentrant.
• Mega Mario, not to be confused with Molto Mario, is a clone of the Super Mario Bros. engine. Which one? Well, all of them, sort of. The graphics and sound are taken from the three NES games and the SNES Super Mario World, which gives a very weird effect. You haven't lived until you've seen SMW Mario stomp a SMB1 Goomba. Ironically, the Goomba has lived right up to that point. In what was surely the defining feature of this software project, Mega Mario features monochrome red blood spurting in precise arcs out of you and the creatures you stomp.

  The mechanics are kinda off: for instance, your Mario is just slightly too big which means that if you don't position yourself just right you'll punch two bricks instead of one. I don't generally make a big deal out of these little issues but c'mon, it's-a me, Mario! The Mario games are classics because of their polished, flexible gameplay. Anyway, the level design is the big problem here. It doesn't have the cleverness of any real Mario game. There are just mazes and coin bonanzas. I've seen the same thing in ZZT games and Rocks 'n' Diamonds boards.

  Speaking of games that rip off the Mario sprites, Super Mario PAC looks like fun but I can't play it because it's a Windows game. I think it's a game where Mario forms an organization to buy off Mushroom Kingdom politicians.

• On a very special Game Roundup, it's Pango: Mudbath, a pygame game by Python guy Moshe Zadka. You're a penguin and you wander around an unobstructed 2D field shooting at squares. Because squares are getting you down, man. Pretty good start on a game, but the name is too similar to the arcade classic Pengo.
• 304. This card game has not been modified.
• Paratim is a clone of an old DOS game called Paratrooper. Unfortunately, the original is totally superior in every respect except for the DOSsy circumstances of its birth. The original was a game that played the Toccata and Fugue in D Minor--not the Phantom of the Opera part but the part that sounds like a helicopter--and then tries to kill you with helicopters. This game can't live up to that.
• A while ago I rounded up a game called Tower of Doom, which it turned out had changed into a totally different game called Warp Rogue. Despite the promising name the game is not that good as a Roguelike. There's no exploring, only tactical combat and corpse-looting for the resale value. It does have a nice advancement system (you spend experience points on new abilities and improved attributes).
• And now what I hope to be a new feature of Game Roundup, Interactive Fiction Roundup. I haven't played any IF in years and I have lost track of the "scene" assuming it is the kind of thing that has a "scene" and not just a room with a description and geographically-defined exits. But the release of Inform 7 rekindled my interest, so I'm working to catch up on what people have done since, say, 2001. The results of my working will make it into future Game Roundups.

  And this one! Today we have Chris Klimas's Blue Chairs, a 2004 game that is kind of a modern version of IF chestnut A Fable. Fable was not even a real game, only part of a game, written by Stan Heller in 1985 to test out GAGS (game development precursor to AGT). Somehow it got included as one of the sample AGT games, which was a bad idea because from a game design perspective it's not good (even by AGT standards). I liked the writing though. You wandered around a landscape experiencing vignettes from your past.

  Anyway, the game I'm actually reviewing. Blue Chairs has all the good aspects of Fable, but it also has a real (surreal) plot, good NPC interaction, and really good descriptions. It mixes linear and nonlinear elements well, with many endings that are nonetheless easy to all see with a strategically placed saved game. It's also got this (spoiler), and the near perfect example of using characterization as a puzzle element.

(3) Tue Jul 11 2006 18:30 Systemantics: The working epigram for my current project comes from a book called Systemantics:

A complex system that works is invariably found to have evolved from a simple system that worked.

(The first sentence of Gall's law)

I decided to buy this book and read it to make sure I wasn't quoting a book that said crackpot things about freemasonry or something. Of course, now that I've bought it I'm able to search on a key phrase and find the whole first edition online.

I had my hopes for this book. In my mind it was a romantic relic of the age of cybernetics. It would have feedback diagrams and equations vaguely remembered from CS112. It would have lots of other quotable bits backed up by distilled essence of case study. It would be the Mythical Man-Month or the Peopleware of systems design.

The truth is more prosaic: it's a pop-psychology book. The bibliography includes pop-psych classics I'm OK, You're OK and Games People Play. It mentions General Semantics on the third page. It talks about Murphy and Parkinson and Peter--pop psychologists all--as though they were Weiner and Taylor. There are no data and not even many anecdotes, just thought experiments.

Plus it's got dumb Watergate jokes. Hey, that's real funny to you guys in 1975 but I'm reading your book in the twenty-first century. You savvy my future-speak? To us, that happened decades ago, and we have higher standards for jokes about it. (Sample Watergate joke that's still funny. Non-broken image link for same.) Nothing wrong with trying to make a book funny, but the Poignant Guide to Ruby is pretty funny, and it's also got code you can run that works.

Okay, so it's a bunch of thought experiments. That's not too bad if you run the experiments and they jive with your experience, which is true here for the most part. The book has some more good quotes ("Systems work best when designed to run downhill") as well as good ideas I'd never thought of ("the confusion of Input and Output"). I'll probably pick up the revised edition of the book, The Systems Bible, which will surely have fewer Watergate jokes. But the pop psychology emphasis really decreases the authority of a quote from this book.

I could say "A complex system that works is invariably found to have evolved from a simple system that worked." myself, and it would have the same force as quoting John Gall in this book. I've probably got more hard evidence than the book does, because I live in the future. I quote him the way you quote Mark Twain: because you like the way he put it. So maybe I should just find a quote about systems design from Mark Twain.

(1) Tue Jul 11 2006 21:47 Loaf Is Weird Food: Exhibit A: Jennifer McCann's Vegan Loaf Web Application at Vegan Lunch Box, one of what I hope will eventually be one of many applications that let you explore a parameterized space of related dishes. In this case, dishes made out of separately-edible items all mashed together with binder.

As with anything, if you put it on the web I'll randomize it. Thus, my latest feature, Mystery Loaf. It names and creates a new dish every five minutes. Eating them is your problem.

Wed Jul 12 2006 17:41 wadl.rb: If you like REST web services I have something for you. A while ago Marc Hadley came up with a service description language called WADL. I now have a primitive proof-of-concept Ruby WADL parser/client that can read a WADL file and call simple web services like the Yahoo News search. (I emboldened that link because there are so many other links in this entry, and that's the one this entry is about.)

Right now every REST web service is slightly different. You access them all with simple tools: your language's HTTP library, and probably its XML parser. Which is great because it keeps the services simple. What's not so great is that you have to learn each REST service separately.

What's more, it's useful to package and distribute your implementation of a particular service in a particular language. There's PyAmazon and Ruby/Amazon and a4j, PySearch and Yahoo-Ruby. That's always seemed wrong to me.

With a SOAP/WSDL service you just load up the WSDL file and call or generate the native language methods. Even though the underlying technology has more layers and is much more complex, when WSDL is one of those layers (and nothing goes wrong), everything just works. Sometimes there are language-specific wrappers on top of a SOAP/WSDL service (PyGoogle and Ruby/Google), but they don't do much: they make your code look nicer, or compensate for the shortcomings of the underlying API.

WADL is the best way I've found to get that ease of use for REST services, without creating documents that don't make sense to humans, or selling out the resource-oriented architecture of the web. WADL is very human-readable and it describes resources, dammit. I like it a lot and I think it'll make things easier for programmers if it gets widely adopted.

(3) Wed Jul 12 2006 21:21 IF Score System Design, Plus, a Writer's Plea: The above-any-adequate-alliteration-allowance 'First-Timer Foibles' guide to writing IF got passed around a lot among my friends, so I want to talk about that a little bit. This is mostly based on email conversation with Adam Parrish, who I just realized has the awesome job of studying interesting things like IF. I knew about his job, but not that when we talk about this over email I'm slacking off and he's working. Actually now I might be just making stuff up.

First a note about pages like that, which as a good IF writer but a middling static fiction writer, I find kind of frustrating. The page is oriented towards beginning writers. Like most web pages that allege to help you be a writer, it's heavy on "don't misspell words" and less heavy on "don't have a hackneyed plot" and "don't create puzzles that make no freaking sense" and things I don't know or can't articulate. Such pages turn bad writers into readable bad writers, but won't get anyone's work up to really good quality. As I've found out the hard way, good ideas and a good grasp of English don't automatically translate into readable stories. There are additional skills.

These pages chop the head off of a Sturgeon's Power Law that says the vast majority of bad writing is bad for obvious reasons (scroll down to the handy "context of rejection"). With static fiction I can routinely hit the midway point on this particular ring-the-bell carnival game, but I haven't found many good resources for getting higher. I have found one extremely useful page, but the higher-level craft seems to be something you have to learn one-on-one with someone who already knows, or something subjective you learn with practice. Or something that no one has written about because existing documents are enough to get the unintersting people out of your hair.

The problems in the first-timer list are divided into problems of fiction and problems of game design. I'm not going to discuss the problems of fiction because I don't find them very interesting. I can, however, tackle the problems of game design because much less work has been done exploring elementary problems in game design. Because this entry is already huge I'm going to cover one item at a time, over a period of several centuries.

Item 1: bad point systems. There are actually two possible problems here. The first is a poorly-scaled point system where you get 100 points for finding a key. The second is an overgenerous point system where you get 5 points for getting out of bed.

If a piece of IF has a scoring system, it imposes a limit on the score. I don't know of any counterexamples. In general, your score goes up when you do a one-time action that progresses you toward the conclusion of the plot. Your score at any given time is a measure of your progress (Zork III is a notable, and obnoxious, exception).

But if a video game has a scoring system, it imposes no limit on the score (except any imposed by the hardware). Again, I don't know of any counterexamples, though I can conceive of games where your score is represented as a percentage. The score of a video game has nothing to do with the game per se: your advancement towards the end of the game is measured by things like stage numbers. It's just a way of keeping... score. I think this problem is caused by treating a piece of IF like a video game.

It's fine to give the player 100 points for finding a key in a video game, but ridiculous to do the same thing in a piece of IF. This is because people don't handle big numbers well. If your maximum score in an IF game is a big number, it's difficult to tell how close you are to the maximum, and how much 100 points really contributes towards the maximum. A big inscrutable number is cool when you're claiming to be an awesome dude, but it's not as cool when everyone who completes the game gets approximately that same score.

By the same token, you shouldn't give out points for actions that don't advance the plot or don't involve any cleverness. If you do, the score will cease to have meaning as a way of measuring the state of the plot and your cleverness so far.

Thu Jul 13 2006 23:53 Trivia: A while ago I discovered a Victorian book of how-to miscellany called Enquire Within Upon Everything. It turns out that Tim Berners-Lee came across that book when he was a kid, and named his first hypertext system (1980) ENQUIRE after it. Amazing trivia!

Sun Jul 16 2006 16:31 Shot in the heart: And you're to svn blame.

Mon Jul 17 2006 06:29 Monde mondial: From a French message board:

BeautifulSoup, c'est l'un des meilleurs parseurs HTML du monde mondial, c'est ce que j'utilise pour mon script de conversion blueflagz => RSS

"Monde mondial"? Does that mean "real-world"? If so, that's a great idiom.

I don't know when I'm going to be fixing Rubyful Soup to have the features of Beautiful Soup 3.0, but it's probably going to be after a bunch of WADL work.

Mon Jul 17 2006 13:22: One good thing about being President of the United States: the food is great!

When he entered, President Bush walked within two feet of me, and for some reason, he turned and looked me in the eye and said, "Let's eat!".

Ned Batchelder and his wife's White House adventure: parts 1 and 2.

(5) Mon Jul 17 2006 20:31 Maple Syrup Vengeance: In California you can go to a pretty nice breakfast place and they'll serve you real maple syrup in a glass with a spout, or a little syrup ramekin. If you go to a cheaper place they might have fake maple syrup but it'll be in a glass with a spout so you can pretend it's real. Even IHOP does this.

In New York, right next to the place where the maple syrup comes from, they don't do any of this. I've been to fancy breakfast restaurants, as fancy as any I ever went to in California, and they all treat maple syrup like cheap jelly. It's made by Smuckers out of corn syrup and it comes in little plastic tubs with pull-off tops, like non-dairy creamer. They have to stock this stuff because all the restaurants here do delivery, and how are you going to deliver maple syrup in a little syrup ramekin. But they take it a step further and also give it to the people who came over to their restaurant in person! So when I go out to breakfast I never order anything with a dependency on maple syrup. I guess I could be a huge snob and bring my own little container of maple syrup, but then why don't I stay home and make my own waffles?

I've eaten at places in the south (IHOP being the exception), where the restaurant just buys a 20-pack of Log Cabin squeeze bottles and plunks one down on each table. New Yorkers probably think their system is better, but I don't see how. I never thought I'd have anything good to say about IHOP's syrup, but at least it doesn't look obviously fake.

Incidentally, Rocks'n'Diamonds is turning into ZZT.

(2) Wed Jul 19 2006 07:07 I Am A: Mike Popovic heard my earlier song I Am A Cowboy and filked it for his daughter Zoe. He turned it into I Am A Pirate, which I dutifully recorded. Both songs are now part of a concept album called I Am A, which explores the spectrum of authenticity and total fakiness.

Where will the trail of improvisation lead next? Ninjas? Robots? Some other nerdy role-play?

(1) Wed Jul 19 2006 22:51 Experiments: Today I did a thing Sumana has long suggested I do, and took my writing on the road. Down to the cafe about two blocks away. I had lemonade and a sandwich, and wrote book stuff on Sumana's laptop.

I'm not a huge fan of doing that kind of thing; I managed to stay about an hour and a half before it got to me and I had to leave. Did it help my writing? Maybe, but not $13 worth, which is what I spent on food. I'm going to try the cafe three blocks away tomorrow, but I think that mid-day work-trips to cafes will just be occasional treats to eat a lunch I didn't make.

Speaking of which, I went early this morning to the farmer's market in Union Square, which has really picked up since the dead of winter when I first went. Now it's full of cheap carrots and radishes and stone fruits, reasonably priced tomatoes, slightly expensive berries, and expensive garlic. Someone said "These tomatoes are severely deformed! Are you going to charge me extra for this?". I also saw a woman in chef garb pushing a metal cart around with some food on it--could this be one of the "celebrity chefs" who frequent the Union Square market, buying "celebrity food" for their "celebrity restaurants"? Who cares, really? Remind me to tell you about restaurant food sometime.

Usually when I go into Manhattan I try to wait until at least 10, because otherwise you're crammed into a subway car with a bunch of commuters and it feels like a demonstration of Avogadro's number. But today I went along with Sumana so I could buy stuff at the farmer's market and get home in time to make gazpacho and give it lots of time to soak. Sumana invited some people from Fog Creek over for dinner and the theme was to be "cold".

So I made gazpacho, and a tasty couscous/tofu/cucumber/tomato dish, and sliced deformed tomatoes with salt (not really cold, but you have to eat that at every opportunity while the farmer's market tomatoes are around). For dessert: vanilla ice cream with rasperries. Theme: embodied!

On an absolute timescale, dinner preparations kept me busy enough that I didn't write as much as I'd have liked. But I did discover some interesting facts which will go in the book.

Thu Jul 20 2006 21:56 Couscous: By popular demand the couscous recipe.

• 1 box firm tofu
• 2 cups vegetable broth
• 2 tsp olive oil
• 2 cups couscous
• 1/2 red onion, chopped
• 1 cucumber, chopped
• 2 tomatoes, chopped
• 1 bunch parsley, chopped

Okay. Drain the tofu, slice it into rectangles, fry it in peanut oil, slice it into strips.

Boil the olive oil and broth. Pour it on the couscous and fluff with a fork. Dump in all the other ingredients and mix it up some more.

Tastes like a particulate Cobb salad or something.

Thu Jul 20 2006 22:13 I've Got Radioactive Blood: Went to the dentist today, where there was a Spider-Man poster in the X-ray room. The dentist is oriented towards kids, but is that the best place for Spider-Man?

Relatedly, my cousin Jill came to visit with her friend today. Said friend recounted a story of overhearing one of her sons saying to the other, "No, let it bite you, and you'll get superpowers like Spider-Man!" Friend put a stop to that right quick.

Can't think of a punchline, so I'll delegate to Penny Arcade. It's one of those ones where the punchline is in the first panel.

(1) Sun Jul 23 2006 10:42 Only to Find Gutenberg's Bible: Yesterday we planned to go to the Transit Museum in Brooklyn. Instead we went to the research library (the one with the lions) and saw the handwritten Declaration of Independence. It's riddled with apostrophe errors! ("laying it's foundation on such principles") What would my mother think? Incidentally, that link does not have the exact version we saw, but it's close.

Among the topics I need to write about is my antipathy towards libraries that don't let you browse the stacks. But the really huge libraries with closed stacks also tend to be the ones that have things on display like a freaking Gutenberg Bible. Just right there in a case in the middle of an otherwise boring room. Do people know about this? I didn't.

To overcome my closed-stack anguish we went across the street to the mid-Manhattan library and checked out some books. This was great, especially the self-service machine you use to check the books out. But on the way out there was a different form of anguish, a lengthly fracas with the guy who makes sure you don't take books out of the library without checking them out. That's the totality of this poor guy's job, so when I triggered his "unchecked books being removed from library" alarm he went right to work on me.

"You need to check those out." "I did check them out." "Sir, those are library books. You need to check them out." "I did check them out. Do you want to see the receipts?" "Sir, you need to check those books out." "What do you do to check them out besides use the checkout machine?" "You can't take books without checking them out." Yes, I KNOW HOW A LIBRARY WORKS. (I didn't say that. I only say rude things like that afterwards, in weblog entries.)

Eventually he looked at the receipts and conceded that I had probably checked out my library books, deviously disguising myself as an honest citizen and cheating him out of a juicy apprehension. Paperwork beats alarm in this rock-paper-scissors. Sumana suggests that the self-service checkout machine demagnetizes or magnetizes some RFID-like object in the book, and I'd picked my books back up before the machine had had a chance to do that.

Then we decided to go to the transit museum for real, so we took the subway. Sumana has a greedy algorithm for taking the subway through a bottleneck, but one of its implicit assumptions is that the subway graph looks like San Francisco's, with few paths through a bottleneck. Now that we've moved, the algorithm needs generalization. The end result is that, due to stereotypically unintelligible service announcements, we ended up near the Brooklyn Bridge with no simple subway path to the museum. So we just walked the bridge. Which was fun, but once we got to the museum Sumana sat down on a bench and fell asleep, leaving me to explore the museum by myself.

This entry is long, so I'll publish it and write another one.

Sun Jul 23 2006 11:43 Brooklyn Transit Museum: This is a museum in an unused subway station. The subway level has a switching tower with a live view of the nearby subway lines. In fact, the whole subway level is full of old restored subway cars. Which is awesome. You can time travel throughout the twentieth century by going from car to car.

The cars were mostly decorated with period advertising placards, which were pretty interesting, but which destroyed the time-travel metaphor in a Vonnegutian way. Instead of having ads from a specific time, each car had a collection of ads from the entirety of its run, all unstuck in time and smushed together in one car. There were WWII-era ads next to Prohibition-era ads, including one that claimed "One day all women will vote... for [brand of soap]". I don't think that ever happened. Incidentally, the soap ad was one of those ads that features terrible and irrelevant racist cartoons.

Another soap ad had a Depression-era mother saying, "My children must purify hands before eating." Well, I hope your children enjoy starving, since nothing they do will ever be good enough for you.

I walked around for a while thinking these cynical thoughts. Not helped by an ad for subway ad space I'd seen upstairs in a historical exhibit, which claimed that subway riders "have learned to believe implicitly in the advertising displayed in the New York City Surface Cars because it merits and invites confidence."

This attitude was mocked by a contemporaneous newspaper cartoon where, having nowhere else to put ads, the conductor was looping signs over the necks of the passengers. It was from the golden age of political cartoons, when there were lots of strange details but not the loopy word balloons full of cursive dialogue that's poorly blocked and illegible, and not worth reading since it just explains in great detail "Oh! The Situation depicted in this cartoon, though considered by the reader a japish Metaphor, is something that truly does affect my corporeal Form!"

The truth is that most of the subway ads did, and do, not merit or inspire confidence. The golden age of the subway ads were the 1950s, a time when advertisers had forgotten that words had connotations as well as denotations, when you had ridiculous claims like "84 out of 100 women prefer men who wear hats" (an ad for hats, or for men?) and needed gorgeous MAD magazine-style paintings to make up for them.

Trivia: the original name of the Brooklyn Dodgers was the Brooklyn Trolley Dodgers, answering my decades-old question of what the Dodgers were dodging. Also, an old sign once posted near an elevator says "Meddlers Take Notice" in a preemptive anti-Scooby-Doo move.

Sun Jul 23 2006 11:59: Sumana's reading a Dave Barry book. Yesterday she read a bit of it to me about how Abner Doubleday "invented a game that included virtually all of the elements of modern-day baseball, including Bob Costas and the song 'Who Let the Dogs Out'. This led to the Civil War."

"Actually there is a connection between Abner Doubleday and the Civil War," I said, ever the vigilant connection-spotter and nitpicker. I looked it up in Bully for Brontosaurus and, yes, Abner Doubleday did start the Civil War. He was a Union officer who fired the first shot from Fort Sumter.

Actually his Confederate counterpart is the one who really started the war, but it's more true to say that he started the Civil War than to say he invented baseball.

Sun Jul 23 2006 20:00 Ruby Cookbook Promotion: The Ruby Cookbook is either available or almost available, depending on who you ask (Amazon sales rank right now: an amazing 7230; incidentally, authors, I have a small script that records and graphs a book's Amazon rank, since I know you're all obsessed with that). So it's time to start promoting the sucker.

I wrote an article for the O'Reilly Network (which I think is technically not part of O'Reilly, somehow) about how we tested the code in the Cookbook. Which I should nudge the editors about. Lucas lives in Portland, so he doesn't have to pay for a plane ticket to go to OSCON and talk peoples' ears off about the book.

My ill-formed plan was to do a virtual book tour: guest-post on technical weblogs about the coolest recipes from the book. Other people have done this in the past, notably Greg Knauss, but I don't think it's been done with a technical book before. And I never actually emailed anyone about doing this, so it's probably too late now. I got an invitation to join the O'Reilly Ruby group weblog, so maybe I'll just post once a week on there about one of the cool recipes.

Does anyone else have ideas for book promotion? I want to focus on my current project but I should set aside some time for selling what I've already written.

Sun Jul 23 2006 20:09 Mars Needs B-Roll: I forgot to mention that if you ever go to the Union Square famer's market on a Saturday, there'll always be at least three camera crews filming. Some of them are obviously students doing assignments, but for the rest of them, how many puff pieces about the farmer's market does one city, even a city of 8 million people, need? Is it all going to stock footage? "Man with rowdy child buys arugula (0:32)"

(1) Mon Jul 24 2006 22:14 Need to move the earth?: I'm thinking about designing a pocket multi-tool that gives you all of the simple machines. It would have a little lever, an inclined plane, a fold-out pulley, etc. The supplementary gadget: a combination straightedge/compass!

Tue Jul 25 2006 22:00 Beloved Ruby Cookbook Recipes #1: "Getting book information with Ruby/Amazon": You know those old-time radio shows, where Doris Day always guest-starred, and Bob Hope ended up wandering onto the field at Muroc and getting shot? Well, that particular one never aired, but you know the kind of show I mean. The shows that were sponsored by random household objects like soap, or a broom, where the straight man would set up a line and the funny man would always turn it into a little ad for the sponsor.

"Lou, I think we ought to stick around a while longer." "You know, Acme Wallpaper is the only thing today's handyman needs to 'stick around' the house. It's inexpensive, durable, and when properly applied it forms a Penrose tiling! Ask your dealer to see a catalog today." Thus allowing the straight man to set up the exact same line again: "I say, Lou, that we ought to stick around a while longer and watch after Doris Day."

This is the model for this series of weblog entries, where I show you how to do cool things with Ruby but it turns out all to be a scheme to plug the Ruby Cookbook, bad boy of the O'Reilly cookbook family. In the first three entries of this series, I'll show you the system I rigged up to track the sales of my books. I don't really have a plan after that, but I can work from a list I made of the coolest recipes in the Cookbook.

The Book Sales Trilogy:

1. Getting book information with Ruby/Amazon
2. Generating graphs with Gruff.
3. Generating sparklines with the sparklines gem.

The problem

When you write books but don't publish them, it's difficult to know how many you've sold. The publisher gives you a total maybe quarterly. So a lot of authors (including me) use their Amazon sales rank as a proxy for books sold.

There are numerous problems with this, stemming from two facts: 1) Amazon sales rank only covers books sold through Amazon, and 2) it only conveys how well your book is selling _compared to all other books_. Booms and crashes in the book market as a whole won't make any difference to your sales rank. But you can't beat something with nothing, and sales rank is the only way I know of to get anything like real-time sales figures for a book. I talked with Mike Loukides, my O'Reilly editor, today, and he said that giving authors access to sales information is a low-priority project. And we all know about low-priority projects.

So, whenever a book of mine goes up on Amazon (first Learning Python and now the Ruby Cookbook), I register its ISBN with a script that runs once an hour and uses Amazon Web Services to collect the sales rank numbers. My goal is to encapsulate this logic into a class which maps an ISBN to a text file full of timestamped sales ranks. It will also work with Amazon-specific ASINs: for books, the ASIN is just the ISBN.

I use the Ruby/Amazon library, which is covered in Recipe 16.1, "Searching for Books on Amazon". The recipe focuses on keyword searches: finding books or other objects to match criteria. Here, we search by ASIN.

The script is based around a SalesReport class that encapsulates data about one product's sales rank over time. Here's the first draft of SalesReport.

#!/usr/bin/ruby
require 'amazon/search'
class SalesReport < Array
  attr_accessor :name, :asin
  def initialize(asin, name)
    @asin, @name = asin, name
    open(self.class.filename(@asin)).each do |line| 
      self << line.split.collect { |x| x.to_f }
    end
  end

  def self.filename(asin)
    asin + '.txt'
  end

When you create a SalesReport object for a product, it opens up a file named after that product's ASIN and reads in sales data.

Where does that file come from? The update! method is in charge of creating and maintaining it. This is also where we make the actual AWS request (in bold).

  # Updates sales rank for a number of products, and yields a
  # SalesReport object for each one.
  def self.update!(aws_key, *asins)
    now = Time.now.to_i
    req = Amazon::Search::Request.new(aws_key)
    req.asin_search(asins) do |product|
      asin, name = product.asin, product.product_name

      # Update the file with the new sales rank.
      open(self.filename(asin), 'a') do |f| 
        f << now << ' ' << product.sales_rank << "\n"
      end

      # Parse the file into an SalesReport object and yield it.
      yield self.new(asin, name)
    end
  end
end

Three things about this method's signature. First, like filename this is a class method: it's a factory for SalesReport objects rather than something you run on a particular SalesReport. For more on this, see recipe 8.18, "Creating Class and Singleton Methods". The self here is the SalesReport class itself.

Second, this method takes a variable number of ASIN arguments, for which see Recipe 8.11, "Accepting or Passing a Variable Number of Arguments".

Third, as I've mentioned in every book I've ever written, to use AWS you need to sign up for a unique string called a "developer token". This method uses that string to create an Amazon::Search::Request object, which will be our interface to Amazon Web Services.

You can reuse a Request object to do multiple searches, but we're only doing one. To save time and bandwidth, we pass all of our ASINs in a single search: "give me info on these products". The query returns a big XML document, which Ruby/Amazon parses into struct-like Ruby objects. It yields each object to a code block.

Inside the code block, I have access to a variable product which represents what Amazon knows about one particular book. I take its sales rank reading and stick it along with a timestamp into that book's file. Within the code block I also build a SalesReport object, which I yield to another code block somewhere outside this function. The code that calls SalesReport.update! gets yielded a series of SalesReport objects to do with what they will. In this case, my will is to generate cool-looking graphs.

If code blocks confuse you (they confused me when I was new to Ruby), the first few recipes of chapter 7 will help. If you're not familiar with file access, the basics (including appending to a file) are covered in Recipe 6.7, "Writing to A File".

Okay, that's all the SalesReport code for today. Here's how you'd call SalesReport.update! with a code block:

key = 'my AWS developer token whatever'
SalesReport.update!(key, "0596523696", "0764596543") do |product|
  puts "I updated #{product.name} (Rank: #{product[-1][1].to_i})"
end
# I updated Ruby Cookbook (Cookbooks (O'Reilly)) (Rank: 1412)
# I updated Beginning Python (Programmer to Programmer) (Rank: 219370)

So now I've got two files on Crummy's hard disk full of sales rank readings. What to do with that data? That code block doesn't really do anything., Well, I'd love to compare the sales ranks to hard sales numbers, but I've never gotten any for the Python book, and the one data point I got from Michael today doesn't mean much. So I've been comparing these numbers to their earlier selves by graphing them versus time. This leads into tomorrow's episode, Graphic Violence, featuring Chico Marx.

Wed Jul 26 2006 13:48 Oatmeal: Argh. Remind me not to start promotional tutorials until I've written all the code. I had to totally redo yesterday's entry to get a good architecture for doing big graphs and sparklines. Plus because of it, I found an erattum in the Cookbook code (in the MIDI music recipe). So I have the weird feeling where you've been busy all day but haven't accomplished anything new: you've just cleaned up your old mistakes.

On the plus side, the code is much nicer now, and it turns out errata get fixed in the next printing of the book, not the next edition as I'd always thought for some reason.

Also, you know those allergen disclaimers on food that say "may contain traces of nuts"? Today I believe I succeeded in isolating the nuts! First, a recipe:

Natur-Swiit Oatmeal

• 1 cup soy milk
• 1/2 cup non-soy milk (I've been bumping up the soy milk percentage but 100% soy milk still tastes gross in this)
• 1 handful dried fruit
• pinch salt
• 3/4 cup oats

Instructions: Boil the fruit in the milks and the salt. Add oats and cook for 5 minutes on low heat.

Okay, it's just the instructions from the back of the oatmeal canister, except for the fruit. But the fruit is magic. When you boil the fruit in the milk, it swells up and sweetens the oatmeal, so you don't have to add brown sugar to the oatmeal. Adding fresh fruit afterwards never makes it sweet enough for me.

I eat this for breakfast about half the time. Usually I make it with cut up dried apricots, but today I went with all different types of fruit. I dumped them in the saucepan, turned on the heat, and then went away to work on the promotional tutorial and forgot about the boiling milk. Then I remembered and ran back into the kitchen, where the milk proteins had formed huge bubbles in the pan. And suspended in the bubbles were... tiny pieces of walnut skin!

I'm pretty sure that's what they were; they could have been pieces of fruit skin but they sure looked and felt like walnut. I'm pretty sure I've seen the fabled "traces of nuts" and my life is now complete.

(1) Wed Jul 26 2006 16:50 Best-Loved Ruby Cookbook Recipes of the American People #2: "Generating graphs with Gruff": After a long wait the Ruby Cookbook is now available almost everywhere, even on BookFinder. I urge you to purchase what I hope is the strangest O'Reilly book ever published (suggestions for competitors welcome; the only one I can think of is Stephen Feuerstein's Oracle PL/SQL Programming). In addition to hundreds of tired and frankly predictable geek in-jokes (Star Trek, Rogue, Cryptonomicon, the GNU Virtual Fridge, ad nauseum), it features talking frogs, coffinfish, two-faced politicians, corpses in freezers, dispute resolution through ritual combat, Orwellian doublecode, a heart-pounding Novel on Rails, and Dr. Bronner's Peppermint Soap. Special guest star: T-Rex.

It also features graphs! Recipe 12.4, "Graphing Data", introduces Geoffery Grosenbach's Gruff library, by amazing chance also the subject of today's promotional tutorial. Gruff makes it easy to turn data structures into graphs and write them to PNG files. So the old clock on the wall says it's time for part two of the Book Sales Trilogy:

1. Getting book information with Ruby/Amazon.
2. Generating graphs with Gruff.
3. Generating sparklines with the sparklines gem.

The hardest part of Gruff is installing the dang ImageMagick or RMagick libraries and their dependencies in the first place. It's easy on Debian and other systems with a good packaging system, but otherwise it can be a real pain. The second hardest part is working around Gruff when its simplifying assumptions don't apply to you. I glossed over these in the book but I'll tackle the second one a little bit in this tutorial.

Anyway, yesterday I showed you code to take periodic readings of books' Amazon sales rank. And then I showed it to you again today because the code I wrote yesterday was crap. So read that entry even if you read it earlier. The new code makes the rest of the trilogy much easier to present.

You'll recall (from like ten seconds ago when you read it) that we have a SalesReport class that encapsulates sales rank information from a book. Yesterday, though, I didn't show you anything interesting to do with this information. But the night is ours! Tonight, we graph!

Let's open up the SalesReport class again and make a sales report capable of expressing itself as a line graph:

require 'rubygems'
require 'gruff'

class SalesReport
  # Make a Gruff graph for the sales of this product.
  def make_graph(graph_path)
    g = SalesRankGraph.new(800)
    g.title = "Salesrank over time: #{name}"	
    g.theme_37signals
    g.colors = ["black"]
    g.title_font_size = 20
    g.hide_legend = true

This is mostly self-explanatory setup code. In the book I claim that most of the Gruff themes are ugly, but that theme_37signals is okay. Well, that's just, like, my opinion, man, but incontrovertible fact is -- and I should have mentioned this in the book -- that theme_37signals's idea of a good time is to graph the first dataset with a yellow line on what's basically a white background.

That's a really bad idea. I believe there's a UI maxim to the effect of: "Some other color than yellow on white, graph-reader's delight. White under yellow, dangerously confuse a fellow." So I go with theme_37signals but tell Gruff to draw the data line in black: the original high-contrast color for white backgrounds.

The other thing I do is hide the legend, because this graph is only for one product. I would like to graph sales for all of my books on a single graph, but I haven't figured out a good way to do it yet. One of Gruff's simplifying assumptions is that all your data points are spaced evenly along the X-axis starting at X=0. I'd have to insert a bunch of bogus data points for books that came out later; worse, most of my timestamps don't line up precisely, so I'd have to write code to group multiple times into a single data point. So right now I just do one book per graph.

Now we've got one line of code that's very important, because it's where I decide how the data will be represented.

    g.data(@name, collect { |date, rank| 1/rank })

The data method takes an array, and adds it to the graph as a data set. SalesReport is an array of dates and ranks, so I could just pass in the ranks, but that would yield a graph like this:

This is a lousy graph. Unimportant details (long stretches early on where no one bought the book) are the most obvious features, and you can't even see the release of the book. But the data isn't useless; it's just not presented well. We're accustomed to seeing charts go up when the numbers go up (see: any TV commercial featuring a chart), but a good sales rank is very small. Also, as all Web 2.0 types know, book sales follow a power law distribution. A book at 400K sells one copy and jumps to 200K, but you have to sell a mess of books to go from #100 to #90. Displaying the sales rank as though it were linear distorts the data.

I don't know the exact distribution for book sales, but simply taking the inverse of the sales rank gives the graph the right shape. In this graph, the release of the book is obvious, and the time leading up to it makes sense:

What about those labels on the X-axis? Where do they come from? They come from this code:

    label_hash = {}      
    [0, (self.size/2).round, self.size-1].each do |i|
      label_hash[i] = Time.at(self[i][0]).strftime('%m/%d/%Y') if self[i]
    end
    g.labels = label_hash

The graph's labels are a hash that maps positions on the X-axis to strings. Remember, the positions on the X-axis are the indices to the array(s) you passed into the data method. You don't get to choose these values. The X axis starts at zero, and ends at the maximum index of the largest array you passed into data. I choose three labels: one at the beginning, one at the end, and one halfway between. Here's a graph with a lot more history than the Ruby Cookbook one:

Finally, having created the graph, we write it to disk:

    g.write(File.join(graph_path, "#{asin}-salesrank.png"))    
  end
end

Well, not quite finally. I sneakily referenced a class called SalesRankGraph a while back, and never defined it. That class derives from Gruff::Line, but if you do this graph with a Gruff::Line it'll have weird numbers on the Y-axis:

Those labels are just what you'd think: they're the numbers being graphed. This mighty graph stretches from about zero to about 0.001. Of course, the graph is "really" measuring the inverses of those numbers, but there's no way to put that in the labels. It's another of Gruff's simplifying assumptions. You can choose your X-axis labels but not your X-axis points; you can choose your Y-axis points but not your Y-axis labels. I couldn't find an easy way to fix this, so I just hacked the draw_line_markers to not draw them. You can shut off both the X- and Y-axis labels by setting hide_line_markers, but I like the X-axis labels.

class SalesRankGraph < Gruff::Line
  def draw_line_markers
  end
end

So now I've got some pretty nice-looking graphs to track my sales rank. But I don't have time to look at graphs! I should have spent today working on my new project, but instead I wasted the morning fixing problems with the preivous entry in this series, and then spent the afternoon making pizza sauce and writing this entry! What to do? If only there were some post-literate infographic that would convey sales rank information at a glance! Something like the graphics I laboriously put on the crummy.com homepage this afternoon! Stay tuned for tomorrow's episode, The Spark of Line!

Fri Jul 28 2006 14:23 Dream Update:

Sumana: I dreamed I was having dinner with the president, and I was cracking all these jokes, but nobody was laughing.
Leonard: You dreamed you were Stephen Colbert?

Fri Jul 28 2006 14:46 Google Code: A couple weeks ago I talked with Jason Robbins about my new book project. Jason works at Google, along with Greg Stein and several of the other people I respected most at CollabNet.

Jason pointed out that usually when you make a decision it's like the poem from high school, where you never find out what would have happened to your life if you'd chosen differently. But occasionally you do get a glimpse. He said that soon I would get a glimpse of what I'd be doing if I'd gone to work at Google with Greg and him.

And so I have. It seems I would be working on Google Code, a site for hosting open source projects a lot like Tigris and the other CollabNet software development sites.

Hum. Okay, Google is probably a more fun place to work than CollabNet, and Google Code has some great features (I like the radically simple approach to bug tracking). But I don't think Counterfactual Leonard would be happy quitting one job in frustration only to go do another one that's basically the same. I think I'm happier on the road less travelled.

Fri Jul 28 2006 17:47: No one's reading this because Crummy's nameserver died. Kevin's putting together another one. In the meantime, I'd like to point out that I'm not the only one doing promotional tutorials. Among other things, Lucas gave an OSCON talk on distributed programming with DRb and Rinda. He had everyone with a laptop stop typing snide comments into an IRC channel and run his distributed Ruby code, creating an ad hoc SETI@Home project to find prime numbers. Pretty slick.

(2) Sat Jul 29 2006 12:01:

Sumana: There should be a Lord of the Rings II.
Leonard: What would be in this Lord of the Rings II?
Sumana: The ring is back! But now we have machine guns.

Sun Jul 30 2006 17:17: I believe I have found the ultimate Wikipedia page: the list of incomplete lists.

(2) Sun Jul 30 2006 19:42 Anthology Of Ruby Cookbook Recipes That's Not The Ruby Cookbook Itself #3: Sparklines: Okay, thanks to much work by Kevin the nameserver is back up, and I can return you to the Book Sales Trilogy, where I sell the Ruby Cookbook using software that tracks how many copies people have already bought. That's what I call recycling!

On previous installations I showed you how to get sales rank information about a book from Amazon, and how to create a graph of the data over time. Now my vengeance will be complete: I will unleash sparklines upon the world!

Sparklines are really interesting: bits of anonymous data that add quantitative analysis to text, in some cases without breaking the flow of a sentence. The meaning of a sparkline depends on context or a tiny textual label, not on big sets of axial markings. I've been a fan of sparklines for a while, and what better way to propagandize them than by inclusion into a book of cool tricks? THERE IS NONE. So here we go with the final part of the trilogy:

1. Getting book information with Ruby/Amazon.
2. Generating graphs with Gruff.
3. Generating sparklines with the sparklines gem.

As you know, Bob, in previous episodes we defined a SalesReport class which encapsulates information about a product's sales rank over time. Then we extended the class to give it the ability to write out a big graph describing the history of its sales rank. Now we're going to extend it again, and give it the ability to write out a sparkline.

Like Gruff, the sparklines gem is a great piece of code by Geoffrey Grosenbach. It's simpler than Gruff because sparklines are simpler than graphs. It does have some issues you need to watch out for, though. I cover sparklines in Recipe 12.5, "Adding Graphical Context with Sparklines". In the book I give some silly examples focused on embedding sparklines into HTML pages with the data: URI scheme, and incorporating sparklines into Rails views with the sparkline_generator gem. Here, I'll show you to show how to write sparkline graphics to static PNG files.

Those images from the Crummy homepage (Cookbook and Beginning Python ) show sales rank values for the past 30 days. How do I make them? Let's see.

class SalesReport
  # Make a sparkline for the sales of this product.
  def make_sparkline(graph_path, time_units=30, samples_per_unit=24)
    path = File.join(graph_path, "#{@asin}-salesrank-sparkline.png")

    # Gather a sample of the data
    sample = []
    (size-(time_units*samples_per_unit)-1).step(size-1, samples_per_unit) do |i| 
	sample << 1/self[i][1]
    end

A sparkline needs to be small, and unlike Gruff, sparklines doesn't compress data points. Gruff takes an image size in pixels, and whether you give it 8 data points or 800, your graph is that size. But if you give sparklines 800 data points, you get a really long sparkline. I can't use all the data I've gathered since these books showed up on Amazon: I need to economize.

This is fine. I don't need the whole history for a sparkline like I do with a big graph. I'll settle for the sales rank history from the past 30 days. But I run my data collector every hour, and that's still 24*30=720 data points. Solution: I go back 720 hours, and skip ahead 24 samples at at time. This way I pick a sample sales rank for every day, ending with the most recent sample. You can change time_units and samples_per_unit to customize your sparkline.

Now I have an array and I just have to send it to be turned into a sparkline:

    Sparklines.plot_to_file(path, scale(sample), :type => 'smooth', 
                            :line_color => 'black')
  end
end

Well, not quite. Earlier I mentioned that sparklines line graphs don't compress data horizontally. There's a fixed number of pixels between each point. Well, sparklines doesn't compress data vertically either. A line graph can handle values between 0 and 100. You can make the sparkline bigger but I'm pretty sure you can't increase that range. If you give values outside that range they get clipped or ignored.

In this case, our numbers are already between 0 and 100, since we're taking the reciprocals of numbers greater than one. But they're also between 0 and 1, as we saw in the previous installment. When the range is 0-100, this makes for boring sparklines. If your book rocketed to the top of the Amazon charts and stayed there, its amazing success would look like this: (caution: embedded sparkline not visible in IE).

To make our data visible, we need to stretch it out. We might as well go for maximum stretch: treat the smallest sampled rank as zero, and the largest as 100. That's what the scale method mentioned above does. Here's the definition:

  # Scale data so that the smallest item becomes 0 and the largest becomes 
  # 100.
  def scale(data, bottom=0, top=100)
    min, max = data.min, data.max
    scale_ratio = (top-bottom)/(max-min)
    data.collect { |x| bottom + (x-min) * scale_ratio}
  end

I gave a similar method in the sparklines recipe in the book, but it's hard-coded to scale a range to 0-100. This implementation is more general and you can use it anywhere. I really should have made this a separate recipe because it kept coming up. In 12.14 "Generating MIDI Music" (which I hope to write about later) I reused this formula for a different range and got it wrong, and had to send in an erratum. In 2.12 "Using Complex Numbers" I used a similar formula to scale an ASCII drawing of the Mandelbrot set to any desired size. So it turns out this is a useful method to have.

Anyway, now we can scale any data set to take maximum advantage of the vertical space allotted us by a sparkline. In the book I lightly discuss the ramifications of scaling all these different data sets to the same 0-100 range. The Cookbook sells much better than Beginning Python, but you can't tell that from the sparklines. Which is fine here, because I want these sparklines to show trends.

The sparklines gem lets you do other kinds of sparklines: pie charts are my favorite, as you'll see in the book. I'm really excited about the possibility of sparklines: they're like the little crawls at the bottom of television news stations, except they're classy and related to the main text.

The full sales rank monitor program is available here. I hope other authors find it useful.

What's next for Best of Ye Book of Ruby-Receipts? I don't know. These promotional tutorials take surprisingly long to write: it's almost as much work as was writing the original recipes. So I may give it a rest for a while. But let me know if there's a recipe in the Cookbook you'd like to see me explore in more detail, in the context of a real-world program like the sales rank monitor.

Mon Jul 31 2006 21:24 Tiny IF Roundup: This was going to be a whole game roundup but I got distracted BY WORK. Yes, my leisure time was sucked up by book work. I don't know if this is a problem or not. Seems like it might be this early in the book process.

Anyway, I'm just going to review one game, an IF game, and I'm cheating because it's a game I played and liked when I was younger. It's called Crusade Adventure and it came with AGT as a sample game.

I'm a little surprised that no one has written a program to convert AGT code to Inform. AGT and Inform are abstruse in opposite directions, though, so maybe I'm the only one nostalgic enough about AGT to be interested. And as I look at these old games I become less interested. AGT runs just fine in a DOS emulator, after all.

I remember Crusade Adventure as being full of action, with a caving section more sophisticated than Colossal Cave. Well, it's not full of action, it's full of the action-evasion. And its caving section is only sophisticated in that it's a small adjunct to a larger map. The game relies heavily on random events: you can avoid some puzzles (including a fairly clever one) by relying on chance. The map is disjoint, and you have to use a magic word to move around.

Unlike most AGT games, Crusade Adventure is pretty nonlinear and has good atmosphere that I associate with Sir Walter Scott. Unfortunately, it turns out I only associate the atmosphere with Sir Walter Scott because the game makes explicit reference to Ivanhoe, in what might be the most jarring anachronism I've ever seen in an IF game. As Mark Twain would say, "the genuine and wholesome civilization of the nineteenth century is curiously confused and commingled with the Walter Scott Middle-Age sham civilization". I've never read any Scott but his "dreams and phantoms" fit well into an AGT setting: misty woods, suits of armor, talking skulls.

I remember playing a very slightly racier version of Crusade Adventure. But the one on the IF archive is fine because it makes this game stands in stark contrast to the all-out sleaziness of E. L. Cheney's GAGS games, also often distributed with AGT. There was one really disturbing one set in an ultra-sleazy San Francisco that featured lines like: "Lucy's firm young breasts are the center of your focus, You'd love to play around with them, but do you have the time?" Oh, gee, look at the time! A web search reveals that E. L. Cheney now draws cartoons about golf.

As a palate cleanser for the sleaze let me offer more of Twain's Fennimore Cooper-class vitriol against Scott:

It was Sir Walter that made every gentleman in the South a Major or a Colonel, or a General or a Judge, before the war; and it was he, also, that made these gentlemen value these bogus decorations.

Bam!

Tue Aug 01 2006 23:53 Maps: Worked on book, then worked on short story. (Incidentally, still no word on my submission of my previously finished story.) Pleased to note that Beautiful Soup is now being mentioned in distinguished university lectures.

I had an idea based on Google Sightseeing: a journal of negative results. You could post to this service saying that you'd scanned a certain region and found nothing especially interesting.

Actually, is there a journal of positive results that's machine-accessible? A fully general way to describe parts of the map. No reason the negative results couldn't go in there too.

(3) Wed Aug 02 2006 17:36: Crummy reader crackpot schemes come to life!

(1) Thu Aug 03 2006 14:15: Sumana went with coworkers to see Avenue Q. I'd been looking forward to seeing Avenue Q as well, because it looked like someone had finally done a musical that really explored the symbiotic relationship between Muppet and puppeteer. But much of my interest was crushed when Sumana revealed that the puppeteers are not characters in the musical! Any more than they are on brand-name Muppet shows. The only human characters are the ones who aren't operating Muppets.

The puppeteers are visible, they walk around the stage and they emote along with their Muppets, but in the fiction of the show they're supposed to not exist? Or they're supposed to be the insensate bodies of the Muppet characters? I dunno how it's supposed to work. How do Muppet operators see their Muppets? I always thought it was like the relationship between ventriloquist and dummy.

(1) Thu Aug 03 2006 20:55 Rejection and Acceptance: I came home to discover two emails: "Mallory" got rejected by Futurismic and "Unit Testing a Book" got accepted by the O'Reilly Network. Not that I normally have trouble getting technical articles accepted, but it took away some of the sting of bitter, bitter rejection. With lots of helpful feedback and an invitation to resubmit after a rewrite. Silence, reasonable self!

If I may self-analyze, I think my biggest fiction failing is a pathological fear of writing infodumps. I relentlessly avoid explaining anything except in sidelong glances. The result is a pointillistic story that you can't understand without reading it multiple times. To make you want to read the story multiple times I cram it full of ideas. Which must be explained. In sidelong glances. My stories look like flip books. You have to work to see the plot.

Does anyone else have this problem? (Especially published writers whose work I can look at?) I'll let you know what strategies I can figure out against it.

(2) Fri Aug 04 2006 16:39 That's capitalist coffee, you traitor!: Sumana's coworker Jacob has a Muppet obsession similar to mine. He sent us these 10-second coffee commercials from the 1950s featuring a proto-Kermit beating up on a proto-Waldorf for not liking the brand of coffee advertised. Sample commercial:

Proto-Kermit: Have some Willkins coffee?
Proto-Waldorf: No thanks, gimme Brand X!
[Proto-Kermit shoves an X-shaped branding iron into Proto-Waldorf's chest.]

I've never seen such violent Muppets.

(2) Sat Aug 05 2006 23:31 Chronological Order: I thought I'd avoided the mistakes that first-time writers make, but I think I found one that nobody told me. Your short story should be in chronological order. My story was almost in chronological order. There was one scene that I displayed (in two parts) before it "really" happened. But that little bit of jumping around created four time-shifts in the story, making the whole thing seem disjointed and plotless.

All day I gnashed my teeth trying to figure out how to get that damn scene into its chronological place in the story. So many dependencies, so many facts it establishes that I need later. I finally just dropped the scene into the right place and started hacking at the dangling references. The guts are still showing, but amazingly the story now has a plot. One thing happens, and it makes another thing happen, and things cause other things until the story ends, and that's a plot. There are no random notes from the future coming in and obscuring the connection between things x and x+1.

Maybe it's not that strict a rule, but I'm a well-known anti-fan of flashbacks, so for me it's chronological order or totally disassociated sequence of images. It can't be half and half, it can't be now and then.

(7) Sun Aug 06 2006 23:22 Keyboard Madness: Sumana and I play a Tetris game where we use the same keyboard. It gave me an idea for the game that the most people could play using the same keyboard.

It's a spacewar type game with a separate spaceship for every key on the keyboard. When you press a key, that character shows up on the screen as a spaceship and joins the game. You control the spaceship with the corresponding key. Tapping the key rotates your ship clockwise, and holding it down accelerates.

The health of a spaceship is represented as a color from green to red. Accelerating drains your health, and so does colliding with another ship. You kill other spaceships by ramming them: this does some damage to you but more to the other ship. If your health goes to super-red your ship explodes.

In theory you can have as many players as there are keys on the keyboard, assuming they've all got long tentacles for fingers. In practice you will only have a few players, but each player can try to control multiple ships. Will you use ships as disposable missiles or try to set up pincer movements? Or just try to hit the other player's keys and mess up their navigation?

Mon Aug 07 2006 20:56: For a while Mike Nelson has been doing MST3K-style audio commentaries for movies' DVD releases. Now he's refactored and is selling freelance commentaries as MP3 files.

Mon Aug 07 2006 21:41: Game Set Watch now has a column about Roguelike games. Will it be able to keep up with my level of obsession? Only time will tell.

(3) Tue Aug 08 2006 12:29 Conservation of Commas: When I was about eight my mother told me a story that took place while she was clerking law. There was a lawsuit between someone who wanted something built, and the builder they'd hired to build it. The contract read something like "The builder who is responsible for the inspection shall...". It should have read "The builder, who is responsible for the inspection, shall...".

The builder came to court and shrugged its collective shoulders comically and said that it had never heard about these other builders or which one of them was responsible for the inspection, but the contract gave it no reason to believe that it was that unfortunate builder.

Ever since then I wondered what happened to those commas. Now thanks to Teresa Nielsen Hayden I see that one of them has shown up in Canada. Where's the other one?

Tue Aug 08 2006 21:16 Ruby Cookbook PDF: Lots of people were disappointed that there was no downloadable PDF copy of the Ruby Cookbook for sale. Well, now there is! Lucas and I convinced O'Reilly to make it available as a PDF for half off the cover price. And it looks like they might be headed that way in general. (Update: two sources confirm they are.) Our place in history is secure!

(3) Tue Aug 08 2006 22:24 Dada Maps: In 1996 I cut up a road atlas and pasted it into an AAA map of southern California. Because the scales were different, a major highway in one part of the country would feed into a street on the LA detail of the road map. This was my Dada Road Map.

I got this map laminated, but I always thought my craftsmanship on it was rather shoddy, so I never hung it up. Recently I foisted it on Jake Berendes, who pretended to appreciate it.

But once I got it out of my head, a similar idea took its place. Google Maps has pregenerated static image tiles for maps at different scales, and it wouldn't be hard to write a program that tiled them together like Google Maps does, except not helpfully.

So I present Dada Maps, which uses tiles from all over the world to create a new evocative map every 5 minutes. It's what I wanted my earlier project to look like, and it's not subject to scarcity. Enjoy!

(4) Wed Aug 09 2006 00:39 Joementum: In 2003 and 2004 when I was working on the Clark campaign we always made fun of the Lieberman campaign. We didn't do all that well but we always did better than Lieberman.

I've discussed before how part of being a politician is going out and smiling the morning after a crushing defeat and pretending everything will turn out fine. I've seen my share of this up close, but Joe Lieberman was incredible. All through the 2004 campaign he acted like the nomination was his in all but name, right up until he dropped out. Defeats were recast into milestones that he'd had to meet, and that he'd just met. He had the Joementum.

Except no one could figure out what "Joementum" was. Clearly it was not being used according to its dictionary meaning ("an electroplated filligree added in modern times to a medieval baton or weapon"). Nor could it legitimately be considered any kind of pun on "momentum".

I think I've figured it out. Yesterday Joe Lieberman lost his Senate primary (note to non-US readers, who don't care anyway: this is quite a feat). He's still smilin'. He's going to run as an independent, against the Democratic nominee and the (fairly pathetic) Republican.

That's when it struck me. Joementum is not specific to Joe Lieberman, and it's not an attribute of the campaign, as I'd always thought. It's a personal quality of the candidate. It's just the name for that thing inside you that makes you act like the bad news isn't happening, that everything will work out fine and you'll win. It's the momentum that keeps you physically working even when people aren't voting for you. Joe Lieberman just has more of it than most people in politics.

It's not denial; it's the ability to act as though you were in denial. As I discussed in my earlier entry, this is a neccessary quality in a politician, because admitting that any bad news is real will instantly doom you. To gain any nontrivial victory, the candidate must posess enough Joementum to be stubborn and firm-jawed about bad news and impossible odds. But usually when the bad news is that you lost and conceded the election, you can relax.

Thu Aug 10 2006 07:37: "This book is dedicated to the ninja in everyone's Dad." Why didn't I think of that dedication? Via waxy.

Fri Aug 11 2006 22:12: Subway ad: "Making more of me". Aimed at von Neumann probes?

Sat Aug 12 2006 23:19: Today went to the Met, and met some of Sumana's friends. Too tired to go into detail.

Sun Aug 13 2006 10:38 Name-Dropping: Still not much time to write about the Met because I'm going off on a Long Island adventure with Ruby Cookbook contributor Kevin Marshall soon. So I'll just mention the impressive stuff I hadn't seen before. Because most of the new stuff I saw was fairly modern and had a name associated with it, I can do this in the form of name-dropping: Raphael, Rembrandt, Picasso, Dali, Miro, Goya, Bastien-Lepage, Eames, Sottsass, Jan Stern, and the strangely photorealistic Robert F. Blum.

I also discovered a creepy sub-floor called the Henry R. Luce Center, which is a sort of browsable storage space for the museum. It's full of furniture and paintings and bottles just lined up on shelves behind glass, with minimal signage.

Sun Aug 13 2006 17:37: Ruby Cookbook: #1 technical book at Powell's!

Sun Aug 13 2006 22:05 Plane Crazy Three Ways: Went to Jones Beach with Kevin Marshall. A popular hobby at beaches in New York is hiring planes to fly over the ocean trailing advertising banners. You're not having fun here if nobody's monetizing your experience. We saw three ridiculous banners.

One was for Bud Lite. I guess if you're spending a million dollars a day or whatever on advertising, hiring an airplane to fly over a beach is just a drop in the ocean. But it's not neccessary, because the Budweiser brand is one of those select few that's associated with the concept of obnoxious advertising in general. Ask someone: "Say you're at the beach and someone flies a plane in front of you. What is the plane advertising?" "Uh, Bud Lite?" You don't need to hire an advertising plane, because any advertising plane makes people think of your product.

One was for Snakes on a Plane. It had a picture of a snake, and it said "Snakes + Plane = In theaters August 18th". As much as I would like to believe that there were promotional snakes on the plane that pulled that banner, I know that that's not true. The banner merely called attention to one of the many planes above this planet that contain no snakes at all. Was this a good marketing move? I don't know, but this is a ridiculous B-movie-style movie being marketed as an A-movie.

The other airborne banner was the most ridiculous yet. It was tied to its plane by two tethers, which somehow had folded the banner up so that the advertising was not visible to anyone on the beach. It was as obnoxious as the other two ads, but there was no content. Made me think of Bud Lite for some reason.

Thu Aug 17 2006 02:01 Autograph Apology: To a guy whose Cookbook I autographed at LinuxWorld. I don't remember your name but your last name had two Ts at the end. When I autograph books I write down random cliches for the inscriptions. I do this because I have a mental picture of cliches being harmless by definition, so that I can pick them at random and make people's lives more interesting without offending them. But in your case I picked a cliche used in a dismissive or derogatory manner:"Take a picture, it'll last longer". I hope you don't think that I think this cliche applies to our conversation or our author-reader relationship or some perceived slight on your part; I just picked it at random.

So I'm sorry I gave you a lousy inscription, and I hope you don't feel insulted. If you're reading this, I'm happy to send you a replacement sticker-inscription with the cliche of your choice. Or you can just keep it around to embarass me with my own callousness.

To everyone else: enjoy your cliche-inscribed Cookbooks!

(4) Thu Aug 17 2006 10:59 Wow!: Sales of the Cookbook (at least on Amazon) have skyrocketed! Current sales rank: 850. I can't figure out what caused that. Maybe it's the chance to get a copy that I haven't written some nonsensical inscription on.

Update: Maybe the Dinosaur Comics plug.

Sun Aug 20 2006 17:38 Whooooo: "Some say his ghost still... stays with his body because he's not dead."

Tue Aug 22 2006 00:59 Awesome: STOP THE CASCADIA MEGATHRUST EARTHQUAKE SUBDUCTION EVENT

Wed Aug 23 2006 13:39 Advertising Phrases You Didn't Expect: "Maximum strength homeopathic formula"

(1) Thu Aug 24 2006 23:44 Stock Spam Spam: I get lots of stock spam but today I got spam advertising a stock spamming service. "We are a stock promotion company and we can promote any kind of stocks." They have two products. The first is straightforward spamming to pump a stock:

We can increase the price of your stock and we can increase average day trading volume. We can increase price up to 200-250% in 2-3 weeks and also we can increase volume by 10 times each trading day.

You pay them according to how much they jack up the stock price in one day.

WE GUARANTY LONG TERM GROWTH

WE PROMOTE THE COMPANY AND IT'S BUSINESS

Who could doubt it?

The second product I don't understand, unless it's a degenerate case of the first product. Here's the relevant part of the email:

You want to invest some money and you want to make big profits? Our company can help you in that. We know exactly which stocks will increase in price and wich will be falling down. We will give you advices and according to that advices you will make decisions. We will prove it for 1 day for free. We will tell you the price of the stock in advance and you will see that our stock promotion group works hard in order to make your investment portfolio grow.

Payment: Our price for that is 25-30% from your income. Before the deal you are telling us on which ammount of money you are buying/selling stock and we are counting the future profit and take 30% from that profit. You don't have to pay anything in advance, you pay only then you receive your profit.

So they know how stocks will go, but they want you to buy the stock instead of doing it themselves? Clearly there's spamming going on here ("our stock promotion group works hard"), but maybe they also do something with the knowledge about how much you're going to buy or sell? I don't see it.

This demonstrates that stock spamming has become a service that can itself be packaged and sold via spam. It's a little different from generic spamming services (or spam for spamming tools that you yourself sell through spam), because there's an implication that pumping stocks is hard work best left to the spamming professionals.

Fri Aug 25 2006 17:10 Cross Stitch Riot: Found via search requests: periodic table cross-stitch pattern!

(2) Fri Aug 25 2006 21:58 Greens vs. Chez Panisse: Sumana and I went on vacation to California last week. I didn't mention it on NYCB (or, indeed, write much at all to NYCB) because Sumana doesn't like publicly mentioning that we're away from our house. So now I have stuff to tell you about the vacation.

We ate with Rachel Chalmers at Chez Panisse. We sat right next to the open kitchen and watched people make food. The atmosphere was great and conducive to conversation. We talked about science fiction and Rachel said that I should spend my time writing space opera for her delectation, which is good work if you can get it.

We all had the vegetarian prix fixe. Sumana and Rachel shared a half-bottle of fancy wine and I had two glasses of a really good raspberry Italian soda. We started with bread and butter. Then, the menu!

Appetizer was a pair of crostini, one with tomatoes and one with some green vegetable I don't know what--possibly squash--and cheese. They both were topped with an excellent vinaigrette. The tomato crostini was better, because tomatoes are more awesome than random vegetable.

Then a garlic broth soup with a poached egg floating in it. You cut open the egg with your spoon and the yolk flooded out to be cooked by the soup.

Then the entree, which was a soufflé. It was okay but what I think the non-vegetarian option (something porkish) would have been better. What was really great were the corn, green beans, and other vegetables served around the soufflé. Also there were zucchini blossoms at the top of the soufflé, which was awesome.

Dessert was a great fig tart, and there were some tiny after-dessert cookies, also great.

Chez Panisse was quite an experience, but I was pretty disappointed in the soufflé, a nontrivial part of the meal. I don't really like souflés except for the dessert ones with chocolate in them.

The next day I went to Greens with Adam, Kim, Kris, and Erica. Sumana got sick from the plane trip so she didn't come. I, on the other hand, waited until the vacation was over to get sick from a plane trip. Cough, sneeze.

At Greens I had lemonade and zucchini griddle cakes and pasta with tomatoes and chocolate cake. The lemonade was pretty lame (I officially give up on restaurant lemonade, except for the lemonade at the cafe nearest my house). Everything else was excellent.

I've been to Greens many a time and Chez Panisse only once, but I think Greens has better food, as well as being easier to get into and a little less expensive.

I have a more general entry on this topic percolating, but I think there's a point of diminishing returns at which making restaurant food more expensive doesn't make it proportinately better. I hypothesize that this point comes when you stop getting your food from Sysco or a Sysco wannabe and start getting it from diverse local sources. Greens and Chez Panisse are both at this point.

(3) Sat Aug 26 2006 14:28 Transcendental Transneptunianism: Occasionally Sumana has transcendental experiences associated with revelations about the universe. Incredibly, I always manage to ruin these experiences by making some bonehead remark. I can't remember any of these remarks verbatim, but I have no trouble thinking them up and saying them when the need arises. I'd like to not do that next time because it really brings the transcendental party to a halt with a loud record-scratch sound.

I do know the source of these remarks: they stem from the eternal conflict between two philosophes in my mind. These are the hippy-dippy Carl Sagan atheist who says "You don't need God to have a good time, man; look at all the wonder in the natural universe!", and the cranky Richard Dawkins atheist who says "The wonder is all in your brain, you pothead! Not distributed throughout the universe!"

The Carl Sagan philosophe was formed in a transcendental experience I had when I was six years old. You know those charts that go on the walls in elementary school classrooms? Instead of putting a few of them on the walls, my kindergarten teacher, Jim Murchison, bought an enormous number of them. He punched holes in them and turned them into a huge flip chart. Every day or two he would flip over a sheet and reveal the new topic of discussion. One day he flipped over a sheet and I saw the solar system.

I'd surely been to the Griffith Observatory before then, but somehow I hadn't seen any pictures of the planets before this reveal. Jupiter, in particular, blew me away. Seeing drawings of the planets triggered my first and biggest transcendental experience. Since I always ruin Sumana's transcendental experiences it's only fair that I should ruin my own. Jupiter isn't transcendental; it just looks really creepy. It's a planet, a gasball.

What's more, the universe does not contain a magical kind of thing called a "planet". Planetness is a social construct. The solar system is not the sun and nine planets, as depicted on the flip chart of my youth. It's a fusing gasball, four non-fusing gasballs, a few million rockballs, a few billion snowballs, and a big dust cloud.

Recently, the International Astronomical Union decided to make a proper scientific definition out of the social construct "planet". This used to seem easy and even superfluous, but in recent years it's become clear that to judge every object in the solar system by a uniform standard of planetness, you must make a hard decision. Either Pluto is not a planet, or there are lots of Kuiper Belt objects which are planets on the same criteria as Pluto. These planets are very distant, mostly undiscovered, and not very interesting once discovered. There might be hundreds of these tiny boring planets. So the IAU created a definition that excludes these objects, including Pluto.

It's not nice to fool Mother Nature, but it's even less nice to naturalize Mother Social Construct. People are used to the planets being the non-fusing gasballs, four of the biggest rockballs, and Pluto. A lot of people got mad at what they viewed as a bunch of eggheads usurping their social construct. And now the eggheads are arguing amongst themselves about the definition. It's a huge mess.

The real problem with this definition is that, to get the "right" result, the IAU restricted the definition to apply only to our solar system. It's the Bush v. Gore of scientific definitions! They had to do this because extrasolar systems perform instant reductio on any attempt to turn "planet" into an objective concept. There are too many weird things in the universe. It would have been simpler if they'd just enumerated by fiat the eight things in our solar system that are "real" planets.

Astronomers have found things like two non-fusing gasballs orbiting each other in interstellar space. Are they planets? Only if we decide to call them "planets". What they are is big non-fusing gasballs that orbit each other. A gasball can have natural starness, but it cannot have natural planetness, any more than (as Michael Shermer says) it can have natural meaningness. This is what social constructs are for.

I personally don't care if Pluto is called a planet or not. My money is still on "alien disco ball". But if we must have a definition of planetness, it should recognize the inevitable subjectivity. If you don't do this, you need an artificial ban on discussing what might make an extrasolar object a "planet". Because sooner or later the smart-aleck universe will toss you a planet that doesn't meets the technical criteria, or (less likely) a non-planet that does. Then you have to tweak the definition and it's like adding epicycles to the Ptolemaic model of the solar system.

(6) Mon Aug 28 2006 23:35 Antigamesgeburtstagsdinosauriergewinnspiel: I used to wonder if Germans ever made up really really long words just as a joke. I stopped wondering this when I saw das Antigamesgeburtstagsdinosauriergewinnspiel, a production of German video game weblog antigames.de. Then I translated "Antigamesgeburtstagsdinosauriergewinnspiel" as "Antigames birthday dinosaur contest", and I started wondering again. It sure looks funny but was it intended to be? Presumably that's just how you say "birthday dinosaur contest", and maybe it's acceptable to incorporate the name of your website in a word. The German sense of humor is a mystery to me.

Antigamesgeburtstagsdinosauriergewinnspiel (I can almost pronounce that now) is a Something Awful-style Photoshop contest presenting video game screenshots that have been modified to include a dinosaur. This is in accordance with Kevan's discovery that any game can be made more fun by the addition of dinosaurs. The winners are here and here. Highlights include Tetris, Grand Theft Auto, and Nintendogs.

I discovered Antigamesgeburtstagsdinosauriergewinnspiel via referer logs, when a post-contest entry presented Nethack with dinosaurs, and a commenter pointed out that I already put dinosaurs in Nethack.

(3) Tue Aug 29 2006 10:23 Nethackdinosaurierunddeutschdiskussionsbeitrag: The German long-name-making-up question is resolved thanks to Antigames's own Richard (surname presumably not Leonardson), in his entry on the Nethack dinosaur patch. I turned his entry and my previous one into this exclusive fake interview, in which for the first time both sides of the interview are made of quotes taken out of context:

Leonard: I used to wonder if Germans ever made up really really long words just as a joke.

Richard: Yes, we do. You can stop wondering. Really. Stop.

Leonard: "Antigamesgeburtstagsdinosauriergewinnspiel" sure looks funny but was it intended to be?

Richard: It does and it was.

Leonard: Presumably that's just how you say "birthday dinosaur contest".

Richard: Yes, that' s really how you say it.

Leonard: And maybe it's acceptable to incorporate the name of your website in a word.

Richard: Nope, usually not acceptable. Any OTHER questions?

Leonard: Die deutsche Richtung der Stimmung ist ein Geheimnis zu mir.

Richard: Lake, it' s funny because… uh… wave, because. Oh, what the brightly, it' s A mystery tons of ME, too. Happy now?

Actually I should have realized that the German sense of humor is not that much of a mystery; the Antigamesgeburtstagsdinosauriergewinnspiel proves that Germans speak the universal language of Photoshop contests. Only unresolved question: is there a German word for the habit of making up humorous long German words?

Thu Aug 31 2006 23:28: I'm at a point in the writing of this book where it feels like my time would be better spent writing Vectrex games. On the plus side, Sumana and I just did Sesame Street improv and I had Bert say "Live the pain, Ernie!"

PS: Also, I just said "It is now empathy time."

Fri Sep 01 2006 12:38 Recently on Spam as Folk Art: Spam!: The Musical and The Honesty Spam

Sat Sep 02 2006 12:02 Beautiful Soup Job: If you want to make some money writing a Beautiful Soup program, send me email and I'll hook you up with a guy who's got a project.

Mon Sep 04 2006 23:29 Astronomical Edge Cases: I am feeling burned out right now, and unable to write, so I'm just going to point out some more of the weird things in the universe and the social constructs people come up with to deal with them:

• Planetographic coordinates. Defining the prime meridian of a gasball is trickly.
• Extreeeme extrasolar planets (PSR_B1620-26c is 12.7 billion years old!)
• Asteroid moons, asteroid families, and Kirkwood gaps

Tue Sep 05 2006 15:08: Oh, here's one I forgot: peaks of eternal light.

Wed Sep 06 2006 22:51 Ambush Bugaroo Banzai: Sumana and I went to Midtown Comics and I bought some Ambush Bug comics, including the two specials, which I hadn't read before. Now my collection is complete! IN MY MIND. On my bookshelf I've only got four comic books.

These were the first comic books I've ever bought, so this is a big step for me. I also got an issue of Hero Squared, written by Ambush Bug creator Keith Giffen. It's kind of funny in a wacky-sitcom way but lacks the Marx Brothers anarchy of Ambush Bug which is what made me actually go out and buy a comic book. I also got an issue of the aforementioned Buckaroo Banzai comic book, which is not really very funny at all but is great if you like Earl Mac Rauch's Buckaroo Banzai worldbuilding. Though the passage of time has made Buckaroo Banzai seem less eccentric, and more isolated from society and even vaguely sinister.

Thu Sep 07 2006 09:43 Totally Gross!: My fear has come true. Jake's gross-out candy idea has been assimilated. At least it took four years.

Totally gross! You know what we're talking about. You've skinned your elbow while crashing your bicycle getting extreme over a gnarly jump in your neighborhood.

I never meant to get extreme! It was all a big misunderstanding!

Thu Sep 07 2006 17:55 Stop the Motion!: The best thing about the crop of online video storage sites (YouTube, Google Video, et al.) is that it's a perfect medium for sharing, on someone else's dime, old short films that otherwise no one would ever see. There's the old proto-Muppet coffee commercials, and yesterday Andy Baio pointed to an Art Clokey pre-Gumby film which has a great Lovecraftian Roadside Picnic feel.

Gumbasia reminded me of old films like the ones I saw at the Exploratorium, so I went digging. I found Norman McLaren's Synchromy, which is still excellent though on my computer it was slightly... out of sync. Oskar Fischinger's Composition in Blue I could not find, but there is another film of his, seemingly a commercial that anthropomorphizes cigarettes. It's fun except it was made in 1934 and you're all the time worried it's going to turn into a cigarette Nuremberg rally. Which is totally unfair because Oskar Fischinger's art was degenerate and unwelcome, and he left Germany for Hollywood in 1936, so why am I thinking that?

There's also Norman McLaren's Neighbors, which some crackpot Wired article claimed was one of the inspirations for Star Wars. Why, because there's a swordfight? (Note: I don't feel like finding the reference because the Wired article might actually have a point, and then where would I be?)

(1) Thu Sep 07 2006 19:46 Unit Testing The Whole World!: One of the quality control measures I introduced for the Ruby Cookbook was automatically testing the code in the recipes. Most of the recipes contain worked examples, and the examples can be treated as a partial unit test suite for the forgoing code. I wrote doctest-like code that treated those examples like unit tests and used the results to find bugs in the recipes. You can still see the reports linked from the unofficial Ruby Cookbook page.

I think this technique is pretty interesting and I'm really happy with the quality improvements and extra confidence I got out of it. I even considered going to RubyConf and giving a talk about it. But going to RubyConf to give a talk costs me money, money I don't have, whereas writing an article on the same topic would earn me a pitifully small amount of money. I decided to write an article. This is the sort of business acumen that has made me the financial giant I am today.

Now the article's been published: "Unit testing your documentation". Original title was the less-prescriptive "Unit testing a book", which got changed to the even-less-prescriptive "Unit testing the Ruby Cookbook" but now it seems I'm prescripting away! Whee!

Thu Sep 07 2006 23:10 Hello, My Ragtime Gal!: Wow, it looks like I forgot about and stopped reading the webcomic adaptation of The Frogs right around the point where Michigan J. became a central character. Comic also seems to have Jabberjaw for some reason.

Fri Sep 08 2006 20:08 Calculating God: I liked this book a lot, and I recommended it to Sumana as well, which I think covers a pretty big chunk of your possible science fiction readership. The best thing was its relentlessly mundane treatment of alien contact (though the way the aliens flaunted their pop culture knowledge got kind of tiresome[0]). The second best thing was the Burgess Shale. But the second half of the book kept upping the ante in increasingly disturbing shock-twists, eventually turning into this big Arthur C. Clarke thing, and that wasn't what I was interested in. The plot held together well, despite some hurtful American-stereotyping (sniff).

[0] Tiresome but not neccessarily unrealistic. I could definitely imagine aliens in real life constantly flaunting their pop culture knowledge, but it would also be tiresome in real life.

Fri Sep 08 2006 23:05: After a month of rewrites I resubmitted "Mallory" to Futurismic; hopefully they'll accept it this time. I'm working on another story that's turning into a novella, but who'd publish a whole novella from the slush pile? Seems pretty unlikely. Of course, I could always turn to my good friends at Orion Nebula Publishing.

Sat Sep 09 2006 09:53 Recently in SAFA: It Pays to Increase Your Spam Power, and Kevan's hilarious Multiple Explosions

Sat Sep 09 2006 23:10 Picnic: Today I planned and executed a birthday picnic for Sumana in Astoria Park. Verdict: awesome picnic! Nothin' else to say about it, it was just a fun picnic with sliced tomatoes.

Sun Sep 10 2006 14:03 A Theory of Fun For Game Design: I'd been holding off on reading this book so I could savor it. But I think I was anticipating it would be more full of stuff than it is. It's only got a couple of ideas and those ideas were kind of in my head already. Reading the book did make those ideas much more concrete. And man, it has hilarious crocodile cartoons. There's one (where the crocodile is reading the newspaper) that makes me laugh every time I see it.

(1) Sun Sep 10 2006 21:16 Pythagoras Facts: Someone at the picnic-party saw the book I'm reading, Philosophy Before Socrates (I got it from Zack) and asked if I was reading it for a class. In the way that implies that no one would read that kind of book unless someone was forcing them to. But actually I'm reading it because it's interesting. Before there were standards for judging hypotheses, how did philosophers explain the world? Phase 1: myths. Phase 2: coming up with stories and then kind of discussing how plausible they are.

A lesser known fact is that pre-Socratic philosophy is freaking hilarious. Both phases yield funny results (Everything is made of water! No, earth and water! No, air!). They sound ridiculous now but "everything is made of air" was actually a pretty sophisticated model; it supposed that you got other materials by compressing air to greater or lesser densities. Occasionally the book gives out kudos: "[Anaximenes's] view that the stars give no warmth because of their great distance contradicts Anaximander's theory that the stars are closer to earth than the moon and sun are. It is also true." I believe this author was the first to discover that Anaximenes and Anaximander were actually different people.

I just got to the chapter on Pythagoras, who is portrayed as an L. Ron Hubbard/Paul Bunyan/Chuck Norris figure of folklore and Internet meme. Here are some useful Pythagoras Facts (p79):

His followers were devoted to his sayings, which they collected, memorized, and passed down. He was a charismatic figure who became the subject of legends: he killed a poisionous snake by biting it; a river hailed him by name; he made predictions; he appeared simultaneously in two different places; he had a golden thigh. The people of Croton addressed him as Hyperborean Apollo. Pythagoreans identified three types of rational beings: gods, humans, and beings like Pythagoras.

Wed Sep 13 2006 00:04 More Amazing Pre-Socratic Philosophy: Brendan praised my Sept. 12 entries from previous years, placing me under pressure, but really I got nothing. I can however posit a hypothesis that Heraclitus was the first crackpot philosopher. The Pythagoreans pioneered the proud hacker/crackpot tradition of trying to reduce everything to a single concept, but they chose "number", a concept which has meaning even if you don't accept the philosophy. Aristotle liked number okay, but didn't like the Pythagoreans, and he pointed out the many foibles that result when you say that all is number. It looks like Heraclitus saw this coming and avoided it by explaining everything in terms of a really vague concept: the logos, which in this context seems to mean "system of physics".

My first reaction is to mock Heraclitus for saying that if you just figure out how the universe works you'll see how the universe works; but one thing this book has taught me is that these ideas had to be invented, and it looks like Heraclitus might have invented the idea that there is such a system. Plus he has this to say about phase two of philosophy: "Let us not make random conjectures about the greatest matters."

(2) Wed Sep 13 2006 21:22 Arrested Development as Self-Referential Metaphor: There's no news hook for this, but I've been meaning to write this for a while so now's as good a time as any. I like Arrested Development because it's obsessed with continuity and what I can only call intratextuality (see entry not yet written on The Colbert Report). But what I haven't seen anyone point out is that the plot of the show is a metaphor for the production of the show, and for television in general.

To refresh your memory, the basic plot is that a real estate company is trying to make money by building houses. In the pilot, the company has only a model home: an incomplete shell of a house designed to sell other houses. Over the course of the series they build more houses, but their fortunes never improve, because the company is run by incompetents who care only about polishing their egos and stabbing each other in the back. Now, I'm not a big reader of Variety (or, as I believe it's now called, More of the Same), so I'm not up on all the industry gossip, but that dysfunctional environment seems a lot the one in which you put together a television show. Especially a relatively highbrow show that lacks support from the network.

Maybe the reason no one has pointed this out before is that over time the metaphor was presented in more and more blatant ways, making it seem obvious. The most famous intrusion is at the beginning of the third season (which was cut from 22 episodes to 13), where it's revealed that the company had a contract to build 22 houses but it's been cut to 13. As the show's fortunes got worse and worse, more and more of its fans' anguish and desperate, doomed attempts to save the show bled into the fiction as attempts to save the real estate company. But I think the basic metaphor (Bluth company = Arrested Development, building houses = producing television) was there from the very beginning.

Thu Sep 14 2006 21:55: Sumana's parents are in town, so no time to write a real entry. Instead I give you not-really-random-at-all astronomy link: The Sagittarius Dwarf Elliptical Galaxy. I'm reading Voyage to the Great Attractor in conjunction with my pre-Socratic philosophers and it's great.

(1) Sun Sep 17 2006 18:58 Too Little Too Late: I just realized that the Gobots would have been much more interesting if they'd been marketed as the Godots.

(2) Tue Sep 19 2006 20:29 Where's Eris?: In honor of the official naming of the scattered disk object formerly known as 2003UB313, I'm writing a short story called "The Fungi From Eris" (it's almost done, thanks). I'm pretty sure it's not publishable, so I'll probably just put it up on Crummy, but let me know if you can think of an outlet that might take a near-future semi-Mythos story with a news hook.

Anyway, while trying to bring to my story the scrupulous concern for scientific accuracy that is the hallmark of the Lovecraftian style, I became curious about where Eris is, exactly. Fourmilab's Solar System Live orrery only shows the nine textbook planets, but you can show an additional object on the map by pasting in its orbital elements.

Orbital elements are sets of astronomical measurements that resemble the tables used to enforce copyright protection in old computer games. The orbital elements are published, Kilgore Trout-style, in dirty magazines such as the IAU's Minor Planet Electronic Circular. It is rawther difficult to use the IAU site to find the MPEC issue for a particular minor planet, but it's easy if you use a Google site: search. Eris's orbital elements were published in MPEC 2005-041, and they are as follows:

2003 UB313
Epoch 2005 Aug. 18.0 TT = JDT 2453600.5                 MPC
M 197.53790              (2000.0)            P               Q
n   0.00176902     Peri.  151.31153     -0.91258509     -0.02028701
a  67.7091000      Node    35.87500     -0.34877687     -0.48266077
e   0.4416129      Incl.   44.17700     +0.21340843     -0.87557240
P 557              H   -1.1           G   0.15           U   5

Paste that into Solar System Live and you'll see that Eris is currently in the general direction of Neptune, as seen in the accompanying graphic.

Wed Sep 20 2006 21:24 What is the crackpot?: Earlier I called Heraclitus the first crackpot philosopher, due to his technique of explaining the whole universe with one vague idea. This is a common technique among philosophers who try to arrive at the truth about the universe by thinking really hard about it, rather than poking at bits of it. In this entry I speculate why the ancient Greeks produced so many ideas that now seem crackpot, rather than just nonsensical.

Maybe there's nothing to explain. These guys weren't dumb, so their ideas are usually internally consistent. But they didn't know as much as we do, so it's easy for us to pick holes in their cosmologies. Except that it was also easy for people like Cicero to pick holes in their cosmologies, mainly using the rhetorical form of Something Awful-type mockery. Also, once I got past Parmenides in my book, I stopped seeing general crackpot ideas and started seeing ones that looked a lot like modern crackpot ideas I'd encountered before. I believe that crackpottery took a great leap with the founder of the Eleatic school.

Parmenides was so impressed with deductive logic, he personified it as a goddess. According to my book this was the Greek method of dealing with "anything that exists independently of human will or effort, which is everlasting and has effects beyond human control". I actually think Parmenides himself was not a crackpot, though he had a really weird way of expressing his ideas. He looks to me like a radical skeptic.

Parmenides's goddess of logic convinced him that the universe consists of only one thing, which cannot move, has no parts or distinguishing features, and which is the only possible object of thought. Parmenides's monist universe reminds me a lot of the universe you inhabit when reading Descartes, at least until the third Meditation where Descartes totally sells out. Your senses can't be trusted and the knowable universe consists of your mind, about which you know nothing except that it's a thing that thinks... about itself. The radical skeptic arguments are, IMO, impossible to refute and this issue has never been totally resolved (not even by Prof. Hsu).

But radical skepticism is not very interesting. Your sense impressions keep on having parties and you sit idly by, not trusting them enough to join in. The successors of Permenides can't ignore the radical skeptical arguments, but they also want to do some philosophy. So everyone from that point talks about the unreliability of the senses, but except for the Eleatics (like Zeno), they posit a universe containing multiple things. These two conditions, I believe, formed fertile ground for the Golden Age of Crackpotism.

I think the defining feature of a crackpot idea is that its author expects their idea to explain the universe, but they aren't interested in poking at the universe to affirm or disprove the idea. They would rather use pure reason to determine what qualities the universe must have. Except that their pure reason is not so pure: it incorporates premises that were originally acquired through poking.

The Eleatics didn't bother poking the universe because nothing reliable would come out of it, and anyway there's only one thing, it's alredy been explained, motion is impossible, and nothing exists that might do the poking. Because of the force of these arguments, non-Eleatics were also wary of the value of poking. But unlike Parmenides, these guys took as their starting point for their pure reason exercises the behavior of everyday materials, the weather, and atronomical phenomena. Their theories are brittle because they're based not on pure reason (which would give you radical skepticism) but on a small number of sense impressions.

This is not really the Presocratics' fault, because the topic of organized ex post facto universe-poking didn't come up for many centuries (even Aristotle didn't have a good handle on it). But now that we have it, when someone ignores it and does what the Presocratics did (sense impressions -> theory -X-> more sense impressions), we call it crackpot.

The guy who really made me start thinking about this was Anaxagoras, who believed that the fundamental forces of the universe are Love and Strife. Love makes like attract unlike; Strife makes like attract like. Really elegant. More elegant, in fact, than the similar modern crackpot theory (don't remember whose) that posited Suck and Blow as the two fundamental forces.

Also: Democritus, in addition to writing a dialogue between the mind and the senses, came up with his famous atomic theory. This theory started out pretty well, but ran into huge problems because of his insistence that everything is made of atoms. Even energy and ideas. When you see something or think about something, your atoms are colliding with atoms from the thing you're seeing or thinking about.

This reminds me a lot of a crackpot that I think Martin Gardner discussed in Fads and Fallacies. Said crackpot claimed that sound was a particle and not a wave. Sample argument: it's absurd that a cricket would be able to move such a huge volume of air that you could hear it from far away. Instead, the cricket is filling that same huge volume of air with particles, and you hear the particles.

Incidentally, check out the G.O.B.-like behavior of Empedocles, the non-Eleatic who firmly established the now-canonical four elements that show up in video games.

[One story of his death] is that he leapt into the crater of Mt. Elna "wishing to confirm the report about him that he had become a god." ... Empedocles' ego and flair for showmanship. In public he wore a purple robe, a gold crown, bronze shoes, and a laurel wreath. He wore his hair long, had a retinue of boys to attend him, and adopted a grave demeanor. He was known as a physician and magician (professons by no means distinct in antiquity). According to a widely known story he kept a woman alive for thirty days without breathing or pulse.

No, Socrates, it's my illusion!

(1) Wed Sep 20 2006 22:28: I couldn't fit this anywhere in the previous entry. Today when I come up with an invalid cosmology I generally let it die. But when I was younger I would come up with science fiction stories about it (I had a pretty good one about the planetary model of the atom). Eventually my definition of "science fiction" shifted to exclude this kind of counterfactual, but lately others have been using them.

One of the three stories I liked in Ted Chiang's Stories of Your Life and Others took place in the cosmology of ancient Babylon. In ancient Babylon, the story would have been a 50s pulp story with a lame twist at the end. But because we know the universe doesn't work that way, it was an exploration of large-scale counterfactuals, as big as the ones you get in grand space opera. So this pre-Socratics book is making me think about going back to messing with the cosmology in stories. Except not the way I did when I was twelve.

But also I was thinking, if science fiction had been around in antiquity, it might have explored the plethora of cosmologies offered by the Greeks. Aliens would not just look different from us or have a different psychology; they'd be from a different cosmology. For instance, Diogenes thought that air was the motive force behind life and intelligence. How would that work? What if a person from that cosmology spent a lot of time breathing the same air as someone from our cosmology? I guess we could still find out.

Wed Sep 20 2006 23:33 Solomon Kane: Enough of this gay banter. Let's get lowbrow! I was reading a book of Conan stories and I learned that Robert Howard had created a Puritan swordsman character almost exactly like the character I created in Degeneracy. One day I needed to buy a book at retail or face the terrible fate of being bored on the subway. I bought an edition of Howard's Solomon Kane tales and read 'em.

Okay, dude is so not a Puritan. He wears a big hat, and accasionally he remembers he's Christian. That's about it. Maybe a Puritan swordsman would be interested in... purifying the church? Imposing the priesthood of all believers? Kicking popery's ass? No, Solomon Kane cares not for these things. He does the same stuff as Howard's other heroes, except he does it in a dour brooding way. Current mood: stern.

Solomon Kane spends a lot of time in Africa. He discovers lost civilizations. He fights the undead. He has a magical staff his witch doctor buddy gave him. He's Conan, except smarter and not as fun to be around. The 1920s pulp great-chain-of-being racism stands out in stark relief here; it's not as noticeable when Conan is the viewpoint character.

Solomon Kane's schtick is protecting the weak. His modus operandi is to wander the earth, he knows not why, until he finds or hears about someone who's weak (inevitably a woman), and then he protects the hell out of that person to the exclusion of all other activities. His most interesting trait is this obsessiveness. In these stories he tracks people down for years to get revenge on someone who wronged a weak. More than any of the two-cents-a-word descriptions, this is the bit of characterization that makes Solomon Kane come alive.

The stories are never boring, there's some nice cosmos-mixing, and a great moment where Solomon Kane is the victim of great-chain-of-being racism. But I was really disappointed with the way the character was used. He's written as an AD&D paladin, full of do-gooding crusader spirit but with no specific religious beliefs. And the stories are mostly "lost continent" adventure tales divorced from the historical context. If I'd written these stories I'd have him fighting the Royalists. Okay, undead Royalists.

Thu Sep 21 2006 22:20 WADL I do?: I put stupid puns on the names of technical standards in my entry titles in preparation for the popularization writing I plan to do about WADL. Depending on who's publishing it, that often requires coming up with stupid puns for section titles. Today I took back up the cause of wadl.rb and improved it a bit. There are still two big features I need to add, but it's good to write software again instead of book.

Brendan wrote his own version of "The Fungi From Eris", based on my prelimenary notes.

In related notes, I recently tried to catch the science fiction zeitgeist by writing a transhumance story, but it didn't work out well.

Fri Sep 22 2006 11:15 Amazing Discoveries: Universal Nut Sheller

Mon Sep 25 2006 16:27 The Science of Sleep: Sumana and I saw this movie yesterday. She didn't like it that much because the protagonist is messed up in ways that she doesn't sympathize with. I did sympathize, and liked the film a lot, though the ending glosses over the fact that long-term there's no way things can work out for this guy.

Unstrangely, the night after seeing this movie I had several very vivid dreams, including one that took the form of a MST3K skit. This is fitting because the movie's aesthetic was very MST3Kish. In my dream Joel went off somewhere to fight a space monster. The bots were worried about him, and when he came back unharmed they clamored to hear the story of the fight. Joel gathered them around in his parental way and started rambling on and on about strategy and fight psychology and everything but the fight itself. "And, you know, I'm not double-jointed or anything, us Hodgsons [sic] have always fought with the one set of joints." Etc. I'm not that big of a guy, but I'll crawl ya. The dream also involved Tom Servo's head being used as a disco ball.

(1) Tue Sep 26 2006 23:58 Voyage to the Great Attractor: I think the only books I'd read about cosmology were Steven Hawking's two. They left me with an incomplete picture of the universe. I knew that science had determined that there was a universe, and it was full of galaxies, but I had little idea of the large-scale (supergalactic) structure. But a while ago I looked at this map and all that changed. There are clusters and superclusters of galaxies, and creepy megastructures like the Great Attractor that are dragging our local group through the sky. What do these huge clumps of matter want?

Voyage to the Great Attractor does not answer these teleological questions. It tells of the discovery of the Great Attractor and of the subsequent attempts to demonstrate that the Great Attractor really exists and is not just observational error. Because the G.A. is hard to see: it turns out to be right in the part of the sky blocked from view by the Milky Way. Dastardly attractor! (I wonder whether or not this orientation is an accident. I can do a thought experiment where this happens naturally, but I can also poke big holes in the thought experiment. Since the book contains no speculation on this topic, I'm assuming it's an accident.)

Unlike the Hawking books, this book focuses less on results and more on the observing and computer-programming and arguing leading up to the results. Along the way you get a good overview of current (1990s) cosmology. The book also explains in detail concepts I was fuzzy on but that are neccessary for astronomers to do their work: what the red-shift is used for, why the cosmic microwave background is the frequency it is, etc.

This book contains no string theory, so if you don't believe in string theory you can learn all about the large-scale structure of the universe without having someone's unfalsifiable theory of its small-scale structure crammed down your throat. I'm going for the real niche markets here.

Wed Sep 27 2006 19:55: Buncha Long Now Foundation talks on Google Video.

(8) Thu Sep 28 2006 10:35 Giving Up: I haven't written about this because I was hoping it would have a happier ending. Back in May, after my mother died, we kids divided up her things for our inheritance (actually we did this before she died, with her badgering us to take more stuff). I shipped my things USPS to my address in New York. The low-value things like Tupperware and sheets and books I packed into cardboard boxes or fruit crates. The high-value things -- the compact OED, the scrapbooks my mother made for me, the French oven, the pitcher, the bedside lamp I had when I was young -- I packed into durable plastic tubs and mailed with insurance.

Most of the low-value stuff arrived within a week. The high-value stuff and two-thirds of the books disappeared off the face of the earth. I didn't buy tracking on the packages, so I can't track them from the USPS web site, but the post office has a way of tracking insured packages. They show the packages being accepted into the Bakersfield post office on May 17 and 19. That's it. My inheritance is gone.

As with any disaster I'm always revisiting what I could have done to avoid it. The most obvious thing is that I should have written my address directly on the plastic tubs. That way, no matter what happened I would eventually get the tubs along with whatever was in them at the time. The other obvious thing I should have done is not used USPS. I spent about $200 on postage, and it was only that low thanks to my liberal use of media mail. But the stuff I cared enough to insure -- especially the scrapbooks -- I should have cared enough to send UPS or FedEx.

After much badgering and form-filling and asking people here and in Bakersfield to check the back room, I've given up. After much more badgering and form-filling, today I got the post office to accept my insurance claims. In a few weeks they'll either find my packages or send me some money orders. I didn't buy a lot of insurance: just enough to (I thought) make the post office take my valuable packages seriously. But at least I'll get some closure.

When someone you love dies, the things they leave behind can keep your memories of them alive if you incorporate them into your own life. For a long time the only thing I had of my father's was one of his shirts, which is why I was so happy to find his postcards. Right now it looks like the tangible reminders of my mother's life are just more things that are gone forever.

(1) Fri Sep 29 2006 17:37: How come my account is never pre-disapproved?

Sat Sep 30 2006 10:57 Drag It On Over: The WADL Ruby library is up-to-date with the latest WADL standard, and fairly usable despite a lack of syntactic sugar. See especially this example, which turns NewsBruiser into a HTTP+POX web application over the objections of my previous self. Now I'm going to go back to my previous task of writing about why anyone should care about WADL.

Incidentally, WADL, and the Java implementation by its creator, now has its own webpage.

(1) Sat Sep 30 2006 23:05 Wired Nextfest: Nextfest Nextfest. Nextfest '89. Nextfest Nextfest. I went with Sumana and friends to this alleged fest, thinking it would be a mini World's Fair. I mainly thought this because it featured a corporate mega-sponor with its own "GE Pavilion", including -- I am not kidding -- "The Kitchen of the Future".

However the most prominent feature of Nextfest was the big-ass line that took an hour to get through. You really need to buy a ticket online. Okay, that ordeal over, there was some cool stuff in the various pavillions. For instance, Atari (??) had a pavillion full of entertainment stuff like a Claymation terrarium studio and a real-world fighting game, both products of Finnish ingenuity.

There were also many disappointments. Many exhibits had, instead of a thing, a flat-screen TV showing what the thing would look like if they were here. Example: the Lifestraw, one of the greatest inventions of this century, which is tiny and costs about $5 but they didn't have one on display. Ben was excited about seeing the "hybrid locomotive", until (according to him) he discovered that there was only a small model of said locomotive, and a picture of someone standing next to a larger model of the locomotive.

The "Exploration" (ie. "Space") section was kind of a disappointment -- as, indeed, has exploration itself been recently. There was a good exhibit on the Orion capsule, and according to the brochure an exhibit on detecting black holes which I missed.

I will forever consider the biggest rip-off "The Kitchen of the Future". You may have noticed a pattern here where you see something promised in the glossy color ad, with a description written by the same people who write quick, pat descriptions of new technology for Wired, but the thing itself is nowhere to be seen. There was no kitchen.

"Oh but it's not the future yet Giblets" you say, "You just need to wait til the video of the present becomes the kitchen of the future." Maybe it was the present this afternoon but now it's the future and still no kitchen!

What's more, when you call something "The Kitchen of the Future" and only provide a video, you're calling up memories of all those 1950s and 1960s videos on the same topic: Design For Dreaming et cetera. Here's a good historical overview that includes the new video. We expect some camp or at least some style. But this video was dull and creepy. You seemed to be inside some monolothic kitchen appliance, looking out at the people tapping on the touch-screen interface and following the recipe instructions displayed thereon. Both you and they seemed trapped.

And what are the benefits of this kitchen? RFID readers in the fridge, meal suggestions, shopping lists, automatic oven setting. The same things that these kitchens have been promising us forever; only the terminology has changed. I wrote a song eight years ago (called, yes, "The Kitchen of the Future") making fun of this exact thing. The song isn't very good, which is why no one but Jake has seen it, but I think I've made my point by now. If you can't get to the future in fifty years, then if it's world peace or artificial intelligence you should keep trying, but if it's a new kind of kitchen I think you should take a break and rethink the whole project.

Anyway, so as not to by totally cynical, an honestly cool thing about this kitchen of the future: the mega-appliance manages its energy by tricks like "using oven heat to warm dishwater". I've often thought that people could save a lot of energy with a lightweight, portable heat sink that plugs in elsewhere as a heat source. That's probably not possible though.

Looking through the catalog for other cool things I saw... Laser harp! Not really "future" but certainly futuristic and festive. Oh, we went to a demonstration on robots as entertainment, where we saw a great video from robotlab. They program industrial arm-robots to draw portraits, tune them so that you can play music on them, and so on. Lots of odd musical instruments in this show. The equivalent of the .org Pavillion here was "Robot Row", which was full of robots, including robots built by high school students to compete against other robots in a basketball-type game. That was a lot of fun, though I don't really get robots. You may have noticed that I'm not really a hardware guy.

Cool-sounding stuff I didn't see: text-free user interface, some kind of cameraphone-based barcode reader for doing your microloan business paperwork (?), a biofeedback game, and Pixelroller.

PS: here's a Wikitravel guide to the most recent World's Fair.

PPS: There were a bunch of cars. Boring. Except they were hybrid cars. Sensiboring. Some of them ran on ethanol. This was demonstrated by patches of corn surrounding the exhibit. The corn was fake, ie. plastic, ie. made from oil. Secret of ethanol -- revealed!

PPPS: Here's Design For Dreaming and I'm also gonna toss in a link to Explosives: Tools For Progress just on the basis of the name.

Sun Oct 01 2006 21:54 Applesauce: Recently I've been baking sweets with applesauce instead of butter to make them more healthy. I've had very good results with cakelike substances like brownies (that recipe discovered while I was staying with Jake), but cookies eluded me. I think the explanation is that applesauce can make a cookie moist, but it can't bind the cookie together. Applesauce cookies end up squishy.

Fortunately, you can use a separate binder that's tasty and goes will with cookies: oatmeal. I got this idea from the fatless cookies at a weird restaurant near Sumana's work that caters to bodybuilders. Today I made these cookies that use an insane amount of oatmeal. And they're pretty good. Applesauce brownies make a pretty good substitute for butter brownies. These cookies are a different animal from a typical oatmeal cookie, but they're definitely tasty.

(3) Tue Oct 03 2006 22:59: Netflix has a contest going to improve their recommendation engine's ranking by a certain amount. I've gotta say it's the best-specified contest I've ever seen. I'm leery of these contests because the rules usually have a "we own all your ideas even if you don't win" clause. This one just has a "we can use your idea even if you don't win" clause, which is less onerous. And the prize is a cool million, rather than the more traditional prize of a candy bar.

Since I have previously dabbled in recommendation engine design, some friends have asked me if I'm going to go for this prize. Well, I don't really need the money -- just now I got email telling me I'd won $2,500,000.00 -- but more importantly I don't think my Ultra Gleeper ideas are applicable to this domain. They mainly focus on improving the recommendations, whereas this contest is very tightly focused around predicting users' opinions given specific recommendations.

Not to say I don't have ideas. If I were going for the prize I would use IMDB and Amazon data for the movies to gather hypotheses about why people rate a movie high or low. I would then build profiles for the users, using the set of hypotheses as a basis. Then I would predict the users' future actions based on the profile. That sounds like handwaving but the basic idea (which others have seized on as well) is using objective data about the movies, instead of trying to figure out connections between them solely by how Netflix users rate them. And weighted vectors would probably be involved.

Of course, for all I know Netflix already uses external data to run their recommendation engine, so don't just run with this idea and expect to get anywhere.

Wed Oct 04 2006 22:43: BibliOdyssey has turned up history's first Penny Arcade cartoon.

(1) Thu Oct 05 2006 01:16 Back to the Drawing Board: My low-budget teen monster movie, The Regular-Sized Gila Monster, has flopped. Blahlahlahlahlah.

Thu Oct 05 2006 23:28: I keep having weird ideas tonight and not being able to sleep until I investigate them. Example: Erdös-Bacon numbers. Other example: I figured out how old I was the time I went to the hospital by looking up the release date for Young Einstein, which was advertised heavily on television while I was there. I'd just turned nine, which is weird; I remember it happening a lot later. Also I'd been hoping all this time that Young Einstein was a fever dream and not a real movie.

(3) Fri Oct 06 2006 18:24: Sumana informed me of the latest trend: ego-searching Google Code Search to see how widely your software and memes have been incorporated. My name is mostly in peoples' copies of Beautiful Soup, but I did find a Python implementation of a robotfindskitten game from 2004, long predating my own Python implementation. It's called pirhanafindshappiness and it is a bit different from stock rfk. There's also a dice-rolling program I wrote in QBasic in 1993, which ended up on Da Warren.

(6) Sat Oct 07 2006 23:23 Powers of 2 Breakfast: I make this for Sumana a couple times a week.

• 1 can black beans
• 2 chipotle peppers
• 4 eggs
• 8 corn tortillas

Drain beans and mash with peppers. Scramble eggs. Serve on warm tortillas. Add anything that you would put on a burrito: salsa, yogurt, cheese, greens, guacamole, etc. It's filling and tasty.

Sun Oct 08 2006 22:49 Entertainment by Proxy: I hope I don't turn into the kind of person who just posts video links on his weblog, but here's a video of Joel Hodgson's low-energy prop comedy. Almost exactly the same as his first SNL guest shot, if you've seen that. Includes John Candy, which I wasn't expecting.

Mon Oct 09 2006 17:00: Kids' letters to God

Dear God,

If you let the dinasor not exstinct we would not have a country. You did the right thing.

Jonathan

That's tough but fair!

Mon Oct 09 2006 20:58 As We Did Think: Some old computer-stuff links:

• Sales brochures from the Computer History Museum. Includes The Raytheon Computer Armchair Shopper, which sounds better than it is.
• Atari books
• An unplayable demo of Planetfall 2. Another "sounds better than it is", actually.
• Games on the C64, Amiga, BBC Micro. Remakes of same.
• The first DDR game machine.

Mon Oct 09 2006 22:29: Doing work on the Pocket Wisherman today I looked at my mother's old wish list and saw that it had Dr. Seuss Goes to War, a collection of Seuss's WWII editorial cartoons. A little searching revealed that these cartoons are all online and nicely categorized.

Tue Oct 10 2006 21:06: Achievements of Ancient High Civilizations. Really interesting, but:

[T]hese ancient people achieved much, yet were clearly prevented from achieving more by reason of a certain attitude of mind which seems to have been responsible for their failure to develop the scientific method.

If the pre-Socratic philosophers have taught me one thing, it's that motion is impossible. Also that the scientific method is not something lying around that any civilization would see were it not for attitudes of mind. It's a technology with a lot of prerequisites which must be developed. It's still being developed. We didn't get falsifiability until the 20th century.

If there is an attitude of mind that you have to get rid of, I'd say it's the attitude that induction is cheating or doesn't count towards knowledge. That if you thought really hard about things you could deduce the teleological reasons behind them. It takes a really long time for people to give up on this idea, but I'm skeptical that you'd get the scientific method much sooner by deciding to give it up.

Amateur philosophy of science now ends.

Tue Oct 10 2006 21:23 Flame Design: Sumana reads aggregation websites like Reddit and shows me cool things like the candle flame in microgravity. I tell her she should bruise these things. She says, "no, you should!" Even though she's the one what found it. Anyway, candle flame. More generally, Microgravity Combustion Science. I dunno, man. It is really neat, but I'm sad that microgravity is such a scarce resource that you can still do high school science fair-type experiments and get interesting results.

Tue Oct 10 2006 21:53 I Answer Your Search Requests:

• TOFU GOOD OR BAD: TOFU GOOD
• What is the fancy term that killer whales use for navigation and finding food: They call it "mrrrgrrraighwhaaai!"
• real dinosaurs: Take it up with this kid.
• the myth about awful snowman Once there was a terrible snowman who cleaved the world from primal chaos with one swipe of his corncob pipe! From the huddled masses shivering in the howling winds, he demanded obedience! As a symbol of his power he created a great fire in the sky! Which melted him, causing a big flood. Some say yonder mountain range was once the buttons running down his chest!

Wed Oct 11 2006 18:01 At's-a backed up!: Today I saw a plumbing truck that had a picture of Mario on the side.

(2) Fri Oct 13 2006 00:57: Today's linkdump theme is psyop, the nonlethal military tactic that is either a brilliant way to militarize the skills of drafted philatelists and comic book nerds, or one of your smaller-scale government boondoggles. I can't form an informed opinion because, as with other kinds of spam, there's little information available about the response rate. (I thought everyone knew about psyop, but Evan didn't, and I'm guessing he stands in for a chunk of the NYCB-reading audience.)

I've been browsing psyop sites like this one which is frequently updated, not just when there's a war; and this link-filled one. You can spend many a disturbing hour reading through those articles and browsing the scanned propaganda leaflets.

Two of my favorite articles are these WWII case studies: Gray and Black Radio Propaganda against Nazi Germany and H.M.G.'s Secret Pornographer. "In 1939 it never occurred to me that one day my turn would come to wage war on Hitler by pornography." Click through if you like, but there's no topping that sentence.

I can't find my favorite leaflet, but it was put out by the Germans in 1945 and it basically said "Don't get killed in the last five minutes of the war! We've already lost, so give yourself up and stay safe in a POW camp until Germany officially surrenders!" Not really on message; I think oversight was getting a little lax by that point.

(4) Fri Oct 13 2006 18:32: I came up with a new genre of game by combining two game genres I really like in the abstract, but which cause me only frustration when I play them. These would be the gravity game so beloved of the Finns (old-school example: Lunar Lander) and the robot programming game (old-school example: Core Wars). In the hybrid game, you would program a spaceship to fly through the spaceship caves of Finland, destroying other spaceships and not crashing into the walls. Sounds exciting! Sounds boring.

Sat Oct 14 2006 22:15 Some Lists of Games: We went to Connecticut today -- a totally different state -- and then came back. Doesn't make sense.

NES games: Top 100, bottom 20 with obviousness filters applied. All selections arbitrary.

(4) Sun Oct 15 2006 09:43 Elemental: Can't seem to get games out of my head; maybe because games are fun or something. I was thinking about how often games use the classical model of the elements as a game element. D&D is probably the first game to do this, with its elemental planes, but it also happens in Final Fantasy, ADOM, and probably about 25% of other games with fantasy settings.

Not sure how this happened; I'd blame Captain Planet, but I can't get the chronology to work out. Anyway, I'm tired of this so I was thinking of ways to improve on the concept. The obvious one is to use the modern elements, but that adds about a hundred new gameplay... elements, only a few of which are interesting. Another way is to go with the four fundamental forces. That would lead to some interesting gameplay. This is the Temple of the Strong Nuclear Force!

Mon Oct 16 2006 20:33 Guest Entry: Sumana's weblog is down, so I have given her this space to show you a link.

Hi all! Wife here. The folks competing for the Netflix prizes are having machine-learning fun with a big ol' dataset. There's Customer 305344, who rated more than 17,000 movies. There's constant wishing and jibing about getting a lower root mean squared error. And there's the question: why is Miss Congeniality the most-rated movie on Netflix?

The answer lies at the end of a lovely little essay by benjismith, which includes the SQL command ORDER BY hatred. As it turns out, Miss Congeniality shares something with Fahrenheit 9/11, The Passion of the Christ, and Con Air.

Mon Oct 16 2006 23:12: Niklaus Wirth's "Good Ideas -- Through The Looking Glass" is a really interesting survey of computer science ideas that didn't pan out, either by consensus or by Wirth fiat. I thought it would only be kind of interesting, but I got sucked in.

Mon Oct 16 2006 23:22: Early history of Atari:

In case anyone is mistaken in thinking Grand Theft Auto was the first videogame to have players willingly commit a felony, Watergate Caper tempted gamers to “Break Into Watergate Yourself” 30 years prior.

Here's a Watergate Caper flyer.

Tue Oct 17 2006 10:35 Sumana Guest Entry #2:

Planespotting geeks have been instrumental in breaking the CIA secret prisons story. This doesn't endear them to the military:

A few months later, Ray learned through "back channels" that his post about the Base Camp planes had caused heads to roll somewhere in the shadowy world of military "black ops." Apparently, someone, somewhere, had lost a job because of Ray's photos. At a desert bar, one of his friends -- a man with an unspecified connection to Base Camp -- warned him over beers to "stop messing with those Base Camp guys or you'll wind up dead in the desert with two bullets in the back of your head."

"Wouldn't one bullet be enough?" asked Ray.

Leonard suggests that even the Lone Gunmen would not have said anything so nerdy. I find in Ray's response a kinship to the old joke about the lawyer, the doctor, and the engineer about to be guillotined. The blade doesn't drop on the first two, so they're freed. Then the engineer lies down and says, "Well, there's your problem...."

(5) Wed Oct 18 2006 22:07 Trivia: Did you know that the Three Stooges were briefly Nethack characters?

(1) Thu Oct 19 2006 20:55 The Perfect Powerpoint Slide: If you've worked with me in any professional capacity you know that I have a habit of not finishing sentences when I write. I'll get an idea and skip ahead in the paragraph to jot it down. Sometimes I never come back to the sentence I was originally working on.

This happens within a sentence too. I'll start a sentence, then go somewhere else and tack something on to another sentence. Then, considering the start of the sentence to be a trivial corollary of the thing I just tacked on, I'll get right to work on the subordinate clause. Fortunately, the problem bottoms out here; I don't start writing a word and then go running off to write another word somewhere else.

I bring this up by way of introduction to this sentence I found just now in a book chapter, lying at the end of a paragraph like a gnawed bone:

If we , and , then when

Thanks a lot, prior self.

Fri Oct 20 2006 11:35 Coming Soon to Off-Broadway: The Phantom of the Toolbooth

Sun Oct 22 2006 21:25: I discovered Game Set Watch when it did an article on robotfindskitten, and since then it has maintained a similar level of quality. In the past couple days it's linked to two interesting and related items: Cory Arcangel's Super Mario Movie and I, Mario.

I went looking all over for a video of Super Mario Movie, but of course it's not a video, it's an NES cartridge. Get it from Cory's site. It's 15 minutes long and if I'd done it it would be about 7 minutes, but it's pretty fun and has great 8-bit music. You can also get the source, which reveals that there's a BASIC-like language for the 6502. Here's some info from a class on Game Design for the NES.

I, Mario is pretty funny but less interesting. Rather than making new things with the original Mario graphics, it retells the Mario story as accurately as possible within modern game conventions and using modern, anti-cartoon graphics. Rather like the disturbingly literal paintings of video games that if you know what I'm talking about you know what I'm talking about, and if you don't you probably shouldn't try to find out. It's a good concept but it suffers from a surfeit of world-building and belief that this will turn into a real game, and a shortage of people like the aforementioned painter who will just force their vision on the idea and let the chips fall where they may.

(1) Sun Oct 22 2006 22:30: Not since two-years-ago's beached mola mola, or a-year-ago-today's text adventure documentary, have so many people let me know about something as the Sun server farm in a shipping container. A million cyberpunk fantasies--fulfilled!

(11) Mon Oct 23 2006 19:24 YAAP: Today, after over 15 years of playing Nethack, I finally won the game. I came back to Nethack a week ago after a year or so of playing Slash'EM and ToME, and found that my "crazy wizard" strategy was working a lot better than it used to. After a number of frustrating late-game deaths I managed to get a powerhouse of a crazy wizard who, despite (or because of) my careful planning, tore through the late game with nary a scratch. The charm monster spell is astonishingly effective in the late game.

Anyway, I've now joined the ranks of the ascended. Now to work on that foodless zen pacifist caveman.

Tue Oct 24 2006 12:54 Three counts suspicion of cowabunga: I don't know where I got this dumb idea that I would do reviews of the websites that are advertising on Starslip Crisis, but was clicking around and I found a great 20-minute movie called Laser Force. Laser Force is a movie made by Texans with props from American Science and Surplus. It's cheaper than but greatly superior to that other famous Texan movie, Manos: The Hands of Fate. The mangling of language in the first half is excellent; then it goes a little off the rails. It's a fun watch though.

Also advertising on Starslip: This usually-funny raytraced comic, and a thing I'm not going to link to because it seems pretty sketchy, though it might just be arbitrage.

(2) Tue Oct 24 2006 19:09: Shuttle launch seen from the ISS an unspecified point further up than said launch.

Tue Oct 24 2006 23:53: A neat tool I'd never seen before: c64ize degrades an image so it looks like it would on a C64.

(3) Wed Oct 25 2006 00:19: Hey, Dan, what do you think about Coderific?

Wed Oct 25 2006 13:03: I was interviewed about stock spam for an edition of Marketplace that aired yesterday. In a pleasing departure from the norm, I talked for about half an hour and they used three sentences. That's a whole paragraph!

I used the interview as an excuse to update the Stock Spam Effectiveness Monitor main page to reflect some of the stock spam discoveries and developments in the past two years.

Thu Oct 26 2006 23:01 Metadata and Maps... Together at Last!: Ok, here are some proper pictures from the ISS.

Fri Oct 27 2006 08:13 To pity the fool, and for other purposes: Sumana came in wanting to talk about Barack Obama thinking about running for president.

I said "To get more votes, he should change his name to B.A. Barackus Obama."

Sumana said "You have to post this." So I did.

Update: Then we started talking about economics and I said "I pity the greater fool!" And she said I had to post that too. I don't want to turn this into a Mr. T version of mola mola day, so hopefully it will stop here.

(3) Fri Oct 27 2006 10:53 Oppo, You're The Laziest Man On Mars:

"Senator Bedfellow says he's for American jobs. But this one time, he ate a Toblerone! What's next, Senator--the killing of adorable kittens?"

Note to self: never claim that something would be "too flimsy to use in a campaign ad".

Sun Oct 29 2006 09:19 Belated Birthday Entry: Back in September I asked Ryan North to make Sumana a comic for her birthday, and he came through! AWESOME. Ryan also has an auction-based ad system that looks pretty nice.

Mon Oct 30 2006 20:13: I made another stupid Nethack patch.

Tue Oct 31 2006 23:27: Almost to the point where I can reveal my new book project to the world. Also quixotically working on a new SF story that might or might not be publishable.

Sumana and I have started sharing a private notebook, which turns visibly into sharing a del.icio.us account. Today she found for me a statement of the principles of URI design in rant form. You know that I'm a connoisseur of rants about things that aren't really designed to be ranted about.

(3) Wed Nov 01 2006 15:50 Involuntary NaNoWriMo: Several people I know are doing NaNoWriMo, including some of Sumana's coworkers. Sumana was tempted to do it but I told her she has too much else she has to do. I'm tempted to do it as well, but I already have a book deadline at the end of the month. So it's like I'm doing it already but not getting credit.

Wed Nov 01 2006 20:17: What's New at the Internet Archive. Subscribed!

Thu Nov 02 2006 22:40 Milestones Around My Neck: As of today my book is 150 pages long (public announcement coming soon, honestly, along with lots of me complaining about how things should be other than they are; which, though cranky, will provide interesting content on this weblog). Also, I finished my story today, and it did in fact turn into a novella. Now working on it with Brendan; it's called "Vanilla".

(1) Fri Nov 03 2006 08:51 Disturbing Slash Concepts: CJ from Reginald Perrin and CJ from The West Wing.

(5) Fri Nov 03 2006 17:31 REST Web Services: I'm writing a new book and it's called REST Web Services. It will be be the only book of its kind, and everyone should buy it who writes computer programs that work over the web.

There are lots of books about Big Web Services: complex distributed-object systems that reproduce the mechanics of method calls over HTTP. The problem is 1) these systems are way too big for what they do, and 2) they're on the web but they aren't of the web. They don't use any of the web's features, or interact with anything else on the web. They just use HTTP as a transport protocol. They could just as easily run over TCP and get better performance.

These un-weblike systems were able to take the name "Web Services" away from the competition because the competition is... the web. Just writing programs that interact with the web. Sounds pretty sketchy! The web may be all right as a platform for serving movies, or selling books, or trading stocks, or democratizing publishing, or coordinating huge volunteer projects, or searching much of the information currently in existence, but there's no way it's up to the task of managing Accounts Payable! For that you need... Web Services!

As Big Web Services gathered steam, the pro-web forces rallied behind the banner of REST: a name for the design philosophy that made the human-visible web so successful. They preached simplicity, addressability, statelessness and the uniform interface of HTTP. They also got into lots of heated arguments about what REST really meant.

Some organizations created services that claimed to be RESTful, and others critiqued those services and said they weren't RESTful really, and is "RESTful" even a word? Meanwhile the world's programmers started finding options in their IDEs that generated code for Big Web Service clients and servers, so they wouldn't have to do so much programming.

I got the idea for a REST book when I started seriously trying to figure out what was and wasn't REST. I noticed that though certain best practices showed up repeatedly in REST folklore, they never really progressed beyond that point. I decided to write a book that would set down the folklore and hopefully create some new canon, some common ground. I got Sam Ruby to agree to do the hard parts. And when I started work on the book I discovered a basic rule of thumb, a framing device that really focuses on what's important about REST, and makes it easy to tell what's RESTful and what's not.

We call this framing device the Resource-Oriented Architecture (we're not the first, but the other uses are compatible with ours), and I'm going to be writing about it a lot more in NYCB. It's too good to keep hidden in a book along with a bunch of implementation details.

Some people might say, "Leonard, why are you guys even writing this book? There already are lots of books about the web, and REST is just the web, right? That's the whole point! We wrote those books back in the 90s."

To them I say: "Some people, think about Ajax for a bit. Ajax, as a technology, is only about two years old, but it's composed entirely of technologies that are much older, like Javascript and XMLHttpRequest and the browser DOM. So why are there all these books about Ajax? (Including Christian Gross's Ajax and REST Recipes, coming out later this month and seemingly the only other book to mention REST in the title.) Why does Ajax excite people when DHTML was a big yawn? Because Ajax is a way of framing these old technologies so that people can do awesome things.

"There's nothing on the market that explains what REST really means and how to make it work for you. Nothing about programming the web in a way that's more structured than screen-scraping without falling into the pit of Big Web Services. This is a way of framing preexisting technologies that lets people do awesome things."

Then I add, "Speaking of Ajax, did you know that an Ajax application is basically a REST web service client that runs in your web browser? That's why Christian Gross's book is called what it is, and that's just one of the amazing reality-twisting facts you'll learn from my new book, Jon." Because I'm talking to Jon Stewart! I'm on The Daily Show, plugging my new REST book! It's a dream come true! But wait--I'm not actually on The Daily Show! I'm just on a clip from The Daily Show that someone uploaded to YouTube! And now someone is sending a DELETE request to my URI! I can see the Authorization request header, its value is "DMCA Takedown Notice", and nooooooooooo

Then I wake up. Whew! It was just a regular dream, not a dream come true. I get up and go back to work on my REST book.

Ruby is awesome and the Ruby Cookbook was a lot of fun to write, but it wasn't my idea. It's part of a high-profile O'Reilly series. There was destined to be a Ruby Cookbook, written by somebody, from the moment Rails strapped an E motor to Ruby and hit the igniter. This book was my idea, and Sam and I are driving it. In the coming months I'll be writing a lot more on this topic. There's a lot of work we need to do in the real world, in addition to writing stuff down in a book.

PS: I say this on the homepage for the book, but in case you haven't clicked through and you're interested in REST, please email me or Sam. We'd love to have you help us with the folklore and best practices, and we need lots of reviewers.

Sat Nov 04 2006 19:36 Daylight Come And It Won't Go Home: I probably shouldn't have announced my book project on a Friday afternoon. In my defense, yesterday I thought it was Wednesday. But the book gotten a huge reception to the point that I can only hope we can meet everyone's expectations. A huge number of smart people want to review the book so I feel pretty good about it.

Let me talk about light pollution. Recently I moved to a place where I have around 18 million neighbors. A lot of these people like to leave lights on at night. The effect is that every night the sky glows about as bright as the full moon (there's also a full moon tonight, which isn't helping). With curtains I can get a simalcrum of darkness, but I haven't seen (or, rather, been unable to see) real night darkness in months.

Now, I've seen the stars and they're not going anywhere, I don't need to look at the stars every night, but this lack of darkness is taking a toll on me. We're now at a point where there's about the same amount of ambient light from 5PM to 7AM, and the rest of the time there's normal daylight. I've never had any problems with SAD before I moved to New York, so I think it's not the darkness that's bothering me; it's the fact that it never gets really dark.

(6) Sun Nov 05 2006 00:35 TMLMTBGB: Aaron Swamp Swartz is in town staying with us, and we just went to see the Brendan-endorsed play "Too Much Light Makes the Baby go Blind", which is now also Leonard-endorsed. They have an hour to do 30 pieces of sketch comedy and sketch tragedy, ordered according to audience demand. Highlights included "Legoland Theater Presents: The Medieval Epic" and "Bite this, Reality!". Lowlights included preachy political Colbert Report ripoff. There is audience participation (one of my bugbears, or even owlbears), but it's easy to avoid by not sitting at the edges. Dadaism lives! Kind of. The group is called the Neo-Futurists but there was actually no Futurist content unless you count the idea of man as a slave to machinery (in this case the clock).

Also, almost everyone in the cast looked like someone else I'd already heard of. So you could pretend that the skits were being performed by an ensemble of Owen Wilson, Simon Pegg, Pauline Yates, Sumana's friend Steve Schultz, Aladdin Ullah from Uncle Morty's Dub Shack, and Jacquelyn, who I couldn't think of anyone she looks like.

(1) Sun Nov 05 2006 08:42: I keep getting spam with subjects like "Po..s.sible meeting" and "Fri.en.dship", and thinking that they're advertising hot new Web 2.0 sites.

Sun Nov 05 2006 20:12 Taxonomy of the Programmable Web: As I've learned more about REST (including formulating the ROA) I need to go back and rewrite the first few chapters. This entry is part of my attempt to document what I've learned since then. As I write these entries keep in mind that I'm not any kind of expert, I'm just a guy who decided to figure out REST and write a book about it. So if you disagree or think the something is called the wrong thing, leave me a comment or email and we'll figure it out.

The first problem is to sort out the terminology. There are many ways of classifying services, and we're introducing two more in the book: "Resource-Oriented Architecture", which I've mentioned before, and "The programmable web", which is a way of saying "web services" without implying Big Web Services, or even saying the word "services". We got the latter term from the excellent programmableweb.com. (We also introduce "Big Web Services" itself, which is a way of saying "web services" and implying only the SOAP/WSDL/WS-* stack, but I think I'm going to use that term a lot more in these entries than in the book.)

The main work thing I did today was draw diagrams: a taxonomy and Venn diagram. Don't worry, they'll be redrawn by real graphic designers before they're printed in the book. One or both might make it in; the Venn diagram is simpler but the taxonomy has examples and chapter references.

Both diagrams show two decisions I've made for the book: first, we draw a distinction between Resource-Oriented and Service-Oriented architectures. Second, we define HTTP+POX as a hybrid of the two architectures. The former because it's not useful to pit a technology against an architecture (as in "SOAP vs. REST").

The latter because the web services described as HTTP+POX, the ones that are kind-of-but-not-really RESTful, tend to deviate from REST in the direction of RPC and the Service-Oriented Architecture (rather than having no architecture at all, which was what I expected). Examples include the Flickr API, which includes explicit method names ("flickr.photos.search"), and the del.icio.us API, which has special URIs that you GET to invoke particular methods ("/posts/delete").

Some interesting edge cases: almost all web applications can be considered HTTP+POX web services. They're usually not all that RESTful, but they take standard HTTP requests with non-fancy entity bodies, and serve a non-fancy XML dialect (HTML or XHTML). This is how libraries like WWW-Mechanize look at the web.

In fact, I suspect the combination of URI- and RPC-based architectures to form HTTP+POX comes from our experience writing apps for the human web. You can't do nontrivial REST in a web browser without Ajax. HTML forms only support GET and POST, they don't let you set request headers, and they have hard-coded destination URIs (this is why URI templates are going to be great).

I used to think that, while the lack of PUT and DELETE was lame, people who obsessed about those methods as a determiner of RESTfullness were just being pedantic. But the ROA/SOA classification shows why it's important: if you have to overload POST, you end up creating extra URIs which the client POSTs to in different circumstances. This puts you well down the road to the RPC model. Your URIs no longer point to resources; they point to procedures you can call by POSTing. It's natural to keep following that model when you go from writing a web app for browsers, to writing a web service for computer programs.

Another edge case is that a website full of static pages is technically a RESTful web service. It's composed of resources (files), each with one URI and one representation (the data in the file). Each URI correctly supports a subset (GET and HEAD) of the uniform interface. This is how libraries like Beautiful Soup look at the web.

If you look at the Venn diagram you see that the entire human-visible web (static websites and web apps) is part of the programmable web. This is quite intentional. People were programming to the web long before anyone started talking about "web services". It's dirty work because the human web wasn't designed with software clients in mind, but sometimes it's the only way to get the data.

There's one thing in the diagram I'm not clear on. It might be a minor point or it might be an important aspect of our definition of HTTP+POX. The question is whether there are web applications that are so RPC-oriented they cross the limits of HTTP+POX and move in on the architectural turf of SOAP/WSDL services. I drew the Venn diagram with web apps that were neither REST nor HTTP+POX, thinking of really monolothic web apps that only expose a single URI, and do everything by POSTing to that URI.

The "POX" in HTTP+POX basically means "no SOAP", but what specifically don't HTTP+POX people like about SOAP? Is it the complexity of the SOAP message itself? Plain old XML can get pretty complex too. Or is it the fact that the SOAP message contains information (like the method name) that they think should go in the URI?

There are also, as one emailer pointed out, problems with calling anything a "web service" if we're drawing distinctions between a "service-oriented architecture" and something else. I'm not sure what to do about that. I'm tossing around the idea of switching that label to "RPC-oriented architecture", but I don't want to introduce yet another term of art.

Mon Nov 06 2006 11:30 open-uri ought to be a REST client: I spent a lot of time and used a lot of pages in the Ruby Cookbook showing how to use Net:HTTP to build more and more capable HTTP clients that hid the hairiness of Net:HTTP. But in truth the open-uri library, which we only covered a little, supports almost every feature you could want from an HTTP client library: custom headers, proxies, transparent HTTPS with cert validation, even automatic handling of the redirects it's okay to handle automatically. And the interface couldn't be simpler: it overloads Kernel#open so that you can open an HTTP request like a file, and read the response like data from a filehandle.

I'd like to recommend open-uri for everything, including as a client for REST web services, but there are two big features it's missing. You can't use any HTTP method other than GET, and you can't send any body data (which makes sense if GET is the only supported method). Lately I've been building on my Cookbook work, coming closer and closer towards a general HTTP client library for Ruby, but really all I'm doing is reinventing open-uri with two extra features. It would be a lot better to just add those features to open-uri.

I submitted a patch for open-uri on this topic a while back but have yet to get any response, even from people saying it's stupid. I'm writing REST clients with open-uri under the assumption that that'll work by the time the book is published. For instance, here's some code from my Amazon S3 client:

  # Saves this bucket to S3.
  def save
    open(uri, :method => :put)
    return self
  end

  # Deletes this bucket and all its contents.
  def delete
    open(uri, :method => :delete)
  end

I guess you could argue that these features break the metaphor, that PUTting to a URI isn't really "opening" it, or that it's like opening a file for write access and you should have to write the body data inside a code block the way you do for a file.[0] But c'mon, let's have that argument. I don't want to have to keep telling everyone about fake reimplementations of open-uri just so they can write REST clients in Ruby. There's also precedent, like XMLHttpRequest.open in Javascript.

I'd also like to add caching support to open-uri, like the caching support in Joe Gregorio's httplib2 for Python. But I'll wait on that until the fate of this simpler patch is decided.

[0] I doubt this is really the intent of open-uri, since right now it doesn't support HEAD either. I think the intent of open-uri is to make it trivial to make an HTTP request from Ruby.

(5) Mon Nov 06 2006 12:05 Amazing Science: Pulsating Gels Could Power Tiny Robots. Hey, what couldn't power tiny robots?

(1) Mon Nov 06 2006 20:51 Argh: Have to go another round with the post office. So aggravating.

(2) Mon Nov 06 2006 21:21 Nomenclature: Some good pushback on the Venn diagram and ROA/SOA dichotomy, primarily here at Labnotes. But also in some emails saying what amounts to: "Why are you putting 'services' in the name of your book if you then go and classify REST outside the Service-Oriented Architecture, today's hot buzzword that has 'service' in its very name? Why are you even saying 'web services' at all if you think REST is different from anything service-oriented?"

Good points, and highlighting an increasing source of uneasiness on my part. Which is good, despite the fact I generally don't like uneasiness, because it shows it's paying off to embarrass myself by formulating this book in public. People are interested enough in the topic to form uncoordinated revolts around ideas that are wrong or need work.

First a disclaimer about how much this matters: not a whole lot. This is a book about REST and the ROA. The name of the ROA might change yet for marketing reasons, but we've got its technical features pretty well defined. There is a chapter (#1) that covers the programmable web as a whole, a chapter (#3) whose first half covers services popularly considered RESTful that aren't really, and a chapter (#8) that compares REST/ROA to "the other guys".

I was calling "the other guys" "SOA" but that's starting to seem wrong. My current wild hare is to call them "Message-Oriented Architectures", though that has a bit of a collision with Message-Oriented Middleware. That way "Service-Oriented Architecture" is freed up to describe a higher-level architecture based on providing and consuming web services (ROA or MOA or both). Which seems closer to existing usage of "SOA", insofar as "SOA" means anything besides "your new issue of BusinessWeek is here."

Point is, our taxonomy could change completely and we'd only have to make real changes to those three chapters (and #8 isn't even started yet, partially because we're trying to sort this stuff out first). More important, much more detailed, and hopefully less contentious, will be our description of what goes on inside the "REST" part of the Venn diagram.

Anyway, our task here is the Confucian task of the rectification of names. We're trying to classify the set of techniques and technologies used on the programmable web. If it turns out to be best described with gradients instead of subsets, it might be a 2D or 3D map instead of a Venn diagram or taxonomy.

This doesn't have to be like the Linnean classification system where everyone signs onto it for all time, but it should be done with respect for the way "the other guys" see themselves. Sometimes in the text of the book we'll poke fun at what I call Big Web Services⁰. But we need to have a credible story about what Big Web Services really are, how they differ from RESTful web services, and what lives in between. Ideally someone who disagrees might say "Well, I wouldn't call it that, but I see how you draw the distinction."

So here are some unsolved mysteries of our taxonomy:

I don't think anyone will dispute that there's at least one rival web service architecture to REST/ROA: the RPC architecture that shows up in many web applications, in web services like Flickr's, and in XML-RPC. Can we consider this architecture as similar to the message processing architecture that rarely shows up in web applications, and that is commonly associated with SOAP? That is, granted there are differences, do they matter for a book about REST, or do they both differ from REST in a way we can capture in some snappy term like MOA?

Are all these architectures mutual opposites, or points on a gradient?

It it useful to distinguish between the typical robotic SOAP/WSDL interfaces autogenerated from Java or .NET code at the touch of a button, and the in potentia SOAP architecture that can be anything, even RESTful?

Consider the services popularly considered RESTful but seriously non-thesis-compliant. We currently term those things "HTTP+POX" in deference to common (kind of common) usage. Is that just another case of conflating architecture and technology? Is it worth coming up with another term? And given that there are multiple rival architectures for web services, do these not-quite-RESTful services consistently deviate from REST in the direction of some other architecture, or are they just freakish one-off mutants? Are they the missing links on a gradient between the rival architectures?

⁰ Said like in one of those political ads with lots of black-and-white closeups: "Senator Bedfellow doesn't want you to know about his millions in contributions from Big Web Services. When the FBI came knocking, he started shredding the XML documents. But he forgot to shred the envelopes. Senator Bedfellow: wrong on URIs. Wrong for the web."

Mon Nov 06 2006 21:56: As is all the rage, I have started posting book-related links I want to find or track under the del.icio.us tag restbook. If you want me to see a link, you can post it there. If you're secretly writing a REST book of your own and my tag cramps your style, let me know and we'll settle the namespace collision on the field of honor. Or the field of me renaming my tag to "oreillyrestbook".

Tue Nov 07 2006 00:04: Who would send me a Nethack T-shirt to congratulate me on my first ascension?

He's Seth David Schoen!

(6) Tue Nov 07 2006 22:21 Taxonomy of the Programmable Web! Part Deux: I know you're all for some reason anxious for the next report on my book progress. Well, in public it's all high-level philosophy but really the bulk of my day was spent chasing down Javascript and HTTP proxy hacks so I can finish "Ajax Applications as REST Clients".

However Sam and I did have a long IM conversation today, and cleared up a lot of taxonomy issues, as well as some of my misunderstandings about SOAP. I feel confident enough now to take another stab at my taxonomy of the programmable web. I got about halfway through before Ben came over to watch Battlestar Galactica, and that half is what I present to you tonight.

Again the disclaimer: though people will argue about these topics until the end of time, for purposes of this book we just need to get a coherent story about what it is we're cutting away as we zoom in on the programmable web to focus on the parts called "REST" and "ROA".

There are two major architectures on the programmable web. One is oriented around the passing of messages from place to place, and one is oriented around the manipulation of server-side objects called resources. We're (still tenatively) calling these the Message-Oriented and Resource-Oriented Architectures. Both architectures have real web applications and real web services to their credit.

These architectures can show up elsewhere than on the web, but we're not concerned with that. On the programmable web a service under either architecture uses the same underlying technologies: HTTP; a data format like XML, JSON, or form-encoded key-value pairs; and at least one URI.

Because they're built on HTTP, both architectures have a request cycle where the client asks the server to do something, and the server does it and sends a response. But we've found two questions which the MOA and the ROA answer differently:

1. How is the request scoped? How does the client tell the server what part of the application it wants to operate on? Let's say the purpose of the request is to fetch some data. Why should the server send back data X instead of data Y?
2. How does the client convey to the server what it wants the server to do? Why should the server send the requested data instead of modifying it somehow?

First a brief digression. When you send an HTTP request, you have two main options as to what to send in the entity-body. You can just send your data, or, if your data is an XML document, you can stick it in a box first. The name of the box is SOAP.

Just to avoid confusion: "SOAP" here is referring to the SOAP standard, not to the things people do with SOAP in the real world (that comes next). Sam says: "There isn't much to SOAP. It is three elements and a fault descriptor. Oh, that, and a world view." I'm just talking about the three elements and the fault descriptor. The worldview is the MOA.

Why would you stick your XML document in a box? Well, there are a variety of stickers that can be slapped onto the box without disturbing the contents. These stickers are the WS-* stack. That's the five-cent definition.

With that in mind, let's consider how the Resource-Oriented Architecture answers the two questions. A client for an ROA service scopes its request to a resource, by sending its request to a URI that identifies that resource. In an idealized ROA application there's one resource for every part of the application you might want to manipulate. Every resource has at least one URI, and you've got to send your request to some URI, so you might as well designate your resource by its URI. The URI goes in a special place in the HTTP request and can't be confused with anything else.

When conveying what it wants an ROA service to do to a resource, the client is restricted to a small and standardized list of methods. These methods are the famous HTTP GET, PUT, etc. Again, the HTTP method goes in a special place in the HTTP request and can't be confused with anything else.

If the client wants to convey an action that's not on that list, the problem is almost always that they've chosen the wrong resource. The service ought to expose some other resource for which the client's desired action translates naturally into one of the standard methods. The service might not actually expose this resource, but that just means the service can't do what the client wants. It's no different from the aggravation of a C library that's missing the function you need.

(There are edge cases I don't really understand, about overloading POST for atomic operations that span resources. I'm still trying to figure this part out. I think I may end up modelling this as a tiny MOA architecture inside the ROA.)

Now, depending on the service, when you need to send a document with your request, you might just send it as is, or you might stick it in a SOAP box first. The second possiblity, is, AFAIK, only theoretical. I don't know of any services I would describe as ROA where the client sticks its entity-bodies in a SOAP box before sending them.

This is partly for historical reasons. It's partly because it's simpler for the server to accept the entity-body as a document, instead of parsing it into a SOAP box covered in stickers, plus the "real" document. But it's mainly because HTTP already has stickers (HTTP headers). An HTTP request or response is already a box that can contain data and have stickers slapped on it.

HTTP stickers are more restricted in form than SOAP stickers, but they're good enough for most purposes. To the extent that ROA service programmers covet the features of WS-* stickers, they tend to devise ways of representing the same information in HTTP headers. Example: WS-Security's WSSE "ported" to the ROA as the X-WSSE header.

OK, enough about the ROA for a while. What about the MOA? And where do popular acronyms already in common usage fit in? Tune in tomorrow for the action-packed conclusion.

Wed Nov 08 2006 15:04: Beautiful Soup used to stream election results in Lawrence. As part of big ole chain of hacks. Lovely.

(3) Wed Nov 08 2006 18:10 Book Name: "Leonard is may be overly-self-congratulatory (perhaps with deliberate irony), but it sounds like there is a lot of demand for such a thing."

Always with deliberate irony, but my infernokrusher prose on the REST book page has two other goals. First, to market the book, to make people excited about it so I can ask them for help. Second, to act as a guideline and a goal. As the book nears completion I will find myself wrestling with the question of whether the manuscript does or does not put the "web" back into "web services". If it doesn't then we should probably do some more work on it. Does it drive all opposition before it, or is it just okay? Etc.

I'm exhausted, so no continuation of the taxonomy series tonight. But I would like to give an update on the name of the book. I got about 20 emails about it, almost uniformly against "with Ruby" in the title. On a semirelated note I can report that "with Ruby" is almost certainly gone from the title. The title is not finalized; editor Michael thinks "REST Web Services" is ungrammatical. But I think we agree on what kind of name will capture the goal of the book: an introduction to REST that has contains real examples but also has some philosophical heft to it.

The best argument for "with Ruby", I think, apart from "cheap trick to sell more books", is the same argument for using Ruby as our implementation language. We're not using Ruby because its XML support or HTTP client library is the best (it's not), but because Ruby is where the open source REST work is happening. This gives us lots to talk about without having to be polyglot throughout the whole book. Right now you don't see things like ActiveResource happening in other languages. (Note: this statement is precisely calculated to bring people out of the woodwork saying "you've neglected my awesome non-Ruby server-side REST tool!" so that I can take a look at said tools.)

The counterargument is that most peoples' interest in REST doesn't start on the server side. It starts when they write clients for big-company services like Yahoo!'s and Amazon's and Google's. They hear the term "REST" used (possibly incorrectly), and want to know more about the underlying architecture. There will always be more clients than servers, and the client programmers are using all kinds of languages.

Those whose interest does come from the server side are either trying to decide what to use, or just curious about RESt. Inside the book it's fine to say "Like REST? Your best bet right now is Ruby." But the book will fly past a chunk of its audience if the title says: "Like REST and Ruby? You'll like this book."

(1) Wed Nov 08 2006 19:11: Continuing my habit of clicking on Kris's ads I found the comic strip Station V3, which combines a ShaBot-esque sense of gag-based humor with a pretty good psychological study of being posted to a space station where nothing ever happens and you're surrounded by other dysfunctional people. A dimension that could be exploited to great effect by any number of other big-name media brands (I'm looking at you, Beetle Bailey), but never seems to be.

Strip includes cute fishlike aliens, and outdoes Battlestar Galactica in the low-tech science fiction aesthetic sweepstakes by having not just big clunky phones but big clunky payphones. Also you probably know this, but if you like science fiction comics with big space opera arcs and good character development you should check out Schlock Mercenary.

(7) Thu Nov 09 2006 12:50 Return of the Son of Taxonomy of the Programmable Web: Previously, on Taxonomy of the Programmable Web, I put out a description of the Resource-Oriented Architecture based on its answers to these two questions:

• How is the request scoped? How does the client tell the server what part of the application it wants to operate on? Let's say the purpose of the request is to fetch some data. Why should the server send back data X instead of data Y?
• How does the client convey to the server what it wants the server to do? Why should the server send the requested data instead of modifying it somehow?

Now I'm going to put out a description on the same principles of the Message-Oriented Architecture. Then I think I can make a case for how to classify weird cases like HTTP+POX.

To state the obvious (though it stops being obvious if you or we call the MOA something else), the Message-Oriented Architecture is about passing messages back and forth. Client sends server a message, server takes some action and sends client a message in response. A SOAP box with stickers on it is an example of a message. An HTTP request with headers is also an example of a message.

In an ideal MOA there is no information outside of the message. So if the message is in a SOAP box being transmitted over HTTP, then all the features of HTTP are just distractions and should be used as little as possible. There should be no leakage from the message into the transport protocol.

In the ROA, the fundamental server-side object is the resource. An ROA service usually has an infinite number of resources. In the MOA, the fundamental server-side object is the message processor, sometimes called an endpoint. A MOA service exposes a small number of message processors: usually one. A message processor is the thing that has a URI (or, possibly, the thing that can be the target of the WS-Addressing SOAP sticker).

Side note: A resource is just a message processor with a very limited vocabulary. So there's a pretty easy conceptual translation between the ROA and the MOA. This makes sense because HTTP is itself a message-oriented protocol. Of course, almost all ROA services expose too many "endpoints" to list. You either generate their URIs according to a rule, or you follow links in a document the server sends you.

An MOA might have ten or 200 or a million endpoints, but it's not ROA unless it has an endpoint or "resource" for every object the client might manipulate. Any service that exposes an infinite number of endpoints has at least some of the ROA in it.

This ties back into the famous triangle of "nouns" (resources), "verbs" (HTTP methods), and content types. The ROA constrains the "verbs" and lets the "nouns" run free. The MOA constrains the "nouns" and lets the "verbs" run free.

Side side note: an ROA service can contain a finite number of endpoints, but only if there's a finite number of objects the client might manipulate. A static website (like your cat's homepage, or the site that serves Google Maps image tiles) is an ROA service with a finite number of endpoints. A web service version of Parmenides's model of the universe would only have one endpoint, because there is only one proper object of discussion.

Now let's consider how the MOA answers the two questions:

How is the request scoped? Why should the server do x to data D1 instead of data D2? Well, in the URI it's scoped to the message processor. If your MOA service is actually ROA then that's all you need, because the "message processor" is the "resource" is the part of the application you're operating on. Otherwise, you need some additional scoping, and this is done by putting information in the message itself. This might be in the box or it might be in one of the stickers; the details don't matter.

How does the client convey to the server what it wants the server to do? Why should the server do x to data D1 instead of doing y to data D1? This too is scoped by information within the message itself. Again, this might be in the box or in one of the stickers; it doesn't matter to the MOA. Unlike with the ROA, the vocabulary of this sort of information is not constrained at all. If you constrained it you'd need to create more endpoints to handle the same actions with the more limited vocabulary, and you'd move towards the ROA.

Since in the MOA both of these pieces of information go into the message, there's a tendency to conflate them, and indeed that's what I used to do.

To recap from last time: the ROA puts scoping information in the URI, and puts the action information in the HTTP method. The big exception seems to be overloaded POST, where the action information can go in the entity-body. I'm now fairly certain that overloaded POST in the ROA is best described in terms of the MOA; I'll try to cover that next time.

Now I'd like to talk about a common design style for MOA services: the RPC style. In an RPC service, the scoping information corresponds to the information that would identify an object if this were an object-oriented program. The action information corresponds to a method that should be called on that object. As is proper with the MOA, both of these go into the message.

The message also contains some additional information, which corresponds to arguments passed into the procedure. The scoping information can be found among these arguments: things like the bank account number or the user ID. This is because an RPC service isn't really object-oriented. If it was, you'd be able to work directly with the objects (as, say, resources) instead of going through message processors. But with a little work you can get the same kind of "fake" OO you see in the GNOME project's C code, where the first argument to every function is a pointer to the "object". I put "fake" in scare quotes, possibly sending the most mixed message ever, because I think you can do real OO design in a non-OO system. But it looks strange, and you have to do work to get it.

In an RPC-style service, the HTTP request-response cycle is used to simulate a method or function call in a programming language. The request is the method invocation, and the response contains the return value of the method, or the exception it throws.

We all know that method calls are not neccessarily simple stuff where I ask you for information and you return it in a data structure. This "method call" might do anything: fetch information, store information, put a job in a processing queue, set up callbacks for later, etc. The response message might contain useful information, it might describe an exceptional condition, or it might just be a formality. If you register a callback the real data might be coming in later through the callback. So saying "RPC style" is not a negative value judgement about the capabilities of the style.

Let's take an easy case: XML-RPC. It's obviously an MOA architecture, since the only URI in an XML-RPC service is the URI to the XML-RPC service. It also rejects HTTP headers, methods, and status codes almost entirely, preferring to convey all information information in its XML message document.

It's also obviously an RPC architecture, because 1) it says so right in the name, and 2) its messages are full of incriminating tag names like methodCall and params.

Now, what about SOAP? Is SOAP an RPC style? No, that's a category error. SOAP came from XML-RPC (more accurately: they have a common ancestor) but they aren't the same kind of thing. XML-RPC is a way of describing method calls; SOAP is a way of sticking a message in a box and putting stickers on the box. This makes it useful for any MOA application, but it doesn't make it RPC.

So, riddle me this, Batman: how come basically the entire installed base of services that use SOAP also use the RPC style? Why does the average programmer think SOAP and then think RPC style?

SOAP is the message format of choice for automated tools that take Java or C# code you've already written, and with one click turn it into a web service. These tools are to web services as FrontPage is to web pages. The resulting "web service" exposes only one URI and accepts methods only in a very specific form that a person can't comprehend without a detailed map (a WSDL file) and a software tool on the other side (a WSDL client) that can read the map so you don't have to. When the software tool is done reading the map it's reconstructed something like the method signatures of the original Java or C# code.

This is why REST people dislike these services so much, and why that dislike often spills over into SOAP. This is why people tend to think that anything that doesn't fit this model is "REST". This is why I say these services are not really on the web: they're looking out at the web through a little peephole. To get any information out of them, you have to construct a very specific document, and then you have to put in a box and put the right stickers on it.

These services have the same design as XML-RPC services. They use XML Schema instead of XML-RPC's idiosyncratic data serialization format, and they use WSDL instead of the listMethods extension to XML-RPC. All the work done to make SOAP nothing but a good way of putting an XML document in a box and slapping stickers on the box, has passed these services by.

I don't like this because it's ugly and it pollutes the web. SOAP fans don't like this because it conflates SOAP with RPC when what they use SOAP for isn't RPC at all. I'm not really clear on what MOA styles they are using if not RPC, but I don't think it's going to come up a lot in a book about REST and the ROA, so I can just leave them to it and have Sam act as my lookout in case anything relevant to the book happens.

Who's the culprit here? I think it's those tools. They make it easy to get a web service that has the same interface as your code (that'd be a Procedure-Call interface that works Remotely), disregarding the fact that the architecture of the web doesn't look like that. Combine this disconnect with the complexity of the interface and the idiosyncracies of the tools, and you've made it easy for someone else to work with your service if they've got the exact same setup, and very difficult otherwise. This is pretty disgraceful since the real-world power of the web comes from its ability to connect everyone together.

The secondary culprit is WSDL. WSDL doesn't enforce RPC but it makes life easy for tools that want to force SOAP into an RPC mold. Sam is ambivalent about WADL. I suspect he's afraid that WADL will make it easy to force HTTP into an RPC mold, that tools for doing this will proliferate, and that we'll never see resources again. Sure, WADL makes you specify your resources, but a smart tool could easily subvert that and create services with a single "resource" that handles a whole bunch of method calls.

I think the scenario I've ascribed to Sam is not inevitable. A lot of the mess we see today comes from historical contigencies like SOAP's heritage in XML-RPC. But to prevent a future without resources we need a basic book that's willing to go out and fight for them. It means we need to straighten out the terminology—even if only informally and we only use the terminology locally—so we can create a frame for our argument.

Next time: HTTP+POX.

Fri Nov 10 2006 16:22: OK, here's the deal. On any given day I can do book writing or I can write one of those big ole weblog entries. Today was book writing day. I got the "Writing Clients" chapter more or less done, bringing us to our milestone of having drafts of half the chapters. This is one of those milestones where I get paid, so I'm happy.

The Ajax chapter is also done, and I'd like someone who knows more than I do (ie. something) about Ajax to look at it.

Fri Nov 10 2006 21:06: Picked Sumana up at work and walked down to Chelsea for dinner. My first choice, "Elmo", had no vegetarian entrees, so we went to a nearby place called Food Bar (not Foo Bar) that was good. We then walked to the Strand, where giveaway copies of books go to die, and Sumana found some cheap books she wanted. A nice evening out and a relief from looking at a computer screen all day.

Sat Nov 11 2006 23:27 Picasso: Today hung out with Evan and his father, which was a lot of fun. We went to the Whitney (conveniently hidden in a totally random block of Manhattan) and saw the Picasso exhibit. The theme was to have a Picasso, and then a bunch of paintings by other artists (Lichtenstein, Pollock, Jasper Johns, etc.) who'd ripped off the Picasso. It was pretty neat, and there was an Edward Hopper exhibit as a bonus. That guy's sketches are amazing.

There were two Picasso paintings (the really famous ones) that the MOMA owns and wouldn't lend to the Whitney. So the exhibit just put up the ripoff paintings and had tiny passive-aggressive reproductions of the originals along with explanations along the lines of "Girl Before a Mirror, in the collection of the Museum of Modern Arts since 1944, the cheap bastards!..."

(5) Sun Nov 12 2006 20:07 The Rite of First Sale: Earlier I declared, "I've never done so much work for a hundred dollars." Well, I'm moving up in the world, because now I've never done so much work for two hundred dollars. I got email today saying that Futurismic bought my story Mallory!

This is great news for me as it demonstrates that I can write fiction people will buy. Look for it sometime next year, probably a little before the REST book is published.

(1) Mon Nov 13 2006 12:56 Abbot and Costello Meet the Taxonomy of the Programmable Web: So far I think I've made a coherent case for classifying RESTful and resource-oriented services on one side of the Venn diagram, and message-oriented services on the other. The RESTful and resource-oriented services keep the scoping information (why should the server send you some dataset instead of some other dataset?) in the URI, and action information (why should the server send you the dataset instead of modifying the dataset?) in the HTTP method.

The MOA field is dominated by RPC-style services, most of which use SOAP messages constrained by WSDL files. The client communicates its desire by sending a message (a box with stickers on it) to a message processor, which unpacks the box to find both the scoping information and the action information.

We've got two edge cases to explain: overloaded POST and so-called "HTTP+POX" services. It's been said (see link at end) that I shouldn't try to fit HTTP+POX services anywhere at all because they're just a mishmash, but I press on.

We've also got an unexplained coincidence. I've been describing SOAP as a way of putting a document in a box and slapping stickers on it. But you can look at HTTP the same way. The HTTP request (or response) is the box; the entity-body is the document; and the URI, headers, method name, and response code are all stickers.

HTTP+POX, as they say, is a mishmash, but I think I can identify the things it's a mishmash of. An HTTP+POX service is a message-oriented RPC-style service that uses HTTP as the box format, rather than using SOAP.

This is most obvious if you look at how you modify the dataset in HTTP+POX services (I'm using the del.icio.us API and Flickr's allegedly "REST" API as examples). You either send a GET or an overloaded POST to a URI that's not a resource: it's a description of the procedure you want to call. There's a message processor for every operation, rather than just one (as generally happens with SOAP+WSDL services), or one for every object on which you might want to operate (as with resource-oriented services). Examples: del.icio.us /v1/posts/add and /v1/posts/delete, Flickr's multitude of methodName arguments.

The confusing part is this: when the RPC method you're calling is "get some data", the HTTP method is usually GET, and the arguments to the RPC method usually go into the URI. This URI designates a resource, and you can GET a representation of that resource! This is why I thought that HTTP+POX services are hybrids of the resource-oriented and message-oriented architectures. In many cases, the message is one you'd send if you were using the uniform interface to get a representation of a resource. The service may not have been designed with resources in mind, but this part of it is functionally resource-oriented.

Examples. Consider an endpoint of the del.icio.us API: https://api.del.icio.us/v1/posts/get?tag=restbook That doesn't have the same ring as http://del.icio.us/leonardr/restbook, but it's the same kind of URI. Those are two URIs to two representations of the same resource: "my recent posts tagged with 'restbook'". There are infinitely many URIs of this form, each identifying a different resource.

Similarly for http://api.flickr.com/services/rest?method=flickr.photos.search&api_key=xxx&name=penguin. We're supposed to interpret that as a remote procedure call, but it's also the URI to a resource: "Pictures tagged 'penguin'". Another URI to a different representation of the same resource is http://flickr.com/photos/tags/penguin. Even a URI like /rest?method=flickr.people.findByEmail&find_email=leonardr@segfault.org is the URI to a resource, though a better URI for the same resource might be /people?email=leonardr@segfault.org.

We have our own recommendations about how to structure URIs in a resource-oriented service, but it's pretty small-minded to say that services aren't resource-oriented just because their URIs look funny and contain things that look like method names (like "get" and "method=flickr.photos.search"). From a functional standpoint, these services stop being resource-oriented when they stop using resources. And, in general, they stop using resources when it comes time to modify the data set.

The method names in those URIs are useless to the extent they agree with HTTP's uniform interface. The "get" and the "flickr.photos.search" basically mean HTTP GET, and the client's already using GET. When those strings become useful ("add" or "flickr.photos.comments.deleteComment") it's because the RPC interface supercedes the uniform interface. The URI can no longer be conceived as pointing to a resource.

/rest?method=flickr.photos.comments.deleteComment&comment_id=100 could be one of many URIs to the resource "comment #100". Maybe the "method" argument is just random junk. Maybe if you GET this URI you get the comment, and if you delete this URI you delete the comment. But of course if you GET this URI you delete the comment. This URI doesn't identify the comment: it identifies an operation on the comment. That's not a resource. It's just a procedure call.

You can see this by looking at web applications. A web application like a search engine is pretty resource-oriented. It exposes an infinite number of URIs that slice up the search engine data in various ways: by query, by page, by language searched. All of these URIs point to resources. You can't see any cracks between the message-oriented and resource-oriented models, because 1) the "messages" you're sending are standard HTTP requests, and 2) you can't modify the search engine data through the web interface, so you only use GET and you use it as intended.

Then again, consider NewsBruiser, the weblog program that publishes these words. I wrote NewsBruiser before the term "REST" was coined, but it's pretty typical of examples even today. I've got URIs like /nb.cgi/add, /nb.cgi/comment-add, /nb.cgi/configure, and so on. In general GETting one of these URIs gives you an HTML form, which you fill out and POST to the same URI to modify the data set.

Even putting aside the fact that you can't do PUT or DELETE with HTML forms, this is a very RPC-oriented application. It's got method names in the URIs. Yet when you're not changing the application you're using URIs like /nb.cgi/view/nycb/2006/11/03/1. That's got a method name ("view"), but it's redundant with the name of the HTTP method (GET), just like with the del.icio.us and Flickr examples. The rest of the URI describes a resource: the second News You Can Bruise entry of the third of November, 2006. Indeed, I've got a rewrite rule that cuts out all the redundant and default data so that you can use /2006/11/03/1 instead.

So that's HTTP+POX. It looks like there are two main paths to this architecture. You might come to it by designing web services on the exact same principles as web applications. This overlaps with the resource-oriented architecture when that architecture congrues with the capabilities of web browsers (GETting data from resources identified with URIs, sending data to hard-coded locations). It departs from the resource-oriented architecture when it comes to things you can't do with a web browser (using PUT and DELETE instead of overloading POST so much, sending data to URIs that the client generated dynamically).

Or you might come to it through an attempt to simplify the RPC style. You might be doing pushbutton SOAP+WSDL services and one day say "screw only having a single endpoint; I'm going to have an endpoint for every procedure and it's going to be identified in the URI and then I won't need WSDL". Or you might say "screw putting an XML document in an XML box; I'm already using HTTP, I'll just use it as my box and then I won't need SOAP".

Now a word about the name. I've decided "HTTP+POX" is inaccurate and I think I won't use it in the book except insofar as I need to translate real-world terms into my wondrous precise terminology. Here's the problem: like "AJAX" of old, "HTTP+POX" hard-codes a reference to XML where there might not be any XML.

To reiterate one of my earlier points in new language: what's "Plain Old" about Plain Old XML? What's "Plain Old" is that it's a document, not a box with a document inside it. You've already got a perfectly good box: the HTTP request. But that's a statement about the box. "POX" is a statement about the contents, and those don't have to be XML. An HTTP+POX web service might serve plain text, JSON, HTML microformats, or graphic files in an HTTP box. This holds even if you don't buy my argument that HTTP+POX is an RPC-style architecture.

I can think of two different names. One involves doing what they did to AJAX and lowercasing the "POX", making it a word instead of an acronym: HTTP+Pox (HTTP can stay an acronym because you are using HTTP and only HTTP). This has the advantage that it's an almost invisible change. But depending on how cocky I feel about my classification of these services as RPC services, I may try to push the term HTTP+RPC. Because "Pox" doesn't mean anything, and I don't like the trend towards stripping acronyms of their meaning when it becomes inconvenient (as also happened to SOAP).

The next installments will cover what to do with terms like "Service-Oriented Architecture" in the face of definitions that would a priori prevent REST from getting any of the pie, and also finally tackle the question of overloaded POST in resource-oriented architectures. I think it's going to be a while until the last one, at least, shows up. Before I try to figure out overloaded POST I want to incorporate into the book text my thinking in the series so far.

(2) Tue Nov 14 2006 00:31: This one's pretty short; I just want to respond to Pete's entry about the term "SOA". I have largely bought into Assaf's assertion that defining resource-oriented services out of the SOA cuts them away from a large source of largess and buzz for no real reason. I don't really care about "SOA", but I do care about people spending money in the right places, so I want to leave the rhetorical ground open for those who can make that happen.

Pete quotes Anne Thomas Manes as saying:

Service oriented architecture (SOA) is a software design discipline in which application and infrastructure functionality are implemented as shared, reusable services.

As Pete points out this definition has a dependency on the definition of "service". Pete's definition is "a network available collection of related operations".

Okay, I'll buy that, and I'll also buy a $0.50 contract on Pete's "a resource is not a service". I'd say a resource is a service, albeit a very limited one that can only operate on one piece of data. An RSS feed is a good example of a one-resource service. But a collection of resources is definitely a service. In fact, "service" is what we call a collection of related resources.

A single resource can't have arbitrary operations, but you can expose arbitrary operations by exposing combinations of resources. It's not the same design as an RPC service, but only intertia associates RPC with "service-oriented" in the first place, and a service-oriented system could easily consume a mix of resource-oriented and RPC-style services.

Wed Nov 15 2006 20:38 A Plan Comes Together: with ElementSoup

Wed Nov 15 2006 20:40: Spent most of the day doing Ruby Cookbook errata (new printing going out next week), which was pretty humbling. Also converting the REST book to Docbook, which is AGGRAVATING. Not least because I need to rewrite the first part of the book anyway and I keep conflating the Docbook conversion with the rewrite.

PSEUDO-UPDATE: I finally finished the errata and decided to put up an update. Then I saw I had saved this entry as a draft and not published it. So this isn't a real update. Incidentally, the previous entry was one I wrote back in August and saved as draft instead of publsihing.

(2) Sat Nov 18 2006 00:12: Sorry for no NYCB; been converting the book to Docbook and tearing apart the first part so I can stick Taxonomy of the Programmable Web stuff into it.

Sun Nov 19 2006 09:40: Susanna and John are here for Thanksgiving and sightseeing. Susanna has a baby Beet inside her, as promised. Today we go to the park and the Met.

(4) Tue Nov 21 2006 10:12 Borges: A Life: Been reading this book (the Williamson one) in stolen moments because it needs to go back to Rachel after Thanksgiving. It starts out really slow and boring but about 2/5 of the way in starts to ramp up and gets very interesting. Unfortunately 2/5 of the way is 200 pages, and it turns out you need most of that stuff because it's describing what later turns out to be the neuroses Borges can't get rid of for the rest of his life.

Wed Nov 22 2006 22:04: Tomorrow: Susanna and I are in charge of Thanksgiving. It's going to be great.

I've got much of the REST book converted to DocBook and am putting what I write online. Send me email if you want to do review.

Wed Nov 22 2006 23:03: From email conversation with Danny:

> > I SHALL TAKE THE CHALLENGE!
> Join me or die! Can you do any less?
I can't think of any way I could do less in general, but I'm always on the lookout...

Fri Nov 24 2006 19:15: There are some kids who come out and play in the concrete "backyard" of the apartment building across from ours. That "backyard" is also where people stick their trash, and recently the kids' play has incorporated the trash.

There are two different reactions to this: "We must stop them!" and/or "I remember doing that sort of thing." I am in the latter camp because the kids seem to know what they're doing; they play with the abandoned furniture instead of rooting around in the trash bags.

Sometimes it I think they know what they're doing a little too well. A while ago someone threw out an old wide-screen TV (a big box for a plasma-screen showed up around the same time). The other day some kids were sitting in front of the TV, coloring with markers and pretending to watch TV.

Sat Nov 25 2006 17:21 Post-Thanksgiving Breakfast: Bowl of stuffing with fried egg on top. Delicious.

(2) Sun Nov 26 2006 18:27 Cloisters: Today Sumana and I went up to the tip of Manhattan and visited the mini-Met museum of medieval art at the Cloisters (actual monastery cloisters imported from Europe). It's in a really nice park (Fort Tryon) so we also had fun walking around the park, and had a pretty fancy brunch at the in-park restaurant, New Leaf. We watched the sun set over the George Washington bridge.

Some funny bits of conversation from our day:

"Wel-come to the social! Zune you'll be dead!" <-Cryptkeeper voice

"I don't know how all these little stores can survive."
"They survive because there are millions of people here, and they all demand breakfast sandwiches!"
"And punctuation errors."
"Yes. They 'demand' breakfast sandwich's."

I got a Met membership so that I can go whenever I want without having to pay the exorbitant "recommended donation", or having to deal with the passive-aggressive consequences of trying to get in for less. Last time I was at the big Met I found out that my favorite objects d'art in that museum, the ancient Chinese office supplies, are part of a temporary exhibit that goes away in January. Go see that exhibit! It's great! And it also has a bunch of cool Xu Bing art that I just discovered last time.

Sun Nov 26 2006 22:50 Quick Restaurant Review: Annie's on 3rd Avenue between 78th and 79th. A little bit expensive, but good food and lots of it. Fairly vegetarian-friendly. Good place to eat after seeing museums.

(2) Mon Nov 27 2006 16:52: Misc. tidbits from conversation last night:

• Brandon Bird should paint an Adoration of the Magi in which the Magi are Kirk, Picard, and Sisko.
• Brandon Bird already painted a Last Supper, but perhaps some British equivalent to Brandon Bird could paint a Last Supper featuring all the incarnations of the Doctor from Doctor Who.
• Dun-da-dun-da-dun. Investor Gadget.
• It's well-known that a lot of the intro songs to James Bond movies are really lame and don't even have the same names as the movies. I now plan to write a song cycle of 21 songs, each with the name of a Bond movie.

(3) Mon Nov 27 2006 20:41 Karaoke For Beginners:

• "Tequila"
• "Wipe-out"
• "Meow Mix Commercial" (theme)

(1) Wed Nov 29 2006 16:44 Shake Shack: Today I went to the Shake Shack on my way to the farmer's market. I went because it's supposedly a New York tradition, and pretty soon it closes until spring. I had a hamburger and a "concrete", which is a sort of thick milkshake made from frozen custard and extra stuff.

I've been to a frozen custard place before: Carl's in Fredricksburg, Virginia. So I knew what to expect. And it was good, except here I ordered a concrete that included Valrhona chocolate disks or whatever they're called. I didn't really know what they are so I said "Disks? Sign me up!" Then I got the concrete and remembered what Valrhona chocolate disks are. Those guys are great... FOR BAKING. For putting in frozen drinks they're not so great. It's like putting chopped-up Snickers bars in a milkshake and then crunching on frozen Snickers bar all the time you're drinking it. When you get a concrete, stick to extra things that are soft.

Hamburger: eh. I probably shouldn't even have gotten this since Shake Shack had about 150 health code violations (not an exaggeration) last time it was inspected. It wasn't very good. Andy and I have been going out on excursions where we try out the city's hamburgers, and Shake Shack is pretty lousy. I'd rate it below Burger Joint (trashy take-out burger restaurant in fancy Parker Meridien hotel, primarily famous in class-obsessed New York due to its class-transvestitism; there are celebrity autographs on the wall inc. Stephen Colbert's), which doesn't even use real hamburger buns. In conclusion: Shake Shack is for frozen custard with soft things in it. That stuff is good and you're likely to run afoul of fewer health code violations.

If you're curious, best hamburger discovered so far: Zip Burger at 300 E 52nd. They have Belgian fries too, and non-perfunctory veggie options.

Thu Nov 30 2006 09:22 1995 Follies: Sumana did a presentation for her corporate finance class where she pretended she was a Morgan Stanley investment banker trying to get people to invest in Netscape pre-IPO. I suggested she start off by saying "I bring you tidings... FROM THE FUTURE!". She said "That's basically what she [the professor] explicitly said we couldn't do."

So instead she talked about Netscape's server business and the promise of the web and blah and blah. There were Powerpoint slides, which seems somewhat anachronistic for 1995.

Anyway, this morning we were doing a postmortem, and the conversation turned toward the mid-90s integration of the proprietary online networks (Compuserve, AOL, etc.) into the Internet. Specifically, the common shunning of AOL users. Any mention of shunning calls to Sumana's mind Gentlemen's Agreement, specifically the scene where Gregory Peck can't get a room at a New England inn.

"Yes, I'd like a room." <- Rugged Gregory Peck voice
"Certainly, sir, I think we can accommodate you. If you'll just put your email address down to get on our mailing list."
"gregorypeck22@aol.com..."
"Ah. Oh, I see, sir. I'll be quietly shunning you now, sir. It's a free service we provide... FOR THOSE WE DON'T GIVE ROOMS."

Thu Nov 30 2006 22:08: Well, my performance in this year's Nethack tournament was pretty lousy, mainly because it wasn't until around Tuesday that I thought "Hm, what would be fun to do in spare moments right around now? Oh yeah, Nethack." I did have one good character but killed her off stupidly.

Almost all of REST Web Services so far is rewritten and in Docbook. I planned to get the whole thing done by the end of the month, but that's just another goal I didn't meet: still have half a chapter to redo. I'm very happy with the first 5 chapters so far, though. Total length in O'Reilly's stylesheet is 200 pages. Again, email me if you want to read the REST book as I write it.

Sat Dec 02 2006 22:10: "All these pictures are 360° panoramas projected to look like small planets." All panoramas should undergo such a projection. I have decreed it!

Sun Dec 03 2006 10:54: It looks like the Libre Map Project is paying off! I'm always happy at reductions in the gap between "public domain" and "available online".

Sun Dec 03 2006 22:24 Commercial:

"Hi. I'm a human..."
"...and I'm a Cylon."

Sun Dec 03 2006 22:46: I was reading a Steven Levy book (Insanely Great) about the development of the Macintosh, and after a while it went a little off the rails and started talking about things that were not t.d.o.t.M. But one of the things it mentioned was John Sculley's blue-sky idea of a notebook computer called the Knowledge Navigator, with software agents and voice recognition and basically everything you could want for the Navigation of Knowledge. Sculley even had a bunch of long-form commercials made for it.

Well, it didn't take me long to get up and search for those commercials online, where they now live. They're surprisingly similar to Douglas Adams's much more enjoyable Hyperland. In fact, I think Steven Levy was thinking about Hyperland when he described some aspects of Knowledge Navigator.

(1) Tue Dec 05 2006 00:49: Good news, everyone! Long-time NYCB favorite The Sugar Beet has a book coming out!

Wed Dec 06 2006 23:31: Dan's 20th Century Abandonware has lots of nice photos and screenshots.

Wed Dec 06 2006 23:35 Backup Thanksgiving: When the Thanksgiving leftovers start to run out it's time for Backup Thanksgiving. In recent years I made Backup Thanksgiving because I was learning to cook, or because of the absence of certain foods from the official Thanksgiving table. This year I made it because it's just good food.

This year Backup Thanksgiving was roast chicken, orange-cranberry sauce, dressing, mashed potatoes, mashed sweet potatoes, carrots, green beans, and Parker House rolls. So delicious. But now Sumana's sick of the leftovers and I'm almost done reincorporating them. I made stock from the chicken carcasses and bread pudding from the cranberry sauce. Now I have to start thinking about what to make instead of just pulling stuff out of the fridge.

PS: The first six chapters of the REST book are now done!

Thu Dec 07 2006 12:33 We'll Distribute Anything: Sumana was looking at her old Abbot and Costello tape (which, she now realizes, started her interest in sketch comedy) and was curious about what else the ultra-80s distributor GoodTimes Home Video was in the habit of distributing. A little of everything, it turns out. My fave has got to be: "Form... Focus... Fitness, the Marky Mark Workout".

Thu Dec 07 2006 12:33: mah gets spam from Phillip K. Dick stories

Thu Dec 07 2006 19:28 QOTD: Sumana: "That's a good explanation. Of something else."

(1) Fri Dec 08 2006 23:03 Cool, Refreshing Nilas Ice: Today's EPOD: Nilas Ice

Sat Dec 09 2006 19:26: Rocket mail. Seems not cost effective.

The mail consisted entirely of commemorative postal covers addressed to President of the United States Dwight Eisenhower, other government officials, the Postmasters General of all members of the Universal Postal Union, and so on. They contained letters from United States Postmaster General Arthur E. Summerfield.

Reinforces my belief that the postal service makes most of its money acting as a kind of live-action MMORPG, arbitrarily creating things and declaring them to have value.

(1) Sun Dec 10 2006 07:51: Wikipedia is your great trivia game collaborator. Sumana and I played that game for a couple hours last night and it was a lot of fun. It's hard to find something appropriate by visiting random pages, but you can do well by picking a category and going through it.

You do have to pick something in a field the other person knows fairly well, because the fun of the game is in being able to make a valid guess every time a new piece of information is revealed. I did very poorly when Sumana was going through "Hollywood Squares contestants" and "People from Cleveland". OTOH Sumana did well identifying the honorary Globetrotters (not an official Wikipedia category at the moment, though if you put the Futurama characters in "fictional honorary Globetrotters" you'd have one of the best Wikipedia category names ever). Sumana reminds me that I did well with "Fictional robots with emotion" and "Characters played by a member of the opposite sex". At the intersection of those two lies Gypsy.

(1) Sun Dec 10 2006 23:47: We went up to New Haven today to hang out with the Minutillos. We had lots of fun! They showed us around, we talked and played games on the Wii, dinner at vegetarian restaurant, etc. Recommended. If you know the Minutillos.

(2) Sun Dec 10 2006 23:54: Beautiful Soup Works For You.

Tue Dec 12 2006 00:44 WD-50: Rachel Chalmers was in town! We celebrated by dining at WD-50, New York's ritziest restaurant/lubricant. There were good flavor combinations and textures, especially the dessert tasting menu (peanut + guava + shortbread, chocolate + lime + licorice + avocado). And foams! Horseradish foam, orange blossom foam; you name it, it was a foam. The portions were small by my standards and practically broke the bank, but it was worth eating there at least once. Made me want to go to the Chinatown Ice Cream Factory afterwards and just get a big ol' cone, but I didn't.

Tue Dec 12 2006 23:48 Giant Tuesday Night of Amazing Inventions and Also There is a Game: Sumana finally got me to go to this comedy show just as it's ending (last show is next giant Tuesday). It was a lot of fun, though I was hoping for more inventions, because they're like the MST3K invention exchanges and because Mike Birch plays a great inventor; the perfect combination of Joel's paranoid naivete and the Mads' cruelty.

(1) Wed Dec 13 2006 22:16: Too bad I found this after I bought presents for everyone, because Etsy pushes a lot of buttons. People make things and sell them online through a decentralized marketplace! It plays into everyone's preconceived notions about what the Internet makes possible!

Sumana says "You're really only indie and fresh if Wikipedia deletes you for not being notable."

(2) Thu Dec 14 2006 22:36 List of Fun:

• I got my Futurismic check today! (Plus T-shirt) I'm not yet a published fiction author but I am a paid fiction author.
• I thought of a publishing scheme that combines the street performer protocol with micropayments and the tradition of pulp magazines. I put up descriptions of stories with word counts. You send in payments at magazine rates (5-6 cents a word) and I reveal the appropriate number of words. If people think a story is boring they stop paying for it and it just cuts off in the middle.
• Recently I've noticed two comics that operate under similar constraints. If only there was some way to combine them...
• I got a copy of 1984's The Whole Earth Software Catalog, ostensibly for research for a 1980s steampunk story, but really because contemporaneous reviews of early personal computer software are SOLID GOLD. This is the tradition I like to deface in my Software and Game Roundups (been a while since the last one of those, actually).
• Diane Duane linked to my Star Trek slash pairing generator. This is vicariously exciting as Diane Duane wrote the best Trek-franchise novel among the far too many I read in grade school (that'd be Spock's World, which has the truest-to-life depictions I've seen of everyday life in the Federation).

(1) Fri Dec 15 2006 19:04 Speaking in tongues: I was walking around today and passed a guy who was talking in a strange language. But it was too regular to be any human language. But then it wasn't a special language at all; he was just saying the same thing over and over again, in English. "Rolex rolex watch [unintelligible; presumably a price] rolex rolex rolex watch [price] rolex watch rolex".

He spoke to no one in particular. Most likely he was hawking fake or otherwise black-market Rolexen, getting plausible deniability from the crowd and his deadpan delivery. But I like to think of him as one of the new breed of mad monks, who take their mantras from the Heraclitan river of spam that flows through our lives. Strong buy strong buy one to watch about to explode strong buy...

Fri Dec 15 2006 21:24: Crazy things: CryoPID turns a running process into a disk file.

(2) Fri Dec 15 2006 23:24 Hacker Duel: Ward Christensen versus Ward Cunningham.

Sat Dec 16 2006 16:45: Hung out with Evan today. I told him about Kevin's record collection of punk bands with great names like "Live Skull". As we spoke the radio in the restaurant was playing aggravating commercials with creepy lines like "add a little bit of sour cream, and watch 'em smile!" and "it follows your every move!" It was at this point we decided that all brand names should be replaced with "Live Skull". It works beautifully.

(2) Sat Dec 16 2006 19:37: Interesting fact from the Whole Earth Software Catalog. You've heard of special-purpose word processing machines, but there was once a special-purpose spreadsheet machine: the WorkSlate. It didn't do too well.

Mon Dec 18 2006 10:47 Clipping Service: I'm quoted in an article on stock spam in the Fredericksburg Free Lance-Star.

Tue Dec 19 2006 08:21 Weird Search Requests:

what do the different parts of the christmas cracker mean?

What does a paperclip mean?

Here's one answer, though I can't determine the relative positions of tongue and cheek:

The Christmas cracker was invented, purely by chance, by an English baker called Tom Smith. He took the simple principal of the wrapped sweet or 'bon bon' and added first a love motto then, after much experimenting, a strip of paper impregnated with a compound which would 'crack' when opened. Over the course of time he dropped the sweet, lengthened the wrapper and introduced small novelty gifts.

I guess 2% inspiration and 98% perspiration moves an invention into "purely by chance" territory?

(2) Wed Dec 20 2006 22:10 Goodbye Google SOAP Search Service: You may have heard that Google has deprecated their SOAP-based search service. This comes after Nelson Minar, who worked on that API back when he was at Google, says he'd "never choose to use SOAP and WSDL again."

Much as I dislike the way SOAP is used these days, I'm not inclined to gloat, because the deprecation just means more work for me. I used the SOAP search service as an example in all three of my books. Now I've got to find another free, public, non-obscure SOAP service to use as an example (ideas?).

The official Google narrative is that the SOAP-based web service has been replaced by something called the "Google AJAX Search API". If you take this narrative at face value it means that Google has taken down their web service and put up an AJAX library in its place. What's AJAX? Who knows, it's AJAX. Here's a typical weblog entry on the topic.

Another victory for REST over WS-*? Nope -- Google doesn't have a REST API to replace it. Instead, something much more important is happening, and it could be that REST, WS-*, and the whole of open web data and mash-ups all end up on the losing side.

It's probably my recent proximity to Sam that's doing it, but I'm noticing a tendency in myself to draw fine distinctions. There's a sense in which this narrative is right and a sense in which it's not. I'm going to pick apart the narrative and show what exactly is disturbing about the Google AJAX Search API.

It's not true to say that "Google doesn't have a REST API to replace it." In fact, Google has two REST APIs, and one of them predates even the SOAP API. You've probably used this old API: its primary endpoint is http://www.google.com/.

Yes, the Google website is in fact a very RESTful web service. The downside of this web service is that it's a little bit difficult to use automatically, as opposed to through a web browser. It serves data in a human-oriented format (HTML), and you have to screen-scrape it into a data structure if you want to do anything for it.

There are libraries for doing this, but the other problem is Google doesn't want you to do it. It violates Google's Terms of Service ("No Automated Querying"). Lots of inconsiderate people write scripts that hammer Google's REST API day and night. Google tries to prevent this by sniffing out anything that might not be a web browser and preventing it from accessing the API. (To see this, set your browser's User-Agent to "libwww-perl" and try to use Google.)

But it can't be denied that people outside of Google have a powerful hankering for Google's dataset, so eventually someone (Nelson Minar, it seems) came up with a second web service API that was designed just for use by automated clients. The catch was that you had to sign up for a unique key to use it, and that key would only work for you 500 (later 1000) times a day.

In point of fact Nelson chose a SOAP/WSDL architecture for this web service. But there was no need to use any different architecture at all. Here's a possible different way of implementing the constraints above:

When you make an HTTP request to google.com, we try to figure out whether you're a web browser or an automated client. Ordinarily, if you're an automated client, we shut you out. But here's the deal. Now you can sign up for an "automated client key". When you make an HTTP request to google.com, stick your key into the Authorization header. Not only will we not shut you out, we'll try to make things easy for you. Instead of a human-oriented HTML document, we'll send you the appropriate data in an easy-to-parse XML format. But, we'll only do this for you 500 (later 1000) times a day. Then it's back to shutting you out.

This technique has a number of subtle benefits which I could bore you with for quite a while. But its obvious benefit is that it's got the exact same "API" as the Google website, which everyone knows how to use.

Anyway, instead of going down a route like this (which would, I think, have changed the history of web services quite a bit), Google went down the SOAP/WSDL route. Now they're deprecating the SOAP service in favor of some mysterious "AJAX API". This brings me to the second of Google's REST APIs.

There is no magical thing called an "Ajax request". An Ajax client makes normal HTTP requests, and processes the results automatically, just like a web service client. An Ajax client is a web service client.

What HTTP requests is the Google Ajax client making? I poked around a little bit and it looks like it mainly makes GET requests to URIs that look something like http://www.google.com/uds/GwebSearch?callback=GwebSearch.Raw&context=0&lstkp=0&v=1.0&key=xxxxxxxxxx&term=web+services. That's not exactly http://www.google.com/search?q=web+services, but it's not too far off either.

The Google AJAX API consists of a browser-side Javascript library and a server-side web service. The one acts as a client for the other. From what little I've seen of the web service I'd consider it quite RESTful. In fact, it's architecturally very similar to Yahoo!'s RESTful search API. They both use the same (IMO, fairly unsafe) trick to get a web browser to execute dynamically-generated Javascript code from another domain.

The main difference is that Yahoo's search API can also be made to send data (in JSON or ad-hoc XML format) instead of executable Javascript. That makes it possible for the service to be consumed by automated clients, not just by web browsers running client-side Ajax programs.

Let me just see if I can do something similar with the Google web service. The Javascript it serves is extremely close to also being a JSON document; I should be able to hack it a little and parse it as JSON.

Here's some Ruby code that gives you kind of a command-line Google search like people used to write for the old SOAP API. It requires the json gem.

You can skip the code.

---

#!/usr/bin/ruby
require 'rubygems'
require 'uri'
require 'open-uri'
require 'json'

KEYS = %w{GsearchResultClass unescapedUrl url visibleUrl cacheUrl
          title titleNoFormatting content results adResults
          content1 content2 impressionUrl}

def search(key, term)
  uri = "http://www.google.com/uds/GwebSearch" + 
    "?callback=GwebSearch.Raw&context=0&lstkp=0&v=1.0" + 
    "&key=#{key}&q=#{URI.escape(term)}"
  javascript = open(uri).read
  
  # Hack quotes around the hash keys to make the Javascript string
  # into JSON.
  KEYS.each do |key|
    find = Regexp.compile("\s*#{key}\s*:")
    json.gsub!(find, " \"#{key}\" : ")
  end 

  parsed = JSON.parse(json)
  return parsed["results"], parsed["adResults"]
end

# Command-line interface begins here

(puts "Usage: #{$0} [API key] [search term]"; exit) unless ARGV.size == 2
key, term = ARGV

results, ads = search(key, term)
puts "#{results.size} results for '#{term}':"
results.each do |result|
  puts result['titleNoFormatting']
  puts " #{result['url']}"
  puts " #{result['content'][0..70]}" unless result['content'].empty?
  puts 
end

unless ads.empty?
  puts "Look at some ads while you're at it:"
  puts '------------------------------------'
  ads.each do |ad|
      puts ad['title']
      puts ad['visibleUrl']
      puts " #{ad['content1']}"
      puts " #{ad['content2']}"
      puts
  end
end

---

Now, in old episodes of MacGyver, whenever MacGyver built a bomb out of baking soda and masking tape, the writers would change some crucial detail (like change the masking tape to Scotch tape) so that if kids copied MacGyver they wouldn't blow up the house. I've done something similar here. I've removed a crucial line of code from that program, so that people don't just go copying it and running it all over the place.

Why did I do that? Because when it works, that program violates the Google AJAX Search API Terms of Service. "The API is limited to allowing You to host and display Google Search Results on your site." I can use the old SOAP API to write a command-line search tool, but I can't use the new, RESTful API in that kind of application. My users can only access the RESTful API through a specific library (Google's Javascript library), running in a specific way (in their web browsers), for a specific purpose (displaying search results).

Wait a minute... running only in a web browser? Terms of Service? Bootleg scripts that hack the output into something a parser can understand? This REST web service is made available on exactly the same programming-unfriendly terms as the Google website "REST web service"!

Instead of screen-scraping a web page, I'm now screen-scraping a web service. I'm reverse-engineering undocumented URI formats, just like I do when I screen-scrape. So far, there's nothing on Google's end that sniffs my user-agent to make sure the web service only runs in a browser, but you can bet there will be as soon as that becomes a problem for Google.

The "blow to web services" narrative is incorrect. Google did in fact deprecate their SOAP API and expose a RESTful API. A win for REST!

Though incorrect, the "blow to web services" narrative is also correct. Google deprecated their SOAP API, exposed a RESTful API, and then erected a bunch of technological and legalese barriers around any attempt to actually use the RESTful API. You're only allowed to use it through one library in one language in one environment for one purpose. A loss for everyone!

On the level of technological choices, this move is a big improvement. They've gone from SOAP, which has a lot of overhead, to plain old HTTP, which has strictly less. Gone from an RPC style, which doesn't play well with the web, to a RESTful style, which does. This makes an enormous amount of technological sense. From its first day on the web, Google has exposed its dataset through a RESTful interface that gets orders of magnitude more traffic than any "web service" it might expose. In a sense, all they're doing now is unifying the architectures.

When it comes to getting information into the hands of people who can use it, Google has taken a big step backwards. The SOAP interface was serious overkill, but what you did with it was your business (though you could only do it 1000 times a day). The new RESTful interface is a technical improvement, but it's encumbered with restrictions that make it a museum piece. Unless you're writing an Ajax application using Google's library, its true value can only be obtained illicitly. And that's the other reason why I'm not inclined to gloat.

(4) Wed Dec 20 2006 22:43 Entities: I got a Christmas present shipped to me. The FedEx sticker on the front says it's from "Fran&amp;#146;s Chocolates".

(1) Wed Dec 20 2006 22:54: I talked to Riana today and she told me about Knuth 3:16. Hilarity!

Thu Dec 21 2006 13:41 rest-open-uri: My open-uri hack isn't going into Ruby anytime soon, so rather than have to devote a bunch of space in the book to hacking or reimplementing, I packaged it as the rest-open-uri gem. You can use it wherever you'd use open-uri, and it supports entity-bodies and all the HTTP methods. It's the only Ruby HTTP client library I need! And I think it should work even in programs that also use open-uri (so long as you require it after open-uri).

Thu Dec 21 2006 16:49 QOTD:

Leonard: I'll do it, but I'll be using scare quotes the whole time! "Scare quotes"!
Sumana: Yes, Leonard, scare quotes. That's what keeps you from being bourgeois.

I laughed and laughed.

Thu Dec 21 2006 19:00 Giant Tuesday: Last night we went to the last episode of Giant Tuesday and it was great. Highlights included the sentient George Foreman grill that talked like George Foreman, and the best-executed fart joke I've ever seen. Too bad it's now GONE FOREVER.

(1) Fri Dec 22 2006 08:02 Artificial Beach: I'd known for a long time about the artificial beach theme park in Japan. I've been to an artificial beach myself (though not such an opulent one), so I didn't think there was anything especially odd about it until immediately after waking up this morning, when I realized that Japan is an island chain and they've already got more beach than they know what to do with. It's like having a Vegas casino where the theme is "desert and gila monsters".

Indeed, according to this random weblog entry the artificial beach is right next to a real beach. I guess that makes sense logistically but it takes us into the territory of a Vegas casino whose theme is "ripping off the other casinos on the Strip". Which undermines my point, because that would be the most awesome casino ever.

Sat Dec 23 2006 22:48: Not much to report. Nandini came over for Christmas, which is fun. We watched Brick again.

I finally got my sample web service written for the book, though it's still missing one feature. It's about 16K of code, which I don't know how many pages that translates into.

(1) Sun Dec 24 2006 09:38: "Sigmund Freud, scribbling in the pages of a Swiss hotel register, appears to have left the answer to a question that has titillated scholars for much of the last century: Did he have an affair with his wife’s younger sister, Minna Bernays?"

Clearly this is why my mother always signed registries as "Sigmund Freud".

Mon Dec 25 2006 23:04: We had present-opening fun and then went to Ben's house for a long party. I got a blowtorch! (Part of a creme brulee kit from Susanna and John) Happy Christmas to all, and to all a good night.

Tue Dec 26 2006 21:42 Awesome APOD: "The gegenschein is sunlight back-scattered off small interplanetary dust particles."

Thu Dec 28 2006 07:16 DUN DUN: Law and Order: Sesame Street

Fri Dec 29 2006 18:09 Egg Thing: It's difficult to search for casserole recipes on the web because most of the recipes are full of mushroom soup. So I went with my gut and did the following egg bake thing for dinner last night. I guess I could have checked the Julia Child, but my gut was closer.

• 2 chopped up zucchini
• 2 chopped up leeks
• 1 cup milk
• 4 eggs
• shredded hard cheese
• bread crumbs

Preheat oven to 375 degrees. Saute the leeks. Beat together milk, eggs, and cheese. Stack the zucchini in a 9-inch baking pan, mixed with the leeks. Pour the milk mixture on top and put bread crumbs on top of that. Bake for 50 minutes. It turned out great!

(1) Fri Dec 29 2006 19:08: For a while I was trying out AdSense ads on this site. They never brought in more than a paltry sum, and I stopped doing them yesterday, rather abruptly, when Google cancelled my AdSense account.

The web is full of stories of these arbitrary cutoffs, which I can now confirm in their relevant details. You get an email saying "we've cancelled your account based on classified information" and that's it. You get a pro forma chance to defend yourself, but it's not much good because you don't know what happened. It also doesn't make much difference, because the response always comes back: "we re-ran our classified algorithm on our classified dataset and it gave the same result as before, buh-bye".

Fortunately, I have a friend in the online ad business. Ryan's Project Wonderful mostly serves ads for webcomics right now, which is fine by me, as it means fewer "KEYWORD? Buy it now at creepysite.com" ads.

Fri Dec 29 2006 22:00: Some minor bugfixes to wadl.rb. Still no official release, still not much in the way of syntactic sugar.

Sat Dec 30 2006 12:11 Some books about computer history: I mentioned a while back that I was reading Steven Levy's book about the development of the Macintosh. I guess that triggered something in my to clear out my stock of books about computer history, because then I read The Dream Machine, a book about ARPA by M. Mitchel Waldrop (almost as ponderous a name as J. C. R. Licklider), and Andrew Hodges's famous Alan Turing: The Enigma. And there was also the matter of the intermittently fascinating Whole Earth Software Catalog.

I bought the Waldrop book as a biography of Licklider (actually I bought it because of the BRIGHT YELLOW COVER that blocked out all other books in the bookstore), but it really strains to fill a whole book with his life and make it interesting. Fortunately the book's method of straining is to bring in all the other characters of American computer science from the 1940s through the 1970s, with a theme of Licklider as the networker between them, and that was very entertaining. Though I admit part of the entertainingness was the celebrity-spotting equivalent of those old Looney Tunes cartoons that feature charicatures of all the Warner Bros. contract stars of the 1940s. Oh look, it's {Peter Lorre,John McCarthy}.

The most interesting new information I found in the book is the history of ARPA, specifically of the Information Processing Techniques Office (Licklider was IPTO head twice). The major theme is the Vietnam-era mission anticreep that slowly pushed ARPA from advanced research projects, to the more circumscribed realm of advanced research projects that we can use on the battlefield within two years without fundamentally changing anything (somewhat parallel to the problems Xerox PARC had). Meanwhile Licklider's trying to scrape together a couple million to fund his crazy "Intergalactic Network" project, and his office is sticking boilerplate defense justifications onto incoming grant proposals so they can get ARPA money.

"All that language about military rationale wasn't in the Stanford version of the proposals," explains Ed Geigenbaum: it was slapped on at the very end by the ARPA funding officers back in Washington. "The only people who ever saw it were the students who would later dig it up under the Freedom of Information Act. Then they'd bring it on campus and say, 'See, McCarthy is working on such and such.' McCarthy would say, 'What do you mean? I never heard of that!'"

But the book also ties together things that I learned about in college with no that they were connected. For instance, ALOHA and Ethernet use the same collision resolution mechanism not because it's an obvious fact about the universe, but because Bob Metcalfe read a paper slamming ALOHA and decided to prove it could work. And multitasking came directly from time-sharing, as cheaper computers made time-sharing less of a selling point.

Fun fact: the book made it sound like ARPA was originally planned to encompass space research, and that NASA was created as a separate agency after a bureaucratic turf war. So the space program could have been part of ARPA.

Understanding the book requires no technical knowledge, but I don't think I would have enjoyed it as much if I didn't already have all these pieces in my mind ready to be connected. The book made me more interested in another item on my wishlist, John Markoff's What the Dormouse Said.

This entry got kind of long, so I'll talk about the Turing book in another entry.

(1) Sat Dec 30 2006 16:11 The Enigma: This was a weird book to read because I'd already internalized much of it. It's the standard narrative of Turing's life I grew up with, and if you read it for the details you can actually see Neal Stephenson coming up with the plot for Cryptonomicon. So unlike the Licklider book I don't have a whole lot to say about this one, except that it was definitely worth reading.

One thing that jumped out of me was, reading the WWII sections, how much of the codebreaking relied not on abstract mathematical ingenuity but in ingenuity applied to exploiting the adversary's mistakes. For instance, the Germans often used a less secure version of Enigma for unclassified things like the weather report, not realizing that if Bletchley cracked the weather report they'd be 80% done cracking the secure cipher for that day. Not surprising that this got largely left out of Cryptonomicon, because fiction is usually more interesting when the adversaries are competent.

Sat Dec 30 2006 16:35 Nethack Interlude: Despite my earlier promises to myself that I'd stop playing Nethack after ascending, I got hooked earlier this month on the alt.org Nethack server, which yields up many interesting bones files. I started trying to ascend a tourist (to my mind the lamest Nethack class), and this morning I succeeded, achieving #1490 on the alt.org high score board. Since the various conducts have never really appealed to me, hopefully this is the end of Nethack addiction.

(1) Sat Dec 30 2006 22:57 The Anthropic Principle of Open Systems: There's a common theme in the Licklider book and the Tim Berners-Lee book, the theme of contigency. At every stage in the development of the Internet, there was an overwhelming chance that the project would fail. There were interests vested against openness, and then strong competitors that didn't share the philosophy of openness. But the open solution won. The same thing happened, in miniature, for the Web on top of the Internet. I mentioned this in passing last year.

This creates a problem analagous to the problem of the anthropic principle in physics. It's not a perfect analogy because the development of the Internet took place within space-time, but I think you can see it. Why did all these unlikely contigencies happen?

I can think of three possible responses. The first is that the contigencies weren't unlikely at all. There is some hidden force of society that makes sure open solutions tend to win. This is the knee-jerk free software response. I like it okay, but I've grown more dissatisfied with it as I've read these books, because there are lots of other situations in the world where the open solutions lost big time. When does the hidden force work?

The second response is that we gravitate to whatever open solutions we can find to solve our problems. The general public can't use something unless it's open. If things had turned out differently we might be speculating on how unlikely it was that we would develop a tradition of collaborative biology or open letter-writing.

The third is to deny that there's anything special about the contigencies. We don't talk about things that don't happen, so if these contigencies hadn't happened, we wouldn't be talking about them. Saying that the Internet and the Web worked out the way they did is just a tautology. This response has its points when applied to the physics anthropic principle, but I find it unsatisfying in this case, because it looks like the same sorts of contigencies turned out the same way twice.

Anyway, just some high-concept rambling to see out the year.

(4) Sun Dec 31 2006 21:50: Today I walked around Manhattan with Evan. I mentioned my desire to own a nice topcoat and Evan took me to basically every clothing store in crowded SoHo. Finally we found a topcoat in the basement of J. Crew. I guess they haven't been selling well lately what with the oppressive non-cold.

I sustained an actual enthusiasm for clothes shopping for a record two stores, but by this time I was willing to settle for just trying on a topcoat and seeing how I looked in it. This resolve was strengthened by the $200 price tag on the topcoat.

The trying-on was a disaster. I'd thought I could just wear a topcoat on top of whatever and it would be warm and make me look classy, but I looked like a slob who'd stolen a topcoat from somebody. I was like the woman in Cryptonomicon who starts wearing stockings and then discovers that stockings dictate your entire wardrobe and lifestyle.

The topcoat works on Evan because Evan already has a compatible wardrobe, but it's totally out of my demographic. This was a fairly distressing realization. The coat would not transfer its class to me merely through a mercenary monetary transaction. This is the difference between class and mere wealth, and I don't even have the consolation that I became extremely wealthy and then discovered it.

(1) Sun Dec 31 2006 23:31: Apart from my crushing last-minute defeat by a topcoat, 2006 was an okay year for me. I got married and got used to New York. I got a book published, and sold a book proposal and a SF story. I saw a lot of friends and had a lot of ideas and played a lot of games.

I'm glad I got to spend a lot of time with my mother before she died, and I'm glad we got to see Susanna and John for Thanksgiving. My main 2006 regret is along the same lines: I wish we could have all been together for Christmas; I think we should start making plans for next year and get it set up as early as possible. Someone said that with Mom dead we would need to put in work to see each other, because we'd no longer have a natural gathering place, and I'm starting to see the truth in that.

My current plans for next year are writing-related: the REST book and lots more SF. Let me know if you want to read the noir space opera I'm trying to sell.