<M <Y
Y> M>

[Comments] (1) Recursive Love: I got one of those "I ♡ NY" plastic bags when I bought some cheese. But inside the heart was the word "Love", for those who speak English but don't know the connotation of a heart. But the word "Love" was itself spelled with a heart: "L♡VE". So why stop there?

: Went with Sumana and Adam to a party last night which was way too loud. But we did hear a story about a theme restaurant in Manhattan called Ninja New York. It sounded like a very expensive Chuck E Cheese for adults. I say "adults" only because it serves this weird Asian fusion food that I suspect kids would hate. The New York Times hated it.

Confusing the point of a restaurant with the mission of a "Saturday Night Live" skit...

In the name of "new style sushi" Ninja employs rice cakes as beds - or sometimes graves - for a rectangle of truffle-flecked omelet (it tasted like soggy French toast), a sliver of sautéed foie gras (pleasant, but how could it not be?) and a finger of seaweed-crowned mackerel (fishy in the extreme).

Adam thought this was orientalism at its finest, but this restaurant is an offshoot of one in Tokyo, so there's clearly some self-orientalism going on as well.

The only cool part: the restaurant is camoflauged. It looks like an apartment building and you go down in the elevator and the door opens to reveal a ninja standing in front of you. They should cut the dining experience short right there and call it "Be Scared by a Ninja For $5".

Blurb Brag: Oh, I don't think I mentioned that the Ruby Cookbook has back-cover blurbs from why and Matz! You can see them both at the Ruby Cookbook Official Unofficial Homepage.

[Comments] (1) : Pictures from my trip to Worcester. Includes photos of the game boards I "modded" for Jake's store. Includes picture of me and Jake together, which will not convince Kevin that Jake is a real person.

: Check out these musical notations. Cornelius Cardew's notation looks like the Ferengi language.

And what did I see?: why has devised hpricot, a permissive HTML parser with tree-crawling functions. It's written in C, so it's fast. Just another great piece of software named after apricots.

[Comments] (2) : Today I realized that we've lived in New York for six months and I haven't put up any pictures of our neighborhood in Astoria. Then I realized that this was because I didn't have a camera. But now I do, though it's falling apart and doesn't take pictures half the time. So yesterday when I was walking around I took some pictures. Since it was the Fourth I paid special attention to flags. Here are the pictures. I tried to get one of the statue of Athena, but that was one of the pictures the camera didn't really take.

[Comments] (1) Early Birthday: This is my first birthday, I think, where I've really become obsessed with the mortality symbolism of the birthday. So bah to that! Let's concentrate on PRESENTS. We hung out with Andy today and he surprised me with a Not For Tourists guide to Queens (not that Queens is really a big tourist destination anyway). And Sumana presented me with an awesome print of The Death of Jennifer Sisko! Susanna sent me something I don't know what. Yes, physical objects will keep my mind off the eventual termination of my physical existence. Wait, how is that supposed to work?

: Here's a great lecture series on the Byzantine Empire.

: I improvised a song for Sumana today, which was pretty good (cf.), so I recorded it. It's called I am a Cowboy.

: How can we stimulate production of awesome deep-sea photography? Hold a contest! Pictures must be of the high end of the ocean depth gradient, which as is well known is correlated with awesomeness.

[Comments] (2) : Common IF author mistakes. Alert: alotta alliteration allocated afore all asterisked areas.

[Comments] (1) : I wasn't planning on doing this until later, but I made myself a birthday cake today.

The Answer Is Yes: "I pity the fool who has to ask me if I still pity the fool." From Joe Mahoney.

[Comments] (1) A Game Divided Cannot Roundup:

[Comments] (3) Systemantics: The working epigram for my current project comes from a book called Systemantics:

A complex system that works is invariably found to have evolved from a simple system that worked.

(The first sentence of Gall's law)

I decided to buy this book and read it to make sure I wasn't quoting a book that said crackpot things about freemasonry or something. Of course, now that I've bought it I'm able to search on a key phrase and find the whole first edition online.

I had my hopes for this book. In my mind it was a romantic relic of the age of cybernetics. It would have feedback diagrams and equations vaguely remembered from CS112. It would have lots of other quotable bits backed up by distilled essence of case study. It would be the Mythical Man-Month or the Peopleware of systems design.

The truth is more prosaic: it's a pop-psychology book. The bibliography includes pop-psych classics I'm OK, You're OK and Games People Play. It mentions General Semantics on the third page. It talks about Murphy and Parkinson and Peter--pop psychologists all--as though they were Weiner and Taylor. There are no data and not even many anecdotes, just thought experiments.

Plus it's got dumb Watergate jokes. Hey, that's real funny to you guys in 1975 but I'm reading your book in the twenty-first century. You savvy my future-speak? To us, that happened decades ago, and we have higher standards for jokes about it. (Sample Watergate joke that's still funny. Non-broken image link for same.) Nothing wrong with trying to make a book funny, but the Poignant Guide to Ruby is pretty funny, and it's also got code you can run that works.

Okay, so it's a bunch of thought experiments. That's not too bad if you run the experiments and they jive with your experience, which is true here for the most part. The book has some more good quotes ("Systems work best when designed to run downhill") as well as good ideas I'd never thought of ("the confusion of Input and Output"). I'll probably pick up the revised edition of the book, The Systems Bible, which will surely have fewer Watergate jokes. But the pop psychology emphasis really decreases the authority of a quote from this book.

I could say "A complex system that works is invariably found to have evolved from a simple system that worked." myself, and it would have the same force as quoting John Gall in this book. I've probably got more hard evidence than the book does, because I live in the future. I quote him the way you quote Mark Twain: because you like the way he put it. So maybe I should just find a quote about systems design from Mark Twain.

[Comments] (1) Loaf Is Weird Food: Exhibit A: Jennifer McCann's Vegan Loaf Web Application at Vegan Lunch Box, one of what I hope will eventually be one of many applications that let you explore a parameterized space of related dishes. In this case, dishes made out of separately-edible items all mashed together with binder.

As with anything, if you put it on the web I'll randomize it. Thus, my latest feature, Mystery Loaf. It names and creates a new dish every five minutes. Eating them is your problem.

wadl.rb: If you like REST web services I have something for you. A while ago Marc Hadley came up with a service description language called WADL. I now have a primitive proof-of-concept Ruby WADL parser/client that can read a WADL file and call simple web services like the Yahoo News search. (I emboldened that link because there are so many other links in this entry, and that's the one this entry is about.)

Right now every REST web service is slightly different. You access them all with simple tools: your language's HTTP library, and probably its XML parser. Which is great because it keeps the services simple. What's not so great is that you have to learn each REST service separately.

What's more, it's useful to package and distribute your implementation of a particular service in a particular language. There's PyAmazon and Ruby/Amazon and a4j, PySearch and Yahoo-Ruby. That's always seemed wrong to me.

With a SOAP/WSDL service you just load up the WSDL file and call or generate the native language methods. Even though the underlying technology has more layers and is much more complex, when WSDL is one of those layers (and nothing goes wrong), everything just works. Sometimes there are language-specific wrappers on top of a SOAP/WSDL service (PyGoogle and Ruby/Google), but they don't do much: they make your code look nicer, or compensate for the shortcomings of the underlying API.

WADL is the best way I've found to get that ease of use for REST services, without creating documents that don't make sense to humans, or selling out the resource-oriented architecture of the web. WADL is very human-readable and it describes resources, dammit. I like it a lot and I think it'll make things easier for programmers if it gets widely adopted.

[Comments] (3) IF Score System Design, Plus, a Writer's Plea: The above-any-adequate-alliteration-allowance 'First-Timer Foibles' guide to writing IF got passed around a lot among my friends, so I want to talk about that a little bit. This is mostly based on email conversation with Adam Parrish, who I just realized has the awesome job of studying interesting things like IF. I knew about his job, but not that when we talk about this over email I'm slacking off and he's working. Actually now I might be just making stuff up.

First a note about pages like that, which as a good IF writer but a middling static fiction writer, I find kind of frustrating. The page is oriented towards beginning writers. Like most web pages that allege to help you be a writer, it's heavy on "don't misspell words" and less heavy on "don't have a hackneyed plot" and "don't create puzzles that make no freaking sense" and things I don't know or can't articulate. Such pages turn bad writers into readable bad writers, but won't get anyone's work up to really good quality. As I've found out the hard way, good ideas and a good grasp of English don't automatically translate into readable stories. There are additional skills.

These pages chop the head off of a Sturgeon's Power Law that says the vast majority of bad writing is bad for obvious reasons (scroll down to the handy "context of rejection"). With static fiction I can routinely hit the midway point on this particular ring-the-bell carnival game, but I haven't found many good resources for getting higher. I have found one extremely useful page, but the higher-level craft seems to be something you have to learn one-on-one with someone who already knows, or something subjective you learn with practice. Or something that no one has written about because existing documents are enough to get the unintersting people out of your hair.

The problems in the first-timer list are divided into problems of fiction and problems of game design. I'm not going to discuss the problems of fiction because I don't find them very interesting. I can, however, tackle the problems of game design because much less work has been done exploring elementary problems in game design. Because this entry is already huge I'm going to cover one item at a time, over a period of several centuries.

Item 1: bad point systems. There are actually two possible problems here. The first is a poorly-scaled point system where you get 100 points for finding a key. The second is an overgenerous point system where you get 5 points for getting out of bed.

If a piece of IF has a scoring system, it imposes a limit on the score. I don't know of any counterexamples. In general, your score goes up when you do a one-time action that progresses you toward the conclusion of the plot. Your score at any given time is a measure of your progress (Zork III is a notable, and obnoxious, exception).

But if a video game has a scoring system, it imposes no limit on the score (except any imposed by the hardware). Again, I don't know of any counterexamples, though I can conceive of games where your score is represented as a percentage. The score of a video game has nothing to do with the game per se: your advancement towards the end of the game is measured by things like stage numbers. It's just a way of keeping... score. I think this problem is caused by treating a piece of IF like a video game.

It's fine to give the player 100 points for finding a key in a video game, but ridiculous to do the same thing in a piece of IF. This is because people don't handle big numbers well. If your maximum score in an IF game is a big number, it's difficult to tell how close you are to the maximum, and how much 100 points really contributes towards the maximum. A big inscrutable number is cool when you're claiming to be an awesome dude, but it's not as cool when everyone who completes the game gets approximately that same score.

By the same token, you shouldn't give out points for actions that don't advance the plot or don't involve any cleverness. If you do, the score will cease to have meaning as a way of measuring the state of the plot and your cleverness so far.

Trivia: A while ago I discovered a Victorian book of how-to miscellany called Enquire Within Upon Everything. It turns out that Tim Berners-Lee came across that book when he was a kid, and named his first hypertext system (1980) ENQUIRE after it. Amazing trivia!

Shot in the heart: And you're to svn blame.

Monde mondial: From a French message board:

BeautifulSoup, c'est l'un des meilleurs parseurs HTML du monde mondial, c'est ce que j'utilise pour mon script de conversion blueflagz => RSS

"Monde mondial"? Does that mean "real-world"? If so, that's a great idiom.

I don't know when I'm going to be fixing Rubyful Soup to have the features of Beautiful Soup 3.0, but it's probably going to be after a bunch of WADL work.

: One good thing about being President of the United States: the food is great!

When he entered, President Bush walked within two feet of me, and for some reason, he turned and looked me in the eye and said, "Let's eat!".

Ned Batchelder and his wife's White House adventure: parts 1 and 2.

[Comments] (5) Maple Syrup Vengeance: In California you can go to a pretty nice breakfast place and they'll serve you real maple syrup in a glass with a spout, or a little syrup ramekin. If you go to a cheaper place they might have fake maple syrup but it'll be in a glass with a spout so you can pretend it's real. Even IHOP does this.

In New York, right next to the place where the maple syrup comes from, they don't do any of this. I've been to fancy breakfast restaurants, as fancy as any I ever went to in California, and they all treat maple syrup like cheap jelly. It's made by Smuckers out of corn syrup and it comes in little plastic tubs with pull-off tops, like non-dairy creamer. They have to stock this stuff because all the restaurants here do delivery, and how are you going to deliver maple syrup in a little syrup ramekin. But they take it a step further and also give it to the people who came over to their restaurant in person! So when I go out to breakfast I never order anything with a dependency on maple syrup. I guess I could be a huge snob and bring my own little container of maple syrup, but then why don't I stay home and make my own waffles?

I've eaten at places in the south (IHOP being the exception), where the restaurant just buys a 20-pack of Log Cabin squeeze bottles and plunks one down on each table. New Yorkers probably think their system is better, but I don't see how. I never thought I'd have anything good to say about IHOP's syrup, but at least it doesn't look obviously fake.

Incidentally, Rocks'n'Diamonds is turning into ZZT.

[Comments] (2) I Am A: Mike Popovic heard my earlier song I Am A Cowboy and filked it for his daughter Zoe. He turned it into I Am A Pirate, which I dutifully recorded. Both songs are now part of a concept album called I Am A, which explores the spectrum of authenticity and total fakiness.

Where will the trail of improvisation lead next? Ninjas? Robots? Some other nerdy role-play?

[Comments] (1) Experiments: Today I did a thing Sumana has long suggested I do, and took my writing on the road. Down to the cafe about two blocks away. I had lemonade and a sandwich, and wrote book stuff on Sumana's laptop.

I'm not a huge fan of doing that kind of thing; I managed to stay about an hour and a half before it got to me and I had to leave. Did it help my writing? Maybe, but not $13 worth, which is what I spent on food. I'm going to try the cafe three blocks away tomorrow, but I think that mid-day work-trips to cafes will just be occasional treats to eat a lunch I didn't make.

Speaking of which, I went early this morning to the farmer's market in Union Square, which has really picked up since the dead of winter when I first went. Now it's full of cheap carrots and radishes and stone fruits, reasonably priced tomatoes, slightly expensive berries, and expensive garlic. Someone said "These tomatoes are severely deformed! Are you going to charge me extra for this?". I also saw a woman in chef garb pushing a metal cart around with some food on it--could this be one of the "celebrity chefs" who frequent the Union Square market, buying "celebrity food" for their "celebrity restaurants"? Who cares, really? Remind me to tell you about restaurant food sometime.

Usually when I go into Manhattan I try to wait until at least 10, because otherwise you're crammed into a subway car with a bunch of commuters and it feels like a demonstration of Avogadro's number. But today I went along with Sumana so I could buy stuff at the farmer's market and get home in time to make gazpacho and give it lots of time to soak. Sumana invited some people from Fog Creek over for dinner and the theme was to be "cold".

So I made gazpacho, and a tasty couscous/tofu/cucumber/tomato dish, and sliced deformed tomatoes with salt (not really cold, but you have to eat that at every opportunity while the farmer's market tomatoes are around). For dessert: vanilla ice cream with rasperries. Theme: embodied!

On an absolute timescale, dinner preparations kept me busy enough that I didn't write as much as I'd have liked. But I did discover some interesting facts which will go in the book.

Couscous: By popular demand the couscous recipe.

Okay. Drain the tofu, slice it into rectangles, fry it in peanut oil, slice it into strips.

Boil the olive oil and broth. Pour it on the couscous and fluff with a fork. Dump in all the other ingredients and mix it up some more.

Tastes like a particulate Cobb salad or something.

I've Got Radioactive Blood: Went to the dentist today, where there was a Spider-Man poster in the X-ray room. The dentist is oriented towards kids, but is that the best place for Spider-Man?

Relatedly, my cousin Jill came to visit with her friend today. Said friend recounted a story of overhearing one of her sons saying to the other, "No, let it bite you, and you'll get superpowers like Spider-Man!" Friend put a stop to that right quick.

Can't think of a punchline, so I'll delegate to Penny Arcade. It's one of those ones where the punchline is in the first panel.

[Comments] (1) Only to Find Gutenberg's Bible: Yesterday we planned to go to the Transit Museum in Brooklyn. Instead we went to the research library (the one with the lions) and saw the handwritten Declaration of Independence. It's riddled with apostrophe errors! ("laying it's foundation on such principles") What would my mother think? Incidentally, that link does not have the exact version we saw, but it's close.

Among the topics I need to write about is my antipathy towards libraries that don't let you browse the stacks. But the really huge libraries with closed stacks also tend to be the ones that have things on display like a freaking Gutenberg Bible. Just right there in a case in the middle of an otherwise boring room. Do people know about this? I didn't.

To overcome my closed-stack anguish we went across the street to the mid-Manhattan library and checked out some books. This was great, especially the self-service machine you use to check the books out. But on the way out there was a different form of anguish, a lengthly fracas with the guy who makes sure you don't take books out of the library without checking them out. That's the totality of this poor guy's job, so when I triggered his "unchecked books being removed from library" alarm he went right to work on me.

"You need to check those out." "I did check them out." "Sir, those are library books. You need to check them out." "I did check them out. Do you want to see the receipts?" "Sir, you need to check those books out." "What do you do to check them out besides use the checkout machine?" "You can't take books without checking them out." Yes, I KNOW HOW A LIBRARY WORKS. (I didn't say that. I only say rude things like that afterwards, in weblog entries.)

Eventually he looked at the receipts and conceded that I had probably checked out my library books, deviously disguising myself as an honest citizen and cheating him out of a juicy apprehension. Paperwork beats alarm in this rock-paper-scissors. Sumana suggests that the self-service checkout machine demagnetizes or magnetizes some RFID-like object in the book, and I'd picked my books back up before the machine had had a chance to do that.

Then we decided to go to the transit museum for real, so we took the subway. Sumana has a greedy algorithm for taking the subway through a bottleneck, but one of its implicit assumptions is that the subway graph looks like San Francisco's, with few paths through a bottleneck. Now that we've moved, the algorithm needs generalization. The end result is that, due to stereotypically unintelligible service announcements, we ended up near the Brooklyn Bridge with no simple subway path to the museum. So we just walked the bridge. Which was fun, but once we got to the museum Sumana sat down on a bench and fell asleep, leaving me to explore the museum by myself.

This entry is long, so I'll publish it and write another one.

Brooklyn Transit Museum: This is a museum in an unused subway station. The subway level has a switching tower with a live view of the nearby subway lines. In fact, the whole subway level is full of old restored subway cars. Which is awesome. You can time travel throughout the twentieth century by going from car to car.

The cars were mostly decorated with period advertising placards, which were pretty interesting, but which destroyed the time-travel metaphor in a Vonnegutian way. Instead of having ads from a specific time, each car had a collection of ads from the entirety of its run, all unstuck in time and smushed together in one car. There were WWII-era ads next to Prohibition-era ads, including one that claimed "One day all women will vote... for [brand of soap]". I don't think that ever happened. Incidentally, the soap ad was one of those ads that features terrible and irrelevant racist cartoons.

Another soap ad had a Depression-era mother saying, "My children must purify hands before eating." Well, I hope your children enjoy starving, since nothing they do will ever be good enough for you.

I walked around for a while thinking these cynical thoughts. Not helped by an ad for subway ad space I'd seen upstairs in a historical exhibit, which claimed that subway riders "have learned to believe implicitly in the advertising displayed in the New York City Surface Cars because it merits and invites confidence."

This attitude was mocked by a contemporaneous newspaper cartoon where, having nowhere else to put ads, the conductor was looping signs over the necks of the passengers. It was from the golden age of political cartoons, when there were lots of strange details but not the loopy word balloons full of cursive dialogue that's poorly blocked and illegible, and not worth reading since it just explains in great detail "Oh! The Situation depicted in this cartoon, though considered by the reader a japish Metaphor, is something that truly does affect my corporeal Form!"

The truth is that most of the subway ads did, and do, not merit or inspire confidence. The golden age of the subway ads were the 1950s, a time when advertisers had forgotten that words had connotations as well as denotations, when you had ridiculous claims like "84 out of 100 women prefer men who wear hats" (an ad for hats, or for men?) and needed gorgeous MAD magazine-style paintings to make up for them.

Trivia: the original name of the Brooklyn Dodgers was the Brooklyn Trolley Dodgers, answering my decades-old question of what the Dodgers were dodging. Also, an old sign once posted near an elevator says "Meddlers Take Notice" in a preemptive anti-Scooby-Doo move.

: Sumana's reading a Dave Barry book. Yesterday she read a bit of it to me about how Abner Doubleday "invented a game that included virtually all of the elements of modern-day baseball, including Bob Costas and the song 'Who Let the Dogs Out'. This led to the Civil War."

"Actually there is a connection between Abner Doubleday and the Civil War," I said, ever the vigilant connection-spotter and nitpicker. I looked it up in Bully for Brontosaurus and, yes, Abner Doubleday did start the Civil War. He was a Union officer who fired the first shot from Fort Sumter.

Actually his Confederate counterpart is the one who really started the war, but it's more true to say that he started the Civil War than to say he invented baseball.

Ruby Cookbook Promotion: The Ruby Cookbook is either available or almost available, depending on who you ask (Amazon sales rank right now: an amazing 7230; incidentally, authors, I have a small script that records and graphs a book's Amazon rank, since I know you're all obsessed with that). So it's time to start promoting the sucker.

I wrote an article for the O'Reilly Network (which I think is technically not part of O'Reilly, somehow) about how we tested the code in the Cookbook. Which I should nudge the editors about. Lucas lives in Portland, so he doesn't have to pay for a plane ticket to go to OSCON and talk peoples' ears off about the book.

My ill-formed plan was to do a virtual book tour: guest-post on technical weblogs about the coolest recipes from the book. Other people have done this in the past, notably Greg Knauss, but I don't think it's been done with a technical book before. And I never actually emailed anyone about doing this, so it's probably too late now. I got an invitation to join the O'Reilly Ruby group weblog, so maybe I'll just post once a week on there about one of the cool recipes.

Does anyone else have ideas for book promotion? I want to focus on my current project but I should set aside some time for selling what I've already written.

Mars Needs B-Roll: I forgot to mention that if you ever go to the Union Square famer's market on a Saturday, there'll always be at least three camera crews filming. Some of them are obviously students doing assignments, but for the rest of them, how many puff pieces about the farmer's market does one city, even a city of 8 million people, need? Is it all going to stock footage? "Man with rowdy child buys arugula (0:32)"

[Comments] (1) Need to move the earth?: I'm thinking about designing a pocket multi-tool that gives you all of the simple machines. It would have a little lever, an inclined plane, a fold-out pulley, etc. The supplementary gadget: a combination straightedge/compass!

Beloved Ruby Cookbook Recipes #1: "Getting book information with Ruby/Amazon": You know those old-time radio shows, where Doris Day always guest-starred, and Bob Hope ended up wandering onto the field at Muroc and getting shot? Well, that particular one never aired, but you know the kind of show I mean. The shows that were sponsored by random household objects like soap, or a broom, where the straight man would set up a line and the funny man would always turn it into a little ad for the sponsor.

"Lou, I think we ought to stick around a while longer." "You know, Acme Wallpaper is the only thing today's handyman needs to 'stick around' the house. It's inexpensive, durable, and when properly applied it forms a Penrose tiling! Ask your dealer to see a catalog today." Thus allowing the straight man to set up the exact same line again: "I say, Lou, that we ought to stick around a while longer and watch after Doris Day."

This is the model for this series of weblog entries, where I show you how to do cool things with Ruby but it turns out all to be a scheme to plug the Ruby Cookbook, bad boy of the O'Reilly cookbook family. In the first three entries of this series, I'll show you the system I rigged up to track the sales of my books. I don't really have a plan after that, but I can work from a list I made of the coolest recipes in the Cookbook.

The Book Sales Trilogy:

  1. Getting book information with Ruby/Amazon
  2. Generating graphs with Gruff.
  3. Generating sparklines with the sparklines gem.

The problem

When you write books but don't publish them, it's difficult to know how many you've sold. The publisher gives you a total maybe quarterly. So a lot of authors (including me) use their Amazon sales rank as a proxy for books sold.

There are numerous problems with this, stemming from two facts: 1) Amazon sales rank only covers books sold through Amazon, and 2) it only conveys how well your book is selling _compared to all other books_. Booms and crashes in the book market as a whole won't make any difference to your sales rank. But you can't beat something with nothing, and sales rank is the only way I know of to get anything like real-time sales figures for a book. I talked with Mike Loukides, my O'Reilly editor, today, and he said that giving authors access to sales information is a low-priority project. And we all know about low-priority projects.

So, whenever a book of mine goes up on Amazon (first Learning Python and now the Ruby Cookbook), I register its ISBN with a script that runs once an hour and uses Amazon Web Services to collect the sales rank numbers. My goal is to encapsulate this logic into a class which maps an ISBN to a text file full of timestamped sales ranks. It will also work with Amazon-specific ASINs: for books, the ASIN is just the ISBN.

I use the Ruby/Amazon library, which is covered in Recipe 16.1, "Searching for Books on Amazon". The recipe focuses on keyword searches: finding books or other objects to match criteria. Here, we search by ASIN.

The script is based around a SalesReport class that encapsulates data about one product's sales rank over time. Here's the first draft of SalesReport.

require 'amazon/search'
class SalesReport < Array
  attr_accessor :name, :asin
  def initialize(asin, name)
    @asin, @name = asin, name
    open(self.class.filename(@asin)).each do |line| 
      self << line.split.collect { |x| x.to_f }

  def self.filename(asin)
    asin + '.txt'

When you create a SalesReport object for a product, it opens up a file named after that product's ASIN and reads in sales data.

Where does that file come from? The update! method is in charge of creating and maintaining it. This is also where we make the actual AWS request (in bold).

  # Updates sales rank for a number of products, and yields a
  # SalesReport object for each one.
  def self.update!(aws_key, *asins)
    now = Time.now.to_i
    req = Amazon::Search::Request.new(aws_key)
    req.asin_search(asins) do |product|
      asin, name = product.asin, product.product_name

      # Update the file with the new sales rank.
      open(self.filename(asin), 'a') do |f| 
        f << now << ' ' << product.sales_rank << "\n"

      # Parse the file into an SalesReport object and yield it.
      yield self.new(asin, name)

Three things about this method's signature. First, like filename this is a class method: it's a factory for SalesReport objects rather than something you run on a particular SalesReport. For more on this, see recipe 8.18, "Creating Class and Singleton Methods". The self here is the SalesReport class itself.

Second, this method takes a variable number of ASIN arguments, for which see Recipe 8.11, "Accepting or Passing a Variable Number of Arguments".

Third, as I've mentioned in every book I've ever written, to use AWS you need to sign up for a unique string called a "developer token". This method uses that string to create an Amazon::Search::Request object, which will be our interface to Amazon Web Services.

You can reuse a Request object to do multiple searches, but we're only doing one. To save time and bandwidth, we pass all of our ASINs in a single search: "give me info on these products". The query returns a big XML document, which Ruby/Amazon parses into struct-like Ruby objects. It yields each object to a code block.

Inside the code block, I have access to a variable product which represents what Amazon knows about one particular book. I take its sales rank reading and stick it along with a timestamp into that book's file. Within the code block I also build a SalesReport object, which I yield to another code block somewhere outside this function. The code that calls SalesReport.update! gets yielded a series of SalesReport objects to do with what they will. In this case, my will is to generate cool-looking graphs.

If code blocks confuse you (they confused me when I was new to Ruby), the first few recipes of chapter 7 will help. If you're not familiar with file access, the basics (including appending to a file) are covered in Recipe 6.7, "Writing to A File".

Okay, that's all the SalesReport code for today. Here's how you'd call SalesReport.update! with a code block:

key = 'my AWS developer token whatever'
SalesReport.update!(key, "0596523696", "0764596543") do |product|
  puts "I updated #{product.name} (Rank: #{product[-1][1].to_i})"
# I updated Ruby Cookbook (Cookbooks (O'Reilly)) (Rank: 1412)
# I updated Beginning Python (Programmer to Programmer) (Rank: 219370)

So now I've got two files on Crummy's hard disk full of sales rank readings. What to do with that data? That code block doesn't really do anything., Well, I'd love to compare the sales ranks to hard sales numbers, but I've never gotten any for the Python book, and the one data point I got from Michael today doesn't mean much. So I've been comparing these numbers to their earlier selves by graphing them versus time. This leads into tomorrow's episode, Graphic Violence, featuring Chico Marx.

Oatmeal: Argh. Remind me not to start promotional tutorials until I've written all the code. I had to totally redo yesterday's entry to get a good architecture for doing big graphs and sparklines. Plus because of it, I found an erattum in the Cookbook code (in the MIDI music recipe). So I have the weird feeling where you've been busy all day but haven't accomplished anything new: you've just cleaned up your old mistakes.

On the plus side, the code is much nicer now, and it turns out errata get fixed in the next printing of the book, not the next edition as I'd always thought for some reason.

Also, you know those allergen disclaimers on food that say "may contain traces of nuts"? Today I believe I succeeded in isolating the nuts! First, a recipe:

Natur-Swiit Oatmeal

Instructions: Boil the fruit in the milks and the salt. Add oats and cook for 5 minutes on low heat.

Okay, it's just the instructions from the back of the oatmeal canister, except for the fruit. But the fruit is magic. When you boil the fruit in the milk, it swells up and sweetens the oatmeal, so you don't have to add brown sugar to the oatmeal. Adding fresh fruit afterwards never makes it sweet enough for me.

I eat this for breakfast about half the time. Usually I make it with cut up dried apricots, but today I went with all different types of fruit. I dumped them in the saucepan, turned on the heat, and then went away to work on the promotional tutorial and forgot about the boiling milk. Then I remembered and ran back into the kitchen, where the milk proteins had formed huge bubbles in the pan. And suspended in the bubbles were... tiny pieces of walnut skin!

I'm pretty sure that's what they were; they could have been pieces of fruit skin but they sure looked and felt like walnut. I'm pretty sure I've seen the fabled "traces of nuts" and my life is now complete.

[Comments] (1) Best-Loved Ruby Cookbook Recipes of the American People #2: "Generating graphs with Gruff": After a long wait the Ruby Cookbook is now available almost everywhere, even on BookFinder. I urge you to purchase what I hope is the strangest O'Reilly book ever published (suggestions for competitors welcome; the only one I can think of is Stephen Feuerstein's Oracle PL/SQL Programming). In addition to hundreds of tired and frankly predictable geek in-jokes (Star Trek, Rogue, Cryptonomicon, the GNU Virtual Fridge, ad nauseum), it features talking frogs, coffinfish, two-faced politicians, corpses in freezers, dispute resolution through ritual combat, Orwellian doublecode, a heart-pounding Novel on Rails, and Dr. Bronner's Peppermint Soap. Special guest star: T-Rex.

It also features graphs! Recipe 12.4, "Graphing Data", introduces Geoffery Grosenbach's Gruff library, by amazing chance also the subject of today's promotional tutorial. Gruff makes it easy to turn data structures into graphs and write them to PNG files. So the old clock on the wall says it's time for part two of the Book Sales Trilogy:

  1. Getting book information with Ruby/Amazon.
  2. Generating graphs with Gruff.
  3. Generating sparklines with the sparklines gem.

The hardest part of Gruff is installing the dang ImageMagick or RMagick libraries and their dependencies in the first place. It's easy on Debian and other systems with a good packaging system, but otherwise it can be a real pain. The second hardest part is working around Gruff when its simplifying assumptions don't apply to you. I glossed over these in the book but I'll tackle the second one a little bit in this tutorial.

Anyway, yesterday I showed you code to take periodic readings of books' Amazon sales rank. And then I showed it to you again today because the code I wrote yesterday was crap. So read that entry even if you read it earlier. The new code makes the rest of the trilogy much easier to present.

You'll recall (from like ten seconds ago when you read it) that we have a SalesReport class that encapsulates sales rank information from a book. Yesterday, though, I didn't show you anything interesting to do with this information. But the night is ours! Tonight, we graph!

Let's open up the SalesReport class again and make a sales report capable of expressing itself as a line graph:

require 'rubygems'
require 'gruff'

class SalesReport
  # Make a Gruff graph for the sales of this product.
  def make_graph(graph_path)
    g = SalesRankGraph.new(800)
    g.title = "Salesrank over time: #{name}"	
    g.colors = ["black"]
    g.title_font_size = 20
    g.hide_legend = true

This is mostly self-explanatory setup code. In the book I claim that most of the Gruff themes are ugly, but that theme_37signals is okay. Well, that's just, like, my opinion, man, but incontrovertible fact is -- and I should have mentioned this in the book -- that theme_37signals's idea of a good time is to graph the first dataset with a yellow line on what's basically a white background.

That's a really bad idea. I believe there's a UI maxim to the effect of: "Some other color than yellow on white, graph-reader's delight. White under yellow, dangerously confuse a fellow." So I go with theme_37signals but tell Gruff to draw the data line in black: the original high-contrast color for white backgrounds.

The other thing I do is hide the legend, because this graph is only for one product. I would like to graph sales for all of my books on a single graph, but I haven't figured out a good way to do it yet. One of Gruff's simplifying assumptions is that all your data points are spaced evenly along the X-axis starting at X=0. I'd have to insert a bunch of bogus data points for books that came out later; worse, most of my timestamps don't line up precisely, so I'd have to write code to group multiple times into a single data point. So right now I just do one book per graph.

Now we've got one line of code that's very important, because it's where I decide how the data will be represented.

    g.data(@name, collect { |date, rank| 1/rank })

The data method takes an array, and adds it to the graph as a data set. SalesReport is an array of dates and ranks, so I could just pass in the ranks, but that would yield a graph like this:

This is a lousy graph. Unimportant details (long stretches early on where no one bought the book) are the most obvious features, and you can't even see the release of the book. But the data isn't useless; it's just not presented well. We're accustomed to seeing charts go up when the numbers go up (see: any TV commercial featuring a chart), but a good sales rank is very small. Also, as all Web 2.0 types know, book sales follow a power law distribution. A book at 400K sells one copy and jumps to 200K, but you have to sell a mess of books to go from #100 to #90. Displaying the sales rank as though it were linear distorts the data.

I don't know the exact distribution for book sales, but simply taking the inverse of the sales rank gives the graph the right shape. In this graph, the release of the book is obvious, and the time leading up to it makes sense:

What about those labels on the X-axis? Where do they come from? They come from this code:

    label_hash = {}      
    [0, (self.size/2).round, self.size-1].each do |i|
      label_hash[i] = Time.at(self[i][0]).strftime('%m/%d/%Y') if self[i]
    g.labels = label_hash

The graph's labels are a hash that maps positions on the X-axis to strings. Remember, the positions on the X-axis are the indices to the array(s) you passed into the data method. You don't get to choose these values. The X axis starts at zero, and ends at the maximum index of the largest array you passed into data. I choose three labels: one at the beginning, one at the end, and one halfway between. Here's a graph with a lot more history than the Ruby Cookbook one:

Finally, having created the graph, we write it to disk:

    g.write(File.join(graph_path, "#{asin}-salesrank.png"))    

Well, not quite finally. I sneakily referenced a class called SalesRankGraph a while back, and never defined it. That class derives from Gruff::Line, but if you do this graph with a Gruff::Line it'll have weird numbers on the Y-axis:

Those labels are just what you'd think: they're the numbers being graphed. This mighty graph stretches from about zero to about 0.001. Of course, the graph is "really" measuring the inverses of those numbers, but there's no way to put that in the labels. It's another of Gruff's simplifying assumptions. You can choose your X-axis labels but not your X-axis points; you can choose your Y-axis points but not your Y-axis labels. I couldn't find an easy way to fix this, so I just hacked the draw_line_markers to not draw them. You can shut off both the X- and Y-axis labels by setting hide_line_markers, but I like the X-axis labels.

class SalesRankGraph < Gruff::Line
  def draw_line_markers

So now I've got some pretty nice-looking graphs to track my sales rank. But I don't have time to look at graphs! I should have spent today working on my new project, but instead I wasted the morning fixing problems with the preivous entry in this series, and then spent the afternoon making pizza sauce and writing this entry! What to do? If only there were some post-literate infographic that would convey sales rank information at a glance! Something like the graphics I laboriously put on the crummy.com homepage this afternoon! Stay tuned for tomorrow's episode, The Spark of Line!

Dream Update:

Sumana: I dreamed I was having dinner with the president, and I was cracking all these jokes, but nobody was laughing.
Leonard: You dreamed you were Stephen Colbert?

Google Code: A couple weeks ago I talked with Jason Robbins about my new book project. Jason works at Google, along with Greg Stein and several of the other people I respected most at CollabNet.

Jason pointed out that usually when you make a decision it's like the poem from high school, where you never find out what would have happened to your life if you'd chosen differently. But occasionally you do get a glimpse. He said that soon I would get a glimpse of what I'd be doing if I'd gone to work at Google with Greg and him.

And so I have. It seems I would be working on Google Code, a site for hosting open source projects a lot like Tigris and the other CollabNet software development sites.

Hum. Okay, Google is probably a more fun place to work than CollabNet, and Google Code has some great features (I like the radically simple approach to bug tracking). But I don't think Counterfactual Leonard would be happy quitting one job in frustration only to go do another one that's basically the same. I think I'm happier on the road less travelled.

: No one's reading this because Crummy's nameserver died. Kevin's putting together another one. In the meantime, I'd like to point out that I'm not the only one doing promotional tutorials. Among other things, Lucas gave an OSCON talk on distributed programming with DRb and Rinda. He had everyone with a laptop stop typing snide comments into an IRC channel and run his distributed Ruby code, creating an ad hoc SETI@Home project to find prime numbers. Pretty slick.

[Comments] (2) :

Sumana: There should be a Lord of the Rings II.
Leonard: What would be in this Lord of the Rings II?
Sumana: The ring is back! But now we have machine guns.

: I believe I have found the ultimate Wikipedia page: the list of incomplete lists.

[Comments] (2) Anthology Of Ruby Cookbook Recipes That's Not The Ruby Cookbook Itself #3: Sparklines: Okay, thanks to much work by Kevin the nameserver is back up, and I can return you to the Book Sales Trilogy, where I sell the Ruby Cookbook using software that tracks how many copies people have already bought. That's what I call recycling!

On previous installations I showed you how to get sales rank information about a book from Amazon, and how to create a graph of the data over time. Now my vengeance will be complete: I will unleash sparklines upon the world!

Sparklines are really interesting: bits of anonymous data that add quantitative analysis to text, in some cases without breaking the flow of a sentence. The meaning of a sparkline depends on context or a tiny textual label, not on big sets of axial markings. I've been a fan of sparklines for a while, and what better way to propagandize them than by inclusion into a book of cool tricks? THERE IS NONE. So here we go with the final part of the trilogy:

  1. Getting book information with Ruby/Amazon.
  2. Generating graphs with Gruff.
  3. Generating sparklines with the sparklines gem.

As you know, Bob, in previous episodes we defined a SalesReport class which encapsulates information about a product's sales rank over time. Then we extended the class to give it the ability to write out a big graph describing the history of its sales rank. Now we're going to extend it again, and give it the ability to write out a sparkline.

Like Gruff, the sparklines gem is a great piece of code by Geoffrey Grosenbach. It's simpler than Gruff because sparklines are simpler than graphs. It does have some issues you need to watch out for, though. I cover sparklines in Recipe 12.5, "Adding Graphical Context with Sparklines". In the book I give some silly examples focused on embedding sparklines into HTML pages with the data: URI scheme, and incorporating sparklines into Rails views with the sparkline_generator gem. Here, I'll show you to show how to write sparkline graphics to static PNG files.

Those images from the Crummy homepage (Cookbook and Beginning Python ) show sales rank values for the past 30 days. How do I make them? Let's see.

class SalesReport
  # Make a sparkline for the sales of this product.
  def make_sparkline(graph_path, time_units=30, samples_per_unit=24)
    path = File.join(graph_path, "#{@asin}-salesrank-sparkline.png")

    # Gather a sample of the data
    sample = []
    (size-(time_units*samples_per_unit)-1).step(size-1, samples_per_unit) do |i| 
	sample << 1/self[i][1]

A sparkline needs to be small, and unlike Gruff, sparklines doesn't compress data points. Gruff takes an image size in pixels, and whether you give it 8 data points or 800, your graph is that size. But if you give sparklines 800 data points, you get a really long sparkline. I can't use all the data I've gathered since these books showed up on Amazon: I need to economize.

This is fine. I don't need the whole history for a sparkline like I do with a big graph. I'll settle for the sales rank history from the past 30 days. But I run my data collector every hour, and that's still 24*30=720 data points. Solution: I go back 720 hours, and skip ahead 24 samples at at time. This way I pick a sample sales rank for every day, ending with the most recent sample. You can change time_units and samples_per_unit to customize your sparkline.

Now I have an array and I just have to send it to be turned into a sparkline:

    Sparklines.plot_to_file(path, scale(sample), :type => 'smooth', 
                            :line_color => 'black')

Well, not quite. Earlier I mentioned that sparklines line graphs don't compress data horizontally. There's a fixed number of pixels between each point. Well, sparklines doesn't compress data vertically either. A line graph can handle values between 0 and 100. You can make the sparkline bigger but I'm pretty sure you can't increase that range. If you give values outside that range they get clipped or ignored.

In this case, our numbers are already between 0 and 100, since we're taking the reciprocals of numbers greater than one. But they're also between 0 and 1, as we saw in the previous installment. When the range is 0-100, this makes for boring sparklines. If your book rocketed to the top of the Amazon charts and stayed there, its amazing success would look like this: (caution: embedded sparkline not visible in IE).

To make our data visible, we need to stretch it out. We might as well go for maximum stretch: treat the smallest sampled rank as zero, and the largest as 100. That's what the scale method mentioned above does. Here's the definition:

  # Scale data so that the smallest item becomes 0 and the largest becomes 
  # 100.
  def scale(data, bottom=0, top=100)
    min, max = data.min, data.max
    scale_ratio = (top-bottom)/(max-min)
    data.collect { |x| bottom + (x-min) * scale_ratio}

I gave a similar method in the sparklines recipe in the book, but it's hard-coded to scale a range to 0-100. This implementation is more general and you can use it anywhere. I really should have made this a separate recipe because it kept coming up. In 12.14 "Generating MIDI Music" (which I hope to write about later) I reused this formula for a different range and got it wrong, and had to send in an erratum. In 2.12 "Using Complex Numbers" I used a similar formula to scale an ASCII drawing of the Mandelbrot set to any desired size. So it turns out this is a useful method to have.

Anyway, now we can scale any data set to take maximum advantage of the vertical space allotted us by a sparkline. In the book I lightly discuss the ramifications of scaling all these different data sets to the same 0-100 range. The Cookbook sells much better than Beginning Python, but you can't tell that from the sparklines. Which is fine here, because I want these sparklines to show trends.

The sparklines gem lets you do other kinds of sparklines: pie charts are my favorite, as you'll see in the book. I'm really excited about the possibility of sparklines: they're like the little crawls at the bottom of television news stations, except they're classy and related to the main text.

The full sales rank monitor program is available here. I hope other authors find it useful.

What's next for Best of Ye Book of Ruby-Receipts? I don't know. These promotional tutorials take surprisingly long to write: it's almost as much work as was writing the original recipes. So I may give it a rest for a while. But let me know if there's a recipe in the Cookbook you'd like to see me explore in more detail, in the context of a real-world program like the sales rank monitor.

Tiny IF Roundup: This was going to be a whole game roundup but I got distracted BY WORK. Yes, my leisure time was sucked up by book work. I don't know if this is a problem or not. Seems like it might be this early in the book process.

Anyway, I'm just going to review one game, an IF game, and I'm cheating because it's a game I played and liked when I was younger. It's called Crusade Adventure and it came with AGT as a sample game.

I'm a little surprised that no one has written a program to convert AGT code to Inform. AGT and Inform are abstruse in opposite directions, though, so maybe I'm the only one nostalgic enough about AGT to be interested. And as I look at these old games I become less interested. AGT runs just fine in a DOS emulator, after all.

I remember Crusade Adventure as being full of action, with a caving section more sophisticated than Colossal Cave. Well, it's not full of action, it's full of the action-evasion. And its caving section is only sophisticated in that it's a small adjunct to a larger map. The game relies heavily on random events: you can avoid some puzzles (including a fairly clever one) by relying on chance. The map is disjoint, and you have to use a magic word to move around.

Unlike most AGT games, Crusade Adventure is pretty nonlinear and has good atmosphere that I associate with Sir Walter Scott. Unfortunately, it turns out I only associate the atmosphere with Sir Walter Scott because the game makes explicit reference to Ivanhoe, in what might be the most jarring anachronism I've ever seen in an IF game. As Mark Twain would say, "the genuine and wholesome civilization of the nineteenth century is curiously confused and commingled with the Walter Scott Middle-Age sham civilization". I've never read any Scott but his "dreams and phantoms" fit well into an AGT setting: misty woods, suits of armor, talking skulls.

I remember playing a very slightly racier version of Crusade Adventure. But the one on the IF archive is fine because it makes this game stands in stark contrast to the all-out sleaziness of E. L. Cheney's GAGS games, also often distributed with AGT. There was one really disturbing one set in an ultra-sleazy San Francisco that featured lines like: "Lucy's firm young breasts are the center of your focus, You'd love to play around with them, but do you have the time?" Oh, gee, look at the time! A web search reveals that E. L. Cheney now draws cartoons about golf.

As a palate cleanser for the sleaze let me offer more of Twain's Fennimore Cooper-class vitriol against Scott:

It was Sir Walter that made every gentleman in the South a Major or a Colonel, or a General or a Judge, before the war; and it was he, also, that made these gentlemen value these bogus decorations.


<M <Y
Y> M>


Unless otherwise noted, all content licensed by Leonard Richardson
under a Creative Commons License.