News You Can Bruise

I spent a lot of time and used a lot of pages in the Ruby Cookbook showing how to use Net:HTTP to build more and more capable HTTP clients that hid the hairiness of Net:HTTP. But in truth the open-uri library, which we only covered a little, supports almost every feature you could want from an HTTP client library: custom headers, proxies, transparent HTTPS with cert validation, even automatic handling of the redirects it's okay to handle automatically. And the interface couldn't be simpler: it overloads Kernel#open so that you can open an HTTP request like a file, and read the response like data from a filehandle.

I'd like to recommend open-uri for everything, including as a client for REST web services, but there are two big features it's missing. You can't use any HTTP method other than GET, and you can't send any body data (which makes sense if GET is the only supported method). Lately I've been building on my Cookbook work, coming closer and closer towards a general HTTP client library for Ruby, but really all I'm doing is reinventing open-uri with two extra features. It would be a lot better to just add those features to open-uri.

I submitted a patch for open-uri on this topic a while back but have yet to get any response, even from people saying it's stupid. I'm writing REST clients with open-uri under the assumption that that'll work by the time the book is published. For instance, here's some code from my Amazon S3 client:

  # Saves this bucket to S3.
  def save
    open(uri, :method => :put)
    return self
  end

  # Deletes this bucket and all its contents.
  def delete
    open(uri, :method => :delete)
  end

I guess you could argue that these features break the metaphor, that PUTting to a URI isn't really "opening" it, or that it's like opening a file for write access and you should have to write the body data inside a code block the way you do for a file.[0] But c'mon, let's have that argument. I don't want to have to keep telling everyone about fake reimplementations of open-uri just so they can write REST clients in Ruby. There's also precedent, like XMLHttpRequest.open in Javascript.

I'd also like to add caching support to open-uri, like the caching support in Joe Gregorio's httplib2 for Python. But I'll wait on that until the fate of this simpler patch is decided.

[0] I doubt this is really the intent of open-uri, since right now it doesn't support HEAD either. I think the intent of open-uri is to make it trivial to make an HTTP request from Ruby.

Okay, thanks to much work by Kevin the nameserver is back up, and I can return you to the Book Sales Trilogy, where I sell the Ruby Cookbook using software that tracks how many copies people have already bought. That's what I call recycling!

On previous installations I showed you how to get sales rank information about a book from Amazon, and how to create a graph of the data over time. Now my vengeance will be complete: I will unleash sparklines upon the world!

Sparklines are really interesting: bits of anonymous data that add quantitative analysis to text, in some cases without breaking the flow of a sentence. The meaning of a sparkline depends on context or a tiny textual label, not on big sets of axial markings. I've been a fan of sparklines for a while, and what better way to propagandize them than by inclusion into a book of cool tricks? THERE IS NONE. So here we go with the final part of the trilogy:

1. Getting book information with Ruby/Amazon.
2. Generating graphs with Gruff.
3. Generating sparklines with the sparklines gem.

As you know, Bob, in previous episodes we defined a SalesReport class which encapsulates information about a product's sales rank over time. Then we extended the class to give it the ability to write out a big graph describing the history of its sales rank. Now we're going to extend it again, and give it the ability to write out a sparkline.

Like Gruff, the sparklines gem is a great piece of code by Geoffrey Grosenbach. It's simpler than Gruff because sparklines are simpler than graphs. It does have some issues you need to watch out for, though. I cover sparklines in Recipe 12.5, "Adding Graphical Context with Sparklines". In the book I give some silly examples focused on embedding sparklines into HTML pages with the data: URI scheme, and incorporating sparklines into Rails views with the sparkline_generator gem. Here, I'll show you to show how to write sparkline graphics to static PNG files.

Those images from the Crummy homepage (Cookbook and Beginning Python ) show sales rank values for the past 30 days. How do I make them? Let's see.

class SalesReport
  # Make a sparkline for the sales of this product.
  def make_sparkline(graph_path, time_units=30, samples_per_unit=24)
    path = File.join(graph_path, "#{@asin}-salesrank-sparkline.png")

    # Gather a sample of the data
    sample = []
    (size-(time_units*samples_per_unit)-1).step(size-1, samples_per_unit) do |i| 
	sample << 1/self[i][1]
    end

A sparkline needs to be small, and unlike Gruff, sparklines doesn't compress data points. Gruff takes an image size in pixels, and whether you give it 8 data points or 800, your graph is that size. But if you give sparklines 800 data points, you get a really long sparkline. I can't use all the data I've gathered since these books showed up on Amazon: I need to economize.

This is fine. I don't need the whole history for a sparkline like I do with a big graph. I'll settle for the sales rank history from the past 30 days. But I run my data collector every hour, and that's still 24*30=720 data points. Solution: I go back 720 hours, and skip ahead 24 samples at at time. This way I pick a sample sales rank for every day, ending with the most recent sample. You can change time_units and samples_per_unit to customize your sparkline.

Now I have an array and I just have to send it to be turned into a sparkline:

    Sparklines.plot_to_file(path, scale(sample), :type => 'smooth', 
                            :line_color => 'black')
  end
end

Well, not quite. Earlier I mentioned that sparklines line graphs don't compress data horizontally. There's a fixed number of pixels between each point. Well, sparklines doesn't compress data vertically either. A line graph can handle values between 0 and 100. You can make the sparkline bigger but I'm pretty sure you can't increase that range. If you give values outside that range they get clipped or ignored.

In this case, our numbers are already between 0 and 100, since we're taking the reciprocals of numbers greater than one. But they're also between 0 and 1, as we saw in the previous installment. When the range is 0-100, this makes for boring sparklines. If your book rocketed to the top of the Amazon charts and stayed there, its amazing success would look like this: (caution: embedded sparkline not visible in IE).

To make our data visible, we need to stretch it out. We might as well go for maximum stretch: treat the smallest sampled rank as zero, and the largest as 100. That's what the scale method mentioned above does. Here's the definition:

  # Scale data so that the smallest item becomes 0 and the largest becomes 
  # 100.
  def scale(data, bottom=0, top=100)
    min, max = data.min, data.max
    scale_ratio = (top-bottom)/(max-min)
    data.collect { |x| bottom + (x-min) * scale_ratio}
  end

I gave a similar method in the sparklines recipe in the book, but it's hard-coded to scale a range to 0-100. This implementation is more general and you can use it anywhere. I really should have made this a separate recipe because it kept coming up. In 12.14 "Generating MIDI Music" (which I hope to write about later) I reused this formula for a different range and got it wrong, and had to send in an erratum. In 2.12 "Using Complex Numbers" I used a similar formula to scale an ASCII drawing of the Mandelbrot set to any desired size. So it turns out this is a useful method to have.

Anyway, now we can scale any data set to take maximum advantage of the vertical space allotted us by a sparkline. In the book I lightly discuss the ramifications of scaling all these different data sets to the same 0-100 range. The Cookbook sells much better than Beginning Python, but you can't tell that from the sparklines. Which is fine here, because I want these sparklines to show trends.

The sparklines gem lets you do other kinds of sparklines: pie charts are my favorite, as you'll see in the book. I'm really excited about the possibility of sparklines: they're like the little crawls at the bottom of television news stations, except they're classy and related to the main text.

The full sales rank monitor program is available here. I hope other authors find it useful.

What's next for Best of Ye Book of Ruby-Receipts? I don't know. These promotional tutorials take surprisingly long to write: it's almost as much work as was writing the original recipes. So I may give it a rest for a while. But let me know if there's a recipe in the Cookbook you'd like to see me explore in more detail, in the context of a real-world program like the sales rank monitor.

After a long wait the Ruby Cookbook is now available almost everywhere, even on BookFinder. I urge you to purchase what I hope is the strangest O'Reilly book ever published (suggestions for competitors welcome; the only one I can think of is Stephen Feuerstein's Oracle PL/SQL Programming). In addition to hundreds of tired and frankly predictable geek in-jokes (Star Trek, Rogue, Cryptonomicon, the GNU Virtual Fridge, ad nauseum), it features talking frogs, coffinfish, two-faced politicians, corpses in freezers, dispute resolution through ritual combat, Orwellian doublecode, a heart-pounding Novel on Rails, and Dr. Bronner's Peppermint Soap. Special guest star: T-Rex.

It also features graphs! Recipe 12.4, "Graphing Data", introduces Geoffery Grosenbach's Gruff library, by amazing chance also the subject of today's promotional tutorial. Gruff makes it easy to turn data structures into graphs and write them to PNG files. So the old clock on the wall says it's time for part two of the Book Sales Trilogy:

1. Getting book information with Ruby/Amazon.
2. Generating graphs with Gruff.
3. Generating sparklines with the sparklines gem.

The hardest part of Gruff is installing the dang ImageMagick or RMagick libraries and their dependencies in the first place. It's easy on Debian and other systems with a good packaging system, but otherwise it can be a real pain. The second hardest part is working around Gruff when its simplifying assumptions don't apply to you. I glossed over these in the book but I'll tackle the second one a little bit in this tutorial.

Anyway, yesterday I showed you code to take periodic readings of books' Amazon sales rank. And then I showed it to you again today because the code I wrote yesterday was crap. So read that entry even if you read it earlier. The new code makes the rest of the trilogy much easier to present.

You'll recall (from like ten seconds ago when you read it) that we have a SalesReport class that encapsulates sales rank information from a book. Yesterday, though, I didn't show you anything interesting to do with this information. But the night is ours! Tonight, we graph!

Let's open up the SalesReport class again and make a sales report capable of expressing itself as a line graph:

require 'rubygems'
require 'gruff'

class SalesReport
  # Make a Gruff graph for the sales of this product.
  def make_graph(graph_path)
    g = SalesRankGraph.new(800)
    g.title = "Salesrank over time: #{name}"	
    g.theme_37signals
    g.colors = ["black"]
    g.title_font_size = 20
    g.hide_legend = true

This is mostly self-explanatory setup code. In the book I claim that most of the Gruff themes are ugly, but that theme_37signals is okay. Well, that's just, like, my opinion, man, but incontrovertible fact is -- and I should have mentioned this in the book -- that theme_37signals's idea of a good time is to graph the first dataset with a yellow line on what's basically a white background.

That's a really bad idea. I believe there's a UI maxim to the effect of: "Some other color than yellow on white, graph-reader's delight. White under yellow, dangerously confuse a fellow." So I go with theme_37signals but tell Gruff to draw the data line in black: the original high-contrast color for white backgrounds.

The other thing I do is hide the legend, because this graph is only for one product. I would like to graph sales for all of my books on a single graph, but I haven't figured out a good way to do it yet. One of Gruff's simplifying assumptions is that all your data points are spaced evenly along the X-axis starting at X=0. I'd have to insert a bunch of bogus data points for books that came out later; worse, most of my timestamps don't line up precisely, so I'd have to write code to group multiple times into a single data point. So right now I just do one book per graph.

Now we've got one line of code that's very important, because it's where I decide how the data will be represented.

    g.data(@name, collect { |date, rank| 1/rank })

The data method takes an array, and adds it to the graph as a data set. SalesReport is an array of dates and ranks, so I could just pass in the ranks, but that would yield a graph like this:

This is a lousy graph. Unimportant details (long stretches early on where no one bought the book) are the most obvious features, and you can't even see the release of the book. But the data isn't useless; it's just not presented well. We're accustomed to seeing charts go up when the numbers go up (see: any TV commercial featuring a chart), but a good sales rank is very small. Also, as all Web 2.0 types know, book sales follow a power law distribution. A book at 400K sells one copy and jumps to 200K, but you have to sell a mess of books to go from #100 to #90. Displaying the sales rank as though it were linear distorts the data.

I don't know the exact distribution for book sales, but simply taking the inverse of the sales rank gives the graph the right shape. In this graph, the release of the book is obvious, and the time leading up to it makes sense:

What about those labels on the X-axis? Where do they come from? They come from this code:

    label_hash = {}      
    [0, (self.size/2).round, self.size-1].each do |i|
      label_hash[i] = Time.at(self[i][0]).strftime('%m/%d/%Y') if self[i]
    end
    g.labels = label_hash

The graph's labels are a hash that maps positions on the X-axis to strings. Remember, the positions on the X-axis are the indices to the array(s) you passed into the data method. You don't get to choose these values. The X axis starts at zero, and ends at the maximum index of the largest array you passed into data. I choose three labels: one at the beginning, one at the end, and one halfway between. Here's a graph with a lot more history than the Ruby Cookbook one:

Finally, having created the graph, we write it to disk:

    g.write(File.join(graph_path, "#{asin}-salesrank.png"))    
  end
end

Well, not quite finally. I sneakily referenced a class called SalesRankGraph a while back, and never defined it. That class derives from Gruff::Line, but if you do this graph with a Gruff::Line it'll have weird numbers on the Y-axis:

Those labels are just what you'd think: they're the numbers being graphed. This mighty graph stretches from about zero to about 0.001. Of course, the graph is "really" measuring the inverses of those numbers, but there's no way to put that in the labels. It's another of Gruff's simplifying assumptions. You can choose your X-axis labels but not your X-axis points; you can choose your Y-axis points but not your Y-axis labels. I couldn't find an easy way to fix this, so I just hacked the draw_line_markers to not draw them. You can shut off both the X- and Y-axis labels by setting hide_line_markers, but I like the X-axis labels.

class SalesRankGraph < Gruff::Line
  def draw_line_markers
  end
end

So now I've got some pretty nice-looking graphs to track my sales rank. But I don't have time to look at graphs! I should have spent today working on my new project, but instead I wasted the morning fixing problems with the preivous entry in this series, and then spent the afternoon making pizza sauce and writing this entry! What to do? If only there were some post-literate infographic that would convey sales rank information at a glance! Something like the graphics I laboriously put on the crummy.com homepage this afternoon! Stay tuned for tomorrow's episode, The Spark of Line!

You know those old-time radio shows, where Doris Day always guest-starred, and Bob Hope ended up wandering onto the field at Muroc and getting shot? Well, that particular one never aired, but you know the kind of show I mean. The shows that were sponsored by random household objects like soap, or a broom, where the straight man would set up a line and the funny man would always turn it into a little ad for the sponsor.

"Lou, I think we ought to stick around a while longer." "You know, Acme Wallpaper is the only thing today's handyman needs to 'stick around' the house. It's inexpensive, durable, and when properly applied it forms a Penrose tiling! Ask your dealer to see a catalog today." Thus allowing the straight man to set up the exact same line again: "I say, Lou, that we ought to stick around a while longer and watch after Doris Day."

This is the model for this series of weblog entries, where I show you how to do cool things with Ruby but it turns out all to be a scheme to plug the Ruby Cookbook, bad boy of the O'Reilly cookbook family. In the first three entries of this series, I'll show you the system I rigged up to track the sales of my books. I don't really have a plan after that, but I can work from a list I made of the coolest recipes in the Cookbook.

The Book Sales Trilogy:

1. Getting book information with Ruby/Amazon
2. Generating graphs with Gruff.
3. Generating sparklines with the sparklines gem.

The problem

When you write books but don't publish them, it's difficult to know how many you've sold. The publisher gives you a total maybe quarterly. So a lot of authors (including me) use their Amazon sales rank as a proxy for books sold.

There are numerous problems with this, stemming from two facts: 1) Amazon sales rank only covers books sold through Amazon, and 2) it only conveys how well your book is selling _compared to all other books_. Booms and crashes in the book market as a whole won't make any difference to your sales rank. But you can't beat something with nothing, and sales rank is the only way I know of to get anything like real-time sales figures for a book. I talked with Mike Loukides, my O'Reilly editor, today, and he said that giving authors access to sales information is a low-priority project. And we all know about low-priority projects.

So, whenever a book of mine goes up on Amazon (first Learning Python and now the Ruby Cookbook), I register its ISBN with a script that runs once an hour and uses Amazon Web Services to collect the sales rank numbers. My goal is to encapsulate this logic into a class which maps an ISBN to a text file full of timestamped sales ranks. It will also work with Amazon-specific ASINs: for books, the ASIN is just the ISBN.

I use the Ruby/Amazon library, which is covered in Recipe 16.1, "Searching for Books on Amazon". The recipe focuses on keyword searches: finding books or other objects to match criteria. Here, we search by ASIN.

The script is based around a SalesReport class that encapsulates data about one product's sales rank over time. Here's the first draft of SalesReport.

#!/usr/bin/ruby
require 'amazon/search'
class SalesReport < Array
  attr_accessor :name, :asin
  def initialize(asin, name)
    @asin, @name = asin, name
    open(self.class.filename(@asin)).each do |line| 
      self << line.split.collect { |x| x.to_f }
    end
  end

  def self.filename(asin)
    asin + '.txt'
  end

When you create a SalesReport object for a product, it opens up a file named after that product's ASIN and reads in sales data.

Where does that file come from? The update! method is in charge of creating and maintaining it. This is also where we make the actual AWS request (in bold).

  # Updates sales rank for a number of products, and yields a
  # SalesReport object for each one.
  def self.update!(aws_key, *asins)
    now = Time.now.to_i
    req = Amazon::Search::Request.new(aws_key)
    req.asin_search(asins) do |product|
      asin, name = product.asin, product.product_name

      # Update the file with the new sales rank.
      open(self.filename(asin), 'a') do |f| 
        f << now << ' ' << product.sales_rank << "\n"
      end

      # Parse the file into an SalesReport object and yield it.
      yield self.new(asin, name)
    end
  end
end

Three things about this method's signature. First, like filename this is a class method: it's a factory for SalesReport objects rather than something you run on a particular SalesReport. For more on this, see recipe 8.18, "Creating Class and Singleton Methods". The self here is the SalesReport class itself.

Second, this method takes a variable number of ASIN arguments, for which see Recipe 8.11, "Accepting or Passing a Variable Number of Arguments".

Third, as I've mentioned in every book I've ever written, to use AWS you need to sign up for a unique string called a "developer token". This method uses that string to create an Amazon::Search::Request object, which will be our interface to Amazon Web Services.

You can reuse a Request object to do multiple searches, but we're only doing one. To save time and bandwidth, we pass all of our ASINs in a single search: "give me info on these products". The query returns a big XML document, which Ruby/Amazon parses into struct-like Ruby objects. It yields each object to a code block.

Inside the code block, I have access to a variable product which represents what Amazon knows about one particular book. I take its sales rank reading and stick it along with a timestamp into that book's file. Within the code block I also build a SalesReport object, which I yield to another code block somewhere outside this function. The code that calls SalesReport.update! gets yielded a series of SalesReport objects to do with what they will. In this case, my will is to generate cool-looking graphs.

If code blocks confuse you (they confused me when I was new to Ruby), the first few recipes of chapter 7 will help. If you're not familiar with file access, the basics (including appending to a file) are covered in Recipe 6.7, "Writing to A File".

Okay, that's all the SalesReport code for today. Here's how you'd call SalesReport.update! with a code block:

key = 'my AWS developer token whatever'
SalesReport.update!(key, "0596523696", "0764596543") do |product|
  puts "I updated #{product.name} (Rank: #{product[-1][1].to_i})"
end
# I updated Ruby Cookbook (Cookbooks (O'Reilly)) (Rank: 1412)
# I updated Beginning Python (Programmer to Programmer) (Rank: 219370)

So now I've got two files on Crummy's hard disk full of sales rank readings. What to do with that data? That code block doesn't really do anything., Well, I'd love to compare the sales ranks to hard sales numbers, but I've never gotten any for the Python book, and the one data point I got from Michael today doesn't mean much. So I've been comparing these numbers to their earlier selves by graphing them versus time. This leads into tomorrow's episode, Graphic Violence, featuring Chico Marx.

Oh, I don't think I mentioned that the Ruby Cookbook has back-cover blurbs from why and Matz! You can see them both at the Ruby Cookbook Official Unofficial Homepage.

Floyd from The Fresh Loaf, who presumably knows from bread and bread byproducts, tells me that the sourdough soup I had at the Polish restaurant was probably zurek, and looking at the recipe I agree. Zurek is made with rye sourdough starter, so it is actually sourdough soup. Inventive.

Floyd also mentioned that he dabbles in Ruby, so that's one. I am going through the Poignant Guide, which is funny and gonzo and Fafbloggish, and I have a simple question about idiom. I see code like the following, which sets up a block that deals with a single item:

File::open('foo.txt', 'w') do |f|
  f << "text"
end

This idiom makes sense when you're iterating over a bunch of items, but why create a new block to deal with a single item? Is it somehow more Rubic (Rubyic?) than the following code, which seems much more straightforward?:

f = File::open('foo.txt', 'w')
f << "text"

Let me know. I'm trying to spare you the details of my mastery of Ruby (and puppets), but this is undoubtedly the first of many questions.

There seems to be some interest in a Ruby port of Beautiful Soup, and I'm thinking of doing the port myself. I need to stop being a dilletante and do a good-sized Ruby project, and that's a good place to start. Do any Ruby people read this weblog? Would you like to have this?