News You Can Bruise

• "Introduction to Computing" (Drexel)
• "Design and Analysis of Algorithms" (Stanford)
• "Object Oriented Problem Solving" (Colorado State)
• "Fundamentals of Artificial Intelligence" (UCLA)
• "Design and Architecture of Computer Systems" (UC Riverside)
• "Computer Security" (UC Berkeley)
• "Operating Systems" (Harvard)
• "Building High Performance Servers" (Brown)

At the risk of revealing all my secrets, I'm getting the #retrorocket pictures by harvesting the Internet Archive's NASA Images site, one year at a time. I've combed through 1969-1988, and my technique will break down around 1994, when there start to be thousands of pictures for each year (even 1969 wasn't that bad). But 1994 is not all that retro, really, is it? Yeah, it's kinda retro.

Despite the name, NASA Images has a lot of non-images: mainly movies and technical documents. Recently one tech document caught my eye: Technology Utilization Network System, a 1987 document laying out NASA's recommendations for off-the-shelf PC hardware and software.

This document pulls no punches, naming brand names, presenting huge benchmark tables, and spelling out just what it takes to outfit an effective mid-80s networked office on the taxpayer's dime. This is a time before Windows, in an office environment without Unix. Let's take a look at this document and see which products have the Right Stuff, and which fizzle on the launch pad. (n.b. Entertainment Weekly didn't want this for some reason, and I'm too lazy to change the segue.)

We'll start with the basic PC recommendation. "The recommended workstation for TUNS is the Compaq DeskPro 286 with a 40 Mb hard disk." Each should be equipped with an EGA card and a color monitor. "The estimated GSA cost... is $4,087." (About $7700 in 2010 dollars.)

The use as a workstation of one of the newer 386 machines (based on the Intel 80386 CPU chip) is not currently recommended. The power provided by the 386 system is more than that required of the workstation.

Unlike a lot of offices in 1987, NASA understands the importance of networking. That's why each workstation is fitted with a 3Com Etherlink card ($451) and connected to a local file server running Novell Advanced Netware/286. Netware wins out over now-obscure competitors like 3Com's 3+ Share, Banyon VINES/286, Fox 10-net, and Lee Data's LANMASTER. The Internet stack is not even considered: I'm sure NASA's scientific installations are on the Internet by now, but it's not really an option for IBM PCs, and the term "intranet" doesn't exist.

The Compaq 286 used for workstations can also act as the file server on a small LAN. But for a larger LAN, you do need to buy that 386, and for the really big installations, it's got to be "a Novell T286B with a 183 Mb of hard drive space". And remember to buy name-brand:

Although many vendors claim to sell "AT clones," ISN has occasionally found very subtle differences in the performance of these "clones," which may result in problems during system integration.

For connecting to external sites and databases, each workstation is outfitted with a Hayes Smartmodem 2400 at $579.

ISN does not recommend installation of 9600 baud modems at this point. The lack of standard protocols, error-correction methods and data compression techniques for 9600 baud communications means that two modems from different vendors will rarely communicate with each other at 9600 baud.

Printer time! TUNS spells out recommendations for cheapo dot-matrix printers (the Epson FX-286e, $527), letter-quality daisywheel printers (the Diablo D80IF, $1523), and laser printers for impressing the boss (the HP Laserjet Series II, $1795). Note that the most expensive printer is half the price of the workstation PC.

Two printers, the Brother Twinwriter 5 and the Fortis DH-45, include both dot-matrix and daisy-wheel print mechanisms. The two companies are actually marketing the same printer under different labels. Although this printer was initially viewed as an exciting combination of functionality at a reasonable price, it was excluded from further consideration after the Twinwriter 5 vendor reported extremely poor reliability and great customer dissatisfaction.

What about software? For the most part, TUNS recommends off-the-shelf 80s office software, Lotus 1-2-3 (GSA cost: $305) and WordPerfect 4.2 ($173):

Although the evaluation scores for WordPerfect and Microsoft Word show only a two point difference, the evaluation team highly recommends WordPerfect because the evaluation team found it much easier to use WordPerfect's ACSII-to-document transfer features.

But they choose Unify, a database I've never heard of, over 80s heavyweights like dBase and Clipper. Why? Because Unity just has more stuff, like a C library.

Oracle was eliminated as a possibility because a LAN version of the software, although in development for some time, is still not available.

Each 286 workstation is loaded with about $100 worth of utilities: Popular ones like Sidekick and Norton Utilities, as well as some more obscure ones: "Sideways to aid in the printing of large spreadsheets; ScreenSave to protect the life of the monitors; KeyBuffer to allow the user to enter characters from the keyboard at a faster rate than acceptable under DOS; and FilePath to aid in the use of multiple directories and sub-directories."

But the best part of the whole document is the section on email. The document identifies a large number of requirements for an email system, such as:

The sender must be able to identify a single recipient, multiple recipients, and a "group" mailcode consisting of multiple mail identification codes but addressed as one unit.

And then presents a huge table comparing the competitors (again, Internet email is not even on the radar). And then decides to just keep using "the NASAMAIL system currently available throughout NASA." NASAMAIL was actually an installation of Sprint Telemail, which you can read a little more about in RFC 1168, including a mention of a Telemail-Internet mail gateway at NASA Ames.

It's a little odd that hardware of this description almost never shows up in the NASA Images archive! (I did see one mid-80s IBM PC sitting on someone's desk, but when I went back to look for it as an illustration I couldn't find it.) I think these recommendations were mostly for clerical workers and managers, and that the engineers and scientists (who show up quite a bit in the archive) used minicomputers, Unix workstations, and mainframes into the 90s.

I'm probably not gonna go through any more of these documents in any detail, but here are two others I found really interesting: the public affairs plan for STS-1, the first Space Shuttle flight, and the original press kit for Apollo 13.

The main problem is the arbitrary restriction of Amazon Web Services where you can't get more than 250 pages of query results. If it weren't for this restriction, you could pick a random number from a huge range and do a Browse Node search to get the corresponding page of results. This is still a pretty good solution if you want to search for products in a restrictive set like the set of science fiction books: find the corresponding Browse Node and do a browseBestSellers.

As it is, the simplest solution is to only search for a single word, instead of Jarno's two. Here's some code that does that and works pretty well.

import amazon
import random

def random_product(keyword, max_page=250, associate="crummthesite-20"):
    result = None
    while result == None:
        page = random.randint(1, max_page)
        try:
            results = amazon.searchByKeyword(keyword, page=page, type="lite",
                                             associate=associate)
            if results:
                result = random.choice(results)
        except amazon.AmazonError:
            results = None
        if not results:
            max_page /= 2
    return result

def format_product(p):
    print '<a href="%s">%s</a>' % (p.URL, p.ProductName)

keyword = random.choice(open("american.0").readlines()).strip()
format_product(random_product(keyword))

Sample book: Reutilization of Waste Materials.

The other solution is screen-scraping. As covered earlier, there is an Amazon URL such that you can randomly vary a parameter and get a more or less random set of books. Here's some code that fetches one of those URLs and grabs an ASIN at random. Then it does an AWS lookup of that ASIN because life is too short to scrape all the appropriate information off that big ol page. This is a little less reliable than the earlier implementation, but it gets a better random distribution of books (since the other one picks a word to search on, and word distribution is not random).

from BeautifulSoup import BeautifulSoup
import amazon
import random
import urllib2
import re
import socket

URL = "http://www.amazon.com/exec/obidos/tg/new-for-you/top-sellers/-/books/all/books/0/1/%s/1/"
def random_book(max_page=150000, associate="crummthesite-20"):

    product = None

    while product == None:
        url = URL % random.randint(1, max_page)

        req = urllib2.Request(url)
        try:
            soup = BeautifulSoup(urllib2.urlopen(req).read())
            asins = soup.fetch("input", {"type" : "hidden",
                                         "name" : re.compile("asin\.*")})
            if asins:
                asin = random.choice(asins)['name'][5:]
                product = amazon.searchByASIN(asin, "lite", associate=associate)[0]
        except socket.error:
            product = None
        if not product:
            max_page /= 2
    return product

def format_product(p):
    print '<a href="%s">%s</a>' % (p.URL, p.ProductName)

print format_product(random_book())

Sample book: Choice, Welfare and Measurement

Update: I don't know anything.

It's a good idea though, until someone gets mad that you're changing their RSS feeds, which I estimate would take 16 seconds.

I was clearly a little off on my timing there, because it's been at least 18 seconds and I don't see anyone who's complained about this yet. I guess all the people who would get mad are still diverted by the spectacle of Greasemonkey modifying their web pages.

Fortunately you can get around some of the problems by hacking together stuff using the web service APIs. They don't do nearly as much as I'd like, but Amazon wish lists with all their problems are still the only online book tracker that provides any web service API at all. Another comment in that weblog entry is the inevitable "we're doing it better with Rails and we need beta testers", which is worth a shot; as is (still) LibraryThing if you just want something that works. The Z39.50 gateway is looking better to me all the time; the software equivalent of beer goggles.

Book update: files chapter basically done.

In the web services portion of Beginning Python I demonstrate a web service client that goes through an Amazon wish list looking for bargains. A bargain is defined as an item available used through Amazon for some small fraction of its new price.

There are lots of bargains because many people think they can make money selling used books online for a penny and making it up in volume. Actually they think they can make it up by skimming money off the $3.50 shipping surcharge that Amazon charges the buyer. But they might as well be trying to make it up in volume, because $3.50 is not that much when you think of the labor involved in determining the book's condition, doing the data entry, then when someone buys the book finding it, packaging it, and shipping it out. That kind of low wage is usually associated with foolhardy professions like writing the books in the first place.

But people persist in listing books online at sub-yard-sale prices, and even when you tack on $3.50 for shipping you can find lots of good deals on books you wanted to buy anyway. The formula I would use today (slightly different from the one in the book) would be to add the UsedPrice for an item to the shipping cost (assumed $3.50), then consider the item a "bargain" if the total cost is 25% or less of Amazon's OurPrice.

Ideally you would have a script that generated an RSS feed containing all the current bargains on your wish list. I haven't written this useful program yet because my pressing concern right now is slowing down the rate at which I acquire books from my wish list.

Recently I was given an Amazon gift certificate. You can't use gift certificates on used items because you're not really buying from Amazon. The rational way to use them is on things that aren't available used, or that cost just as much used as new. In other words, anti-bargains. Tweak the threshold algorithm and by bargain finder script became an anti-bargain finder.

The anti-bargain finder relies on the assumption that used availability on Amazon is proportional to used availability in the real world. If there were some book that crowded used bookstores in the real world but wasn't being sold used on Amazon at all, it would be falsely flagged as an anti-bargain. This is unlikely to happen because so many of the used listings on Amazon come from real-world used bookstores in the first place.

I think I had another hack idea in this category but I can't remember it at the moment. These are pretty simple ideas but they can save you money.

If you only want to make an annotated list of all your books then use LibraryThing. It's fast, easy to use, and the data is reliable. You can simulate multiple discrete lists by using tags. It costs $10, which is not much compared to other things you've paid for. You could also use All Consuming, which is free and does non-books, but was really slow when I tried to use it (which was a long time ago). You could also use one of the 200 bookshelf applications sure to show up on Ning. You could buy Delicious Library for your Mac, which uses a webcam as a barcode scanner. You could, in short, use whatever incarnation of this particular project tickles your fancy, because your fancy is not tickled by external APIs.

If you want to build a list of books and then use it as a data structure, then Amazon wish lists are the only game in town operating on a higher level than a Z39.50 binding: see for instance this NYCB from last year, before I got REST religion. Amazon provides some interesting data not available through other APIs, like prices and sales rank, but you can join on ISBN to get that data, so a lot of the things in the series will be applicable to people with Tyrannioware or other Z39.50 systems.

Link from waxy, which has seemingly picked up an interest in recommendation engines.

I just know I'm gonna turn into one of those crotchety computer geezers who had an idea back in 1972 and spends their whole life complaining that nobody Gets It.

Maybe software packages need subtitles, the way every nonfiction book nowadays has a subtitle that explains the catchy main title in way too much detail. "Beautiful Soup: The Hidden History Of The Python *ML Parser That Handles Bad Input And Provides Tree Traversal Methods".

I'm really proud of the web chapter (and undoubtedly I will remain proud until I get the reviewers' comments), partly because it brings me full circle back to the student class on CGI I gave six years ago. But mainly because in this chapter I suggest a definition of web services that I've only seen one other place, but that I'd like to see more often. I think it's a pretty intuitive definition that will hopefully set the younger generation on the right path.

In the networking chapter I don't do anything like that. For one, I don't know nearly as much about low-level networking as I do about web programming (though I know a hell of a lot more about networking than I did when I started writing that chapter). But mainly it's because on the network and the transport layer, we already had the fight between the simple geeky incrementally-developed protocol and the huge corporate designed-by-committee protocols, and just like in the movie, the ragtag band of geeks and their scruffy but loveable protocol won. Not much to do now except to show people how to use it.

We had that same fight on the application layer for email and chat and "content delivery" and you name it, and almost all the time the geeks won. And just when you thought the fight had been appealed all the way up the protocol stack and settled once and for all, now we're having the same fight on top of HTTP! Being too young to have been involved in the earlier fights, I figured I might as well do my bit for this one, and throw in my lot with the idea of web services I think fits the best with the Python philosophy.

Update: The guy doing the Hacks book (okay, it's Danny, if you must know) says of my hack: "I'm speechless!". Nothing like a shot of praise to bolster my spirits in preparation for the clobbering I fear the tech reviewers will give my web chapter.

I know everyone hates software that's hard to install and that the Gleeper is currently such a software, but I was surprised at how for many people the breaking point was having to set up a MySQL database. If most of that is balking at setting up MySQL, then SQLite will probably solve the balking problem. I haven't installed it but I don't see how it could be harder to install than MySQL. Plus the database files it keeps can live in the same directory as the app instead of in /var/, which I think is probably the big draw though it doesn't make much difference setup-wise.

Because people (including me) are all about self-containment, I'm planning to distribute a copy of SQLObject with future versions of the Gleeper. That should short-circuit the SQLObject version fiasco, and it's the only external library the Gleeper depends on. Including a copy of every single external library works well with NewsBruiser, and as a matter of fact it works with the Gleeper, which already includes about ten external modules in its lib/ directory. Why not SQLObject too?

I don't want to distribute SQLite in the package, because it's not a Python library. I'd have to either distribute platform-specific binaries or hook up with SQLite's build system. That said, I don't have a problem with changing over to SQLite if people think that's easier. I had all sorts of doubts about SQLite, but then I realized they're the same doubts people who use proprietary databases have about open source databases, so they're probably WRONG. SQLite looks really nice.

The other thing I could do is write a script that takes you through the installation process, instead of a document that tells you how to do it. A very powerful tool for making installation documents look less complicated is to hide the verbiage at the other end of hypertext links that might not be followed. I didn't do this because I didn't have time before CodeCon. I'm still not doing it because I'm working on a different project, with a deadline. Spooky! To me, anyway. Whoooo!

PS: This is also why there's no new Beautiful Soup, or whatever else it is I promised you I'd do.

So in addition to the anti-spam measures I took yesterday, this morning I updated my subwiki installation and started toughening it. I put in robots meta tags similar to those in NewsBruiser, and email notification of changes so it'll poke me if it gets spammed. Unfortunately un-spamming a page in subwiki is pretty difficult, which is why I was so lazy in the first place, so now I need to write a 'revert' function. Then I can have a 'click here to revert' link in the update email like I do with "click here to mark as spam" in the NewsBruiser comment emails.

I wish Bayesian techniques would work well on wikis, but it seems like a just-change-the-links attack would be even easier to do for wiki pages than for weblog comments.

First, the spam I was getting seemed to be manually entered. There were long (by web bot standards) and irregular pauses between hits from spammers, and slightly differing spam methodologies. Sometimes the log message for a spammy change had a solitary "d" or "df" in it, the mark of a person used to scribbling crap data into web forms. It was kind of sad to imagine the people whose job it is to manually spam wikis, like an email spammer who must manually type in SMTP commands. Maybe I'm missing all the bot-based attacks because I use a relatively unpopular wiki software (just as the only comment spam I've ever seen on NewsBruiser was manually entered)

What is new and interesting, though, is the way the wiki spammers locate wikis to spam. To get things started there must be some bots that can spam SubWiki, or particularly persistent manual wiki spammers. But once there is any spam at all in a wiki, a pheremone trail has been laid down and the hordes close in.

You see, the majority of manual wiki spammers seem to be free riders who use search engines to find wikis that have already been spammed, then go in and replace the preexisting spam with their spam. The internicene warfare rages without end, as spammers destroy each others' contributions to the wikispamosphere while making a mockery of the work of the original spammer who went through all the trouble of finding that wiki in the first place, or writing a bot that could spam SubWiki. For shame!

There are a variety of engines and sub-techniques in use, but the most common one is to search for "wiki" plus the name of a site to which wiki spam points. Then, for each hit, go into the wiki and replace the (spam) text of the page with your own spam text. This anti-wiki-spam organization has documented this behavior, but not its comical implication.

Until I can figure out a better solution which hopefully doesn't involve me doing a whole lot of SubWiki development or switching wiki software, I am going to do a little free riding of my own. I've implemented a couple solutions that protect the NewsBruiser wiki only because it's not worth five seconds of a spammer's time to figure out what's going on, when there are so many other wikis they could be spamming.

<!  DOCTYPE HTML PUBLIC...

That is, as opposed to:

<!DOCTYPE HTML PUBLIC...

More generally, is it okay to have whitespace between the <! that begins a declaration and the actual name of the declaration? The W3C validator says it's fine, and I couldn't find anything in the standard about it, and the general SGML-based-language attitude towards whitespace is "whitespace is all in your head, maaaaan." However, there are intimations on various webpages that it's invalid, and the Python HTML parser chokes on it (this is a leading cause of complaints against Beautiful Soup). I have a fix for the Python HTML parser but before I put all the effort into making sure it doesn't break anything else I'd like to know if it's even something that's "supposed" to work.

When I read stories in which the characters have software companies (which is not that often) I wonder why the author doesn't just start that software company themselves if it's such a great idea. I hoped I'd get insight into the question with the company in my story, but I honestly don't know why I don't start it myself; it seems like there's about a five-year window coming up where it'd be pretty profitable. Maybe there's not as much of a market for the product in the real world. Also, as we've established, I'm just not a company-starter, which is a shame as I've decided that's the only way to really get rich.

Well, I thought, I'll encode the numbers x,y as a product of powers of primes 2^x*3^y, just like you would to encode a formula into Gödel numbering. Then I'll have one number that, by the Fundamental Theorem of Arithmetic, uniquely represents some list of numbers. On the other side I'll "just" decode the list from that number.

Then I figured out that the real answer was to put the numbers in a list and pass in the list.

In conceptually related news, you might remember how a year or so ago I was obsessed with configuration interfaces. I even wrote a paper about them. Well, today Anthony Baxter talks about his client-side GUI implementation of a configuration interface.

I almost said Jake's program worked in a Dissociated Press fashion. It doesn't, but I bet that would sound really neat. You could turn a music library into an insane never-ending medley by analyzing the waveforms, finding smooth transitions, and pasting together one-second chunks.

I was going to do a minor NewsBruiser release tonight, but it misbehaves really badly when you install it on Python 2.3, so I'll do it tomorrow.

On the plus side, while seeing if anyone else had duplicated my puny results, I came across the awesome Trading Agent Competition, best described as Core Wars for those intelligent agents we hear so much about that look for the best prices online. In fact, that link deserves to be the star of this entry. Okay, I'll move this paragraph above the diagram so you'll see it. I couldn't find source for the agents, but here's an overview of the strategies employed.

And here's the chess diagram:

+--------+
|33333333|
|33333333|
|23333332|
|11111111|
|11111111|
|11111111|
|00000000|
|00000000|
+--------+

The only thing remotely interesting about that diagram is that it's not symmetrical, because of the positions of the king and queen.

Another chess-mentioning entry coming later today, if I write it.

In this spirit is Brian Fitzpatrick's broken-keyboard.el, which sneaks around your Emacs keybindings breaking one key at a time. It's one for the ages, or at least all ages that have keyboards.

PS: If you know of any cool Wikis, post them in comments.

Suppose you want the glory of Wiki contributorship, but you also crave the greater proportional glory that comes from contributing to a Wiki that lacks thousands of contributors. Check out the Wiki Cookbook, the most promising of the WikiBooks. It has few specific recipes, but as with Wikipedia the potential is there to focus our individual obsessions on a single semicanonical web site. Check out, for instance the Caesar salad recipe--they have the original recipe from Caesar's Bar along with a bastardized modern version. Completeness mania strikes again! Huzzah! A global namespace ensures that variants of recipes will be explored within the context of the larger dish rather than stuck somewhere else. I hope.

My only complaint: I can't find any way to get any of the special Wiki pages (random page, recent changes) for a particular book (ie. the cookbook). There's one big dataset of all the changes made to every WikiBook from which these pages take their orders.

Even so, I would only care in an academic sense, but sometime in the past Wikipedia surpassed the point where the articles were tedious clean-room implementations of preexisting encyclopedia entries. Now they tell me about interesting things like The Fundamental Theorem of Poker (which inspired me to seek out other fundamental theorems) and what the SysRq key is for.

In celebration of the article and the server-crushing load it will no doubt send Wikipedia's way, I declare today WikiDay on News You Can Bruise. I can think of two other Wiki-related things to write about, which should be good enough for a Day.

Kris figured they took down the interactive portion because of 9/11, which makes total sense. Previously, terrorists would only have had to obtain a nuclear device if they wanted to blow up a city. Now, they have to obtain a nuclear device and a map and compass! But what about other methods of horrible destruction, as yet out of human control? Are there convenient, non-terror-enabling online forms that show you how dead you will be? I decided to do some searching and find out.

• Earthquake: You can find out when you felt the tremors from past earthquakes, but that doesn't do you much good. However, see Asteroid Strike below. Update!: If you live in the SF Bay Area, Adam Parrish has a deal for you! Maps and everything!.
• Volcano Eruption: All I could find was this distance calculator to volcanoes in the Pacific Northwest. If you don't mind doing a little web browsing, you can locate volcanoes in your area at this Smithsonian site or this USGS site, which is less comprehensive but has maps and other volcano freebies.
• Tsunami: Nothing I could find.
• Asteroid Strike: Here's what I'm talking about. Takes projectile characteristics and calculates all types of damage: crater ejecta, shock wave, thermal radiation, etc.

  Plus, since an asteroid strike causes an earthquake, you can work out a rough correspondence of impact criteria to Richter scale magnitude and simulate any kind of earthquake that way.

  My only complaint: With the right asteroid you can create an impact crater bigger than the earth. Indeed, you can specify an asteroid much bigger than the earth and it doesn't, as it should, treat the collision as a minor inconvenience to the asteroid.

• Tornado: Well, if you know the windspeed of a particular tornado, you can plug it in here and it will give you an index on the meaningless-except-to-tornado-geeks "Fujita Scale". But it will also give you wiseacre Midwestern comments, a decent proxy for "How dead will you be?" (Eg. "Too late to head for the cellar!")
• Flood: Well, this is pretty interesting. It's not a calculator per se, but a tutorial and script for simulating flooding in a GIS system. The reason I say it's interesting was I've been wondering how to do the same thing on a grand scale: see what the world would look like if the sea level were to rise or fall.
• Godzilla attack: Nothing. Not even a damage table you can roll on.

Well, I think that covers most of the SimCity disasters. Maybe the raw power of a nuclear explosion or an asteroid strike are what let you make a calculator that doesn't get hung up on the topography of the landscape or the size of a particular volcano.

For programmers about to enter the real world (or otherwise about to take responsibility for software projects), it's a good technical companion to the more lifestyle-oriented Things I Wish I'd Been Told. Along with How to Build a Fort That Girls will Visit, these two documents form the essential post-college geek starter kit trilogy. (Thanks to Paul for looking in the Internet Archive when I was too lazy.)

[0] Supposedly you learn to work in a team in college, but it doesn't work. Working on an open source project is better preparation. You could pick up pair programming in college, if you went in determined to learn that way.

Update: Your Earth FAQ has the answer.

Coming soon, if you're interested: easily-derivable-from-first-principles secrets of how to get the most bang for your Freshmeat announcement buck.

New comment features: you can automatically shut down comments on old entries, and you can turn on comments for specific entries instead of for the whole site.

Joe says that there was a glowing review of NewsBruiser in the UK magazine Linux Format. Since they don't do aspiring weblog software authors the favor of putting their glowing reviews online, Joe is sending me the magazine and I'm gonna type it up. And as long as we're discussing Linux magazines with weird names, how about India's Linux For You?

PS: stack's day job project is out.

On the other end of the What Would Be Nice supply chain, It Would Be Nice If Freshmeat's XML-RPC interface could be used to grab the newly added/updated projects listed on Freshmeat instead of only being usable for submitting new versions of projects. This would make my Software Roundup rounds a lot easier.

PS: Hey, Past Leonard, you saw The Two Towers with your family and Return of the King with the rest of the technical team from the presidential campaign where you work.

Past Leonard: Yeah, right. Stop calling me here!

Anyway, I see that Saruman has not been idle since the War of the Ring: his love of perverting all that is good and natural is surely the driving force behind Perlthon. Look at the author's name! "Manura"? An obvious anagram.

Future Leonard: Hey, Present Leonard, you will watch the extended DVD version of LOTR to pass the time on your colony ship's voyage to Sirius.

Stop wasting my time!

This Program Lets You Write Letters is crucial to the use of or coordinates with a number of full blown applications. Ok, I only know of, like, two. Well, there was that screen play thing, but that went straight out the window.

+-----------------------+   +---------------------------------------+
| Wrote wacky flowchart |-->| Incorporated suggestions into version |
+-----------------------+   | 2 of wacky flowchart                  | 
                            +---------------------------------------+
                                              |
                                              v
                    +---------------------------------------------+
  +-----------------| Revealed 6 new flowcharts at company picnic |
  |                 +---------------------------------------------+
  v
+------------------------------------------------------------+
| Came up with trendy Cabbage Patch Dolls flowchart, big hit |
+------------------------------------------------------------+
                         |
                         v
         +------------------------------------+
   +-----| Cabbage Patch Doll market collapse |
   |     +------------------------------------+             
   |
   |   +---------+    +---------------------------------------------+
   +-->| Despair |--->| Slashed wrists with own flowchart templates |
       +---------+    +---------------------------------------------+
                                           |
                                           v
                                     +-------------+
                                     | NO PROBLEM! |
                                     +-------------+

As flowcharts became obsolete, a few of the surviving wacky flowchart writers managed to make the transition to wacky UML diagrams, but their audience ever shrinks and becomes more technical. Some of the old stuff is really good, though. For instance, today on Crooked Timber they posted a funny, multi-page Meta-Cosmology Theory Flowchart. That was actually the whole point of this entry, but it's sort of overshadowed by the graphic. HEY! LOOK OVER HERE! There, that should do it.

Even today, flowcharts are sometimes funny, but it's usually unintentional.

powerpoint presentation in wisdom tooth surgery

Anyway, that's the software side of things, and on the hardware side there's the collection of old scanned computer manuals, truly a labor of love. A random skim through the manuals reveals, eg. the Hayes Stack Chronograph, which is an external system clock. I thought external CD-ROM drives were old school, but it turns out that that school was built on the ruins of an even older school.

[Wait, under what circumstances did you swear revenge on Ned Batchelder's weblog? -Ed. Such questions are not productive in these times of transition, as we seek to reopen friendly relations with Ned Batchelder's weblog. How come I never heard about this? Is this one of those delusional things you make up as you type to make an entry more interesting? -Ed. You're a spy for Ned Batchelder's weblog, aren't you? Fie! This means war!]

OK, here you go. Resign Patterns, including "Simpleton" and "Flypaper". From Ned Batchelder's weblog.

Through our service, you compose and send your lover a normal-looking "Greeting Card" saying "I Love you" or a similar message. Because the e-mail appears to be a regular greeting card, the recipient will open the e-card and LoverSpy will be automatically and silently installed!

Ha ha! Sucker! Uh, dear.

Throughout my plugin adventure I've been bothered by Robert X. Cringeley's argument against refactoring. Was I refactoring to improve NewsBruiser or just futzing around with the code to give myself something to do? I kept reminding myself that I was doing this to make it easier to introduce new features that other people had written. Now that it's done, new features are easier to write and don't clutter up the main code. Lousy programming conscience! Take that, Robert X. Cringely!

I shouldn't complain about the over-ambitious goal, since a trivial corollary is to use the Wiki to annotate Cryptonomicon as well. Quicksilver sounds more annotation-worthy, though, like a cross between a history of science and those weird alternate-history and shadow-history books where Benjamin Franklin teams up with a young Antoine Lavoisier to stop an underground cabal of animal magnetists from Venus. Hmm, that's actually not a bad--glaaaah! Damn you, Thomas Paine, and your fiendish future-tech remote control garotte!

[0]Actually, Patrick Tufts probably set it up on his behalf.

I keep going to the Five Card Nancy CGI, building some stuff, then getting bored and quitting. It's like a game of FreeCiv, but much quicker. Here are my best ones so far: Short attention span Nancy, hyperactive Nancy.

Hi. I've got Internet access at home now.

root is not in the sudoers file. This incident will be reported.

Consider the archetypal replicator command: "Tea. Earl Grey. Hot." Captain Picard never orders anything else from the replicator, yet either there's no way for him to get the replicator to know that when he wants tea he wants hot Earl Grey; or there is such a way, he can't figure out how to set it up, and he's too proud to ask anyone for help.

What's more, the captain's utterance implies that if he just asked for "Tea. Earl Grey.", either the replicator would give him iced Earl Grey or the ship's computer would ask "At what temperature do you want the Earl Grey?" like a text adventure parser. How come they had time to do a molecular scan of a cup of Earl Grey but not time to put into the 24th-century equivalent of Cyc that Earl Grey is the sort of tea you drink hot?

This is not rocket science; it's the sort of thing that contemporary programmers use as fanciful examples in their EuroPython talks. "Let's take a hypothetical Person, Jean-Luc. Jean-Luc likes Tea, so he sets that as his default Drink, but there are many different kinds of Tea, so he can set a default for Tea as well..." It wouldn't be hard to do the interface either; after you got your elaborately specified tea you could say "Bookmark. Tea." and thereafter it would just be "Tea." Except for the inevitable wacky malfunction, where the attempt to bookmark tea would give you a tea-colored bookmark, you would be fine.

Next time: I tackle the question of what happened to all the industrial designers between Enterprise and TOS.

[0] Also the "Kobayashi Maru" way.

Unfortunately, both of the authors were the ones who got stuck with fixing this problem. Also, because one of them is a big whiner, he kept complaining that `we knew this all along ­-- something really stank for iterations and now I'm stuck with fixing it.'

--Recognizing and Responding to 'Bad Smells' in Extreme Programming (from Jeremy Hylton)

There's also a Chinese Restaurant Name Generator which comes up with better names than the one I wrote for my PyCon paper, but it's some weird Javascript thing that only works once, like the guy's variety act in that Warner Brothers cartoon.

One great use for Downhill is finding new weblogs. That's also a great use for writing Downhill and having people link to it. Once the buzz dies down, an Eater-style postmortem where I talk about new weblogs I found (also coming soon: the not-really-that-shocking results of an ad hoc experiment I ran against the readers of pyblagg). I hope the announcement that I'll do a postmortem won't affect the way people link to me.

Then there's the awesome ezPyCrypto, a GPLed (so beware)library which makes it incredibly easy to do crypto in Python. Encrypt everything, just because you can! Encrypt your own source code in some sort of obfuscation/trusted computing thing! Encrypt the very name of your program! Encrypt your cat!

Q: Where do they keep the pirate Constitution?
A: In the National Arrrrchives.

Q: How do they protect the pirate Constitution from deterioration?
A: Arrrrgon.

Q: What's a pirate's favorite web portal?
A: Yoho!.

And two of mine:

Q: What was an evolutionary precursor to the modern pirate?
A: yo-ho-homo erectus.

Q: What is a pirate's favorite physical constant?
A: Plank's Constant.

(Apparently I can't remember any of Riana's pirate jokes.)

Also, when I came in, Zack mistook me for Cory Doctorow. Cory did not show up, leaving the nagging question in the back of my mind: could I have succesfully impersonated Cory Doctorow for the duration of the party? Maybe there's a story on the possibility.

I showed Seth and Nick Moffitt my new game, and they laughed uproariously. Hopefully I'll finish the game tomorrow, and then you too can laugh uproariously or not get it.

1. 20030512: N. Korea claims new drink for techies
2. 20030516: N. Korea training hackers, Seoul says

More similar word lists at wordlist.sourceforge.net, including a list with part-of-speech data which made me waste time this evening writing an eater that replaces a word with another word of the same part of speech. It's not up yet because I'm not yet sure how good it is.

cvs admin -m "#.##:`cat /path/to/new/commit/message`" path/to/affected/file

Where #.## is the version number for the file.

If you need to retroactively change a CVS commit message for a commit that affected more than one file, you are doomed to repeat that action for every file. A little utility script would be useful here.

Before you write code, you're supposed to write unit tests. It turns out I can bring myself to do this most of the time, but I do it grudgingly because I don't like writing the infrastructure for a new type of test. The way I generally do it is to write the tests but leave the expected value for the assertions blank (unless the expected value is really easy to figure out, or is actually defined and not just the result of applying some transform to some data I just made up). Then I write the code and all the assertions fail. I use the code I wrote to figure out what the asserted values should be, fixing bugs as I do, and paste in the final, correct values for later use.

Where unit tests are really great is when you find a bug in your code. If you've put in the work to write that infrastructure for that sort of bug, it's easy to record your finding of the bug in a way that will automatically notify you if it ever shows up again. This is lesson one: unit tests give you a place to codify the results of the debugging process. By putting in some incremental effort you can ensure that you're notified should the bug you're fixing ever recur. Usually it doesn't recur, but your real worry is not the bug recurring; it's the bug recurring without your knowledge. With unit tests, you know.

Sometimes when I refactor code I worry that some portion of the code I never exercise has broken due to refactoring. This is the worst thing about Python: unless you're using Leo (which I'm still not, despite my earlier touting of it), when you refactor the code the indentation changes, and when that happens you have to redo the indentation because the automatic indenter never works right, and all sorts of problems can crop up.[0] Unit tests find the problems, and if there are any you fix them and that's that. Lesson two: with unit tests, instead of worrying about code you broke while refactoring, your mind is free to worry about less rational things, like bloodthirsty packs of feral hamsters roaming the streets.

I have unit tests for all my Eaters, and some decent coverage for the HTML parser, but nothing yet for the user interface (even though I wrote the user interface to be drivable from Python code) or the thing that grabs URLs. This is because of lesson three: the closer a piece of code is to an actual user, a piece of hardware, or data from the outside world, the more aggravating it is to write unit tests for it. I am convinced this demonstrates a close affinity between unit tests and Lisp.

The big wrapper lesson is that unit tests are a way of creating a grammar with which to describe the behavior of the system so that you can offload to the computer the task of checking that behavior. Unit tests are easier to write when the elements of the grammar have simple APIs that you control, and harder to write when the elements are complex things with outside dependencies, like URLs and mouse movements. Even if you only write unit tests for the simple stuff, it means your agonizing over the more complicated stuff will be untroubled by the suspicion that the problem is something lower-level and simple (of course, then it turns out to actually be something lower-level and simple, and the code says "Gotcha!" like Lucky Ducky and struts off, but I can't help you there).

[0] Eventually I'm going to write my own autoindenting Emacs hook which operates on the very simple principle of moving the cursor line to where autoindent takes it, then moving every child line left or right the same number of spaces. I don't know why other indenters have these fancy algorithms that don't work, when that algorithm works fine. With my algorithm you have to go down arbitrarily many lines before moving the first line, to see how many lines are underneath the first one, then go back up to the top and go through all those lines again, but on the plus side it actually works.

I have not looked into whether or not this actually exists, because the longer I put it off, the longer I can pretend it exists in some dusty section of the HTML spec. I assume it doesn't exist because I can already feel the people who are in charge of adding things to the HTML spec breathing down my neck, telling me to use one user-visible control per hyperlink destination. Don't make me use Javascript!

PS: PAP is the second game I know of in which feature creep is a bogeyman.

1. All my troubles seemed so far away.
2. I went to see Seth.

At Seth's place I showed him my PyCon presentation. Then we went through my code. Both presentation and code were well-received. Seth complimented my use of object-oriented design, and we came up with some minor improvements to I Want Options (for instance, IntegerOption should be called CountingNumberOption).

Then Seth showed me the features of Movable Type he likes. I have all of them except for user accounts and permissions (which isn't going to happen anytime soon) and a clever entry list in edit mode. I also like the category list in edit mode.

I passed along to Seth the copy of The Bug, about which more later. I have to pack for DC.

It is possible... that the recent trend in congress [nb. he means this] will extend from fries and toast to film (France's other great export to the States); in which case, you should get on the bandwagon early and switch your "French film" references to "Freedom film" references. To facilitate this, I wasted the following time:

http://adamparrish.warped.com/cgi/french-to-freedom.pl

(I suggest trying it with the following URL: http://www.lambweston.com/company.asp?navid=29&chain=7,29 ... tee hee)

I think this proves that the marketplace of ideas has disintermediated our elected officials, whom we used to pay to think up this sort of picayune gesture of international chumship or saber-rattling. Perhaps someone should tell them... but they're having so much fun!

Coming soon: this.

This happened with the "kitchen sink" Mozilla example I linked to earlier, and I don't think that's even really an Easter egg; it's an interesting example of what you can do with Mozilla. It just looks like an Easter egg because it's not boring like the other Mozilla examples. Other cases of this I can think of: there was a rather lame Easter egg in the xclock application of Red Hat 4.2 or something, which caused a similar rash of complaints. A later version of Red Hat offered "Redneck" as a localization option, and I'm sure the same people complained about that. [0]

This is the point at which I cast aspersions on the amount of code the people who complain have committed to open source projects, but that's an ad hominem argument which doesn't address their actual argument, which is that open source programmers should optimize their actions to devote their energies to the next most important thing. Sure, if open source programmers were perfect utilitarians (and their idea of what is important were the same as yours), that's what they'd do. But they're not. They have the same desires as real people, and they have an additional weakness for cool-looking things like sinks implemented in XHTML and Javascript, and you have no right to try to allocate other people's voluntary labor, blah blah blah.

At the same time, I don't think putting Easter eggs into an open source application fulfills any psychological need. Most of the time I've seen this it's been a parody of feature catch-up ("Windows has Easter eggs, Red Hat 4.2 must have them too!"), or of an application's reknownedly robust feature set ("{Mozilla,Emacs,Nethack} has everything but the kitchen sink!").

Proprietary software programmers create Easter eggs are because it gives them an outlet for creativity they feel they're not expressing, or because it's the only way they can get their name into the software they wrote. If an open source programmer feels like they have leftover creativity, they can switch projects or work on a side project for a while. And of course open source programmers get their names all over their software. So the same dynamics don't apply.

I personally don't put Easter eggs in my code (except for my text adventures, which I sort of see as a command line interface to an enormous number of Easter eggs). In general, I'd rather implement the Easter egg idea as a separate program, or just write a NYCB entry about it. In lieu of Easter eggs, I give my projects silly names, and sometimes I put in silly comments where only other programmers will see them.

The silly project names make people who like catchy names remember the project, and where my software is in danger of being used by humorless people who think "catchy name = crappy software", there's invariably a marketing department conveniently placed between me and them, who will create boring aliases for my projects no matter what I call them.

I have no data to back this up, but I imagine most open source programmers who have the urge to Easter egg fritter away that energy in separate, more whimsical applications. The Unix philosophy, after all, is no friend to Easter eggs, and thanks to the Internet, your cool little program doesn't need to piggyback on your big boring program with an established delivery mechanism.

[0] Adding "Redneck" or some other bastardized version of your native language as a supported language is actually useful when you're adding localization support to an application, because it allows developers who only speak one language to use a "foreign" localization bundle that they can understand. That's probably not why Red Hat put it in, but I'm just saying.

PS: I left the Mozilla sink running in another tab while I wrote this entry, and when I switched back to the tab and noticed this, I momentarily panicked.

PPS: I forgot to mention that the whole reason I wrote this entry was to mention the following Easter Egg comment in Java Cracklib; it looks to be taken from the original C source:

 /* 
  * alternative use for a boomerang, number 1: a standard throwing 
  * boomerang is an ideal thing to use to tuck the sheets under 
  * the mattress when making your bed. The streamlined shape of 
  * the boomerang allows it to slip easily 'twixt mattress and 
  * bedframe, and it's curve makes it very easy to hook sheets 
  * into the gap.
  */

The point of this little stroll down memory lane is to point you to an essay Rohit Khare wrote, called Who Killed Gopher? Gopher is still being developed, at a stately pre-bubble pace. It's like 1993, only with Subversion.

(Found via referer logs thanks to Mike's tendency to mention GTV! and Degeneracy everywhere; thanks, Mike!)

At the very least, Leo is the outline editor I've been looking for, and I can pretend to be one of those people, like Joseph Allen, who uses an editor named after them. At most, I will become one of those people who is a fanatic about Leo on wikis.

Note: I had to recompile Python to get Leo to work, because my preinstalled Python defined Py_UNICODE as wchar_t even though wchar_t on my system is 32 bits, in direct contradiction to what The Python/C API Reference Manual says. I mean, geez.

Cracklib Rosie, get on board...

• Bash: echo $USER > /dev/murder
• Python: import murder
• Perl: use Death::Murder;
• E: Alice <- murder
• Basic: 10 MURDER MEMORY
• Lisp: (murder 'lesser_languages)
• C: #import "murder.h"
• Objective-C: [NSProgrammer murder]
• O'Reilly: Murder: The Definitive Guide

Update: Scott James Remnant adds Murder In A Nutshell

Now, Mark Wooding of Cambridgeshire has written Quine, a C program capable of transforming any C program into a quine. This makes me happy because I don't think I've ever written any quines, except in made-up languages in which such things were trivial. Now there's a quine transformation program and I can scoff and say "Oh, quine is a solved problem."

(Found via the humbling sweetcode, which for every "Well, that's out of the way now" generates ten "Man, I wish I {had thought of that, could have written} that"s.)

(Spread is not really a buzzword, but it deserves to be.)

Many say, "MS Windows has won, and the Linux Desktop is dead!" Now the Linux Desktop is UNDEAD, and it has risen from its crypt in search of virgin blood.

(From LinuxOrbit via LWN)

Update: Aaron reveals that he too has written an RSS-to-email aggregator. "I want to make it a hosted service, but I haven't done it," says Aaron.

Accusations Fly In Clickolinkostan Parliament As Cabinet Resigns Over showtime Scandal

Opposition Leader: "limit Was The Last Straw"

Disgraced PM Neakums Accepts Responsibility For Covert tz Modification

[0] May not actually be the cutest little thing you've ever seen.

obj.method( arg1, arg2 );

What the hell is that? A parenthesis provides horizontal space just fine without adding even more space before or after it.

Oh, uh, the program is called 'pv', and it's really nice. Our version would have been called 'pwc' or something like that, and another reason it's just as well we didn't write it is that if we had, the cloned version would be called 'gpwc'.

Because of a programming error the distribution of cospeciation may be slightly biased towards rejecting the null hypothesis of cospeciation if the host and parasite phylogenies are small.

That's one of those technobabble sentences that people quote to point out that something makes no sense to them, but that actually does make sense to me, so I guess I'm quoting it for the novelty of such a thing happening.

<dumont> CAP-tain SPAUL-ding!

The conceit of an IRC infobot is that it hangs out on an IRC channel as a user, dispensing definitions of terms and engaging in limited social interaction with the human users. Dumont knows nearly 100,000 facts, including such old standbys as the entire Jargon File[0] and airport and area codes. She He also contains various handy features such as the ability to do Google searches and domain lookups, not to mention clickolinko the automatic weblog. Dumont was written by Walter M. Gibbs.

Random example of dumont interaction (thanks, Pete):

<pedro> mola mola?
<Dumont> mola mola is The Mola mola (ocean sunfish) is the world's largest known bony fish. At least one estimate over 3000 lb. has been recorded and fish reaching 3m from fin tip to fin tip have been seen. It is found in all oceans in tropical and temperate climes. It eats jellyfish and small fishes and algae. In the eastern Pacific, Mola mola is normally found from B.C. to South America, although in El Nino events it has been recorded as far north as Alaska.

[0] I used "entire" there to make it sound more impressive, but there's really nothing that interesting about loading an ASCII file into a computer program; the interest lies entirely in which file you choose. The metaphor is taken from reference to human memory, where memorizing the entire Jargon File would be an actual feat. I don't really know where this footnote is going, so... aaah! It's fallen off the edge of the Internet!

"It's actually pretty fun to be really tall in the Far East, because you feel a little like Godzilla." -- Robert Bennefield, our 7'-ish director of Ops.

There's software that will determine whether two pieces of code are similar, for purposes of detecting plagiarism in class assignments. But let's say you had software that could detect 'similarity' between two pieces of software. You could also run it on two different parts of the same program to automatically find places where you could benefit from factorization. Shazam! It's like a reduce-to-the-halting-problem proof, only it gives you something good instead of something bad!

Of course, 'similar' for purposes of detecting plagiarism is a much easier concept than 'similar' for purposes of detecting factorability, so you probably can't use existing software similarity software for this purpose. But software to do that would be a very useful tool, especially if you're like me and find it boring to inspect code for factorability.

"Math documents, for example, should run through a math compiler that not only verified that the documents can run but that they are true..."

Nuh-uh-uh.

The link going around CollabNet is this graph of OpenOffice compile dependencies. I think it would make a good technology graph for FreeCiv, although you'd have to get rid of that big bottleneck. Every sentence in this paragraph containes at least one BiCapitalized word.