News You Can Bruise

Since listening to KUSC in college I've been a fan of the old BBC radio program(me) My Word!, an ur-quiz show with a focus on chin-stroking erudition, shameless bluffing when erudition fails, and cornball shaggy dog stories. About ten years ago my fandom took a big hit when the BBC stopped pouring decades-old My Word! reruns down whatever transcontinental pipe eventually got it broadcast on American radio stations' streaming websites. But recently I discovered a large cache of episodes uploaded to the Internet Archive in 2020, including a bunch of episodes I'd never heard. Jackpot!

I really wanted to get this cache turned into an RSS feed so I can listen to episodes alongside my podcasts. Kevan's Fourble service can do this pretty easily, and in fact it already has, but what I'd really really like is an RSS feed that incorporates the information about broadcast dates and shaggy dog stories found in this particular item's carefully written description. That will dramatically improve the usability of the "podcast" and allow me to listen to the episodes in rough chronological order, rather than alphabetically according to the first vocabulary word lobbed at the panelists.

This is, in fact, a job for The Syndication Automat, a project I created in 2004, the semi-early days of RSS. Back then it was sometimes necessary to employ vigilante justice to make RSS feeds for websites that didn't have them. This was actually the original use case for Beautiful Soup!

Of course, hard times soon struck the Automat as every website got its own RSS feed, RSS feeds themselves were ditched in favor of Twitter and Facebook, and then Twitter and Facebook melted down, leaving us with nothing. (I'm extrapolating a little here.) 2009 was the last time any of the Automat's old feeds were updated. But podcasts still stand, the cockroaches of syndication, so it makes perfect sense to bring back the Automat one more time to host The Doubly-Unofficial, Partially Chronological "My Word!" Podcast Feed. Painstakingly hand-crafted by a script I painstakingly hand-crafted to deal with tons of edge cases like "two shows that use the same vocabulary word" and "shows where the filename doesn't precisely match the vocabulary word" and "shows where the general era of the show is known but not the exact broadcast date". I took care of all that stuff; all you have to do is listen.

If you just want to make a podcast out of the MP3 files in an Internet Archive item, and not do any other processing, you can use my very tidy, edge-case-free Python script, which depends on the modules internetarchive, feedgen, and pytz

from datetime import datetime
from feedgen.feed import FeedGenerator
from internetarchive import get_item
import pytz
import sys
import time

def utc(dt):
    return dt.replace(tzinfo=pytz.utc)

class IACollectionFeed(object):
    def __init__(self, ia_item, destination_url):
        self.item = self.fetch_item(ia_item)
        self.feed = FeedGenerator()
        self.feed.link(href=destination_url)
        self.feed.description(self.item.metadata['description'])
        self.feed.title(self.item.metadata['title'])
        for file in self.item.get_files():
            if file.format != 'VBR MP3':
                continue
            self.add_entry(file)

    def fetch_item(self, ia_item):
        return get_item(ia_item)
                 
    def add_entry(self, file):
        entry = self.feed.add_entry(order='append')
        entry.id(file.url)
        entry.title(file.metadata['name'])
        mtime = utc(datetime.fromtimestamp(int(file.metadata['mtime'])))
        entry.updated(mtime)
        entry.enclosure(file.url, str(file.size), "audio/mpeg")
        return entry
        
    def __str__(self):
        return self.feed.rss_str(pretty=True).decode("utf8")

           
if __name__ == '__main__':
    print(IACollectionFeed(sys.argv[1], sys.argv[2]))

To generate a fast and cheap version of my DUPCMW!PF, I'd invoke the script with this command line:

$ python roughdraft.py bbcmyword https://www.crummy.com/automat/feeds/myword.xml

While producing this post I discovered that not only is there another, smaller, differently organized collection on the Internet Archive, but there's a significantly larger (but less well described) archive on RadioEchoes, which also has an even bigger archive of My Word!'s inevitable but lesser companion, My Music!.

To celebrate the anniversary of the first moon landing, I packaged up a project I came up with a while back: Apo11o ll, a generative piece that performs Queneau assembly on the Apollo 11 transcripts (from The Apollo 11 Flight Journal and The Apollo 11 Surface Journal).

Duke: Rog. [Long pause.]

Armstrong: That's one small step for (a) man; one giant leap for mankind.

McCandless: Roger, 11. I have a T13 update for you. AOS Tananarive at 37:04, Simplex Alpha. Readback. If you want to go that way, crank it up, and then you can drive it around and look where you want. Over. 11, this is Houston. And we copy the V_I.

Aldrin: Does it look to you like the [garble] the right way? Yes, they were working out - this elaborate scheme.

Collins: Unless you'd rather sleep up top, Buzz; I like - you guys ought to get a good night's sleep, going in that damn LM - How about - which would you prefer? I say the leak check is complete, and I'm proceeding with opening the hatch dump valve.

Aldrin: That enough?

McCandless: Apollo 11, this is Houston at 1 minute. Over.

First Mashteroids and now this? How am I doing all this Queneau space-magic? The answer is simple: Olipy, my library for artistic text generation (focusing on Queneau assembly, because it's the best). Check it out of Github and you'll have everything you need to create home versions of many of my works. It's like my own personal Boîte-en-valise! Want to create something new? Just grab some data and feed it to an Assembler class.

Last night I'd finally had enough with all of my Twitter bots not working due to sending POST requests to a resource that was 401 Gone. The one I really need to keep going is Frances Daily, and that one's on break right now because the planner page for June 1988 was unfortunately missing. But we're running out of June, so I fixed it.

To do that I had to fix Sycorax, the way-too-advanced piece of software that enacted an elaborate running commentary during the serialization of Constellation Games, a commentary that about eighty people saw. Since I'm pretty sure I'm the only person using Sycorax, I've decided to stop doing a tarballed release every time I change something, and just put the code up on Github.

Robot roll call!

• @FrancesDaily (Pan left.)
• @EveryBrendan (I'm not ready!)
• @DadaLimericks (What a cool guy.)
• @CrowwwwwwwwdBoardGames (I'm different! Specifically, I was rewritten to use Kicktraq's RSS feed instead of attempting to scrape every Kickstarter project page!)

The robot in the shop is @RoyPostcards, which I'll fix around the same time I get some more postcards ready to put up.

Update: @CrowdBoardGames prayed for a friend, and he came! His name is Timmy!

When I meet someone and they ask what I "do", I gauge their level of tech knowledge to determine how much detail to go into. Have they heard of Ubuntu? (I work for Canonical.) Have they heard of Linux? (I work for a company that puts out a Linux distribution.) Maybe at they've used a computer? (I write software.) I do this rather than just going straight to "I write software" (from which I've only had to back off once), because when, as sometimes happens, I'm asked this question by an Ubuntu user, the question of "what do you do" turns from a social nicety into a real conversation. And it's really nice to get appreciation from someone just because of where you work.

This week, Canonical employees are writing about how their day jobs interact with the open source community. For some people, like the desktop team, the connection is pretty obvious. For me it's a little more subtle. I give open source developers control over their accumulated software artifacts, and I give them back some time that they'd otherwise spend doing boring things.

The projects I work on are open source, but for the most part that just means I'm doing my work in public. My main project, lazr.restful, is used a fair amount within Canonical, but not much outside of it. So saying "hey, I write open source software" is cheating. What's really going on?

Launchpad is going on. Launchpad is a website for doing software development. When you work on a project, you're generating document-type artifacts (stuff that's part of your completed project) and process-type artifacts (stuff detailing the actions you took to get to the completed project). Document-type artifacts need to be version-controlled, and process-type artifacts need to be shepherded through a workflow.

Such sites are nothing new. For almost my whole showing-up-for-work-every-day career, I've been working on one of these sites: before Canonical, it was CollabNet. What's new about my work on Launchpad is that with the web service I give you (the developers using the site) direct access to your process-type artifacts. This benefits you in two ways:

1. Automation

   Not all the code you write goes into your project. Some of it is utility code you use to maintain the project itself.

   For instance, I wrote a little shell script to run whenever I do a new release of lazr.restful. Make a tarball, sign it with my GPG key, create a new release series on Launchpad (if appropriate), upload the tarball and GPG key as a release for that series, upload the release to PyPI, use bzr tag to tag the current revision with the release number. All I have to do is type my GPG passphrase.

   Well, you know how to tarball and GPG-sign with shell commands, and bzr tag is a shell command, but PyPI and launchpad are websites, and to tell them about a new release requires a web service client. The setup.py script is the web service client for PyPI, and the web service client for Launchpad is launchpadlib, the code I wrote. If Launchpad's web service didn't publish release series and release files as resources, I'd have to do that step by hand.

   And just as there's code you write but don't release, there's code you never write at all—you execute it yourself, over and over and over, using a web browser. You're pushing process-type artifacts—bugs, questions, code reviews—through some kind of workflow.

   A tool can provide the framework for a workflow, but the work won't flow on its own. Someone has to operate the pump, and that can get boring. Some rules are easy to describe algorithmically but time-consuming to execute. Like "If a commit message mentions that this commit fixes a bug, go ahead and mark the bug fixed." Or "When making a new milestone, move in all the bugs that didn't get addressed in the last milestone."

   As the developers of the Launchpad website, we could force everyone into a specific workflow and optimize for that workflow. Or we could try to create a tool that's so general you could use the website to define arbitrary rules like the commit-message rule. Or, we can publish the software development artifacts as HTTP resources and get out of the way.

   When I think about the Launchpad web service, I mostly think about allowing developers to scratch their own itches and automate the automatible parts of their jobs.

   (BTW, the Launchpad portion of my release script is in lp:launchpadlib/contrib/upload_release_tarball.py.)

2. Integration

   This is the code that gets published—applications where communication with Launchpad is a feature. For instance, if an application crashes while you're using the Ubuntu desktop, Apport pops up and gives you a chance to report a bug against that application. Apport reports those bugs through the Launchpad web service. There are also programs like Ground Control, which let you manage your software development artifacts from your desktop instead of your web browser. These all bring Launchpad to your desktop.

   You're just using these applications, you didn't write them yourself, so you don't know that the Launchpad web service is involved at all. But all these products—better bug reports, easier administration—feed data and saved time back into the open source ecosystem.

And for me it is fundamentally about saving people time. There's a story from the development of the Macintosh that has stuck with me for years. Steve Jobs is doing what he does best: berating developers. In this case he's telling them to make the boot time faster.

Well, let's say you can shave 10 seconds off of the boot time. Multiply that by five million users and thats 50 million seconds, every single day. Over a year, that's probably dozens of lifetimes. So if you make it boot ten seconds faster, you've saved a dozen lives. That's really worth it, don't you think?

I don't have fifty million users, but I'm also not in the business of saving each of my users ten seconds. I want to save you three hours, or a week would be better. That's why I do what I do—you can see this thread running through every job I've had, both my nonfiction books, and my most successful open source projects. Check the intro to Ruby Cookbook: I just realized I wrote the same damn thing in there.

This book is meant to save you time. Time, as they say, is money, but a span of time is also a piece of your life. Our lives are better spent creating new things than fighting our own errors, or trying to solve problems that have already been solved. We present this book in the hope that the time it saves, distributed across all its readers, will greatly outweigh the time we spent creating it.

Beautiful Soup is showing its age, but I don't think it's totally crazy to estimate that it's saved thirty or forty person-years since 2004. Someday I'll die, and I don't want the main thing people say about me to be "The time he saved other people was a greater-than-unity multiple of his own lifespan." But that is an important part of my personal philosophy. It's Tim O'Reilly's "Create more value than you capture" applied to time.

Version 2 of Dada Chess features textual representations of the games for those with chessless browsers, highlighting of the piece that just moved, and improved stalemate rules so the game ends instead of the kings chasing each other all over the board.

I also heard back from the original ChessPy author, who for complex reasons invited me to make my bug fixes public by forking the project. So here you go. It's also got unit tests for the stuff I changed. It's really easy to use, and recommended if you have some Dada Chess-like project that needs to run simulated chess games that don't require an AI.

It's no secret that I admire Marcel Duchamp. The mad genius who cackled "I'll destroy them all!" from his bell tower, and then went ahead and destroyed them all. But only today did I create the ultimate tribute to Duchamp: Dada Chess! The latest addition to the crummy.com family of dada toys is an infinite series of chess games in which both players play at random.

I thought this project would take me days to realize, but thanks to Will McGugan's great Python chess library it only took a few hours. I did find a few bugs w/r/t what the chess library considers to be "check" and "checkmate", but I fixed them and sent in a patch.

While I was at it I fixed Dada Maps and Spurious, whose bits had rotted.

We've taken the code that powers the Launchpad web service and moved it into standalone server and client libraries. This is still what I'm working on, so it'll get more useful soon.

In other news, I've started using the word "blog". Argh.

Update: Strange that it took me a day to realize this, but it means that for the first time in about seven years my job is to work on open source software.

I wrote a tiny script that runs my del.icio.us tags through the Eater of Meaning's WordEndingEater to get different words, and prints links to the tag pages for those words. Sometimes these tags don't go anywhere, like "carpets" or "antedates", but sometimes they're pretty interesting, like "macro" (wonderfully ambiguous) and "twenty". It's fun--until you try it yourself!

No packaging; I'll just put it inline for those interested. You need the Eater, of course, and Beautiful Soup (or one of those fancy del.icio.us-Python interfaces with all the XML dependencies).

#!/usr/bin/python
#Del Eat
#by Leonard

import urllib2
import os
import sys
from BeautifulSoup import BeautifulSoup
from eater import WordEndingEater

if len(sys.argv) != 4:
    print 'Usage: %s [username] [password] [path to Eater prefix file]' \
          % sys.argv[0]
    sys.exit(1)

username, password, path = sys.argv[1:]
if not os.path.exists(path):
    print 'No such Eater prefix file: %s' % path
    sys.exit(1)

authinfo = urllib2.HTTPBasicAuthHandler()
authinfo.add_password('del.icio.us API', 'http://del.icio.us', 
                      username, password)
opener = urllib2.build_opener(authinfo)
xml = BeautifulSoup(opener.open('http://del.icio.us/api/tags/get').read())

eater = WordEndingEater(path)
print '<ul>'
for tag in xml('tag'):
    eaten = eater.eatWord(tag['tag'], 'text')
    print '<li><a href="http://del.icio.us/tag/%s/">%s</a></li>' % (eaten,
                                                                    eaten)
print '</ul>'

I kept wondering about the suitability of Bayesian textual analysis to such-and-such a problem domain. Would it magically solve the problem as it does with spam, or would it be like trying to evolve an artificial intelligence by rating randomly generated sentences? Eventually I decided that the time had come for action! So I hired this guy to write a really simple generalized Bayesian rating application which you could use for just long enough to see if the problem was tractable. Yes, I don't just sit around--I make others do my bidding! But then he subcontracted it to a company in the Phillippines, and they went out of business, and their contracts were picked up by a floating libertarian utopia on a raft built out of abandoned oil drums, so make a long story short I ended up doing it myself for about half the price.

It's called The Bayes Motel and it does a pretty good job. By which I mean I wrote applications using it for a domain where Bayesian rating obviously works, and one where it almost works, and... it works and it almost works. TBM also does something I've been wanting for a while without realizing it: different tokens are displayed with different colors, to help you visualize how a piece of text got a certain Bayesian score. Pretty slick! And inefficient. Oh well, it's not supposed to be a big heavy-duty application anyway, just a prototyping tool. I hope you find it useful.

If I were a real hacker I would forget this Bayesian nonsense and do a Python implementation of Fast and accurate text classification via multiple linear discriminant projections. Even in accomplishment I feel guilty!

Big ole weblog entry on recommendation systems mentions the Ultra Gleeper, and Paolo Massa, whose paper I referenced in the UG paper, has also noticed it. I find a lot of good weblogs to subscribe to from backlinks and being del.icio.used.

I have from the beginning felt that my paper is a much better contribution to the field of computer science than the Ultra Gleeper itself. The Gleeper, like all flesh, is frail and has shortcomings, but the paper exists in the realm of ideas, and solves all the damn problems and crowns me king of the world already.

Regrettably I have used this fact as an excuse to not work on making the Ultra Gleeper easier to install and use. I am a mess, I have writing assignments and even despite that I'm lazy and don't do work (this is also the reason I have not responded to your email). Also I beat myself up about the absolute amount of time I waste, even though by relative standards I'm a pretty productive guy.

Oh yeah, as long as I'm touting accomplishments that just highlight my lack of follow-through. blogs.gnome.org has started using NewsBruiser for their user weblogs, and they say nice things about me that I don't think are deserved. Also Benjamin Kahn ported one of those standard weblog themes (ie. one of the ones you see everywhere on weblogs of all shapes and sizes) to NewsBruiser. It looks nice!

This is kind of a neat tool I wrote but I'm not sure what to call it. It's a wrapper around the famous PyRSS2Gen library. It provides a simple pickle-based backend storage for state relating to an RSS feed that's screen-scraped from a web page. Some of the state it stores is redundant with the actual content of the RSS feed, but some of it is contextual information like "when was the first time a screen-scrape attempt found the item that has this guid"? (I am too tired to explain why this information is useful, but trust me, it quite often is.)

The tool provides convenience functions for fetching a new version of the web page, and does the good-citizen Etag and Last-Modified thing, so all you have to do is write a hook method that scrapes the webpage into a bunch of RSS items and adds them to the feed. As items go on the top of the feed, older ones automatically drop off the end.

In conjunction with Beautiful Soup) this makes it incredibly easy for me to screen-scrape a web page into an RSS feed and have it keep working over time. Up to this point I've been trying to run the various Syndication Automat feeds out of NewsBruiser notebooks. It's a clever idea but cleverness is just about all it's got going for it. It's clunky and awkward, and there are some cases I just can't handle, such as the Dover page where the items I want to RSSify don't have any dates on them. (That's why the contextual information mentioned earlier in this entry is useful, BTW)

I thought any alternative to using NewsBruiser would be a lot of backend work, but it's not. My module's about 150 lines of code: here it is in a temporary location until I come up with a real name for it: RSSHelper.py. Here's a sample script that uses it and Beautiful Soup and ASCII, Dammit (actually the "HTML, Dammit" subset) to make a Dover automat feed.

It originally took me an hour of work and eventual failure to get a NewsBruiser notebook-backed feed for the Dover site. It took about five minutes to write that script linked above. And the new script actually works, instead of putting the same books into the feed over and over again.

I've got the same feeling with this as I did when I stopped writing custom parsers for screen-scraping and started using Beautiful Soup. Is this as cool as I think it is? Do things like this already exist? What should I call it?

PS: Danny deserves a midwife credit since I came up with this and wrote it while working on my second hack for the Life Hacks book.

The Beautiful Soup 2.0 beta is available for a limited time only, before it becomes the real 2.0. Before I bless it I want other people to try porting their old code to the new version, but mainly to see if the new tree builder works as well as I think it does deciding how to build a parse tree out of bad HTML. All the other stuff is new features, so if you have to change your code, it's because your old code was a victim of the weird ways version 1.x parsed bad HTML. One fun way to test it is just load an HTML file into a BeautifulSoup and prettyPrint() it to see if the tags line up the way you think they should. It all depends on your idea of what's fun, of course.

Beautiful Soup 2.0 only works with Python 2.2 and above, which is unfortunate but I had to do that so I could subclass the string and Unicode types. Should a sufficiently large mob form outside my castle, I'll consider doing a version of the the new version that works with Python 1.5.2.

Okay, I've got the new Beautiful Soup running an extremely sensible parsing algorithm based (more or less) on the difference between block tags and inline tags. I gave it a pretty-print mode so I could eyeball the parse tree, and it looks good everywhere I've tried it. Major features I still have to implement:

• Make fetch() capable of searching for text, not just tags.
• Add the nextSibling and prevSibling members for tags that have siblings on the same level of the parse tree.

It's now in Debian! How 'bout that.

Beautiful Soup is on a roll lately, getting lots of buzz at PyCon, and at the BangPyPers, the Bangalore Python users' group. Now that I've got a little free time I worked on version 2.0 this evening, which should make people happy. But then I watched The Emperor's New Groove with Sumana, which made me happy.

I can't think of any other Disney films that are out-and-out comedies. Usually they've got a bunch of songs and morality crap and it's like a dang variety show up there. I really dislike that, but this movie was great. Even its failed jokes were funnier than the comic relief in a typical Disney animated feature. Of course, I stopped seeing these movies a long time ago; maybe they're all like this now.

Oh, the thing I forgot to mention in that enormous entry is that Michael Josephson put my OnDemandAmazonList class in the "contributed code" section of the PyAmazon project. Ordinarily to iterate over an Amazon list you have to iterate over the first page, then grab the second page, iterate over it, etc. There's also at least two different things that might happen when you go over the end of the list. OnDemandAmazonList hides that complexity and lets you iterate over a wish list or result set like a regular Python list. Now I might be able to get away with not including that code in my chapter, which would be nice.

I've got an Amazon wish list where I toss in whatever books strike my fancy in my online travels. The idea is that I'll know what to look for when I'd otherwise just be aimlessly staring at the shelves of used bookstores. The flaw in my plan, which you have no doubt realized, is that when I'm in those bookstores I don't have access to my wish list. Oh, the exquisite irony!

Yeah, I've had enough of that lousy irony. A while ago someone proposed the obvious solution: print out your wish list and keep it in your notebook. The problem is that while Amazon gives you a wide variety of ways to sort your wish list, none of them do you any good in a physical store. No alphabetical by author, or title, no sorting by genre. So yeah, you've got a hard copy of your wish list, but when you've got a wish list like mine with over 250 items, you'd spend all your time context-switching between the printout and the bookshelves.

I figured I could do better, because Amazon exposes their product and wish list info through a REST interface, and I've been messing around with it for a while (other bits of messing-around to show up in my book). So I wrote The Pocket Wisherman. It grabs a wish list, figures out the most-represented genres (for things that have genres), and sorts everything by genre and (where appropriate) author or artist. I wrote a formatter to print a big HTML card of the information that you can print out.

Except. I can't control printed dimensions in HTML. But I wanted 3x5 cards to put in my little birthday notebook. So I wrote a thing that generates the cards in PDF, four to a page. I also wrote a thing to generate a big HTML list, so now I can autogenerate the kind of obsessive page that other nerds would spend days keeping up to date (if only I could do the same for the books I've read... wait, I guess I can).

The Test

But how well does it work? Yesterday, I put it to the test by taking my printout cards to Black Oak Books in Berkeley. I'd made many previous visits, while my wish list was accumulating its items, but on any given trip I'd never found more than maybe 5 books I really wanted. How many would I find if I had easy access to my wish list? How easy would it be to correlate the wish list and the store bookshelves?

I ended up buying 11(!) books from my wish list, and 2 by authors on my wish list (Stephen Baxter and Fernand Braudel) where the bookstore didn't have the specific book I'd put on there. I got 3 books not on my wish list, which is about my usual total haul when I go to a used bookstore. Total elapsed time: about 2.5 hours, which is slightly longer than I would have spent in Black Oak without a list.

There were 6 books on my wish list that I didn't get because they seemed too expensive (including the out-of-print novel with the hilarious name, "Buddy Holly Is Alive And Well On Ganymede", available in a signed first edition for $45). There was also one book I looked at and then decided I didn't really want it. Total books found because of an entry on my wish list: 20. I'm confident I found about 70 percent of the books on my wish list of which Black Oak had used copies. Pretty good.

How easy is it to apply this program? For ze so-called "genre fiction" it is incredibly easy. The Pocket Wisherman automagically figures out which genres to sort by, so that science fiction is shown as Science Fiction and not Fiction (unless you don't have much science fiction). Genre fiction classifications are pretty monolithic, so you just go to that section of the bookstore and run through your list and the bookshelves in parallel. One pass and you're done.

For nonfiction it is difficult. The biggest section of my wish list is "History", but history is not a monolithic classification in most used bookstores. Black Oak has separate sections for the history of every region of the world. It puts biography in with history, though some bookstores have them separately. I went all over the place trying to remember what the books were about (the downside of the truncated author names and titles that make the card display so compact) and in which subgenre I might find them.

General fiction was also a pain, but I think that's just because there was so much of it and so little of it was stuff I wanted.

I had a big "unclassified" section that I didn't really check at all. It's made of high-hanging fruit: really obscure stuff, stuff I don't think a used bookstore would sell, and stuff that might be categorized in any of a number of places.

The categorization itself slips up in a couple places. Historical fiction in my list gets categorized under History instead of Fiction. Sometimes the Amazon data is just wrong (why is the big Far Side collection filed under "History"?)

Sumana points out that if the bookstore has a terminal where you can look things up, or has a lot of staff (unlike Black Oak), you can just hog the terminal or a staff member for a while and avoid a lot of these problems.

Anyway, give it a try. If there's enough demand I might host a web application version of it, but that could be pretty ugly due to the one-query-per-second-per-IP-address limitation of AWS.

So what did I find out today but that the Ultra Gleeper, which still needs to be rejiggered so it's easier to install, is famous! It's mentioned in an article on CodeCon and Demo on B1 of today's Wall Street Journal (though they devalued it to a mere "Gleeper").

And when I tried a Google News search to see if Google had some sneaky deal with the WSJ such that I could use it to link through their subscription firewall (answer: nope), I discovered that the Ultra Gleeper got a much more gushing treatment in Red Herring, in an interview with Christian Leybold, a VC whose company sponsored the "underground convergence of algorithmic minds". Doesn't this mean I get a free round of funding?

At CodeCon I was mobbed with people who wanted to know why I made the Ultra Gleeper run against MySQL instead of the hip self-contained database, SQLite. Actually I think it was Danny O'Brien, who just plugged the Gleeper in this week's NTK and maybe one other person, but I'm no stranger to micro-mobs. Anyway the answer is that I barely heard of SQLite and I've been using MySQL installations more or less continuously at work and at play for years, so I just went with what I was used to.

I know everyone hates software that's hard to install and that the Gleeper is currently such a software, but I was surprised at how for many people the breaking point was having to set up a MySQL database. If most of that is balking at setting up MySQL, then SQLite will probably solve the balking problem. I haven't installed it but I don't see how it could be harder to install than MySQL. Plus the database files it keeps can live in the same directory as the app instead of in /var/, which I think is probably the big draw though it doesn't make much difference setup-wise.

Because people (including me) are all about self-containment, I'm planning to distribute a copy of SQLObject with future versions of the Gleeper. That should short-circuit the SQLObject version fiasco, and it's the only external library the Gleeper depends on. Including a copy of every single external library works well with NewsBruiser, and as a matter of fact it works with the Gleeper, which already includes about ten external modules in its lib/ directory. Why not SQLObject too?

I don't want to distribute SQLite in the package, because it's not a Python library. I'd have to either distribute platform-specific binaries or hook up with SQLite's build system. That said, I don't have a problem with changing over to SQLite if people think that's easier. I had all sorts of doubts about SQLite, but then I realized they're the same doubts people who use proprietary databases have about open source databases, so they're probably WRONG. SQLite looks really nice.

The other thing I could do is write a script that takes you through the installation process, instead of a document that tells you how to do it. A very powerful tool for making installation documents look less complicated is to hide the verbiage at the other end of hypertext links that might not be followed. I didn't do this because I didn't have time before CodeCon. I'm still not doing it because I'm working on a different project, with a deadline. Spooky! To me, anyway. Whoooo!

PS: This is also why there's no new Beautiful Soup, or whatever else it is I promised you I'd do.

One of the things I like about the Ultra Gleeper is its RSS feed, which not only gives you recommendations but also the interface you for rating the recommendations, right in your aggregator. It works really well for me because I use Feed On Feeds, so my aggregator supports Javascript (incidentally, only recently did I find out that "DHTML" is just a code word for "Javascript"). It should work in any aggregator since they any aggregator has to be web-enabled enough to handle clicking on a link, even if it doesn't support Javascript.

If ever again I get into a segfault.org-like situation where every day I have to go through a bunch of user submissions and publish or kill them, it'd be easy to do that as a series of operations in my RSS aggregator. There's an emerging consensus that anything non-urgent you have to keep an eye on should be exposed through RSS, but I'd like to actually be able to take action on those things without having to click over to the main site. You'd see the item, and then any controls neccessary to edit the item, and buttons to publish or kill it. About as easy as such things get.

The only drawback to this idea is that you have to authenticate to get a customized RSS feed, so either your browser needs to store username and password or you need to stick it into the feed URL.

A similar idea in the same vein, which I didn't actually use, was the idea of wrapping any RSS feed at all in Gleeper clothing. The result would look almost exactly like the old RSS feed, but each item would be annotated with a set of rating controls for each link in the entry. I never did this because it'd be way too much trouble to be an RSS feed man-in-the-middle, and change all my subscriptions to point to the cache. It's a good idea though, until someone gets mad that you're changing their RSS feeds, which I estimate would take 16 seconds.

Update: It occurs to me that one could combine these ideas. Suppose you have a website that keeps track of tasks for you, and provides an RSS feed for the tasks, but you have to click through the RSS feed to actually manipulate the task. You could write a script that annotates the RSS feed with the HTML forms and links you need to manipulate the task, and do it all from your aggregator.

It turns out the Ultra Gleeper requires version 0.5.3 of SQLObject, not version 0.6 or up as I previously stated. It looks like I started working on it about three days before 0.6 was released. I know Ian Bicking reads this weblog, so maybe he could speculate on whether porting to 0.6 would be fraught with peril or as easy as changing the import statements.

Update: I just released 1.0.2, which fixes RSS bugs and also moves the variables specific to my installation into cfg.py where you'll run into them while setting up the database and change them to your stuff.

I'm rebooting my Ultra Gleeper installation to get a feel for how long it really takes to set up the dataset, and to write the getting-started document. So far it's taking a loooong time; I think because I'm running it on a machine with less bandwidth. Best find so far: Code Names, a map of secrets.

Gleeper TODO note to myself: (all of this has to be done by Monday)

• Find out why some pages show up twice in a recommendation list.
• http://foo.com won't be created if there's http://foo.com/, but the opposite isn't true.
• Finish and publish paper
• Put up updated code
• Write presentation slides

Am thinking of getting a big standalone freezer to supplement the meager freezer on top of my fridge. I like making food and freezing it to heat up later. How much sense does this make? I'm not sure where in my house it would go.

I'll try to keep posting every day despite the month and a half of constant activity that looms before me. I like posting to NYCB but too often it turns into an excuse to procrastinate, which is my big problem. In addition to Ultra Gleeper, where I've finally pulled out of the rest stop and onto the straightaway, this new project is the textbook example of something you don't want to do if you want to retain free time. I guess I can prove the Ultra Gleeper is worth its salt by just spending a couple minutes a day posting links I find through it.

I've been putting off working on the Ultra Gleeper but today I did most of what I still need to do before CodeCon and it wasn't so bad. Now it's mostly a matter of writing the paper. I feel like I need to take a break from my existing projects and commitments and only work on new things for a while. Maybe after CodeCon, or after CodeCon plus Beautiful Soup 2.0.

I added an RSS feed of the new books published by Dover to the Automat. Now it's easy to keep track of what's going on at every nerd's favorite not-quite-general-purpose publishing house. Don't miss Spacecraft Attitude Dynamics or Country and Suburban Houses of the Twenties: With Photographs and Floor Plans. Sometimes Dover's books are awesome and sometimes they remind me of Andy's father's geology books that we made fun of in ZZT games. "Why drink coffee when you can read Origin of Sedimentary Rocks, or the heart-pounding, best-selling Principles of Geomorphology."

Cannibalism story is done in rough draft, which is good, but now I'm back to my usual uncaring state about story-writing. I was hoping I'd get writing fever and end up going NaNoWriMo and writing the whole novel of which that story is but a set piece, but it seems that's not what my brain has in store for me. I did discover that making up a story while handwriting it goes a lot faster than doing so while typing it.

But, I won't absolutely insist that you treat me right. The subject of my CodeCon presentation, long a secret, has been revealed as the Ultra Gleeper, a recommendation engine for web pages. Yes, I created the thing people have been complaining about not having for nine years. And it does a pretty good job.

It still needs work, but I've got a whole month still to put off doing that work. I meant to work on it this weekend, but instead I wrote a goofy story about cannibalism. Good job, me.

Hey, Beautiful Soup fans (all others can ignore this entry). Among my other projects I am designing Beautiful Soup version 2.0, which should be much more coherent and powerful, as well as generating better parse trees and having better Unicode support. This will come at the expense of Python 1.5 compatibility and (as always) backwards compatibility with previous versions. Thanks to several incredibly useful contributed patches, I have almost everything figured out, but I have two unresolved issues about nomenclature and operator overloading, which follow. I know there is nothing Python programmers love better than arguing about nomenclature, so have at it.

1. Consider the following HTML:

   <p><b>Foo <i>bar</i></b> <u>baz</u></p>
   <div>Some more text</div>

   *ML is parsed by Beautiful Soup (and other parsers) into a treelike structure, like so:

   [root]
    |
    +-P
    | +-B
    | | |
    | | +-"Foo"
    | | |
    | | +-I
    | |   |
    | |   +-"bar"
    | +-U
    |   |
    |   +-"baz"
    |
    +-DIV
      |
      +-"Some more text"
   

   One of the two defining features of Beautiful Soup is that it tries harder than other parsers to build this tree even when the markup is bad, and will always give you some sort of tree, on the assumption that if you wanted a ParsingIsHardLetsGoShoppingException you would have created one yourself. But one of the things missing from Beautiful Soup v1 is (to refer to the tree above) any notion of the relationship between the P tag and the DIV tag, which I've discovered can be a very important relationship to have access to when you're screen-scraping.

   Looking at the tree it's obvious that the P tag and the DIV tag are siblings; they're right next to each other on the same level of the tree. But there's no easy way in Beautiful Soup to get from the P tag to the DIV tag. The .next member of the P tag is the B tag, because it was the thing *parsed* immediately after the P tag. You have to get the parent of the P tag (the root of the document), then get the list of its children, then go through it looking for the P tag, then see what the next thing is.

   In Beautiful Soup v2, the P tag is going to have some pointer to the DIV tag, and vice versa. This will only be useful for relatively well-formed HTML, but when you need it, you need it.

   However I'm not sure what to call these new members. previous and next are already taken as referring to "previous/next thing parsed" and I want to leave that alone. The only other ideas I have are previousSibling and nextSibling. Do you have any other suggestions?

2. The other defining feature of Beautiful Soup is that it comes packaged with tree-traversal methods, inefficient to run but very efficient to not have to write yourself. There are two such methods. The main one is called fetch and it searches the tree for whatever you're looking for, returning a list of everything that matches. This is aliased (in v1 and v2) to the method call operator __call__, so in the example above you'd write soup.fetch("b") or just soup("b") to get a list containing the only B tag.

   But if you know there is only one B tag, it's a pain to call fetch and then take the first item in the resulting list. So there's a helper method called first that does it for you. In Beautiful Soup v2 I want some operator-overloading magic for the first operator. I can't think of anything suitably Pythonic, though.

   Overriding __getitem__ would look OK, but I'm already using it to get a tag's attributes, ie. a['href']. So I've got two current contenders. The first is the dot operator (__getattr__). The dot operator looks the nicest (soup.head.title is lovely) but it also happens to be used for member and method access on objects. I'd rather not have a solution that does one thing when you call .title and another thing when you call .fetch, especially when you might be parsing some XML or made-up markup language that has a "fetch" tag. It also seems like it would slow down the parsing a lot.

   The second contender is the % operator (__mod__). I can't explain why I like (soup % head) % title except that I think it's funny. Unfortunately the joke is too complicated to express in words, so you'll either get it too or you'll just have to take my word for it. On the other hand, if you took the modulus operator at its word you'd expect soup % head to give you the whole document but without the HEAD tags, which is totally not what first does. The other possible reference is to the string interpolation operator, but first is searching for a 'string', not inserting one. So it's not very intuitive unless you share my sense of programming humor.

   I may just not create any shorthand for first since it is already shorthand for fetch()[0]. It would be nice to have something clever and elegant, though, the way we use the method call operator as an alias for fetch. Again, I need ideas.

"Instead of writing ad hoc HTML browsers, use Beautiful Soup". AWESOME

Anthony Baxter, who's contributed patches to Beautiful Soup before, sent in a special parser class for dealing with the aggravations particular to SOAP-generated XML. The class name is BeautifulSoap.

I am a novelty vampire. I can have a cool idea but if it starts eating up all my nights and weekends I think it's not so cool anymore. This felt like it was taking forever, but it only ate three of my nights and now it's more or less done: the fearsome Stock Spam Effectiveness Monitor.

Said monitor tries to get a picture of how people respond to spam by graphically correlating spam that advertises a stock with the price of that stock. I chose this because stock spam isn't as obscure as other spam, due to a gentleman's agreement with the spamees, a greater need for verisimilitude, or some nearly-toothless SEC rule (a la "The publisher of this newsletter discloses the receipt of ten thousand dollars from a third party... for the circulation of this report."). Also because it's the only kind of spam I know of where the sales numbers during a certain time period are public knowledge.

So far the results are not very enlightening, but I'll keep watch over it for a while and eventually we'll figure out just what happens when someone sends out stock spam. Maybe nothing happens! I bet those third parties would like to know that, before they continue shelling out ten thousand dollars a pop for the circulation of this report. Yes, I think they'd indeed be interested third parties.

At last I got some time to mess around with Nethack. But what to do? Something simple to start with. What is Nethack missing, I asked myself? The obvious answer was dinosaurs. It's got all sorts of fantastical creatures, but none of the truly fearsome beasts from Earth's own past. That was my internal sales pitch, anyway.

Study of the sources and this helpful document showed me the way to creating new Nethack monsters, and I was ready to zap my wand of meta-create monster. But which dinosaurs to pick? I thought about that old AD&D monster manual table that lists about 50 kinds of dinosaurs in great detail and somehow manages to make dinosaurs boring. No, I would only pick a representative sample of dinosaurs.

My patch provides but three painstakingly selected lizard-beasts: the fastest dinosaur ever, the smartest dinosaur ever, and the most popular dinosaur ever. The balance is probably off and there's a lot more that could be added (treatment of the fact that, while many dinosaurs have hands, they can't really do much with them; also, I guess you could saddle and ride the smaller dinosaurs), but it makes me happy.

In a follow-up to yesterday's deadly revelation[0] on the post-campaign fate of the code I wrote for the Clark campaign, today I found out from Sumana that the Kerry campaign is using a hacked version of my eBlocks software for distributed phone banking. I'd talked to some Senate campaigns and state parties about it over the months, but this is the first time I've seen it in use. Even more neat!

[0] Okay, so it wasn't deadly. What of it? I thought you liked sensationalism! At least that's what I heard from your ALIEN LOVE SLAVE!

My not-that-bad fears have been confirmed: "Your Spam Name" generators are multiplying! But instead of multiplying forward in time, like skinnable MP3 players or open source CMSes, they are multiplying backward in time. Kevan informs me that after writing his "Your Spam Name" script he discovered an even earlier one, which spits out spammer names at random rather than taking a seed. However, it provided me with the best spam name I've ever seen: Hospitable B. Inhospitality.

I'm sure this comes as a surprise to no one, but frozen green beans are no good. I've been making green beans recently from the fresh ones you have to trim and it's been time-consuming but the green beans are great. Last night I was lazy and used frozen beans and they were soggy like the grave. Well, they were soggy like the ones I hated when I was a kid, the ones that come in a pack with tasteless corn and lima beans that get stuck in your molars (but not your molas) when you chew.

Also, I just noticed that Daily Kos is using my old Clark Community Network rolling open thread code for sites where people love to have huge random conversations. I assume it's my code because the wording is almost exactly the same. Neat.

Yesterday I wrote a little toy called Your Spam Name. In a coincidence that boggles the mind, it turns out that Kevan wrote the exact same toy a couple weeks ago and never told anyone about it. Because I think my toy looks nicer than Kevan's, I'm going ahead and releasing it anyway. After all, what spammer would settle for just one spam name? Anyway, if we don't get the word out people will just keep writing them.

Oh yeah, the other thing I did yesterday was fix up Beautiful Soup to give you more ways to search for tags with certain attributes. I can't take all the credit since Jonathan Ellis came up with the basic idea, but I will take most of the credit.

Automat mania continues; I added a new feed for new things added to the Interactive Fiction Archive. I'm not entirely sure how it's going to work out; probably you'll get new files right away and then again once they get processed. Not a big deal.

Because I've been able to remember to update it for a while, the deliverables weblog is back in business. Let me know if you actually find it useful.

The not-posting life is not the life for me, but there should be no more interruptions of that length in the forseeable future. (I still can't believe I'm doing this silly thing.) Anyway, the Syndication Automat got mentioned on CoolGov, and I added to the Automat a feed for the University of Pennsylvania's Online Books Page, which aggregates links to free online books from all over the web. It has a lot more technical books, religious books, and generally modern stuff than Project Gutenberg. I've been looking at it sporadically for years but now I can keep up with it.

I promised Seth I'd add some other feeds to the Automat, so I'll probably do that this weekend. It's good easy useful work.

Today's my day for publicity, as Beautiful Soup gets a favorable mention in Uche Ogbuji's xml.com column, which is actually about converting HTML into XML. Beautiful Soup tries to reduce the number of times you have to convert HTML into XML, but if you do have to there are tools for it.

libxml2's tree object looks like something I could use as a model for a future version of Beautiful Soup (I do need to rewrite a big chunk of it; I'm painfully aware of numerous embarassing flaws, but it's still the best screen-scraping library IMO).

Who knows why it took me so long to come up with this, but I finally did last night while talking to Sumana (who is coming home tomorrow). Continuing my love of randomly generated entertainment, Dadasaurus Rex does for Dinosaur Comics what Dada Pokey did for Pokey the Penguin (or, to look at it another way, what Pokey the Penguin did for itself). Every minute it puts together a new comic using the hard-to-install-on-non-Debian-systems but wonderful Python Imaging Library.

Eventually this system will expand to consume Dada Pokey and its spin-offs, and I'll also add a Five-Card Nancy style game and more different webcomics, and everything will be happy because I'll have finally gotten rid of that horrible 1998 Dada Pokey code. For now, at least most is happy; love those comics!

Update: in personal communication, Ryan North says Dadasaurus Rex is "FANTASTIC".

If you're messing around with del.icio.us and your intentions are honorable, resist the temptation to hit that hidden link at the bottom of various pages. It does exactly what you think it does.

More sense than it was making before, anyway. I changed it to turn relative times into absolute times, and got rid of the weird "URLs I've seen before" cutoff in favor of a time-based cutoff. It's still over here.

There's lots of Python code that pings weblogs.com, but I couldn't find a Python library for the other side of the equation; one that parses the big XML file of recent pings. That's why I developed the Ass-Kicking Laser Algorithm, which does that very thing. Even with the paucity of things you could conceivably do with the weblogs.com XML file, I think it has more options than are wise. Once I start actually using it I may trim it down. But like I say, one piece at a time.

The name is Kris'. He came up with it in 1998 when I was complaining that the algorithms I was learning about in class didn't live up to their cool names, like the gift wrapping algorithm and Graham's Scan. Since any interesting name I could give this library would set the user up for an anticlimax, I figured I'd go all the way. Thanks, Kris.

In retrospect, though, Graham's Scan is pretty cool.

Sometimes people get ideas about how there should be a special attribute you can give to an <A> tag that indicates just how you feel about the awful, awful thing you're linking to. So that search engines and other programs that analyze links don't misconstrue your linking as endorsement. Unfortunately due to the vagueness of the idea and the fact that it's kind of hard to search for things like "html" and "link", I haven't been able to find anybody's ideas on this topic. Does anyone know of any? I don't want to make up my own ideas because that will defeat the purpose of building some ad hoc consensus.

Somewhat relatedly, some of my readers may not know (and others may know all too well) that there's a whole sub-economy based on gaming search results. The currency of this sub-economy is units of Google PageRank, and it is kind of weird.

I made some changes to Dog Bites Dog to make it get better (ie. "gramatically correct more often") headlines but most of my attempts backfired. I think I improved the average headline quality a little, though. I'm now into heuristic territory (while "man" can be a verb, it's exceedingly unlikely that it is the verb in a headline).

Also, I had a dinner party last night, and I made an ice cream pie which was one of the tastiest things I've ever eaten. Also an artichoke-spinach-cheese dip which I'm having for dinner tonight with my first loaf of French bread. Anyway, at the party were Riana and Seth and Riana's friend Alexey. Riana and Seth have this parrrrrpetual Talk Like A Pirate Day thing going on when they're arrrround each other. Like that. Eventually I pointed out how easy it would be to add a mode to the Eater of Meaning that did the same thing, and well, one thing led to another and before you could say "Let us time how long it takes for you to implement the mode of which we have been speaking", Seth had started his stopwatch and I was running to the computer room. Elapsed time: 3 minutes. Though I put some more time into it afterwards fixing edge cases so that you can discuss things like the aRRRRRTs sound server.

Also, the Eater of Meaning goes further and further off its original goal of destroying all human communication, and is quickly becoming just a framework for writing web filters really quickly. I swear I don't plan these things, they just happen.

The Weekly World News is weird. I don't mean their articles are weird, though the good ones are. I mean they have no consistent narrative about themselves. For instance, The Onion has a conceit that it's a very serious newspaper and is not funny at all, which is actually pretty close to the truth. But the WWN, while not actually claiming to be a joke, doesn't even pretend you're supposed to believe it. It's supposed to shock you, but you can't be shocked by something you know isn't real and were kind of expecting anyway.

Evidence: this contest where you have to make up a headline for the B-movie quality photo. Lots of sites have photo caption contests, but you get the feeling WWN is using the caption contest to figure out the best angle on the story that will eventually go with the picture. Also, every week someone submits "Michael Jackson's New Shocking Look" as a headline, and it's got a pretty good win/loss record, which is just sad.

The other thing is that if they should stumble upon an actually interesting fact, they obfuscate it. Look at this article on typos in printings of the Bible. Now, typos do happen in printings of the Bible and there are lists of them. But the WWN article, while clearly based on such a list, bears the marks of the writer saying "I can think up funnier typos than this!", failing, and publishing their list anyway in preference to the real one which is funnier. It's like they have some code of reverse journalism where they can't publish anything true.

Anyway, I didn't write this to pick on WWN, though it turned out that way because honestly the magazine itself as a physical object gives me the creeps--it's something about the paper. Okay, gotta concentrate. Get to the point and don't say anything else negative about WWN. THE POINT IS that I added a couple more news sources to Dog Bites Dog, on an auxilliary page. It's got WWN headlines and a huge conglomoration which mixes up headlines from the 3 other news sources, and which is quickly becoming my favorite.

Does anyone have more suggestions for sources? It has to be something that usually uses "Crazed Noun Verbs As Bystanders Look On Helplessly" type headlines, because that's all it can parse.

OK, this one takes some explaining. Almost 10 years ago Newt Gingrich wrote an article for his PAC that contained big lists of positive and negative buzzwords for use in political discussion. The point, as is usually the case in politics, was to put attitudes directly into people's heads by piggybacking on peoples' idea of language as a means of communication.

Of course you don't really need the list. Once you've read and written enough political speech the buzzwords just flow from your fingers as you type. But all sorts of irrelevant, non-buzzwords flow from your fingers as well. How can you be sure people see the buzzwords?

Enter the Eater of Meaning. Since political writing contains little meaning to begin with, this eater works a little differently from the others. It simply highlights the buzzwords (green for positive buzzwords, red for negative buzzwords), and leaves everything else alone, letting your readership see the tone of the document at a glance and be appropriately alarmed, reassured, or inspired. Kind of the way Onion articles work, come to think of it. I don't have a good Eater-themed name for it yet; any suggestions?

Examples: #1, #2, vs. a control site in case you think I'm kidding. The list of buzzwords is kind of dated, but on the whole it holds up well. Enjoy.

Update: Danny calls this "'They Live' style shades for political hotwords", which is an excellent summary but which is not Eater-related. My eating metaphor has backed me into a corner while simultaneously painting that corner!

Thanks to PyRSS2Gen, Dog Bites Dog now has RSS feeds. This should be fun. Unfortunately, as always seems to happen with me, Feed on Feeds doesn't parse the RSS file correctly even though Feed Validator (Feeeeeeed Validator) says it's valid. Steve?

Update: Bizarrely, it works now.

I dreamed about RSS feeds and woke up too early, and got Dog Bites Dog running against a source of non-offbeat news. I'm debating whether to leave it up because it turns out shuffling is a headline mood multiplier: it makes "funny" headlines actually funny, but it makes depressing headlines really depressing ("Powell says US Jailed for Lying", "Life Won't Be Freed").

There are some good ones, though, like "Flying ants End Airbus Subsidies", "Ruling Resigns, Admits Gay Affair", and "Category 4 Charley Raises $2.4 Million Near Seattle". In recognition of this, I decided to just get it to not use the most morbid headline fragments. See what I've done? I'm already making the compromises that result in offbeat news in the first place.

This one's for Kris, who a long time ago pointed out that the headlines of "offbeat" news stories become orders of magnitude funnier if you shuffle the subjects and the predicates. Well, Kris has been feeling a little low lately so I thought I'd cheer him up by writing Dog Bites Dog (title a Sumana production), the script which does just that every five minutes. It's not as good as doing it by hand, but what did you expect?

I'm spookily intrigued by the possibility of applying this (very simple) algorithm to 'real' news headlines. But right now I'm more intrigued by the possibility of sleeping. Enjoy, all-night surfers and people on the other side of the world.

I have the darnedest luck with the websites I scrape for the Automat. They either change their website formats while I'm working on them or they stop updating before I can make sure the automatic import works. Case in point: the Catalog of Government Publications ("Pueblo, Colorado stuff", says Kevin), which has been averaging 25-30 new online documents a day but which only had one today. Hope it picks back up soon; I've actually found several very interesting documents in this stream (examples later, when I don't have to go to sleep).

What are they doing, anyway? About two-thirds of the documents they put in the database are not online, and lack even a reliable bibliography link, so I haven't been able to put them in the RSS feed yet. Are there 75 and only 75 new documents published by the government every day? I have no idea whether or not that number even sounds right.

Supposedly the GPO is required to keep a master index of everything the government publishes, but is that really all of it? They're still releasing documents from 2000 and 1994. That implies that there's a big delay in getting some information from the government agency to the GPO. It's a mystery. I am driven by the cool idea of having an RSS feed listing every document the government publishes, but even if this feed had all the not-online ones I can't shake the feeling it's only scratching the surface. Does anyone know?

My latest contribution to the LazyWeb: The Syndication Automat! I suck websites into NewsBruiser weblogs and show you the RSS feeds that result. Eventually I'm sure I'll scrape ConGlomGo's site and anger the corporate gods, but for now I'm sticking to getting information out of the government, Project Gutenberg, and other well-meaning mastodons. Hit the wiki page if there's a feed you want and you think I might want it too. Life is too short for me to mantain a bunch of scraped feeds I'm not actually interested in.

This was the main reason I wrote Beautiful Soup, so it's been in my head a while. I've been building it up in pilfered snatches of time and it's good enough to release now. It's also the first project created under the aegis of the new Center for High-Energy NewsBruiser, the name I like so much I'm going to use it for my other NewsBruiser-as-web-platform projects.

Give it a try if anything there interests you. I'm especially proud of the text excerpts in the Gutenberg feeds, and the clickable maps in the piracy report.

Because Beautiful Soup's reception greatly exceeded expectations (I think I tapped a real market need here), I made a cute little web page for it with lots of examples and Tenniel graphics. I finally get to pay tribute to the line from Carroll that made me actually roll around on the floor laughing when I was 9 and my father was reading The Annotated Alice aloud, one chapter a week, complete with all the incomprehensible-to-children annotations.

Let me toot a slightly different horn here, if I may. A while ago I put out a call for a Python library that parsed an HTML file into a DOM-like data structure. I needed something that worked on Python 1.5.2 with no external dependencies, something that either fixed or forcibly parsed bad HTML, something that made screen-scraping easy; yea, something that both sliced and diced.

You may be surprised to learn that none of my faithful readers' helpful suggestions met with the approval of my demanding eye, and so I wrote my own library, which I like to call Beautiful Soup and which I mentioned in passing yesterday. It's the HTML parser that just doesn't care. If you give it perfect HTML, it'll give you a perfect data structure, just like the big-name parsers. But other parsers know too much about HTML. They choke on or try to rewrite bad markup. They assume you care about the whole document. A pirate might make you walk the plank, but only a parser would make you walk the whole tree.

Not so with Beautiful Soup. What you wrote is what you get. If the HTML is horrible, so is the data structure you get--but you get something, and if you're screen-scraping, you don't care about the whole data structure. You're not writing a web browser. You want to grab some data and get out. Beautiful Soup hides all the tree traversal behind a couple of methods that let you slurp up all the links, all the headings of a certain class, the specific span that contains the train schedule time, whatever. It's similar in philosophy to Aaron's xmltramp.

If, like Lucy Ricardo, you got some 'scrapin' to do, give it a try. I love it. But then, I love all my children. The ones I designed to go on rampages no less than the ones where it just happens.

PS: Since Beautiful Soup knows very little about HTML, and it's based on SGMLParser, you could probably use it on anything that looks like HTML, eg. XML or your domain language that has the same structure as HTML but different tags.

This is seriously the best NewsBruiser yet. It's got all the features on Zack's "stuff NewsBruiser can't do" wish list, and you know Zack's a tough customer. You want NewsBruiser to look like the rest of your website, you got it. You want a blogroll, you got it (albeit slightly ugly to manage). The search engine is smarter and much more robust; it's fun to search where before it was kind of a drag, as though your baby had been untrue.

And if you like to import things, you're in luck. Not only can you hook a NewsBruiser weblog up to a Moveable Type weblog and suck down all the entries, comments, and trackbacks; it now comes with a lumbering Tor Johnson of an HTML machete (deserving of its own post, tomorrow) that you can use to exhume weblog entries (and things-that-should-be-weblog-entries) from their nonstandard HTML coffins.

You'll forgive me if all that functionality makes me wax poetic for a bit. Over the past year or so I've been thinking about weblogs, syndication, online communities, information retrieval, the next step in computer-mediated communication... the whole bit. I haven't been talking about it much, because I don't do the grand vision thing well. In my experience, a grand vision is largely a pheremonal advertisement for the person with the vision. I see instead a lot of little strategies and experiments. The world doesn't need another wannabe visionary, but I figure if I can make some incremental improvements and implement some ideas, I can hold my head high with the rest of them.

This ties into NewsBruiser like so: you know how a lot of open source software projects start out with an idea to write a program to do one task, then get it into their heads to do a generalization of that task, and keep integrating the task over some variable and never get anything done? I am fortunate and happy to report that the history of NewsBruiser development has been the good twin of that phenomenon.[0] All this time I've been envisioning and implementing slight tweaks to NewsBruiser concepts that open up new vistas, restricted liftings of restrictions that make it more powerful. Such changes add up, and somewhere during the past year the application became capable of being a platform. Those slight tweaks are becoming more powerful, and my crackpot ideas for fearsome electronic beasts are resolving themselves into simpler and simpler transforms on the NewsBruiser codebase.

But as they say, talk is cheap. I'll let you know when the killer app is done. For now, we've just got a really good weblog system.

[0] The urge to generalize is natural, and it's not hard to generalize without biting off more than you can chew: you start with a working system; you make incremental improvements; you make sure all improvements have a working system as their endpoint; and you try very hard to make the new system as simple to use as the old system, assuming you don't care about the new features. Above all, be patient. It will be better and come faster, and you'll get a better understanding of the problem domain if you don't try to do everything at once. Slow and steady wins the race. A stitch in time saves nine. Neither a borrower nor a lender be.

PS: Zack, Mike, everyone else who's interested: give this version a try. Let me know if you have problems or feature requests.

I finally got off my duff and did something about the awful performance of crummy.com (mainly caused by my brazen sacrifice of performance for convenience and UI tweakability, running everything through a Noodle-esque wrapper script). I've now got everything except the actual CGIs being rendered from my abstract source files with their dislike of headers and footers to static HTML files, The more dynamic files (people's weblogs) are re-rendered once a minute. It's still a hack, but at least it's no longer a slow hack.

Let me know if you see anything I missed.

Can someone point me to an HTML parser that turns an HTML document into a nested data structure like what I sketched out below? I'm sick of having to jump through hoops to collect the text of a link. I know there's something similar for XML because I heard about it at EuroPython. Stop me before I write my own! Must work or be makable to work in Python 1.5.2. Offer not valid in "Mirror, Mirror" universe.

Sorry, no spectacular thing to show you today. I've got only one of the three things working that I wanted to have, and (sad to say) it's not the one that hits people in the gut and makes them think This Is Important, insofar as any of them do that.

This is an HTML document.

->

["This is an", [TAG name='a' attrs={href: 'http://www.w3.org/MarkUp/', title: "Brought to you by HTMLCorp"} child=[TAG name='b' child=["HTML"]], "document"], "."]

Update, much later: I wrote Beautiful Soup because I wasn't happy with any existing parser.

The parser in the htmllib library is useless for actually parsing HTML. Use the one in sgmllib instead. I learned this once when writing the Eater of Meaning and I'm learning it again with this new project I'm working on. Hopefully I'll have something to show you tonight.

Sumana got me a copy of Twisty Little Passages, a book I've had my eye on for a little bit. I will probably start reading it tonight since I'm too burned out to do anything else (more productive things I could be doing: making Nutellamisu as I'd planned, making any kind of food at all, continuing to unpack/organize all my junk, putting up pictures from the CollabNet retreat, working on NewsBruiser, writing a story, writing something down for the panel--wait, I have to write an entry about that). OK, other entry coming up.

Degeneracy Part II: This Time It's Monetized! (from Sumana)

Brendan brought to my attention that Scott McCloud, possibly the only cartoonist to be lampooned in a Tonight's Episode (what, you're going to check?), has in an act of preemptive revenge gone back in time and ripped off Dada Pokey with Five Card Nancy. A non-Scott McCloud person implemented a nice CGI that lets you build a never-ending Nancy strip by selecting from randomly-presented panels. This would be a really good mode for Dada Pokey.

I keep going to the Five Card Nancy CGI, building some stuff, then getting bored and quitting. It's like a game of FreeCiv, but much quicker. Here are my best ones so far: Short attention span Nancy, hyperactive Nancy.

Aaron Swap Swartz writes:

Because this scrambling letters except for the first and last thing seems to be all the rage on the blogs, I thought I'd write a EoM add-on.

And so he has. Thanks, Aaron. The Eater Of Meaning: your trivial text transformation testbed!

You can now exclude URLs from consideration in your Downhill search. I implemented this in response to Colin, who complained about the appearance in shortest paths of weblogs.com, blogdex, and other things that are not weblogs. Rather than volunteering to mantain in perpituity a list of non-weblogs that nonetheless show up in the ecosystem, I made it a do-it-yourself thing that also makes it possible to incrementally search a space of related weblogs.

hebig.org said Downhill should have an API. Well, now it does. It doubles as a weblog ecosystem API, since I couldn't find one anywhere.

New Downhill; the main new features are a much larger dataset and improved speed. Phil responded to my friendly email by putting up newer copies of the ecosystem data and mentioning me in his weblog, complete with link to embarrassing sophomore-year-in-college first NYCB entry. I've even got my own category now.

Hey, everyone from New Zealand that I've met has been cool and friendly. Are there any right jerks in New Zealand at all? Do you keep them down on the farm where they can't harm nothin'? Or have I just been fortunate?

Here's the thing: Downhill. It's basically an Oracle of Bacon for weblogs. It uses the weblog ecosystem data to find the shortest path between any two weblogs, measured by outgoing and incoming links. I was surprised not to find a preexisting implementation of this, but now it's been done. Just my contribution to the LazyWeb.

A little while back I set up a CVS repository on the machine that hosts Crummy. Today I spent some time importing stuff and setting up ViewCVS, and the result is Leonard's Repository Of Fun. There's also anonymous CVS access; directions are on the RoF page. So far I've put up codebases for the following programs:

• Degeneracy
• Guess The Verb!
• The Eater of Meaning
• ksethdavidschoen
• The Me Software Map
• Tonight's String
• What Fools These Mortals

More coming soon (don't worry, I won't list here every single one I add). I put some work into importing old backup codebases I had for my IF games, so if you're interested you can poke around in the history and find old design docs, like the stream-of-consciousness trancript which was the first thing I ever wrote for Guess The Verb!.

Also, please let me know if I've made some horrible CVS-hosting mistake.

Belated catchy yet inexplicably unusable alternate title for What Fools These Mortals: "God Mode".

A million years ago, I made an offhand comment about the theology of Nethack. The thing I was thinking of was the weird conception Nethack has of atheism, which it doesn't distinguish from theism that doesn't want any divine intervention. But another part of it is a gameplay problem common to adventure games: your character has a powerful backer who has a lot invested in your success, yet who sometimes acts against that interest in the name of a less compelling interest, like greed or aggravation.

The stereotypical example of this is the shopkeeper who wrings his hands and says "You must save Freedonia!" and then tries to nickel-and-dime you on the beef jerky you're trying to buy so you can freaking save Freedonia! I've often felt Nethack deities were acting the same way: look, do you want me to get the Amulet of Yendor or not? Then give me a hand, stop me from turning to stone!

That's why I wrote this songgame: What Fools These Mortals. It lets you simulate being the deity in a game of Nethack. The unexpected thing was, while writing the game, I actually figured out why the Nethack gods are so capricious: it's a lot more fun to smite the player than to help them out, and they're probably just going to die anyway, so why not? This also explains the attitude of the shopkeeper: he's seen heroes come and go, and if he gave beef jerky discounts to everyone he'd go out of business (I don't know why the shopkeeper's financial affairs are persistent across games, when nothing else is, but maybe not all the heroes are you).

Anyway, try out the game; it's got some fun stuff. The code is a mess, but it should be easy to hack if you're so inclined.

If it's enormous word lists you crave, try YAWL, a word list about 5 times the size of the one that comes with Red Hat. YAWL comes with Scrabble cheating help anagram and multi-word anagram finder multi, which according to the README "can provide some cheap thrills when a party gets dull." Maybe you should just adjourn the party early.

More similar word lists at wordlist.sourceforge.net, including a list with part-of-speech data which made me waste time this evening writing an eater that replaces a word with another word of the same part of speech. It's not up yet because I'm not yet sure how good it is.

I buckled to popular demand (well, demand from Aaron SwatchSwartz, who's pretty popular), and created an RSS feed for Tonight's Episode.

The ten minutes I spent tweaking the Eater's definition of "word" so that the NTK line art would show up correctly, has paid off.

Broke my internal promise and worked more on the Eater tonight; I wrote an extension to the WordReplacingEater that lets you copy the content from one URL into the layout of another URL, but I don't think it's good enough to put up yet, and I need to make some dinner.

The Eater is at 1.3. Includes caching; better knowledge about what should be proxied; and three new eaters, including the obsessive-compulsive sort-the-letters-in-each-word eater, and a spooky new get-rid-of-all-content eater. I think I'm done with this for a while.

A glowing review of the Eater over at Maccessibility.

1.2 is up, thanks to some help from Todd and some good old-fashioned elbowgrase (sryy for typos, mt elbowf are slipoery),. It now correctly handles previously troublesome sites like oblomovka.com, zeldman.com, sites with meta redirects, sites with self-closing tags, sites with frames, etc. etc. There's also a new eater which turns a page into lorem ipsum. (Obligatory "meta overload" link)

Crud, it just got linked on MetaFilter. I was going to talk about unit tests but I should work on a cache instead.

Update: OK, much better.

PS: the name "Eater of Meaning" comes from a song I wrote called "Eaters of Meaning". I've had that song stuck in my head for the past 3 days.

So, with that spot of unpleasantness out of the way, we're free to talk about the really exciting thing: unit tests! I wrote the Eater because I like writing that sort of thing, but also to get into the habit of writing unit tests. Here are my observations on the process:

Before you write code, you're supposed to write unit tests. It turns out I can bring myself to do this most of the time, but I do it grudgingly because I don't like writing the infrastructure for a new type of test. The way I generally do it is to write the tests but leave the expected value for the assertions blank (unless the expected value is really easy to figure out, or is actually defined and not just the result of applying some transform to some data I just made up). Then I write the code and all the assertions fail. I use the code I wrote to figure out what the asserted values should be, fixing bugs as I do, and paste in the final, correct values for later use.

Where unit tests are really great is when you find a bug in your code. If you've put in the work to write that infrastructure for that sort of bug, it's easy to record your finding of the bug in a way that will automatically notify you if it ever shows up again. This is lesson one: unit tests give you a place to codify the results of the debugging process. By putting in some incremental effort you can ensure that you're notified should the bug you're fixing ever recur. Usually it doesn't recur, but your real worry is not the bug recurring; it's the bug recurring without your knowledge. With unit tests, you know.

Sometimes when I refactor code I worry that some portion of the code I never exercise has broken due to refactoring. This is the worst thing about Python: unless you're using Leo (which I'm still not, despite my earlier touting of it), when you refactor the code the indentation changes, and when that happens you have to redo the indentation because the automatic indenter never works right, and all sorts of problems can crop up.[0] Unit tests find the problems, and if there are any you fix them and that's that. Lesson two: with unit tests, instead of worrying about code you broke while refactoring, your mind is free to worry about less rational things, like bloodthirsty packs of feral hamsters roaming the streets.

I have unit tests for all my Eaters, and some decent coverage for the HTML parser, but nothing yet for the user interface (even though I wrote the user interface to be drivable from Python code) or the thing that grabs URLs. This is because of lesson three: the closer a piece of code is to an actual user, a piece of hardware, or data from the outside world, the more aggravating it is to write unit tests for it. I am convinced this demonstrates a close affinity between unit tests and Lisp.

The big wrapper lesson is that unit tests are a way of creating a grammar with which to describe the behavior of the system so that you can offload to the computer the task of checking that behavior. Unit tests are easier to write when the elements of the grammar have simple APIs that you control, and harder to write when the elements are complex things with outside dependencies, like URLs and mouse movements. Even if you only write unit tests for the simple stuff, it means your agonizing over the more complicated stuff will be untroubled by the suspicion that the problem is something lower-level and simple (of course, then it turns out to actually be something lower-level and simple, and the code says "Gotcha!" like Lucky Ducky and struts off, but I can't help you there).

[0] Eventually I'm going to write my own autoindenting Emacs hook which operates on the very simple principle of moving the cursor line to where autoindent takes it, then moving every child line left or right the same number of spaces. I don't know why other indenters have these fancy algorithms that don't work, when that algorithm works fine. With my algorithm you have to go down arbitrarily many lines before moving the first line, to see how many lines are underneath the first one, then go back up to the top and go through all those lines again, but on the plus side it actually works.

Kevan and Seth enjoy EoM, and now it's even better. I implemented Kevan's suggestion for an Eater that changes a word to a different word that starts with the same three letters; this makes the text look less purely random and look more like something out of a dream. "I've done it for years with song lyrics, in my head," says Kevan (I quote that herebecause I need to reference it in a future NYCB entry no another topic). Seth just says, "The Eater of Meaning is _so_ great!"

TEoM reminds me of the Troll Book Club order forms I got once a month or so in grade school. My friends and I had great fun erasing or crossing out letters to turn "time" into "tie", etc. and making hash out of the descriptions of Caldecott winners. There is no EoM mode to do this, but I might write one and recapture those halcyon days.

After much additional work, The Eater of Meaning is up. Enjoy its sudsy head, fine oak bouquet, and backbiting aftertaste.

This evening I worked on a little side project, the aim of which was to get me started writing unit tests. Things went pretty well until I reached the point where it was almost screenshot-worthy, then I threw my unit tests to the winds and started fixing things and adding tiny featurelets so I could show you something before I go to sleep. Here you go. I call it the Eater of Meaning.

1. All my troubles seemed so far away.
2. I went to see Seth.

At Seth's place I showed him my PyCon presentation. Then we went through my code. Both presentation and code were well-received. Seth complimented my use of object-oriented design, and we came up with some minor improvements to I Want Options (for instance, IntegerOption should be called CountingNumberOption).

Then Seth showed me the features of Movable Type he likes. I have all of them except for user accounts and permissions (which isn't going to happen anytime soon) and a clever entry list in edit mode. I also like the category list in edit mode.

I passed along to Seth the copy of The Bug, about which more later. I have to pack for DC.

I put up The Power of Greypoint, a tiny script I wrote to turn my ReST presentation file into HTML slides. Everything else I could find turned presentation definitions into images, which I'm not interested in.

Submitted my Python paper. Thanks to Jason, Kevin, Josh, and Greg Stein for their comments on it.

A bunch of NewsBruiser cleanup. It's like cleaning the house, except it doesn't just immediately get dirty again. I added I Want Options to the Leonard Software Map, as well as the little HTML-generation module that comes with NewsBruiser, Leonard's Obsequious Markup Producer.

I submitted my abstract. Whew! I'm afraid of getting bonked on some procedural thing like providing examples of prior art instead of references to papers. I guess I'm more afraid of getting bonked because they think my idea is stupid. Should I put up the abstract so everone can read it? <--my gut reaction for everything, as noted below

Thanks to Jason for proofreading the abstract and giving me lots of help on how to write an abstract. Thanks to Kevin for making it all the way through the motivation before falling asleep.

Right now (or sometime around now), Sumana is introducing Bruce Sterling. I wonder how that's going.

Got the second draft of my abstract done, and not much else. I can't find any real prior work on this, which is weird; can anyone help? I'm looking for frameworks which allow you to get this sort of interface by defining in a text file the options you want to be settable. The closest things I can find are Scarab and a feature of Zope. Both of those are oriented toward dynamically creating simple data gathering forms rather than letting the developer define a static set of settable configuration options with arbitrarily complex logic. There's also Intake, which lets you develop all your form interaction this way, sort of.

I wrote a rough draft of an abstract for a presentation for the Python conference. The topic is the advantages of CGI-based configuration systems, and the implementation of same. I sent the abstract to Jason, who's published papers galore. Anyone else want to see it?

Jon Carroll quotes Guess The Verb!. And it's not a quote published (to my knowledge) anywhere else, which means that he's actually played the game. Mike has also quoted Dr. Prugrave, so this makes him my most-quoted fictional character of all time. Unfortunately, he's too mad to appreciate the honor. (Thanks to Sumana, who noticed this)

I finally remembered where the idea of the little permalink triangle came from: not, as I once thought, from Doc Searls' site, but from a project I did with Peter Hodgson where we did a hypertext version of Diary of a Madman. The triangle interrupted the main text to signal an entry into the thicket of hypertextual readings below.

Why, you ask, did we not instead make use of that marvel of HTML, the text link? I'm not sure (I'd never even asked myself that question until just now). Probably because I was trying for an exact lookalike of the earlier HTML version, which was itself a lookalike of a version done in some proprietary hypertext system, which was itself a lookalike of a version done in a third hypertext system, and so on back 10 or 15 years to the dawn of PC hypertext systems (my version is called 'madman7', and there's talk between Peter and I of a madman8). Somewhere in the DOS era there was probably a hypertext system that didn't know about or didn't want to rely on terminal bold or reverse video to signify linked text, and so displayed the link as an asterisk next to the linked text. I actually think madman7 looks nicer with the triangle than it would with linked text, but I may have to revisit that.

Anyway, here's the triangle, the colors darkened a little for NewsBruiser: . I like it better than the old triangle I made by hand , so I'm going to start using it for NYCB and after I check with Peter I'm going to make it the NewsBruiser default permalink image.

Note the subtle use of attachments there.

Yesterday at lunch the Dan/Kevin/Josh/Leonard entity came up with the new killer app: a site that, when you sign up, uses an RSS aggregator to send you an email newsletter every day containing the new items in those feeds you've told it you want. Such things already (probably) exist, but if they do they rely on screen-scraping and as such have a limited range. If you know of such a thing that exists, tell me so that Dan and Manoj will stop bugging me about adding a newsletter feature to NewsBruiser.

You can tell this was partially my idea because it's an aggregator aggregator.

One beneficial side effect of the Me Software Map is that it gives you incentive to package up software you've written in a usable-by-others form. Where's he going with this?, you think, stealthily reaching for the nearest blunt object. Well, I combined the two Tonight's Episode CGIs into a single file, spruced it up a bit, and released the whole thing as Tonight's String. Ow!

My programming philosophy is "Once is a special case, twice is a copy-and-paste, three times is a generalization."[0] I've revamped my devel page once since I put it up, and I needed to do it again, if only to add ksethdavidschoen. But it's such a pain to write all that HTML with the anchors and the alphabetization and the blah and blah. So I generalized: behold The Me Software Map! (And its concretion, The Leonard Software Map) Now I keep all the project metadata in a simple, no-frills (well, one-frill) configuration file and serve it up to you in a nice HTML form when you request it.

[0] Canonically, you're supposed to generalize the second time, but I've found it more helpful to have three cases (assuming that copy-and-paste is significantly easier than generalizing) because with the triangulation thus provided you're less likely to generalize in the wrong direction. (Also, if I copy-and-paste maybe there won't be a third time and I'll get away with it; sometimes my time is more valuable than the well-factoredness of the code.)

I wrote a CGI to manage the ever-growing and slowly-shrinking list of Tonight's Episodes. I can specify a particular one for the next day, or failing that one will be chosen at random. It'll be easy to set up an archive script, should I ever feel so inclined. Plus, Tonight's Episode is now syndicated, though I'm uncertain whether or not "here, use this string" is a valid syndication format.

Bugs fixed. Djinn game submitted. I just realized that I did a really stupid thing, generating code when i should have had static code and a generated data structure. Fixing now.

So it turns out I was using the 6/10 Inform library for Degeneracy, so I made some diffs and packaged up the source code. Here's the April link to Degeneracy. I should add a third navbar up at the top that I change when I add a new deliverable to the site. Note: the presence of the word "deliverable" in this entry signifies irony.

I'm uncertain as to the value of releasing the source code to and bug list for a work of IF immediately after releasing the work itself. Of course, those who want spoilers at any cost can always disassemble the .z5 file. IF is sort of a special case in programming, and even in game programming, since it's so story-driven and artsy. I don't think I could develop an IF story publicly. I mean, I could do it. Psychologically I would adjust. But it wouldn't feel right, and I don't think it would be as good as it would be if I was able to sit on my vision for the game and not tell anyone about it until I was ready.

That said, I think the basic game structure of Guess the Verb! could have been done a lot better, and that a design session with someone else early on could have made it a much better game. (See upcoming GTV! post-mortem.)

Speaking of which, I really like the developer-written game post-mortems at Gama Sutra, even though I've never played one of the dissected games. I wish that post-mortems could be done for more types of software, but games are 1) relatively unimportant, obviating the need for compulsive litigators to sue because a developer mentioned publicly that a product actually has a defect; 2) a category where, eventually, you can actually claim to be done, rather than just having attained a certain version number.

Back to IF: a whopping *two* bugs were found in the djinn game. I am amazed and flabbergasted, and both bugs have been fixed. It's true that there is always one more bug...

Bah! I have Lock and DoorLock<-Lock classes in Degeneracy, but I get warnings about Lock not being used when I instantiate only DoorLock objects. Fortunately a non-DoorLock lock is in the cards for as soon as I get DoorLock to work.

You can tell I'm running out of things to do to a game when I start implementing the locks on doors as separate objects. I'm only doing this because Mike saw fit to try "put key in lock" instead of the standard "lock door" and I pledged that I would, within reason, make every command typed by a beta tester do the Right Thing.

If nothing else, The Lone Gunmen show takes place in the correct universe, the comedy universe, in which characters can get into funny situations without having to spend a week in the hospital and/or a month in court due to a misguided attempt by the authors of their situations to keep things realistic. So many alleged "comedies" are given the quotation mark treatment by me because, while they claim to be funny, they do not take place in a funny universe. So that's one thing The Lone Gunmen has going for it. The other is the Lone Gunmen themselves, who are almost always funny.

My dilemma here is severalfold; in a bizarre sense, and in common with who knows how many others, I consider the Lone Gunmen show to have been my idea. It's implemented pretty much the way I would have done it (except my version would have taken place in a universe with encryption software), and it airs a year and a bit after I had the idea; about the time I'd think it would take to turn an idea into a television series. And yet, now that it's on the screen, I really can't see where it can go, other than a retelling as farce of X-Files, which is a good idea but which is bound to get tiresome after a while.

Also, any mental commitment I might make to watch TLG next week and see how it comes out is a) likely to be forgotten, since four times out of five I forget to watch TV when I mean to, and b) the exact same mental promise I made for Star Trek: Voyager, and we all know how that turned out.

Mike just sent me a bunch of code for the pig from Degeneracy (not to be confused with the girl from Ipanema). He's been doing more work on that game than I am.

I got zero work done on Degeneracy this weekend. I got quite a bit of work done on the LoTechComp game, though. I know my priorities are inverted (I promised Degeneracy this month, the LoTechGame deadline isn't until tax day), but Degeneracy is a drag to work on. I need to just knuckle down and finish it so I will be free of its deadly curse.

The new game is really fun, almost in the same way robotfindskitten is fun. In fact, in a certain, well-defined sense it is MORE FUN than robotfindskitten. Blasphemy? Perhaps. I can't give you the name because if I give you the name then you know exactly what it is; let's call it "The Djinn Game" (doh) for now.

But check it out: I originally wrote it using the default Inform library. It was about 40K. Since almost none of the Inform library functionality was being used in the game, I wrote my own parser loop, taking it down to 7K (including a skeleton narrative). Since then I've added about 3K of text. I can and hope to add arbitrary quantities of text, which is what takes it into the robotfindskitten vein. Before entering the game into the contest I also need to work out a better theoretical underpinning for the game to make it more consistently challenging.

The most complicated part of my next game is now implemented. Only one other part is anywhere near as complicated.

I'm hoping to start a January beta-test of this game, and release it in February. But now, I sleep.

Grr. I had to update Guess The Verb! again because of a mis-statement about the Second Incompleteness Theorem. The version on gmd.de will make me look like a spaz.

New release of Guess The Verb!, this one the official Release 2. Now includes hints! Cf. my first Usenet posting ever. I could probably have posted the announcement from some UNIX account or other so as not to have looked like the Usenet equivalent of an AOL user, but what the hell.

Hey hey. Monty Hall's Hall of Doors is linked to by the web page for a statistics class.

Campbell was talking about the Monty Hall Paradox today, and I looked at the source code for Monty Hall's Hall of Doors, and I thought, "This Perl is so dingy!". So I rewrote it. MHHD 2.0! Easier to follow, easier to understand, more wacky options than ever before! The only thing I took out was the "play at home" option, cause it messed up the code and I really don't think anyone ever used it.

No new TATC functionality, save the ability to unbind the server, but the interface is a lot better and it's all nicely object-oriented now. I've run it both locally on fire and from kuato, so there really is RMI happening and I'm not just fooling myself. I'll probably spend the rest of the day hacking on this and then Sunday on stuff for Peter.

It seems that my document tree generator does not generate valid 4.0 HTML. Maybe I should actually find out what the DD tags do before using them all over.

I'll fix it later. Right now I'm redoing the notebook program.

Ho ho ho ho ho. I have a basic RMI grep up and running. The hard part is over. The other part is also hard, but is more interesting. My aim is to be able to plug in some keywords and have the server look in the source text crossword-puzzle-wise for your words, a la The Bible Code. The part I'm goig to do for CS131 is just a windowed grep, possibly a multidimensional windowed grep.

I need to find out how to make an RMI server unregister itself from the registry instead of killing the registry every time I change the server. I smell it in java.rmi.Naming.

I finally wrote the mail lister (or most of it) I've been meaning to write. Unfortunately, all the mail I want listed lives on sampo. Oh well.