News You Can Bruise

I've put up a transcript of the talk I gave at the PyCon US Maintainers Summit last month, about the lessons I learned while being the solo maintainer of Beautiful Soup, over 20 years and through two periods of professional burnout:

How to maintain a popular Python library for most of your life without with burning out

The quick takeaway is that strong boundaries are important: both the software boundaries provided by published APIs and packaging dependencies, and the decision as to where your volunteer open source work ends and the rest of your life begins. I have some suggestions for the ways the two interact, and an anecdote about how we mentally rewrite our memories of our struggles to make ourselves more active participants. If you're the maintainer of an open source project, I recommend checking it out!

Right under my self-imposed deadline I've reached my 2019 goal of porting olipy and botfriend to Python 3. Enjoy it! I sure am.

This year I spent some time doing the pleasant part of botmaking (ideation, data gathering) without the boring part (handling dozens of edge cases, signing up for accounts). One of the bots I never completed was "Cassini GIFs".

All the photos taken by the Cassini probe are online, and each photo has associated metadata. By looking for photos taken by the same instrument at evenly spaced times, you can find frames that would look good as an animation. Here's a nice example: a "movie" of Saturn's moon Dione taken on November 1, 2011:

It was really fun to make these animations that probably only a few people before me had seen. And the fun doesn't need to stop at Saturn: the SETI Project's OPUS3 has aggregated imagery from across NASA's outer planet missions.

The bad news for anyone else who wants to try this out is that the Cassini data is huge. You can download individual frames pretty easily, but the metadata is bundled in enormous tarballs and stored in an ad hoc 1990s file format. To provide a booster seat for the future, I converted the metadata into NDJSON format and put it up as cassini-metadata. Here's one more GIF to whet your interest: Saturn's rings on July 15, 2014:

I've come back to working on botfriend as a break from writing, and I've got some new stuff to show you. It's all based around code I put in olipy for dealing with the Internet Archive.

The Archive has so much stuff that if you're not looking for something specific, random selection is the best way to experience it. So I've made it really easy to pick a random item from an IA collection, and (if it's a textual collection) pick any page from that item and get it as an image.

Here's Junk Mail Bot, which provides a random-sample view of this collection. Just look at this cool plotter! Every item has a link to the IA web viewer so you can see it in context.

Yorebooks is the exact same bot but for the collection of yearbooks. I'm partial to this jaunty slice of 1940s writing from the Illinois State Normal university.

These bots were inspired by Rob Manuel's excellent YORE COMPUTER, and it's easy to make your own bots of this sort. In botfriend, a "random page from an Internet Archive collection" bot fits comfortably in fifty lines of code.

But maybe my cool Mastodon bots don't impress you. You miss the days when I would put new bots on Twitter, and Twitter would randomly suspend them. Actually maybe it wasn't random; maybe my bots insulted a Nazi or something. Anyway, I'm not coming back to Twitter but I can offer a compromise: how about a podcast?

The Internet Archive has over 150,000 archived episodes of podcasts, and people almost never listen to old episodes. Again, the "random sample" technique is appropriate here.

So I created Podcast Roulette, which picks a random episode of a random podcast every day and puts it into an RSS feed, creating a fun meta-podcast. In a doubly-meta twist, a lot of podcast episodes focus on one specific mostly-forgotten instantiation of something old, like Episode 384 of the Superman Fan Podcast, which covers a single issue of Action Comics.

I've been putting the sampled podcasts on my MP3 player and although I haven't yet found one that I want to put into heavy rotation, it gives me a feeling I haven't had in a while, the feeling of moving across the radio dial. Update: After a few more days, I realize that this also grants the thrill of eavesdropping.

I'm happy to announce the formal release of two artistic software packages I've been working on for a while. Olipy (PyPI: olipy) is a set of art supplies for manipulating text. It's got sophisticated tools for random selection, a Queneau assembly library, an easy-to-use Python interface to corpora, the *_ebooks algorithm, etc. etc. A lot of my bots are built off the code in here.

Speaking of bots, the second package is Botfriend (PyPI: botfriend). This takes care of all of the boring parts of bot-writing (coding to the Twitter and Mastodon APIs, picking items from a backlog, scheduling posts), allowing you to focus on the fun of creating playful interventions into your friends' depressing social media experiences, bringing joy to all!

I've been using Botfriend to run my personal bots for about a year now. I recently packaged it, improved the docs greatly, and made it really easy to run from within a virtual environment. All you have to do is write the creative bit and put your publishing credentials in a config file. I hope it's useful to you!

Thanks to Allison Parrish for helping me through the realization that I could exploit the pip installation process to install Botfriend's user interface. It feels like an exploit, anyway.

That's my name for the formerly-unnamed technique I used in "Board Game Dadaist". It all started in April, the night I was guest critic for Adam's ITP class. Afterwards I went out to dinner with Adam and Rob, and Adam was talking up Markov chains. Dude loves him some Markov chains. I said "Man, I'm tired of Markov chains. Markov chains are so 70s, they have little coke spoons dangling from them. I'm gonna come up with a better algorithm for creating generative text."

Big talk, but fortunately I didn't have to come up with a better algorithm, because I already had. Back in 2008 I released a project called "Spurious", which generates new Shakespearean sonnets by picking lines from the existing sonnets. It generates two sonnets at once using two different algorithms. Algorithm B (the one lower down on that page) is totally random: you could get a new sonnet made entirely of the first lines of other sonnets. But Algorithm A (the first one on that page) creates what I'm calling a Queneau assembly. The first line of a new sonnet is the first line of some existing sonnet. The second line is the second line of some other sonnet. The third line comes from the set of third lines, and so on to the end.

Oulipo founder Raymond Queneau did something very similar in his 1961 book "Hundred Thousand Billion Poems". This may be where I got the algorithm I used in "Spurious", though I don't think it was a conscious homage. In "Hundred Thousand Billion Poems" there are ten sonnets bound such that you can "turn the page" for a single line of the sonnet, changing that line while leaving the rest of the poem intact. Each generated poem feels like a sonnet because it starts with a "first line" and ends with a "last line" and every line in between is placed where it was in some manually generated sonnet.

I've named the technique in honor of Queneau because I can't find anyone who used it earlier. It's not universally better than a Markov chain, because it only works in certain cases:

• It doesn't work on free text, only on texts with a small-scale internal structure (a poem, a paragraph, a short biography).
• You must have many source texts with that structure.
• You might not like how much of the original text it preserves.

That said, the Queneau assembly gives very entertaining results, and it's now my go-to dada technique, promoted over Markov chains and even unadulterated randomness.

The simple algorithm

I've come up with a number of algorithms for making Queneau assemblies. I'll talk about the simplest first, just so you'll see how this works. This is a refined version of the algorithm I used for "Mashteroids" (yes, those asteroid descriptions were me reinventing Queneau assemblies). It's not the algorithm I used for "Board Game Dadaist"; I'll talk about that later.

You've got a body of N texts, T₀, T₁, ..., T_N-1. Each text can be split into some number of chunks, eg. T₀₀, T₀₁, ..., T_0M-1.

Split each text into chunks and assign each chunk to one of three buckets. The first chunk from each text goes into the "first" bucket. The last chunk from each text goes into the "last" bucket". All the other chunks go into the "middle" bucket.

Also keep track of how text lengths are distributed: how likely it is that a text consists of one chunk, how likely that it consists of two chunks, and so on.

Now you're ready to assemble. Pick a length for your new text that reflects the length distribution of the existing texts. Then pick a chunk from the "first" bucket. If your target length is greater than 1, pick a chunk from the "last" bucket". If your target length is greater than 2, pick chunks from the "middle" bucket intil you've got enough. Concatenate the chunks first-middle-last, and you've got a Queneau assembly!

Paragraphs made from sentences

Now let's look at the scales on which you might create a Queneau assembly. Outside of poetry, the paragraph is the Queneau assembly's natural habitat. A pragraph has a flow to it, especially when you've got something like a description of a board game or an asteroid that's only one paragraph long.

You need to handle things like quotes and parentheses that open in one sentence and don't close by the end of the sentence, or that close without having opened. I wrote code for this in BGD but it doesn't catch all the cases.

Phrases made from words

In "Board Game Dadaist", the names of games are also Queneau assemblies. Here the chunks are words. I take the first word from the name of game A, the second word from the name of game B, and so on. So "Pirates! Denver" might come from "Pirates! Miniature Battles on the High Seas" and "Monopoly: Denver Broncos".

Quotes and parentheses are still problems, though it's not as bad. The big problem I ran into was repeated words, and words like "the" which are not allowed to end a game name. (The simple algorithm, with its "last" bucket, prevents "the" from showing up last unless it showed up as the last word of an existing game. In the algorithm I used for "Board Game Dadaist", I had to special-case this.)

In general, Queneau assemblies will not create coherent English sentences. Much as it pains me to admit, a Markov chain is better for that. It works for board game titles because we allow titles a lot of creative license, even up to the point of suspending the rules of grammar. "Pirates! Denver" makes no sense as a sentence, but it's a perfectly good game title.

Words made from letter-chunks

Many games have single-word titles, eg. "Carcassonne". I wanted to have single-word titles in BGD, but I didn't want to duplicate real names. So I applied the Queneau assembly algorithm on the word level.

Here, the chunk is a run of letters that's all vowels or all consonants. So "Carcassonne" would be split into the chunks ["C", "a", "rc", "a", "ss", "o", "nn", e"]. I keep two sets of buckets, one for vowels and one for consonants. If the first chunk was a vowel chunk, the second chunk is a consonant chunk, and I alternate til I reach the end.

This means that single-word BGD titles are almost never English words, but they do capture the feel of those one-word titles that aren't words (examples: "Zajekan", "Fraseda", "Kongin", "Q-blardo").

The BGD algorithm

Now that you see how it works, I'll explain the algorithm I actually use for "Board Game Dadaist". Instead of three buckets, I have a lot of numbered buckets. When I split a text into T₀₀, T₀₁, ..., T_0M-1, I put T₀₀ into bucket 0, T₀₁ into bucket 1, and so on, with T_0M-1 going into bucket M-1. I create an assembly by picking from bucket 0, then bucket 1, and so on until I've reached the target length.

This is the algorithm that "Hundred Thousand Billion Poems" uses, and when the texts have more structure than "beginning/middle/end", this algorithm works a lot better. I don't think it matters much for BGD descriptions, but I do think it matters for game names. I would like to combine this algorithm with the "last" bucket from the simple algorithm, because right now board game descriptions sometimes end abruptly with a sentence like "Contents:".

God created the natural numbers. Adam created the Networked Byte Organ [See action-packed issue 1024! -Ed.], which listens to network traffic and plays it as music. I created YABO, a Ruby script which takes any data and creates a MIDI file. Samples include a bit from Mega Man 2 (compare Adam's), a bit from an OGG file of a square wave, and a midi representation of yabo.rb itself. Crank up the BPM and you've got a modem. Requires the midilib gem. YABO? Yatta!

Sometimes people express the opinion that Beautiful Soup should be in the standard library. Given the current state of the Python standard library, and the nature of standard libraries in general, I'd rather it not. The standard library is extremely resistant to change—look at sgmllib. It's easier to start over—look at urllib2 and httplib2.

The two other reasons I've seen given for not including BS in the standard library are that it has unspecified Unicode bugs, and that the name is silly. Dude, just rename it. Also, you're using a programming language named after a sketch comedy show. As for the Unicode bug(s), I fixed the one people were complaining about, and the result is Beautiful Soup 3.0.4.

I was actually doing a very large change that no one had complained about, and it wasn't working out well. I decided to just do a release full of easy stuff, since a lot of people were having the Unicode problem.

I'm also seriously thinking about retiring Rubyful Soup, because hpricot is a lot faster.

Sumana informed me of the latest trend: ego-searching Google Code Search to see how widely your software and memes have been incorporated. My name is mostly in peoples' copies of Beautiful Soup, but I did find a Python implementation of a robotfindskitten game from 2004, long predating my own Python implementation. It's called pirhanafindshappiness and it is a bit different from stock rfk. There's also a dice-rolling program I wrote in QBasic in 1993, which ended up on Da Warren.

The WADL Ruby library is up-to-date with the latest WADL standard, and fairly usable despite a lack of syntactic sugar. See especially this example, which turns NewsBruiser into a HTTP+POX web application over the objections of my previous self. Now I'm going to go back to my previous task of writing about why anyone should care about WADL.

Incidentally, WADL, and the Java implementation by its creator, now has its own webpage.