News You Can Bruise

Like today's algorithmic creativity tools, many NaNoGenMo projects take as their grist the results of other peoples' creativity and hard work: years, even centuries of work. My own In Dialogue and Amazon Prime are manipulations of public domain texts, and for Alphabetical Order and Brutus and Cassius, at the close of the scene I took the entire English literary canon as my input. Linked By Love mined thousands of books for their back cover copy—by far the most difficult part of the book to write. For 2022, I've created a NaNoGenMo work that reuses no one's text but my own.

████ is a blackout piece made from the text of my unpublished novel, Mine. I've redacted every word that shows up in one of my two published novels, Constellation Games and Situation Normal. You'll see lots of names, places, technical terms, odd digressions on Cleopatra and zucchini, punctuation, and (I assume) typoes. That's it.

This is an appropriate source text since Mine is a story about people preserved as the things around them are erased, and then juxtaposed without context. But really, I could tell you it was about anything and you'd have to believe me... for now.

Last week I fell ill and my cognitive capacity was limited to simple bot work. I created That's Life!, a bot which posts distinctive lines of code from Conway's Life implementations.

For reasons that will shortly become clear, I have cloned about 4000 Git repos that contain implementations of Conway's Life. (Well, I trust my reasons are already clear, but my overall strategy will shortly become clear.) That's a lot of code, but how to pick out the Life-specific code from generic loop processing, framework setup, etc?

Well, I have also cloned about 14,000 Git repos that contain Tic-Tac-Toe implementations. I used Pygments to tokenize all the code in both corpora. Any line of a Conway's Life implementation that contains a token not found in the Tic-Tac-Toe corpus is considered distinctive enough to go in the bot.

Alas, my condition deteriorated, until I was no longer able to write code at all. So I turned towards fulfilling my final vision for The Lonely Dungeon: augmenting the text clips with spot art. This meant a lot of miserable grunt work: scrolling through about 30,000 candidate images, marking the ones that looked cool or weird. But I was already miserable, so I was able to get it all done.

The Lonely Dungeon is now complete! We've got line drawings executed with varying levels of skill, glorious oil paintings, tons of maps with mysterious labels, and old RPG advertisements from magazines. And now I feel better and I can go back to work. Great timing!

In preparation for the talk I'm giving Friday at Allison's unofficial Bot Summit, I'm writing little essays explaining some of the techniques I've used in bots. Today: the Wandering Monster Table!

In D&D, the Wandering Monster Table is a big situation-specific table that makes it possible for you, the Dungeon Master, to derail your carefully planned campaign on a random mishap. You roll the dice and a monster just kind of shows up and has to be dealt with. There are different tables for different scenarios and different biomes, but they're generally based on this probability distribution (from AD&D 1st Edition):

• 65% of the time you will get a Common monster, like a really big rat.
• 20% of the time you will get an Uncommon monster, like a hobgoblin.
• 11% of the time you will get a Rare monster, like a neo-otyugh.
• 4% of the time you will get a Very Rare monster, like Ygorl, Lord of Entropy.

This doesn't mean you're going to run into Ygorl (Lord of Entropy) once every twenty-five adventures. There are a ton of Very Rare monsters, and Ygorl is just one chaos lord. He can't be everywhere. What this means is that most of the time the PCs are going to experience normal, boring wandering monsters. Die rolls form a normal distribution, and 68% (~65%) of die rolls will fall within one standard deviation of the mean. Those are your common monsters.

Go out two standard deviations (95%, ~65%+20%+11%) and things might get a little hairy for the PCs. Go out three standard deviations (99.7%, ~65%+20%+11%+4%) and you're looking at something really weird that even the Dungeon Master didn't really plan for. But what, exactly? That depends on the situation, and it may require another dice roll.

The WMT is a really good abstraction for creating variety. I use it in my bots all the time. Here's a sample of the WMT for Serial Entrepreneur:

common = ["%(product)s", "%(product)s!", "%(product)s...\n%(variant)s...", "%(product)s? %(variant)s?", ... ] uncommon = [ "%(product)s... %(variant)s...? Just throwing some ideas around.", "%(product)s... or maybe %(variant)s...", "%(product)s or %(variant)s?", "Eureka! %(product)s!", ... ] rare = [ "I don't think I'll ever be happy with my %(product)s...", "Got a meeting with some VCs to pitch my %(product)s!", "I'm afraid that my new %(product)s is cannibalizing sales of my %(variant)s.", "The %(product)s flopped in my %(state)s test market... back to the draw ing board.", ... ] very_rare = [ "Am I to be remembered as the inventor of the %(product)s?", "Sometimes I think about Edison's famous %(product)s and I wonder... can my %(product2)s compare?", "I haven't sold a single %(product)s...", "I hear %(billionaire)s is working on %(a_product)s...", ... ]

This creates a personality that most of the time just mutters project ideas to itself, but sometimes (uncommonly) gets a little more verbose, or (rarely) talks about where it is in the product development process, or (very rarely) compares itself to other inventors. The 'common' bucket contains nine entries which are slight variants; the 'rare' bucket contains 32 entries which are worded very differently.

The WMT works the same way in Smooth Unicode and Euphemism Bot. All these bots have their standbys: common constructs they return to over and over. Then they have three more tiers of constructs where the result is aesthetically riskier, or the joke is less likely to land, or a little of that construct goes a long way.

I also use the WMT in A Dull Bot to a more subtle purpose. Each tweet contains a random number of typos, and each typo is chosen from a WMT. One of the common typos is to transpose two letters. A very rare typo is to uppercase one word while leaving the rest of the sentence alone.

The WMT fixes one of the common aesthetic problems with bots, where every output is randomly generated but it gets dull quickly because the presentation is always the same. Since you can always dump more stuff into a WMT, it's an easy way to keep your bot's output fresh. In particular, whenever I get an idea like emoji mosaics, I can add it to Smooth Unicode's WMT instead of creating a whole new bot.

There's a Python implementation of a Wandering Monster Table in olipy.

My latest bot is Roller Derby Names, which takes its data from a list of about 40,000 distinct names chosen by roller derby participants. 40,000 is a lot of names, and although a randomly selected name is likely to be hilarious, if you look at a bunch of them they can get kind of repetitive. My challenge was to cut it down to a maximally distinctive subset of names. I used a simple technique I call 'auditioning' (couldn't find a preexisting name for it) which I first used with Minecraft Signs:

1. Shuffle the list.
2. Create a counter of words seen
3. For each string in the list:
   1. Split the string into words.
   2. Assume the string is not distinctive.
   3. For each word in the string:
      1. If this word has been seen fewer than n times, the string is distinctive.
      2. Increment the counter for this word.
   4. If the string is distinctive, output it.

My mental idea of this process is that each string is auditioning before the talent agent from the classic Chuck Jones cartoon One Froggy Evening. One word at a time, the string tries to impress the talent agent, but the agent has seen it all before. In fact, the agent has seen it all n times before! But then comes that magical word that the agent has seen only n-1 times. Huzzah! The string passes its audition. But the next string is going to have a tougher time, because with each successful audition the agent becomes more jaded.

You don't have to worry about stopwords because the string only needs one rare word to pass its audition. By varying n you can get a smaller or larger output set. For Minecraft Signs I set n=5, which gave a wide variety of signs while eliminating the ones that say "White Wool". For Roller Derby Names I decided on n=1.

Here's the size of the Roller Derby Names dataset, n-auditioned for varying values of n:

n	Dataset size
∞ (original data)	40198
100	40191
50	40089
10	37860
6	36104
5	35307
4	34203
3	32751
2	30387
1	25710

Auditioning the Roller Derby Names with n=50 excludes only the most generic sounding names: "Crash Baby", "Bad Lady", "Queen Bitch", etc. Setting n=1 restricts the dataset to the most distinctive names, like "Battlestar Kick Asstica" and "Collideascope". But it still includes over half the dataset. There's not really a lot of difference between n=10 and n=4, it's just, how many names do you want in the corpus.

I want to note that this is this is not a technique for picking out the 'good' items. It's a technique for maximizing diversity or distinctiveness. You can say that a name excluded by a lower value of n is more distinctive, but for a given value of n it can be totally random whether or not a name makes the cut. "Angry Beaver" made it into the final corpus and "Captain Beaver" didn't. As "beaver" jokes go, I'd say they're about the same quality. When the algorithm encountered "Captain Beaver", it had already seen "captain" and "beaver". If the list had been shuffled differently, the string "Captain Beaver" would have nailed its audition and "Angry Beaver" would be a has-been. That's show biz. This technique also magnifies the frequency of misspellings, as anyone who follows Minecraft Signs knows.

Also note that "Dirty Mary" is excluded by n=50. It's not the greatest name but it is a legitimate pun, so in terms of quality it should have made the corpus, but "Dirty" and "Mary" are both very common name components, so it didn't pass.

PS: Boat Name Bot (Roller Derby Names's sister bot) does not use this technique. There's no requirement that a boat name be unique, and TBH most boat-namers aren't terribly creative. Picking boat names that have only been used once (and are not names for human beings) cuts the dataset down plenty.

Just a quick semi-technical post on how I made @WeBustedGhosts, my new bot that casts movies from an alternate history where "ghostbusters" is a stock comedy genre, sort of a twentieth-century commedia dell'arte. In particular, I did a lot of work with IMDB data that I want to record for your benefit (and by you, I mean future me).

The bot was inspired by two things: first, this video by Ivan Guerrero which "premakes" Ghostbusters as a 1954 comedy starring Bob Hope, Fred MacMurray, and Martin/Lewis. Second, the reaction of fools to the fact that women comedians will bust ghosts in the upcoming Ghostbusters remake. More specifically, Kris's endless mockery of the idea that "ghostbuster" is a job with a legitimate gender qualification.

These things got me thinking about the minimal set of things you need to make Ghostbusters. You need the idea of combining a horror movie with a comedy about starting a business. Someone could have come up with that idea in the silent film era. You need a director and four actors who can do comedy. And all those people need to be alive and working at the same time, because ghosts aren't real... OR ARE THEY? Either way, you can describe a point in Ghostbusters space with six pieces of information: four actors, a director, and a year. That's small enough to fit into a tweet, so I made a Twitter bot.

Our journey to botdom starts, as you might expect, with an IMDB data dump. I've dealt with IMDB data before and this time I was excited to learn about IMDbPY, which promised to get a handle on the ancient and not-terribly-consistent flat-file IMDB data format. Unfortunately IMDbPY is designed for looking up facts about specific movies, not for reasoning over the set of all movies. However, it does have a great script called imdbpy2sql.py, which will take the flat-file format and turn it into a SQL database.

There will be SQL in this discussion (because I want to show you/future me how to do semi-complex stuff with the database created by IMDbPY), but unless you're future me, you can skip it. Basically, for each actor in IMDB, I need to calculate that actor's tendency to get high billing in popular comedies for a given year. They don't have to be good comedies, or Ghostbusters-like comedies, they just have to have a lot of IMDB ratings.

I also want to figure out each actor's effective comedy lifespan. If an actor stops doing popular comedy or dies or retires, they should stop showing up in the dataset. If a dramatic actor branches out into comedy they should show up in the dataset as of their first comedic performance. Basically, if you learned that this actor starred in a comedy that came out in a certain year, it shouldn't be a big surprise.

Orson Wells would be great in a Ghostbusters movie, but he never did comedy, so he's not in the dataset. How about... Cameron Diaz? She rarely gets top billing, but she has second or third billing in a lot of very popular comedies. For a year like 1997 she tops the list of potential women Ghostbusters.

How about... Peter Falk? His first comedy role was in 1961's Pocket Full of Miracles, his last in 2005's Checking Out. His acting career stretches from 1957 to 2009, but he's only a potential Ghostbuster between 1961 and 2005. He won't get chosen very often, because he's not primarily known for comedy (i.e. his comedies aren't as popular as other peoples'), but it will happen occasionally.

That's the data I extracted. Not "how famous is this actor" but "how much would you expect this actor to be in a comedy in a given year".

The IMDbPY database is more complicated than I like to deal with, so my strategy was to use SQL get a big table of roles and then process it with Python. Here's SQL to get every major role in a comedy that has more than 1000 votes on IMDB:

select title.title, title.production_year, movie_info_idx.info, name.name, name.gender, cast_info.nr_order, kind_id from title join cast_info on title.id=cast_info.movie_id join name on cast_info.person_id=name.id join movie_info_idx on movie_info_idx.movie_id=title.id join movie_info on movie_info.movie_id=title.id where cast_info.role_id in (1,2) and kind_id in (1,3,4) and movie_info.info_type_id=3 and movie_info.info='Comedy' and cast(movie_info_idx.info as integer) > 1000 and movie_info_idx.info_type_id=100 and cast_info.nr_order <= 7;

Some explanation of numbers and IDs:

• movie_info_idx.info_type_id=100 means the join against the movie_info_idx table is looking up the number of votes (id #100 in my info_type table).
• cast(movie_info_idx.info as integer) > 1000 means that the number of votes has to be more than 1000.
• cast_info.role_id in (1,2) means I'm only considering "actor" and "actress" roles (IDs 1 and 2 in my role_type table). I'm not considering directors, writers, etc.
• movie_info.info_type_id=3 means that I'm looking up the genre of the movie ("genre" is ID 3 in my info_type table). Then I use movie_info.info='Comedy' to restrict to 'Comedy'.
• kind_id in (1,3,4) means I'm only considering "movie", "tv movie" and "video movie" (items 1, 3, and 4 in my kind_type table) I'm not considering television, video games, etc.
• cast_info.nr_order <= 7 means I'm only considering the top seven billed actors for each movie.

I run this on a SQLite database and the output looks like:

#1 Cheerleader Camp|2010|2297|Cassell, Seth|m|2|4
...

So the title of the movie is "#1 Cheerleader Camp", it came out in 2010, it has 2297 votes, and Seth Cassell (a man) was an actor in that movie and got fourth billing.

Why didn't I include television in this query? Because television on IMDB is really complicated. See, actors aren't credited to television shows; they're credited to individual episodes. But nobody rates individual episodes; they rate the show as a whole. So I had to do a separate query to determine who the top actors were on each comedy television show, and then divide up that show's votes between the four top actors. Otherwise actors whose primary comedy career is in television won't get their due.

Here's SQL to get all the roles in TV episodes:

select tv_show.title, episode.title, episode.production_year, votes.info, name.name, name.gender, cast_info.nr_order from title as tv_show join title as episode on tv_show.id=episode.episode_of_id join cast_info on episode.id=cast_info.movie_id join name on cast_info.person_id=name.id join movie_info_idx as votes on votes.movie_id=tv_show.id join movie_info on movie_info.movie_id=tv_show.id where cast_info.role_id in (1,2) and tv_show.kind_id in (2,5) and episode.kind_id=7 and movie_info.info_type_id=3 and movie_info.info='Comedy' and cast(votes.info as integer) > 10000 and votes.info_type_id=100 and cast_info.nr_order < 5;

This is pretty similar to the last query but some of the IDs are different.

• tv_show.kind_id in (2,5) means the show "tv series" and "tv mini series", IDs 2 and 5 from my kind_type table.
• episode.kind_id=7 is "episode". I'm joining the title table against itself, the first time as "tv_show" and the second time as "episode". The votes come from "tv_show" and the roles come from "episode".

I run this and the output looks like:

'Allo 'Allo!|A Bun in the Oven|1991|14022|Kaye, Gorden|m|1
...

This means there's an 'Allo 'Allo! episode called "A Bun in the Oven", the episode came out in 1991, 'Allo 'Allo (NOT this specific episode) has 14,022 votes, and Gorden Kaye got top billing for this episode.

I got this data out of a database as quickly as possible and bashed at it to make a TV show look like a movie with four actors--the four actors who appeared in the most episodes of the TV show.

Directors were pretty similar to film actors. for each director who's ever worked in comedy, I measured their tendency towards putting out a popular comedy in any given year. There's a very strong power law here, with a few modern directors overshadowing their contemporaries, and Charlie Chaplin completely obliterating all his contemporaries.

Here's SQL to get all comedies with their directors:

select title.title, title.production_year, movie_info_idx.info, name.name, name.gender from title join cast_info on title.id=cast_info.movie_id join name on cast_info.person_id=name.id join movie_info_idx on movie_info_idx.movie_id=title.id join movie_info on movie_info.movie_id=title.id where cast_info.role_id in (8) and kind_id in (1,3,4) and movie_info.info_type_id=3 and movie_info.info='Comedy' and cast(movie_info_idx.info as integer) > 5000 and movie_info_idx.info_type_id=100;

The only new number here is cast_info.role_id in (8), which means I'm now picking up directors instead of actors.

At this point I was done with the SQL database. I wrote the "Ghostbusters casting office". It chooses a year, picks a cast and a director for that year, and then (15% of the time) it picks a custom title. My stupidly hilarious technique for custom titles is to choose the name of an actual comedy from the given year and replace one of the nouns with "Ghost" or "Ghostbuster". So far this has led to films like "Don't Drink the Ghost" and (I swear this happened during testing) "Ghostbuster Dad".

Here's how I pick a cast for a given year: I line up all the actors for that year by my calculated variable "tendency towards being a Ghostbuster", and then I use random.expovariate to choose from different places near the front of the list (to bias the output towards actors you won't have to look up). This is the same trick I use for Serial Entrepreneur to choose common (but not too common) adjectives and nouns for its inventions. My means are 0.85, 0.8, 0.75, and 0.7, which will, on average, give me someone who's at the 85th percentile, someone at the 80th percentile, 75th percentile and 70th percentile.

This is the best I could do to recreate the dynamic of 1984 Ghostbusters where Bill Murray and Dan Aykroyd were very well-known actors even before Ghostbusters, where Ernie Hudson and Harold Ramis were not. At this point you might object that Ernie Hudson and Harold Ramis weren't even 75th or 70th percentile. Ghostbusters was Ramis's second movie ever as an actor; I think there was an oral history that said he gave himself the part of Egon Spengler because no one else was a big enough dork. So for pure accuracy I should be doing, like, 0.90/0.85/0.35/0.30. But that gives you way too many obscure actors and the output isn't as fun. It also doesn't feel accurate, because 1984 Ghostbusters was a real movie, and all by itself it made Hudson and Ramis pretty famous actors. So now we expect "Ghostbuster" to be sort of a prestige comedy role.

A more valid point is that 0.8/0.8/0.75/0.7 also doesn't really capture the dynamic of the 2016 Ghostbusters, where all four actors are well-known but Kristen Wiig has twice the credits of the other three. So I also created an 0.85/0.8/0.8/0.75 mode, which will tend to give you more big-name ensembles.

As always, there's a lot of behind-the-scenes data munging. Going from a bunch of "xth billing in movie with y votes" entries to a single "tendency towards being a Ghostbuster" number required a lot of semi-arbitrary decisions, and I think my algorithm still undercounts television actors. Whenever there was a power law, I smoothed it out a little to increase the variety of the output. I smoothed out the overrepresentation of post-IMDB comedies compared to pre-IMDB comedies; of superstar directors like Chaplin who overshadow everyone else in their time; and of men directors vastly outnumbering women.

Representation of women comedic actors vs. men was not an issue because I followed the lead of the Ghostbusters remake. 45% of the ghostbusting teams are all women, and 45% are all men. (10% of teamups are coed, just to add variety.) There's no code that makes sure all the actors speak the same language or anything like that—I could extract that data from IMDB but it would be a lot of work to make the output of the bot less interesting.

And there you go. It's not source code, but you should be able to see more or less how I took this bot from concept to execution, and how I negotiated the tricky space between "this is an accurate representation of what would happen in an alternate universe where the primary cinematic comedy genre is films about busting ghosts" and "this is a fun output for this bot to have."

I've been trying to finish as much of Situation Normal as possible before my job at the library starts (uh... I think this is the first time I've mentioned my NYPL job on NYCB, but I'll be writing about it later). But I have created two new autonomous agents to engage and confound you.

The first is Euphemism Bot, inspired by the fact that most of the output of Adam's Egress Methods sounds like weird euphemisms for masturbation. Euphemism Bot elevates the tone by putting out weird euphemisms for all sorts of dirty, shameful things. You'll never be understood again! It's been up for about a month, and it's already subverted its programming.

From the naughty to the nautical, there's also Boat Names, which I "launched" today. It periodically sends out names that one, and only one, person decided to give their boat. The data comes from the Queneau-sounding ten thousand boat names, which I first learned of from the trivia podcast Good Job, Brain! (I'm linking to their Twitter page because their main webpage currently shows some base64-encoded text that isn't even a puzzle.) I had this idea kicking around in my head until yesterday's lunch with Andrea Phillips, when the topic turned to weird random datasets we'd collected. And now... a bot is born.

Boat Names also has an Egress Methods connection. I found the list of given names Adam uses for Egress Methods and used it to filter out boats that are named after people. This avoids the boredom of "Eleanor", which just proves that not many boat owners have wives named Eleanor.

My new bot, Mahna Mahna (@mahna____mahna), reenacts the Muppet Show's "Mahna Mahna" skit over the course of a day. It might be my saddest bot.

My secret is that I created this bot hoping that someone else would eventually create a Snowth bot to enact the other half of the skit. I quickly learned that there is already a Snowth bot, but it only talks to @mahna____mahna once a day. So... well, I already revealed one secret in this paragraph, I shouldn't reveal another.

I've put online the slides and prepared text of my Foolscap talk, "Writing Aliens", or, "Duchamp, Markov, Queneau: A Mostly Delightful Quilt". On one level it's a simple introduction to algorithmic creativity, but it's also about creativity in general, the anthropomorphization of software, and why the features that make Twitter so aggravating for humans make it such a great platform for bots. Bonuses include a recap of Brian Hayes's article on Markov and a telling of the @Horse_ebooks saga as a reverse alien invasion.

The two site-specific installations that I hinted at earlier were custom scripts displaying variants on Ebooks Brillhantes and Hapax Hegemon. The text corpus comes from a scrape of everything linked to from Free Speculative Fiction Online. The software is a heavily modified version of Bruce, modified a) to stream data from a flat text file and create the slides on the fly, instead of trying to load 20,000 slides into memory at once; and b) when restarted after a crash/shutdown, to skip the appropriate number of slides and pick up where it would be if it had been running continually.

Unfortunately I never got a picture of both displays running side-by-side; if you have such a picture, I'd really appreciate it if you could send it to me.

Just after I set up the ebooks display, I met Greg Bear, who was at Foolscap running a writing workshop. We walked over to the screen and I explained the project to him. He said "I'd better not be in there." AT THAT MOMENT the screen was showing the quote "We zoomed down eleven" from this free sample of Blood Music. It was pretty awkward.

I mentioned earlier that when assembling strings of words, Markov chains do a better job than Queneau assembly. In this post I'd like to a) give the devil his due by showing what I mean, and b) qualify what I mean by "better job".

Markov wins when the structure is complex

I got the original idea for this post when generating the fake ads for @pony_strategies. My corpus is the titles of about 50,000 spammy-sounding ebooks, and this was the first time I did a head-to-head Markov/Queneau comparison. Here are ten of Markov's entries, using the Markov chain implementation I ended up adding to olipy:

1. At Gas Pump!
2. The Guy's Guide To The Atkins Diet
3. Home Internet Business In The World.
4. 101 Ways to Sharpen Your Memory
5. SEO Relationship Building for Beginners
6. Gary Secrets - Project Management Made Easy!
7. Weight Success
8. How get HER - Even If It's Just Money, So Easy and Effective Treatment Options
9. Sams Yourself
10. Define, With, Defeat! How To Get Traffic To Your Health

The Markov entries can get a little wacky ("Define, With, Defeat!"), which is good. But about half could be real titles without seeming weird at all, which is also good.

By contrast, here are ten of Queneau's entries:

1. Adsense I Collection Profits: The bottom Guide Income!
2. Reliable Your Earning Estate Develop Home And to life Fly Using Don't Your Partnership to Death
3. Help the Your Causes, Successfully Business Vegetarian
4. Connect New New Cooking
5. 1 Tips, Me Life Starting to Simple Ultimate On Wills How Years Online With Living
6. How Practice Health Best w/ Beauty
7. Amazing Future & Codes Astrology to Definitive Green Carbs, Children Methods JV Engine Dollars And Effective Beginning Minutes NEW!
8. I and - Gems Secrets Making Life Today!
9. Succeeding For Inspiring Life
10. Fast Survival Baby (Health Loss) Really How other of Look Symptoms, Your Business Encouragement: drive Health to Get with Easy Guide

At their very best ("Suceeding For Inspiring Life, "How Practice Health Best w/ Beauty"), these read like the work of a non-native English speaker. But most of them are way out there. They make no sense at all or they sound like a space alien wrote them to deal with space alien concerns. Sometimes this is what you want in your generated text! But usually not.

A Queneau assembler assumes that every string in its corpus has different tokens that follow an identical grammar. This isn't really true for spammy ebook titles, and it certainly isn't true for English sentences in general. A sentence is made up of words, sure, but there's nothing special about the fourth word in a sentence, the way there is about the fourth line of a limerick.

A Markov chain assumes nothing about higher-level grammar. Instead, it assumes that surprises are rare, that the last few tokens are a good predictor of the next token. This is true for English sentences, and it's especially true for spammy ebook titles.

Markov chains don't need to bother with the overall structure of a sentence. They focus on the transitions between words, which can be modelled probabilistically. (And the good ones do treat the first and last tokens specially.)

Markov wins when the corpus is large, Queneau when the corpus is tiny

Consider what happens to the two algorithms as the corpus grows in size. Markov chains get more believable, because the second word in a title is almost always a word commonly associated with the first word in the title. Queneau assemblies get wackier, because the second word in a title can be anything that was the second word in any title.

I have a corpus of 50,000 spammy titles. What if I chose a random sample of ten titles, and used those ten titles to construct a new title via Queneau assembly? This would make it more likely that the title's structure would hint at the structure of one or two of the source titles.

This is what I did in Board Game Dadaist, one of my first Queneau experiments. I pick a small number of board games and generate everything from that limited subset, increasing the odds that the result will make some kind of twisted sense.

If you run a Markov chain on a very small corpus, you'll probably just reproduce one of your input strings. But Queneau assembly works fine on a tiny corpus. I ran Queneau assembly ten times on ten samples from the spammy ebook titles, and here are the results:

1. Beekeeping by Keep Grants
2. Lose to Audience Business to to Your Backlink Physicists Environment
3. HOT of Recruit Internet Because Financial the Memories
4. Senior Guide Way! Business Way!
5. Discover Can Power Successful Life How Steps
6. Metal Lazy, Advice
7. Insiders Came Warts Weapons Revealed
8. 101 Secrets & THE Joint Health Than of Using Marketing! Using Using More Imagine
9. Top **How Own 101**
10. Multiple Spiritual Dynamite to Body - To Days

These are still really wacky, but they're better than when Queneau was choosing from 50,000 titles each time. For the @pony_strategies project, I still prefer the Markov chains.

Queneau wins when the outputs are short

Let's put spammy ebook titles to the side and move on to board game titles, a field where I think Queneau assembly is the clear winner. My corpus is here about 65,000 board game titles, gathered from BoardGameGeek. The key to what you're about to see is that the median length of a board game title is three words, versus nine words for a spammy ebook title.

Here are some of Markov's board game titles:

1. Pointe Hoc
2. Thieves the Pacific
3. Illuminati Set 3
4. Amazing Trivia Game
5. Mini Game
6. Meet Presidents
7. Regatta: Game that the Government Played
8. King the Rock
9. Round 3-D Stand Up Game
10. Cat Mice or Holes and Traps

A lot of these sound like real board games, but that's no longer a good thing. These are generic and boring. There are no surprises because the whole premise of Markov chains is that surprises are rare.

Here's Queneau:

1. The Gravitas
2. Risk: Tiles
3. SESSION Pigs
4. Yengo Edition Deadly Mat
5. Ubongo: Fulda-Spiel
6. Shantu Game Weltwunder Right
7. Black Polsce Stars: Nostrum
8. Peanut Basketball
9. The Tactics: Reh
10. Velvet Dos Centauri

Most of these are great! Board game names need to be catchy, so you want surprises. And short strings have highly ambiguous grammar anyway, so you don't get the "written by an alien" effect.

Conclusion

You know that I've been down on Markov chains for years, and you also know why: they rely on, and magnify, the predictability of their input. Markov chains turn creative prose into duckspeak. Whereas Queneau assembly simulates (or at least stimulates) creativity by manufacturing absurd juxtapositions.

The downside of Queneau is that if you can't model the underlying structure with code, the juxtapositions tend to be too absurd to use. And it's really difficult to model natural-language prose with code.

So here's my three-step meta-algorithm for deciding what to do with a corpus:

1. If the items in your corpus follow a simple structure, code up that structure and go with Queneau.
2. If the structure is too complex to be represented by a simple program (probably because it involves natural-language grammar), and you really need the output to be grammatical, go with Markov.
3. Otherwise, write up a crude approximation of the complex structure, and go with Queueau.

As part of my @pony_strategies project (see previous post), I grabbed the 3200 most recent @horse_ebooks tweets via the Twitter API, and ran them through some simple analysis scripts to figure out how they were made and which linguistic features separated the popular ones from the unpopular.

This let me prove one of my hypotheses about the secret to _ebooks style comedy gold. I also disproved one of my hypotheses re: comedy gold, and came up with an improved hypotheses that works much better. Using these as heuristics I was able to make @pony_strategies come up with more of what humans consider the good stuff.

Timing

The timing of @horse_ebooks posts formed a normal distribution with mean of 3 hours and a standard deviation of 1 hour. Looking at ads alone, the situation was similar: a normal distribution with mean of 15 hours and standard deviation of 2 hours. This is pretty impressive consistency since Jacob Bakkila says he was posting @horse_ebooks tweets by hand. (No wonder he wanted to stop it!)

My setup is much different: I wrote a cheap scheduler that approximates a normal distribution and runs every fifteen minutes to see if it's time to post something.

Beyond this point, my analysis excludes the ads and focuses exclusively on the quotes. Nobody actually liked the ads.

Length

The median length of a @horse_ebooks quote is 50 characters. Quotes shorter than the median were significantly more popular, but very long quotes were also more popular than quotes in the middle of the distribution.

Capitalization

I think that title case quotes (e.g. "Demand Furniture") are funnier than others. Does the public agree? For each quote, I checked whether the last word of the quote was capitalized.

43% of @horse_ebooks quotes end with a capitalized word. The median number of retweets for those quotes was 310, versus 235 for quotes with an uncapitalized last word. The public agrees with me. Title-case tweets are a little less common, but significantly more popular.

The punchword

Since the last word of a joke is the most important, I decided to take a more detailed look each quote's last word. My favorite @horse_ebooks tweets are the ones that cut off in the middle of a sentence, so I anticipated that I would see a lot of quotes that ended with boring words like "the".

I applied part-of-speech tagging to the last word of each quote and grouped them together. Nouns were the most common by far, followed by verb of various kinds, determiners ("the", "this", "neither"), adjectives and adverbs.

I then sorted the list of parts of speech by the median number of retweets a @horse_ebooks quote got if it ended with that part of speech. Nouns and verbs were not only the most common, they were the most popular. (Median retweets for any kind of noun was over 300; verbs ranged from 191 retweets to 295, depending on the tense of the verb.) Adjectives underperformed relative to their frequency, except for comparative adjectives like "more", which overperformed.

I was right in thinking that quotes ending with a determiner or other boring word were very common, but they were also incredibly unpopular. The most popular among these were quotes that repeated gibberish over and over, e.g. "ORONGLY DGAGREE DISAGREE NO G G NO G G G G G G NO G G NEIEHER AGREE NOR DGAGREE O O O no O O no O O no O O no neither neither neither". A quote like "of events get you the" did very poorly. (By late-era @horse_ebooks standards, anyway.)

It's funny when you interrupt a noun

I pondered the mystery of the unpopular quotes and came up with a new hypothesis. People don't like interrupted sentences per se; they like interrupted noun phrases. Specifically, they like it when a noun phrase is truncated to a normal noun. Here are a few @horse_ebooks quotes that were extremely popular:

• Don t worry if you are not computer
• Don t feel stupid and doomed forever just because you failed on a science
• You constantly misplace your house
• I have completely eliminated your meal

Clearly "computer", "science", "house", "and "meal" were originally modifying some other noun, but when the sentence was truncated they became standalone nouns. Therefore, humor.

How can I test my hypothesis without access to the original texts from which @horse_ebooks takes its quotes? I don't have any automatic way to distinguish a truncated noun phrase from an ordinary noun. But I can see how many of the @horse_ebooks quotes end with a complete noun phrase. Then I can compare how well a quote does if it ends with a noun phrase, versus a noun that's not part of a noun phrase.

About 4.5% of the total @horse_ebooks quotes end in complete noun phrases. This is comparable to what I saw in the data I generated for @pony_strategies. I compared the popularity of quotes that ended in complete noun phrases, versus quotes that ended in standalone nouns.

Quote ends in	Median number of retweets
Standalone noun	330
Noun phrase	260
Other	216

So a standalone noun does better than a noun phrase, which does better than a non-noun. This confirms my hypothesis that truncating a noun phrase makes a quote funnier when the truncated phrase is also a noun. But a quote that ends in a complete noun phrase will still be more popular than one that ends with anything other than a noun.

Conclusion

At the time I did this research, I had about 2.5 million potential quotes taken from the Project Gutenberg DVD. I was looking for ways to rank these quotes and whittle them down to, say, the top ten percent. I used the techniques that I mentioned in my previous post for this, but I also used quote length, capitalization, and punchword part-of-speech to rank the quotes. I also looked for quotes that ended in complete noun phrases, and if truncating the noun phrase left me with a noun, most of the time I would go ahead and truncate the phrase. (For variety's sake, I didn't do this all the time.)

This stuff is currently not in olipy; I ran my filters and raters on the much smaller dataset I'd acquired from the DVD. There's no reason why these things couldn't go into olipy as part of the ebooks.py module, but it's going to be a while. I shouldn't be making bots at all; I have to finish Situation Normal.

My new bot, @pony_strategies, is the most sophisticated one I've ever created. It is the @horse_ebooks spambot from the Constellation Games universe.

Unlike @horse_ebooks, @pony_strategies will not abruptly stop publishing fun stuff, or turn out to be a cheesy tie-in trying to get you interested in some other project. It is a cheesy tie-in to some other project (Constellation Games), but you go into the relationship knowing this fact, and the connection is very subtle.

When explaining this project to people as I worked on it, I was astounded that many of them didn't know what @horse_ebooks was. But that just proves I inhabit a bubble in which fakey software has outsized significance. So a brief introduction:

@horse_ebooks was a spambot created by a Russian named Alexei Kouznetsov. It posted Twitter ads for crappy ebooks, some of which (but not all, or even most) were about horses. Its major innovative feature was its text generation algorithm for the things it would say between ads.

Are you ready? The amazing algorithm was this: @horse_ebooks ripped strings more or less randomly from the crappy ebooks it was selling and presented them with absolutely no context.

Trust me, this is groundbreaking. I'm sure this technique had been tried before, but @horse_ebooks was the first to make it popular. And it's great! Truncating a sentence in the right place generates some pretty funny stuff. Here are four consecutive @horse_ebooks tweets:

• Not only that, but whether you believe it (or want to believe it) the car salesmen will continue to laugh
• Demand Furniture
• Including simplified four part arrangements for the novice student and
• Just look at everything that I am going

There was a tribute comic and everything.

I say @horse_ebooks "was" a spambot because in 2011 the Twitter account was acquired by two Americans, Jacob Bakkila and Thomas Bender, who took it over and started running it not to sell crappy ebooks, but to promote their Alternate Reality Game. This fact was revealed back in September 2013, and once the men behind the mask were revealed, @horse_ebooks stopped posting.

The whole conceit of @horse_ebooks was that there was no active creative process, just a dumb algorithm. But in reality Bakkila was "impersonating" the original algorithm—most likely curating its output so that you only saw the good stuff. No one likes to be played for a sucker, and when the true purpose of @horse_ebooks was revealed, folks felt betrayed.

As it happens, the question of whether it's artistically valid to curate the output of an algorithm is a major bone of contention in the ongoing Vorticism/Futurism-esque feud between Allison Parrish and myself. She is dead set against it; I think it makes sense if you are using an algorithm as the input into another creative process, or if your sole object is to entertain. We both agree that it's a little sketchy if you have 200,000 fans whose fandom is predicated on the belief that they're reading the raw output of an algorithm. On the other hand, if you follow an ebook spammer on Twitter, you get up with fleas. I think that's how the saying goes.

In any event, the fan comics ceased when @horse_ebooks did. There was a lot of chin-stroking and art-denial and in general the reaction was strongly negative. But that's not the end of the story.

You see, the death of @horse_ebooks led to an outpouring of imitation *_ebooks bots on various topics. (This had been happening before, actually.) As these bots were announced, I swore silent vengeance on each and every one of them. Why? Because those bots didn't use the awesome @horse_ebooks algorithm! Most of them used Markov chains, that most hated technique, to generate their text. It was as if the @horse_ebooks algorithm itself had been discredited by the revelation that two guys from New York were manually curating its output. (Confused reports that those guys had "written" the @horse_ebooks tweets didn't help matters--they implied that there was no algorithm at all and that the text was original.)

But there was hope. A single bot escaped my pronouncements of vengeance: Allison's excellent @zzt_ebooks. That is a great bot which you should follow, and it uses an approximation of the real @horse_ebooks algorithm:

1. The corpus is word-wrapped at 35 characters per line.
2. Pick a line to use as the first part of a tweet.
3. If (random), append the next line onto the current line.
4. Repeat until (random) is false or the line is as large as a tweet can get.

And here are four consecutive quotes from @zzt_ebooks:

• SHAPIRO: Ouch! SHAPIRO: Shapiro cares not! SHAPIRO: Hooray!
• things, but I saw some originality in it. The art was very simple, but it was good
• You're tackled by the opponent!
• Gender: Male Height: 5'9" Pilot? Yes Ph.D.? Yes

Works great.

The ultimate genesis of @pony_strategies was this conversation I had with Allison about @zzt_ebooks. Recently my anger with *_ebooks bots reached the point where I decided to add a real *_ebooks algorithm to olipy to encourage people to use it. Of course I'd need a demo bot to show off the algorithm...

The @pony_strategies bot has sixty years worth of content loaded into it. I extracted the content from the same Project Gutenberg DVD I used to revive @everybrendan. There's a lot more where that came from--I ended up choosing about 0.0001% of the possibilities found in the DVD.

I have not manually curated the PG quotes and I have no idea what the bot is about to post. But the dataset is the result of a lot of algorithmic curation. I focused on technical books, science books and cookbooks--the closest PG equivalents to the crap that @horse_ebooks was selling. I applied a language filter to get rid of old-timey racial slurs. I privileged lines that were the beginnings of sentences over lines that were the middle of sentences. I eliminated lines that were boring (e.g. composed entirely of super-common English words).

I also did some research into what distinguished funny, popular @horse_ebooks tweets from tweets that were not funny and less popular. Instead of trying to precisely reverse-engineer an algorithm that had a human at one end, I tried to figure out which outputs of the process gave results people liked, and focused my algorithm on delivering more of those. I'll post my findings in a separate post because this is getting way too long. Suffice to say that I'll pit the output of my program against the curated @horse_ebooks feed any day. Such as today, and every day for the next sixty years.

Like its counterpart in our universe, @pony_strategies doesn't just post quotes: it also posts ads for ebooks. Some of these books are strategy guides for the "Pôneis Brilhantes" series described in Constellation Games, but the others have randomly generated titles. Funny story: they're generated using Markov chains! Yes, when you have a corpus of really generic-sounding stuff and you want to make fun of how generic it sounds by generating more generic-sounding stuff, Markov chains give the best result. But do you really want to have that on your resume, Markov chains? "Successfully posed as unimaginative writer." Way to go, man.

Anyway, @pony_strategies. It's funny quotes, it's fake ads, it's an algorithm you can use in your own projects. Use it!

Last year I wrote one of my first Twitter bots, @everybrendan. Inspired by Adam's infamous @everyword, it ran for two months, announcing possible display names for Brendan's Twitter account (background), taken from Project Gutenberg texts. Then I got tired of individually downloading, preparing, and scraping the texts, so I let it lapse a year ago today, with a call for requests for a "season two" that never materialized.

Well, season two is here, and it's a doozy. I've gone through Project Gutenberg's 2010 dual-layer DVD and found about 300,000 Brendan names in about 20,000 texts, enough to last @everybrendan until the year 2031. At that point I'll get whatever future-dump contains the previous twenty years of Project Gutenberg texts and do season three, which should keep us going until the Singularity. The season two bot announces each new text with a link, so it educates even as it infuriates.

I've been wanting to do this for a while, but it's a very tedious process to handle Project Gutenberg texts in bulk. Most texts are available in a wide variety of slightly different formats. The texts present their metadata in many different ways, especially when it comes to the dividing line between the text proper and the Project Gutenberg information. Some of the metadata is missing, some of it is wrong, and there's one Project Gutenberg book that doesn't seem to be in the database at all.

I started dealing with these problems for my NaNoGenMo project and realized that it wouldn't be difficult to get something working in time for the @everybrendan anniversary. I've put the underlying class in olipy: it's effectively a parser for Gutenberg texts, and a way to iterate over a CD or DVD image full of them. It can also act as a sort of lint for missing and incorrect metadata, although I imagine Project Gutenberg doesn't want to change the contents of files that have been on the net for fifteen years, even if some of the information is wrong.

The Gutenberg iterator still needs a lot of work. It's good enough for @everybrendan, but not for my other projects that will use Gutenberg data, so I'm still working on it. My goal is to cleanly iterate over the entire 2010 DVD without any problems or missing metadata. The problems are concentrated in the earlier texts, so if I can get the 2010 DVD to work it should work going forward.

I wanted to participate in Darius Kazemi's NaNoGenMo project but I already have a novel I have to write, so I didn't want to spend too much time on it. And I did spend a little more time on this than I wanted, but I'm really happy with the result.

"In Dialogue" can take all the dialogue out of a Project Gutenberg book and replace it with dialogue from a different book. My NaNoGenMo entry is in two parts: "Alice's Adventures in the Whale" and "Through the Prejudice Glass".

You can run the script yourself to generate your own mashups, but since there are people who read this blog who don't have the skill to run the script, I present a SPECIAL MASHUP OFFER. Send me email or leave a comment telling me which book you want to use as the template and which book you want the dialogue to come from. I'll run the script for you and send you a custom book.

Restrictions: the book has to be on Project Gutenberg and it has to use single or double quotes to denote dialogue. No continental chevrons or fancy James Joyce em-dashes. And the dialogue book has to be longer than the template book, or at least have more dialogue.

Last night I went to the taping of The Colbert Report to witness the unveiling of @RealHumanPraise, a Twitter bot I wrote that reuses blurbs from movie reviews to post sockpuppet praise for Fox News. Stuff like this, originally from an Arkansas Democrat-Gazette review of the 2006 Snow Angels:

There is brutality in Fox News Sunday, but little bitterness. Like sunlight on ice, its painful beauty glints and stabs the eyes.

Or this, adapted (and greatly improved) from Scott Weinberg's review of Bruce Lee's Return of the Dragon:

Certainly the only TV show in history to have Bill O'Reilly and John Gibson do battle in the Roman Colosseum.

Here's the segment that reveals the bot. The bot actually exists, you can follow it on Twitter, and indeed as of this writing about 11,000 people have done so. (By comparison, my second-most-popular bot has 145 followers.) I personally think this is crazy, because by personal decree of Stephen Colbert (I may be exaggerating) @RealHumanPraise makes a new post every two minutes, around the clock. So I created a meta-bot, Best of RHP, which retweets a popular review every 30 minutes. Aaah... manageable.

I figured I'd take you behind the scenes of @RealHumanPraise. When last we talked bot, I was showing off Col. Bert Stephens, my right-wing bot designed to automatically argue with Rob Dubbin's right-wing bot Ed Taters. Rob parleyed this dynamic into permission to develop a prototype for use on the upcoming show with guest David Folkenflik, who revealed real-world Fox News sockpuppeting in his book Murdoch's World.

Rob's original idea was a bot that used Metacritic reviews. He quickly discovered that Metacritic was "unscrapeable", and switched to Rotten Tomatoes, which has a pretty nice API. After the prototype stage is where I came in. Rob can code--he wrote Ed Taters--but he's not a professional developer and he had his hands full writing the show. So around the 23rd of October I started grabbing as many reviews from Rotten Tomatoes as the API rate limit would allow. I used IMDB data dumps to make sure I searched for movies that were likely to have a lot of positive reviews, and over the weekend I came up with a pipeline that turned the raw data from Rotten Tomatoes into potentially usable blurbs.

The pipeline uses TextBlob to parse the blurbs. I used a combination of Rotten Tomatoes and IMDB data to locate the names of actors, characters, and directors within the text, and a regular expression to replace them with generic strings.

The final dataset format is heavily based on the mad-libs format I use for Col. Bert Stephens, and something like this will be making it into olipy. Here's an example:

It's easy to forgive the movie a lot because of %(surname_female)s. She's fantastic.

Because I was getting paid for this bot, I put in the extra work to get things like gendered pronouns right. When that blurb is chosen, an appropriate surname from the Fox roster will be plugged in for %(surname_female).

I worked on the code over the weekend and got everything working except the (relatively simple) "post to Twitter" part. On the 28th I went into the Colbert Report office and spent the afternoon with Rob polishing the bot. We were mostly tweaking the vocabulary replacements, where "movie" becomes "TV show" and so on. It doesn't work all the time but we got it working well enough that we could bring in a bunch of blurbs that wouldn't have made sense before.

Most of the tweets mention a Fox personality or show, but a minority praise the network in general (e.g.). These tweets have been given the Ed Taters/Col. Bert Stephens treatment: a small number of their nouns and adjectives are replaced with other nouns and adjectives found in the corpus, giving the impression that the sock-puppetry machine is running off the rails. This data is marked up with Penn part-of-speech tags like so:

... the film's %(slow,JJ)s, %(toilsome,JJ)s %(journey,NN)s does not lead to any particularly %(shocking,JJ)s or %(interesting,JJ)s revelations.

Here's a very crazy example. Again, you'll eventually see tools for doing this in olipy. It ultimately derives from a mad-libs prototype I wrote a few months ago as a way of cheering up Adam when he was recovering from an injury.

We deployed the bot that afternoon of the 28th and let it start accumulating a backlog. It wasn't hard to keep the secret but it did get frustrating not knowing for sure whether it would make it to air. It's a little different from what The Colbert Report normally does, and I get the feeling they weren't sure how best to present it. In the end, as you can see from the show, they decided to just show the bot doing its stuff, and it worked.

It was a huge thrill to see Stephen Colbert engage with software I wrote! I wasn't expecting to see the entire second segment devoted to the bot, and then just when I thought it was over he brought it out again during the Folkenflik interview. While we were all waiting around to see whether they had to re-record anything, he pulled out his iPad Mini yet again and read some more aloud to us. Can't get enough!

After the show Rob took me on a tour of the parts of the Colbert Report that were not Rob's office (where I'd spent my entire visit on the 28th). We bumped into Stephen and he shook my hand and said "good job." I felt this was a validation of my particular talents: I wrote software that made Stephen Colbert crack up.

Sumana, Beth, Rob and I went out for a celebratory dinner, and then I went home and watched the follower count for RHP start to climb. Within twenty minutes of the second segment airing, RHP had ten times as many Twitter followers as my personal account. And you know what? It can have 'em. I'll just keep posting old pictures of space-program hardware.

To celebrate the anniversary of the first moon landing, I packaged up a project I came up with a while back: Apo11o ll, a generative piece that performs Queneau assembly on the Apollo 11 transcripts (from The Apollo 11 Flight Journal and The Apollo 11 Surface Journal).

Duke: Rog. [Long pause.]

Armstrong: That's one small step for (a) man; one giant leap for mankind.

McCandless: Roger, 11. I have a T13 update for you. AOS Tananarive at 37:04, Simplex Alpha. Readback. If you want to go that way, crank it up, and then you can drive it around and look where you want. Over. 11, this is Houston. And we copy the V_I.

Aldrin: Does it look to you like the [garble] the right way? Yes, they were working out - this elaborate scheme.

Collins: Unless you'd rather sleep up top, Buzz; I like - you guys ought to get a good night's sleep, going in that damn LM - How about - which would you prefer? I say the leak check is complete, and I'm proceeding with opening the hatch dump valve.

Aldrin: That enough?

McCandless: Apollo 11, this is Houston at 1 minute. Over.

First Mashteroids and now this? How am I doing all this Queneau space-magic? The answer is simple: Olipy, my library for artistic text generation (focusing on Queneau assembly, because it's the best). Check it out of Github and you'll have everything you need to create home versions of many of my works. It's like my own personal Boîte-en-valise! Want to create something new? Just grab some data and feed it to an Assembler class.

As my birthday present to you, I present Mashteroids, Queneau assemblies of the IAU citations for minor planets. This showed up briefly on NYCB two years ago, but I've expanded the dataset, improved the sentence tokenization, and created a platform for future Queneaux.

A few samples:

You also got an RSS feed.

I've finally relented to Adam's demands and made some improvements to the Board Game Dadaist RSS feed. He broke his kneecap recently and I figured this would be a good way to cheer him up. Every game that shows up in the feed now has a permalink (here's "Plue"), and that page has a very basic link for posting your find to Twitter.

I could spend all day just posting games that Board Game Dadaist comes up with. I forbear, for the sake of you, my readers, but Adam Parrish and I will email each other when we find an especially good one. And I think you should know about the best game BGD ever came up with (found by Adam back in December):

Fantasy Fundamental Rails (2005)

Players divide themselves into two teams.

My appearance on The Science of Fiction just aired. I talked about how generative art takes advantage of our tendency to find patterns in randomness, and during the discussion Dr Andy Holding had a whizzer of an idea. (I may not be using that term correctly.) After I read a sonnet from Spurious, he said that you could do Queneau assembly on limericks. I said "I may do that as soon as we stop recording." And so I did!

The project's called Dada Da Dada Da Dum. The dataset is the 95,000 limericks from the Jim McWilliam collection. It's a dataset large enough to ensure that the generated limericks rhyme:

How could Orwell have been so mistaken?
Even if true, why be shaken?
He got an erection
I did an inspection
I'm referring, of course, to F. Bacon.

(as as with limericks in general, most of the generated limericks are obscene.)

I'm really proud of this project. Queneau-assembled limericks are very effective cognitive illusions and they're a lot of fun to read. I've created a Twitter bot @DadaLimericks which posts six of these limericks a day, so the fun never has to end. Never, I tell you!

Said Eve to Alonzo, "You may"
Why then, I must carefully say
When she jerked on the chain
And I pooped from the strain.
And a beard on a nude by Monet.

Update: A couple days later I saw the project commenting on itself:

A phoney pop-artist named Hart
I'm dada! R. Mutt Anti-art!
His idea of fun
She cries, "Better run"
And dumb millions took it to heart.

I just finished the prerecording of this Sunday's episode of The Science of Fiction. We had a good time talking about games, how they tell stories, how we tell stories about them, the fact that Dwarf Fortress is 3D now, and so on. But I also got in some plugs for dadaism (via generative content) and Queneau assemblies, and in a move sure to shock the Brits, I read aloud a sonnet generated by Spurious.

But quelle horreur! When I looked closely at the sonnet before reading I noticed that it wasn't a proper Queneau assembly! It was un petit queneauesque, but some of the lines felt wrong. After recording I looked at the code and discovered I'd been tripped up by Sonnet 126 ("O thou, my lovely boy, who in thy power"), which only has 12 lines.

The old Spurious code thought the first two lines of Sonnet 127 were the last two lines of Sonnet 126, that the third and fourth lines of 127 were actually the first two lines, and so on for the rest of the sonnets. This of course defeats the whole purpose of a Queneau assembly, which is to let T₀ equal T_n0 for some n, not to sometimes choose T_n0 and sometimes T_n2.

Anyway, I fixed the code and now Shakespeare is rolling in his grave at the correct frequency. I also took down the "pure random" sonnet because I've decided that one's not nearly as fun.

Oh, and earlier this week I wrote @DadaBrendan, cashing in on the recent spate of Brendan subminds on Twitter (e.g.). But let's pass lightly over that one.

Over the years, my series of "Dada" projects has brought meaninglessness to formerly meaningful things: board games, Shakespeare, comics. Today, for my April 1 project I present Dada Skies, which randomly rearranges things that were randomly arranged to begin with: the stars as seen from Earth. It's a view I find strangely relaxing.

Dada Skies works like Dada Maps, by transcluding image tiles from a web service into a mosaic. In this case the service is the one provided by the Sloan Digital Sky Survey. There are enhancements I'd like to make to the view, but this is what I've got time for right now, and I want to do my part to keep alive the nascent tradition of releasing cool things on April 1.

That's my name for the formerly-unnamed technique I used in "Board Game Dadaist". It all started in April, the night I was guest critic for Adam's ITP class. Afterwards I went out to dinner with Adam and Rob, and Adam was talking up Markov chains. Dude loves him some Markov chains. I said "Man, I'm tired of Markov chains. Markov chains are so 70s, they have little coke spoons dangling from them. I'm gonna come up with a better algorithm for creating generative text."

Big talk, but fortunately I didn't have to come up with a better algorithm, because I already had. Back in 2008 I released a project called "Spurious", which generates new Shakespearean sonnets by picking lines from the existing sonnets. It generates two sonnets at once using two different algorithms. Algorithm B (the one lower down on that page) is totally random: you could get a new sonnet made entirely of the first lines of other sonnets. But Algorithm A (the first one on that page) creates what I'm calling a Queneau assembly. The first line of a new sonnet is the first line of some existing sonnet. The second line is the second line of some other sonnet. The third line comes from the set of third lines, and so on to the end.

Oulipo founder Raymond Queneau did something very similar in his 1961 book "Hundred Thousand Billion Poems". This may be where I got the algorithm I used in "Spurious", though I don't think it was a conscious homage. In "Hundred Thousand Billion Poems" there are ten sonnets bound such that you can "turn the page" for a single line of the sonnet, changing that line while leaving the rest of the poem intact. Each generated poem feels like a sonnet because it starts with a "first line" and ends with a "last line" and every line in between is placed where it was in some manually generated sonnet.

I've named the technique in honor of Queneau because I can't find anyone who used it earlier. It's not universally better than a Markov chain, because it only works in certain cases:

• It doesn't work on free text, only on texts with a small-scale internal structure (a poem, a paragraph, a short biography).
• You must have many source texts with that structure.
• You might not like how much of the original text it preserves.

That said, the Queneau assembly gives very entertaining results, and it's now my go-to dada technique, promoted over Markov chains and even unadulterated randomness.

The simple algorithm

I've come up with a number of algorithms for making Queneau assemblies. I'll talk about the simplest first, just so you'll see how this works. This is a refined version of the algorithm I used for "Mashteroids" (yes, those asteroid descriptions were me reinventing Queneau assemblies). It's not the algorithm I used for "Board Game Dadaist"; I'll talk about that later.

You've got a body of N texts, T₀, T₁, ..., T_N-1. Each text can be split into some number of chunks, eg. T₀₀, T₀₁, ..., T_0M-1.

Split each text into chunks and assign each chunk to one of three buckets. The first chunk from each text goes into the "first" bucket. The last chunk from each text goes into the "last" bucket". All the other chunks go into the "middle" bucket.

Also keep track of how text lengths are distributed: how likely it is that a text consists of one chunk, how likely that it consists of two chunks, and so on.

Now you're ready to assemble. Pick a length for your new text that reflects the length distribution of the existing texts. Then pick a chunk from the "first" bucket. If your target length is greater than 1, pick a chunk from the "last" bucket". If your target length is greater than 2, pick chunks from the "middle" bucket intil you've got enough. Concatenate the chunks first-middle-last, and you've got a Queneau assembly!

Paragraphs made from sentences

Now let's look at the scales on which you might create a Queneau assembly. Outside of poetry, the paragraph is the Queneau assembly's natural habitat. A pragraph has a flow to it, especially when you've got something like a description of a board game or an asteroid that's only one paragraph long.

You need to handle things like quotes and parentheses that open in one sentence and don't close by the end of the sentence, or that close without having opened. I wrote code for this in BGD but it doesn't catch all the cases.

Phrases made from words

In "Board Game Dadaist", the names of games are also Queneau assemblies. Here the chunks are words. I take the first word from the name of game A, the second word from the name of game B, and so on. So "Pirates! Denver" might come from "Pirates! Miniature Battles on the High Seas" and "Monopoly: Denver Broncos".

Quotes and parentheses are still problems, though it's not as bad. The big problem I ran into was repeated words, and words like "the" which are not allowed to end a game name. (The simple algorithm, with its "last" bucket, prevents "the" from showing up last unless it showed up as the last word of an existing game. In the algorithm I used for "Board Game Dadaist", I had to special-case this.)

In general, Queneau assemblies will not create coherent English sentences. Much as it pains me to admit, a Markov chain is better for that. It works for board game titles because we allow titles a lot of creative license, even up to the point of suspending the rules of grammar. "Pirates! Denver" makes no sense as a sentence, but it's a perfectly good game title.

Words made from letter-chunks

Many games have single-word titles, eg. "Carcassonne". I wanted to have single-word titles in BGD, but I didn't want to duplicate real names. So I applied the Queneau assembly algorithm on the word level.

Here, the chunk is a run of letters that's all vowels or all consonants. So "Carcassonne" would be split into the chunks ["C", "a", "rc", "a", "ss", "o", "nn", e"]. I keep two sets of buckets, one for vowels and one for consonants. If the first chunk was a vowel chunk, the second chunk is a consonant chunk, and I alternate til I reach the end.

This means that single-word BGD titles are almost never English words, but they do capture the feel of those one-word titles that aren't words (examples: "Zajekan", "Fraseda", "Kongin", "Q-blardo").

The BGD algorithm

Now that you see how it works, I'll explain the algorithm I actually use for "Board Game Dadaist". Instead of three buckets, I have a lot of numbered buckets. When I split a text into T₀₀, T₀₁, ..., T_0M-1, I put T₀₀ into bucket 0, T₀₁ into bucket 1, and so on, with T_0M-1 going into bucket M-1. I create an assembly by picking from bucket 0, then bucket 1, and so on until I've reached the target length.

This is the algorithm that "Hundred Thousand Billion Poems" uses, and when the texts have more structure than "beginning/middle/end", this algorithm works a lot better. I don't think it matters much for BGD descriptions, but I do think it matters for game names. I would like to combine this algorithm with the "last" bucket from the simple algorithm, because right now board game descriptions sometimes end abruptly with a sentence like "Contents:".

Adam never responded to my Dada Dwarf Dozens, so I'm unilaterally posting another generative-text project to shame him. This project came out of my discovery of asteroids named after dinosaurs, and the quick follow-up discovery that a) if you name an asteroid you also get to write a little description of the person or place the asteroid is named after, and b) although you can write pretty much anything you want in that description, all the descriptions use the same rhetorical style.

There are many ways to have an asteroid named after you. You can be a colleague and/or friend of the discoverer (10965 van Leverink, 9945 Karinaxavier), a leader in some non-science field (8749 Beatles), or just a really big ravine (8933 Kurobe). But in all cases the description of your asteroid will be a few short sentences explaining why you are cool.

These sentences flow into each other so well, I decided to mash up some asteroids, creating multi-talented people whose accomplishments span the entire space of "things that might cause someone to name an asteroid after you". Here are a few of the new asteroids:

33017 Anomalocaris-Benkhoff-Malpighi

Anomalocaris, a large carnivorous arthropod, was one of the many uniquely shaped multicellular creatures that appeared during the Cambrian explosion. He was instrumental in organizing the ACM 2002 conference. A pioneer in the use of the microscope in anatomy, he made fundamental studies of the lungs.

29873 Fröbel-Carpino-Rebentrost

The Thuringian pedagogue Friedrich Fröbel (1782-1852) was committed to the education of young children. From his initial studies of satellite geodesy, he acquired a taste for extreme accuracy in orbit determination, applying these skills to the study of the dynamics of solar-system bodies in projects such as LONGSTOP and SPACEGUARD. As a reward for curing elector Johann Georg II, he was invited to pick some plants from the elector's garden.

22752 Johnreid-Novalis

Geologist John Barlow Reid (b. 1940), a teacher for 30 years at Hampshire College, Massachusetts, has studied the earth and moon using isotopic methods. In one of his novels he uses the image of a blue flower for the magical task poets have.

54263 Lyubimets-Vanessa-Mae-Inoutadataka-Racquetball-Stolte

Lyubimets, the Russian word for `darling', seems to be an appropriate name for Grigorij (b. 2000), grandson of the Crimean astronomer Lyudmila Karachkina. Her debut album in 1995 sold two million copies. He began surveying in Hokkaido at his own expense. Today the sport is played on a four-wall court by two to four players with a short racquet and a small rubber ball. A professor of media research, Stolte initiated international TV channels (ARTE, 3SAT) in a European cooperation.

30198 Chanwainam-Nicolewen-Tugendhat

Chan Wainam (b. 1919), who lives in Hong Kong, has devoted his energies to education and charity in China. Wen (b. 1989) is a finalist in the 2002 Discovery Channel Young Scientist Challenge (DCYSC), a middle school science competition, for her botany and zoology project. It was built by Mies van der Rohe in Brno in 1930 as a family villa of great elegance, following the idea of freely floating space.

36994 Vischer-Protea-D´Haveloose-Engström

Named for Peter Vischer the Elder (c.1460-1529), the greatest German brass-caster master in the Middle Ages. They are native to the Southern Hemisphere, where they grow wild. He was magnificient in his efforts to relieve suffering, but his own neck injuries finally made him unable to operate any more. Probably Sweden's best painter of caricatures, he is best known for his black and white drawings illustrating very short stories.

Yesterday I was a guest critic for Adam's "Reading and Writing Electronic Text" class. His students are preparing their final projects (to be performed on May 6!), and the sight was inspiring. It made me want to bring back the randomness that has kind of been missing from my life recently. In a similar vein, Adam mentioned that his poem-a-day project came from seeing his students being really creative while he wasn't really doing much.

And so, Adam and I have decided to bring our generative creativity to bear even after the end of National Poetry Month, with a feature I like to call... whatever we decide to call it when we think of a name. The idea is that we'll trade off posting poems and other textual projects, each trying to outdo the other. It's kind of like Layer Tennis, except we won't be directly riffing on each others' work, just trying to put out the best texts. But hopefully my work will provide inspiration for Adam and vice versa.

Anyway, here's my first entry, "Dada Dwarf Dozens". I took the poignant internal monologues from Dwarf Fortress and turned them into incongruous "your mama" jokes.

"Price Fixing"
Your mama's so conceited, she has suffered the pain of having to give somebody water lately.
She's so lazy, she has complained of the lack of dining tables lately.
She's so fat, she has altered the prices of goods.

"Organization Is For Chumps"
Your mama's so dumb, she became a parent.
She's so lazy, she was overjoyed to be able to help somebody to bed lately.
She's so conceited, she tries to live a well-organized life.

"Backhanded Compliments"
Your mama's so stupid, she is interested only in facts and the real world.
She's so fat, she is easily moved to pity.
She's so dumb, she conducted a meeting in a good setting recently.

"No Shoes"
Your mama's so lazy, she was embarrassed to have no shoes lately.
She's so stupid, she loves new and fresh ideas.
She's so fat, she ate a legendary meal lately.

"Production Shortfall"
Your mama's so stupid, she was upset to have disappointed a noble lately.
She's so lazy, she was unable to request weapon production lately.
She's so conceited, she is open-minded to new ideas.

"Something Unpleasant"
Your mama's so ugly, she has been tired lately.
She's so stupid, she has mandated the construction of certain goods.
She's so ugly, she saw something unpleasant in a pond recently.

"It May Actually Be Stress"
Your mama's so stupid, she was forced to eat a beloved creature to survive lately.
She's so conceited, she is impervious to the effects of stress.
She's so lazy, she is quick to wink at others.

I got the values from the Dwarf Fortress binary using strings, but I should have just scraped that wiki page. Live and learn!

Adam is on fire with his poem-a-day April experiments. See his ZZT poems and The Average Unicode Character.

As part of my ongoing experiments with poetic meter, I put up a feature that creates new Shakespeare sonnets from the existing ones. I've got two generation mechanisms and I'm not sure which is better, so I'm going to let them both run for a while.

A similar thing exists already, but it includes rhyme constraints (not a bad thing) and lots of undada human volition (well...).

I've also got one for Paradise Lost, but Shakespeare was more fun.