<Y

[Comments] (3) Mon Jan 02 2012 19:32 Frances Daily: We're out of the gate with the first Crummy project of the new year. I spent Christmas in Utah, and while I was there my sister Susanna foisted on me my mother's old Franklin day planner. She's already typed up my mother's Franklin diary from the 80s and 90s, a diary very similar to the weblog she kept from 2001 until she died in 2006, so all that was left of the day planner was an enormous block of calendars.

It's a very detailed calendar that will be useful for family history purposes but quite boring otherwise. Except, at the beginning of each month there was an "index" page on which my mother had written a brief summary of every day of that month. There were index pages from January 1987 to November 1990, with a few sporadic months from later on.

The cumulative effect of these daily summaries was incredibly powerful. This was a time of major upheaval in my family, leading up to (but not ending with) my father's death in 1992. The monthly summaries show my mother trying to keep it together while studying for a graduate degree and raising three kids pretty much on her own. It's inspiring and a bit horrifying.

I've started a Twitter account, @FrancesDaily, which is using Sycorax to reprint the daily summaries 25 years after my mother wrote them. (Here's the RSS feed.) The summary for each day will go up at 4 PM Pacific time. It's a little spotty in January, but once it picks up she doesn't miss many days for the next three years.

I don't know what the effect of the summaries will be when experienced in real time. Probably when I read one I'm going to mentally compare my day against the day Frances had 25 years earlier, and you might want to do the same.

This is a much lower-bandwidth project than Roy's Postcards, but where my father's writing almost never showed any emotion, these summaries pack a lot into just a couple words. Susanna read the index and said "Mom was really hard on herself." So please take entries like "wasted day" as accurate depictions of my mother's mental state, but not necessarily of reality.

A note about the names. I did the thing you frequently see in old journals that have passed through the hands of the journal-writer's descendants, and replaced most of the names with initials. For instance, the January 18 summary, "Mario's Eagle", became "M's Eagle". The Mario mentioned is Mario Canton, one of my dad's Boy Scouts and later a family friend. I initialized most of the names because you don't have the context to know who all these people are, and I think giving each entry an explanation significantly longer than the entry itself would ruin the effect.

In a few cases, making peoples' names into initials was also necessary to protect their privacy. I left a few names intact: mostly my mother's close relatives like her aunt LeJeune ("Jeuney"), her sister Anne, my father, and of course me and my sisters.

Again, that Twitter account is @FrancesDaily. Here's the RSS feed.

[Comments] (1) Tue Jan 03 2012 08:49 CG Author Commentary #5: "The Stars My Screensaver": Yeah, you know it's getting serious now. The microblog archive is up, I'm feeling good and it's time for some commentary:

And there's the commentary. Stay glued to the proverbial set for chapter 7, when Ariel will say, "Well, her hardware's Chinese..."

Image credits: Gisela Giardino, The United States Department of State, and the East German postal service.

<- Last week Next week ->

Wed Jan 04 2012 11:38 GeekDad Reviews Constellation Games: Wired's Jonathan Liu got an advance copy and calls it "a perfect blend of aliens, video game geekiness, and modern social media." Other quotes relevant to my interests include "absolutely loved it — it might not be the best book I read in 2011, but it’s certainly in the top 10."

C'mon, folks, do I have to draw you a picture? Because I can't draw very well. I use words instead.

Sat Jan 07 2012 10:18 Dear Santa: Dinosaurs:

My niece's Christmas stocking project.

Mon Jan 09 2012 06:30 Connected Tragedies: Sumana noticed this. 1, 2.

Tue Jan 10 2012 08:43 CG Author Commentary #7: "Party Creation": Ariel's Twitter will be pretty quiet this week because the entire chapter takes place over the course of one day. Your only solace is farmers market quail sausage, and this COMMENTARY:

Stay tuned for chapter 8, a chapter I think is one of the best in the book, the chapter that got me to give up on the first draft and rewrite the entire book to be more like it. The only chapter in which Jenny will say, "Wait a minute, are you naked?"

Image credits: I got the first image from Flickr user marsmet462, not sure if they put in enough transformative elbow grease to put their own license on it. Second image comes from Sven-S. "☃" Porst .

<- Last week | Next week ->

[Comments] (8) Tue Jan 10 2012 11:42 Hostile Witness: The Vechs Interview: Minecraft can be a tough game. The controls are kind of blocky, the best resources are hidden deep in the map alongside deadly lava rivers, and the night hosts monsters that will kill you just as soon as look at you (or, in one case, just as soon as you look at them). But it's not that tough. All that terrain is generated by algorithm. It's not like the random number generator is trying to kill you.

But there's this guy named Vechs who is trying to kill you. His "Super Hostile" series of custom Minecraft maps offer challenges that prohibit or subvert every survival strategy you learned in vanilla Minecraft. Even in his easier maps you'll find bottomless pits, world-spanning ceilings that block Minecraft's all-important sunlight, swarms of monsters pouring from hacked spawners, and TNT in unfortunate places. Just getting your first tree is a challenge. Complete a Super Hostile map, and vanilla Minecraft will seem easy.

But Vechs' maps are not just tough: they're creative, fun to play, and they look great. Vechs uses landscape features and lighting to grab the player's attention, direct the flow and pacing of their playthrough, and give them a spectacular environment to build in once they've conquered the map.

I've raved about Super Hostile a couple times before here on NYCB, but with the release of "Spellbound Caves", the tenth entry in the series, I knew it was time to get serious. I sat down with Vechs (I assume he was sitting down, anyway) and interviewed him over minecraftforum.net's private message feature. My goal was to pick up where this interview from July 2011 left off, with in-depth questions about his style and his mapmaking wish-list. The interview contains some Minecraft jargon, but anyone with an interest in game design should get something out of it.

Leonard: You play a kind of character on your maps, an angry trickster god who hates his players and taunts them by writing things on signs. But clearly you're not actually like that. I've played maps made by people who really did hate me, who wanted me to farm cobblestone for an hour or dig through obsidian without a tool, and I said "screw this" and quit the map. I don't think you'd ever do that, right? What's the difference between you and the "Vechs" in your maps?

Vechs: It has to do with challenge. There is a difference between making the player use skill or ingenuity, and making the player do something tedious. Sometimes a solution to an area can involve using lots of blocks (Like the player making a cobblestone tube for them to safely move through.), but these are usually just one option of many the player can use to conquer an area.

Sometimes the "Vechs character" in my maps is pretty mean, and just downright spiteful, especially when it comes to traps. In real life, I'm not like that at all.

In the Obsidian Block interview you say that you recently graduated from college and are looking for a career as a game designer or world designer. What did you study in college?

I am a Media Arts major. I studied everything from digital image editing, video editing, to stage lighting, to writing scripts and screenplays for movies, and more. I'm glad to have a diverse background, even though my passion is still game design.

What would be your ideal job? Would you rather work on a big-budget project with high production values, or an indie project where you have more creative control?

My first choice would actually be to have my own studio and bring to life some of the game ideas I have. One idea I've had for a while, and as far as I know, nobody has ever made a game like it. I wouldn't mind making it all myself, but that means I would have to re-learn a lot of programming. I've programmed some text-based games in C++, but programming is not my main forte.

That said, I also wouldn't mind working for a major company. Like, for example, Valve. Love those guys.

What other games have you made maps for? You mention Duke Nukem 3D in the Obsidian Block interview; what else?

Just off hand: Red Alert, Warcraft II, Warcraft III, Neverwinter Nights, Total Annihilation, TA:Spring, Terraria, Command and Conquer (and several sequels), The Elder Scrolls series, and obviously I'm the world designer for the RPG games I've worked on, using the XP and VX engines.

Are you currently making maps for any games other than Minecraft?

At this moment, no, but I have been meaning to make some maps for Team Fortress 2.

Have you ever heard of ZZT or Megazeux, or am I just incredibly old?

You're old! *grins* I looked them up, and I think my version of that would be the RPG-series of game engines.

There are a lot of memorable set pieces in the Super Hostile series. Now that you've put out ten maps, would you mind taking a look back and sharing some of your favorites?

The first 15 minutes on just about any of my maps. I love that feeling of just starting off and scrambling for resources. I like the rail station in "Sea of Flame II", and how it goes out in the area with the huge pillars, and "Spellbound Caves" is just full of nice vistas and "scripted" events.

Most of my maps feature at least one "death fortress" as an end-game area. These are intended to be where the player gets to use all the resources and items he has been collecting through the whole map. TNT, lava, swords, bow and arrow, even TNT cannons... bring your whole arsenal and have some fun!

Can you describe the evolution of your design philosophy over the course of the series?

Try to improve in at least one area every time I make a new map. Push the Minecraft engine to its limits. Make an awesome and memorable experience for the player.

What are the biggest challenges in re-balancing Super Hostile for Minecraft 1.0?

Armor and blocking.

Does 1.0 have anything to do with the fact that you recently flattened the difficulty levels in your map descriptions, so that "Sunburn Islands" and "Legendary" are now both considered "Easy"?

Yes and no. I feel that recently I have been drifting away from the theme of "Super Hostile" and I want to get back to my roots. Being able to respawn forever, over and over kind of takes the risk out of a map. Even in "Legendary", unless you really mess up and drop all the wool in lava or something, you can just set your bed spawn near an area, and try over and over until you get it right. I think that's pretty Easy on the player, even if the area you are attempting is challenging.

Call me nostalgic, but I kind of miss (sometimes) the GAME OVER screens from older video games. Modern video games, in the name of convenience, typically feature unlimited lives, save games, checkpoints, the works. But beating a modern video game, I have to admit, is much less satisfying than beating some of those old NES games. You can just bang your head against the game until you get lucky and get through an area. Heh, man this makes me feel old! "In my day, we didn't have all those checkpoints! We had three lives! One hit deaths! And we were happy!" *shakes cane*

Anyway, I do think this is a legitimate point of concern on modern game design, is risk versus reward. It is possible to make games so easy that they are very unsatisfying...

I'm an admirer of your ability to create new genres of map. Have you made experimental Minecraft maps that just didn't work? What's in your "abandoned projects" folder?

The only thing I've actually stopped on, is "Race for Wool #3: Common Ground". Because it basically became "Capture the Wool".

Have you ever made maps for a game that featured scriptable events? If so, do you miss that capability in Minecraft?

I have used C++ to code some text-based games. I have also used various scripting languages in the process of making mods or making my own games with existing engines. You do have some limited "scripting" ability in Minecraft, using redstone. Check out the Rumbling Caverns in my tenth map and you will see what I mean. :)

But yes, I would love some even rudimentary scripting in Minecraft. I believe a while ago, I proposed invisible effector blocks, that you can place with Creative or MCedit, that modify the immediate environment around them. Like, an invisible block that makes monsters not spawn within 50 blocks. Or one that doubles monster spawning within 50 blocks. Or one that makes it snow. Or one that makes a ray of sunlight always be shining on that spot. Or one that makes the temperature freezing so any water turns to ice. Simple stuff like that. They would show up faintly in Creative mode, but be invisible while in survival mode.

What would you like to see added to Minecraft? On your forum thread you mention that you'd like to add sharks and underwater plants to "Endless Deep". What else?

Bow enchantments... more mining enchantments, such as area mining. Check out episodes 04 and 05 of my Spellbound Caves Dev Commentary.

For bow enchantments, I would like:

I think these enchants for bows would make bow combat much, much more fun. It's currently fairly slow paced, and a bit boring. Imagine a bow with Toxic, Piercing, and Phantom Spreadshot on it! It would be so much fun to shoot groups of enemies with a bow like that.

You have a creative relationship with some of the people who do Let's Play videos of your maps. It's a kind of relationship I've never seen before: the way people play your maps in public affects the way you design later maps. How did these relationships develop?

Very organically. Zisteau agreed to LP my very first map, "Sea of Flames" version 1.0, and ever since then, he's been involved in playing my maps, and giving feedback.

There's a very clever trap in "Spellbound Caves", [location redacted]. It's clever for many reasons, but I'm asking about it because it doesn't seem to have any triggering mechanism. I went in afterwards and took the walls apart and couldn't figure out how it works. What's the secret? Or is there a pressure plate somewhere that I missed?

I has a seekret. Oh, also, I hate you, die in a fire.

POSTSCRIPT: With my interviewee uncooperative, I had no choice but to load a fresh version of "Spellbound Caves" into an editor to get to the bottom of the mystery. What I found was a trigger that did not shock me to the core of my being. But it is a cool design.

The trigger is a proximity sensor: a shaft behind a wall, with a creeper spawner at the top of the shaft and a pressure plate at the bottom. When the player gets within 16 blocks of the spawner, it activates and spawns a creeper, which drops onto the pressure plate, triggering the trap. The resulting explosion obliterates both creeper and spawner, leaving no trace of the trigger.

And that's what you get with Vechs' maps: MacGyver-like use of everything the game engine provides, to create confounding and unexpected effects. Seriously, game studios: hire this guy. Everyone else: play his maps.

PPS: Hey, people from minecraftforums.net, thanks for coming over. I've written other articles about Minecraft (1 2 3 4), and if you like my stuff, you might want to check out my novel about alien video games.

[Comments] (1) Thu Jan 12 2012 09:17: My sister talks about her miscarriage.

The only thing worth saying is "I'm sorry." I may think those things. John and I may even say those things to each other. But don't impose beliefs or possibilities or happy thoughts on me.

[Comments] (1) Mon Jan 16 2012 10:50 Findings: My writing life has settled down a bit so I'm finally going to write about Findings, the social reading startup where I worked last summer. This is more an essay about what I see in Findings than an introduction to the site--you can see lots of general introductions linked to from this Findings blog post, including co-founder Steven Johnson's introduction, and the Business Insider article whose title is the perfect elevator pitch, "Findings is GitHub for Ideas".

If what I'm about to say sounds interesting to you, there are development jobs open at Findings right now. Just as a reminder, I myself don't work at Findings anymore, and even when I did, only the foggyheadedest knave would have taken my personal opinions as representative of company policy.

Let me start out with this quote I took from Darwin's The Descent of Man, not because the quote itself proves anything, but because the quote is an important part of my reading of Darwin:

Brehm gives a curious account of the instinctive dread, which his monkeys exhibited, for snakes; but their curiosity was so great that they could not desist from occasionally satiating their horror in a most human fashion, by lifting up the lid of the box in which the snakes were kept. I was so much surprised at his account, that I took a stuffed and coiled-up snake into the monkey-house at the Zoological Gardens, and the excitement thus caused was one of the most curious spectacles which I ever beheld.

If you want to learn about evolutionary biology, read Steven Jay Gould's essays. Darwin's a good writer and he got it basically right, but he didn't know about genes or DNA. I read Darwin to experience the origins of the field. I didn't expect (though perhaps I should have) to encounter endless artifacts of the days of two-fisted Baconian science.

When Charles Darwin needs to figure something out, he carries out an experiment, no matter how tedious or disturbing. He takes snakes to the zoo, he puts kittens' feet in his mouth, he floats 94 kinds of plants in seawater, he hacks aphids. If someone has the temerity to question his conclusions he's all "Citation needed? I'll give you citations, motherfucker!"

When you read a book, it has an effect on your mind. You're a slightly different person after reading it. You've created something new: a reading of the book: Here's an apropos quote from Alberto Manguel's A History Of Reading, which I read on paper and typed in. Manguel is talking about Petrarch's Secretum:

What Augustine (in Petrarch's imagining) suggests is a new manner of reading: neither using the book as a prop for thought, nor trusting it as one would trust the authority of a sage, but taking from it an idea, a phrase, an image, linking it to another culled from a distant text preserved in memory, tying the whole together with reflections of one's own -- producing, in fact, a new text authored by the reader.

Readings are ephemeral. Life goes on, and the memory fades. Ken Macleod's The Star Fraction had a huge influence on me, probably leading to whatever career I now enjoy as an author of fiction, but I read it ten years and 600 books ago, and now I don't remember a damn thing about it.

That's why we dog-ear pages and highlight passages. We're instantiating our reading of the book so we can go back later and approximate the mental state it gave us without re-reading the whole thing. Even if all we got out of a book was "this bit was funny", it's better to have the funny bit at hand than not. Even if you never go back to the highlighted passage, the act of highlighting replays that passage and deepens your initial memory of it.

Liberate your readings

I've been typing in quotes from the paper books I read, like I did with the Manguel. Of course, with an electronic book, you don't have to do this. The act of highlighting creates an electronic record of your reading of the book. When I was in college I read about the first e-ink research coming out of MIT, and I knew that this was the future. Indeed it was the future, because I had to wait ten years for the technology to make it to market. But, sour grapes, we've got ebook readers now.

Ebook readers have big problems, but at this point the problems are mostly political, not technical. For instance, you can highlight passages when reading a book on your Kindle, but because of a deal between Amazon and the publisher, your book's metadata may include restrictions, which the Kindle will obey, on how much you can highlight. And your highlights and notes—the "new text authored by the reader"—are stuck on a website that Amazon didn't put a lot of work into because they don't consider your reading of a book important to their business.

Findings takes advantage of the fact that Amazon is wrong about this. Findings liberates your highlights and makes them searchable and shareable. Your reading of a book is a big part of your relationship with that book, and Findings gives you access to it.

You can also use Findings to take a reading of a web page, creating a record of what would otherwise be an ephemeral activity. I'm not as interested in this feature, but people are using it quite a bit, and my interest does increase as the length of the web page I'm reading approaches the length of a book.

Browse readings

So that's what Findings can do for you personally. Now let me pitch you the network effects. Take a look at this screenshot which shows the Findings global stream:

You can't see the global stream without logging in, which I think is a shame because I think this is what really sells Findings. We have here a stream of little bits of text, like Twitter used to have on their front page. Except here, every bit of text is a quote that someone liked well enough to save. It's very high-quality stuff. At the top you can also see some recently added books, and by clicking on a book you can see someone's condensed reading of the book.

Basically, Findings gives you browsing access to a large library, not of books, but of readings. It's easy to discover new books, people who read books you like, and—this is new—people who read books in ways you like.

There are a ton more useful things I could mention, but they're mostly behind-the-scenes things where Findings makes things "just work" (like consolidating multiple editions of the same text), or they depend on features that haven't been implemented yet. So I'm going to close by mentioning the social signalling feature.

Strut your stuff

One underappreciated feature of paper books is signaling to other people that you are cool. You read books! Fancy books, like Ulysses! You care enough about books to make space for them in your house. You take them on the subway even though they're heavy. Darwin would say it's like the peacock's tail. But if you have an ebook reader, nobody knows how cool you are. You're just a person with an ebook reader.

By letting you publicize your reading of an ebook, Findings reinstates your ability to send those social signals. The downside is that you have to actually read the book. You can't just put a big book on your coffee table: the thing you're sharing is what you got out of the book. (Well, you can fake it, but it's probably about as much work as reading the book legitimately.)

So that's Findings. I don't use it as much as I thought I would, because I'm still trying to draw down my stack of paper books, but when I read a book on my Kindle, it stays read, thanks to Findings.

I mentioned this before, but the last thing I did at Findings was design a web service for them, which they're hopefully working on now. Once the web service launches, you'll be able to write programs that import readings into Findings from non-Kindle sources.

Do it yourself

One final note: If you have a Kindle, connect it to your computer and look on its filesystem. All your highlights are kept in a structured-text file located at documents/My Clippings.txt. This file includes highlights taken from PDF files and other ebooks not recognized by Amazon, which don't get synced to kindle.amazon.com. Even if you don't use Findings, take control of your highlights by backing up this file.

Image credits: McKay Savage, Romana Klee, and André Fincato.

[Comments] (1) Tue Jan 17 2012 10:10 CG Author Commentary #8: "They Came For Our Twinkies": K'chua! Such a useful word. This week, Curic does her part to Keep Austin Weird. Here's the (tiny) Twitter archive from last week.

Some exciting news from the world of commerce: the Constellation Games paperback drops April 17. If you're waiting for the paperback, do yourself a favor and pre-order at the $20 level. Once it's released, the paperback will cost $20 on its own, but if you pre-order, you'll also get a bunch of extras, including three short stories that all pass the Bechdel test.

The seventeenth of April is also the day we serialize chapter 21, "Her". I'm going to keep posting my commentaries once a week along with the serialization, even though a growing number of you will have read the whole book and know how it turns out. Then you'll know how I feel right now!

I'm also thinking of having a celebratory book launch dinner at Hill Country, a famous Austin-area barbecue joint that fortuitiously has a branch in New York City. Let me know if you're interested in attending.

...and we're back from commercial. Here's the commentary for chapter 8:

Stay tuned for the inevitable letdown next Tuesday, when Curic will say, "I did not pee in your sink."

Image credits: (CC) Larry D. Moore and Wikimedia Commons users Solkoll and SeppVei.

<- Last week | Next week ->

[Comments] (1) Fri Jan 20 2012 17:40 Beautiful Soup 4 Beta 2: Thanks to some help from Ezio Melotti I've got the Beautiful Soup test suite passing on Python 2.7 and Python 3.2. Here's a tarball containing the original Python 2 module in bs4, plus the same code autoconverted to Python 3.2 in py3k/bs4.

I'm still not sure about the best way to distribute this package, either while it's beta or afterwards. I'll probably end up creating a new project on PyPi, because otherwise people who install programs that easy_install beautifulsoup will crash due to the module's new name. Does that make any sense?

Anyhow, we're almost at the end of this fitfully travelled road. Once I figure out distribution and rewrite the documentation, a) no one should need to use BS3 anymore if they don't want to, and b) it should be possible to get lxml-like performance or html5lib-like flexibility with a Beautiful Soup API, by actually using lxml or html5lib as the underlying parser.

PS: remember, it's now from bs4 import BeautifulSoup.

[Comments] (1) Sun Jan 22 2012 09:43 Beautiful Soup 4 Benchmark: This is going to go into the Beautiful Soup 4 documentation, but you might find it interesting. It's my first legitimate benchmark of BS4, and the first benchmark of this stuff I've seen since Ian Bicking's excellent 2008 benchmark.

Ezio Melotti pointed me to a list of the top 10,000 domains worldwide, according to some random source. It looked legit, so I wrote a script to download the homepages of the top 200 domains as served to a desktop web browser. My dataset included many pages written in Chinese, Japanese, Russian, Portuguese, Polish, and German.

For every parser I was interested in, I parsed each homepage and timed the parse. This gave me 200 numbers for every parser. To reduce that to a single non-huge number I calculated a mean: how many kilobytes of real-world HTML the parser could process in a second. I also noted each parser's success rate: how many of the 200 homepages it had handled without raising an exception.

Here are the results, ordered by their performance under Python 2.7.

Python 2.7 Python 3.2
Parser Speed (KB/s) Success rate Speed (KB/s) Success rate
Beautiful Soup 3.2 (SGMLParser) 211 100% - -
html5lib (BS3 treebuilder) 253 99% - -
Beautiful Soup 4.0 + lxml 255 100% 2140 96%
html5lib (lxml treebuilder) 270 99% - -
Beautiful Soup 4.0 + html5lib 271 98% - -
Beautiful Soup 4.0 + HTMLParser 299 59% 1705 57%
html5lib (simpletree treebuilder) 332 100% - -
HTMLParser 5194 52% 3918 57%
lxml 17925 100% 14258 96%

Note that the "HTMLParser" tests don't actually produce anything. HTMLParser is an event-based parser, so when the HTML is parsed, nothing comes out because I didn't include any handler code. All the other tests build a parse tree in memory.

Another thing to keep in mind about the html5lib results: html5lib is kind of the opposite of BS4. BS4 always builds a tree of Beautiful Soup objects, but you can tell it to generate that tree using html5lib, lxml, or HTMLParser. Whereas html5lib always uses its own parser, but you can tell it to build a tree of lxml objects, a tree of BS3 objects, etc.

The big surprise for me is that on Python 2.7, lxml is the worst choice for a parser to drive BS4. It's a worse choice than html5lib! How did that happen? I have no idea. I was hoping to cash in on the lxml magic (see below), and it's not working. I need to look into this. Notice that html5lib takes a performance hit from using lxml's treebuilder. If the magic's not in the treebuilder and it's not in the parser, where is it?

Unless I can find that magic and exploit it, it remains the case that if you're paying by the minute for computer time, you should use lxml. It's written in C, and on Python 2.7 it builds a parse tree sixty times faster than BS4, three times faster than a pure-Python parser that does absolutely nothing with the data. Even on Python 3, lxml alone is seven times faster than BS4+lxml. I said stuff like this in the BS3 documentation, but I think I need to be more forceful about it in the BS4 docs.

The good news is that Beautiful Soup is 6-8 times faster on Python 3 than it is on Python 2, and even at its slowest, BS4 is noticeably faster than BS3.

The big caveat is that my definition of "success" is pretty minimal. Just because the parser parsed the file without crashing doesn't mean it will give you a useful parse tree.

Another caveat: on Python 3, I couldn't get HTMLParser to take raw bytes as input, so I ran the data through UnicodeDammit first. I counted this time as part of the parse time. This probably explains HTMLParser's slower speed on Python 3 and its higher success rate.

Update: Argh, I found out about this a year ago. The problem is that Unicode, Dammit is incredibly slow in some cases. Here are the results on 2.7 if I take out the prepare_markup methods in the builders for HTMLParser and lxml, and just assume everything's UTF-8:

Python 2.7 Python 3.2
Parser Speed (KB/s) Success rate Speed (KB/s) Success rate
Beautiful Soup 4.0 + lxml 2287 96%260096%
Beautiful Soup 4.0 + HTMLParser 2069 48%168057%

That's more like it! The problem is that reliability suffers. Both parsers crash in the 4% of cases where it's not UTF-8 but the encoding is declared in a <meta> tag. And there's an unknown number of cases where the data's not UTF-8 but the conversion doesn't crash, leading to garbled data. But at least now I remember this problem.

Also note that on Python 3.2, getting rid of Unicode, Dammit doesn't matter nearly as much. (It doesn't matter for HTMLParser at all.) Presumably Python 3.2 has better built-in support for encoding autodetection.

Mon Jan 23 2012 11:43 To This Basic Game Hedgehogs Are Added: I bought a cute game about hedgehogs, Der Igelwettkampf ("The hedgehog contest"), as a Christmas present for my niece. On Der Igelwettkampf's BoardGameGeek page I noticed that it was classified under the game family "Animals: Hedgehogs/Porcupines". I'd thought "Family" was for boring things like grouping together the endless versions of Ticket to Ride, but turns out it's also used to group together all the games about hedgehogs.

The question then arises: what's the best game about hedgehogs? According to BGG it's Igel Ärgern + Tante Tarantel, a double bill in which Tante Tarantel might be doing some of that work because Igel Ärgern on its own is rated a bit lower.

More importantly, what's the worst hedgehog game? Indubitably it's Hedgehog's Revenge, "The GAME where the hedgehog STRIKES BACK!", whose BGG description includes the now-hopefully-immortal saying "To this basic game hedgehogs are added."

At this point I was on a roll... of the dice! I went back to my now-old BGG data dump, sorted the board game families by how many games they contained, and picked out interesting groupings for use in Loaded Dice. We've got Games about animals (most popular: dogs) Game versions of sports (soccer), and Games about countries (the Roman Empire, in a landslide). That page shows the top-rated game and the lowest-rated game, so get ready to load a lot of cover images.

I did a couple other lists, like media tie-ins (champion: Disney) and "families" that are strongly tied to one single game (the 889-strong "Monopoly" family), but I think the three lists I put up are the most interesting.

Bizarre trivia abounds! Did you know that crows are board game gold? The worst game about crows (The Crow and the Pitcher) has a BGG rating of 6.32, which isn't that bad at all. (Longtime fans will remember the median rating is 6.0).

Did you know there are twenty rodeo-themed games? Apparently you didn't, since only one of those games has more than five ratings. How many wargames take place in Switzerland, a country that doesn't fight wars? Only two: Switzerland must be Swallowed and Zürich 1799.

My data is six months old now and it's starting to show some cracks. There are BGG families for Russia and Antarctica which were created after I took my dataset, so they don't show up in the country list even though most of their games are in my data. After getting the Switzerland idea I ran the "What percentage of a country's games are wargames?" test on all countries, but wargames were drastically undercounted. For instance, all but one "Vietnam" game on BGG is a wargame (the exception being Venture Vietnam), but only 35% of those games were classified under a general "Wargames" category.

But, the lists are still a lot of fun and there are some interesting games in there. I'll leave you with the board game equivalent of the dusty World Book Encyclopedia sitting on the shelf at your grandparents' house: Trivial Pursuit - The Year in Review - Questions about 1992, the worst-rated game (3.90) in the 155-strong Trivial Pursuit family. Also available in 1993 flavor!

[Comments] (6) Tue Jan 24 2012 09:06 CG Author Commentary #9: "Import System": Last week and this week have some of my favorite Twitter bits (e.g.) because the CDBOEGOACC is finally available in English. Sunday night while working on Loaded Dice I realized that one of the reasons I really like playing around with the BoardGameGeek dataset is it's like a real-life CDBOEGOACC.

The flip side is this chapter doesn't have a lot of plot. But hopefully you're okay with that because of all the fun mini-stories like the Sea Level game/food. It's supposed to represent the design phases of a software project, where you're throwing around a lot of ideas but not much is being produced.

Next week is a set piece, and after that the plot won't let up until the cliffhanger that ends Part One. Before that happens, I need to get some solid exoludology in to bring in topics that are important later, like Sayable Spice and Ariel's unsuccessful attempts to translate it.

Before beginning the chapter 9 commentary, I want to get something off my chest about the first sighting of the Farang in chapter 1. In that chapter, Ariel compares their antennacles to the oral tentacles of a "cerebrophage". In the second draft I just out and said "mind flayer". My writing group said I should change it because readers might not know what a mind flayer is. ("Did you mean: mind flower?") Taking their advice to heart, I changed the reference to a made-up reference that nobody will get. Well, at least we're all in the same boat now!

And here's chapter 9. Vent your egg sacs before reading this commentary:

Be sure to tune in next Tuesday, when Dana will say, "This application will terminate due to suspected theft or circumvention."

Oh, and you might want to keep an eye on @Tetsuo_Milk.

Image credits: Flickr user krusty, Guillaume Piolle, and Flickr user CoffeeGeek.

<- Last week | Next week ->

Sat Jan 28 2012 09:58 Fruit to Fruit: Time for another crummy.com Apples to Apples variant (previous editions), this one discovered last week by Pat.

On every green A2A card there's the name of the card, like "Handsome", but there are also three related words, like "attractive", "elegant", "fine". In Fruit to Fruit, you don't read the name of the card. You just read the related words. Sometimes the related words are so similar that you might as well be reading the name of the card, but usually something goes missing (such as the masculinity of "handsome"), leading to funnier red cards being put down. The name of the card is finally revealed during judging.

We had a great time with this and played it in conjunction with the Apples to Placebos variant, even though there were four players. You might think this overkill, but at this point A2A is more a social activity than a game. Anyway, it says right on the box "The game of hilarious comparisons!", so anything that makes the comparisons more hilarious is legit.

While seeing if anyone else had come up with this variant I discovered Apples to Trivial Pursuit, and the improv comedy variant. I also discovered that the game is patented, and that there is an entire patent classification system for "means... by which contests of skill or chance may be engaged in among two or more participants, where the result of such contests can be indicated according to definite rules."

[Comments] (2) Tue Jan 31 2012 09:18 Constellation Games Author Commentary #10: "K.I.S.S.I.N.G.": This is Dana Light's big chapter, and I'm having trouble writing commentary because it's pretty self-contained. A problem is introduced and Ariel solves it by the application of technology. If I hadn't been writing a novel when I came up with Dana, this chapter would have become a short story, maybe part of a sequel to "Mallory". It would have been about the way evil psychologists use game mechanics and the ELIZA effect to manipulate users into spending money, and the way people get real pleasure from spending money on things designed to manipulate them.

Although evil psychology does show up in Constellation Games, I didn't have as much space for it as I'd like. Instead this chapter shows the first grown-up thing we see Ariel do. In a world in which sub-human-level AI has suddenly become very common, Ariel decides to empathize with it.

He doesn't anthropomorphize Dana. Dana doesn't pass the Turing test, she isn't terribly smart or self-aware, but she's capable of happiness and she doesn't deserve to be deliberately made unhappy by evil psychologists. This attitude is what ultimately makes Ariel a hero, not just a POV character. The consequences of his decision to empathize will run through the entire book, and then overflow the book into "Dana no Chousen," and I still don't know when and whether Ariel does the right thing w/r/t Dana. But you gotta have empathy.

Apart from that, I don't have much to say. Here are a few miscellaneous notes:

Tune in next week for action, intrigue, and romance between people at the same level of sentience. It's the only chapter when Ariel will say: "I just have a slight fear of being a tiny speck in the infinite cosmic void." But not the only chapter when he'll think that.

PS: Due to an error on my part, the chapter 9 Twitter feeds ran as part of chapter 8, and chapter 10's Twitter feeds ran last week. This really can't go on, because next week's feeds are tightly integrated with chapter 11. So except for a brief bit of bonus material I just wrote, there will be no Twitter stuff this week. Sorry about that!

Photo credits: Kevin Trotman and Peter Anderson.

<- Last week | Next week ->

[Comments] (4) Thu Feb 02 2012 11:59 easy_install beautifulsoup4: This is an HTMLized version of an email I sent to the Beautiful Soup discussion group, about the impending release of Beautiful Soup 4.

Introduction

When Beautiful Soup was first released in 2004, the state of HTML parsing in Python was appalling. Over the past eight years, things have improved so dramatically that Beautiful Soup's HTML parser is no longer a competitive advantage. I don't want to duplicate other peoples', work, so I'm getting Beautiful Soup out of the parser businesss. Beautiful Soup's job is now to provide a Pythonic screen-scraping API on top of a data structure created by a third-party parser.

This will be Beautiful Soup 4, and I've been planning it for years. With help from Thomas Kluyver and Ezio Melotti, I've now met the three main goals of Beautiful Soup 4:

  1. Make a single codebase that works under Python 2 and Python 3.
  2. Stop using SGMLParser (removed in Python 3) and make it possible to swap out one parser for another.
  3. Support two major Python parsers (lxml and html5lib) as well as Python's (not currently very good) batteries-included parser, html.parser.
The first version of BS4 is almost ready for release, and I'd like you to test it out, if you haven't already. I still to fix some things, in particular some performance problems. But, note that even with the performance problems, BS4 is faster than BS3 across the board.

On Python 2 or Python 3 you can install the BS4 beta with this command:

easy_install beautifulsoup4

You can also get the source tarball.

The documentation has been completely rewritten. You may find the section on porting BS3 code to BS4 especially interesting.

There are three major things I'd like your feedback on before completing the release.

Hall of Fame

The BS3 documentation lists open-source projects that use Beautiful Soup. I stopped maintaining this list many years ago because there are hundreds of these projects, and since most of them are screen-scrapers, they're pretty ephemeral.

I'd like to bring this feature back as a "hall of fame", featuring applications of Beautiful Soup that grab a reader's attention. People who used Beautiful Soup in a high-profile way or to tackle a big issue. Projects that are interesting to hear about even if the software doesn't work anymore, or uses an old version of Beautiful Soup, or if Beautiful Soup was used internally and the public only saw the results.

My bias is towards projects having to do with space, science, journalism, politics and social justice. Here are some examples so you know the kind of thing I'm thinking of:

If you did anything of this sort, or know of someone who did, I'd like to hear about it.

Do you prefer lxml or html5lib?

Right now, the parser ranking goes lxml, html5lib, html.parser. I like lxml because it's incredibly fast and it can parse anything. But I'd like to see what you think of the trees it generates. Would html5lib, with its web-browser-like heuristics, be a better default?

substitute_html_entities

BS3 had a number of overlapping and inconsistent ways of turning HTML/XML entities into Unicode characters, and possibly turning Microsoft smart quotes into HTML entities at the same time. In BS4, all this stuff is gone. HTML and XML entities are *always* converted into Unicode characters.

This is great but there's one problem: output. If you want to turn those Unicode characters back into entities when outputting as a string, you need to call soup.encode(substitute_html_entities=True), which is a little clunky. I'm thinking of adding an output_html_entities attribute that you can set on a soup or tag to control whether this substitution happens. Do you like this idea?

I think I also need to ensure that characters like "&" and "always converted to XML entities on output, even though this will hurt performance a bit.

Conclusion

What you install with easy_install beautifulsoup4 is a beta release. If I hear of a problem soon, there's still time to fix it, even if it means a major change to the API. So please try it out and give me feedback.

Mon Feb 06 2012 17:55: Earlier I ran some speed/accuracy tests of Beautiful Soup driven by various parsers. Python's built-in HTMLParser scored very poorly, parsing only 52% (Python 2.7.1) or 57% (3.2.2) of my test pages without raising an exception. Well, Ezio Melotti, the maintainer of HTMLParser, has been working for a while on improving HTMLParser's handling of bad HTML. Most of this code is in Python 3.2.2, so I should have been getting the benefit, but it wasn't working for me because of a semi-related bug in HTMLParser, which is fixed in the as-yet-unreleased 3.2.3.

After talking with Ezio today, I was able to monkeypatch BS4 to avoid the bug in 3.2.2. This means on Python 3, BS4 with no external parser installed will give reliability comparable to BS4+lxml (98% versus 99%). It's still about 50% slower, though, parsing about 1300 kb of HTML per second, versus 2100 kb/second for BS4+lxml.

[Comments] (4) Tue Feb 07 2012 08:48 Constellation Games Author Commentary #11: "Launch Title": Love those title puns! This blockbuster episode sends Ariel TO THE MOON and introduces two major new characters, Tetsuo Milk and Ashley Somn. Also a minor but important character: Linda Blum, Ariel's mom.

Here's last week's Twitter archive, which ran two weeks ago due to my own errors. Twitter service has now resumed, but because this plot arc is so compressed (the rest of Part One crams two weeks of frantic activity into five weeks of real time), most of it is going to come out on Tuesdays and Wednesdays. Don't be afraid, I'll be here the whole time with long-winded commentary:

What a huge commentary, and this plot arc's just getting started. Be sure to tune in next week, when Ariel will say, "I do not use sex to maintain social cohesion."

Image credits: Andy Bernay, Joe Mabel, Linda Salzman Sagan, unknown.

<- Last week | Next week ->

Wed Feb 08 2012 11:02 Beautiful Soup 4 Beta 4: Beautiful Soup 4 beta 4 is out! You can install it with easy_install beautifulsoup4 or pip install beautifulsoup4. You can also download the tarball or check out the Bazaar repository.

Big changes:

Thu Feb 09 2012 17:11 Beautiful Soup 4 Beta 5: Just going to link to my description message this time. Today I focused on clearing out the bug backlog. It's mostly minor stuff, but I'd like opinions on one change, relating to how a tag is treated if it has multiple CSS classes.

Mon Feb 13 2012 09:13 nanDECK: I have a little side project creating a print-and-play board game. The game has a lot of cards, but I don't need to design each card individually--I can generate them programatically. Or I could, if I were capable of writing the program.

First I tried ReportLab, the Python library for making PDFs. I'd used it for the sadly-now-defunct Pocket Wisherman, and I thought it would be perfect for putting lots of little squares on a piece of paper.

Not so fast! The Pocket Wisherman puts lots of squares on a piece of paper, but in that program text flows from one square to another. That can't happen on a playing card. The closest I could come with ReportLab was a table, and since I couldn't add spacing between the table cells the way you can in... HTML...

It was easy to get something in HTML that looked right on screen (these cards are pretty simple), but not so easy to get them to look good when printed. So I went back to searching for tools optimized for card design. I delved deep, past many people talking about the best way to manufacture cards for print-and-play-games, and then I found nanDECK by Andrea Nini.

I'm gonna complain a lot about nanDECK so I want to make it really clear that nanDECK solved my problem. In about an hour I went from having two failed Python scripts and no cards, to having cards as nice as my design skils could make them. If I got some design help from someone else I can make the cards nicer still, from within nanDECK.

Now, let the complaining begin! Actually, I'm not even gonna complain. I'll just phrase my complaints as helpful hints. nanDECK is a Windows IDE for a domain-specific markup/programming language. It runs fine in WINE. The prominently-linked manual is actually a reference guide--tutorials and examples are linked further down the homepage.

The interface features so many buttons that the "visual edit" button might get lost in the shuffle (ha), but that button is going to help you so much. You won't have to remember all the arguments to the language directives, and you can lay out elements visually on the card rather than guess at measurements over and over again. In the end I couldn't get the linked-data feature to work (possibly an interaction with WINE), so I figured out the layout for a single card within nanDECK and then wrote a Python program to generate the nanDECK script for my entire deck.

Whew! Kept it positive. If you want to design cards for a game, and you don't want to lay them all out manually (which you shouldn't), I think nanDECK is your best option. Thanks, Andrea Nini!

Tue Feb 14 2012 09:00 Constellation Games Author Commentary #12: "Monsters From Space": Welcome to another chapter full of laughter and embarrassing faux pas. This week we learn why Curic scanned Ariel's house, and get our first glimpses of the ancient, not-particularly-wise Ip Shkoy.

Before the commentary begins, I want to bring up something serious that I could save for next week but I don't want to. Dr. Janice Voss died on February 6 at 55. She was a scientist, a NASA astronaut who flew on five shuttle missions, and later the science director for the Kepler Space Telescope. She was a big science fiction fan. I met her once in 2007, in what was certainly the highest-wattage dinner I've ever attended (photos), and she made a huge impression on me.

The only major character in Constellation Games you haven't met yet is an astronaut, Tammy Miram. She's introduced next week. If I hadn't met Janice Voss, Tammy Miram would not exist, and I have no idea what the novel would look like from next week on.

I don't mean that Tammy Miram is "based on" Janice Voss, or that the character is a way to tell Janice's story in a fictional setting. I only met Janice Voss once and I have no idea what her story would look like. (Spoiler) Also, Janice was a very well-adjusted person, and Tammy is not. But a dinner-length conversation with Janice was enough to move the societal role of "NASA astronaut" out of my mental category "archetypes useful in science fiction stories" and into "interesting jobs I can give to my characters."

R.I.P., Janice Voss. Ad astra per aspera.

Here's last week's Twitter feed, as it was meant to be seen (i.e. without a weird UTF-8 encoding issue). And now, this week's commentary:

Tune in next week, when Curic will say, "Infiltration? Cold reading? Propaganda? Torture? Extracting false confessions?"

Image credits: NASA, NASA again, Kabir Bakie, Alain r.

<- Last week | Next week ->

Thu Feb 16 2012 09:16 Beautiful Soup 4 Beta 6, Beautiful Soup 3.2.1: There are two ongoing serials here at crummy.com: Constellation Games and Beautiful Soup 4. Here's the announcement message for the latest installment in the latter saga.

The big news is a new release of the 3.x series, Beautiful Soup 3.2.1. This fixes a pretty bad problem that can let through cross-site scripting attacks if you use Beautiful Soup to sanitize HTML. If that's you, you should upgrade ASAP.

That was certainly worth fixing, but I don't do much work on Beautiful Soup 3 anymore. I mean, if I fixed every bug in BS3, I'd have... Beautiful Soup 4, which is now almost done. All the bugs are closed out. There's one more big feature I may add, and some minor cleanup I want to do, but mainly I want to make sure people are comfortable with the new API.

Thanks to Stefano Rivera, BS4 is now in Debian unstable and Ubuntu Pangolin, as beautifulsoup4. So the clock is ticking on freezing the API. This would be a great time to try to port your BS3 scripts to BS4, and let me know how difficult it was and what you had to change.

[Comments] (2) Mon Feb 20 2012 07:32 Where's That Golden Age?: A couple weeks ago Samuel Arbesman posted an entry to Wired's science blog called "How to search for the golden age of television", an entry that's been driving me crazy since I read it. Not because I disagree with his analysis of the IMDB dataset, but because I don't like his starting point. Arbesman uses "each television show’s running time, in number of episodes, as a very rough proxy for quality". It's true that there's probably a positive correlation, but that metric has a couple problems. First, it severely discounts the present. A show on the air today may have several seasons to run, but we don't know that yet, so it'll look worse than an old show of equivalent quality. Second, the IMDB dataset features a much more direct proxy for quality: user ratings.

I don't think ratings are a great proxy for quality--a look at the highest-rated TV shows will put a stop to that nonsense. And the run length of a show is at least an objective fact. But I think our collective opinion of a TV show today is a better proxy of quality than how long the network was originally willing to keep it going. And if you use ratings, I think you can get closer to answering the question "what would a golden age of television look like?"

My guess is, Arbesman didn't use ratings because it's kind of annoying to get that information out of the IMDB dataset. But I'd already done a lot of work on the dataset for The MST3K-IMDB Effect, so in this post I crunch the numbers my way and see what falls out.

If you're expecting controversy, I can't provide. My findings don't contradict Arbesman's, they just provide a different way of looking at the data.

Step 1: Get the data

(If you're impatient, you can skip to the graphs.)

It all starts with IMDB's plain-text data dumps. I downloaded release-dates.list.gz and ratings.list.gz from the FTP site. I also downloaded distributors.list.gz, but it turned out that data wasn't useful.

Step 2: Identify shows, episodes, and air dates

release-dates.list lists all movies, TV shows, and episodes of TV shows. TV shows are in quotes, and episode names are in curly brackets. 

Point Break (1991)					USA:12 July 1991
"Star Trek: Voyager" (1995)				USA:16 January 1995
"Star Trek: Voyager" (1995) {Caretaker (#1.1)}		USA:16 January 1995

Unfortunately, web series look just like TV shows, which is going to mess with the data for recent years: 

"The Angry Video Game Nerd" (2006) {A Nightmare on Elm Street (#1.13)}	USA:31 October 2006

I tried some tricks to get rid of web series, like only considering shows with a listed television distributor (distributors.list), but there are tons of dinky cable reality shows that have exactly the same data characteristics as web series. So I'm leaving them in. Just know that when I say "TV shows", I'm talking about TV shows + web series.

To make the initial dataset smaller, I used grep to remove everything except the US premieres of TV shows, and of episodes of TV shows. (And web series.) Then I wrote a Python script that turns this information into a picklable data structure.

The script ties a show to all of its known episodes, and parses out each episode's release date along with the premiere date of the show itself. I want to know every year in which an episode of the show premiered in the US. This has some problems--it makes the original "Star Trek" show up as a 1988 show because that's the first time the original pilot was aired--but they're pretty minor.

Step 3: Add the ratings

Now I know when every show started, and in many cases I know every year a show was on the air. In the next step I load in another file and add ratings to shows and episodes.

Ratings are kept in ratings.list. They look like this: 

      0000001212   11245   7.5  "Star Trek: Voyager" (1995)
      0000012111    1558   7.1  "Star Trek: Voyager" (1995) {Caretaker (#1.1)}

There's lots of cool stuff here like a histogram (0000012111 means 10% of people rated the premiere of Voyager a 6, 20% of people a 7, and so on), but what we're after are the IMDB ranks: 7.5 stars and 7.1 stars in this case.

Unfortunately, there's a lot of boring stuff in ratings.list like the top 250 movies. Fortunately, I already wrote code to parse this file during my investigations into the MST3K-IMDB effect.

Step 4: Graphs!

Now I'm going to break out numpy and pychart. Let me start with a calibration run, a graph Arbesman also did. How many shows were on the air in a given year?

Pretty similar to Arbesman's graph. My graph doesn't go down at the end, because I cut the data off at 2011, the last full year of data. I also start later, with the first year for which there were five rated TV shows. I'm picking up some shows he's not, possibly because I'm counting a show in every year it aired, possibly because I'm picking up shows that don't have any episodes listed on IMDB, possibly because he found some way I didn't think of to exclude web series. But it's a similar shape.

Now here's the graph you've been waiting for: mean rating over time:

It's a sad story of precipitous drops in quality: one between 1959 and 1980, one between 1999 and 2005. By this measure, 2005 was the worst year in television history. If you only looked at mean rating over time, you'd say that there was one golden age of television, from 1955 to 1965, and that the 1980-2000 period was a period of stagnation interrupting an otherwise steady decline.

The graph of median rating over time tells much the same story, so I won't transclude it, but you can follow this link to see it.

But, mean rating isn't the whole story. Let me pull out the only statistics trick I know: look at the standard deviation of the ratings over time.

1959, the year with the highest mean rating, is also a year of extreme homogeneity. Less than one star of difference separates the very good shows from the very bad shows. After 1959, the good shows get better, and the bad shows get worse, relative to the mean. In 1980 the standard deviation was 1.37 stars, and in 2011 it was almost two stars. Remember that ratings are not normally distributed, so two stars is quite a lot. (Even one star, as in 1959, ain't nothing.)

Combine this with the skyrocketing number of shows (which begins in the late 90s and goes into overdrive once we start counting web shows) and you can see how that 2000-2005 decline happened. Over 1300 distinct shows aired in 2005. Of course the mean show is going to be crap! The amazing thing is that things have gotten better since 2005, even as we now make over twice as many shows per year. (And web series! Can't forget those!)

Another factor is that people aren't even bothering to rate the bad shows. Here's the percentage of shows that aired in a given year that don't have IMDB ratings because they haven't gotten enough votes. For 2011, this was a majority of shows!

Old shows aren't rated because nobody remembers them. New shows aren't rated because... well, I did a bunch of spot checks, and they fall into three categories. 1) web series, 2) shows that were never aired and maybe never even produced, 3) crap. Only #3 can properly be considered part of "television". The mean rating would certainly be lower if every show had a rating, but I don't know how much lower.

That's where we stand: television is bad, and it's getting worse. That trend may have been reversed recently, or the decline may have been masked by web shows with passionate fans, or things may have gotten so bad that people stopped even bothering to rate the crap. But! Would you exchange the television of today (mean rating: 6.2) for the television of 1973? (mean rating: 7.3). I wouldn't, and I don't think you would either. What's going on?

Well, we don't watch the mean television show. We only watch the good shows. (If you've read this far, I'm gonna go ahead and make that assumption.) And if you look at the good shows, the picture looks very different.

Here's what the shows look like one standard deviation above the mean. This is basically the top 16% of shows:

At the high end, the decline in quality is reversed in the 80s and early 90s. The gains are undone in the late 90s (2005 is still terrible), but then quality shoots back up. This is very similar to Arbesman's graph of show length over time.

What if you're even more selective? Let's graph the value 1.5 standard deviations above the mean for each year. I don't know what percentile this would correspond to, but it's something like the top 5%. This is the very best stuff you can find on TV in a given year:

This graph, I think, is the best answer to "what would a golden age look like"? It would look like the 60s, when there were three channels under tight quality control, and you could turn on the television at any given time and probably find something good. Or it would look like right now, when a huge number of shows are being produced, and it's easy to be a snob and only watch the very best. This is why we don't remember 2005 as being the worst year of TV in the history of the medium, and this is why I'd never trade today's TV for 1973's TV, even though 1973 looks pretty good on that graph.

So, there you have it--another way of looking at the IMDB data. More to come! Next up: a little thing I like to call "Worst Episode Ever".

Tue Feb 21 2012 09:10 Constellation Games Author Commentary #13: "Your Day Job": The lucky chapter thirteen introduces the novel's last major character, Mission Specialist Dr. Tammy Miram. She gets right to work, kicking off a subplot that won't be wrapped up til the very last chapter. Let's look at a bunch of commentary, most of which is about her. But first, Twitter archive from last week! Okay, here we go:

That's all I got. Stay tuned for the huge chapter 14, a chapter full of deepening mysteries and used game trade-ins, the chapter where Ashley finally says, "Ariel was distracted by my beautiful ovipositor."

Image credits: NASA, Mark Phillips, Allen Garvin.

<- Last week | Next week ->

Tue Feb 21 2012 19:33: Remember when this weblog used to be about fun links? I don't either, but I think it was somewhere in there. Well, check this out: last year when I went to PAX my most enjoyable experience was the panel "Videogames Antiques Roadshow." It worked just like you think: people would bring old game stuff up on stage, and distinguished collectors would estimate the value of the old stuff. Here are some pictures from that panel. In fact, you can see me in the second photo, fourth row center.

Kind of got distracted there--the point of this post is not to look at a crowd scene that includes me. I meant to say that they brought the panel back at PAX Prime, and this time there's video. And it's now called "Retrogaming Roadshow", possibly due to trademark issues. In addition to bringing to light cool bits of history like the PCjr edition of M.U.L.E., I love the way these panels illustrate the social construction of value. Highly recommended if you've got an interest in this stuff.

[Comments] (6) Wed Feb 22 2012 16:39 Worst Episode Ever: Time for some more IMDB fun. Last time I looked at whole years of television. This time, I'll graph the ratings for individual episodes of TV shows. Can we watch shows get better or worse over time?

We sort of can. The problem is that only a true fan bothers to go to IMDB and rate individual episodes of a TV show. So you can't really trust the episode ratings--they're too high. But we can visualize trends in show quality, as percieved by the fans.

For these visualizations you want long-running series with lots of die-hard fans. So let's start with Star Trek:

(Note the very last data point in that one. That's the series finale, which everyone hates.)

There's a lot of scatter, but you can generally see the common Star Trek pattern of the show getting better as the ensemble cast comes together. Except for the original series, which ended with a lousy season. Now let's look at another nerd favorite, "Buffy the Vampire Slayer":

Beth requested that one. I've seen exactly one episode of Buffy so I wasn't expecting anything in particular. It looks like a show that's consistently good, but wildly inconsistent within the bounds of "consistently good". It doesn't really get better over time. Maybe the Voyager and DS9 graphs look the same to someone who's not a Trek fan.

But compare "Mystery Science Theater 3000", which gets drastically better over time. When I was younger I would have disputed this finding, but now I basically agree with this graph:

I did a lot more graphs, but I'll just show two more. Here's the graph for "The Simpsons", a very long-running show with a very fickle fan base (see title of this post):

Wow! I love this graph! I don't know enough about the history of the show to name the historical trends, but I'm pretty sure a Simpsons fan will be see a big part of their life history reflected in this graph.

I wanted to see if this sort of coherent shape was just an artifact of the fact that "The Simpsons" has been on the air for over 20 years, so I graphed another long-running show notorious for huge variation in quality, "Saturday Night Live":

You can definitely see where things went wrong, but even within a season there's huge variation in quality. The Simpsons is created by the same people every week, where SNL has two wild cards every week: its guest host and musical guest. And since it's sketch-based, three good or three awful minutes can make or break the entire episode.

Next up, the third and possibly final part of this analysis, in which I'll pit fans of a show against the general public.

PS: For the record, according to IMDB data, the actual worst episode ever of "The Simpsons" was #9.11, "All Singing, All Dancing".

Update: People in comments had questions I can't answer because I only know how to do very basic statistics, but they also had questions about how many people rated the episodes, which I can answer. This table shows how many people have rated each series as a whole, as well as the median and mean numbers of ratings for every episode that has any ratings. I also included how many people rated the first episode, how many rated an episode in the middle, and how many rated the last/most recent episode.

Series Series ratings Show ratings (median) (mean) (std)First showMiddleMost recent
"Buffy the Vampire Slayer" (1997) 34564 498 553.41 224.888625111091
"Enterprise" (2001) 8843 140 189.27 242.282397130152
"Mystery Science Theater 3000" (1988) 6650 57 65.54 47.412178131
"Saturday Night Live" (1975) 10151 15 19.86 15.651121160
"Star Trek" (1966) 12695 419 480.95 222.836683891923
"Star Trek: Deep Space Nine" (1993) 9779 172 188.32 107.371501151290
"Star Trek: The Next Generation" (1987) 16974 329 375.62 354.4921893184580
"Star Trek: Voyager" (1995) 11245 153 169.08 110.961558177348
"The Simpsons" (1989) 15578 319 355.07 173.09221430996

So SNL actually has very few ratings per episode, while The Simpsons is on par with ST:TNG. It's common for the first episode and the finale to have many more ratings than others. And here's a graph of the number of people who have rated "The Simpsons" over time:

Fri Feb 24 2012 11:17 Beautiful Soup 4 Beta 8: I didn't even mention beta 7 on NYCB because it was oriented towards getting rid of test failures. Test failures that had a lot to do with what versions of what parsers were installed, but nothing to do with whether or not Beautiful Soup itself was broken.

Beta 8 adds very basic namespace awareness. By "basic" I mean:

  1. Handle documents that include namespaced tags and attributes without crashing or mangling the document on output.
  2. If the parser provides namespace information for a tag or attribute, store it for the user's reference instead of discarding it.

That's it. No one responded to my request for namespace-related feature requests, so I'm doing the bare minimum.

[Comments] (2) Mon Feb 27 2012 12:59 Incorrectly Regarded As Good: In this third and final part of my IMDB data adventure, I want to switch from graphs to tables, and shed light on the eternal struggle between fans and non-fans. If fans are the ones who care enough to rate individual episodes, non-fans are the ones more likely to rate the show as a whole. I looked at every show that has at least 100 ratings, plus at least 100 rated episodes. I divided the show rating by the mean episode rating to get a "fan appreciation quotient". (I used mean because the show rating itself is a mean, calculated by IMDB.)

Shows with high FA quotients are more beloved by fans than by the general IMDB-using public:

FA quotientShowShow ratingMean episode rating
1.63"Entertainment Tonight" (1981)3.76.0
1.34"Melrose Place" (1992)5.77.6
1.28"Dynasty" (1981)5.97.6
1.28"The Rosie O'Donnell Show" (1996)3.64.6
1.26"Mighty Morphin' Power Rangers" (1993)6.07.5
1.24"Full House" (1987)6.07.4
1.20"Ghost Whisperer" (2005)6.47.7
1.20"Fear Factor" (2001)4.95.9
1.16"Dharma & Greg" (1997)6.77.7

Note that since this is a quotient, it has nothing to do with the magnitude of the ratings. "The Rosie O'Donnell Show" got terrible ratings even from the people I'm assuming are fans; it's just that the show as a whole did even worse.

OK, smarty pants, what about a low FA quotient? How can a show appeal more to the mainstream than to its own fans? Well, I think a low FA quotient means that a show seems better in retrospect than it actually was. Or, more positively, it means that a show was more than the sum of its parts. Either way, here are the shows with the lowest FA quotients:

FA quotientShowShow ratingMean episode rating
0.78"Bonanza" (1959)7.35.7
0.78"NYPD Blue" (1993)7.76.0
0.77"In Living Color" (1990)7.96.1
0.75"Teenage Mutant Ninja Turtles" (1987/I)8.16.0
0.73"Gunsmoke" (1955)8.05.8
0.71"What's My Line?" (1950)8.96.3
0.71"Saturday Night Live" (1975)8.15.7
0.68"House of Payne" (2006)2.51.7
0.62"Ellen: The Ellen DeGeneres Show" (2003)7.34.6
0.60"MADtv" (1995)6.74.0

Look how much sketch comedy there is on that list! I think I'm on to something. Two of my favorite shows, ST:TNG and MST3K, also have low FA quotients of 0.83 and 0.84 respectively.

And right in the middle we have the shows that are exactly as good (or bad) as you remember them:

FA quotientShowShow ratingMean episode rating
1.00"Becker" (1998)7.67.6
1.00"Cold Case" (2003)7.57.5
1.00"Dancing with the Stars" (2005/I)4.84.8
1.00"Hercules: The Legendary Journeys" (1995)6.66.6
1.00"MacGyver" (1985)7.87.8
1.00"Mission: Impossible" (1966)8.18.1
1.00"Project Runway" (2004)6.66.6
1.00"Rawhide" (1959)8.28.2
1.00"The Practice" (1997)7.77.7

Haters

Similar to the struggle between fans and non-fans is that between fans and antifans, a.k.a. haters. Fans of a show will give it a very high rating, and haters will give it a very low rating. We can detect this by looking for shows whose ratings have high standard deviations. IMDB doesn't make the standard deviation available directly, but it does provide a ten-character ASCII string that represents the distribution of ratings.

Star Trek: The Next Generation has been rated 16,974 times. Its rating distribution string looks like this: "0000000124". The "4" means that the number of ten-out-of-ten votes is somewhere between 40% (6,790) and 49% (8,316) of those 16,974 votes. The "2" means that between 20% and 29% of the votes are nine-out-of-ten, the "1" means that between 10% and 19% of the ratings are eight-out-of-ten. The zeroes mean that the other star ratings account for between 1% and 9% of ratings each. You can see the conversation about TNG is very heavily dominated by the fans.

I reconstructed the original rating distribution very roughly by treating the character "0" as five percent of the total votes, "1" as fifteen percent, and so on, up to "9" meaning 95 percent of the votes. How rough is the reconstruction? Well, for TNG, the reconstructed distribution has 20,363 data points, where the actual distribution (whatever it is) only has 16,974.

When I take the standard deviation of the reconstructed distribution for ST:TNG, I get 2.74 stars. This particular number is not trustworthy because of the assumptions made in reconstructing the distribution. But by making the same assumptions for every show, we can see which shows are the most divisive. Here are the shows with the largest standard deviations, among all shows with more than 1000 ratings:

Standard deviationShowRatingVotesDistribution
3.85"Laguna Beach: The Real Orange County" (2004)3.721703000000003
3.76"Barney & Friends" (1992)3.712554000000002
3.76"Jon & Kate Plus 8" (2007)5.427162000000004
3.76"The Hills" (2006)3.358284000000002
3.75"Shake It Up!" (2010)4.810132000000003
3.75"Paranormal State" (2007)4.514383000000002
3.75"Flavor of Love" (2006)4.512542000000003
3.75"The Simple Life" (2003)3.429563000000002
3.75"The Jerry Springer Show" (1991)3.916313000000002
3.75"Jersey Shore" (2009)4.531303000000002
3.75"Hannah Montana" (2006)3.919273000000002
3.75"Big Brother" (2000/III)4.016213000000002

That list has a bottom, but it's not interesting--it's the shows about whose quality there is general consensus. All right, here it is:

Standard deviationShowRatingVotesDistribution
2.38"Mork & Mindy" (1978)7.017460000012211
2.38"Around the World in 80 Days" (1989/I)6.914460000012211
2.38"Amazing Stories" (1985)7.314670000012211
2.38"V" (1984)7.225570000012211
2.38"Crusade" (1999)7.011330000012211
2.34"Impact" (2008)5.616330000111000
2.31"Nuremberg" (2000)7.227540000012311
2.22"Moby Dick" (1998)6.519670000112100
2.15"Golden Years" (1991)5.014590001211000
2.12"Covert One: The Hades Factor" (2006)5.710110000122000
2.12"The Andromeda Strain" (2008)6.158580000122100

I experimented with a different mapping of the distribution, e.g. saying that "0" meant 2 percent of the votes, "1" meant ten percent, "2" meant 20 percent, and so on. This made the standard deviations into smaller numbers, but it didn't change the ordering of shows very much.

Variability

We can also measure how much a show varies in quality by taking the standard deviation of the ratings given to its episodes. For this I looked at shows which had at least ten episodes that had been rated at least ten times. Here are the results—the "Variability" is the standard deviation of the episode ratings, in IMDB stars.

VariabilityShowShow rating
3.32"The Tonight Show Starring Johnny Carson" (1962)8.3
2.74"The Late Late Show with Craig Ferguson" (2005)8.6
2.62"Jimmy Kimmel Live!" (2003)6.4
2.60"Beauty and the Geek" (2005)5.9
2.37"Late Night with Conan O'Brien" (1993)8.5
2.23"Late Show with David Letterman" (1993)6.9
2.04"Silk Stalkings" (1991)6.1
1.89"The Tonight Show with Jay Leno" (1992)5.3
1.87"Superboy" (1988)6.3
1.70"Duck Dodgers" (2003)8.2
1.68"The Virginian" (1962)7.7
1.68"Ellen: The Ellen DeGeneres Show" (2003)7.3

There's a lot of late-night talk here. If I loosened the restriction on number of ratings per episode, I also got a lot of soap operas (most of whose episodes have no ratings at all).

And here's the bottom of that list: the most consistently good (or, in theory, bad) shows on TV:

VariabilityShowShow rating
0.20"Day Break" (2006)8.3
0.20"Lucky Louie" (2006)8.1
0.20"Boardwalk Empire" (2010)8.9
0.20"Hung" (2009)7.5
0.19"Outsourced" (2010)7.7
0.19"The Ben Stiller Show" (1992)7.3
0.18"Happy Endings" (2011)8.1
0.18"Lewis" (2007)7.9
0.08"Planet Earth" (2006)9.7

I looked into the variability of the ratings distribution for individual episodes, hoping to find the most/least controversial TV episodes ever aired, but most of what I found looked like ratings juking. For instance, "Friday Night Lights" and "The Shield" show a hater/fan dynamic on the episode level: some people rating every individual episode very low and others rating every episode very high.

I think that's enough for now, but I'll come back to the data as I have more ideas, and maybe I'll even learn more than basic statistics for you.

Mon Feb 27 2012 22:26: Last year I learned about the LEGO model of the International Space Station. Today I learned that sometime last year Satoshi Furukawa assembled the LEGO ISS on board the real ISS. In a glovebox, so the pieces wouldn't fly away. There are educational videos.

Tue Feb 28 2012 10:04 Constellation Games Author Commentary #14: "The Wave Function Of The Universe": Damn, the time is flying. Part One ends in three weeks. And today there's a lot of non-commentary stuff I want to talk about, so the commentary itself will be pretty light.

First, I want to tell you that Jeremy Penner implemented Chapter 5's Gatekeeper in HTML5 for the 2012-in-One Glorious Developers Konference Kollection. You can play it online. I wouldn't classify Gatekeeper as fan art, though Jeremy is a fan, because he did it for me as a Kickstarter reward. But either way, it's pretty great!

Second, I want to talk about the process of designing the cover art. You don't have to read the book to "get" the cover—that wouldn't exactly help sales—but the design details are a product of in-world thinking. And at this point you've seen enough of the universe that I can go through that thinking without big spoilers.

The cover is by Chris Sobolowski, who wants me to mention his email address and let y'all know that he's available for graphic design work. So if your contract with Jenny Gallegos fell through due to her being a fictional character, contact Chris, who's a real person.

The process went like this: first, Kate and I laid out a huge number of cover ideas (some of which I've mentioned in earlier commentaries), and decided we wanted a cover themed around the ET hardware. At this point Kate got Chris involved, and Chris came up with a couple sketches that made the book look like a handheld computer. Here's one of them, next to the cover we ended up using:

I've spent months looking at the finished cover instead of this first draft, and what strikes me now is how similar they are. But what struck me at the time was that the computer looks like a piece of military hardware. It's dark and brooding, like one of Batman's gadgets. I wanted something flashy and colorful, like one of Batman's gadgets. Or like the Hitchhiker's Guide, to not use the same analogy twice in a row.

But I'm not the artist, and I'm also not a writer who thinks he can do the artist's job. So instead of demanding specific changes I wrote two different in-world histories for this handheld computer, and presented them to Chris.

In one story, the computer was a product of the Dhihe Coastal Coalition, the Farang civilization that produced the Brain Embryo. This explained the military appearance, and it had certain implications for changes he should make to the design. (E.g. making the buttons much smaller).

In the other story, the one we went with, the computer is an Ip Shkoy ripoff of a Dhihe design, produced by Perea, the conglomerate that also put out the game reviewed in this chapter, A Tower of Sand. (The glyphs on the final cover's buttons say "pe" "re" "a".) This has its own implications: the colors are now so bright as to verge on the garish, making the computer look more like a consumer product and making the book look more like a comedy and less like a technothriller.

In this story, the only remaining Farang detail is the Brain Embryo-esque mother-of-pearl finish. Stylistically it's reminiscent of the wood grain on an Atari 2600, but it tells a different story. When you were a kid, electricity was an advanced technology. Then all these space aliens showed up handing out blueprints for handheld computers. You want something that looks as different as possible from the wooden toys you had when you were young.

The cocktail cabinet-like second set of controls at the top comes from this bit I wrote about the computer's social context:

Why would the notoriously social Ip Shkoy build a single-user game system? It probably has something to do with sex. Imagine this portable computer as a product for the swinging bachelor, full of "sophisticated" adult games to break the ice, contact management applications to replace one's little black book, and a vibrator peripheral for when the night's inevitable failure leaves you alone in your crappy apartment.

This device would need to have some two-person controls, so that you can play those icebreaker games with your would-be conquest, but the overall feel would be that this is my computer, but I might let you use it.

Chris took the Ip Shkoy story and produced something that's very close to the final cover. Here's another side-by-side comparison:

After that, there was a lot of back and forth on trivial details like how much and what kinds of wear should be visible on the computer. Around this time Adam was designing the Pey Shkoy language for Tetsuo's Twitter feed, so I asked him to also design a script for use on the cover. This is also the point where Kate got the idea for a "Berlitz Traveler's Lexicon," which became "Pey Shkoy Benefits Humans."

I haven't mentioned the back cover, but at this point I think I've reached or exceeded the limit on how long this discussion can be without getting dull, so let's move on to chapter 14 commentary. But not before linking to the archive of last week's Twitter fun.

OK, that's plenty for this week. Next week: IT BEGINS. Oh, and Curic says, "Silence, puny human!"

Image credits: Jeremy Penner, Chris Sobolowski, NASA

<- Last week | Next week ->

Tue Feb 28 2012 10:51 Beautiful Soup 4 Beta 9: The latest beta is the first one I'm calling a release candidate, so if you've been waiting to try it out, now's your chance.

Fri Mar 02 2012 09:12 Beautiful Soup 4 Beta 10: Hey, two in a row. The "release candidate" thing was a lie; the big change is that I ported and incorporated Simon Willison's soupselect project. So you can now combine the Beautiful Soup API with CSS selectors. Except I just realized that I ported an old version of the code, so I'll be doing another release. Anyway, here it is in the BS4 docs.

[Comments] (4) Sat Mar 03 2012 09:59 No Sirens On Titan: Recently I read a 2001 book by Jeffrey Kluger, Moon Hunters, about unmanned missions to non-planetary Solar System bodies. It was a little out of date but there was a lot of good early stuff, like how every time one of the Ranger missions failed, Khrushchev would use it as a laugh line in a speech. ("The Soviet pennant on the moon has been awaiting an American pennant for a long time. It is starting to become lonesome.")

And the book's its very out-of-dateness reminded me of something I'd forgotten about. The Cassini probe was launched when I was in college (I remember a flyer for an anti-Cassini protest at JPL, the point being that Cassini might explode on the launchpad like a Ranger and contaminate Cape Canaveral with radioactivity), and in Moon Hunters it's on its way to Saturn. But now it's there, like a jump cut!

And (this is the part I'd forgotten) Cassini included a probe, Huygens, whose job it was to land on Titan. That's why it was always called "Cassini-Huygens" on the news. It wasn't just NASA and ESA fighting over the name of the mission. And Huygens was instrumented with a microphone. Wow!

So I went to the Internet looking for the microphone data, and I was not disappointed. By that I mean: I found some sound files. The Planetary Society offers 'sounds from the Huygens "Microphone"', and those quote marks should be a clue as to how this is going to turn out. This semi-technical description of the Acoustic Sensor Unit explains all: the "microphone" is part of a set of instruments that examined Titan's atmosphere during the descent. It's designed to detect a thunderstorm. It takes a sample once every two seconds, and its share of the Huygens bandwidth is a measly 480 bits per second. It's basically taking Polaroid pictures of the ambient sound—not something the human sense of hearing can deal with.

But the Planetary Society gamely processed the data into sound files approximating what you would hear if the microphone was much better. And... it sounds like wind, because Huygens is falling through atmosphere. No thunderstorms. There are files reconstructed from the data recorded while Huygens was sitting on the surface. (Well, it's still sitting on the surface, but from back when the battery worked.) Unfortunately, according to Peter Falkner of ESA, "all the sound we can hear is likely internal to the microphone."

So in terms of the gee-whiz factor, the microphone is a bit of a bust. It doesn't help that Huygens's only visible-light image from Titan's surface looks like a daguerreotype of Mars (see comparison). No wonder I forgot all about Huygens. As an antidote, I recommend Cassini's amazing photos of Titan from orbit, including radar images of the hydrocarbon lakes.

This wasn't the Planetary Society's first venture into astroacoustics. In the 1990s, three Berkley scientists developed "The Mars Microphone", an actual human-ear-like microphone that would work on Mars. Unfortunately it went to Mars with the Mars Polar Lander, which was lost during landing. Another Mars Microphone was supposed to go on the ESA Netlander mission, but that mission was canceled for being too expensive.

The Phoenix lander had a Huygens-like low-resolution microphone as part of its Mars Descent Imager, but (I'm synthesizing contradictory reports here) MARDI was not turned on during descent because it could have screwed up the landing. The MARDI microphone was turned on after landing, but no data was received.

It's a legacy of heroic striving towards almost certain disappointment, but there's another MARDI on the Mars Science Laboratory, so let's check back in August.

Image credits: ESA/NASA/JPL/University of Arizona, Roel van der Hoorn/NASA

Sun Mar 04 2012 14:45: I just randomly discovered that a friend of mine, Will Thompson, cohosts a radio show about science fiction, and last week he put in a little plug for Constellation Games. Tune in at around 51:00 to hear me finally achieve my goal of having my work compared to Ken Macleod's. Admittedly by someone who hasn't read Constellation Games and doesn't seem very into Ken Macleod. But we don't get to choose how that kind of goal is achieved.

Tue Mar 06 2012 10:09 Constellation Games Author Commentary #15: "777": A few weeks ago I described the moment when I realized I'd written a novel that didn't pass the Bechdel test. I went back trying to "fix" the "problem". Should be easy, right? Five of the eight main characters are women. Well, I'm counting Curic as a woman because that's how Ariel thinks of her.

Actually, that's the problem: the whole novel is tight third-person limited from Ariel's POV. The women definitely have conversations that don't involve Ariel, but it's all off-camera. To dramatize such a conversation from Ariel's POV, he'd have to be spying on them or something.

Fortunately, there's a cheap fix: pull a Starbuck on male stock characters. I did this twice. In this chapter, I gender-swapped the Senator who gives Kinki Kwi the runaround. A similar thing will happen next week. In neither case is Ariel a direct party to the Bechdel-passing conversation. In this chapter, Curic recounts the conversation to him; in chapter 16 it's something he overhears on television.

So annoyed was I at the difficulty of a non-cheap fix, I decided to write all the bonus stories from the POV of the women. This made passing Bechdel trivial. Jenny talks to Bizarro Kate, Jenny talks to Curic. Done. You just have to be interested in what women might talk about.

(Attn. Bechdel nitpickers: if you're calling shenanigans because Curic never names the Senator, wait for chapter 16, geez.)

I hope you're hanging off a cliff. Here's last week's Twitter archive, and now the miscellaneous commentary:

And on that cheery note we end this week's commentary. Tune in next week for "False Daylight," the HEART-BEATING CONCLUSION to Part One, in which special guest star Charlene Siph will say, "Pardon my French."

Image credits: U.S. Congress (x2), Tropenmuseum of the Royal Tropical Institute, Wikimedia Commons user Silver_Spoon_Sockpop, NASA.

← Last week | Next week →

Thu Mar 08 2012 13:20 Worst Best Picture: Last night I dreamed I was teaching a college-level class on the history of film. Despite my total lack of qualifications, the class went well, because I focused more on film metadata than on history or craft. One of the things I did in dream-class was compare different measures of film quality, as I've been doing recently on NYCB with TV shows. In particular, I compared the winner of each year's Best Picture Oscar to IMDB's highest-rated movie of that year.

Well, prepare for a dream come true, because when I woke up I created that comparison in real life, using my old standby, IMDB data. I also brought in Wikipedia data, because it looks like IMDB doesn't publish any machine-readable information about awards or nominations. Wikipedia doesn't either, but you may have heard of a little library called Beautiful Soup.

Without further ado, here's the table. Well, I need a little more ado to explain what the headings mean.

YearBest PictureBPIMDBBPrankBest IMDB RatingBIRIMDBBIRnomAlignment
1927Wings7.80#9Metropolis8.40 0%
1928The Broadway Melody6.50#8La passion de Jeanne d'Arc8.30 0%
1929All Quiet on the Western Front8.10#1Chelovek s kino-apparatom8.40 0%
1930Cimarron6.30#20All Quiet on the Western Front8.10 0%
1931Grand Hotel7.70#11City Lights8.60 12%
1932Cavalcade6.40#24Trouble in Paradise8.20 10%
1934It Happened One Night8.30#1It Happened One Night8.30✓✓25%
1935Mutiny on the Bounty7.90#4A Night at the Opera8.10 41%
1936The Great Ziegfeld6.90#57Modern Times8.50 30%
1937The Life of Emile Zola7.40#31La grande illusion8.20 50%
1938You Can't Take It With You8.00#4The Lady Vanishes8.10 30%
1939Gone with the Wind8.20#3Mr. Smith Goes to Washington8.40✓40%
1940Rebecca8.40#1Rebecca8.40✓✓40%
1941How Green Was My Valley7.90#7Citizen Kane8.60✓60%
1942Mrs. Miniver7.70#12Casablanca8.80 30%
1943Casablanca8.80#1The Ox-Bow Incident8.20✓30%
1944Going My Way7.40#31Double Indemnity8.60✓20%
1945The Lost Weekend8.10#4Les enfants du paradis8.30 20%
1946The Best Years of Our Lives8.30#2It's a Wonderful Life8.70✓40%
1947Gentleman's Agreement7.40#25Out of the Past8.10 0%
1948Hamlet7.90#12Ladri di biciclette8.50 40%
1949All the King's Men7.60#18The Third Man8.50 20%
1950All About Eve8.50#2Sunset Blvd.8.70 20%
1951An American in Paris7.30#44Strangers on a Train8.30 0%
1952The Greatest Show on Earth6.70#86Singin' in the Rain8.40 20%
1953From Here to Eternity7.90#10Le salaire de la peur8.30 0%
1954On the Waterfront8.40#3Shichinin no samurai8.80 20%
1955Marty7.70#18Les diaboliques8.30 0%
1956Around the World in 80 Days6.8?The Killing8.20 0%
1957The Bridge on the River Kwai8.40#512 Angry Men8.90✓60%
1958Gigi6.90#77Vertigo8.50 20%
1959Ben-Hur8.20#5Ningen no jôken8.80 20%
1960The Apartment8.40#2Psycho8.70 20%
1961West Side Story7.70#28Ningen no jôken8.80 20%
1962Lawrence of Arabia8.50#1Lawrence of Arabia8.50✓✓40%
1963Tom Jones7.00#74Pour la suite du monde8.50 0%
1964My Fair Lady7.90#15The T.A.M.I. Show8.40 0%
1965The Sound of Music7.90#20Obyknovennyy fashizm8.40 0%
1966A Man for All Seasons8.00#8Il buono, il brutto, il cattivo.9.00 20%
1967In the Heat of the Night8.00#8Cool Hand Luke8.30 20%
1968Oliver!7.50#41C'era una volta il West8.80 20%
1969Midnight Cowboy8.00#8Le chagrin et la pitié8.40 20%
1970Patton8.10#4Brigada Diverse intra în actiune8.10 20%
1971The French Connection7.90#12A Clockwork Orange8.50✓20%
1972The Godfather9.20#1The Godfather9.20✓✓20%
1973The Sting8.40#1The Sting8.40✓✓20%
1974The Godfather Part II9.00#1The Godfather: Part II9.00✓✓40%
1975One Flew Over the Cuckoo's Nest8.90#2Hababam sinifi9.00 40%
1976Rocky8.10#8Tosun Pasa8.70 20%
1977Annie Hall8.20#9Star Wars8.80✓20%
1978The Deer Hunter8.20#5Selvi boylum, al yazmalim8.50 20%
1979Kramer vs. Kramer7.70#27Apocalypse Now8.60✓20%
1980Ordinary People7.90#8The Shining8.50 40%
1981Chariots of Fire7.20#74Raiders of the Lost Ark8.70✓20%
1982Gandhi8.10#8Maratonci trce pocasni krug8.40 0%
1983Terms of Endearment7.40#42El sur8.20 0%
1984Amadeus8.40#4Balkanski spijun8.50 20%
1985Out of Africa7.00#83Zügürt Aga8.50 0%
1986Platoon8.20#4Aliens8.50 20%
1987The Last Emperor7.80#17Muhsin Bey8.40 0%
1988Rain Man8.00#12Nuovo Cinema Paradiso8.50 0%
1989Driving Miss Daisy7.40#45Indiana Jones and the Last Crusade8.30 0%
1990Dances with Wolves8.00#3Goodfellas8.80✓40%
1991The Silence of the Lambs8.70#1The Silence of the Lambs8.70✓✓60%
1992Unforgiven8.30#3Reservoir Dogs8.40 20%
1993Schindler's List8.90#1Schindler's List8.90✓✓40%
1994Forrest Gump8.70#4The Shawshank Redemption9.20✓60%
1995Braveheart8.40#3Se7en8.70 20%
1996The English Patient7.30#73Freebird... The Movie8.30 20%
1997Titanic7.40#51La vita è bella8.50 20%
1998Shakespeare in Love7.30#83American History X8.60 20%
1999American Beauty8.60#3Fight Club8.80 60%
2000Gladiator8.40#3Memento8.70 20%
2001A Beautiful Mind8.00#13The Lord of the Rings: The Fellowship of the Ring8.80✓20%
2002Chicago7.30#155The Last Just Man9.40 40%
2003The Lord of the Rings: The Return of the King8.80#1The Lord of the Rings: The Return of the King8.80✓✓20%
2004Million Dollar Baby8.20#7Eternal Sunshine of the Spotless Mind8.50 0%
2005Crash8.50#12Babam Ve Oglum8.70 0%
2006The Departed8.50#3Das Leben der Anderen8.50 20%
2007No Country for Old Men8.20#8Jogo de Cena8.70 0%
2008Slumdog Millionaire8.20#7The Dark Knight8.90 0%
2009The Hurt Locker7.80#30Puskás Hungary8.60 30%
2010The King's Speech8.30#17Inception8.90✓20%
2011The Artist8.20#15Drive9.00 0%

Ok, "Alignment". Take 1941 as an example. There were ten Best Picture nominees in 1941 (although it was called something different back then). So we take the top ten movies of 1941 by IMDB rating. Six of the Best Picture nominees are also in the top ten by IMDB rating, so the alignment for 1941 is 60%. At the other extreme, none of the five 1983 Best Picture nominees are in the IMDB top five for that year, so the alignment for 1983 is 0%.

For a few years I couldn't calculate BPrank, generally because the IMDB year of the Oscar winner differs from the year it won an Oscar. Early on this happens a lot because until 1933 the Academy Awards covered parts of two years. That's why "All Quiet on the Western Front" got the nod in the 1929 Oscars, and then showed up as the best-rated IMDB film of 1930. The "1929" Oscars weren't just held in 1930, they actually covered some movies released in 1930. But sometimes the dates just don't match up. Casablanca is the top-rated film of 1942 and the winner of the 1943 Oscar. This still happens: The Hurt Locker won Best Picture in 2009 but IMDB says it was released in 2008. In most cases I was able to find the year the film was released, according to IMDB, and put down down its ranking within that year for BPrank.

My dataset excludes TV shows, video games, direct-to-video releases, and shorts. (Excluding shorts required cross-referencing against IMDB's genre.list file.) I also excluded movies with fewer than 150 votes on IMDB. I did what I could to exclude movies that are mainly concert footage, although Freebird... The Movie still made it on there. I did not exclude documentaries or foreign films.

Finally, to fulfil the promise of this post's title. According to IMDB, the worst movie ever to win Best Picture is 1930/1931's winner, "Cimmaron" (IMDB:6.3). But if you look relative to what else came out the same year, the worst Best Picture is "Chicago" (IMDB:7.5), which IMDB data ranks at the 155th-best movie of 2002. However you look at it, the best movie ever to win Best Picture is 1974's The Godfather: Part II (IMDB:9.00).

PS: Why are the Oscar nominees linked and the IMDB champions not linked? Because IMDB DATASET DOESN'T INCLUDE ANY URLS ARGH.

PPS: I did something similar for board games as part of Loaded Dice. I called it the "People's Spiel des Jahres." I didn't put up the table because the results were uninterestingly full of wargames. But wargames generally don't get nominated for Spiel des Jahres, so maybe I should exclude them and try it again.

[Comments] (7) Tue Mar 13 2012 08:44 Constellation Games Author Commentary #16: "False Daylight": Here it is, the season finale! We've got the whole contact mission going to shit, plus a game review! Don't worry, everything will turn out fine. Maybe.

Last Friday I went to the Brooklyn Museum to take some pictures for my final Constellation Games commentary. (And if you can somehow turn that into a spoiler, I salute you.) It's a fun museum, like a much less formal version of the Met. While walking through the room of Indian sculpture I passed a curator cleaning one of the sculptures with a Shop Vac and a brush. When I showed a flinch of uncertainty about where the stairwell was, a security guard told me and talked my ear off about what I should see next, then opened up the cabinet containing the emergency fire hose and took out a "What's Happening" brochure, which she used for reference and then gave to me. Also, the neighboring Botanic Garden was free to get in because it's winter and everything's dead.

Friend of the show and beta reader Brendan Adkins has been writing erudite-ass essays about the novel's symbolism, and I'd make fun of him for being pretentious except he's right about most of it. My earlier coyness notwithstanding, I did reuse some of the character of Ariel from The Tempest, the guy with magic powers who gets bossed around all the time. Don't you think The Tempest would be more interesting if it were more about the PEOPLE WITH MAGIC POWERS and less about the Renaissance douchebags? We can only dream. For now, we sup the slender soup of the Twitter archive and this week's commentary:

With that, I'd like to thank you for following me through "Hardware", the first part of Constellation Games. After a short season break of seven days, we'll pick up with Part Two, "Software." It all starts next Tuesday, when Ariel will say, "Probably the most expensive penis in history."

Image credits: Wikimedia Commons user Anynobody, Paul Mutant, U.S. Air Force

← Last week | Next week →

Wed Mar 14 2012 12:26 Beautiful Soup 4.0.1: It's been nearly two weeks since the release of the last BS4 beta, and no one has reported problems with the code. I'm sure there are still problems, but at this point the best way to find them is to do an official release. So, I present the first full release of Beautiful Soup 4, 4.0.1![0]

If you're just tuning in, Beautiful Soup 4 is nearly a complete rewrite that works on Python 2 and Python 3. Instead of a custom-built parser from 2006, Beautiful Soup 4 sits on top of lxml (for speed) or html5lib (for browser-like parsing) or the built-in HTMLParser (for convenience). Methods and attributes are renamed for PEP 8 compliance, and Beautiful Soup 4 incorporates the soupselect project to provide basic CSS selector support. I completely rewrote the documentation, Beautiful Soup's secret weapon since 3.0, for clarity and completeness.

That's the major stuff. Even though most of the code has changed, my goal was not to add a bunch more features, but to make sure Beautiful Soup will still be usable and useful years into the future.

Beautiful Soup 4 is mostly but not entirely backwards compatible with Beautiful Soup 3. Most users should be able to switch from 3 to 4 just by changing an import line. In the Python tradition of sticking a number on the end of your module name when you break backwards compatibility, I've released it as a separate package, beautifulsoup4.

This release also inaugurates the Beautiful Soup Hall of Fame, featuring the uses of Beautiful Soup that I personally find the coolest or highest-profile.

So, try out Beautiful Soup 4 the next time you need to do some screen-scraping. If you've used Beautiful Soup 3, I think you'll be pleasantly surprised. If not, I'll just say I hope you like it.

I've thanked them before, but special thanks are once again due to Thomas Kluyver and Ezio Melotti for helping me get everything working under Python 3.

[0] The first release is called 4.0.1 instead of 4.0.0 because I've been bitten by clever packagers before and I don't want them thinking "4.0.0" is an earlier version than "4.0.0b10".

[Comments] (5) Mon Mar 19 2012 08:39 The Pitch!: Hey, folks, Leonard here, telling you that if you haven't bought your copy of Constellation Games, still the greatest novel about video games from outer space, now is probably the single best time to buy.

Sure, you were skeptical at first. Ever since standing in line for that midnight showing of The Phantom Menace, you've been wary of things that seem awesome. You thought, "can this guy bring to comedic science fiction the same epic scope we saw in RESTful Web Services?" But now Part One of the novel has been sent to subscribers, and random commentary readers are calling it "STONE COLD BRILLIANT" and "some of the most fun I've had in years". Even normally reputable publications like Wired's GeekDad have called it a "wild ride" that's "so much fun to read".

Now here's where your late-adopterhood pays off: with the completion of Part One, all subscribers have been given access to a compiled PDF of the novel so far. That's about 50,000 words in a single unencumbered file you can drop onto your ebook reader or your fancy smartphone.

This means you can subscribe to CG for $5, read the first sixteen chapters in one huge gulp, and then start reading the rest of the story as the chapters come out every Tuesday. Or you can subscribe at the $20 level, read the PDF at a more leisurely pace, and finish the whole story when the paperback comes out next month. For $20 you'll also get three bonus stories that take place before/during/after the novel, and an irreverent guide to a pathologically strange alien language.

With all this stuff on the table, you silently think, why not keep waiting? Won't we just offer more in the future? THE ANSWER IS NO. Once the paperback comes out, the bonus stories and language guide stop being pack-ins and become "sold separately"s. The paperback on its own will cost $20. (I don't know exactly how this is going to happen, but that's the gist of it.) So the best deal is to shell out $20 now for early access to Part One and a lot of preorder bonuses. If you hate paper, you can pay $5, catch up on the novel the way you would a web comic, and buy the bonus material later.

Friend, don't let the fact that I seem to think it's a great idea to call you "friend" in a sales pitch, dissuade you from shelling out your hard-earned PayPal balance for this quality entertainment. Here's the subscription page, and here are the first two chapters so you can see what you're getting. The whole thing could be yours for the cost of a really, really enormous gumball, a gumball that won't fit in your mouth so why even bother? This is a much better deal.

[Comments] (1) Mon Mar 19 2012 13:25 Archive: On Friday I decoded a BCDIC punch card that my dad used to sign up for classes at UCLA in 1968. It says, "C 6088312496U40" What drove me to this? Well:

Some addenda acquired from readers while I performed that blob of text on identica/Twitter:

[Comments] (8) Tue Mar 20 2012 08:59 Constellation Games Author Commentary #17: "Their First Contact Was Better": This chapter has the best title in the whole book. Just gettin' that out of the way. This week sets up the plot for the next couple months while focusing the action on the emotional core of Part Two: Ariel's relationships with other humans people from Earth.

I really liked the comments from last week's commentary--two people I didn't know were reading said hello, Brendan responded to my evaluation of his reader commentary, and my friend Zack (whose name I stupidly misspelled) disputed my use of Creative License. If you're enjoying these commentaries, please do say hi in the comments.

Look on last week's Twitter archive, ye mighty, and despair. Tetsuo won't be posting for a while because of the Internet blackout. Here's this week's commentary:

And there we go. Be sure to tune in next week, when Ariel's all like, that's right, motherfucker, you're not the only one who can use paper. Oh, and Tetsuo writes a game review!

Image credits: NASA, unknown, Flickr user puuikibeach.

← Last week | Next week →

Tue Mar 20 2012 16:54 DLC Upsell: Did you buy one of the really cheap Constellation Games packages, and are now regretting your decision? Sorry, no refunds. Oh, you want a package with more stuff? You're in luck! Use the Candlemark & Gleam contact form to ask for an upgrade, and Kate will upgrade your subscription and invoice you for the difference.

Be sure to say which package you want. "Gold" ($12) is the one with the (electronic) phrasebook and bonus stories.

Thu Mar 22 2012 09:20 Schmeckel Needs a Van: Schmeckel, the Jewish transgender punk band most familiar to NYCB readers, has a Kickstarter project to get a sweet tour van. This lets me combine two great things in one post: crowdfunding and cool doods like Schmeckel frontman Lucian Kahn, who will probably get the Teenage Mutant Ninja Turtles airbrushed onto the tour van.

I'm still going through the Kickstarter firehose every day, but the number of projects I've backed has gone way down since the heady Month of Kickstarter. This one's an auto-back, though. Backing a crowdfunding project is like being pen pals with the Internet, so give it a shot.

[Comments] (1) Tue Mar 27 2012 09:03 Constellation Games Author Commentary #18: "The Amazing Colossal Man-in-the-Middle": Be warned! This week's commentary goes deep into the workings of a scene that was originally a disaster, and the rewrite process that made it hopefully only a minor disappointment. Fortify yourself with last week's Twitter archive before proceeding.

Tune in next week for the shocking chapter 19, in which Ariel travels the well-worn road from "unreliable narrator" to "flat-out liar." The chapter in which BEA Agent Fowler will say the ridiculous line everyone tried to get me to cut, but I refused! For you, my readers.

Image credits: Flickr user windygig, Pepe Medina, Danny Cornelissen.

← Last week | Next week →

Sun Apr 01 2012 09:45 Dada Skies: Over the years, my series of "Dada" projects has brought meaninglessness to formerly meaningful things: board games, Shakespeare, comics. Today, for my April 1 project I present Dada Skies, which randomly rearranges things that were randomly arranged to begin with: the stars as seen from Earth. It's a view I find strangely relaxing.

Dada Skies works like Dada Maps, by transcluding image tiles from a web service into a mosaic. In this case the service is the one provided by the Sloan Digital Sky Survey. There are enhancements I'd like to make to the view, but this is what I've got time for right now, and I want to do my part to keep alive the nascent tradition of releasing cool things on April 1.

Mon Apr 02 2012 10:22 Constellation Games Game Design Promo: My editor has an extra advance copy of Constellation Games, which she asked me to give away in a gala promotional event! This put me in a pickle: I love giving things away, but I really hate "promotional events". So I came up with something fun: you can win a copy of the paperback (or, if you're already a subscriber, a subscription upgrade), by fleshing out the Constellation Database of Electronic Games of a Certain Complexity.

For those who haven't read Constellation Games: the CDBOEGOACC is an enormous XML document containing metadata for entertainment software from countless alien civilizations, as well as the computers and peripherals necessary to run that software. Like, imagine if Jason Scott worked for the Culture. To give you the idea, here are some CDBOEGOACC quotes from the narrator's Twitter feed:

Every CDBOEGOACC entry is a tiny science-fiction story about an alien culture and someone who responded to their culture by making a game. If you like this idea, you are the target audience for Constellation Games and you should enter this contest to win a copy. The CDBOEGOACC is the part of the book that was the most fun to write, and I can't think of a better gala promotional event than asking you to come up with these mini-stories.

You can write something tweet-sized like the quotes above, or you can flesh out an idea a bit more and put it on your weblog, or whatever. It doesn't matter to me, so long as you somehow make sure I know about it. Only entries I know about will be judged. Surefire ways to make sure I know about it: send me email, post a link in the comments below, or use the unwieldy hashtag #cdboegoacc on Twitter or Identica.

The contest ends when I wake up in the morning on April 10, one week from tomorrow, and pick my favorite CDBOEGOACC entry. We'll mail the winner their advance copy right away, which means you'll get the book a few days before its official release on April 17. (As a bonus, this uncorrected proof contains a huge continuity error in chapter 35, which you can find and then feel superior to me.) I'll also pick a random winner, who will get a free base subscription to the serial, plus a collected ebook once the serialization is done.

Since the goal is to make cool things, you can enter more than once to improve your chance of getting the paperback, but the random drawing is one entry per person. I will judge entries on the twin criteria of "sensawunda" and "comedy", the binary star by which I steer my novelist's ship. Have fun!

[Comments] (6) Tue Apr 03 2012 09:07 Constellation Games Author Commentary #19, "Implementation Details": We're more than halfway through the book! Traditionally, it's around this time that an author starts to have doubts about having signed on for this huge project. It's happened on all three of my books and it's happening again with this commentary. I have a great time writing this stuff every week, and I hope you're enjoying it, but it does take a lot of time. Time which might be better spent working on my second novel, "A Fire Upon the Derp".

Uh, anyway, last week's Twitter archive. Don't forget about the game design contest, which ends next week. This week we meet Dana Light 2.0, and the totally different human person who's not Dana, Svetlana Sveta.

That's the week! Stay tuned for next week's special tabletop gaming episode, when Ariel will say, "I think we should just nuke each other once and get it out of our systems."

Image credits: Wikimedia user Ecelan, Joshua Kaufman, Flickr user ePi.Longo.

← Last week | Next week →

Wed Apr 04 2012 14:27 TUNS: For a while I've been gathering cool space-themed pictures to illustrate the Constellation Games commentary, but there's way more and they're way more varied than I need. Rather than abandoning this embarrassment of visual riches, I recently started posting a couple pictures a day to my microblog using the hashtag #retrorocket, so named because I mostly focus on cool-looking old tech and people working with or building it.

At the risk of revealing all my secrets, I'm getting the #retrorocket pictures by harvesting the Internet Archive's NASA Images site, one year at a time. I've combed through 1969-1988, and my technique will break down around 1994, when there start to be thousands of pictures for each year (even 1969 wasn't that bad). But 1994 is not all that retro, really, is it? Yeah, it's kinda retro.

Despite the name, NASA Images has a lot of non-images: mainly movies and technical documents. Recently one tech document caught my eye: Technology Utilization Network System, a 1987 document laying out NASA's recommendations for off-the-shelf PC hardware and software.

This document pulls no punches, naming brand names, presenting huge benchmark tables, and spelling out just what it takes to outfit an effective mid-80s networked office on the taxpayer's dime. This is a time before Windows, in an office environment without Unix. Let's take a look at this document and see which products have the Right Stuff, and which fizzle on the launch pad. (n.b. Entertainment Weekly didn't want this for some reason, and I'm too lazy to change the segue.)

We'll start with the basic PC recommendation. "The recommended workstation for TUNS is the Compaq DeskPro 286 with a 40 Mb hard disk." Each should be equipped with an EGA card and a color monitor. "The estimated GSA cost... is $4,087." (About $7700 in 2010 dollars.)

The use as a workstation of one of the newer 386 machines (based on the Intel 80386 CPU chip) is not currently recommended. The power provided by the 386 system is more than that required of the workstation.

Unlike a lot of offices in 1987, NASA understands the importance of networking. That's why each workstation is fitted with a 3Com Etherlink card ($451) and connected to a local file server running Novell Advanced Netware/286. Netware wins out over now-obscure competitors like 3Com's 3+ Share, Banyon VINES/286, Fox 10-net, and Lee Data's LANMASTER. The Internet stack is not even considered: I'm sure NASA's scientific installations are on the Internet by now, but it's not really an option for IBM PCs, and the term "intranet" doesn't exist.

The Compaq 286 used for workstations can also act as the file server on a small LAN. But for a larger LAN, you do need to buy that 386, and for the really big installations, it's got to be "a Novell T286B with a 183 Mb of hard drive space". And remember to buy name-brand:

Although many vendors claim to sell "AT clones," ISN has occasionally found very subtle differences in the performance of these "clones," which may result in problems during system integration.

For connecting to external sites and databases, each workstation is outfitted with a Hayes Smartmodem 2400 at $579.

ISN does not recommend installation of 9600 baud modems at this point. The lack of standard protocols, error-correction methods and data compression techniques for 9600 baud communications means that two modems from different vendors will rarely communicate with each other at 9600 baud.

Printer time! TUNS spells out recommendations for cheapo dot-matrix printers (the Epson FX-286e, $527), letter-quality daisywheel printers (the Diablo D80IF, $1523), and laser printers for impressing the boss (the HP Laserjet Series II, $1795). Note that the most expensive printer is half the price of the workstation PC.

Two printers, the Brother Twinwriter 5 and the Fortis DH-45, include both dot-matrix and daisy-wheel print mechanisms. The two companies are actually marketing the same printer under different labels. Although this printer was initially viewed as an exciting combination of functionality at a reasonable price, it was excluded from further consideration after the Twinwriter 5 vendor reported extremely poor reliability and great customer dissatisfaction.

What about software? For the most part, TUNS recommends off-the-shelf 80s office software, Lotus 1-2-3 (GSA cost: $305) and WordPerfect 4.2 ($173):

Although the evaluation scores for WordPerfect and Microsoft Word show only a two point difference, the evaluation team highly recommends WordPerfect because the evaluation team found it much easier to use WordPerfect's ACSII-to-document transfer features.

But they choose Unify, a database I've never heard of, over 80s heavyweights like dBase and Clipper. Why? Because Unity just has more stuff, like a C library.

Oracle was eliminated as a possibility because a LAN version of the software, although in development for some time, is still not available.

Each 286 workstation is loaded with about $100 worth of utilities: Popular ones like Sidekick and Norton Utilities, as well as some more obscure ones: "Sideways to aid in the printing of large spreadsheets; ScreenSave to protect the life of the monitors; KeyBuffer to allow the user to enter characters from the keyboard at a faster rate than acceptable under DOS; and FilePath to aid in the use of multiple directories and sub-directories."

But the best part of the whole document is the section on email. The document identifies a large number of requirements for an email system, such as:

The sender must be able to identify a single recipient, multiple recipients, and a "group" mailcode consisting of multiple mail identification codes but addressed as one unit.

And then presents a huge table comparing the competitors (again, Internet email is not even on the radar). And then decides to just keep using "the NASAMAIL system currently available throughout NASA." NASAMAIL was actually an installation of Sprint Telemail, which you can read a little more about in RFC 1168, including a mention of a Telemail-Internet mail gateway at NASA Ames.

It's a little odd that hardware of this description almost never shows up in the NASA Images archive! (I did see one mid-80s IBM PC sitting on someone's desk, but when I went back to look for it as an illustration I couldn't find it.) I think these recommendations were mostly for clerical workers and managers, and that the engineers and scientists (who show up quite a bit in the archive) used minicomputers, Unix workstations, and mainframes into the 90s.

I'm probably not gonna go through any more of these documents in any detail, but here are two others I found really interesting: the public affairs plan for STS-1, the first Space Shuttle flight, and the original press kit for Apollo 13.

Tue Apr 10 2012 09:18 Constellation Games Author Commentary #20, "Feature Creep": This week: Dana earns her paycheck, we learn the shocking (if you're a Farang) secret of Sayable Spice, and Ariel stresses out and gets a little stalkery.

I'm not sure who put a bunch of tags on CG's LibraryThing page, but they're pretty great. (and full of spoilers) Apparently CG is a bildungsroman about cosplay, douchebags, mecha-godzilla, real replicas, and vastening. I don't disagree!

Come for the Twitter archive, stay for the commentary. CDBOEGOACC contest winners will be announced as soon as I finish the post.

Part Two's plot kicks into high gear next week, with the terrifying chapter 21, "Her". The chapter in which Tetsuo will say (but not demonstrate) "Sexual pair bonding!" Also next week: you can get a paper copy of the book and read the whole rest of the story! Then complain about how I'm not putting up this commentary fast enough.

Photo credits: US Army, US Department of Defense, Flickr user my_eye.

← Last week Next week →

Tue Apr 10 2012 09:56 CDBOEGOACC Contest Results: I was worried that no one would enter the CDBOEGOACC contest and it would be like a party where no one showed up. But ten people put in 29 entries, which is a pretty good party. I'm pretty sure all the entrants are Constellation Games subscribers, so I gotta work on my crossover appeal, but I'm happy with the turnout.

Once again, the grand prize is a galley copy of Constellation Games which will hopefully be delivered a few days before the paperback comes out. Even I don't have a galley copy, so you know it's exclusive. There'll also be a random drawing, and the winner will get a free basic-level subscription. Although since everyone who entered is already a subscriber, I don't know if you want that. Maybe you can give it to a friend, maybe you can negotiate a different prize with Kate, the publisher.

Anyway, let's take a look at the entries:

All the other entries were short concept quotes posted to Twitter. I've archived all of them here because I really hate the way Twitter's UI consigns the past to a dark, eternal oubliette.

I thought all the entries were really good, although Adam may have been phoning it in. C'mon, Adam, this ain't Apples to Apples. Anyway, I'm excluding Adam and Brendan from consideration because they were beta readers. Here are my three favorites (apart from Andrew's, which I've already spilled the beans that it didn't win):

And the winner of the Constellation Games galley copy is... Ornithopter! I loved their game concept because it tied into a theme I don't really explore in CG: the repurposing of really awful historical situations as entertainment simulations.

But don't give up yet, non-disqualified entrants! We've still got the random drawing. And here's some Python code to perform it:

>>> import random
>>> entrants = ["Andrew Perry", "Ornithopter", "George Buckenham", "Benhimself", "Gus Andrews", "Tikitu de Jager", "Mirabai Knight"]
>>> random.choice(entrants)
'Andrew Perry'

OK, well, that worked out. Andrew Perry will receive the random drawing prize, and we'll just call it the CDBOEGOACC Jury Prize.

And that's it for the gala CDBOEGOACC giveaway contest! I hope it was a fun time. I certainly enjoyed watching people come up with this stuff.

Mon Apr 16 2012 12:06 Beautiful Soup 4.0.4: I haven't been mentioning all the Beautiful Soup releases I've been doing, because they're just maintenance releases, but I'll mention them occasionally because fixing bugs (and determining what's a bug and what's not) still takes up a fair amount of my time. We're up to 4.0.4 and I've fixed/worked around a number of bugs, including one that prevented Beautiful Soup from parsing an XML document larger than about 512 bytes.

I've also updated the docs quite a bit to help people solve common problems. I'm not sure where to stop, because Beautiful Soup is the first Python library a lot of people use, so it gets caught up in questions like "how do I install Python packages on Windows?" (It's not easy.)

Mon Apr 16 2012 14:58 CS161: You could get a computer science education just by taking classes called CS161:

[Comments] (7) Tue Apr 17 2012 09:14 Constellation Games Author Commentary #21, "Her": This is one of the most important chapters in the book. I need you to start feeling the weight of the Constellation as a geological-time, astronomical-scale project, not just as the country where Tetsuo and Curic were born. The best way is to show you the Earth contact mission through the eyes of the one who's seen it all: the Her superorganism.

Twitter was quiet last week, and will be even quieter this week because the whole chapter takes place between 3 and 5 AM on a Tuesday. I think you should do some work this week instead of checking Twitter all the time.

Today is the official release of the paperback edition of Constellation Games! How does this work? I have no clue. I believe those of you who ordered the paperback will be getting it sometime this week, and those who have been resolutely refusing to preorder will soon be able to order it from the online store or get it from a local indie. In the meantime, how about a bulleted list? I know all about those.

On that note: tune in next Tuesday (or read the paperback) for Ariel's terrible, horrible, no good, very bad day. During the course of which he will say, "I'm more worried about my friend's problem than in coming up with the perfect urine-related analogy for the problem."

Image credits: Joachim Barrande, Flickr user fubsan

← Last week | Next week →

Mon Apr 23 2012 09:55 Some Interesting Game Aliens: Editor Kate pointed me to this list of the best and worst aliens in video games. Without wanting to say anything bad about that list, I noticed that it (and similar lists I've found online) focuses heavily on the visual design of humanoid aliens from first-person shooters. So I thought I'd make my own list, in honor of the print release of Constellation Games (Publishers Weekly calls it "fun"!), highlighting some video game aliens that I find interesting from a game design perspective. I'm sure there are plenty more I haven't heard of, so if you have any additional suggestions, I'd like to hear about them in comments.

The invaders (Space Invaders)

Just gonna get this one out of the way. Among the most iconic aliens ever devised, the invaders in the middle row have come to symbolize video games as a whole. Apart from their visual style there's nothing there, but the style is great.

The blobs (A Boy and his Blob)

These were my gut-reaction nomination for "best", because they're the only aliens I can think of that only make sense in a video game. The blob in A Boy and his Blob is a game mechanic personified: a sentient inventory. This causes serious problems if you try to think about the species outside the context of the game—when your toilet clogs on Blobolonia, do you feed your friend a jellybean and turn them into a plunger? But within the game, it works great.

The Melnorme (Star Control II)

If you want vivid alien characterizations, Star Control II is your game. Unfortunately, most of those characterizations are based on asinine stereotypes. That's why the Melnorme win it for me. It would be easy to make "the trader race" greedy and sleazy—in fact, SC2 does this with the other "trader race", the Druuge. But the Melnorme are friendly cosmopolitans who're fun to talk to. And they occasionally drop ominous hints that are never followed up on anywhere in the game.

That said, there's nothing game-y about the Melnorme, they just happen to be in a game. Every race in SC2 could guest on Star Trek, and many of them have. So I'm not pushing this one very hard. At least they're not humanoid.

Honorable SC2 mentions: the Zoq-Fot-Pik, who are silly and fun; and the Orz, who are similar to but not as well-executed as...

The Endermen (Minecraft)

From another dimension rather than from outer space, but aliens nonetheless. The Roadside Picnic of video game aliens; the Endermen follow rules that make perfect sense... to them. Their random rearrangement of blocks and sudden fits of aggro bear a twisted resemblance to your own behavior in Minecraft. Like you, they are interlopers in the game world, and their behavior was designed to challenge your dominance of it.

Giygas (Earthbound)

I almost didn't count Giygas for the same reason I'm not counting the Meteor from Maniac Mansion: I already gave the "cool aliens that happen to be in a game" nod to Star Control 2. But the final battle of Earthbound does some interesting things with the game's generic JRPG battle system, so sure, I'll count it.

"Them" (The Legend of Zelda: Majora's Mask)/The Martians (Metal Slug 2)

These aliens are composed entirely of pop culture cliches. The interesting thing is not their design but the fact that they show up at all. These aliens aren't just from another planet: they're from another genre. The Martians show up and abruptly turn your tasteless WWII run-and-gun into a '50s saucer flick. And "They" show up in a Zelda game. Albeit a Zelda game that also features a time loop and travel to the moon.

When I asked on Twitter for peoples' favorite video game aliens, the only response I got (thanks, Laura!) was also in the vicinity of this category: Crypto from Destroy All Humans, which I haven't played but which looks just like the movie Mars Attacks!.

Board game bonus! The Loser (Cosmic Encounter)

Cosmic Encounter is all about embodying game mechanics into alien species, and the Loser is the best, because it forces you to have debates about what it means to "win" a game. Whatever chaos is happening due to the other players' equally unbalanced species choices, the Loser multiplies it. My absolute fave.

And there you go. Let me know of any you think I missed—this is a bizarrely underexplored field, though maybe I just think it's bizarrely underexplored because I spent a long time writing a novel about it. I mean, I also thought it was weird no one had explained how game titles work.

Image credits: Flickr user philosofia, DeviantArt user aeonpants, DeviantArt user dczanik, DeviantArt user EliteParanoid, SNK, Felicia Cano.

Mon Apr 23 2012 15:21 Handheld Device:

[Comments] (4) Tue Apr 24 2012 09:38 Constellation Games Author Commentary #22, "Nerfed": This chapter is bad news for Ariel but good news for me, because yesterday I got my box of author copies. That means those of you who ordered paperbacks should be getting them soon. This EXCLUSIVE SIDE COVER REVEAL shows the playtime synergies possible for those who shelled out for the Adamantium package with its USB key.

In a couple weeks I may do a special spoiler post so that those who've read the whole book can ask me questions about stuff that hasn't been serialized yet, rather than waiting until the appropriate chapter comes up in the serialization.

There's a solitary tweet in last week's microblog archive. Let's move on to the commentary:

I came into this chapter treating it like a chess problem. I had all the pieces on the board and the question was how Ariel would outsmart the BEA again, the way he did with Dana. I sat down and puzzled over this and had a long conversation with Beth trying to figure out how to get Ariel out of this scrape.

And after about forty-five minutes of being totally stumped I asked myself the obvious question: Why am I trying to get Ariel out of this?

At this point I knew how the book would end (it turns out I only knew how Part Two would end). I didn't have the plot planned out between now and then, but at some point I needed to break Ariel. This is the perfect time to start. He's stressed out from Sayable Spice work, shaken from his encounter with Her and the revelation of Curic's ambivalence. Let's just go for it.

So I destroyed Ariel's house. He tries all the clever gambits I thought of for him, and they don't work, and he loses. And that was the single best thing that ever happened to the book. From this point on the characters are developed enough that I can do whatever horrible things I want to them. They'll either figure something out and come out on top, or they'll lose, and either way it'll be interesting reading.

A lot of this week's commentary is me telling stories that are at best tangentially related to the chapter. But if that's good enough for big-name DVD commentaries, it's good enough for me:

Pretty chunky commentary this week, hope you enjoyed it. Be sure to tune in next week for chapter 23, when Jenny will say, "I don't masturbate to it."

Image credits: yours truly, the city of Austin, NASA/Ames, Wikimedia Commons user Hoshie.

← Last week | Next week →

Fri Apr 27 2012 12:01 o Invasion: There are some rogue os in the Constellation Games acknowledgements. "N. K. Jemison" should refer to the Nebula and Hugo-nominated author N. K. Jemisin, not to a person who doesn't exist. And "Beth Lermon" is of course my friend Beth Lerman.

Surely these are far from the only typos in the book [I originally wrote "on the book"], but they're really bad, and deserve a special correction, and my apologies.

Sat Apr 28 2012 08:55 Hidden Treasures: There's a competition going on right now called "Hubble's Hidden Treasures", a competition to identify amazing but overlooked images within the Hubble Space Telescope's massive twenty-year data archive. I have no special expertise in image processing or astronomy, but I so coveted the prize of an iPad[0] that I thought I'd try my hand.

See, most people who enter this contest are taking pictures of nebulae and galaxies and setting up the colors to represent different wavelengths of light. Here's a nice example. These images have aesthetic value but are merely emblematic of Hubble's scientific value, which comes from the raw data. As long as we're playing that game, why not find aesthetic satisfaction in Hubble's glitches? My innovative thinking will surely net me the prize.

Just as an example, here's a super-washed-out image that could have the date and place of your punk show written in the middle of it. Lots of images have glitchy edges, and my original plan was to make a collage of the glitchy edges. But then I found the image to the right, which blew my stupid idea out of the water.

I call this ghostly image "hst_05909_01_wfpc2_fr418n18_pc", because that's its Hubble dataset ID, but if I were hanging it in an art gallery I'd call it "Cygnus", because that's the search I used to find it. Here it is big and zoomable. Here's a larger image (it's the full WFPC2 image--the one to your right comes from the Planetary Camera bit of the WFPC2) which I think makes it clear the feature you see is a photographic artifact and not a real thing in space.

I don't have anything more to say about hst_05909_01_wfpc2_fr418n18_pc; it's just a nice piece of abstract photography. The Hubble Legacy Archive is great stuff in general, though.

[0] Can the winner get 5 minutes of Hubble observation time or something contest-specific? Just trying to think outside the box here. The box labeled "box of iPads for use as contest prizes".

[Comments] (4) Tue May 01 2012 09:27 Constellation Games Author Commentary #23, "Trust Us, We're Expert Systems": I do believe it's time for a super dark relationship chapter. That's what I believed when I wrote this, anyway. Clearly I was eager to keep landing the body blows on Ariel after chapter 22.

It's always sadder when characters bring about their own destruction than when someone else screws them over. In the second draft this episode was a little less of a downer, because while Ariel was living in coffee shop exile he had a great idea for a mobile app he could write very quickly and sell to recoup some of his losses from the last chapter. So all the awful stuff between him and Dana and Jenny still happened, but at least we ended on a positive note. Who needs that, right? Just hang tough.

I cut out the "mobile app" subplot because it added a lot of story complexity for no real benefit. Ariel's already working on a software project, so why add another one? He puts out a press release for it next chapter, but I just turned it into a press release for the Sayable Spice: Earth Remix demo. Works fine.

I'll talk about the proposed app after the miscellaneous commentary, because this week's is a little light and next week's will probably be huge. (Spoiler: Tetsuo comes to Earth.)

Oh, and here's last week's Twitter archive.

Okay, about that mobile app. One thing that barely shows up in Constellation Games, but was very important in "Vanilla", is the contact audit. To sponsor an ET for an American visa (as Ariel did for Curic and Bai is now doing for Tetsuo), you need to register with the BEA as a contactee. You're supposed to do the same if you have any prolonged or repeated contact with ETs, although the Greenland Treaty is quickly making that unenforcable.

All registered contactees need to come in to their local field office twice a year for a sit-down interview about all the ETs they've encountered over the past six months. It's generally a formality; the point is to make contact with ETs a pain in the ass and, on the margin, discourage people from having anything to do with the Constellation.

Ariel's mobile app idea was a "contact manager", a way of taking the pain out of your contact audit. Whenever you meet an ET you just take a picture of them—something you were going to do anyway—and enter their name. Then your contact audit is effectively just a slideshow.

In the second draft, Ariel's key business insight was that although relatively few people really need this app, a lot of people want to be the sort of person who needs it, so they'd buy it aspirationally. Clever idea, but not really necessary for the story, so out it went.

That's all I got. Tune in next week for the TETSUOUS continuation, in which Ariel will say "Jesus Christ the great moral teacher!"

Image credits: Tim Patterson, Doug Kline.

← Last week | Next week →

[Comments] (6) Tue May 08 2012 09:07 Constellation Games Author Commentary #24, "Homebrew": Tetsuo's back, and he brought exposition! This week we take a break from beating up Ariel, and just startle him a lot while he's high.

This week's Twitter feed is almost entirely devoted to Tetsuo's first day on Earth. Today also marks the start of the Great Microblog Bonus Content Migration. Prior to this point, Ariel's feed was where it was at. But Ariel's now too busy to tweet a lot, and he'll stay busy until the end of the book. Tetsuo's feed will be picking up the slack, chronicling his adventures on Earth and showing what the other characters are doing as the focus of the novel tightens around Ariel. If you're following Ariel but not Tetsuo, this is the week to get on the Tetsuo Train (patent pending).

(NB. I won't be setting a Twitter profile image for Tetsuo because the default image is a much better depiction of him than anything I could come up with.)

Speaking of Twitter feeds, here's last week's. And before we get started, some extratextual comments:

Now that the paperback is out, you can get it from your regular source for paperbacks: Barnes and Noble or Amazon, or order it from a bookstore through Ingram, or is there any chance a bookstore might proactively stock it based on the radioactively glowing Publishers Weekly review? I wouldn't depend on it, but that would be nice. Note that the paperback is the only thing you can get from your usual source—bonuses are only available from the C&G store, and the ebook edition won't be out until serialization wraps up at the end of July.

I'm not sure when people who are getting bonus stories and USB keys will be receiving them, so lemme just tell you this now. For our mutual peace of mind, I ask that you hold off reading those stories until you finish the novel. "Dana no Chousen" takes place after the novel; "Found Objects" casually blows two of the Part Three reveals; and "The Time Somn Died" is, in my opinion, actually incomprehensible unless you've read the whole book and know a lot more about Ashley and the Constellation than you do now.

You can read "Pey Shkoy Benefits Humans" anytime, even though it "takes place" after the novel. It's got basic spoilers like "Tetsuo still teaches at UT Austin", but guess what, I just spoiled you on that.

Finally, an obligatory reminder: although has been an instance where the week's chapter didn't show up in the web archive, the emails are consistently sent out every week, and if you didn't get a chapter it's almost certainly in your spam folder.

Now on to real commentary. I wrote the contact event as a positive catastrophe, a shocking world-changing event out of nowhere which is absolutely wonderful. These days a catastrophe leaves a maelstrom of frantic Internet communication in its wake, a stew of information and guesses and wishful thinking and propaganda that slowly settle into an agreed-upon set of facts and opinions and crackpot theories.

This process has been happening in the background throughout the novel. You've only seen glimpses of it (the bits that Ariel contributes), but it's very important, because that's how I've been controlling the flow of the worldbuilding: flooding the zone with misinformation and letting the truth precipitate out when I'm ready to use it dramatically.

There must be CDBOEGOACC games about Ragtime and the Slow People. But I can't tell you all this stuff at once. There'd be no space for a story. My Creative License-ish solution is there's lots of information about this stuff once you know where to look, but no human consensus about what information is accurate. It's a mess of half-assed opinions mixed up with misinformation and conspiracy theory, with no way of judging the truth of the matter. (Bai will complain about this next week.)

It was easy to control the flow of information early in the novel, when I had the world's governments working on my behalf. In "Found Objects" Jenny has a hard time getting some basic information, because that story takes place during chapter 5. But with the Greenland Treaty in effect, the half-life of secrets has declined dramatically, and the worldbuilding is starting to flood the story.

But I still have control over one thing. Ariel is the narrator. There are secrets he has to keep, details he considers unimportant, and one thing he just doesn't want to tell you. Eventually he'll figure out the central mystery of the book, and he won't tell you that either. (Don't worry, I won't leave you hanging.) With Tetsuo blabbing all the stuff the Constellation played down in the first half of the book, Ariel's scheming and obstinacy and fear of embarrassment are my secret weapons for maintaining a relatively even pacing.

That was the big-think piece, now for the misc:

The beefiest commentary yet? I'm not going back to check. Instead I'm looking forward, to next week, when Tetsuo will say, "What were you smoking? Perhaps it was crack!"

Image credits: Tim Patterson, Matt Lancashire, Mark, Doug Kline.

← Last week | Next week →

[Comments] (7) Wed May 09 2012 12:53 Constellation Games Spoiler Conversation: I don't know how much traffic this will get, but now that the paperbacks are being sent out, I'm setting up this post for readers of my commentary posts who have read the whole book. Here you can comment on and ask questions about the chapters that haven't been serialized yet, or the novel as a whole. So have at it! I'll compile anything interesting that comes out of this and include it in the appropriate commentary posts.

[No comments] Sun May 13 2012 10:05 I WILL FLY: My in-laws gave us a sheet-feed scanner, so I've been scanning a box of my old school stuff, saved for posterity (which is now) by my mother. I'll be putting the prize of the collection online tomorrow, but in honor of Mother's Day I wanted to share this thing I drew in 1985, which was too big to scan so I took a picture:

It's a drawing full of mysteries. Some of my drawings were labeled, either by myself ("Triciratops") or by Mom ("helicopter"), but I don't know what that thing hovering above the ocean is or what inside the ocean is saying "I WILL FLY". (Maybe another flying fish like the one on the left?) I do know what's with the diacritical marks. I think the spelling book we used (probably Basic Goals in Spelling--I remembered it used "snurks" to refer to words not spelled as pronounced) taught us to mark up words that way to indicate their pronunciation.

Happy Mother's Day!

[Comments] (3) Mon May 14 2012 14:11 A Time Machine And Other Poems: Among my recent childhood scans were a number of books, written in pencil and bound with staples and tape. One of the earliest is a six-page chapbook of poetry called A Time Machine and Other Poems.

For the first time ever, I now present A Time Machine as it was originally intended to be seen: on the Internet. I wrote these poems sometime between the ages of 6 and 8, and I'm much happier showing them to you than the poems I wrote when I was a teenager. I think you will see that certain themes have been constants in my writing my entire life.

A note on the text: The poems were originally formatted as free verse, but they're clearly not free verse, so I reformatted them. I've corrected the spelling throughout except in one case where it was ambiguous. Strangely, there is no poem called "A Time Machine".

A Time Machine and Other Poems

Written and illustrated by Leonard Richardson

A time poem
There's no such thing as a time machine.
Even so you may sometimes wonder
If you could hear ancient thunder
If you could see an ancient beam.
If you could swim in an ancient stream.
So build a pretend time machine if you please.
And go and feel an ancient breeze.

The dinosaurs have died
The dinosaurs have died you see.
Even in the great big sea.
So when you're swimming in the sea,
Beware of dinosaurs, you and me.

Tyrannosauruses are red
Tyrannosauruses are red
Allosauruses are blue.
When you're near them,
Run away to. [sic]

How did the dinosaurs die out?
How did the dinosaurs die out?
Was it a whale with its spout?
No one knows for sure I know
But in a time machine I will go.

Other books in this series

Oh man. That "whale with its spout" line gets me every time. And the first poem's ABBAACC rhyme scheme is pretty nice.

I'm sure I wrote the "Other books in this series", but the only one I still have is Better Homes and Gardens ("WITH QUESTIONERES AND CHECKLISTS!!!). It's nowhere near as good as A Time Machine—like most magazines, it's full of padding—but it does include the immortal line: "Now buy the stuff you don't have." Better Homes and Gardens has its own "Other books" list, which promises a fifth book, Computer Games—also lost to history.

[Comments] (3) Tue May 15 2012 09:24 Constellation Games Author Commentary #25, "The Infiltration Path": I accidentally wrote a lot of this commentary as chapter 24 commentary, because the ambivalence scene really wants to be part of 24. It may have originally been in 24, but I had to move it out because there was too much stuff in there already.

One bit in the final chapter is presented out of order, but there's it's still chronological from a certain standpoint. Here in chapter 25, I just wasn't a good enough writer to present the events of the novel in strict chronological order. I don't know why this sort of thing bothers me so much. (Actually, I do.)

I'm tired of getting interrupted every week to write the commentary, so last week I made them my main project. I've completed commentaries up to the end of chapter 33 (but haven't chosen the images, which takes a while on its own). After chapter 36, there will be some short commentaries on the bonus stories and "Pey Shkoy Benefits Humans", and possibly one more on the book as a whole.

If you've read the paperback, the spoiler thread from last week is still open for your questions and comments. Here's last week's Tetsuo-licious Twitter feeds, and now commentary:

Now for all you loyal commentary readers, it's time for the first ever Constellation Games deleted scene. Early in the second draft, this chapter ended with Ariel and Tetsuo on the commuter train to Ariel's parents' house in College Station. I'll present the train conversation and then explain why I cut it:

"Do people ever ask you what your real name is?" asked Tetsuo. "When you tell them your name?"

"No," I said, "but I'm not a space alien who took a Japanese name."

"It seems very rude," said Tetuo.

"What is your real name?"

"Why do you ask me the instant I tell you I don't like to be asked? Tetsuo Milk is my real name."

"What was your name before you learned a human name to change it to?"

Tetsuo made a reluctant sound and then said "Don't transliterate that in your blog."

"That's pronouncable," I said. "Why'd you change it?"

"We always adopt local names on contact missions," said Tetsuo. "We've got to prove we're the most adaptable species in the universe. We're pretty conceited, honestly."

"Hey," I said, "that's our schtick. Humans are the most adaptable species."

"According to who?"

"That's just how it works. Everybody's the best at something. Farang are the strongest, Barbarians are the fastest, Her is the creepiest. Humans are the most flexible."

"Are you designing a role-playing game?"

"Better me than somebody who doesn't know basic rules of game balance."

"Everybody thinks their species is the most adaptable," said Tetsuo. "It's like patriotism. You like the Longhorns, your parents like the Aggies, who's to say who's right?"

"Those are football teams," I said. "Patriotism is for countries."

"Well, you get what I'm saying."

It's a pretty funny conversation, which is why I present it now, but I cut it because it has serious problems and I make better use of its ideas later on:

Basically, a much better scene in chapter 28 killed off this scene. Let's let it rest... in peace.

Tune in next week for the family reunion, during which Tetsuo will say, "Your brother's not a turtle."

Image credits: Thomas Deusing, Dave Herholz, NASA, Maureen Didde, Flickr user Perro Viejo.

← Last week | Next week →

[Comments] (1) Fri May 18 2012 12:54 ASINs that Spell Words: In the past year I've learned more about Amazon's ASIN product identifiers than... well, probably more than the median person wants to know, but not more than I want to know. One thing I've learned is that the ASIN for a print book is the same as its ISBN, but the ASIN for the Kindle edition of that same book is an Amazon-specific code. And where ISBNs are all numeric, non-ISBN ASINs tend to contain letters. Which means in theory you could have a Kindle book (or other Amazon product) whose product identifier was obscene.

Well, enough chitchat, let's look at some books whose ASINs end in five- or six-letter words! Courtesy of an Amazon site map and a word list:

[Comments] (1) Mon May 21 2012 11:33 118 Fifth Avenue: Marcel Duchamp's readymade urinal Fountain plays a small but important part in Constellation Games. As I was writing the commentary for the chapter where it shows up, something started nagging at me. Something that had been bothering me for a while in a low-key sort of way.

It's well known to Duchamp fans that ol' Marcel bought the Fountain urinal from the J. L. Mott Iron Works at 118 Fifth Avenue in New York. That's the story, anyway; I don't really trust anything Duchamp says about his readymades, for reasons documented elsewhere in this weblog. Anyway, the thing that's been bothering me is: what's at 118 Fifth Avenue now, ninety-five years later? Can you still buy a urinal there?

Probably not. Over the past hundred years all the plumbing supply stores have been pushed off Fifth Avenue and onto the side streets. But maybe there's an ABC Carpets there that sells overpriced toilet lids or something. I decided to go check.

The stupid thing is, I pass 118 Fifth Avenue all the time. It's right near City Bakery and the Union Square farmers market. But I never bothered to figure out which building was 118, until Saturday, when I went out with my camera and verified that 118 Fifth Avenue is now a Gap Body.

To the right you can see the same building in 1911, when it was the iron works. (Presumably just the showroom; the NYT says the factory was moved to New Jersey in 1902.) The facade clearly hasn't changed since then. Here's a Google Street View link that tries to copy the angle of the 1911 drawing.

Gap Body is exactly the sort of disappointment I was expecting, so I'm not really disappointed. I am glad to know exactly when I'm passing that little bit of art history. And here's a free idea for all you up-and-coming artists: buy a readymade tank top from that Gap Body and present it as a sequel to Fountain.

PS: this 2010 post on Ephemeral New York talks about statues atop 118 Fifth. Those statues are actually across the street, atop 91 Fifth, currently a J. Crew. This confused me greatly while I was out there, so I want to put an explicit correction on the web.

Image credits: Alfred Steiglitz, J. L. Mott Iron Works, yours truly.

[Comments] (2) Tue May 22 2012 09:02 Constellation Games Author Commentary #26, "Everyone With Cartoon Violence": This week Ariel faces his greatest self-imposed challenge yet: getting along with his parents. Let's listen in, shall we?

Actually, before we shall, I have some good non-novel news. I've sold my story "Four Kinds of Cargo" to Strange Horizons! Look for it late this year. It's not a Constellation story, but I think fans of the book will like it.

Here's last week's extensive Twitter feeds. I retconned one of Tetsuo's posts because he mentioned his cocktail experiments, which don't happen until this week. And now, a bulleted list:

Before I go, a little bonus deleted snippet from the Ariel/Curic conversation, which might be of interest:

"The mature thing to do is to send for help before you end up like the Inostrantsi."

"The Inostrantsi are still around," I pointed out.

"The Inostrantsi reproduce by budding," said Curic. "They didn't have much genetic diversity to lose. They're also immortal, so the surviving individuals had good memories of pre-collapse society. Let's not push our luck, Ariel."

Okay! Good commentary, everybody. Tune in next week for Tetsuo's first college lecture, during the course of which he will say, "I'm sorry, I just assumed there was space travel."

Image credits: Azmeen Afandi, Brian Sterling, yours truly.

← Last week

[Comments] (1) Fri May 25 2012 11:25 Crazy the Scorpion: A cooperative card game: I've been having a great time with a card game Beth Lerman and I invented, a game which I'm calling "Crazy the Scorpion", for the same reason "Exquisite Corpse" is called what it is. If you want a less interesting title it can also be called "Newsworthy".

Crazy the Scorpion is based on the "fun variant" of Man Bites Dog invented in 2010 by Kevan Davis, Holly Gramazio, and myself, but it's even more fun, and replayable to boot. I've tested it with two and three players. It should work with four, but probably not more than four. It plays in 20-30 minutes. I'm releasing these rules, and the print-and-play cards (see below) into the public domain.

Parts

To play Crazy the Scorpion, you need two decks of cards:

  1. A copy of Man Bites Dog.
  2. A stack of Trivial Pursuit cards.

Trivial Pursuit cards should be easy to find--in my experience, the Trivial Pursuit family is the single most common board game find at thrift stores and yard sales. You can also use Once Upon a Time cards or red Apples to Apples cards (not playtested).

Man Bites Dog is tougher to find, but I've made a print-and-play deck of 128 headline words. I constructed the words by looking at a news site, independently of Man Bites Dog, and the words are optimized for Crazy the Scorpion and not Man Bites Dog, and you can't play Man Bites Dog with my deck anyway because the cards have no point values.

Goal

The goal is to construct a 5x5 magic square of headlines, out of headline cards and Trivial Pursuit answers. The best way to explain the game is with a...

Sample of play

I start the game. I draw the Man Bites Dog card "SCAM", and a Trivial Pursuit card with these answers:

I lay down the cards like so, and designate the Trivial Pursuit card as the "Gopher" card.

The headline reads "Gopher Scam". Other legal layouts include "Scam The Montreal Canadiens" and "A Pen Scam". Anything that could conceivably be a headline in any universe. I could have laid out the headline horizontally or vertically.

Now it's your turn. You draw the Man Bites Dog card "DEVOTED" and a Trivial Pursuit card with these answers:

You lay down your cards like so, and designate your Trivial Pursuit card the "Abraham Lincoln" card:

Now there are two headlines: "Devoted Gopher Scam" and "Devoted Abraham Lincoln". Other legal placements would create headlines like "Gopher: Scam A Goalie", "Devoted Gopher" (created by placing "Devoted" above "Gopher"), and "Scam Prancer Devoted".

"Gopher A Goalie" is an illegal placement: it would put two Trivial Pursuit cards next to each other, which violates suggestion #1. "A Pen Devoted" is also illegal: it would rename "Gopher" to "A pen", violating suggestion #2.

Now it's my turn again. I draw the Man Bites Dog card "DRUGS" and a Trivial Pursuit card with the following answers:

I lay down my cards like so, and designate my Trivial Pursuit card the "A quantum" card:

Now there are six headlines:

  1. Devoted Gopher Scam
  2. Abraham Lincoln Drugs A Quantum
  3. Devoted Abraham Lincoln
  4. Gopher Drugs
  5. Scam A Quantum

Among other legal moves, I could have formed "Devoted Gopher Scam Drugs Dennis Rodman" instead.

Halfway through my third turn, we might have a nice 3x3 magic square that looks like this:

Or, in textual form:
DevotedGopherScam
Abraham LincolnDrugsA quantum
JudgeA netTourist

Forming these six headlines:

  1. Devoted Gopher Scam
  2. Abraham Lincoln Drugs A Quantum
  3. Judge A Net Tourist
  4. Devoted Abraham Lincoln
  5. Gopher Drugs A Net
  6. Scam A Quantum Tourist

(More likely, that early in the game we wouldn't have a magic square at all. But this makes for a better illustration.)

By the end of the game we'll have headlines like "The Ladybug Judge A Net Tourist, Charlemagne Blasts" and "Nicotine-Devoted Abraham Lincoln: Judge Buckminster Fuller's Movie."

Now I think you're ready for the...

Official Rules

The goal is to construct a 5x5 magic square of headlines. Or, for the adventurous, a 6x6 square. Players take turns drawing two cards (one from each deck) and placing them in a grid. The game ends when the magic square is complete. The game may end in the middle of one player's turn.

When playing a Trivial Pursuit card, the player names the card after one of its answers. The card is considered to have that name for the rest of the game.

Each played card must be orthogonally adjacent to at least one card already played.

Suggestions

The game is better if you follow these suggestions, but in specific cases you might get funnier headlines by breaking them.

  1. Man Bites Dog cards should not touch other Man Bites Dog cards, and Trivial Pursuit cards should not touch other Trivial Pursuit cards. You want to get a nice checkerboard pattern.
  2. Don't rearrange, rename, or remove cards once they're played.
  3. Headlines must make some kind of sense at every stage. This is more a requirement that you come up with a story about each headline, than an admission that there's some sequence of words that cannot conceivably be a headline.

Analysis

The Man Bites Dog cards are full of words that clearly belong in headlines, but which (for the sake of generality) include no details. Trivial Pursuit answers are disconnected references to newsworthy topics. Combining them yields sentences that feel like real, specific headlines, but make no sense whatsoever.

Have fun!

<Y
[Main]

Unless otherwise noted, all content licensed by Leonard Richardson
under a Creative Commons License.