News You Can Bruise

Beautiful Soup 4 Beta 4

Wed, 08 Feb 2012 15:02:03 GMT

Beautiful Soup 4 beta 4 is out! You can install it with easy_install beautifulsoup4 or pip install beautifulsoup4. You can also download the tarball or check out the Bazaar repository.

Big changes:

If you're using Python 3.2, the built-in html.parser is now reliable enough to use on its own. You don't need to install lxml or html5lib just to parse bad HTML (but lxml is still a lot faster). The forthcoming Python 2.7.3 should also work this way.
This is of course a feature of Python, but due to a pretty bad bug in html.parser, I wasn't taking advantage of it. I worked with Ezio Melotti to monkeypatch that bug from within BS, and now we're back in the very good situation of not needing any external dependencies.
new_tag() will follow the rules of whatever tree builder was used to create the original soup. For example, a new <p> tag will look like "<p />" if you're dealing with XML, but it'll look like "<p></p>" if you're dealing with HTML.
There's now a new_string() method to go along with new_tag().
There are two new methods for manipulating the tree: PageElement.insert_before() and PageElement.insert_after().
I replaced the substitute_html_entities argument with the more general formatter argument. You can do all sorts of crazy stuff with this.
The default formatter converts bare ampersands and angle brackets to XML entities, but doesn't touch HTML entities. I think it's kind of America-centric to convert characters like é to é by default, but I might make the default a "punctuation" formatter that converts things like curly quotes to HTML entities.

Constellation Games Author Commentary #11: "Launch Title"

Tue, 07 Feb 2012 12:48:44 GMT

Love those title puns! This blockbuster episode sends Ariel TO THE MOON and introduces two major new characters, Tetsuo Milk and Ashley Somn. Also a minor but important character: Linda Blum, Ariel's mom.

Here's last week's Twitter archive, which ran two weeks ago due to my own errors. Twitter service has now resumed, but because this plot arc is so compressed (the rest of Part One crams two weeks of frantic activity into five weeks of real time), most of it is going to come out on Tuesdays and Wednesdays. Don't be afraid, I'll be here the whole time with long-winded commentary:

It's such a relief to be able to talk about Tetsuo! So much happens in this chapter, he doesn't get a lot of time with Ariel, but that changes starting next week. Tetsuo is great, I love him a lot, but... he's a scene stealer. Anything I wrote, he would grab and run off in some weird direction. When the Aliens were choosing human names, Tetsuo is the guy who picked a name because it means "iron man."
Tetsuo reminds me of Londo Mollari from Babylon 5, in that he starts out a comic relief character (insofar as a comedy can have designated comic relief) and over time reveals more serious facets of his personality. But unlike Londo, Tetsuo never stops saying goofy shit. Tetsuo is the infrafictional author of the subscriber bonus tome "Pey Shkoy Benefits Humans", which is set about six months after the end of the novel, and he's still at it.
Ashley Somn is not a scene-stealer, so her husband kind of overshadows her for most of the book. But she's an awesome character on a slow burn. She's got a high-drama character arc that revs up in the last third, and which I fill in with the bonus story "The Time Somn Died." (Title is not a spoiler.)
I keep forgetting that Tetsuo is orange and Ashley is green; I always imagine it the other way around. They're bright neon colors with darker spots, like tropical frogs. Why? Are Aliens poisonous? I dunno. Lick one and see!
The short scene with BEA Agent Krakowski in back of the strip club is the very last thing I wrote for Constellation Games. Its main purpose is to dramatize the sub rosa assignment Krakowski gives Ariel, an assignment which becomes very important in Part Two. But I also threw in the talking rat, to introduce you to something else that's very important in Part Two: the idea that Ariel might not be the most reliable narrator.
I don't know why Krakowski and Fowler were at a strip club in the middle of the day, but I'm sure it was work-related.
The part of this chapter that's an excerpt from Ariel's Twitter feed will not be shown on his actual Twitter feed, because that would be annoying. But that section was the main inspiration for the in-character feeds in the first place.
Original title for this chapter: "MAN WALKS ON FUCKING MOON."
One of the big problems with the second draft was that for most of it, the tone was emotionally distant. It took me a while to understand the characters. The blog format didn't help, and I moved further away from it in every draft. Ariel is a guy who shows vulnerability in huge dramatic bursts and won't open up otherwise. Etc. etc. Anyway, I worked on this a lot in the third draft, and one of the big changes is that at Andrew Willett's suggestion I modeled Ariel's lunar excursion on this Narbonic strip.
Ariel's reaction to being launched into space is taken directly from what I imagine would happen to me. Also possibly taken from my reaction as a kid to the Disneyland exhibit "Mission to Mars", which had a similar setup where you saw the ground drop out from underneath you while feeling absolutely no acceleration.
The sculptures of the figures from the Pioneer plaque (not "Carl Sagan's gold record", as Ariel mistakenly believes—that's the Voyager record) are another moment of not-quite-understanding taken from "Vanilla". Ariel's initial description of the docking bay is the opening parenthesis of a pretty huge piece of bookending, so watch for that.
One of the imaginary book covers I had in my head while writing was a design based on the Pioneer plaque, except with (clothed) Ariel and Jenny.
Ariel mentions some Eritrean refugees living on Ring City, but they're only the biggest and most famous group of refugees—the ones Ariel knows about. Human Ring is also home to smaller groups of refugees from around the world, and to miscellaneous individuals living under the radar. The actual population of Human Ring at this point is closer to 700 than to 500.
The refugees come up a few times later on, but they don't play a big part in the novel because I don't currently feel I've got the literary chops to tell their stories. But I knew that not mentioning refugees would be unrealistic. It would imply that humanity's governments were able to coordinate to completely lock down the planet, or that the Constellation was sending away asylum seekers. What we have is a compromise, not one I'm happy with, but I think the best I could do.
I know this is already super long, but I want to introduce a new segment here on CG Author Commentary, a little recurring bit I like to call Creative License. Sure, I write silly stories about space aliens visiting Earth seemingly in violation of the Fermi Paradox, but that doesn't mean I can just make stuff up. At the same time, I want Constellation technology to appear very close to the "indistinguishable from magic" line. Creative License explores that tension by pointing out things that probably can't exist in real life, and the made-up reasons I use to justify their existence in the story.
First we have the shuttles the Constellation uses for short hops to Earth and Luna. I have no idea how they go as fast as they do, but I have a vague idea how they achieve a smooth ride: they use ports to maintain an acceleration differential between the inside of the shuttle and the outside, so the inside accelerates at a gravity-like rate while the outside powers up towards some horrendous speed. Ports are very important bits of Constellation tech and need their own segment on Creative License. I'll probably do them next week, after we see one in action.
But this week we also have the Constellation spacesuits. Inflatable spacesuits are nothing new, but Ariel's suit folds up when not inflated and doesn't seem to have any space for hard parts like air tanks, a fluid recycler, the comm system he plugs his phone into, or a way of dissipating heat. Creative License Solution: as you'll see later, the Constellation does pretty amazing things with origami. I imagine all that fine machinery is packed flat and inflates to the correct shape with the rest of the suit.

What a huge commentary, and this plot arc's just getting started. Be sure to tune in next week, when Ariel will say, "I do not use sex to maintain social cohesion."

Image credits: Andy Bernay, Joe Mabel, Linda Salzman Sagan, unknown.

<- Last week

Mon, 06 Feb 2012 21:55:09 GMT

Earlier I ran some speed/accuracy tests of Beautiful Soup driven by various parsers. Python's built-in HTMLParser scored very poorly, parsing only 52% (Python 2.7.1) or 57% (3.2.2) of my test pages without raising an exception. Well, Ezio Melotti, the maintainer of HTMLParser, has been working for a while on improving HTMLParser's handling of bad HTML. Most of this code is in Python 3.2.2, so I should have been getting the benefit, but it wasn't working for me because of a semi-related bug in HTMLParser, which is fixed in the as-yet-unreleased 3.2.3.

After talking with Ezio today, I was able to monkeypatch BS4 to avoid the bug in 3.2.2. This means on Python 3, BS4 with no external parser installed will give reliability comparable to BS4+lxml (98% versus 99%). It's still about 50% slower, though, parsing about 1300 kb of HTML per second, versus 2100 kb/second for BS4+lxml.

easy_install beautifulsoup4

Thu, 02 Feb 2012 15:59:19 GMT

This is an HTMLized version of an email I sent to the Beautiful Soup discussion group, about the impending release of Beautiful Soup 4.

Introduction

When Beautiful Soup was first released in 2004, the state of HTML parsing in Python was appalling. Over the past eight years, things have improved so dramatically that Beautiful Soup's HTML parser is no longer a competitive advantage. I don't want to duplicate other peoples', work, so I'm getting Beautiful Soup out of the parser businesss. Beautiful Soup's job is now to provide a Pythonic screen-scraping API on top of a data structure created by a third-party parser.

This will be Beautiful Soup 4, and I've been planning it for years. With help from Thomas Kluyver and Ezio Melotti, I've now met the three main goals of Beautiful Soup 4:

Make a single codebase that works under Python 2 and Python 3.
Stop using SGMLParser (removed in Python 3) and make it possible to swap out one parser for another.
Support two major Python parsers (lxml and html5lib) as well as Python's (not currently very good) batteries-included parser, html.parser.

The first version of BS4 is almost ready for release, and I'd like you to test it out, if you haven't already. I still to fix some things, in particular some performance problems. But, note that even with the performance problems, BS4 is faster than BS3 across the board.

On Python 2 or Python 3 you can install the BS4 beta with this command:

easy_install beautifulsoup4

You can also get the source tarball.

The documentation has been completely rewritten. You may find the section on porting BS3 code to BS4 especially interesting.

There are three major things I'd like your feedback on before completing the release.

Hall of Fame

The BS3 documentation lists open-source projects that use Beautiful Soup. I stopped maintaining this list many years ago because there are hundreds of these projects, and since most of them are screen-scrapers, they're pretty ephemeral.

I'd like to bring this feature back as a "hall of fame", featuring applications of Beautiful Soup that grab a reader's attention. People who used Beautiful Soup in a high-profile way or to tackle a big issue. Projects that are interesting to hear about even if the software doesn't work anymore, or uses an old version of Beautiful Soup, or if Beautiful Soup was used internally and the public only saw the results.

My bias is towards projects having to do with space, science, journalism, politics and social justice. Here are some examples so you know the kind of thing I'm thinking of:

"Movable Type", a work of digital art on display in the lobby of the New York Times building, uses Beautiful Soup to scrape New York Times feeds.
Alexander Harrowell uses Beautiful Soup to track the business activities of an arms merchant.
The Lawrence Journal-World used Beautiful Soup in 2006 and 2010 to gather election results.
The NOAA's Forecast Applications Branch uses Beautiful Soup in TopoGrabber, a script for downloading "high resolution USGS datasets."

If you did anything of this sort, or know of someone who did, I'd like to hear about it.

Do you prefer lxml or html5lib?

Right now, the parser ranking goes lxml, html5lib, html.parser. I like lxml because it's incredibly fast and it can parse anything. But I'd like to see what you think of the trees it generates. Would html5lib, with its web-browser-like heuristics, be a better default?

substitute_html_entities

BS3 had a number of overlapping and inconsistent ways of turning HTML/XML entities into Unicode characters, and possibly turning Microsoft smart quotes into HTML entities at the same time. In BS4, all this stuff is gone. HTML and XML entities are *always* converted into Unicode characters.

This is great but there's one problem: output. If you want to turn those Unicode characters back into entities when outputting as a string, you need to call soup.encode(substitute_html_entities=True), which is a little clunky. I'm thinking of adding an output_html_entities attribute that you can set on a soup or tag to control whether this substitution happens. Do you like this idea?

I think I also need to ensure that characters like "&" and "always converted to XML entities on output, even though this will hurt performance a bit.

Conclusion

What you install with easy_install beautifulsoup4 is a beta release. If I hear of a problem soon, there's still time to fix it, even if it means a major change to the API. So please try it out and give me feedback.

Constellation Games Author Commentary #10: "K.I.S.S.I.N.G."

Tue, 31 Jan 2012 13:18:31 GMT

This is Dana Light's big chapter, and I'm having trouble writing commentary because it's pretty self-contained. A problem is introduced and Ariel solves it by the application of technology. If I hadn't been writing a novel when I came up with Dana, this chapter would have become a short story, maybe part of a sequel to "Mallory". It would have been about the way evil psychologists use game mechanics and the ELIZA effect to manipulate users into spending money, and the way people get real pleasure from spending money on things designed to manipulate them.

Although evil psychology does show up in Constellation Games, I didn't have as much space for it as I'd like. Instead this chapter shows the first grown-up thing we see Ariel do. In a world in which sub-human-level AI has suddenly become very common, Ariel decides to empathize with it.

He doesn't anthropomorphize Dana. Dana doesn't pass the Turing test, she isn't terribly smart or self-aware, but she's capable of happiness and she doesn't deserve to be deliberately made unhappy by evil psychologists. This attitude is what ultimately makes Ariel a hero, not just a POV character. The consequences of his decision to empathize will run through the entire book, and then overflow the book into "Dana no Chousen," and I still don't know when and whether Ariel does the right thing w/r/t Dana. But you gotta have empathy.

Apart from that, I don't have much to say. Here are a few miscellaneous notes:

As you might expect, a lot of this stuff will come up again in "Dana no Chousen". But the callback you probably won't notice unless I point it out is that Dana loves popcorn.
I enjoy many bits of this chapter but my favorite is Bai's big moment of lucidity, when he immediately detects and shoots down Ariel's Manic Pixie Dream Girl fantasy. (And you can bet that's gonna come up again.)
I'm sure that G'go Investigation: When You Gotta Die makes sense in cultural context. Like, imagine if the first thing you learned about 21st-century Earth was Mario Kart: Double Dash.
I really like the design of the chainable memory cylinders on the Simulates Hi-Def False Daylight. In the second draft, "[False Daylight] games were distributed as a set of ROM chips, snapped onto standardized circuit boards, and enclosed in a removable plastic case to be plugged into the computer's game slot." This led to chips popping out, hiding in the carpet and stabbing people in the foot. That's a design in keeping with the generally poor quality of Ip Shkoy consumer goods, but it doesn't fit with the fact that the False Daylight is a clone of the Brain Embryo, so I switched to the much cooler chainable cylinders.
Originally I transliterated Bai's "bro" as "bra". Everyone hated this. I changed it to "brah". The hate did not abate. What is wrong with you people? "Brah" is an accurate transliteration! It's so accurate.

Tune in next week for action, intrigue, and romance between people at the same level of sentience. It's the only chapter when Ariel will say: "I just have a slight fear of being a tiny speck in the infinite cosmic void." But not the only chapter when he'll think that.

PS: Due to an error on my part, the chapter 9 Twitter feeds ran as part of chapter 8, and chapter 10's Twitter feeds ran last week. This really can't go on, because next week's feeds are tightly integrated with chapter 11. So except for a brief bit of bonus material I just wrote, there will be no Twitter stuff this week. Sorry about that!

Photo credits: Kevin Trotman and Peter Anderson.

<- Last week | Next week ->

Fruit to Fruit

Sat, 28 Jan 2012 13:58:33 GMT

Time for another crummy.com Apples to Apples variant (previous editions), this one discovered last week by Pat.

On every green A2A card there's the name of the card, like "Handsome", but there are also three related words, like "attractive", "elegant", "fine". In Fruit to Fruit, you don't read the name of the card. You just read the related words. Sometimes the related words are so similar that you might as well be reading the name of the card, but usually something goes missing (such as the masculinity of "handsome"), leading to funnier red cards being put down. The name of the card is finally revealed during judging.

We had a great time with this and played it in conjunction with the Apples to Placebos variant, even though there were four players. You might think this overkill, but at this point A2A is more a social activity than a game. Anyway, it says right on the box "The game of hilarious comparisons!", so anything that makes the comparisons more hilarious is legit.

While seeing if anyone else had come up with this variant I discovered Apples to Trivial Pursuit, and the improv comedy variant. I also discovered that the game is patented, and that there is an entire patent classification system for "means... by which contests of skill or chance may be engaged in among two or more participants, where the result of such contests can be indicated according to definite rules."

CG Author Commentary #9: "Import System"

Tue, 24 Jan 2012 13:06:35 GMT

Last week and this week have some of my favorite Twitter bits (e.g.) because the CDBOEGOACC is finally available in English. Sunday night while working on Loaded Dice I realized that one of the reasons I really like playing around with the BoardGameGeek dataset is it's like a real-life CDBOEGOACC.

The flip side is this chapter doesn't have a lot of plot. But hopefully you're okay with that because of all the fun mini-stories like the Sea Level game/food. It's supposed to represent the design phases of a software project, where you're throwing around a lot of ideas but not much is being produced.

Next week is a set piece, and after that the plot won't let up until the cliffhanger that ends Part One. Before that happens, I need to get some solid exoludology in to bring in topics that are important later, like Sayable Spice and Ariel's unsuccessful attempts to translate it.

Before beginning the chapter 9 commentary, I want to get something off my chest about the first sighting of the Farang in chapter 1. In that chapter, Ariel compares their antennacles to the oral tentacles of a "cerebrophage". In the second draft I just out and said "mind flayer". My writing group said I should change it because readers might not know what a mind flayer is. ("Did you mean: mind flower?") Taking their advice to heart, I changed the reference to a made-up reference that nobody will get. Well, at least we're all in the same boat now!

And here's chapter 9. Vent your egg sacs before reading this commentary:

This chapter represents the absolute end of the abandoned first draft. Beyond this point everything is from the second or third draft.
In a questionable move on my part, Ariel gets an Alien computer before he meets any Alien characters, requiring that I introduce you to the species with an infodump ("eight-foot monkey-lizards"). Don't worry, in just a couple weeks, Alien characters will show up and run off with the whole damn book.
Speaking of infodumps, I want to do a little infodump of my own, about the Ip Shkoy. The Ip Shkoy were an ancient civilization of Aliens, much like the Roman Empire was an ancient civilization of humans. "Ip Shkoy" is not the native-language name for the Alien species. I tried to make this abundantly clear, but I've noticed well-intentioned people calling the Alien species "the Ip Shkoy" or ascribing to modern Aliens the (frequently pretty awful) opinions of the Ip Shkoy. Which would be like Curic thinking that Ariel regularly offers sacrifices to Jupiter Optimus Maximus.
Star Trek has conditioned us to see an ET species as having a single homogenous culture that never changes, and this sort of confusion is why they do that on Star Trek. That said, I don't think this is anyone's fault but my own. If I'd presented modern Alien society in as much detail as I present the Ip Shkoy, the other probably wouldn't crowd out the one. It doesn't help that certain features are shared by both cultures, such as transitive pair bonding (aka polyamory).
Recapture That Remarkable Taste, the Ip Shkoy remake of Sayable Spice, is not to be confused with the new William Gibson anthology, Distrust That Particular Flavor.
Charlene Siph is mentioned again, which gives me a good excuse to talk about Alien names. The Aliens on the contact mission have all taken human first names, but their surnames are monosyllables which I usually generated by truncating creepy English words ("siphon", "somnolent") to four letters. The impression I want is of someone who's trying to be accommodating but doesn't quite have it down.
This is another detail imported from "Vanilla", one that I'm really happy with, one that even becomes important to the plot in one place. And if you like symbolism, check this out: "Ariel Blum" could be an Alien name.
In the second draft I wrote a whole review of Proty's Big Escape, but turns out it's a dumb idea to insert one game review wholesale into another game review. Let me know if you want the Proty review, and I'll make it the first CG Deleted Scene, even though it's only five paragraphs long.

Be sure to tune in next Tuesday, when Dana will say, "This application will terminate due to suspected theft or circumvention."

Oh, and you might want to keep an eye on @Tetsuo_Milk.

Image credits: Flickr user krusty, Guillaume Piolle, and Flickr user CoffeeGeek.

<- Last week | Next week ->

To This Basic Game Hedgehogs Are Added

Mon, 23 Jan 2012 15:43:02 GMT

I bought a cute game about hedgehogs, Der Igelwettkampf ("The hedgehog contest"), as a Christmas present for my niece. On Der Igelwettkampf's BoardGameGeek page I noticed that it was classified under the game family "Animals: Hedgehogs/Porcupines". I'd thought "Family" was for boring things like grouping together the endless versions of Ticket to Ride, but turns out it's also used to group together all the games about hedgehogs.

The question then arises: what's the best game about hedgehogs? According to BGG it's Igel Ärgern + Tante Tarantel, a double bill in which Tante Tarantel might be doing some of that work because Igel Ärgern on its own is rated a bit lower.

More importantly, what's the worst hedgehog game? Indubitably it's Hedgehog's Revenge, "The GAME where the hedgehog STRIKES BACK!", whose BGG description includes the now-hopefully-immortal saying "To this basic game hedgehogs are added."

At this point I was on a roll... of the dice! I went back to my now-old BGG data dump, sorted the board game families by how many games they contained, and picked out interesting groupings for use in Loaded Dice. We've got Games about animals (most popular: dogs) Game versions of sports (soccer), and Games about countries (the Roman Empire, in a landslide). That page shows the top-rated game and the lowest-rated game, so get ready to load a lot of cover images.

I did a couple other lists, like media tie-ins (champion: Disney) and "families" that are strongly tied to one single game (the 889-strong "Monopoly" family), but I think the three lists I put up are the most interesting.

Bizarre trivia abounds! Did you know that crows are board game gold? The worst game about crows (The Crow and the Pitcher) has a BGG rating of 6.32, which isn't that bad at all. (Longtime fans will remember the median rating is 6.0).

Did you know there are twenty rodeo-themed games? Apparently you didn't, since only one of those games has more than five ratings. How many wargames take place in Switzerland, a country that doesn't fight wars? Only two: Switzerland must be Swallowed and Zürich 1799.

My data is six months old now and it's starting to show some cracks. There are BGG families for Russia and Antarctica which were created after I took my dataset, so they don't show up in the country list even though most of their games are in my data. After getting the Switzerland idea I ran the "What percentage of a country's games are wargames?" test on all countries, but wargames were drastically undercounted. For instance, all but one "Vietnam" game on BGG is a wargame (the exception being Venture Vietnam), but only 35% of those games were classified under a general "Wargames" category.

But, the lists are still a lot of fun and there are some interesting games in there. I'll leave you with the board game equivalent of the dusty World Book Encyclopedia sitting on the shelf at your grandparents' house: Trivial Pursuit - The Year in Review - Questions about 1992, the worst-rated game (3.90) in the 155-strong Trivial Pursuit family. Also available in 1993 flavor!

Beautiful Soup 4 Benchmark

Sun, 22 Jan 2012 13:43:27 GMT

This is going to go into the Beautiful Soup 4 documentation, but you might find it interesting. It's my first legitimate benchmark of BS4, and the first benchmark of this stuff I've seen since Ian Bicking's excellent 2008 benchmark.

Ezio Melotti pointed me to a list of the top 10,000 domains worldwide, according to some random source. It looked legit, so I wrote a script to download the homepages of the top 200 domains as served to a desktop web browser. My dataset included many pages written in Chinese, Japanese, Russian, Portuguese, Polish, and German.

For every parser I was interested in, I parsed each homepage and timed the parse. This gave me 200 numbers for every parser. To reduce that to a single non-huge number I calculated a mean: how many kilobytes of real-world HTML the parser could process in a second. I also noted each parser's success rate: how many of the 200 homepages it had handled without raising an exception.

Here are the results, ordered by their performance under Python 2.7.

	Python 2.7		Python 3.2
Parser	Speed (KB/s)	Success rate	Speed (KB/s)	Success rate
Beautiful Soup 3.2 (SGMLParser)	211	100%	-	-
html5lib (BS3 treebuilder)	253	99%	-	-
Beautiful Soup 4.0 + lxml	255	100%	2140	96%
html5lib (lxml treebuilder)	270	99%	-	-
Beautiful Soup 4.0 + html5lib	271	98%	-	-
Beautiful Soup 4.0 + HTMLParser	299	59%	1705	57%
html5lib (simpletree treebuilder)	332	100%	-	-
HTMLParser	5194	52%	3918	57%
lxml	17925	100%	14258	96%

Note that the "HTMLParser" tests don't actually produce anything. HTMLParser is an event-based parser, so when the HTML is parsed, nothing comes out because I didn't include any handler code. All the other tests build a parse tree in memory.

Another thing to keep in mind about the html5lib results: html5lib is kind of the opposite of BS4. BS4 always builds a tree of Beautiful Soup objects, but you can tell it to generate that tree using html5lib, lxml, or HTMLParser. Whereas html5lib always uses its own parser, but you can tell it to build a tree of lxml objects, a tree of BS3 objects, etc.

The big surprise for me is that on Python 2.7, lxml is the worst choice for a parser to drive BS4. It's a worse choice than html5lib! How did that happen? I have no idea. I was hoping to cash in on the lxml magic (see below), and it's not working. I need to look into this. Notice that html5lib takes a performance hit from using lxml's treebuilder. If the magic's not in the treebuilder and it's not in the parser, where is it?

Unless I can find that magic and exploit it, it remains the case that if you're paying by the minute for computer time, you should use lxml. It's written in C, and on Python 2.7 it builds a parse tree sixty times faster than BS4, three times faster than a pure-Python parser that does absolutely nothing with the data. Even on Python 3, lxml alone is seven times faster than BS4+lxml. I said stuff like this in the BS3 documentation, but I think I need to be more forceful about it in the BS4 docs.

The good news is that Beautiful Soup is 6-8 times faster on Python 3 than it is on Python 2, and even at its slowest, BS4 is noticeably faster than BS3.

The big caveat is that my definition of "success" is pretty minimal. Just because the parser parsed the file without crashing doesn't mean it will give you a useful parse tree.

Another caveat: on Python 3, I couldn't get HTMLParser to take raw bytes as input, so I ran the data through UnicodeDammit first. I counted this time as part of the parse time. This probably explains HTMLParser's slower speed on Python 3 and its higher success rate.

Update: Argh, I found out about this a year ago. The problem is that Unicode, Dammit is incredibly slow in some cases. Here are the results on 2.7 if I take out the prepare_markup methods in the builders for HTMLParser and lxml, and just assume everything's UTF-8:

	Python 2.7		Python 3.2
Parser	Speed (KB/s)	Success rate	Speed (KB/s)	Success rate
Beautiful Soup 4.0 + lxml	2287	96%	2600	96%
Beautiful Soup 4.0 + HTMLParser	2069	48%	1680	57%

That's more like it! The problem is that reliability suffers. Both parsers crash in the 4% of cases where it's not UTF-8 but the encoding is declared in a <meta> tag. And there's an unknown number of cases where the data's not UTF-8 but the conversion doesn't crash, leading to garbled data. But at least now I remember this problem.

Also note that on Python 3.2, getting rid of Unicode, Dammit doesn't matter nearly as much. (It doesn't matter for HTMLParser at all.) Presumably Python 3.2 has better built-in support for encoding autodetection.

Beautiful Soup 4 Beta 2

Fri, 20 Jan 2012 21:40:15 GMT

Thanks to some help from Ezio Melotti I've got the Beautiful Soup test suite passing on Python 2.7 and Python 3.2. Here's a tarball containing the original Python 2 module in bs4, plus the same code autoconverted to Python 3.2 in py3k/bs4.

I'm still not sure about the best way to distribute this package, either while it's beta or afterwards. I'll probably end up creating a new project on PyPi, because otherwise people who install programs that easy_install beautifulsoup will crash due to the module's new name. Does that make any sense?

Anyhow, we're almost at the end of this fitfully travelled road. Once I figure out distribution and rewrite the documentation, a) no one should need to use BS3 anymore if they don't want to, and b) it should be possible to get lxml-like performance or html5lib-like flexibility with a Beautiful Soup API, by actually using lxml or html5lib as the underlying parser.

PS: remember, it's now from bs4 import BeautifulSoup.

CG Author Commentary #8: "They Came For Our Twinkies"

Tue, 17 Jan 2012 14:10:43 GMT

K'chua! Such a useful word. This week, Curic does her part to Keep Austin Weird. Here's the (tiny) Twitter archive from last week.

Some exciting news from the world of commerce: the Constellation Games paperback drops April 17. If you're waiting for the paperback, do yourself a favor and pre-order at the $20 level. Once it's released, the paperback will cost $20 on its own, but if you pre-order, you'll also get a bunch of extras, including three short stories that all pass the Bechdel test.

The seventeenth of April is also the day we serialize chapter 21, "Her". I'm going to keep posting my commentaries once a week along with the serialization, even though a growing number of you will have read the whole book and know how it turns out. Then you'll know how I feel right now!

I'm also thinking of having a celebratory book launch dinner at Hill Country, a famous Austin-area barbecue joint that fortuitiously has a branch in New York City. Let me know if you're interested in attending.

...and we're back from commercial. Here's the commentary for chapter 8:

Hopefully this chapter doesn't seem too special now, but when I first wrote it it was a revelation. It was a narrative with a plot! Curic was an active character and you could see her chemistry with Ariel as they physically interacted! My writing group indicated in no uncertain terms that this was the stuff they had been waiting for.
I wrote a couple more chapters in the old, boring style but it wasn't long before I gave up and started the second draft, which tried to make the whole book more like this chapter. To this end I introduced innovations like the long IM conversations, and Jenny. Imagine reading up to this point, except Jenny has only been mentioned once and Curic has only had three lines of dialogue. That was the first draft.
Because of its importance to the book's history, and also because it's such a great set piece, I kept coming back to this chapter. Near the end of the book, I recontextualize it by showing the visit to Earth from Curic's perspective. And Jenny's remark that the plastic fractal "looks like Skewer Sue's bracelet" fed, as I hinted a few weeks ago, into one of the book's most important scenes.
Which almost didn't happen. At this point I'd decided that Jenny was a big comics nerd and wanted her to reference a kinda bland comic book character name. My first try was "Titania". Well, the joke was on me because Titania is an actual Marvel villain. Sumana says she showed up during Dan Slott's run on She-Hulk, so that's probably where I got it, but I don't remember her. Bland, but already taken. Titania was out. Then I came up with "Skewer Sue", which is not a top-tier name like "Wonder Woman", but is definitely not bland. Then I decided "why should I deliberately come up with bland names?" and went with Skewer Sue. Believe me when I say that if I'd been able to name that character Titania, the offhand reference would have stayed offhand forever.
...And I can't even tell you all the other things in this chapter that become important later. Because they're all huge spoilers. But in one case also because it's quite embarrassing to admit how long I took to realize that I could reuse something. I should have called this chapter "Guns on the Mantelpiece". (Actually, I should have called the second chapter "Guns on the Mantelpiece"; I never liked "Corner Pieces".)
Jenny's nephew Eduardo does not really show up again, except in the Twitter feeds and one of the deleted scenes, but if you want to make your protagonists look good you should show them doing nice things for kids. Just a little tip I picked up in Sid Meier's screenwriting class.
Eddie was originally the son of Jenny's brother, James, who never shows up at all. I don't remember why James became Jenny's sister, but when I changed it I de-named the character so as not to use up a "named character" slot in your head.
Actually, rereading this, I notice I didn't give Eddie any lines. That was kind of sloppy. I'd retcon it by saying he's intensely shy around Curic, but Curic's account of the day contradicts this! Oh no.
The star-draw was added in the third draft. There's another star-draw in chapter 20, but a lot of other stuff is happening in that scene, and Ariel wasn't actually there for it, so the exposition was very difficult. Putting a star-draw here takes some of the load off the chapter 20 scene, and--only later did I realize this was much more important--shows you a fluid overlay in action. Albeit a two-person fluid overlay because Jenny and Bai are slackers and Eddie's a little kid.
The celebrity voices used for the different ET vocalizers are a tiny import from "Vanilla", in which (among other things) the protagonist was shaken down by an ET who sounded like Garrison Keillor. Now that I write that down, I realize that I've only got one joke about this, because there's a very similar incongruous-voice bit in chapter 21. Maybe I should have declined to import this particular thing from "Vanilla"... it feels like an artifact from a less mature stage of my style.
The bit where Bai introduces Dana to Curic was added after I sold the book, at editorial request to get more Dana early in the story. You'll thank my editor later.
And I don't know when all of this came together, but I love Curic's mix of anthropological curiosity, disinclination to tolerate bullshit, and bizarre opinions on what's bullshit. Her reactions to Dana, to "Eddie wants to be an astronaut", and to the revelation that "fuck" is a swear word... she's got all the best lines. And she does the game reviews! This is the chapter where Curic came alive for me. Metaphorically.

Stay tuned for the inevitable letdown next Tuesday, when Curic will say, "I did not pee in your sink."

Image credits: (CC) Larry D. Moore and Wikimedia Commons users Solkoll and SeppVei.

<- Last week | Next week ->

Findings

Mon, 16 Jan 2012 14:50:56 GMT

My writing life has settled down a bit so I'm finally going to write about Findings, the social reading startup where I worked last summer. This is more an essay about what I see in Findings than an introduction to the site--you can see lots of general introductions linked to from this Findings blog post, including co-founder Steven Johnson's introduction, and the Business Insider article whose title is the perfect elevator pitch, "Findings is GitHub for Ideas".

If what I'm about to say sounds interesting to you, there are development jobs open at Findings right now. Just as a reminder, I myself don't work at Findings anymore, and even when I did, only the foggyheadedest knave would have taken my personal opinions as representative of company policy.

Let me start out with this quote I took from Darwin's The Descent of Man, not because the quote itself proves anything, but because the quote is an important part of my reading of Darwin:

Brehm gives a curious account of the instinctive dread, which his monkeys exhibited, for snakes; but their curiosity was so great that they could not desist from occasionally satiating their horror in a most human fashion, by lifting up the lid of the box in which the snakes were kept. I was so much surprised at his account, that I took a stuffed and coiled-up snake into the monkey-house at the Zoological Gardens, and the excitement thus caused was one of the most curious spectacles which I ever beheld.

If you want to learn about evolutionary biology, read Steven Jay Gould's essays. Darwin's a good writer and he got it basically right, but he didn't know about genes or DNA. I read Darwin to experience the origins of the field. I didn't expect (though perhaps I should have) to encounter endless artifacts of the days of two-fisted Baconian science.

When Charles Darwin needs to figure something out, he carries out an experiment, no matter how tedious or disturbing. He takes snakes to the zoo, he puts kittens' feet in his mouth, he floats 94 kinds of plants in seawater, he hacks aphids. If someone has the temerity to question his conclusions he's all "Citation needed? I'll give you citations, motherfucker!"

When you read a book, it has an effect on your mind. You're a slightly different person after reading it. You've created something new: a reading of the book: Here's an apropos quote from Alberto Manguel's A History Of Reading, which I read on paper and typed in. Manguel is talking about Petrarch's Secretum:

What Augustine (in Petrarch's imagining) suggests is a new manner of reading: neither using the book as a prop for thought, nor trusting it as one would trust the authority of a sage, but taking from it an idea, a phrase, an image, linking it to another culled from a distant text preserved in memory, tying the whole together with reflections of one's own -- producing, in fact, a new text authored by the reader.

Readings are ephemeral. Life goes on, and the memory fades. Ken Macleod's The Star Fraction had a huge influence on me, probably leading to whatever career I now enjoy as an author of fiction, but I read it ten years and 600 books ago, and now I don't remember a damn thing about it.

That's why we dog-ear pages and highlight passages. We're instantiating our reading of the book so we can go back later and approximate the mental state it gave us without re-reading the whole thing. Even if all we got out of a book was "this bit was funny", it's better to have the funny bit at hand than not. Even if you never go back to the highlighted passage, the act of highlighting replays that passage and deepens your initial memory of it.

Liberate your readings

I've been typing in quotes from the paper books I read, like I did with the Manguel. Of course, with an electronic book, you don't have to do this. The act of highlighting creates an electronic record of your reading of the book. When I was in college I read about the first e-ink research coming out of MIT, and I knew that this was the future. Indeed it was the future, because I had to wait ten years for the technology to make it to market. But, sour grapes, we've got ebook readers now.

Ebook readers have big problems, but at this point the problems are mostly political, not technical. For instance, you can highlight passages when reading a book on your Kindle, but because of a deal between Amazon and the publisher, your book's metadata may include restrictions, which the Kindle will obey, on how much you can highlight. And your highlights and notes—the "new text authored by the reader"—are stuck on a website that Amazon didn't put a lot of work into because they don't consider your reading of a book important to their business.

Findings takes advantage of the fact that Amazon is wrong about this. Findings liberates your highlights and makes them searchable and shareable. Your reading of a book is a big part of your relationship with that book, and Findings gives you access to it.

You can also use Findings to take a reading of a web page, creating a record of what would otherwise be an ephemeral activity. I'm not as interested in this feature, but people are using it quite a bit, and my interest does increase as the length of the web page I'm reading approaches the length of a book.

Browse readings

So that's what Findings can do for you personally. Now let me pitch you the network effects. Take a look at this screenshot which shows the Findings global stream:

You can't see the global stream without logging in, which I think is a shame because I think this is what really sells Findings. We have here a stream of little bits of text, like Twitter used to have on their front page. Except here, every bit of text is a quote that someone liked well enough to save. It's very high-quality stuff. At the top you can also see some recently added books, and by clicking on a book you can see someone's condensed reading of the book.

Basically, Findings gives you browsing access to a large library, not of books, but of readings. It's easy to discover new books, people who read books you like, and—this is new—people who read books in ways you like.

There are a ton more useful things I could mention, but they're mostly behind-the-scenes things where Findings makes things "just work" (like consolidating multiple editions of the same text), or they depend on features that haven't been implemented yet. So I'm going to close by mentioning the social signalling feature.

Strut your stuff

One underappreciated feature of paper books is signaling to other people that you are cool. You read books! Fancy books, like Ulysses! You care enough about books to make space for them in your house. You take them on the subway even though they're heavy. Darwin would say it's like the peacock's tail. But if you have an ebook reader, nobody knows how cool you are. You're just a person with an ebook reader.

By letting you publicize your reading of an ebook, Findings reinstates your ability to send those social signals. The downside is that you have to actually read the book. You can't just put a big book on your coffee table: the thing you're sharing is what you got out of the book. (Well, you can fake it, but it's probably about as much work as reading the book legitimately.)

So that's Findings. I don't use it as much as I thought I would, because I'm still trying to draw down my stack of paper books, but when I read a book on my Kindle, it stays read, thanks to Findings.

I mentioned this before, but the last thing I did at Findings was design a web service for them, which they're hopefully working on now. Once the web service launches, you'll be able to write programs that import readings into Findings from non-Kindle sources.

Do it yourself

One final note: If you have a Kindle, connect it to your computer and look on its filesystem. All your highlights are kept in a structured-text file located at documents/My Clippings.txt. This file includes highlights taken from PDF files and other ebooks not recognized by Amazon, which don't get synced to kindle.amazon.com. Even if you don't use Findings, take control of your highlights by backing up this file.

Image credits: McKay Savage, Romana Klee, and André Fincato.

Thu, 12 Jan 2012 13:17:39 GMT

My sister talks about her miscarriage.

The only thing worth saying is "I'm sorry." I may think those things. John and I may even say those things to each other. But don't impose beliefs or possibilities or happy thoughts on me.

Hostile Witness: The Vechs Interview

Tue, 10 Jan 2012 15:42:33 GMT

Minecraft can be a tough game. The controls are kind of blocky, the best resources are hidden deep in the map alongside deadly lava rivers, and the night hosts monsters that will kill you just as soon as look at you (or, in one case, just as soon as you look at them). But it's not that tough. All that terrain is generated by algorithm. It's not like the random number generator is trying to kill you.

But there's this guy named Vechs who is trying to kill you. His "Super Hostile" series of custom Minecraft maps offer challenges that prohibit or subvert every survival strategy you learned in vanilla Minecraft. Even in his easier maps you'll find bottomless pits, world-spanning ceilings that block Minecraft's all-important sunlight, swarms of monsters pouring from hacked spawners, and TNT in unfortunate places. Just getting your first tree is a challenge. Complete a Super Hostile map, and vanilla Minecraft will seem easy.

But Vechs' maps are not just tough: they're creative, fun to play, and they look great. Vechs uses landscape features and lighting to grab the player's attention, direct the flow and pacing of their playthrough, and give them a spectacular environment to build in once they've conquered the map.

I've raved about Super Hostile a couple times before here on NYCB, but with the release of "Spellbound Caves", the tenth entry in the series, I knew it was time to get serious. I sat down with Vechs (I assume he was sitting down, anyway) and interviewed him over minecraftforum.net's private message feature. My goal was to pick up where this interview from July 2011 left off, with in-depth questions about his style and his mapmaking wish-list. The interview contains some Minecraft jargon, but anyone with an interest in game design should get something out of it.

Leonard: You play a kind of character on your maps, an angry trickster god who hates his players and taunts them by writing things on signs. But clearly you're not actually like that. I've played maps made by people who really did hate me, who wanted me to farm cobblestone for an hour or dig through obsidian without a tool, and I said "screw this" and quit the map. I don't think you'd ever do that, right? What's the difference between you and the "Vechs" in your maps?

Vechs: It has to do with challenge. There is a difference between making the player use skill or ingenuity, and making the player do something tedious. Sometimes a solution to an area can involve using lots of blocks (Like the player making a cobblestone tube for them to safely move through.), but these are usually just one option of many the player can use to conquer an area.
Sometimes the "Vechs character" in my maps is pretty mean, and just downright spiteful, especially when it comes to traps. In real life, I'm not like that at all.

In the Obsidian Block interview you say that you recently graduated from college and are looking for a career as a game designer or world designer. What did you study in college?

I am a Media Arts major. I studied everything from digital image editing, video editing, to stage lighting, to writing scripts and screenplays for movies, and more. I'm glad to have a diverse background, even though my passion is still game design.

What would be your ideal job? Would you rather work on a big-budget project with high production values, or an indie project where you have more creative control?

My first choice would actually be to have my own studio and bring to life some of the game ideas I have. One idea I've had for a while, and as far as I know, nobody has ever made a game like it. I wouldn't mind making it all myself, but that means I would have to re-learn a lot of programming. I've programmed some text-based games in C++, but programming is not my main forte.
That said, I also wouldn't mind working for a major company. Like, for example, Valve. Love those guys.

What other games have you made maps for? You mention Duke Nukem 3D in the Obsidian Block interview; what else?

Just off hand: Red Alert, Warcraft II, Warcraft III, Neverwinter Nights, Total Annihilation, TA:Spring, Terraria, Command and Conquer (and several sequels), The Elder Scrolls series, and obviously I'm the world designer for the RPG games I've worked on, using the XP and VX engines.

Are you currently making maps for any games other than Minecraft?

At this moment, no, but I have been meaning to make some maps for Team Fortress 2.

Have you ever heard of ZZT or Megazeux, or am I just incredibly old?

You're old! *grins* I looked them up, and I think my version of that would be the RPG-series of game engines.

There are a lot of memorable set pieces in the Super Hostile series. Now that you've put out ten maps, would you mind taking a look back and sharing some of your favorites?

The first 15 minutes on just about any of my maps. I love that feeling of just starting off and scrambling for resources. I like the rail station in "Sea of Flame II", and how it goes out in the area with the huge pillars, and "Spellbound Caves" is just full of nice vistas and "scripted" events.
Most of my maps feature at least one "death fortress" as an end-game area. These are intended to be where the player gets to use all the resources and items he has been collecting through the whole map. TNT, lava, swords, bow and arrow, even TNT cannons... bring your whole arsenal and have some fun!

Can you describe the evolution of your design philosophy over the course of the series?

Try to improve in at least one area every time I make a new map. Push the Minecraft engine to its limits. Make an awesome and memorable experience for the player.

What are the biggest challenges in re-balancing Super Hostile for Minecraft 1.0?

Armor and blocking.

Does 1.0 have anything to do with the fact that you recently flattened the difficulty levels in your map descriptions, so that "Sunburn Islands" and "Legendary" are now both considered "Easy"?

Yes and no. I feel that recently I have been drifting away from the theme of "Super Hostile" and I want to get back to my roots. Being able to respawn forever, over and over kind of takes the risk out of a map. Even in "Legendary", unless you really mess up and drop all the wool in lava or something, you can just set your bed spawn near an area, and try over and over until you get it right. I think that's pretty Easy on the player, even if the area you are attempting is challenging.
Call me nostalgic, but I kind of miss (sometimes) the GAME OVER screens from older video games. Modern video games, in the name of convenience, typically feature unlimited lives, save games, checkpoints, the works. But beating a modern video game, I have to admit, is much less satisfying than beating some of those old NES games. You can just bang your head against the game until you get lucky and get through an area. Heh, man this makes me feel old! "In my day, we didn't have all those checkpoints! We had three lives! One hit deaths! And we were happy!" *shakes cane*
Anyway, I do think this is a legitimate point of concern on modern game design, is risk versus reward. It is possible to make games so easy that they are very unsatisfying...

I'm an admirer of your ability to create new genres of map. Have you made experimental Minecraft maps that just didn't work? What's in your "abandoned projects" folder?

The only thing I've actually stopped on, is "Race for Wool #3: Common Ground". Because it basically became "Capture the Wool".

Have you ever made maps for a game that featured scriptable events? If so, do you miss that capability in Minecraft?

I have used C++ to code some text-based games. I have also used various scripting languages in the process of making mods or making my own games with existing engines. You do have some limited "scripting" ability in Minecraft, using redstone. Check out the Rumbling Caverns in my tenth map and you will see what I mean. :)
But yes, I would love some even rudimentary scripting in Minecraft. I believe a while ago, I proposed invisible effector blocks, that you can place with Creative or MCedit, that modify the immediate environment around them. Like, an invisible block that makes monsters not spawn within 50 blocks. Or one that doubles monster spawning within 50 blocks. Or one that makes it snow. Or one that makes a ray of sunlight always be shining on that spot. Or one that makes the temperature freezing so any water turns to ice. Simple stuff like that. They would show up faintly in Creative mode, but be invisible while in survival mode.

What would you like to see added to Minecraft? On your forum thread you mention that you'd like to add sharks and underwater plants to "Endless Deep". What else?

Bow enchantments... more mining enchantments, such as area mining. Check out episodes 04 and 05 of my Spellbound Caves Dev Commentary.
For bow enchantments, I would like:

Piercing (Arrows go through monsters and cause damage in a straight line.)
Toxic (Arrows cause poison.)
Fire (Arrows ignite enemies.)
Knockback (Arrows cause knockback.)
Phantom Spreadshot (In addition to your one normal arrow, you fire out additional ghost arrows (higher ranks provide more) that act like normal arrows, only you cannot pick them up, and they quickly vanish after hitting the ground.)
Explosive (Arrows cannot be reclaimed and cause a small explosion, could possibly be combined with the fire enchant, so arrows function like ghast projectiles, causing an explosion and leaving fire.)
Charge Speed (You charge your bow faster.)
Unbreaking (Your bow lasts longer.)

I think these enchants for bows would make bow combat much, much more fun. It's currently fairly slow paced, and a bit boring. Imagine a bow with Toxic, Piercing, and Phantom Spreadshot on it! It would be so much fun to shoot groups of enemies with a bow like that.

You have a creative relationship with some of the people who do Let's Play videos of your maps. It's a kind of relationship I've never seen before: the way people play your maps in public affects the way you design later maps. How did these relationships develop?

Very organically. Zisteau agreed to LP my very first map, "Sea of Flames" version 1.0, and ever since then, he's been involved in playing my maps, and giving feedback.

There's a very clever trap in "Spellbound Caves", [location redacted]. It's clever for many reasons, but I'm asking about it because it doesn't seem to have any triggering mechanism. I went in afterwards and took the walls apart and couldn't figure out how it works. What's the secret? Or is there a pressure plate somewhere that I missed?

I has a seekret. Oh, also, I hate you, die in a fire.

POSTSCRIPT: With my interviewee uncooperative, I had no choice but to load a fresh version of "Spellbound Caves" into an editor to get to the bottom of the mystery. What I found was a trigger that did not shock me to the core of my being. But it is a cool design.

The trigger is a proximity sensor: a shaft behind a wall, with a creeper spawner at the top of the shaft and a pressure plate at the bottom. When the player gets within 16 blocks of the spawner, it activates and spawns a creeper, which drops onto the pressure plate, triggering the trap. The resulting explosion obliterates both creeper and spawner, leaving no trace of the trigger.

And that's what you get with Vechs' maps: MacGyver-like use of everything the game engine provides, to create confounding and unexpected effects. Seriously, game studios: hire this guy. Everyone else: play his maps.

PPS: Hey, people from minecraftforums.net, thanks for coming over. I've written other articles about Minecraft (1 2 3 4), and if you like my stuff, you might want to check out my novel about alien video games.

CG Author Commentary #7: "Party Creation"

Tue, 10 Jan 2012 12:43:39 GMT

Ariel's Twitter will be pretty quiet this week because the entire chapter takes place over the course of one day. Your only solace is farmers market quail sausage, and this COMMENTARY:

In this chapter we meet Bai's girlfriend, Dana Light. Sort of. Dana is the bane of this minimal-spoiler commentary because it's a bit of a spoiler even to treat her as an important character instead of a bit of character development for Bai. But she is important--Bai was defined in terms of Dana, not the other way around, and I found Dana interesting enough to make her the star of the first bonus story, "Dana no Chousen."
Chapter 10 will be a very Dana-heavy chapter. If I hadn't been writing a novel when I came up with the character, that chapter would have become a standalone short story. Of course, once I had the character, Dana quickly became important to the story (spoiler!!) and after selling the book I moved bits from Chapter 10 backwards into this chapter so as to introduce her earlier.
More on Dana and Dana/Bai in a few weeks when I commentate chapter 10, but for now you might enjoy going back to chapters 3 and 4 and re-reading the earlier references to her.
Bai is pretty ditzy when it comes to women, because that was the only way I could get him into this relationship without making him stupid or a Love Plus-style otaku. If I was a better writer I might have come up with a better solution, but I'm at the level I'm at. However, there are a couple moments later on when Bai is surprisingly perceptive about Ariel's love life.
The other introduction in this chapter is Sayable Spice, probably the game most important to the plot. In the second draft, Ariel talked about Sayable Spice a whole lot, but it wasn't until chapter 14 that he mentioned that he was working on a remake. Now Jenny forces Ariel to commit and puts a stop to that nonsense right quick. Thanks, Jenny.
When I got rid of Bruce, the character you'll never see, I gave most of his scenes to Bai. (Eg. it was originally Bruce playing Knockdown Dragout in chapter 4.) I'm pretty sure the scene in the backyard here is the only Bruce scene I gave to Jenny. Nobody cares!
This is also the only chapter in which Bizarro Kate has a speaking part. In the second and third drafts, she hooked up with Bruce at this chapter's cookout, and later in the novel they were living together. Hers was a supporting role to begin with, and when I eliminated her major-character boyfriend, she got cut down to just a walk-on. But I really like the character, and I gave her a chance to shine (ie. be cranky) in "Found Objects", the Jenny story.
I think Tennis For Two is the only real-life video game I mention in the novel. There was no other way to get the joke to work.
The tattoo joke I had to cut from chapter 19 has landed on Twitter. Relive the saga: 1 2 3 4. I don't think I needed to actually present the translation. Oh well.
Around this time I planned on changing Ariel's Twitter profile image to the duck from the Crispy Duck Games logo, but it didn't look good, so I think I'm/he's going to stick with the pissed-off pony.
The only other thing I want to say about this chapter is you see a little bit more of how old ET video games just aren't as interesting to other people as they are to Ariel.

Stay tuned for chapter 8, a chapter I think is one of the best in the book, the chapter that got me to give up on the first draft and rewrite the entire book to be more like it. The only chapter in which Jenny will say, "Wait a minute, are you naked?"

Image credits: I got the first image from Flickr user marsmet462, not sure if they put in enough transformative elbow grease to put their own license on it. Second image comes from Sven-S. "☃" Porst .

<- Last week | Next week ->