News You Can Bruise for 2013 December 4

Permalink [Comments] (2) Wed Dec 04 2013 14:55 Secrets of (peoples' responses to) @horse_ebooks—revealed!:

As part of my @pony_strategies project (see previous post), I grabbed the 3200 most recent @horse_ebooks tweets via the Twitter API, and ran them through some simple analysis scripts to figure out how they were made and which linguistic features separated the popular ones from the unpopular.

This let me prove one of my hypotheses about the secret to _ebooks style comedy gold. I also disproved one of my hypotheses re: comedy gold, and came up with an improved hypotheses that works much better. Using these as heuristics I was able to make @pony_strategies come up with more of what humans consider the good stuff.

Timing

The timing of @horse_ebooks posts formed a normal distribution with mean of 3 hours and a standard deviation of 1 hour. Looking at ads alone, the situation was similar: a normal distribution with mean of 15 hours and standard deviation of 2 hours. This is pretty impressive consistency since Jacob Bakkila says he was posting @horse_ebooks tweets by hand. (No wonder he wanted to stop it!)

My setup is much different: I wrote a cheap scheduler that approximates a normal distribution and runs every fifteen minutes to see if it's time to post something.

Beyond this point, my analysis excludes the ads and focuses exclusively on the quotes. Nobody actually liked the ads.

Length

The median length of a @horse_ebooks quote is 50 characters. Quotes shorter than the median were significantly more popular, but very long quotes were also more popular than quotes in the middle of the distribution.

Capitalization

I think that title case quotes (e.g. "Demand Furniture") are funnier than others. Does the public agree? For each quote, I checked whether the last word of the quote was capitalized.

43% of @horse_ebooks quotes end with a capitalized word. The median number of retweets for those quotes was 310, versus 235 for quotes with an uncapitalized last word. The public agrees with me. Title-case tweets are a little less common, but significantly more popular.

The punchword

Since the last word of a joke is the most important, I decided to take a more detailed look each quote's last word. My favorite @horse_ebooks tweets are the ones that cut off in the middle of a sentence, so I anticipated that I would see a lot of quotes that ended with boring words like "the".

I applied part-of-speech tagging to the last word of each quote and grouped them together. Nouns were the most common by far, followed by verb of various kinds, determiners ("the", "this", "neither"), adjectives and adverbs.

I then sorted the list of parts of speech by the median number of retweets a @horse_ebooks quote got if it ended with that part of speech. Nouns and verbs were not only the most common, they were the most popular. (Median retweets for any kind of noun was over 300; verbs ranged from 191 retweets to 295, depending on the tense of the verb.) Adjectives underperformed relative to their frequency, except for comparative adjectives like "more", which overperformed.

I was right in thinking that quotes ending with a determiner or other boring word were very common, but they were also incredibly unpopular. The most popular among these were quotes that repeated gibberish over and over, e.g. "ORONGLY DGAGREE DISAGREE NO G G NO G G G G G G NO G G NEIEHER AGREE NOR DGAGREE O O O no O O no O O no O O no neither neither neither". A quote like "of events get you the" did very poorly. (By late-era @horse_ebooks standards, anyway.)

It's funny when you interrupt a noun

I pondered the mystery of the unpopular quotes and came up with a new hypothesis. People don't like interrupted sentences per se; they like interrupted noun phrases. Specifically, they like it when a noun phrase is truncated to a normal noun. Here are a few @horse_ebooks quotes that were extremely popular:

Clearly "computer", "science", "house", "and "meal" were originally modifying some other noun, but when the sentence was truncated they became standalone nouns. Therefore, humor.

How can I test my hypothesis without access to the original texts from which @horse_ebooks takes its quotes? I don't have any automatic way to distinguish a truncated noun phrase from an ordinary noun. But I can see how many of the @horse_ebooks quotes end with a complete noun phrase. Then I can compare how well a quote does if it ends with a noun phrase, versus a noun that's not part of a noun phrase.

About 4.5% of the total @horse_ebooks quotes end in complete noun phrases. This is comparable to what I saw in the data I generated for @pony_strategies. I compared the popularity of quotes that ended in complete noun phrases, versus quotes that ended in standalone nouns.

Quote ends in Median number of retweets
Standalone noun 330
Noun phrase 260
Other 216

So a standalone noun does better than a noun phrase, which does better than a non-noun. This confirms my hypothesis that truncating a noun phrase makes a quote funnier when the truncated phrase is also a noun. But a quote that ends in a complete noun phrase will still be more popular than one that ends with anything other than a noun.

Conclusion

At the time I did this research, I had about 2.5 million potential quotes taken from the Project Gutenberg DVD. I was looking for ways to rank these quotes and whittle them down to, say, the top ten percent. I used the techniques that I mentioned in my previous post for this, but I also used quote length, capitalization, and punchword part-of-speech to rank the quotes. I also looked for quotes that ended in complete noun phrases, and if truncating the noun phrase left me with a noun, most of the time I would go ahead and truncate the phrase. (For variety's sake, I didn't do this all the time.)

This stuff is currently not in olipy; I ran my filters and raters on the much smaller dataset I'd acquired from the DVD. There's no reason why these things couldn't go into olipy as part of the ebooks.py module, but it's going to be a while. I shouldn't be making bots at all; I have to finish Situation Normal.

dada

Comments:

Posted by Franz Poekler at Thu Dec 05 2013 07:30

That's very interesting about interrupting noun phrases, I was noticing just yesterday while updating a ranking of the humorousness of words prior to integrating it with wordnet data that nouns are consistently funnier than other types of words.

My system doesn't use external feedback, it's just a kludge of various guesses involving scrabble scores, word length, frequency of appearance in this corpus or that one, etc.

You get surprisingly far just by looking for words that are spelled funny, because they're usually loanwords, which means they're usually words for things that are novel and/or specific, both of which make things funny. Bassoon, zucchini, umbrella, &c.

I have yet to rescore the words based on the average humor value of wordnet's various "bins", such as body parts, weather, abstractions, etc. but here's what I currently have for the top 500 funniest words:

Gorilla gorilla gorillaraisinreeferrollerpinkiebananazombiestereosensorwafflephlegmGorilla gorillaweaselpill rollerMormoncone peppercowboy bootminionmidgetvacuum pumplemon sharkcowboycherry peppermascotpepperoni pizzapepperpretzelkettlelemmingquirkrainbowtrolldryercereal oatpencil eraserbuddy-buddyomegapandahippoblotterpancakepencilservermedianfreezersleighoyster mushroomMoronhaircutvinegarhamburger bunraviolirazor clamgorillacarrotpopcornHessian bootpillowballoonpicklecannonfeedback loopcrampwedgeavatarspice rackgum booteyebrow pencilchillstirrup pumpacronymatlaswrenchcerealbell pepperworker beedioramabroccolishackmustardpizzasleigh bellpin wrenchbreezepumaalphadinosaurpeppermint gumnettlesharkexcerptoysterlamb currysnowballvacuumhiccuphazardthreatbeefsteak begoniafurnacetheoremloan sharkriddletwistbannermoccasinfedorapuffbedtimeMormonsstickerbouillon cuberazorbroomtiger sharkmazesyndromepepper spraypin cherrypillow lavabarbecuecocoamelodypartridge peabetel peppertank furnacecocoa beandeltahailstonemargaritacroissantcrosswordspaceshipvacuum cleanerstadiumhamburgermemogrammarpepper millhyacinth beanchalkpin oakbuttermilk pancakerampcobwebpillsheet musicprankMormon Tabernaclekarmahobobrush kangarooblackjack oakcannon crackerjodhpur bootmailmanabyssal zonetack hammertwaddlemedia gurufunkfalloutumbrella magnoliasheetmusic rackhyacinthgrocery storeboobbumpyardstickpseudonymsheet pilealgorithmoyster crackermechanicarpeggiocrowmustacheayatollahhallmarksockjunk pilemuckfudgecrescendobeefsteak morelmallmathhedgehogbluebelllemonadeweekend warriorconedisco biscuitstilettotiger rattlesnakeleverfurbelowstamp millmannequindeadbeatpepper rootpumpmandalajunkshortsplaintiffbootmathsrackleechlemonmosquito netpea beanbillowscarecrowteenbuttercuppepper saucebelt buckleflashbulbbio labtrademarkkangarooreelbuddyspark leverLabrador Peninsulamackerel sharkaffidavithoneycombstampnightclubinsomniacknickknackslackdiaphragmnonsensebunnynotebookmosquitoteensnerdsaxophonesnackmustard saucetarantulacorkscrewdrumstickboardwalkriffraffdittowaistcoathighchairnincompoopumbrellaeconomylollipopmausoleumbureaucratwhirlpoolpine hyacinthpepperoniasparagus peasweetheartcappuccinoputtyfootball stadiumslacksrapturemiasmagangplankbarbecue sauceasteriskpocketbookmillenniumleprechaunfootageutensilgirlfriendguerrillasidekicktoadstoollogarithmbeefsteaklimboquesadillazucchinicafeteriaambassadordyefudge sauceluggage rackdomeenthusiasmburnerwithdrawal symptomsweat socklooparchbishopumbrella pinegymnasiumpeninsulalaundry detergenthydrangeapilot balloonfridgeteeter-totterbirthmarkfontwinkmultivitamin pilldisco musickidney begoniabedfellowbasketballlemonade mixwindowpane oysterpoinsettiatoothpickbumblebeechow chowkangaroo mousemyelomagreenhousekerosene heaterbathroom cleanercheesecakechestnut oakoak chestnutbarbershopleewayespressomaul oakHoover Damcherryscratch sheetsagebrush buttercupJuggernautmacadamiacinnamon bunpizza parlortobacco mildewcube rootwindowsillparodywavelengthgestalttableclothquestHooverrattlesnake rootshrimp sauceearthquaketattletalebell morelcandelabrawheelchaircurry saucelemon groveitalicshenaniganRamadanfellowshipportmanteaumushroom poisoningbasket oakhorseshoemechanicstroubadourpeppermintloan-blendbuttermilk biscuitstackworkerarrhythmiacrochet stitchpilot burnerassassingoonotchpin curlgutpostmortemphenomenonbackupemployeeshrimpnanosecondpreviewwheelbarrowion pumpcannonballgrapefruitBaptistchinchillahummingbirdstacksmushroom saucechukka bootdesktopgraffitiwaxworkpterodactylmaniacurrybuggymedicoluggage carrierballroom musicgroceryfeedbackgutsaficionadopinchotitis mediahaddockwandGorilla gorilla beringeiGorilla gorilla grauricorncobpixelcrackerjacklorydotMohammedan calendaroatalpha and omegatwilight zoneleekkidney beaniceberg lettucedungareeYahoocashewiceberghieroglyphmallardpuff pastespark coilcephalopod molluskbookmarkupkeepmicrowave ovencottonmouth moccasingeckogenomecubehessiancakewalknaparistocratpinwheeltagautumn pumpkinlogbookflamencobridesmaidbeefireworkpurl stitchllamalaundry basketcaterpillarasparagus beaneraserflowerpotlemon mintrattlesnakeoakteatyokelopossum shrimprepresentative samplestamp albumzigzag goldenrodstorage tankmarshmallowpeaGalilean satellitesump pumplabpalookapicketbunnihilismglockenspielomelette panstopgaprimepitaphneutrinobylawextravaganzaspicesatelliteoptometristcuedrumsneak previewbunslowbrowpistachiostrawberry daiquirivanilla beangumflakemushroomhammerhead sharkmolehillbasiliskthrow pillowbeltpan-fryclampointerplywoodspaghetti saucesidewalkvestloop-the-loopscrew wrenchhousetopvolleyball netderrierelambloop gainpilot biscuitmittenbubble gumoreellipsispockmarkmelanomabedroom furnitureFrisbeeneighborhoodrefrigeratorpineconeegorobot pilotcarriersausage balloonfrillemployee ownershiproguestirruptonic epilepsyclamsmatzofolk musicvaquerorhododendronflattopsausage pizzajaguarpushupwindmillabyssalmousetrapdoorsteptradeoffhorseshoestear sheetmessengerbreakthroughstem lettuceschnookrack of lambfriezemassage parlorheatermantillathresholdramtangentgarbanzoeyesorevesselkettle holeoutpostcrow pheasanttackspreadsheethangoutcorpus delicticrochet needlesandbar sharklesser pandanicotine poisoningbandannanubbinbouillonboomantennapoppereggnogfattyoyster shellburgerdudgeonwrinkleescape cockaerosolcruisewontonnovellatchlaptopchrysanthemumbedrockhandicapwhisk broommenschhypochondriacwashtubcockravenfleckserifgimcracksheet benddoorknobArgonautpinprickCalypsotabletopaftermathkinkajoulaughingstockspin dryerdartsalvoduelcicadahoosegowwoolflashbackfunk holedogwoodCremonapancake batterpine grosbeakyearbooktomahawkbasepepper-and-saltoodlespartridgeclick-clacklunchroombilberrybanana splitamphibianraspberryscrub oaktilt anglesubdiscoclipreservoirraisin cookietextbookbatchhamperSavannahbracerobroom-weeddesperadoalawarriorOhmobesityscenarioBaptistswoodchuckzigzagkippernorthern pin oakwildebeestbonushallooheartburnbegoniagoddessbean sproutpunditmarijuana cigarettegazpachojeansdartsknockoffepidemic roseolamacaronilemon extracthammerkibbutzmustard seedpuff batterbaseballsnowstormrose leekseed oysterracetrackmuskrateyebrowtoyslicestapleworkshopaviatrixloofahbaseball mittjerkinthunderstormbathroomfilbertrigatonitonicHalloweenmouse nestcentavozonerogue elephantenterprise zonetitbitfurloughlaundrydaiquiritiercerainbow smeltlever scalerailroad engineertarragondewlaponlookerzwiebacksessioneyetoothsuburbbandwidthnarratorscreen fontgoodnessscallywagducatmouse clickNorsetreadmillkilowattcutoutbarleyschmaltzcybersexrose gumkinklifetimewicker basketweekendgospeltankardpoppetdoughnutcluster beanbanana peelmochavendettamaharajaSelenevirtuosorack railwaypeekcluster headachefillerbarracudablowuphaw-hawmicrofilmcarawayprairie rattlesnakelasersamplebiscuitclam dipaftershockcookie sheetrestaurant attendantattorneysymposiumbookshelflabyrinthhollyhockinstinctpinlamb choplamb-chopgrommetwisecrackmassagepaprika saucemildewhedgerowJapanese bananacalicoyeshivaheartbeatphotostathootmelodramagooseneck barnacleJapanese umbrella pinejejunumpine mouseheirloomclassroomcutoffethernetanacondahammer throwcoilpillow slipgnocchifairy shrimpnail-tailed kangaroopineapplecatnapwhetstonehellcatrobotpitchforkmaraboukeepsaketaeniahiccup nutbellhamburger rollclaptrapflapjackblow dryercheckbookattendanturban guerrillaabomasumpush broomhurricane lanternhashfaceliftgutterfan beltstencilink eraserdemand loanfeminismshiatsucobbleprotocolzygotedredgebiconvexneocortexfallensnap peapesteralter egocataclysmcolaballerinatourniquetimamcancelephemeraduffeldoodadLabrador-Ungava Peninsuladeodorantarmchairhaberdashery storeavalancheglowwormsheepherderQuechuablackjacktrendgarbage pickupAshkenazisabbatsauce vinaigrettemesodermairplane mechanicsJapanese oysterGospelsschemeatlas vertebracataracthandoffwaterfallhindsightwaylayleitmotifpickle relishrattlesnake weedlockboxsteppepillow talktouch basewiggle nailcesspoolseed shrimpvanishestoppelsunspothelperjackknife clamgolliwogstitchopponentcutthroatpumpkinsalt lickcatchphrasestilthorseradish rootballyhoopummelmainsailanagramcrosswalkcrossbowmarkerpillow fightenchiladabaklavacorneamaharajahhobgoblinlever lockdiazepamclausepitch pinesymptomgoalpostmezuzahcookbookpoplinexercisemaplecockatoomestizoneoprenechipmunkstemurticariacalendardoohickeydecampOmega Centaurigalootriffmolluskhemispherecottonwoodlemon peelleotardteeterswivel pinreddenbordellokumquatsolar furnacetoothachewithdrawalgliomasalaambanana quitscrub palmettoclenchlifespancuckoo-bumblebeepeanutjonquillardmishmashrebootberiberiocarinaTeapot Domeageratumtottermetatarsaloligarchwindow oysterkangaroo applepupabloodbathspaghettichalk talktesserabaconCanal Zonecapuchin

Posted by Leonard at Sat Dec 14 2013 08:34

For posterity's sake, I'd like to record that there was a spam comment posted here whose text was "Wizard Who Was Scared Of men."

Leave a comment