<D <M <Y
Y> M> D>

[Comments] (4) Thoughtcrime Experiments Lab Report #2: What The Slush Pile Looks Like: There are three rails, you say? Let's take a look at this one!

As a writer, I used to feel like I didn't have a good picture of the short genre fiction market as a whole. You send stories in, they come back. You can get together with other writers in a writing group and commiserate, but there's no way that's a representative sample. One reason I started this anthology was to take a good look at a real slush pile and get some sense of what it really takes to have a story worth publishing.

This is a taboo subject, judging from how little editors talk about it. I've been told superficially encouraging things like (not a direct quote) "If you can write grammatical English and tell a story, you're above average." Okay, but that's just saying that baseline competence is relatively rare. How big is "above average" and what does it look like? Magazines don't publish above-average stories, they publish the best. How many people are the best?

It's a truism that the slush pile is huge, that there's a vast oversupply of terrible science fiction. I started Thoughtcrime Experiments partly to test the hypothesis that this oversupply masks a similar oversupply of high-quality science fiction, endlessly circulating from editor to author to editor like some action-packed Sargasso. With no prior fiction-editing experience, can I take a core sample of the slush pile and grab a bunch of good stories that no one else wanted to publish? All for rates that, while decent by industry standards, are penurous compared to the effort it takes to write a story? Is the problem with the slush pile that too much of it is bad, or that too much of it is good?

There were 241 total submissions to Thoughtcrime Experiments. A couple more might come in, from people of whom we asked another story, but they won't affect this general analysis. There were so many stories we've forgotten about most of them by this point, but we have recorded information about our opinions at the time, which I put into a database. My goal for this entry is to try to draw some conclusions about stories in the aggregate.

Here's our review process. A story comes in. Sumana reads it and sticks it into one of five tiers, A through E. A story goes into one tier or another based on how well it satisfies the following fitness function: Would we regret passing up the opportunity to publish this story? From A to E the tiers are "absolutely not", "no", "eh", "yes", and "yes!" In retrospect, we probably only needed three tiers ("no", "yes", "yes!"), but having five does make the graphs you're about to see more interesting. And at the beginning we didn't know we would get any "yes!" stories. We thought we might have to publish "yes" stories with some "eh" for filler.

In terms of the world-famous Context of Rejection, I'd say tier A corresponds to 1 through 5, tier B is 6 through 8, tier C is 9 through 11, tier D is 12 and 13, tier E is 14. There's some overlap and it's not a perfect match, since the CoR is for novels, but you get the picture. The inflection point is between 11 and 12, the difference between "I could see someone publishing this" and "We should publish this".

Getting back to the process: Sumana flags certain stories in tier C for my attention, and I also read all D and E stories. Sumana and I have many differences of opinion but we've never differed by more than one tier. We talk about the stories. The fruits of our discussion get turned into feedback for the authors. Sumana makes occasional passes over the pending stories and sends out rejection letters.

Currently we've sent out rejection notices for tiers A through C, and for most of D. We've bought one story and we're left with about 25 that we really want to publish. Now is a good time to mention that we're going to publish more stories than the originally-planned five, but not too many more. Certainly not 25. So we need to cull that herd. We're trying to decide on a good mix of stories, which might mean sacrificing an E story for a D--and we're not going to be able to publish all the E stories anyway, as you'll see if you look at the graph below.

The fitness function works well for focusing quickly on the relatively small number of stories that really stand out to us. But it's not so good for scientific purposes because it conflates a lot of variables: technical skill, inventiveness of plot, vividness of character, adjacence to our personal tastes in spec-fic, etc.

For instance, all the terribly-written stories ended up in tier A, but so did most of the horror stories we got, regardless of quality. I said in the submission guidelines that I was open to horror if it was clever, but it turns out I wouldn't know clever horror if it rose from the grave and bit me in the ass with decaying teeth held together only by metal-amalgam fillings. Some especially well-written horror got into tier B, and one really funny zombie story clawed its way into tier C. Long story short, there'll be no horror (the genre, not the emotion) in TE. It turns out that's not what we want to publish.

Another example: we got an extraordinarily well-written story that had no fantastic element in it at all. It went into tier D, not because we wanted to publish it, but because it seemed like a wasted opportunity not to publish something by this person. We made inquiries and got a less mainstream flash story from this person, which we're considering offering to buy for half price. So really, that should have been a tier A story, but we used tier D as a shorthand for "do something about this."

So here it is, the graph you've all been waiting for. How many stories went into each tier? What does the slush pile look like?

This is going to get really unscientific, because what we start out with is our subjective opinions of the stories. But I think there's some interesting stuff in this graph.

This has more than a passing resemblance to a normal curve. Of course, if it were a normal curve, there's a missing tier to the left. It would look more like a normal curve if we split tier A into two tiers, one for terribly-written stories and one for stories that were just extraordinarily incongruent with our tastes. I'm glad we didn't do this, because the correct response in both cases is prompt rejection, but it's an interesting observation.

It's a category error to try to do statistics on the tiers, but what the hell. Let's assign a tier A story 0 points and a tier E story 4 points. Then the mean tier is about 1.54, halfway between tier B ("at least it's grammatical...") and tier C ("might be publishable, but not by us"). So, yes, indeed, if you write a story that is grammatical and tells a coherent story, your story is above average.

What's the interpretation of this graph? Is the "slush pile" really an endless flood of terrible stories? Well, that's a writer's way of phrasing the question. Writers try to write "good" stories and think that "good" stories should be published. But the editor's fitness function is a lot more complex than "good". A lot of stories published in big-name mags in the 70s and 80s couldn't get published today.[0] Did they become "bad" over time, or did the fitness function change? I think a lot of boring stuff gets published, but clearly the relevant slush readers and editors found those same stories "good".

But however you slice it, most of the 241 stories we were sent did very poorly on our fitness function. Only 39 made us think "it would be cool to publish this" for any extended period. So I would say that going through the slush pile is indeed a chore, just as editors say. For Sumana it was a demanding full-time job. But it's not an unrewarding chore. It's not some post-apocalyptic wasteland where you scrounge for a precious tin of canned meat. There's a lot of really good stuff in the slush pile.

Let's take a look at the good stuff. Here's a graph of the 39 stories in D and E, after our first round of culling. After this point, the boundaries between D and E grow fuzzy. Objectively speaking, we have a resistance to rejecting any more of these stories, because we've been dithering over it for a while now.

This is saying nothing more than "we've rejected about half of the tier D stories", but it's nice to have a visual. Sumana and I really liked all these stories, but a lot of them haven't even made it this far, and the vast majority of the remainder will ultimately be collecting another rejection note. By the principle of reversion to the mean, the next editor they're sent to probably won't like them as much as we did. It's possible that some of these stories will never be published. So I think my hypothesis is confirmed: not only is there too much science fiction, there's too much good science fiction.

Maybe we're softies. Maybe these stories aren't really that good. Well, we can do cross-checks. The Context of Rejection says 60-75% of the submissions will be in CoR levels 1-7. We're not an established print market like Tor, we're not in those big "writer's guide" books, so we didn't get as many crazies as Tor does. Maybe that's why only 50% of our stories are in tiers A and B. But maybe we were too lenient and a lot of tier C stories correspond to CoR 6 and 7.

CoR says 95-99% of the submissions are at 10 or below. Well, only 84% of our submissions were in tier C or below, so again, maybe we were too lenient. Maybe we should only be considering the stories in tier E. That's the top 5% of the stories. So maybe we're softies, maybe the rules are different for stories vs. novels, maybe the incredible technical sophistication required to use the Internet filtered out all but the most forward-looking writers. In any event, we're not off by orders of magnitude. We're about 10 percentage points off from the guidelines given in the Context of Rejection. And since we started having difficulties after rejecting 90% of the stories, we're in compliance with Sturgeon's Law.

5% of 241 stories is 11 stories. That's twice as many as we originally planned to publish, and more than we're going to publish even now that we've expanded the anthology. There are way too many stories we don't want to publish, but also too many stories that we do want to publish. Or to look at it another way, there are not enough well-paying markets, and not enough editors with different tastes.

This is why I specifically asked for stories that had been rejected multiple times. I wanted to see whether there were stories that I would really like, but that get consistently overlooked. This is also (one reason) why I ignored people who said my choice of Creative Commons license, or some other aspect of the way I was doing things, would scare away all the talented writers. Honestly, I think the Creative Commons thing scared off more untalented writers than talented ones.

I can see how editors might not want to go this deep into the analysis with aspiring writers. I don't want to do anything that would discourage people from submitting stories. But it seems quite possible to write a great story, send it to twenty editors who all love it, and get twenty rejections, because there's too much good stuff. A great story will get nicer rejections than if you'd sent in a scrap of paper on which you'd written in pencil "MAN HAVE SPACEGUN. explode!! NOW IS SAVE"[1] But from a monetary and publication-credit standpoint, it's the same.

Next time: what to do about this. Assuming I can figure out what to do about this. Anyway, back to work. Editors with more experience can tell me if I'm out of line.

PS: I will not tell you which tier we put your story in. Seriously. It can only cause pain for everyone to act like two peoples' sorting mechanism is some reductive measure of your story's worth. If we have specific critiques, we offer them when we reject the story. Use the feedback or ignore it, and try to sell the story to someone else with different opinions. Tier data is only interesting in the aggregate, as a measure of how we responded to the slush pile as a whole.

Example of how subjective this is: someone sent us a tier A story. I don't remember what story or what we didn't like about it. Rejected. They sent us another story. Again, I don't remember anything about the story--I just saw this pattern when running random queries on the submission database. Tier B. Better than the first, but still not what we want. Rejected. Then they sent another story. Man, how long is this person going to torment us with stories we don't want? Tier E. It's an awesome story and we're very likely to buy it. So what right do we have to imply that their first story was "bad"? Because I can put in all the caveats I want, but that's the message that will come through. More pragmatically, would we have gotten the third story if we had implied that?

[0] Sumana would frequently say of a story, "This would be a great story for Asimov's in 1981." That is, someone else (in another time) might publish it, but not us. Tier C.

[1] I figured I might as well publish this online, since Analog wouldn't take it.


Unless otherwise noted, all content licensed by Leonard Richardson
under a Creative Commons License.