As a writer, I used to feel like I didn't have a good picture
of the short genre fiction market as a whole. You send stories
in, they come back. You can get together with other writers in a
writing group and commiserate, but there's no way that's a
representative sample. One reason I started this anthology was to
take a good look at a real slush pile and get some sense of what it really takes to have a story worth publishing.
This is a taboo subject, judging from how little editors talk
about it. I've been told superficially encouraging things
like (not a direct quote) "If you can write grammatical English
and tell a story, you're above average." Okay, but that's just saying that baseline competence is relatively rare. How big is "above average" and what does it look like? Magazines don't publish above-average stories, they publish the best. How many people are the best?
It's a truism that the slush pile is huge, that there's a vast
oversupply of terrible science fiction. I started Thoughtcrime Experiments partly to test the hypothesis that
this oversupply masks a similar oversupply of high-quality science
fiction, endlessly circulating from editor to author to editor like some action-packed Sargasso. With no prior fiction-editing experience, can I take a core
sample of the slush pile and grab a bunch of good stories that no
one else wanted to publish? All for rates that, while decent by
industry standards, are penurous compared to the effort it takes
to write a story? Is the problem with the slush pile that too much of it is bad, or that too much of it is good?
There were 241 total submissions to Thoughtcrime
Experiments. A couple more might come in, from people of whom
we asked another story, but they won't affect this general
analysis. There were so many stories we've forgotten about most of them by this point,
but we have recorded information about our opinions at
the time, which I put into a database. My goal for this entry is to try to draw some
conclusions about stories in the aggregate.
Here's our review process. A story comes in. Sumana reads it
and sticks it into one of five tiers, A through E. A story goes
into one tier or another based on how well it satisfies the
following fitness function: Would we regret passing
up the opportunity to publish this story? From A to E the tiers are "absolutely not", "no", "eh", "yes", and "yes!" In retrospect, we probably only needed three tiers ("no", "yes", "yes!"), but having five does make the graphs you're about to see more interesting. And at the beginning we didn't know we would get any "yes!" stories. We thought we might have to publish "yes" stories with some "eh" for filler.
In terms of the world-famous Context
of Rejection, I'd say tier A corresponds to 1 through 5, tier
B is 6 through 8, tier C is 9 through 11, tier D is 12 and 13,
tier E is 14. There's some overlap and it's not a perfect match,
since the CoR is for novels, but you get the picture. The
inflection point is between 11 and 12, the difference between "I
could see someone publishing this" and "We should publish
this".
Getting back to the process: Sumana flags certain stories in
tier C for my attention, and I also read all D and E stories. Sumana and I have many differences of opinion but
we've never differed by more than one tier. We talk about the
stories. The fruits of our discussion get turned into feedback
for the authors. Sumana makes occasional passes over the pending
stories and sends out rejection letters.
Currently we've sent out rejection notices for tiers A through
C, and for most of D. We've bought one story and we're left with
about 25 that we really want to publish. Now is a good
time to mention that we're going to publish more stories than the
originally-planned five, but not too many more. Certainly not
25. So we need to cull that herd. We're trying to decide on a
good mix of stories, which might mean sacrificing an E story for
a D--and we're not going to be able to publish all the E stories
anyway, as you'll see if you look at the graph below.
The fitness function works well for focusing quickly on the
relatively small number of stories that really stand out to us. But
it's not so good for scientific purposes because it conflates a
lot of variables: technical skill, inventiveness of plot,
vividness of character, adjacence to our personal tastes in
spec-fic, etc.
For instance, all the terribly-written stories ended up in
tier A, but so did most of the horror stories we got, regardless
of quality. I said in the submission guidelines that I was open
to horror if it was clever, but it turns out I wouldn't know
clever horror if it rose from the grave and bit me in the ass
with decaying teeth held together only by metal-amalgam
fillings. Some especially well-written horror got into tier B,
and one really funny zombie story clawed its way into tier
C. Long story short, there'll be no horror (the genre, not the emotion) in TE. It turns out
that's not what we want to publish.
Another example: we got an extraordinarily well-written story
that had no fantastic element in it at all. It went into tier D,
not because we wanted to publish it, but because
it seemed like a wasted opportunity not to publish something by
this person. We made inquiries and got a less mainstream flash
story from this person, which we're considering offering to buy
for half price. So really, that should have been a tier A story, but we used tier D as a shorthand for "do something about this."
So here it is, the graph you've all been waiting for. How many
stories went into each tier? What does the slush pile look like?
This is going to get really unscientific, because what we
start out with is our subjective opinions of the stories. But
I think there's some interesting stuff in this graph.
This has more than a passing resemblance to a normal curve. Of course, if
it were a normal curve, there's a missing tier to the left. It
would look more like a normal curve if we split tier A into two
tiers, one for terribly-written stories and one for stories that
were just extraordinarily incongruent with our tastes. I'm glad
we didn't do this, because the correct response in both cases is
prompt rejection, but it's an interesting observation.
It's a category error to try to do statistics on the tiers,
but what the hell. Let's assign a tier A story 0 points and a
tier E story 4 points. Then the mean tier is about 1.54, halfway
between tier B ("at least it's grammatical...") and tier
C ("might be publishable, but not by us"). So, yes, indeed, if
you write a story that is grammatical and tells a coherent story,
your story is above average.
What's the interpretation of this graph? Is the "slush pile"
really an endless flood of terrible stories? Well, that's a
writer's way of phrasing the question. Writers try to
write "good" stories and think that "good" stories should be
published. But the editor's fitness function is a lot more
complex than "good". A lot of stories published in big-name mags
in the 70s and 80s couldn't get published today.[0] Did they
become "bad" over time, or did the fitness function change? I
think a lot of boring stuff gets published, but clearly the
relevant slush readers and editors found those same
stories "good".
But however you slice it, most of the 241 stories we were sent
did very poorly on our fitness function. Only 39 made us
think "it would be cool to publish this" for any extended
period. So I would say that going through the slush pile is
indeed a chore, just as editors say. For Sumana it was a
demanding full-time job. But it's not an unrewarding
chore. It's not some post-apocalyptic wasteland where you
scrounge for a precious tin of canned meat. There's a lot of
really good stuff in the slush pile.
Let's take a look at the good stuff. Here's a graph of the
39 stories in D and E, after our first round of culling. After this point, the boundaries between D and E grow fuzzy. Objectively speaking, we have a resistance to rejecting any more of these stories, because we've been dithering over it for a while now.
This is saying nothing more than "we've rejected about half of
the tier D stories", but it's nice to have a visual. Sumana and I
really liked all these stories, but a lot of them haven't even
made it this far, and the vast majority of the remainder will
ultimately be collecting another rejection note. By the principle
of reversion to the mean, the next editor they're sent to
probably won't like them as much as we did. It's possible that some of these stories will never be published. So I think my hypothesis is
confirmed: not only is there too much science fiction, there's
too much good science fiction.
Maybe we're softies. Maybe these stories aren't really that
good. Well, we can do cross-checks. The Context of Rejection says
60-75% of the submissions will be in CoR levels 1-7. We're not an
established print market like Tor, we're not in those
big "writer's guide" books, so we didn't get as many crazies as
Tor does. Maybe that's why only 50% of our stories are in tiers A
and B. But maybe we were too lenient and a lot of tier C stories
correspond to CoR 6 and 7.
CoR says 95-99% of the submissions are at 10 or
below. Well, only 84% of our submissions were in tier C or below,
so again, maybe we were too lenient. Maybe we should only be
considering the stories in tier E. That's the top 5% of the
stories. So maybe we're softies, maybe the rules are different
for stories vs. novels, maybe the incredible technical sophistication
required to use the Internet filtered out all but the most forward-looking writers. In
any event, we're not off by orders of magnitude. We're about 10 percentage points off from the guidelines given in the Context of Rejection. And since we started having difficulties after rejecting 90% of the stories, we're in compliance with Sturgeon's Law.
5% of 241 stories is 11 stories. That's twice as many as we
originally planned to publish, and more than we're going to
publish even now that we've expanded the anthology. There are way too
many stories we don't want to publish, but also too many stories that we do want to publish. Or to look at it another way, there are not
enough well-paying markets, and not enough editors with different
tastes.
This is why I specifically asked for stories that had been
rejected multiple times. I wanted to see whether there were stories
that I would really like, but that get consistently
overlooked. This is also (one reason) why I ignored
people who said my choice of Creative Commons license, or some
other aspect of the way I was doing things, would scare away all
the talented writers. Honestly, I think the
Creative Commons thing scared off more untalented writers than
talented ones.
Next time: what to do about this. Assuming I can figure out what to do about this. Anyway, back to work. Editors with more experience can tell me if I'm out of line.
PS: I will not tell you which tier we put your story
in. Seriously. It can only cause pain for everyone to act like
two peoples' sorting mechanism is some reductive measure of your
story's worth. If we have specific critiques, we offer them when we reject
the story. Use the feedback or ignore it, and try to sell the
story to someone else with different opinions. Tier data is only
interesting in the aggregate, as a measure of how we responded to the slush pile as a whole.
Example of how subjective this is: someone sent us a tier A
story. I don't remember what story or what we didn't like about
it. Rejected. They sent us another story. Again, I don't remember
anything about the story--I just saw this pattern when running
random queries on the submission database. Tier B. Better than
the first, but still not what we want. Rejected. Then they sent another story. Man, how long is this person going to
torment us with stories we don't want? Tier E. It's an awesome story and we're very
likely to buy it. So what right do we have to imply that their
first story was "bad"? Because I can put in all the caveats I want, but that's the message that will come through. More pragmatically, would we have gotten the third story if we had implied that?
[0] Sumana would frequently say of a story, "This would be a great story for Asimov's in 1981." That is, someone else (in another time) might publish it, but not us. Tier C.
[1] I figured I might as well publish this online, since Analog wouldn't take it.
![[Comments]](/nb//themes/Default/comments.png "4 comments")
(4) Thu Feb 19 2009 09:25 Thoughtcrime Experiments Lab Report #2: What The Slush Pile Looks Like:
There are three rails, you say? Let's take a look at
this one!
I can see how editors might not want to go this deep into the analysis with aspiring writers. I don't want to do anything that would discourage people from submitting stories. But it seems quite possible to write a great story, send it to twenty editors who all love it, and get twenty rejections, because there's too much good stuff. A great story will get nicer rejections than if you'd sent in a scrap of paper on which you'd written in pencil "MAN HAVE SPACEGUN. explode!! NOW IS SAVE"[1] But from a monetary and publication-credit standpoint, it's the same.
- Comments:
Posted by Zed Lopez at Thu Feb 19 2009 12:53
In Barry Malzberg's The Engines of the Night (included in the currently in print Breakfast in the Ruins), he wrote about his experience editing "Fantastic." In short, he was blown away by how many good stories from established writers he was getting, and it broke his heart that they hadn't gotten more for them than the 3¢/word his market offered (all the more so that he had to reject most of them.)
Posted by Nathaniel at Thu Feb 19 2009 22:54
A statistic I have sometimes been curious to see (but not yet curious enough to calculate) is something like "total number of unsolicited pro buys per month" (across the whole speculative market). It's what, a dozen or something? Maybe a bit more once you take anthos into account?I guess the corresponding interesting statistic would be "total number of pro-quality stories written per month", but I'm not sure where to get the data on that :-). I guess one could make a Fermi estimate.
Posted by Evan at Sat Feb 21 2009 19:16
If you're curious, a similar analytic post from the submitter side:
http://htmlgiant.com/?p=4677
