This problem has been growing for a long time. I think it was back in the seventies, when I was still in Philly, at a PSFS meeting, we had an (author? editor?) who was saying that at the time, there were 900? 1200? sf&f books published every year, and it was growing then by leaps and bounds, so that no one fan could read everything in the field, as they had been able to do 10-15 years before.

These days, the only way to find the best stuff is a) listen to other fen for what they've been reading and liked, and b) get off your phone and duff, and go to this thing called a "bookstore", where they have dead tree products, and browse.

I've found some great things that way. I've also run into bombs (e.g., I will never buy anything by Kevin J. Anderson, after reading the 5th book of his "Hidden Suns" a couple years ago, and felt completely cheated (as well as the writing going downhill).

PS For Charlie: you didn't show up at the DC 17 party for my single malt....

The only problem with that plan is that the bookstores don't have 90% of what's being published - because it's digital.

I wonder - perhaps there's a space for a bookstore-equivalent space which enables the same sort of browsing of digital works?

That'd be a very interesting experiment...

Where's it all going? Back to the 1930s, I suspect when everybody was skint, but when there were lots of people producing pulp fiction: Serials and serial characters in highly segmented vehicles (back in the 1930s there were sub subgenres such as "airship pulps").

Why? As set up costs drop, and set up becomes more generic (e.g. plant rather than product), shorter projects that can be extended become attractive because they represent a lower risk: Nobody liked the your action series? Fine, retool and do horror.

I see this as ultimately applying across the board. We could be in for a wonderfully creative time.

Interesting - particularly the sub-genres. Is there a list of them anywhere?

Are there any statistics of how many people were writing pulps in those days, or how many pulps were produced?

If this is cyclic, that'd be very useful to know.

If any other industry on earth was over-producing to this degree, a price collapse to more correctly match supply and demand would have laid waste to it some time ago.

The key reason that doesn't happen in the creative industries is that people are not only willing to do the work for free, but are happy to do the work and pay to do it.

The price collapse is happening. But it's not reducing the output. At least not yet, and perhaps not ever.

My prediction for the next 20 years depends on how expensive a book/movie/game is to make in both money and time. If it becomes cheap enough, making a creative story becomes something a few people do for a weekend hobby, mostly sold for free (as with apps).

The problem with this assumption is like with books. The cost in money is VERY cheap if you self publish, but still as expensive as 20 years ago in terms of time. I think that the cost of time will be the limiting factor eventually.

Even under this scenario, big studios will still be making big bucks. Name recognition provides a quick way for someone to find a movie without having to search for it or watch many ads. Plus, big studios will still have the latest technology to show off.

You didn't extend your analysis to songs, but that will reinforce your theory rather than counter it.

As for 50 years in the future, short of basic income existing, I don't have a clue how society would be organized. Heck, I don't know how society would organize even with basic income.

I think that there's a floor there as far as time cost goes, and novels are currently pretty close to it.

Scrivener and similar things make writing somewhat faster, but beyond that I'm not aware of any technologies that might make writing a novel 5x faster or more.

Likewise, I don't know of anything that's going to make writing a screenplay faster, or doing the core design work of a game. And both of those take approximately the same time as writing a novel - screenplays are a bit faster, but not as much as you'd think, and there's a lot more authoring to do beyond the screenplay to create a film or a TV show.

So obviously all those artforms will hit a time floor eventually, and my guess is that it won't be at a level where it's comfortable to do as a hobby, unless you consider being a novelist to be at that level now.

There's another category you should include: YouTubers. Youtube is my primary source of entertainment these days, I have around 20 channels I subcribe to, most post one video a day around 15 minutes long so each day there can be around 5 hours of youtube content for me to watch.

I tend to not watch all of it, aside from finding the time not everything grabs me. But it's obvious how it can become unwieldy.

The thing about youtube that is different to what's mentioned so far is that good youtubers continually post new content, daily. I'd suggest that it's difficult to get hooked on new people not just for time but because every hour spent watching new content is an hour you're falling behind on your regular content. Much of the time that isn't a problem but if you watch a lot of 100 episode Let's Plays or the like it is.

Terrifyingly, there are lots of categories I've excluded.

Podcasts. Bloggers. Documentary filmmakers (who would have doubled the numbers in the "feature film" category above). YouTubers, as you say.

I've really limited the number of outlets I've looked at in this article: the true picture's probably much, much more crowded.

(I agree re YouTube and content production - although there are other models that work the same way. Soap operas, for example, pump out nearly as much content per week as your average YouTuber. Some bloggers and some podcasters are pretty prolific too.)

Perhaps Second Life?

Or maybe the Big River would have a go at (re?)resurrecting VRML?

Unless someone comes up with an AR/VR implementation of L-Space and gives us an endless array of (nightmare) stacks...

As I've commented in other places, my anecdotal experience shouldn't be taken as gospel, but thanks to BookBub and Charlie and other authors and eBooks and smart author tactics I'm reading MORE new authors than before.

BookBub looks for all the books in certain categories (that I choose) on special offer and sends me a daily email. It does so sufficiently well that I now only choose books that a) seem interesting and b) are free. That way if they're no good (IMO) despite the rave reviews, I don't have any hesitation in hitting the delete button.

But, that leads me on to my next thought on the topic: I don't trust the rankings.

For one thing, they skew high. Unless you're a paid critic, do you really go and write up a review on IMDB or whatever of a movie you thought was meh? You MIGHT write a review of a movie you hated but if you're paying for it, you probably don't go to a movie you know you're going to hate. You almost certainly don't review it if you think it's 5/10 movie.

Even allowing for that, it's a measure of popularity (which is important if you're trying to sell something) not necessarily quality, and not necessarily my individual enjoyment. Case in point: Game of Thrones is hugely popular and critically acclaimed. I don't like it. I'm not saying all those fans and critics are wrong, they make their choices and I make mine. It doesn't do it for me.

In a typical year I go to the movies quite a lot. I see about 25-35 feature films. I write, for my own pleasure, a review on each film. I probably miss 1-2 films I'd like to see in a year with illness and other things. 2015 is shaping up to be a pretty typical year. There may be about 70-80 very highly regarded films this year, that doesn't mean there are 70-80 I want to see. For example, American Sniper is supposed to be an excellent movie. I'm not sure if it made your cut or not but I'd be surprised if it didn't. I had at the time and still have no interest in seeing it. An entirely different war movie that I'm pretty sure didn't make your list, The Water Diviner, is probably my top film of the year to date, although the film version of Into the Woods and Inside Out are running it close, and The Hunger Games: Mockingjay Part II I'm hoping will be up there too.

So, for lots of reasons I'm going to question your numbers. Yes, there's more stuff made than we can reasonably expect to consume, and yes, we undoubtedly miss stuff we'd probably like. Yes, that's probably worse for authors where advertising is harder than film and TV. But mechanism exist and work. I have probably ten new authors (all female) whose books I now pay for whose first books I found out about through BookBub and tried free and then bought the rest of the series and I'm waiting for more. I also have 19 new books to read.

I add to that an approximately equal number of books from recommendations on other author's blogs or guest posts. (They're more mixed gender FWIW.)

Yeah, my initial thought was to fire up Unreal Engine 4 and do something in the Oculus Rift.

Hmm, I sense a weekend project coming on...

How would you give it the same accessibility and non-linear feel as a bookshop, though?

Seriously, I might make this. Ideas very welcome!

Good analogy with soap operas. There's been a bit of a trend with some of my Youtube subscriptions that have really highlighted this point; a few of the let's players have started using twitch as well. They then post the recording of the twitch stream to their channel, usually as one block or split up into 20 minute segments, but in any case it ramps up the length of content per day to 3-5 hours.

At first I loved that because hey, more content! But I found myself enjoying it less and less. I realised that the reason was I didn't have time to watch it all so was falling behind on the in-jokes, references etcetera that made it into other videos.

To put it in perspective, assuming 8 hours a novel, you'd need 32 years reading non-stop - no sleep, no food, no toilet breaks - to read this year's output of fiction alone.

This sounds like a job for Harriet Klausner!

Hmm... Procedurally generated (based on subgenres? tags? 'was cover-quoted by X'?) twisty passages, all different, with lots of stacks, racks, bookcases and stairs, taking you up, down and around like the lovechild of Hilbert and Escher, with a 'look inside' or Google Books style look at the actual content to go with the cover and blurb?

Will it make any real difference at all? After all we are Pan narrans the storytelling ape, or so Pterry claimed ....

"...we just say "screw it" and re-read Accelerando instead."

I can honestly say that has not been my experience either. Since I got a kindle about 3 years ago the number of new authors that I've read has increased out of all proportion. With new authors going for a couple of quid I don't spend hours in bookshops umming and ahhing any more I just buy the damn thing.

Very few I dislike, most are OK and a few have joined my 'must buy' list (says the man with two of Jodi Taylor's St.Mary's stories on pre-order).

I'm really positive about the future. Creation always has been 99%+ amateur (think how long PTerry did PR for power stations) but until now there have always been gate keepers deciding what gets published. The drop in production costs means the gate keepers are losing their power. That works for me...

That's a good point, and one that's easy to test.

I went to IMDB and searched for films with the same criteria I used above, but this time rated between 4.0 and 6.0 - the very definition of a "meh" movie.

2,697 results - significantly more than there were 7-and-above movies, and massively more than the 72 films rated 8.5 or above.

Clearly a lot of IMDB users do indeed vote on films they thought were "meh".

Amazon's reviews I'm less certain about, to be fair. Just looking at the results I'd say they skew a bit high, but then I've not randomly sampled the 4.5-and-above reviews to tell.

As to the films you choose to see or not - I completely agree that most people won't want to see all 85% or above films. However, as I mentioned above, most people WILL also be interested in some subset of the sub-85% films. (The Water Diviner has an overall 7.2 rating on IMDB - but interested you enough to see it and love it).

My point is that there are probably dozens of other feature films out there that you'd like, and you have never heard of. Perhaps hundreds. 99% of the features released never make it to your local cinemas.

Incidentally, I think BookBub is another example of the bubbling discovery-based business model, and in fact one of the pioneers. It's a very encouraging sign. For the reasons I mention, film and TV lag way behind in that regard.

Would you say you're reading a lot more novels per year, then? If so, any idea what multiple?

That's interesting. I would love to get similar data on Steam and Netflix to see whether consumption is increasing there too...

But, that leads me on to my next thought on the topic: I don't trust the rankings.

For one thing, they skew high.

Yup. Given you can buy Amazon reviews:

http://www.forbes.com/sites/suwcharmananderson/2012/08/28/fake-reviews-amazons-rotten-core/

You can also get negative reviews removed by having lots users (or user accounts) complain about them. Apparently a common custom in self-publishing circles (give each other good reviews, complain about negative reviews). Sorry, can't locate the reference I had to that (URL no longer valid) but lots of stories online about negative reviews and ratings being deleted by Amazon.

This has happened already in areas more esoteric than fiction. I've seen it said that John von Neumann was the last man in history to know what was going on in mathematics as a whole, and he died something like fifty years ago.

Not mentioned in your list:

Disintegration of the canon, and consequently of "SF fandom". Because without a shared canonical set of works that people have in common, you no longer have fandom.

Hence, Puppygate.

Hence, also, Gamergate. The "there are no gamers, gamers are dead" article that got them hugely up in arms actually said that there are now many different games of different types, and many different types of gamers playing them: and so anyone claiming to represent "gamers" or be a "gamer" was almost certainly just talking about one small subset of gamers who played one small subset of the games that are around. That the days of a simple fan-base who all participate in the same canonical set of games is gone.

Some people found that claim about games and gamers very, very disturbing. (Carl? You're our token gamergate advocate - your view on that?)

If there are 20,000 new SF novels, paradoxically, the sheer volume of choice and difficulty of knowing what to pick means that we just say "screw it" and re-read Accelerando instead

There's a bunch of studies around this sort of thing (although not in the arena of publishing AFAIK). Have a google for "choice overload hypothesis".

(Although they've been tough to replicate — this meta-review gives some pointers).

Speaking strictly from a readers' perspective, my big problem with removing the gatekeepers is that while they kept some good stories from being published, they also kept a lot of bad stories from competing for my limited reading time. (IIRC Charlie mentioned something like 90% of the slushpile being crap at first glance.)

In the brave new world where everyone who wants to tell a story does so, how do I find the (far fewer) who tell good stories?

Genuine question. I've been trying to find new authors on iTunes by reading the free samples, and it's painful — I could write better than a lot of them*. (Not a high bar to pass, in fact a bar lying in a ditch under several feet of muddy water, so not clearing it means truly awful writing.)

*And I suck as a writer. While cleaning I found some old stories I was writing, and am really really glad they never made it online. Clunky prose, leaden dialogue, cardboard characters — embarrassing.

Is there a way to filter your book results by how many reviews they got?

If so, you might want to go with >10 reviews, and resample.

What I'm thinking is how my first self-published book collected at least three glowing reviews, which was very kind of my mom and two other people. High scores, small audience, and no surprise there.

Not that there aren't hidden gems out there, but it would be good to get out from under the Mom and Circle of Friends reviews, and see how many books still make the cut.

FWIW, I did do a casual random sample of the books with >4.5 rating, and most of them had 20-50 reviws or more.

Unfortunately, there's no way to filter beyond that on Amazon (that I'm aware of) without writing a scraper, which was outside the scope of this post.

(I also considered running some Monte Carlo simulations of consumers' reaction to the mass of media, but again, not enough time.

You could try filtering Kindle authors on Amazon by review of 4.5 or above, and then reading a random selection of cheap ones with a lot of reviewers.

Some won't be to your taste and some will have gamed the system, but you might find some interesting reads there.

BookBub also get recommended for this sort of thing.

Beyond that - yeah, discovery is a problem.

So maybe my stories are not so dreadfully bad.

And maybe the fan-fic model will work as a distribution channel. If there's no chance of being paid anyway, why not?

"I expect," said Arabella, "That if the pirates had succeeded he would have been ransomed. And a payment would have been made. It would have bounced around between banks a few times, maybe gotten changed to bearer bonds and faded from sight in the markets of Pallas or Ceres."

There were nods of agreement.

"The astrography is awkward for Pallas and Ceres from Mars. And there are a couple of interesting banking houses on Ganymede." Stagg steepled his fingers. "I'd like you to go to Ganymede. Saunders?"

"Boss... The thing is, Narcotics rather likes to have some funds in slightly dodgy banks, where our field agents can get hold of it, no questions asked. The sort of bank a real gangster would use. And those accounts need to be topped up from time to time. So you'll be selling a stack of bearer bonds on the Bourse de Ganymede. And, Ganymede being Ganymede, expect trouble."

"Of course," said Arabella. "But trouble won't be expecting us."

I need to do some more work on the story, I may change a few things. I was reading through some of it, and I am sure the Puppies won't like it. Should I care?

I saw enough of their storytelling in the Hugo packet. They ought to stick to NaNoWriMo.

And I might get lucky.

5250 games were released on Steam alone last year. Across all platforms there wasn't a single month where less than 1,000 games were released, according to Metacritic.

That's extremely doubtful.

While there's been a rash of releases pushed through of dubious quality, it's no-where near that total.

Steam Reaches 100 Million Users And 3,700 Games

Valve's online platform has added 25 million new accounts since January; more than 1,300 titles added to Steam so far in 2014.

http://www.gamespot.com/articles/steam-reaches-100-million-users-and-3-700-games/1100-6422489/

Steam itself lists 6,243 games total available, however, Steam counts DLC, expansion packs and so on as separate entities.

http://store.steampowered.com/search/?sort_by=Name&sort_order=ASC&category1=998

Rule #1: Never trust Metacritic, it's rubbish.

Speaking personally I'm reading more new authors (although not more books) since I started reading ebooks more. Because things of things like:

• The nearest even vaguely good book store is an hours travel door-to-door, at a location I don't regularly visit. Many places I've lived in the past have been even worse. So until ebooks I wasn't in a position to buy books often. So my time was limited. So I invested that time on safer bets — familiar authors or personal recommendations.
• I can browse new authors at any time on Amazon. See a recommendation on a blog, read a nice review somewhere, read a glowing recommendation on social media, etc. — and less than a minute later I have a kindle sample on my iPad & phone & desktop. Those samples I can consume at any time. My options for new author discovery are now "anytime" rather than "an hour in the bookshop once every month or two".
• Purchase friction is almost zero. I have on several occasions been sitting in my armchair, finished a book, browsed for a new book, purchased it, and continued reading — rather than get off my fat arse and walk six feet to the pile of unread books on the other side of the room.

Don't get me wrong — I do still really, really like browsing in bookshops. However I'm not really sure that for the majority of people they're a better route for discovering new authors that the rather primitive current state of ebooks.

Although — when I said "new authors" what I really should have said was "authors new to me". Which is a whole other thing. Because as well as the massive pile of new incoming books — now the "old" books don't disappear. "Not in stock" or "out of print" don't really happen now.

A new author doesn't just have to fight to get heard among their cohort of new books. They have to fight to be heard above every book out there that I haven't read,

For example — at my current rate of reading I have about 15-20 years of reading on my amazon wish lists right now. And that's just the books out there that I know I'm interested in looking at. Let alone the thousands of others that are out there already that I haven't heard of.

In 50 years time the biggest competition for new authors is probably going to be dead authors…

Actually, wait:

Re-read that statement:

5250 games were released on Steam alone last year. Across all platforms there wasn't a single month where less than 1,000 games were released, according to Metacritic.

12 x 1,000 = 12,000.

12,000 > 5,250

This is all very Kaufman.

Steam is a PC-only distribution platform.

Hence, I went to Metacritic, which includes Steam games, iOS, PS3, PS4, etc, to get a better overview of the situation.

Yeah, that's another of the rocks I didn't turn over in this article :)

My model rather naively assumes that we're not interested in old media any more - but that's just not the case, even a little bit. Even in games, that traditionally most past-hating of media, folks like GOG are making sure that old classics are still available to play. I'll probably be spending some time on "Eye Of The Beholder" this year - released more than 20 years ago.

So in fact the competition is far, far steeper than I've naively assumed here...

Definitely. Reading more but more importantly I'm buying more.

For the last 20 years I'd buy maybe 5-6 novels a year plus get a few as presents. In the last 6 months I've 65 kindle orders and I'd say 2/3rds of those were fiction, albeit of varying lengths.

That's true - however, to take a random sample, of the 100 most recent releases on Steam, 3 appear to be expansion packs, DLC, etc.

The rest are new games.

Steam did not release 5,000+ games last year. The total was roughly 1.7-2k, and that was seen as a huge increase because certain restrictions were removed (notably Greenlighting games became much much easier and there was a rash of 'retro-releases' since license holders noticed the profitability of GoG etc. So, not new games, but either direct re-releases or "remasters")

Firstly, including iOS is meaningless:

Mobile is as tough as ever, being a mature, crowded market. Rose said about 500 games launched every day on iOS in 2014, and 250 a day on Android. "On Steam, at least you get featured in the new releases at least," he said, comparing PC to the mobile market. "That's not the case on mobile…you have to make [exposure] happen yourself."

Gamasutra 2nd March 2015

This is no-where close to PC / Console reality.

The release schedule for 2014 lists ~476 titles by major publishers across all other platforms (Xbox One, PS4, Wii U, 360, PS3, 3DS, Vita, PC)

http://www.gameinformer.com/b/news/archive/2014/07/09/2014-video-game-release-schedule.aspx

The indy market is much harder to measure, but most of those platforms are rarely supported. Let's assume double to be generous - so roughly 1,000.

That's still 1,500 titles in a year, and your point holds. Wasn't being unfriendly, just pointing out hyperbole.

iOS is just... well. 500 / day, I'm sure you can work out that they're not actually games per se. Copy / Paste is generous.

As for Metacritic scores. Not a reliable source of data - the review side is run by corporate PR, the user reports dominated by trolls.

p.s.

I'll try and dig out actual industry numbers if you want.

Ok, again, sorry to do this:

There were no-where near over 9,000 films released last year.

MPAA 2013 [PDF - slide 22, under 800]

MPAA 2014 Report [PDF - slide 21, under 800]

Typo, I guess?

Um, of course.

Ahem.

There's an elephant there.

There might have been 8,200 naughty films released, especially since the internet, cams and amateur snapchats and vines have sliced into the market.

In 2011 the site released a report of compiled data for the year.[14] Included in the statistics released were the number of new titles added that year (9,384) and a ranking of who the busiest performers were.

https://en.wikipedia.org/wiki/Internet_Adult_Film_Database

http://www.iafd.com/ [NSFW]

Very fascinating read and completely in line with my experience as a gamer. For a while it was ok to look at the "what's new" list on steam every week or so and see if there was something interesting. Nowadays there's 10+ new games per day and it's become impossible to figure out if there's actually anything good in there without considerable time investment.

Interestingly the thing that will inevitably need to happen to solve this problem is something Steam is already doing: Improve the tools to filter out things one doesn't care about. Steam allows hiding games, games have tags, and there's a suggestion queue that feeds partly off the user's preferences and partly off of what's new and popular. This means occasionally one can go through that queue, see a list of new or obscure things one might be interested in, and use the tags to quickly decide whether a game has elements one wants or does not want; and then mark it for later, or hide it.

Whenever i interact with other shops (PSN, GOG, Google Play Store, iTunes, etc.) it is a massive pain point that these do not allow filtering things that i do not care for.

Music and TV stores are even worse in that department, and i've seen my parents look at stuff like Amazon Fire Instant Video, which on paper is an amazing offering, and discard it offhand because the first 20 things it offered were stuff they had no interest in, and there was no way to make them go away.

Fascinatingly, i've already seen the software where this evolution will lead to. It exists in the form of taggable image board like https://github.com/r888888888/danbooru They accept tags from the users and are very liberal in allowing users what tags to apply, with some moderation. On bigger boards the content tends to have astounding amounts of tags in surprising detail. At the same time users can use tags to create lists of things they might like, or directly hide things they might dislike.

Steam is moving in that direction, and as the amount of content, both existing and newly-created, becomes evermore massive it seems inevitable that software will end up in that domain.

Shout out to The Swapper (2013) writer btw. (I'd do a spoiler, but the protagonist isn't what you think). Talented and funny guy. The game is about " Two antagonists are just Chalmers and Dennett[...] fight over ownership of players soul". Full disclosure: played The Swapper on release.

He also worked on FTL.

And then worked on The Talos Principle.

He's not exactly an unknown talent in the indy field!

He recently did a talk called How video games will destroy humanity (Aug 31st 2015).

Sorry to derail, but he's hot.

~

To non-derail - I guess you wiggle your nose to find the underlying vibes. Find the noses and wiggle your tentacles at them.

Pricing also makes a difference. A $10 book has to overcome a certain amount of resistance to get me to buy, and a $25 hardcover is a rare purchase. A $3 ebook faces hardly any resistance. I probably won't even bother with the free sample.

At first traditional publishers had the advantage of author recognition, but with time and sampling I've developed a small roster of trusted ebook authors.

I probably shouldn't say it, but I think the amount I'm willing to spend on unknown authors is tending to zero. Doing the rough audit on my spare time media habits results in:

TV (much downloaded) : 7 hours per week YouTube et al : 3 hpw Movies (Blu Ray) : 3 hpw Music : 2 hpw Books (mostly ebooks) : 1 hpw (want ideas SF, which is dying) Gaming : 0 hpw (bored of the repetition)

with the rest taken up with browsing, forums, etc.

I'm not going to spend even $10 on an untrusted book option in that context, I've movies I haven't watched.

I think the only viable model for aspiring authors, going forward, is to GIVE AWAY the book, or at least the beginning third of the book, such that you can hook the buyer (then make it as easy as possible to buy the rest). As such I don't see them as being dead tree authors till they've made it. Hence I don't see bookshops surviving.

What we have is an 'alpha and the rest' market where the top two or three items can make mountains of cash on a global scale, but where that global scale means the level of competition is massively higher as well.

Go large, or go home, is the order of that market.

"In the brave new world where everyone who wants to tell a story does so, how do I find the (far fewer) who tell good stories?"

I've experimented lately with doing this for fanfiction, on AO3 and Fanfiction. I have a few heuristics that seem to work tolerably well for identifying authors whose work I can reliably expect to enjoy.

• Search on fandoms that appeal to me. That's both obvious and not very useful for original fiction, of course.
• Avoid stories that emphasize sexual pairings, even pairings that I like. Doubly avoid stories where one of the pair is an OC ("original character"), as that's likely to be the author's personal avatar, and that kind of fantasy hardly ever appeals to me.
• Be cautious in general about stories that say "OC," even nonsexual ones. Of course OCs are likely to occur in most fanfic, but those that are called out as OC are likely to warp the narrative around themselves, the way Lancelot did to French Arthurian fanfic back in the day.
• Beware of authors who say negative "I know this is crap" things about their writing; they're often right.
• If an author does one thing very well, look at other things of theirs, if you have any interest.
• Sample the first few paragraphs. If the prose is actively appealing, there's a good chance the rest will be. If the prose is distressing, life's too short.

Not all of those transfer over to original fiction, of course. But some of them do. And the general principle of inventing heuristics applies far more widely.

There is so much stuff out there that I could spend the rest of my life reading and enjoying authors who died before I was hatched and not do more than scratch the surface. (And in many cases enjoying them more than current authors because older books tend to have far less sex in.) Ideas for trying authors I haven't read yet arise spontaneously out of random stuff in the general datastream of life at a rate more than sufficient to keep me occupied. There is essentially zero incentive for me to seek out works which are new in absolute terms as opposed to being new to my experience.

The only current author I find sufficiently interesting to keep track of their new publications is The Charlie Himself. I am sure there are other current authors who would interest me just as much, but I have no reason to prefer "current" over "retired" or "dead" (indeed the latter may well be preferable both because they're less likely to include sex and because I can inhale their entire oeuvre in one go rather than messing about waiting for the next instalment, which I hate). And the sheer numbers mean that by the time I get to hear enough interesting stuff about an author to think they might be worth checking out they are unlikely to still be in action.

For at least 15 years, my To Read Pile has been a To Be Read Singularity. Referencing recent posts, I just noticed that I have 4 Tricia Sullivan books on my shelf and I have not read a single one of them. I already have enough entertainment (books, music, films, shows, games) in my house to occupy me not only for the rest of my life, but to make a start on a Pharaonic afterlife. Yet I need and crave new material; so much of this stuff becomes the bedrock substratum of my consciousness as fresh layers are laid down seeking my more immediate attention.

This is a very personal cognitive dissonance which also mirrors the state of much of the world as well.

Yet somehow I still seem to watch the Sopranos almost every year. My extended brain is growing every year, but the original base is calcifying even faster. Sometimes I actually catch myself wondering what is going to happen to my library after I die or become (more) demented, as if it were a living being whose existence is at least as precious as a more recognizable sentience, not a random collection of objects and/or information which will have very little unity without "me" there to impose such a concept on it.

Add fanfiction to your narratophile's dilemma, and even at a 95% crap level, your reader's problem just increased by an order of magnitude.

Most people in Gamergate that I've read/interacted with didn't have a problem a single "Gamers Are Dead" article.

The anger came from multiple such articles appearing over a three day period. That gave rise to suspicions of journalistic collusion and an attempt to set an anti-traditional gamer narrative. Lest you think I'm exaggerating, the following articles appeared between 08-28-2014 and 09-01-2014:

"Gamers" don't have to be your audience. "Gamers" are over (on Gamasutra) We Might Be Witnessing The "Death of An Identity" (on Kotaku) Gaming Is Leaving "Gamers" Behind (on Buzzfeed) A Guide to Ending "Gamers" (on Gamasutra) The death of the "gamers" and the women who "killed" them (on Ars Technica) This guy's embarrassing relationship drama is killing the 'gamer' identity (on Vice) There are gamers at the gate, but they may already be dead (on Destructoid)

Bookstores are also getting harder and harder to find, and seem to carry less and less actual books each year. There are less actual brick & mortar bookstores now than at any time in my memory. I'm just happy to live in an area (Dallas/Fort Worth) with the highest concentration in the world of "Half Price Books" used and remaindered books stores. The nearest new bookstore to me (a Barnes & Nobles) is a twenty minute drive away.

Second Life (which I love) is very ill-suited to selling anything outside of stuff for Second Life. Due to built in limits of number of Avatars allowed in a region, and a dedicated, but niche audience, SL is not the place to sell books. I've seen people attempt it. Mike Stackpole had a virtual bookstore featuring his work in Caledon Mayfair (where I lived in my SL identity of Carl Metropolitan) for two or three years. I almost never saw anyone in it.

Right now, Amazon is the best thing out there to simulate a real browsing experience. And it is only mediocre at that.

Your point about Adult movies is well taken, but it is also possible the figures that Hugh Hancock cited from the IMDB were for all movies worldwide in all languages. If that's the case I can see that number as realistic.

Interesting. From my personal perspective I see an increased reliance on gatekeepers, by which I mean trusted people who can provide on-target recommendations.

To a degree critics have always provided this function, but they are very inefficient; nowadays there are book bloggers, but they have low credibility (cliquiness, corruption, power games... it's become a messed-up field). Consumer reviews on Amazon etc no longer work - they're naturally skewed anyway, but now they are systematically gamed (especially at the new author/self-published end of the field). So who are the new gatekeepers?

I follow recommendations from authors I like and trust, so a word from someone like Charlie can make me pick up an author or a book that I'd otherwise ignore. Secondarily I note tips from commenters on author blogs. I don't even look for a review, just 'this is worth reading.'

Unfortunately many authors (Charlie included) are quite stingy with the recommendations...

I'll see your 300 player MMORPG and give you the in person equivalent, the megagame.

Here's a video from Shut Up and Sit Down about Watch the Skies 2, a 300 person slightly SF megagame of first contact. http://www.shutupandsitdown.com/videos/v/susd-play-megagame-2-pt1/

Bookstores are also getting harder and harder to find, and seem to carry less and less actual books each year. Yes, I've already remarked on one aspect of this ... my local Waterstones' seems utterly pathetic. However, I will be there later today, to see if they have already sold out of The Shepherd's Crown or not .....

Really interesting post, thanks!

I was wondering where are mainstream bundle type campaigns for other media than games? Amazon suggest system does something like this, and there is the Humble Book Bundle, but this could be taken further.

Nope, you need to see the distribution of the votes to determine if there are lots of "meh" scores recorded or not.

Consider some extreme cases with only 10 votes recorded for simplicity. If everyone votes 10, obviously it gets a 10 average. If everyone votes 1 it gets a 1. If 5 vote 1 and 5 vote 10 you get a solid meh of 5.5. If 9 vote 10 and 1 votes 2 you get 9.2 and so on.

I'm sure IMDB has people who do vote obsessively on every film, just like I review every film but I'm pretty sure their results skew high. Amazon actually publish a little bar chart of the star ratings given, you can see it in operation there. 5, 4 and then 1 star reviews typically dominate, even on books with over 500 reviews you often see 0 for 2 and 3 stars. I don't know the Google Play market but the App Store markets both show the same pattern.

Actually, ignoring the actual numbers (and, at least for games, the question of "is $title for PC really a different release from the PSX version, the XBrickN version, the Ninty version, the Android version...?") the fact of large volumes of releases makes me less likely to chance $newauthor (or equivalent) without a supporting recommendation from $groupI_trust (present company very much included) simply due to the "swamping phenomenon" making me reluctant to expend time (never mind money) on them.

I think I might have pushed some limits with that quote I posted.

[[ Yes. You did. As a known commenter you got away with it this time, but don't tigger too much next time - mod ]]

But I seriously wonder if conventional publishing models will be viable for long enough to be worth the efforts of getting noticed. There's some talented people in that business, picking books. They're worth watching. But there are problems. Can the publishing industry maintain a per-book average sales level that supports a search for new writers?

Some of the possible answers to their search problem, such as keeping an eye on Amazon for the self-published authors who rise above the sales median (which is staggeringly low anyway), are distorted by other features of the operation: is it sane to sell through Amazon, so being forced to sign up for the US tax system, for such a minuscule return.

It's a little too easy to feel like the exploited authorial proletariat, with Eric Raymond as Karl Marx (maybe not a good fit) and Amazon as Joe Stalin.

Interesting. From my personal perspective I see an increased reliance on gatekeepers, by which I mean trusted people who can provide on-target recommendations.

To a degree critics have always provided this function, but they are very inefficient; nowadays there are book bloggers, but they have low credibility (cliquiness, corruption, power games... it's become a messed-up field). Consumer reviews on Amazon etc no longer work - they're naturally skewed anyway, but now they are systematically gamed (especially at the new author/self-published end of the field). So who are the new gatekeepers?

Punchline: it turns out ethics in entertainment journalism does matter, but the phrase has been so totally corrupted it can never be talked about again.

Huh. I didn't realise Megagames were still going on. My dad ran one back in the 1990s simulating Gulf War 1: it was held in a church hall using intercoms to comunicate information between factions and the GMs. I beleive the Iraqis managed to force a bloody stalemate, which was pretty impressive.

Oh yeah.

And then we've got short films and short fiction.

There are a lot of stories out there.

The MPAA report is simply the wrong thing to reference in this context - it's a report on theatrically released feature films.

90% of all non-studio features do not get a full theatrical release, or anything like one. They go to Netflix, cable or iTunes directly after a few festivals, maybe a couple of four-wall showings (where the producer pays for the showing), or they simply never get wide release at all.

One of the reasons that the film world is having such problems right now is that the ecosystem simply isn't set up to handle the volume of features being produced. There's a limit to the number of cinema screens, but there's no limit to the number of features.

Which takes us right back to a gatekeeper-based world! Wasn't the Internet supposed to free us from that?

I joke. The interesting thing here is that the cost of setting up as a gatekeeper is also considerably lower than it used to be. I've been waiting for an explosion of gatekeepers for a while, and it hasn't happened yet, though.

Maybe that's because the gatekeepers most people trust are existing, famous figures in the same medium, and those people tend to be too busy to function as full-time critics? We haven't yet really established a trusted-critic class on the Internet, at least outside the games world.

(Unless such folk exist deeper into the literary sphere than I usually read?)

I'll dig into it and see if there's a way to view the distribution. Good thoughts - thanks!

I'm a long-term re-reader, both for financial reasons and also out of personal preference.

The financial reasons are simple - my partner and I are currently "enjoying" our fourth period of unemployment in five years. Consequently, entertainment purchases have dropped waaaaay down the list of potential uses for money. So we're back to re-reading books we already own, borrowing books from the library if we want to read new stuff, playing DVDs we already own, sticking with the stuff available for free on the internet (lots and lots of fan fiction, for example), re-playing games purchased previously and so on.

On the personal side, it's a bit more complicated.

As I'm sure I've mentioned (repeatedly), I have chronic depression. This means for me the question of whether I enjoy something is really problematic and multi-faceted. To begin with, am I currently in a mood where enjoyment is even possible, or am I undergoing one of my periodic fits of anhedonia? If I'm not likely to enjoy something new in the first place, I'm more likely to just leave it to one side, and go back to the tried-and-tested standbys (re-reading Georgette Heyer, playing match-3 style games or solitaire, mahjonng and so on - things which don't really require much motivation or enjoyment to continue). Which means I'm not getting the exposure to new content when I'm going through the anhedonic phases of depression. If I'm in a situation where I can actually take on board new content, I may not be able to do so for very long - for example, if I'm just on the edge of an anhedonic episode, I will be able to read new fiction, but any problematic aspects (for example, if I'm reading fan fic and the writer is misusing words - "loose" instead of "lose" or "taunt" instead of "taut") will tend to throw me out of the story and I'll wind up rage-quitting instead. So on those "edge" days, I'll probably be re-reading stuff as well, or re-playing known safe games, too.

Then there's all the little neurotic questions about whether or not the thing I'm doing/reading/playing/watching will edge too close to the landmines in my skull - if that happens, bang goes the enjoyment factor. Plus, of course, there's the straightforward little physical issues which are becoming more and more important as I get older - can I read the text, or will I have to put on my glasses? Can I make sense of what the actors are saying, or will I need to find the sub-titles? Does this game require the player to have the reflexes of a greased ferret on crystal meth in order to make it playable? Is this a game where the camera perspective isn't fixed, and am I therefore likely to wind up queasy after playing it for about ten to fifteen minutes?

With things I've already tried, already read, already watched, already played, I know the answers to those questions. It makes them far more readily accessible than new stuff.

Then there's the consideration of time. Most weekdays, I spend the day doing housework in instalments. So if I'm reading something, or watching something, or playing something, it needs to be able to be put down so I can do the next bit of housework. Generally I'll try to restrict my breaks between instalments of housework to about thirty minutes each, so I need to be able to reach a "break point" in that thirty minutes (which is why I don't play Final Fantasy games while I'm doing the housework... they're currently reserved either for evenings after the housework is done, or weekends, when I take time off). Again, stuff I'm re-reading/re-playing/re-watching I can put down and not worry about, and pick up fairly easily when I get back.

Incidentally: my TV usage dropped off catastrophically about 18 years ago, when I first got access to the internet, and hasn't picked up since. To the point where these days, our television set is hooked up to the game consoles, but not the (free-to-air) aerial. I'd made the decision back then that life was too short for the stuff on the screen.

William H Stoddard @ 44 - one little heuristic I've found that works very nicely on AO3 is this: if I like what an author has written, I'll generally like the stuff they're recommending, too.

My gatekeepers were never really faceless journalists.

Faced critics were a different matter - in film for example, Barry Norman, Jonathan Ross and now Mark Kermode I can watch and consider their film reviews because I've seen enough of their reviews and films they've reviewed to consider what they're like in what they tell me. Barry Norman was always going to hate anything vaguely SciFi, so you could discount how much he disliked it and listen to the other things he said about it. Wossy on the other hand was going to Fanboy OTT about it... and not all SF movies are worth my time, even if they're worth his!

Nowadays, for books, my gatekeepers are places like this: authors I like who recommend books or invite other authors to write on their blog. People I know in the comments recommending things.

Plus, as I've commented above, free eBooks on BookBub. That's not curated for me beyond a tag and a price. I throw away about half of what I read before I finish it (that's why I only read the free stuff) and more than half of the rest I finish and it goes no further. But I've found a chunk of new authors - including Annie Bellet of Hugos fame for the misfortune of being roped in to the Sad Puppies etc. thing that way.

New systems are emerging and working for some at least.

You could certainly knock up a Library of Babel in a weekend, and then you'd be future proofed.

Having this kind of analysis is really useful for refuting the quality/success fallacy, and feeling better about that backlog.

I try to cope via heuristics, mostly.

'Avoid if free' works pretty well on the consumption side, 'Do it for fun' on the production side.

Ironically, as I'm not charging for what I produce I'm banned from playing my own stuff.

Another very interesting analysis to do would be to look at the quality of free stuff vs paid-for stuff.

Charging sends a powerful quality signal, but is it actually backed up by the data?

I'm not sure how to go about doing that outside the games sphere, though.

The beauty of the heuristic is that it doesn't matter if the quality of paid for stuff is higher or not. As your analysis shows there'll still be more than enough great stuff in either pot.

You could equally go for 'only free' instead, but from experience as an 80s playground games pirate, I value things I've paid for more than things I haven't irrespective of the quality of the thing itself.

Related to Robert Prior's comment above, gatekeepers (or heuristics) may keep some good stories from being published, but in a system where there is an overload of good stories this is a feature, not a bug.

A couple of indy films I've enjoyed are Haphead and Ghosts with Shit Jobs:

http://haphead.com

http://ghostswithshitjobs.com

Haphead is now free, and looking to do a second season. (It was released episodically, although I think watching it as a feature is better.)

"Which takes us right back to a gatekeeper-based world! Wasn't the Internet supposed to free us from that?"

Yes - now anyone can be a gatekeeper. We obviously need meta-gatekeepers to point us to the gatekeepers who point to stuff we might like. However, seriously, some clever AI from the usual suspects may well be the ultimate gatekeeper if it knows what we like well enough. Then we will get what is happening across the Net and in real life - cultural fragmentation and self-ghettoization, where the only things you read and people you talk to are ones you like. Let us celebrate diversity! [...and less than minimum wage for the drones, plus megabucks for the few]

Meta-gatekeepers... Hmm, that's a rather interesting idea...

As a rule of thumb, for books at least, stuff from publishers tends to be higher quality.

Those editors and so on do a good job. They curate for quality.

Unfortunately their biases tend to make them believe white men writer better than anyone else... and if you're a woman changing your name to a man's name can improve your acceptability remarkably, as we've seen in a previous post.

Sturgeon's Law has a 10% of not-crud. And this 10% is much, much larger than it was back when I knew most all TV shows and most all of the SF in the book stores. There is so much more good stuff now. But I gave up trying to see or read it all a half century ago. It's overwhelming.

The numbers are way off, but I like saying that if I had a fortune, instead of being able to do 1% of the things I want to do - I would be able to do 2% of the things I want to do.

Haven't read all of the above comments yet ....

Are your numbers of books published 'net' counts of a particular work/title or is this a number/count of all of the ISBNs released that year? By 'net' I mean all possible formats, sizes, languages, modalities, etc. for that one specific title.

Out of interest, I just Googled 'Terry Pratchett Color of Magic ISBN' and 44 different ISBNs showed up.

Sturgeon's Law has a 10% of not-crud.

That was with gate-keepers. I suspect that, with self-publishing, the crud factor is a lot higher.

The book totals are taken from a search on Amazon for new books in a year. They almost certainly include some noise in the form of duplicates - however, from a reasonably cursory look through the search listings, I'd estimate that noise to be around 5% - 9% of listings total.

If you're not an author yourself, is there an author you would recommend who can illuminate for others your personal experience(s) with being bipolar? What you've just written about your day-to-day life has actually been quite useful/insightful. So, thank you!

How are you (as someone who happens to be bipolar) handling the entertainment deluge, when in funds? (Or, has this also become 'work'?)

This is not a snarky question ... as society becomes more inclusive, we'll need to question some of our default benchmarks re: perceptions/abilities to make sense of the universe day-to-day including entertainment.

Another social comment: Most Western societies are increasingly multicultural and/or multilingual at least this generation, i.e., next 30 years. This means that there's probably a good sized 'foreign-language' market within each of London, NYC, Toronto, Sydney for many English-language authors right now.

That was with gate-keepers. I suspect that, with self-publishing, the crud factor is a lot higher.

That does not obviously follow.

Gatekeepers are a selection process, not an improvement process. (Editorial feedback can be an improvement process, but it's also expensive and doesn't usually happen in an intensive way. Even the "make this clearer" version doesn't necessarily happen these days.) As a selection process, it's substantially selecting for perceived sales; there's lots that's good and memorable and has a relatively small audience. Modern self-publishing mechanisms can get that content out there, and in whatever sense "quality" is meaningful, can act to increase the overall level of quality in the writing pool. (The backlist of the obscure and excellent and out of print emerging as ebooks has a similar effect.)

People self-publishing with an intent of craft (rather than some other motivation) have been exposed to a lot of existing work; they know, at least for themselves, where good is, and what their intent was. They've also got access to some kind of feedback mechanism. (Which can be remarkably good; your editor has constraints your first reader doesn't.)

So I think it's not obvious that the self-published are going to be statistically disadvantaged compared to the traditionally published in terms of quality, whatever that is. (As distinct from sales; they're totally disadvantaged in terms of sales.)

A lot of money for promotion will determine success and failure - just like now, but more so.

The real question is whether money spent on promotion will attract enough paying customers at high enough prices to yield a decent risk-adjusted return on the investment.

Surely it's the selection that limits (limited) the crud? The thousands of previously unpublishable novels that editors routinely declined everywhere now hit the virtual shelves at Amazon... often bolstered by fraudulent or at best dubious reviews. I don't think it unreasonable to assume that the majority of self-published stuff is not very good. I've read some good self-published stuff, but mainly from previously published authors... Most self-pub stuff I've read has been shite (though admittedly mostly when I've been lured out of my safe zone by 99p prices and suchlike, another skewing factor). Yes, publishers used to turn down good stuff. But they turned down more bad stuff than good...

Quoting Charlie from a previous thread:

manuscripts are screened by an intern and the obvious no-hopers -- somewhere around 90% of them -- are rejected on the spot. (Hint: there are a lot of schizophrenics with hypergraphia, and spamming publishers with their word salad is a common activity.) Then if the intern can read the first chapter without bursting into tears it maybe ends on an editor's desk, at which point the editor has to decide whether they like it enough to champion it, and then they have to contact the author and figure out if they can work with this person

Assuming Charlie knows what he's talking about, we've got a 90% immediate rejection rate, followed by some unknown rejection rate of the remainder before it hits an editor. Some unknown percentage gets rejected at the editor stage, some for things that affect quality (refusing copy-edits) and others for things that the reader wouldn't notice (being an asshole, not meeting deadlines). But it looks like the bulk of the rejections is for bad quality.

Self-publishing means that, instead of the publishers filtering out the "schizophrenics with hypergraphia", the purchasers have to do that. Which takes time and effort.

Are interns/editors wrong on occasion? Sure. But I suspect their success rate is pretty good when it comes to weeding out obvious crap.

(Note: I'm not talking about missing 'hits', I'm talking about not passing garbage. Some dreck makes it through, but how much gets filtered out? And how much is that filtering worth to someone with limited time?)

Given that my recreational time is more limited than the material available, I'm willing to miss the occasional 'hit' in exchange for not wasting time on scores of misses. Especially as there are already more 'hits' in my library than I have time to read/watch, even if I stop buying new things for a couple of years.

I can't resist a snarky comment: skim Hubbard's Battlefield Earth, by section. Let's see, Detective Stories, Air Adventure, Jungle Stories, Space Stories, Underwater Adventures.....

Are interns/editors wrong on occasion? Sure. But I suspect their success rate is pretty good when it comes to weeding out obvious crap.

That said, we have it demonstrated several posts ago that this process is biased. Which needs fixing.

But eliminating any selection process whatsoever isn't fixing the problem, really.

Assume men and women produce about the same output, in the same quality ratios. Bias (of whatever cause, at whatever level) means that publishers (dead tree, ebook, whatever) favour men, forcing more women to self-publish.

Now, the women that would have been published, if they'd been men, are as good as anything from a publisher. And some unknown number of men and women were good but were rejected, so their work is also good. But there's a huge amount of crap out there as well, which the purchaser has to wade through/filter somehow to find the good stuff. Statistically, a female self-published work is a bit more likely to be good than a male self-published work — but both are swimming in a sea of crap — Charlie's 'word salad'. Would an average purchaser notice this, or would the remember that the reliably good stuff (from publishers) was mostly male-written, perpetuating the problem?

So to sum up my rather rambling viewpoint, as a purchaser I want reliable gatekeepers of some kind — someone to keep the crap away and tell me enough about the good stuff that I know whether it is likely to appeal to me. I would like those gatekeepers to be unbiased. And I'm willing to accept that they will accidentally filter out some good stuff, in exchange for my not having to wade through crap to find what I want.

Which is why, happy as I am that people like Linda Nagata can self-publish good novels publishers have rejected, I would much rather the problem of bias in publishing was fixed, so they didn't have to.

Sampling ... anthologies

Many consumer products still use sampling to generate trial/purchase. Although online book retailers will allow one to peak into a few pages, that's not at all the same as finding out how the author handles the beginning, middle and end of a story.

Also, for learning about specialty markets, low incidence genres, new authors, I'd be willing to pick up an anthology. If it includes two or three well-known authors, then I won't feel I'm taking as large a risk with my time/money.

I'd also suggest that there's a sales aspect.

One frustrating thing may be that we have a cadre of skilled professionals who know how to sell books that meet a certain set of criteria. These include format, genre, and author, and unfortunately, the default author setting for many formats and genres is heteronormal white male. Equally unfortunately, there are some genres, often regarded as low ranking (e.g. romance and self help) where the default author format is heteronormal white female.

Now, I'm not saying it's fair that they specialize in selling these kinds of books. Perhaps the better question is what happens when they venture out of their comfort zones--how often are they rewarded with enormous sales (e.g. JK Rowling) or awards, and how often do they watch their treasured babies die abandoned by the marketplace, then get grilled by their bosses on the twelve steps they'll take so that they won't make that mistake again? I don't know the answer to any of this, but I'm still trying to my mind around how to make things work for all of us who don't quite fit.

Note that it's not just a problem of SFF and romance or even fiction. I'm still learning how to sell a non-fiction book, and it appears that what I wrote doesn't fall within the normal genre categories of non-fiction writing. Getting it sold is going to be interesting experience.

If it's a book on climate and/or ecology, do you have a place where you'll announce when it's available? I'm always up for more good books on those subjects…

Just checking in with jet lag, but ...

Minor nit: using Amazon scores to distinguish good from crap novels is a losing game. I get one-star reviews on Amazon from customers who think my books are overpriced. (Translation: "reviewer" is actually a whining cheapskate who hasn't actually read the book.) I get one-star reviews on Amazon from customers who thought they were buying a Purple Singing Dinosaur and found they'd bought a Melancholy Elephant instead. (Translation: not a review, more a case of an angry mystery shopper.) I get one-star reviews from matrons in Mississippi who think Scottish detectives swear too much, from folks who think I ought to paint nicer covers for my books, and from folks who think I'm mean to sad puppies.

None of these scores tell you anything about the merit of the work in question. There may be genuine one-star reviews among them, by insightful critics who've identified a yawning chasm of fail in my work that I remain unaware of because I've only been immersed in it for a year while I wrote it, but they're lost in the noise.

A better metric would be to track five-star reviews, or a weighting of five and four star reviews as a ratio against three-star and two-star reviews (which filters out most of the random noise). But again, folks take agin' works of fiction for the damndest reasons.

TLDR: We need better quantitative analysis of reviews. If only AMZN's system, which is probably the easiest and biggest repository to scrape, wasn't so crap ...

Others have pointed some numerical inaccuracies, but I'll say this. The swapper has sold 700k copies. That's huge for an indie game and it's way, way over median on Steam (median is 30k). That game can be considered to be a huge success and well known. It is not an unknown game by any measure (other that maybe you didn't know about it).

An unknown game in Steam doesn't have 4000 user reviews. It has less than one hundred.

I don't disagree with some of your conclusions, but the data and its interpretation is wrong by an order of magnitude in several areas. It's bad. But it's not THAT bad, I think...

I’ll tell you what I’m doing: I create not just books, videos and web sites, but entire cults and tribes to go with them, complete with ways of life, memes, even language. It’s not very lucrative so far, but it’s fun. A world of small tribes defined by lifestyle, ethos, mythos, memes, etc., with bards producing culture for the other members sounds pretty cool to me. Surely more natural for human beings, and more interesting than the vast industrial scale, top-down culture and limited channels we had in the 20th century. Just remember: in such a world, you won’t be able to prevent expressions of “incorrect” culture -- that’s part of letting a thousand flowers bloom; as in nature, many of those flowers will be toxic to you, and some tribes will be your natural enemies.

Agreed. Case in point - the newly-discovered authors that have most delighted me have come through this blog (as guest bloggers, or OGH recommendation); you being one of them. Another is the Transreal bookshop - Mike's recommendations are pretty good.

Gratuitous plug for Graydon - I greatly enjoyed both of your Commonweal books; pardon me if this is a bit OTT, but the world-building reminds me of a Fantasy equivalent to the Culture. The early days of a better world, indeed.

...got any more, or more coming? (Sorry) :)

The Web -- and even earlier, Usenet -- never had gatekeepers preventing schizophrenics with hypergraphia from disseminating their writing. Yet I've rarely had a hard time finding more interesting material to read online than I have hours in a day to read it, and I don't feel like I have been much burdened as a reader doing my own gatekeeping.

Now to contradict myself: the Web has always had gatekeepers, first in the form of people manually curating links to interesting content (early Yahoo, now-hilariously-retro paper books telling you what to visit online) and later in the form of search engines and recommendation engines. There are probably more bytes of machine-generated spam text online than there are human authored texts, even including the schizophrenics, but anti-spam heuristics work well enough that we don't spend most of the day manually rejecting the toxic slush. It seems to me like this minimal level of automated gatekeeping is Good Enough. Free volunteer labor (e.g. people recommending stuff in blog comments) fills most of the other holes left by the absence of professional human curators. I would echo Hugh Hancock's OP that there is more good entertainment out there than there is time to enjoy it all. It's a nice problem to have, for the entertained.

Hmm, very interesting - thanks.

To be honest, I was much less certain about The Swapper than the other examples I gave. I had a suspicion it might be rather better-known than I was giving it credit for.

I'll go look for a better example tomorrow, and check sales figures before I publish!

Just as a retro-spoiler:

Tom Jubert (writer of The Swapper) outlines in the link above to his talk a rather more dystopian vision of gatekeepers and content providers: one in which (Steam / Amazon / Google) battle it out to be the singular content provider for everything.

It's an interesting oppositional stance to OP's piece, worth a few mins if you're interested in computer games / Theory of Mind Lite talks (it's not fantastic though, so skip if time pressed).

~

SA for the day, since the word keeps coming up:

They used an automated speech-analysis program to correctly differentiate—with 100-percent accuracy—between at-risk young people who developed psychosis over a two-and-a-half year period and those who did not. The computer model also outperformed other advanced screening technologies, like biomarkers from neuroimaging and EEG recordings of brain activity.

Atlantic Aug 26th

Automated analysis of free speech predicts psychosis onset in high-risk youths [Paper]

~

Leaving aside questionable claims of 100% reliability (the probability test that humans always fail - 99% accuracy leads to large % of those identified as not actually being sick), the question is:

How comfortable are you with your gatekeepers running such software?

Not just Amazon algo's, but job applications (not having Facebook now a 'questionable' point to note) insurance companies and so on.

Even it were simply helpful (e.g. sends details of support groups, medical contact forms, insurance / health care providers etc), would it know yourself better than you as it trawled through the meta-sphere of data?

I, for one, am going to have a very interesting reading list indeed.

wiggles nose

~

Oh, regarding film / MPAA / Indy / Straight-to-DVD stuff. After slogging through the BFI's 252 page report for 2014 and other vastly boring market research data sets and so on, I boiled it down to:

No-body really knows the exact figures.

I suspect the ratio is not 9:1 for anything with a budget (seriously: like indy games, I suspect the buy-in factor scales and hits # attempts hard, even with miracles such as El Mariachi around), but, again, the general point stands.

I'd expect the ratio is more like 3:1 or 4:1, but that's purely a guesstimate based on crumbs across the internet - I'm all ears to direct links.

Minor correction: the T. Jubert talk was on Jul 31st, not Aug 31st, which would have required some explaining.

And, since no-one is playing, a more pertinent question: Content is not created in a vacuum; it is carefully moulded to a population statistic. (The dreaded 'focus group').

What if the gatekeepers were playing the numbers a little bit beyond the old "Hollywood accounting"? That old chestnut of "Rebellion is useful as a pressure valve, counter-culture is the future trend to be mined and exploited and you all exist within Capital".

Give me a EFT and the right data, the largest data set ever imagined, we could plot the future of entertainment for the next 50 years. The things RAND and cybernetics dreamed of, but real this time...

Quite the sales pitch.

And it was made in 2001-2, to the CIA. You'll probably want to think about which companies went live or public in 2004, and who did the investment portfolios.

Oh, Squirrel time!

Be Seeing You.

I've updated the article to feature a genuinely little-known game - http://store.steampowered.com/app/340360/?snr=1_7_7_230_150_10

(Protip if you're looking for games that fit my "awesome but invisible" criterion: having ANY reviews on Metacritic implies that your game's quite well known, or you've actively worked to make it so. Likewise, any game listed as "Overwhelmingly Popular" on Steam only gets that title based on volume of reviews, not just level of positivity - seems to be around 800 reviews minimum. However, the "Very Positive" level on Steam contains literally hundreds of games with rabidly enthusiastic players and almost no visibility.)

About the sub-sub genres: This has happened to me with music. Grooveshark (now defunct) and now Deezer allow me to find "similar artists". The site also builds a record of what I like and gives suggestions based on that. Now, I've always liked metal, but nowadays I listen to funeral doom pagan/folk metal. That's sub-sub-sub-genre. But I also dislike a certain style of vocals, so about 70% is discarded there. I also prefer melodic music over strictly ambient, so I'm only really listening to 10% of a sub-sub-sub-genre. And I love it.

That's extremely doubtful.

Coming in late (because jet lag) but you know about the HUGE problem with the iOS (and to a lesser extent, Google Play/Android) app store and me-too games? Things called "Angry Boids" or "Angery Birds" and suchlike with superficial design that copies a well-known property minutely, for about the first two levels (hah, more than two levels! That must be expensive to develop)? The idea is much the same as with those polite but misspelled emails I keep getting from the widow of the President of the Ivory Coast asking for help with her banking arrangements, only smaller scale: better to soak 100,000 mugs for $0.99 than 10 mugs for $10,000, it's much less likely that the police will come after you.

And on a less fraudulent note, I've visited colleges in the US that were running degree-level courses in Games Development. Every damn student on those courses is working on a dissertation project, and some of them will eventually see the light of day.

I can totally credit an insane number of games coming out across all platforms. Whether they're any good or not is another matter ...

Unfortunately many authors (Charlie included) are quite stingy with the recommendations...

Reasons I'm stingy with recommendations:

Upshot: authors write, they aren't necessarily good guides to what to read.

Surely it's the selection that limits (limited) the crud? The thousands of previously unpublishable novels that editors routinely declined everywhere now hit the virtual shelves at Amazon... often bolstered by fraudulent or at best dubious reviews.

I am informed that one factor behind the 99.8% rejection rate for slushpile submissions at one major publisher in the late 1990s was schizophrenia with associated hypergraphia. The invention of the word processor made it really easy for unfortunates with hypergraphia[*] to spam lots of publishers simultaneously, and they took full advantage of it; the first step in slushpile reading back then was to check whether the manuscript was written in English words with punctuation that made any kind of sense.

[*] Not all hypergraphia cases are schizophrenics. I've occasionally had fits of hypergraphia -- the first draft of "The Annihilation Score" emerged in 18 days flat: it was a scary experience -- but hypergraphia is a surprisingly common symptom of schizophrenia and as a general rule of thumb the results are not remotely publishable.

Yup. Always knew Herman Melville was nuts. Not so sure about Alexander Dumas, the automated speech analysis algorithm probably used an English translation of Count of Monte Cristo, maybe it was just the translator who was off the deep end.

If it's a book on climate and/or ecology, do you have a place where you'll announce when it's available? I'm always up for more good books on those subjects…

Watch my blog: https://heteromeles.wordpress.com/ .

Anyway, back to the discussion of material.

Speaking of recommendations, how do you feel about recommending source material, rather than other people's fiction?

Just as a general observation, but particularly regarding (2) and (5).

I don't just read your stuff, or even "stuff like your stuff". I do in fact also read biography, history, paranormal romance, space opera (romance) (that's a sub-genre), crime fiction (but not true crime), comedy...

Even worse; you won't find me reading many books for research purposes (I'm much too slow at non-fic; also probably mild undiagnosed ADHD) -- more a case of surfing the web widely, in which case you need to follow my twitter feed because that's where I spontaneously fart interesting URLs on an hourly basis.

Thanks. Bookmarked.

Nah, it's that Sturgeon's Law is fractal.

"unpublishable" doesn't mean bad.

It means some combination of "this won't sell" and "this won't ever sell". (Note Charlie's comments about editors looking for careers as much or more than books.)

Sometimes "this won't sell" because, well, those aren't sentences and not in a clever way.

Sometimes "this won't sell" because not that many people know what haruspicy means and it's being presented in the context of a perl module in a cheese factory. Very rarely stuff like that turns out to have a huge audience (LotR, Harry Potter...) but generally the audience just isn't large enough for commercial success. One consequence of self-publishing getting easy is that stuff is increasingly out there.

Sometimes "this won't sell" because not that many people know what haruspicy means and it's being presented in the context of a perl module in a cheese factory. Very rarely stuff like that turns out to have a huge audience (LotR, Harry Potter...) but generally the audience just isn't large enough for commercial success. One consequence of self-publishing getting easy is that stuff is increasingly out there.

Back in the day, I wrote this weird-ass novelette. The first mate I emailed it to said, "this is great, Charlie -- but you'll never sell it! There aren't enough slashdot readers out there to buy it."

Title was "Lobsters". One Hugo nom, one Nebula shortlisting, and five years later it was the opening chapter of "Accelerando".

Thing is ... that friend of mine was probably right, back in 1998: there wasn't a market for that kind of fiction, it was a bit too weirdly geeky. But by 2005, the market existed.

And if no market for work of a given type is known to exist, that work may reasonably be pigeon-holed as "uncommercial" by an editor.

I'm very glad you enjoyed them!

The Commonweal is finding more friends than I expected, which is inherently cheering.

I expect Commonweal #3 to be out end of March of 2016, should no ill thing arise. (It exists, but needs its copyedit and its cover and suchlike.)

I'm read what I think is all the published Culture novels, and I'm sure something's leaked through from somewhere, but what I was trying for in the Commonweal was the contrapositive of a traditional epic fantasy's social structures -- communitarian instead of authoritarian, egalitarian instead of aristocratic -- and then I had to figure out how it survived and how it got there and it took on something of a narrative life of its own.

(Err... Charlie, etiquette for questions of this character?)

Re: 'TLDR: We need better quantitative analysis of reviews. If only AMZN's system, which is probably the easiest and biggest repository to scrape, wasn't so crap ...'

Suggestion: Using a survey format, ask the reader/rater 2 or 3 multiple choice questions about the book they're rating, such as,characters (who), story line (what), and environment/locale (where)*. Toss out all responders who fail this test. Stuff that might be considered for rating the actual work/novel should probably be selected for better segmenting the reader so that you can point that reader toward other works/authors they might enjoy. (It's doubtful that the reader's ratings will impact what the author being rated is likely to write next, but might help a publisher/bookseller.)

• Potential QC (quality control) of such a book review survey ... consider having about 10 to 12 different (who/what/where/when) multiple choice questions in total from which you randomly present 2 or 3 to any one respondent/survey.

The statistical analysis could be any of a large range of types ... depending on 'n' and other stuff.

"unpublishable" doesn't mean bad.

True, but it sounds like the vast majority of manuscripts submitted are unpublishable because they are bad, not because they are minority tastes.

Which is a big problem for self-publishing authors (especially those without published careers): how do you connect with the small audience that will like your work, and how do they find you, when a large amount of the self-published material out there is bad?

I've self-published on iTunes: two free photography books. (And while I like to think that they aren't bad, they definitely wouldn't be handled by a regular publisher.) Total downloads, less than 500 in two years, and that with showing up on the front page of the iTunes book store when you search for "Mongolia". Doesn't really matter to me — I did it for fun — but the fact that people like me can self-publish as a hobby makes it harder for other people to make a living from writing, and much harder for readers to find their work without some type of gatekeeper or recommendation system.

Or we need Google to develop that agent/avatar thing they patented a few years back, so that it learns our reading tastes well enough (perhaps by tapping into our e-readers and watching how we read things) that it can skim the slush pile data and metadata and find the stuff we want to read...

Of course, at that point, we're toast, because if the machines can read our books better than we can, what purpose is there left for us?

Or we can, just perhaps, suggest that life sucks, that it helps to have obsessive-compulsive disorder if you want to be a professional artist these days (just as it helps if you want to be an academic scientist these days), and try to support the artists whose work we like without worrying too much about all the similar artists we never found.

Or maybe I'm just grumpy because it's a hot day. Hard to tell.

Thanks for replying, Charlie.

First, I'm expressing regret that you don't make more recommendations - I'm not complaining (really, I'm not). The point about your not reading stuff that's actually like what you write is certainly fair enough, and in fact I remember pre-ordering one novel based on a throwaway mention by you and actually never managing to get through the book.

On the other hand... Well, for one thing a recommendation from an author I like will often lead me to spend money, on the grounds that the recommendation is from someone I feel I 'know' and whose taste I am willing to trust. That's not so much the case with, say, something recommended in a Reddit thread. I may not be typical in that, but it does mean that author reccos make me buy books.

As it happens, about half of what I've read in the past year has been prompted by mentions by other authors... including guest bloggers here. You might be stingy with the recommendations, but you're generous with your blog space and that's led me to a few things I've liked (I've read quite a lot of Elizabeth Bear following her appearance here, for example, and she in turn has led me to other names).

It's about trust networks, maybe? Sure, authors aren't necessarily good guides as to what to read, but as a reader I feel a loyalty to the writers I like and I (imagine I) have some kind of insight into their attitudes, tastes, and so on based on what they have written (and also, in not a few cases, based on their blogging!). It's like recommendations from friends and family (which, as any marker can tell you, are orders of magnitude more influential than from any other source).

As I say, I'm not complaining about writers not making recommendations. Your reasons for not doing so are perfectly cromulent. It's merely a shame (for me, at least), in much the way that it's a shame, say, when a friend can't recommend a novel because they don't read fiction.

Robert Pryor has said pretty much what I was going to. (Except for the bit about Mongolian photography...)

I've started at least a couple of novels. Pretty sure they were not up to much, but every now and then I tell myself I should finish one or another of them off (see: sunk cost fallacy). I'm almost certain they'd get rejected by any publisher... but if they were finished, well, at least it wouldn't be wasted time because I could self-publish! At least I'd have a published novel! And I'd have added another to the crapola out there.

Self-publishing means never having to take no for an answer. And hey, maaaaybe I'd be one of those unexpected success stories! Long odds, but better than a lottery ticket a day.

(Err... Charlie, etiquette for questions of this character?)

Feel free to answer any and all; we're past the first hundred comments now. (And it's a series I like.)

[H]ow do you connect with the small audience that will like your work, and how do they find you, when a large amount of the self-published material out there is bad?

Much of the published work out there is bad, too; it will just sell (in the opinion of whoever bought it). So you either go for a statistical definition of "good" or you recognize that the problem isn't so much quality as selection.

Selection is about persistence; it doesn't really matter if it's getting copies into the future or gaining more audience than you lose, you have to keep doing it.

What we know about online content audience is that luck matters and it takes time. Anybody with a serious blog or webcomic audience spent years building it, generally at least a decade.

The other thing that matters is distinguishability; can I tell this stuff from other stuff? This is much the same as saying "can I -- is it possible to -- have a preference for this?" This works in complex ways; there are people in the world who really like NorAm domestic light beer because it's reliably what it is.

So, well, there's an audience. You don't know where it is, who it is, or how large it is.

Really good labels run into the problem that the places that keep the labels aren't really interested in solving the audience-matching problem. They're focused on reputation as a model, rather than implementing an effective matching mechanism. (That is, you should like this because so-and-so is a Good Author, rather than "out of this month's new books, these are the best matches for your recorded tastes and preferences".)

Stability of labels over the necessary decade-plus time scale is also doubtful. It would take some sort of distributed metadata protocol, and everybody who makes ereaders has reasons to be against it.

All of which leads me to think the only approach is persistence; get it out there, keep it out there, and recognize that in a context of making a living from your art what you're doing is buying lottery tickets. (More tickets can shift the odds in your favour, but you need a lot of luck to win.)

Much of the published work out there is bad, too; it will just sell (in the opinion of whoever bought it). So you either go for a statistical definition of "good" or you recognize that the problem isn't so much quality as selection.

I'll grant Kratman et al are counterexamples to 'published = good'.

OTOH, Even if I find his work objectionable, I'm willing to grant that it isn't word salad, that it has a certain technical competence. I've read way too many self-published authors that remind me painfully of high school English assignments. And I haven't encountered that in commercially published work. (Well, outside of a couple of small presses that I suspect are just front names for a circle of self-publishing authors*.)

I'm not talking about taste here. Dan Brown is a competent writer**, and Margaret Atwood a very good one indeed, but I don't like either of them for recreational reading. But neither of them are writing word salad. If they wrote something deliberately non-commercial and sent it blindly into the slushpile, it would make it past the intern and land on an editor's desk.

Maybe that's what I need: an automatic intern. Something to filter out the word salad, and the trilogies written by people who think paragraphs waste space, and all the other stuff like that. That's what I mean by "bad" and "crap" (at least in this context). Maybe I should have been writing "atrociously written" instead, but "bad" is shorter.

*Said suspicion because of the poor quality, on the assumption that a press can afford a decent editor/proofreader, freelance if not staff.

**Absolutely shitty researcher, though. And how the publisher let slip through the proofreading that "Islamic" is a language (among many other historical, geographical, and technical bloopers in the little bit of Angels and Demons I managed to stomach…)

Couldn't find any info re: the Google avatar/agent you mentioned so checked at Google central re: better ways of discovering authors/works that we/readers would enjoy. Below are items 7 and 10 of their 10-point philosophy. (I believe that Google does take suggestions.)

Excerpts:

Once we’d indexed ... our engineers turned their attention to information that was not as readily accessible. Sometimes it was just a matter of integrating new databases into search... Other efforts required a bit more creativity, like adding the ability to search news archives, patents, academic journals, billions of images and millions of books. And our researchers continue looking into ways to bring all the world’s information to people seeking answers.

10.Great just isn’t good enough.

... Even if you don’t know exactly what you’re looking for, finding an answer on the web is our problem, not yours. We try to anticipate needs not yet articulated by our global audience, and meet them with products and services that set new standards. ... Ultimately, our constant dissatisfaction with the way things are becomes the driving force behind everything we do.

Hope you're feeling better soon.

Regards, SFreader

A short story based on those putative Google avatars.

the places that keep the labels [are] ... focused on reputation as a model, rather than implementing an effective matching mechanism.

And then there's Amazon, which recommends hundreds of books that are good matches to the set of books I've read, as far as a quality-indifferent program can tell. They're frequently awful.

And then there's Amazon, which recommends hundreds of books that are good matches to the set of books I've read, as far as a quality-indifferent program can tell. They're frequently awful.

You know how Google's business model is "get more people using the internet" and Microsoft's business model is "if it's hard, we can charge more"?

In that sense, Amazon's business model is "we're the only way". They want to capture all the surplus; not so much disintermediation as transactional parasitism, and they compete strictly on price.

"Which takes us right back to a gatekeeper-based world! Wasn't the Internet supposed to free us from that?"

I don't like this claim. I know people mean "gatekeeper" here kind-of-metaphorically, but I think there's a danger of forgetting what the real gatekeepers were like.

When Charles Stross was 12, my guess is that he saw every single new SF or Fantasy film going. I certainly did that year. And that would mean "Every one on theatrical release or new on TV in the city he lived in". Because that was all that was going. Where I lived, that was 4 films.

The number of actual new SF/F films made was irrelevant. Because pre-VCR, the gatekeepers were the movie theatre programme directors, and the TV programme directors. If it didn't get past them then we didn't get to see it.

When I was 14 I'd ready every SF author in the Children's and Young-Adults section of my local library. Not every work (I didn't like some), but something by every author. By the time I was 17 I'd worked my way through most of those in the rest of the library too. And at the book-store in my city with the best F/SF collection (an entire bookcase! 8 shelves!) I'd read at least a chapter by every author going.

The amount being published was irrelevant. Because the local bookstore buyers, the library buyers: they were gatekeepers. If it didn't get past the gatekeepers, we didn't read it.

And more than that: old stuff was hard to get. The library might stock some of an author's back-catalogue (or not) but the bookshop almost certainly didn't have much and availability in 2nd hand shops was pretty random

Admittedly, those of you with access to decent specialty bookstores were better off. God, I loved those places when I was able to visit one. But only a small fraction of books sold that way.

Conclusions:

1) Yes, finding the good stuff among a mountain new dross is a hard problem. But don't pretend that a review list is a "gatekeeper". We remember gatekeepers.

2) If you step back and look at the big, long-term pictures, then I think the biggest change has been access to back-catalogues. If I find an author I like, I buy and read all their works. My kids discover Miyazaki, I watch all his movies (awesome!!!). So I read more, and watch more, than when there were gatekeepers - but less of it is new. And it doesn't need a deluge of new stuff and a dislike of the new to create that behaviour.

(apologies for the long rant)

In that sense, Amazon's business model is "we're the only way". They want to capture all the surplus; not so much disintermediation as transactional parasitism, and they compete strictly on price.

I've noticed that most mature/static market sectors split in a 90/10 (or maybe 85/15) ratio between a majority incumbent and a rival format -- e.g. Windows/Mac in the PC market, car/pick-up truck (in the US -- pick-ups are a rarity in the UK -- and anyway, SUVs/crossovers are gaining), Android/iOS in smartphones, and so on.

I've also noted that in many segments there's a cost/quality preference split, with the bulk of the market share going to low cost (and low margin) products, but the smaller segment going to a higher cost/quality product. Glaringly visible in smartphones where Apple's iPhone, with about 15% market share, takes about 60-80% of all the profit in the entire business.

This leads me to suspect that there should be a niche for a rival to Amazon that focusses on curation and quality rather than price, at least in the soft goods sectors (books, games, videos, music).

Trouble is, quality is highly subjective -- and how do you assess it? I think Amazon kneecapped themselves in the early days by going for a crude 1-5 star rating system; it's possible to refine your recommendations on Amazon, but the process is time-consuming, tedious, and not very accurate.

Sure, they might have put in a more complex system, but Amazon want to keep things simple.

In the early 1980s, a grand total of 40-45 new SF/F novels were published in the UK in any given year, plus maybe 20 or so foreign imports.

It was quite easy for someone like me to have read almost the entire canon of significant SF novels -- as long as they were published over here. I did eventually find a couple of second-hand bookshops that had rows of battered US paperbacks -- grey market imports -- but you certainly wouldn't see them sold as new in regular bookshops.

But the point is, up until the early 1980s I was pretty much on top of what was going on in the field (albeit too immature to understand a lot of the contexts).

By the early 00s, the UK market had expanded to something like 200-250 SF/F novels a year and it was no longer feasible to stay current with British SF/F, never mind the American field (which was a whole lot larger). Selective reading was then mandatory.

(Also, my reading speed decreased with age. I was probably averaging about 300 books/year from age 10 through age 18. But at age 24 I had a bad detached retina and -- following eye surgery -- read about 4 novels during the subsequent year; I came out of hospital and straight into an accelerated CS degree course and just didn't have time. These days I think I'm back up to 30-60 books/year, but my attention is split with the world wide web, which is a huge time sink.)

Thing is, I don't believe it's possible for anyone to stay current with the English language SF/F genres any more. With specific subsectors within it, yes: I can see someone staying on top of MilSF, or steampunk, or paranormal romance, by the skin of their teeth and by reading voraciously, 2-4 novels a week. But the field as a whole is just too big, even before you include the deluge of self-published work, much of which is shit but some of which is as good as or better than anything that finds a traditional publisher.

Yes, but why do they want to keep it simple?

Hint: Amazon owns Goodreads. Goodreads selects for committed readers. But GR also has a simplistic rating system, and there's no sign of Amazon trying to beef it up.

Not being telepathic, but having experience of working at the bottom end within amazon, they are keeping it simple for several reasons: 1) they have fingers in every pie, I was annoyed to find out they own part of librarything for instance; thus committed readers can use that for things.
2) a complex system that works takes time and money, and Amazon is all about the growth on a shoe string budget (Seriously, they kept the place I worked going but it was obvious that they were focused on minimising costs).
3) they don't want customers to think much about things, they just want them to click on the "buy this book" button, which in our day and age given the dumbed down media etc, is what they expect to happen.

This NYT article seems relevant:

http://www.nytimes.com/2015/08/23/magazine/the-creative-apocalypse-that-wasnt.html?smid=tw-nytimes&smtyp=cur&_r=2

Talking mostly about music, but includes some useful statistics. For example,

According to the O.E.S., songwriters and music directors saw their average income rise by nearly 60 percent since 1999. The census version of the story, which includes self-­employed musicians, is less stellar: In 2012, musical groups and artists reported only 25 percent more in revenue than they did in 2002, which is basically treading water when you factor in inflation. And yet collectively, the figures seem to suggest that music, the creative field that has been most threatened by technological change, has become more profitable in the post-­Napster era — not for the music industry, of course, but for musicians themselves.

And for writing,

The O.E.S. numbers show that writers and actors each saw their income increase by about 50 percent, well above the national average. According to the Association of American Publishers, total revenues in the fiction and nonfiction book industry were up 17 percent from 2008 to 2014, following the introduction of the Kindle in late 2007. . . . The numbers seem to suggest that the market for books may be evolving into two distinct systems. Critically successful works seem to be finding their audience more easily among indie-­bookstore shoppers, even as the mainstream market has been trending toward a winner-­takes-­all sweepstakes.

…After hitting a low in 2007, decimated not only by the Internet but also by the rise of big-box chains like Borders and Barnes & Noble, indie bookstores have been growing at a steady clip, with their number up 35 percent (from 1,651 in 2009 to 2,227 in 2015); by many reports, 2014 was their most financially successful year in recent memory. Indie bookstores account for only about 10 percent of overall book sales, but they have a vastly disproportionate impact on the sale of the creative midlist books that are so vital to the health of the culture.

And most relevant to Hugh, I thing,

The new environment may well select for artists who are particularly adept at inventing new career paths rather than single-­mindedly focusing on their craft.

Anyway, it's a good article. Well worth reading.

And another from the WSJ:

http://www.wsj.com/articles/the-rise-of-phone-reading-1439398395

All of which brings me back to the importance of The Blurb, and good keywords in it, for readers to be able to find the work and get a quick and hopefully accurate impression. I can see a future where misleading blurbs get authors mentally blacklisted by readers.

Yes, but why do they want to keep it simple?

How can you make money off a better rating system?

A better rating system nigh-certainly reduces total sales; people would buy fewer books it turns out they don't like. Individual tastes are fairly narrow and, if facilitated effectively, can get narrower.

Any truly effective rating system would have to build some equivalent of the "components" approach the graphics folks are getting to for compact storage of images, only for complex text semantics. You'd have to make it good enough to work for someone having a fondness for both McPhee and McKillip, to make it work at all. (That is, more effective than the incumbent reviews-and-samples.)

To make it work really well it would need to be time-aware (tastes change!) and you'd probably need user input for mood. Then you would need to make it extremely difficult to poison, which, well. Look at how much of Google's search effort goes to keeping people from gaming the results; any good rating system would need to be a defended commons. Amazon has no business case for defending a commons that will reduce their sales.

Building the thing at all is a big project involving actual cutting-edge research with no certainty of success. Amazon hates those.

Plus, they're the incumbent; incumbents hate change, because change means loss of market share. Their overall business model involves getting to a position where nobody else can get enough capital to compete with them. (Rather like the endgame we're seeing in phones; minimum ante in phones is somewhere north of 10 GUSD.)

This leads me to suspect that there should be a niche for a rival to Amazon that focusses on curation and quality rather than price, at least in the soft goods sectors (books, games, videos, music).

They would have to overcome the problem that bookshops and gaming shops have right now. People come and browse, get recommendations, then buy online because it's cheaper. (And bitch when the local shop goes under, because how will they know what to order now?)

I'd love to see it. But when you pay a premium for an iPhone right now, you get different hardware than if you opted for the cheaper Android phone. But the goods sold in your 'curation and quality' boutique will also be sold in the Amazon megastore*, so people can browse the curated boutique to get good recommendations, then flip to a different browser tab and save a few bucks at Amazon.

Unless I'm really missing what you're getting at, which is entirely possible.

*Unless you were thinking of an exclusive arrangement, which sounds chancy for the authors.

Which prompts me to say "You expect this market thingy to be rational?" in response to Charlie making the reaosnable point that there should be a niche rival to Amazon.

Future Shock - back in the 70s - suggested people would manage exploding media growth by subculture and I think that was a good prediction. I think a lot of great stuff exists and survives only in it's niche.

Then you have people who cross over multiple subcultures - both creators and fans - and the fans in particular are good vectors. So you have people deeply immersed in subcultures, some of whom are indiscriminate, but some of whom are hugely discriminating, and then as a casual follower you find a critical voice that you like.

In short, I think you just accept you aren't going to see/hear/read everything great, but if you can't find good stuff now - it's probably you.

As for the balance of new and old - the old stuff is also becoming more and more accessible. It would be interesting to measure the growth of available back catalog, including YouTube.

but you know about the HUGE problem with the iOS (and to a lesser extent, Google Play/Android) app store and me-too games? Things called "Angry Boids" or "Angery Birds"

Spotify is terrible for this.

My kids wanted to listen to "Rudolf the Red-nosed Reindeer". The home speaker network is connected to Spotify. Should be easy! But I wasn't home to say "Gene Autry". So they gave up, because finding an actual decent version on Spotify was too hard - they didn't want to listen to any of the dozens and dozens of covers by no-hopers.

To be fair, most cover bands don't think of themselves as no-hopers. But nonetheless, I'd pay more for a "Spotify super-premium" if they offered one that had less music choices.

I'm not sure if anyone's still reading this thread, but the things said here don't add up.

Richard Prior refers us to a NY time article that sends us to the OES of the BLA in the USA. That's the Occupational Employment Statistics of the Bureau of Labor Statistics.

It's written as a refutation of the claim from a decade ago that the interwebs would destroy creative jobs because piracy. What it shows is the the number of people employed doing creative stuff hasn't changed much. I don't quite get the BLS's categories system, and found their historic data hard to query, but that's the NYT article's claim about that data.

But how can it be that we're seeing a "tidal wave" of new works, and yet not seeing an increase in the number of people working in this sector?

Is this what economists would call a "productivity miracle" in which we get more output from less labour input? (less typesetters, editors and sound technicians needed, more writers and musicians producing their own stuff)?

Or is this the age of the creative amateur and part-timer?

Or what?