The Just-So Story I learned for Cthulhu's creation is that one day Lovecraft ate some seafood and had a bad allergic reaction.
Since, between Cthulhu and the neo-monarchist crowd, we're moving into "evil god-king" territory, it occurs to me that "I Have No Mouth and I Must Scream" is pretty close to an evil singularity story, with godlike computer power over the physical world instead of natural computer control over an artificial reality.
Perhaps evil singularity stories don't flourish because in a computer simulation there really is no escape: in "I Have No Mouth," people can still be physically destroyed; in the "worldwide total communism" SF stories from the same era, people still occupy a physical reality with physical limitations. Stick authoritarianism in a computer and the oppression is inescapable. You have to turn to transcendent mysticism (The Matrix) to let your protagonists do anything meaningful.
]]>But that's not fundamentally very interesting."
I don't know how you can make sense of Lovecraft without racism. It's like sweeping all that "longing for a semi-legendary British countryside" stuff under the rug when discussing Tolkien--whether or not you find it "interesting," it's absolutely fundamental to getting what he's on about.
To try to answer your question, if you want to find the Lovecraftian singulitarian, try digging around the same sorts of misanthropic and (by modern standards) far-right-wing philososphical groups that descend from Lovecraft's own views of the world: Neo-Straussians, human biodiversity forums, the self-professed intellectual strains of the Christian identity movement. That's where you look to find the combination of cosmic awe and seething dread that fueled Lovecraft's work. The only writer I've read who might qualify is John C Wright, and his views are so distasteful I have little interest in exploring his ideas further.
Lovecraft was not like you, Mr Stross. Don't look for his modern version among the rarefied atmosphere of disappointed British intellectuals. Look for dread, loathing, and disgust.
]]>Rather than gradually abandoning intelligent agency as a causal explanation, I could imagine our little tapeworm scientists moving from a natural explanation of the world, to an "ecosystem"-based model, to the final, paradigm-shattering revelation that they are inside an intelligent creature that is one among many intelligent creatures.
I bet Harlan Ellison already wrote this story.
]]>