« Hiatus ... | Main | How I got here in the end, part eleven: the music stops »

How I got here in the end, part ten: head-first into the Singularity

We human beings are not used to rapid change.

Let me qualify that remark. We may think we're used to rapid change, but for the most part we never see it, up close and personal. We're easily overwhelmed by events if they don't progress in a linear manner. As they spin out of control we find ourselves watching, feeling as if we're moving in slow motion — it's a commonplace in accidents such as car crashes, and what it betrays is a cognitive malfunction as we spend precious moments observing rather than acting.

Some time in late 1998, DataCash, the start-up I was working for, began to go exponential. This is what happened ...

In late 1997, we started out with a ropey demo and about five customers. A grand total of something like 40 credit-card transactions went through the system in November '97 to January '98.

By March-April 98, if I remember correctly, we had a buggy production system, which had been rushed into service far too soon ... but which could deal with two banks rather than one, and supported a bunch of whizzy extra features (whenever I got time to implement them). Features like the ability to handle multiple merchants, each with multiple EPOS terminal IDs, and in some cases trading in currencies other than Sterling. One early roadblock was the discovery that the banks' payment services expected each EPOS terminal ID to be in use for only one transaction at a time; concurrency reared its ugly head very early, because what we were doing was processing transactions on behalf of client software somewhere on the web. We dealt with the concurrency headache by bolting on an extra layer of Virtual terminal IDs — a pool of terminal IDs would be managed by a scheduler — to hide the concurrency problem from the merchants. And we added transaction velocity checking, to prevent end-users mashing their thumbs repeatedly on the "Submit" button and pounding on their credit card. Along the way, we moved from using an ISDN dialup to talk to the banks (basically behaving like a very fast modem dialup) to an X.25 box on BT's commercial backbone; my server, running on a dedicated machine, would open socket connections to the PAD (as many as necessary) and throw transactions at the banks over the dedicated line.

Some time in this period I wrote what was basically a limited-functionality clone of expect in Perl. And then I wrote a server framework in Perl, NetServer::Generic, to make it easier to write pre-forked multiheaded server processes. (NetServer::Generic is on CPAN because I talked Dave, my boss, into agreeing that it wasn't part of our core business, but we could do with having lots of free beta testers to whack on it for bugs. It worked well; I still get badgered by the occasional bug report to this day, even though I stopped maintaining it in 2001.) And I wrote a whole bunch of other stuff; much of it over-hasty, or buggy, or both, because I was being ridden hard by feature creep from one end and maintenance headaches from the other. I was busy: the old truism (first expressed by Tim Berners-Lee, I think) that five years pass on the web for every year in the real world was creeping up on me.

(I gather that Datacash retired the last of the code I'd worked on some time in 2008. As I left the company in 2000, I contemplate this news with mixed emotions. I'm a better author of fiction than I ever was a programmer; and software is notoriously short-lived. I suppose the fact that any of my code survived my departure by more than a month, and that the company kept it running for another few years, speaks for itself.)

By the summer of 1997, we'd begun acquiring new customers. If we had five in December 1997, we had maybe fifty in December 1998. We were still losing money — charging £0.5 per transaction, you need a lot of transactions to pay for two small offices and four salaries and a couple of collocated servers. And I was still a contractor, paid by the day. But things were hotting up. And then they went nuclear.

1998 was the year that the dot-com bubble began to inflate for real. And it nearly killed us. At one point in September, I realized that our transaction throughput was growing at a compound 30% per month, and it kept that up for six months before it began to slacken off. Some time in early to mid 1998, we handled 100 transactions in a month. By the same time in 1999, we had many individual customers who put that volume of credit card transactions through in a day. By late 1999, we were looking at transactions hitting our servers every few seconds; around the time I left, in early 2000, it was looking like credit card transactions per second were the way to measure throughput. Whatever his other faults, Gavin was an excellent salesman. But at the back end, it kept me on my toes. The more your software gets worked, the more likely obscure bugs are to rise to the surface. Sam was having headaches too; his web-based reports were growing like crazy, and the number of customers who needed them was growing too.

I was stressed out for most of two years. I'm an alpha-type personality to begin with, but this wasn't funny. I was writing fiction (and articles for Computer Shopper) as a therapeutic distraction. Around the middle of 1998, I figured that the novel I'd written in 1995-96 Singularity Sky was about ready, and mailed it off in the direction of Tor in New York, where it sat on a certain editorial director's desk for the next eighteen months. I wrote and sold a couple of short stories, and began work on a project which I was workshopping with some other local writers; a strange humorous horror novel/spy thriller about a hapless geek who's fallen into a government department for dealing with ... look, you probably know where this is going, right?. This was strictly a weekend activity, to distract me from the weekday stress cycle: compartmentalising my life helped me deal with Datacash. But it probably didn't help enough.

For most of the end of 1998 and the first half of 1999, the uttermost bane of my life was an ecommerce subsidiary of Bank Paribas called KLELine. They were offering a credit card solution over SSL, which had certain attractions for some of our customers, being (a) French, and (b) able to do some funky and useful things, or so they said. The trouble from my point of view was ... well, they weren't terribly clear on open source, for starters, or on public APIs, which was somewhat more serious. And when I got in deeper, I discovered some horrifying shortcuts in their API. Like, oh, once a credit card transaction hit their servers they'd process it, but the acknowledgement might well disappear into the bit bucket if the poor-quality leased line between London and Paris chose that particular moment to crap out. And the exchange rate for the transaction in question would be pulled out of a hat in accordance with the phase of the moon or something, and a subsequent refund or cancellation request wouldn't go through at the same exchange rate if there was a currency fluctuation.

We were in early-to-mid 1999. We had about four banks up and running, and another three or four in the pipeline to hook up and get ourselves certified by. A couple more programmers had arrived, all (except for Sam and I) being smokers: I was coughing my lungs out by an open window, but still juggling the entire server development and debugging workload. Dave and Gavin had figured out that the one-room model wasn't working; later that summer we were going to move to a new purpose-built office suite with room for 20-40 people, in a refurbished 18th century warehouse in Leith. KLELine was giving me sleepless nights; insomnia, depression, stress. I'd written a wrapper for their broken DLL that worked, most of the time, and took appropriate defensive action when things went pear-shaped, but I had no confidence in them. And Dave's management style was to treat me as an interrupt-driven resource. "Shoot at the monkey's feet; make the monkey dance," as one of my colleagues put it. When you've got one overstressed programmer trying to hold together the mission-critical system the whole company runs on (a company which is now up to about twelve people and mushrooming rapidly) that's not a terribly smart way to manage them — but we were all learning on our feet: it was Dave's first management job, and he was learning by making a lot of mistakes.

One particularly bad Wednesday I came into work, did stuff until I redlined, then sat at my desk for an hour. I got up and went over to Dave's desk. "I'm taking tomorrow and Friday off work," I announced.

"But what about —" Dave looked up and saw my face. "No problem," he said hastily.

I emailed my partner, then booked return tickets to Amsterdam and a weekend in our favourite hotel there. (Amsterdam is cheaper and easier to get to than London, from Edinburgh, not to mention more fun.) We flew out, checked in, and I lay awake until 4am mentally drafting and redrafting my resignation email (for delivery on Monday at 9am). This being the pre-wifi dark ages, I had neither the means nor the opportunity to send it; so we spent Friday morning wandering around Amsterdam in the rain.

Then my mobile rang. I looked at the screen: it was Dave. "Yes?" I asked.

"It's about KLELine," he said. A pause. "They're being shut down by Paribas!" I heard a tinny sound of applause through the phone earpiece. "So you can cross them off your to-do list ..."

It was the applause — from the rest of the office — that convinced me not to send the resignation email. Instead, Feorag and I headed for a pub, where we proceeded to get drunk. And I managed to sleep solidly that night for the first time in a month.

The pub we got drunk in, incidentally, is the Cafe in de Wildeman. And the concentrated sense of future shock I'd been dealing with in my job for the past year — of the pace of change that has accelerated until it's all a whirling blur that you can barely keep track of — came out in the fiction. That weekend I began doodling the intro to a short story on my Psion: "Manfred's on the road again, making strangers rich ..." It was the beginning of the story "Lobsters", which in turn grew into the novel Accelerando, and kept me on the Hugo award shortlists for most of the first half of the past decade.

Then on Sunday we flew home, and it was shortly thereafter that Dave finally twisted my arm into signing on as a full-time employee rather than a contractor; and at about the same time, Jeremy Spencer, then editor of Computer Shopper, offered me the monthly Linux column.

(To be continued ...)




a " isn't closed in the Singularity Sky link

(oh, and I'm enjoying this serie very mych indeed)


Just a nit, but BNP and Paribas were still two separate and unrelated banks in 1999.


Further nitpick: Wildeman with one n.


I assume that Charlie refers to 'BNP Paribas' to avoid confusion with another BNP. . .


And then there is an "or" too much in:

" or on or public APIs "

Keep 'em coming. I really enjoy reading this.


"I'd much written a wrapper[..]"?

While we're picking nits....


As I work in the transaction processing software industry, this hits a lot of buttons. As you were describing the Paribas API issues and such, part of me was trying to analyze the problem and come up with a solution.

I've never experienced the thrill of firing a customer or watching them be dragged under by their own incompetence (and out of the area of my concerns), but I often fantasize of such moments. Sometimes, the stress isn't worth the marginal increase to revenues they represent.

Looking forward to when you have the time and motivation to finish the story.


O/T, but I just had a flashback to "Missile Gap" :)



And of course, Charlie has a book up for Hugo this year, and if you have a membership and haven't voted for the Hugos yet, do so now online because Friday is the last day for it.



Really enjoying this.


Dave @4: Too bad the same typo occurs in the print version of Accellerando at least once on page 9.


I went through some similar exponential waves when the POS transaction market was coming up in the first place. First year I worked for VeriFone, they had $250K revenues. Second year, $4M revenues or 1500% annual growth. Third year, much slower - under $16M revenues, a mere 300% annual growth.

I'm surprised the UK Internet POS market didn't hit until 1997-1998. In 1994, before I left VeriFone to work full-time for my own ISP, they were already hawking various insanely cobbled-together solutions for Internet transactions, and hawking the company itself as Internet payment experts. I think it was one of the main reasons HP bought them (and then had no idea what to do with the actual business.)


Jeez. Just reading this is making me feel dizzy. Did you happen to write Toast in the midst of this? A lot of its future-burn angst seems to parallel what you're writing here.


While reading this my inner voice was chanting "One of us! One of us!". My like experience was with Livingston Enterprises in 95-98, selling access servers and routers to ISPs who were riding the boom. I left shortly after Lucent acquired us. Many of my friends had very similar experiences, and a number of them burned out and/or cashed out and got out of tech to pursue other careers. I've thought about it, but frankly I don't know what I'd do - I'm a geek in my personal life too, tech is where my passion lies.


Clifton: it wasn't the POS market hitting late in the UK, it was the e-commerce market -- POS was well established by 1995, but this new-fangled internet stuff made the bankers' heads hurt, which probably delayed it a couple of years. To which, add: if you wanted to service the UK market you didn't have the relatively simple job of hooking up to Visa and Mastercard (and optionally to Amex); to coer 70% of the market you had to talk to at least five different banks, each of which acted as agents for Visa and Mastercard.

James: the stories in "Toast" were mostly written circa 1988-1998. The intro was written in 2000. So yes, it bracketed my time in the dot-com bubble.

MegaZone: yup. I had the big advantage of having known from an early age that I wanted to write SF; all else was just interesting life experience along the way.


Really good stuff. Just reading this is giving me flashbacks to my last job. The whole

TypeOf(Customer) == GodObject

thing is very familiar. It's a weird feeling when your number one source of stress and work for a long period of time (6 months or more) evaporates suddenly, as if it had never mattered.


Charlie: Yeah, I meant the e-commerce market. There were at least some kind of solutions available in the US by '95 or so, though I think they were at least as bogus as your initial demo. Your explanation as to why probably covers it.


Some smart publisher should realize this biography series has all the makings of a good book.

"Lobsters" was my first Stross experience and I've been a huge fan ever since so it's awesome to read about the circumstances around it's creation.


Charlie I have been reading along and waiting impatiently for the next installment and I have to say they are brilliant, you have the material here for a non-fiction book for certain!

I am the same age as you and my passion for tech in the early days was similar to yours, but my path into the military put screeching halt on going down that rabbit hole. I still am an amateur tech nut, but my skills have atrified beyond repair at this point, so being able to experience it from your perspective has been a very entertaining trip!


Megazone: Oh hey, Livingston! There's a name I haven't heard in a while. I remember the dizzying chain of acquisitions around then: Ascend ate Cascade, Lucent ate Livingston, and then Lucent ate Ascend for dessert. At LavaNet we were mostly buying Ascend Max stuff but I knew a lot of ISP guys who swore by the Livingston Portmasters. And almost everyone was running a million differently-hacked up copies of the Livingston RADIUS server code...

I think it was in 1996 when we were in the process of buying a Cascade 200-port dialup server box costing something like $100K, one of their lead engineers had flown out to help install it, and then the day before we were to do the acceptance test Ascend closed on Cascade and instantly killed the product. (I was watching the engineer while he was on the phone to headquarters; he literally turned white when he got the news.) We weren't going to take title and pay for an already EOLed product, and they didn't want to pay to ship it back, so we had $100K of someone else's hardware sitting around in dead storage for years, until we finally figured it was safe to dispose of it.

Hard to say if that was good times or bad times, but they were certainly interesting.


This isn't really related to this strand but Charlie has been referenced in this book by Carlos Solari: http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470745754.html

Chapter 3 begins:

"In his futuristic cyber crime novel 'The Halting State', Charles stross takes his readers on a trip through Second Life in the year 2019 (not coincidentally) and describes and environment where the events in a 'grid crime' can cross over and inspire crimes in the real world that have real consequences and global impact. Although a purely fictional account, Stross' crime novel, nevertheless, provides an inkling of how web 2.0 might evolve and shape a world in which virtual tools redraw the relationship between humans and machines and narrowing distances between the two and heightening the interaction between them across all the senses. The novel also gives the reader pause for thought by demonstrating what can happen when reliance on technology that was considered a source of power suddenly turns into a vulnerability in the wake of a system failure. The source of the failure is a compromised encryption key that has been implemented to protect all commercial and government functions. The repercussions are far-reaching and absolute. No border is sacred and no business or government agency is spared."


I'm greatly enjoying these, thanks for sharing them with us!


You are going to have to WRITE FASTER, Charlie...
Here's yet another HALTING STATE moment ....


@21 Clifton: Livingston was a damn fun ride. I joined them in August 1995 when they were around 50 people, including manufacturing, etc. I left in February 1998, a little while after they were acquired by Lucent, and they were 250 with multiple offices, including one in Europe. I learned a hell of a lot working there, and it is one of my favorite jobs - I waver on Livingston or Sling Media being my favorite ever.

I also learned some hard lessons about my own limits while working there. After too many 100+ hour weeks, often with 72 hour 'shifts' (with quick runs home for a shower and change), one day I was standing in the corner of a meeting and found myself oddly detached, watching myself slide down the wall, unable to stand. I'd been ignoring a mild fever, just popping pills to keep running, and my body decided that it had had quite enough, thank you, and went on strike. I ended up out of work for a week, most of which was spent in bed. And it took a while longer to feel quite right again, I really pushed things too far that time. I've gotten close to that a few times since, but after that I knew to recognize the signs before I hit the wall at full speed.

Unfortunately things changed when Lucent moved in, and when they acquired Ascend the Ascend folks ended up in charge of the division and they quickly EOL'd the PortMaster products in favor of their own. Which was a damn shame, the PortMaster-4 was a hell of a beast which was never fully realized. And there was a replacement for the PM-3 in the works with 4 T1/E1 ports which would've whipped the MAX6000 like the PM-3 did to the MAX4000. But such is life.

My boss at Livingston, Brian Rice, was awesome. He took the chance on hiring me, a 24 year old one year out of college, and moving me across country. He'd been watching me on USENet groups during my WPI days, and then saw that I'd landed at Xylogics - a Livingston competitor, they made the Annex line - and he wanted me on his team. Brian was pure Californian, an ex-DJ who'd gotten into tech a while back. After the buy out he cashed out and went off to start a book store and was getting back into radio - but unfortunately he died of heart failure before he had the chance. Damn shame, of anyone he really deserved to enjoy the fruits of his labor. He was a real stand-up guy.

Things go in circles - this week my current employer asked me to look into using some old PM-2s as remote serial consoles for our core Cisco gear, for emergency remote access. I need to wake up some long-dormant neurons.