EOF - The Limits of Scale

Linux is like limestone; you can build anything with it. So, while you find limestone in everything from lipstick to pyramids, you find Linux in everything from picture frames to Google.

What brings this analogy to mind is the matter of scale, long regarded as a virtue in the tech world. Getting to scale and staying there are both considered Good Things. But, as with other Good Things, is it possible to have too much? At what point do the biggest things we make with Linux risk turning into pyramids—that is, durable landmarks that are also dead?

These questions came up for me back in January, when two things happened. One was Larry Page replacing Eric Schmidt as Google's CEO. The other was mysterious account deletions at Flickr. Without Linux, there would be no Google or Flickr.

In Google's case, I saw the writing on the wall at the Techonomy conference in Lake Tahoe, August 2010. On stage was Eric Schmidt, amid four other panelists. In the Q&A, Eric said, “If we look at enough of your messaging and your location, and use artificial intelligence, we can predict where you are going to go....Show us 14 photos of yourself and we can identify who you are.” He added:

(You can follow along at wn.com/Eric_Schmidt_at_Techonomy, starting at 21:10. The first question is mine.)

I wanted to freeze time and say “Eric, no! Stop, big guy! Better to say nothing than this kind of stuff!” But I just sat and winced. Two months later in an interview with The Atlantic at the Washington Ideas Forum, Eric said, “We don't need you to type at all. We know where you are. We know where you've been. We can more or less know what you're thinking about.” Spoken like an eyeball on a pyramid.

At this point, it was just a matter of time before one of the founders would return, Steve Jobs-like (and hopefully not Jerry Yang-like) to bring the company back in alignment with Original Principles. That happened in January, followed quickly by a Bloomberg Businessweek cover story titled “Larry Page's Google 3.0”. Said the writers, “The unstated goal is to save the search giant from the ossification that can paralyze large corporations. It won't be easy, because Google is a tech conglomerate, an assemblage of parts that sometimes work at cross-purposes.” The piece goes on to profile a half-dozen “star deputies”. Of them, it says, “Together, their mandate is to help the company move more quickly and effectively—to keep it from becoming yet another once-dominant tech company that sees its mantle of innovation stolen away by upstarts.” Good luck with that.

Flickr's first pyramid moment was a report that photographer Deepa Praveen had her entire Pro account (the kind people pay for) deleted without explanation. The story broke first in Thomas Hawk's blog, and then the action moved to my own blog, with a post titled “What if Flickr fails?” That one racked up 107 comments, including a pair from Yahoo executives. (Flickr belongs to Yahoo.) Nowhere was there anything to relieve fears that an account deletion might come at any time, to anybody, with no chance of recovering whatever was lost. (My own exposure is about 50,000 photos.)

Then Mirco Wilhelm, another Flickr Pro photographer, had his 3,400 photos deleted, in what Flickr eventually admitted was its own error. These were later restored, with much apologizing by Flickr. Still, one had to wonder how much of the problem had to do with Flickr's size. According to the most recent reports at this writing, Flickr hosts more than 5,000,000,000 photos for 51,000,000 registered users, with new photos arriving at more than 3,000 per minute.

One of the best talks on Linux deployment was one given by Cal Henderson at the March 2006 O'Reilly Emerging Technology Conference. It was an all-day tutorial about “launching and scaling new Web services”. I remember being highly impressed at how well Linux allowed a fast-growing pile of digital goods to expand, while still providing near-instantaneous service to everybody who wanted it. I also remember wondering what would happen after Cal left—which he did in 2009.

The answer is workarounds and startups. Here are a few examples, just from the comments that followed my Flickr post: unhosted.org, couchapp.org, www.tonido.com, backupify.com, gallery.menalto.com, pix.am, status.net, thinkupapp.com, piwigo.org, www.zoofoo.com and https://pixi.me, in that order. None yet compete with Flickr, but maybe that's not the idea.

Nature's idea is to take its course. It's as much Linux's nature to start something as it is to grow to the limits of viability. It may help to remember that limestone is made from the corpses of once-living things. Without abundant endings, we wouldn't have beginnings.