EOF - The Limits of Scale

Maybe what's wrong with Too Big is what's right with starting over.

Linux is like limestone; you can build anything with it. So, while you find limestone in everything from lipstick to pyramids, you find Linux in everything from picture frames to Google.

What brings this analogy to mind is the matter of scale, long regarded as a virtue in the tech world. Getting to scale and staying there are both considered Good Things. But, as with other Good Things, is it possible to have too much? At what point do the biggest things we make with Linux risk turning into pyramids—that is, durable landmarks that are also dead?

These questions came up for me back in January, when two things happened. One was Larry Page replacing Eric Schmidt as Google's CEO. The other was mysterious account deletions at Flickr. Without Linux, there would be no Google or Flickr.

In Google's case, I saw the writing on the wall at the Techonomy conference in Lake Tahoe, August 2010. On stage was Eric Schmidt, amid four other panelists. In the Q&A, Eric said, “If we look at enough of your messaging and your location, and use artificial intelligence, we can predict where you are going to go....Show us 14 photos of yourself and we can identify who you are.” He added:

I would make a stronger point—that the only way to meet this set of challenges that we are facing is by much greater transparency and no anonymity. And the reason is that in a world of asymmetric threats, true anonymity is too dangerous....One of the errors that the Internet made a long time ago is that there was not an accurate and non-revocable identity management service....You need a name service for humans.....governments are going to require it at some point.

(You can follow along at wn.com/Eric_Schmidt_at_Techonomy, starting at 21:10. The first question is mine.)

I wanted to freeze time and say “Eric, no! Stop, big guy! Better to say nothing than this kind of stuff!” But I just sat and winced. Two months later in an interview with The Atlantic at the Washington Ideas Forum, Eric said, “We don't need you to type at all. We know where you are. We know where you've been. We can more or less know what you're thinking about.” Spoken like an eyeball on a pyramid.

At this point, it was just a matter of time before one of the founders would return, Steve Jobs-like (and hopefully not Jerry Yang-like) to bring the company back in alignment with Original Principles. That happened in January, followed quickly by a Bloomberg Businessweek cover story titled “Larry Page's Google 3.0”. Said the writers, “The unstated goal is to save the search giant from the ossification that can paralyze large corporations. It won't be easy, because Google is a tech conglomerate, an assemblage of parts that sometimes work at cross-purposes.” The piece goes on to profile a half-dozen “star deputies”. Of them, it says, “Together, their mandate is to help the company move more quickly and effectively—to keep it from becoming yet another once-dominant tech company that sees its mantle of innovation stolen away by upstarts.” Good luck with that.

Flickr's first pyramid moment was a report that photographer Deepa Praveen had her entire Pro account (the kind people pay for) deleted without explanation. The story broke first in Thomas Hawk's blog, and then the action moved to my own blog, with a post titled “What if Flickr fails?” That one racked up 107 comments, including a pair from Yahoo executives. (Flickr belongs to Yahoo.) Nowhere was there anything to relieve fears that an account deletion might come at any time, to anybody, with no chance of recovering whatever was lost. (My own exposure is about 50,000 photos.)

Then Mirco Wilhelm, another Flickr Pro photographer, had his 3,400 photos deleted, in what Flickr eventually admitted was its own error. These were later restored, with much apologizing by Flickr. Still, one had to wonder how much of the problem had to do with Flickr's size. According to the most recent reports at this writing, Flickr hosts more than 5,000,000,000 photos for 51,000,000 registered users, with new photos arriving at more than 3,000 per minute.

One of the best talks on Linux deployment was one given by Cal Henderson at the March 2006 O'Reilly Emerging Technology Conference. It was an all-day tutorial about “launching and scaling new Web services”. I remember being highly impressed at how well Linux allowed a fast-growing pile of digital goods to expand, while still providing near-instantaneous service to everybody who wanted it. I also remember wondering what would happen after Cal left—which he did in 2009.

The answer is workarounds and startups. Here are a few examples, just from the comments that followed my Flickr post: unhosted.org, couchapp.org, www.tonido.com, backupify.com, gallery.menalto.com, pix.am, status.net, thinkupapp.com, piwigo.org, www.zoofoo.com and https://pixi.me, in that order. None yet compete with Flickr, but maybe that's not the idea.

Nature's idea is to take its course. It's as much Linux's nature to start something as it is to grow to the limits of viability. It may help to remember that limestone is made from the corpses of once-living things. Without abundant endings, we wouldn't have beginnings.

Doc Searls is Senior Editor of Linux Journal. He is also a fellow with the Berkman Center for Internet and Society at Harvard University and the Center for Information Technology and Society at UC Santa Barbara.

______________________

Doc Searls is Senior Editor of Linux Journal

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Grid work

Darrel's picture

Doing large scale grid work, we regularlly press various limits of Linux and other systems. If you're in a situation where you're pushing various limits Meat Slicer like open file descriptors and network sockets, here is how to ensure that the limits are large enough.

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState