EOF - The Limits of Scale
Linux is like limestone; you can build anything with it. So, while you find limestone in everything from lipstick to pyramids, you find Linux in everything from picture frames to Google.
What brings this analogy to mind is the matter of scale, long regarded as a virtue in the tech world. Getting to scale and staying there are both considered Good Things. But, as with other Good Things, is it possible to have too much? At what point do the biggest things we make with Linux risk turning into pyramids—that is, durable landmarks that are also dead?
These questions came up for me back in January, when two things happened. One was Larry Page replacing Eric Schmidt as Google's CEO. The other was mysterious account deletions at Flickr. Without Linux, there would be no Google or Flickr.
In Google's case, I saw the writing on the wall at the Techonomy conference in Lake Tahoe, August 2010. On stage was Eric Schmidt, amid four other panelists. In the Q&A, Eric said, “If we look at enough of your messaging and your location, and use artificial intelligence, we can predict where you are going to go....Show us 14 photos of yourself and we can identify who you are.” He added:
I would make a stronger point—that the only way to meet this set of challenges that we are facing is by much greater transparency and no anonymity. And the reason is that in a world of asymmetric threats, true anonymity is too dangerous....One of the errors that the Internet made a long time ago is that there was not an accurate and non-revocable identity management service....You need a name service for humans.....governments are going to require it at some point.
(You can follow along at wn.com/Eric_Schmidt_at_Techonomy, starting at 21:10. The first question is mine.)
I wanted to freeze time and say “Eric, no! Stop, big guy! Better to say nothing than this kind of stuff!” But I just sat and winced. Two months later in an interview with The Atlantic at the Washington Ideas Forum, Eric said, “We don't need you to type at all. We know where you are. We know where you've been. We can more or less know what you're thinking about.” Spoken like an eyeball on a pyramid.
At this point, it was just a matter of time before one of the founders would return, Steve Jobs-like (and hopefully not Jerry Yang-like) to bring the company back in alignment with Original Principles. That happened in January, followed quickly by a Bloomberg Businessweek cover story titled “Larry Page's Google 3.0”. Said the writers, “The unstated goal is to save the search giant from the ossification that can paralyze large corporations. It won't be easy, because Google is a tech conglomerate, an assemblage of parts that sometimes work at cross-purposes.” The piece goes on to profile a half-dozen “star deputies”. Of them, it says, “Together, their mandate is to help the company move more quickly and effectively—to keep it from becoming yet another once-dominant tech company that sees its mantle of innovation stolen away by upstarts.” Good luck with that.
Flickr's first pyramid moment was a report that photographer Deepa Praveen had her entire Pro account (the kind people pay for) deleted without explanation. The story broke first in Thomas Hawk's blog, and then the action moved to my own blog, with a post titled “What if Flickr fails?” That one racked up 107 comments, including a pair from Yahoo executives. (Flickr belongs to Yahoo.) Nowhere was there anything to relieve fears that an account deletion might come at any time, to anybody, with no chance of recovering whatever was lost. (My own exposure is about 50,000 photos.)
Then Mirco Wilhelm, another Flickr Pro photographer, had his 3,400 photos deleted, in what Flickr eventually admitted was its own error. These were later restored, with much apologizing by Flickr. Still, one had to wonder how much of the problem had to do with Flickr's size. According to the most recent reports at this writing, Flickr hosts more than 5,000,000,000 photos for 51,000,000 registered users, with new photos arriving at more than 3,000 per minute.
One of the best talks on Linux deployment was one given by Cal Henderson at the March 2006 O'Reilly Emerging Technology Conference. It was an all-day tutorial about “launching and scaling new Web services”. I remember being highly impressed at how well Linux allowed a fast-growing pile of digital goods to expand, while still providing near-instantaneous service to everybody who wanted it. I also remember wondering what would happen after Cal left—which he did in 2009.
The answer is workarounds and startups. Here are a few examples, just from the comments that followed my Flickr post: unhosted.org, couchapp.org, www.tonido.com, backupify.com, gallery.menalto.com, pix.am, status.net, thinkupapp.com, piwigo.org, www.zoofoo.com and https://pixi.me, in that order. None yet compete with Flickr, but maybe that's not the idea.
Nature's idea is to take its course. It's as much Linux's nature to start something as it is to grow to the limits of viability. It may help to remember that limestone is made from the corpses of once-living things. Without abundant endings, we wouldn't have beginnings.
Doc Searls is Senior Editor of Linux Journal. He is also a fellow with the Berkman Center for Internet and Society at Harvard University and the Center for Information Technology and Society at UC Santa Barbara.
Doc Searls is Senior Editor of Linux Journal
|Designing Electronics with Linux||May 22, 2013|
|Dynamic DNS—an Object Lesson in Problem Solving||May 21, 2013|
|Using Salt Stack and Vagrant for Drupal Development||May 20, 2013|
|Making Linux and Android Get Along (It's Not as Hard as It Sounds)||May 16, 2013|
|Drupal Is a Framework: Why Everyone Needs to Understand This||May 15, 2013|
|Home, My Backup Data Center||May 13, 2013|
- RSS Feeds
- Dynamic DNS—an Object Lesson in Problem Solving
- Making Linux and Android Get Along (It's Not as Hard as It Sounds)
- Designing Electronics with Linux
- Using Salt Stack and Vagrant for Drupal Development
- New Products
- A Topic for Discussion - Open Source Feature-Richness?
- Drupal Is a Framework: Why Everyone Needs to Understand This
- Validate an E-Mail Address with PHP, the Right Way
- What's the tweeting protocol?
- Kernel Problem
4 hours 55 min ago
- BASH script to log IPs on public web server
9 hours 22 min ago
12 hours 58 min ago
- Reply to comment | Linux Journal
13 hours 30 min ago
- All the articles you talked
15 hours 54 min ago
- All the articles you talked
15 hours 57 min ago
- All the articles you talked
15 hours 58 min ago
20 hours 23 min ago
- Keeping track of IP address
22 hours 14 min ago
- Roll your own dynamic dns
1 day 3 hours ago
Enter to Win an Adafruit Pi Cobbler Breakout Kit for Raspberry Pi
It's Raspberry Pi month at Linux Journal. Each week in May, Adafruit will be giving away a Pi-related prize to a lucky, randomly drawn LJ reader. Winners will be announced weekly.
Fill out the fields below to enter to win this week's prize-- a Pi Cobbler Breakout Kit for Raspberry Pi.
Congratulations to our winners so far:
- 5-8-13, Pi Starter Pack: Jack Davis
- 5-15-13, Pi Model B 512MB RAM: Patrick Dunn
- 5-21-13, Prototyping Pi Plate Kit: Philip Kirby
- Next winner announced on 5-27-13!
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?