Is Linux now a slave to corporate masters?

Does it matter who pays the salaries of Linux kernel developers? If so, how much, and in what ways?

The threads fan out from Linux Kernel Development (April 2008) — a report by Greg Kroah-Hartman, Jonathan Corbet and Amanda McPherson. The report covers a lot of ground. Here are the subheads:

  • Development Model
  • Release Frequency
  • Change Rates
  • Source Size
  • Who is Doing the Work
  • Who is Sponsoring the Work
  • Why Companies Support Linux Development
  • Conclusion
  • Thanks
  • Resources.

Guess which one has been getting the most attention?

Let's start with Linux Grows Up and Gets a Job, by Tom Slee. Writes Tom,

One of the highlights: "over 70% of all kernel development is demonstrably done by developers who are being paid for their work". 14% is contributed by developers who are known to be unpaid and independent, and 13% by people who may or may not be paid (unknown), so the amount done by paid workers may be as high as 85%. The Linux kernel, then, is largely the product of professionals, not volunteers.

So Linux has become an economic joint venture of a set of companies, in the same way that Visa is an economic joint venture of a set of financial institutions. As the Linux Foundation report makes clear, the companies are participating for a diverse set of commercial reasons. Some want to make sure that Linux runs on their hardward. Others want to make sure that the basis of their distribution business is solid. And so on, and none of these companies could achieve their goals independently. In the same way, Visa provides services in many different locations around the world in different sizes and types of stores. Some banks need their service mainly in one country, some in another, but when they work together they all get to provide their services all around the world.

...the Linux Foundation report has made clear that open source has crossed its commercial Rubicon, and there is probably no going back.

Nick Carr runs with this in Open source as corporate joint venture:

A new report from the Linux Foundation reveals the extent to which the most famous and successful open source software project - the development of the Linux operating system - has shifted from being a volunteer effort to being a corporate initiative.

Nick goes on to quote Tom Slee's piece, and adds,

There's nothing particularly surprising in the shift from the volunteer to the corporate model - it tends to be what happens when lots of money enters the picture - but it does reveal that while Net-based "social production" efforts may be unprecedented in their scale and unusual in their technology-mediated structure, they are no more immune, or even resistant, to being incorporated into established market systems than any other type of labor that produces commercially valuable goods. The shift in Linux kernel development from unpaid to paid labor, from volunteers to employees, suggests that the Net doesn't necessarily weaken the hand of central management or repeal all the old truths about business organization.

In TechnologyOwl, Timothy Lee pushes back with The Open Source Model Is About Organization, Not Who Signs Your Paycheck. Addressing Nick specifically, Tim writes,

For starters, most of the people contributing to the kernel are professional programmers, and most professional programmers have jobs in the software industry. So it's totally unsurprising that most kernel contributors work for software companies.

But Carr's observation also misses the point in a deeper way. What makes the open source model unique isn't who (if anyone) signs the contributors' paychecks. Rather, what matters is the way open source projects are organized internally. In a traditional software project, there's a project manager who decides what features the product will have and allocates employees to work on various features. In contrast, there's nobody directing the overall development of the Linux kernel. Yes, Linus Torvalds and his lieutenants decide which patches will ultimately make it into the kernel, but the Red Hat, IBM, and Novell employees who work on the Linux kernel don't take their orders from them. They work on whatever they (and their respective clients) think is most important, and Torvalds's only authority is deciding whether the patches they submit are good enough to make it into the kernel. Carr suggests that the non-volunteer status of Linux contributors proves that the Internet "doesn't necessarily weaken the hand of central management," but that's precisely what the open source development model has done. There is no "central management" for the Linux kernel, and it would probably be a less successful project if there were.

In his Know It All column in CIO Insight, Ed Cone posts Decoding the professionalization of Linux, which is comprised mostly of quotage from Clay Shirky. Here it is:

What that kind of analysis is missing is that IBM is paying engineers to work on projects that IBM doesn't own, or solely direct. You pay these engineers -- but of all the relationships between senior management and line employees, the fact you are paying them is about the least important, institutionally. The idea that the minute you pay people to do something, you have the right to manage them and the right to completely take over that work for the benefit of the company -- that's not true.

IBM is not producing that code, IBM engineers are. IBM is paying those people because it's getting value out of them -- Linux creates value for the enterprise, it lowers our cost of managing software, it increases peoples' budgets for hardware and services -- but there's this crazy middle step where Linux is not now and cannot be owned or controlled by IBM. Linux is a brutal technical meritocracy, and there is no senior manager at IBM who can say, "I don't care what the kernel engineers think, I want this." They can't put it into the product without appealing to people who don't work for them. If they announced a strategic change in the kernel they would be laughed out of the room. They have given up the right to manage the projects they are paying for, and their competitors have immediate access to everything they do. It's not IBM's product.

There is a kind of perverse misreading of the change here to suggest that as long there are paid programmers working on the project, it's not developing in any way different from what's going on inside traditional organizations. It badly misunderstands how radical it is to have IBM and Novell effectively collaborating with no contractual agreement between them, and no right to expect that their programmers' work is going to be contributed to the kernel if people external to those organizations don't like it. And that's a huge change.

When people read those statistics, they think, If there's a salary, then all the other trappings of management must go along with it. Not only is that not true, it's actually blinds you to the fact that paying someone a salary without being able to direct their work is probably the biggest challenge to managerial culture within a business that one can imagine.

Now my own few cents worth.

First, if Tim and Clay are right, the language of the report needs some debugging. For example, this line:

What we see here is that a small number of companies are responsible for a large portion of the total changes to the kernel. But there is a "long tail" of companies which have made significant changes.

The operative noun there is companies, not engineers. Then there's this:

The list of companies participating in Linux kernel development includes many of the most successful technology firms in existence. None of these companies are supporting Linux development as an act of charity; in each case, these companies find that improving the kernel helps them to be more competitive in their markets.

While there is a difference between "improving the kernel" and Nick's "an economic joint venture", the stretch isn't a big one. In fact, it's one I'd expect quite a few people to make.

Second, the essential role of Linux in the growing world of utility computing — notably search and big back-end Web services such as Amazon's S3 and EC2 — is right up the alley of Nick's new book The Big Switch: Rewiring the World, from Edison to Google. There Nick describes an emerging networked computing future dominated by a few large companies providing computing and related services as pure utilities. Microsoft's Live Mesh, announced a few days ago, appears to be another one of these.

Third, in all the conversations I've had over the years with kernel developers, none has ever copped to obeying commands from corporate overlords to bias kernel development in favor of the company's own commercial ambitions. In fact, I've only heard stories to the contrary.

This is from my Geek Cruise report in November 2005, where I'm reporting a conversation with Andrew Morton:

Andrew went out of his way to make clear, without irony, that the symbiosis between large vendors and the Linux kernel puts no commercial pressure on the kernel whatsoever. Each symbiote has its own responsibilities. To illustrate, he gave the case of one large company application.

The (application) team don't want to implement (something) until it's available in the kernel. One of the reasons I'd be reluctant to implement it in the kernel is that they haven't demonstrated that it's a significant benefit to serious applications. They haven't done the work to demonstrate that it will benefit applications. They're saying "We're not going to do the work if it's not in the kernel". And I'm saying "I want to see it will benefit the kernel if we put it in".

He adds, "On the kernel team we are concerned about the long-term viability and integrity of the code base. We're reluctant to put stuff in for specific reasons where a commercial company might do that." He says there is an "organic process" involved in vendor participation in the kernel. Earlier this year I had a conversation with IBM's Dan Frye in which he said the same thing, and that it had taken IBM a number of years to lean how to adapt to the kernel development process, rather than vice versa. Andrew explains,

Look for example at the IBM engineers that do work on the kernel. They understand (how it works) now. They are no longer IBM engineers that work on the kernel. They're kernel developers that work for IBM. My theory here is that if IBM management came up to one of the kernel developers and said "Look, we need to do that", the IBM engineer would not say, "Oh, the kernel team won't accept that". He'd say "WE won't accept that". Because now they get it. Now they understand the overarching concern we have for the coherency and longevity of the code base.

Given that now these companies have been at it sufficiently long, they understand what our concerns are about the kernel code base. If IBM need a particular feature, they can get down and put it in the kernel. Just as they would for AIX. There are some constraints about how they do that, however, and they understand that.

But it has to be good for the kernel. And good for supporting, as Andrew puts it, "serious applications".

For those not involved in that process, "Good for the kernel" is a hard concept to grasp. In fact, I'm not sure I would have grasped it myself it I hadn't spent a week on a boat getting schooled by Andrew, Ted Ts'o and a bunch of other kernel developers.

In that same piece, I suggested that Linux development resembles that of a species, rather than of a commercial project:

Kernel development is not about Moore's Law. It's about natural selection, which is reactive, not proactive. Every patch to the kernel is adaptive, responding to changes in the environment, as well as to internal imperatives toward general improvements on what the species is and does.

We might look at each patch, each new kernel version, even the smallest incremental ones, as a generation slightly better equipped for the world than its predecessors. Look at each patch submission -- or each demand from a vendor that the kernel adapt to suit their needs in some way -- as input from the environment to which the kernel might adapt.

We might look at the growth of Linux as that of a successful species that does a good job of adapting, thanks to a reproductive cycle that shames fruit flies. Operating systems, like other digital life forms, reproduce exuberantly. One cp command or ctrl-d and you've got a copy, ready to go -- often into an environment where the species might be improved some more, patch by patch. As the population of the species grows, and more patches come in, the kernel adapts and improves.

These adaptations are more often reactive than proactive. This is even (or perhaps especially) true for changes that large companies such as IBM and HP, which might like to see proactive changes made to the kernel, to better support their commercial applications.

Responding on his blog, Greg called that "what I think is one of the most insitful descriptions about what the Linux kernel really is".

Still, questions about corporate influence on kernel development have been raised. Such as, How do companies influence Linux kernel development, beyond paying developers?

Will the answers expose the kernel as a "corporate initiative?". I doubt it, but I'm not the one who needs to be convinced.

______________________

Doc Searls is Senior Editor of Linux Journal

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Up to 85% are paid -- is this a bad thing?

Michael Harris's picture

I can tell by the tone of comments that my comment is going to be inflammatory. I'm not trying to troll -- I'm an outsider who is really trying to understand. So let me represent my questions / point of view so that I can better understand where you are coming from.

I assume that the open source community desires high quality professionals contributing to its software (not hobbyists programmers). If so, let's assume that these professionals were only contributing on their own time. This would imply that these folks would work all day programming and would come home and spend the evenings and weekends programming open source. For me this does not sound sustainable from a personal health perspective. I'm sure that some folks will be able to keep up the pace, but it seems to me that many (most?) will burn out after a while. Furthermore, I would hope for their sake that life would intrude on their priorities at times. Vacations, family obligations, weddings, the birth of children, the kids' soccer games, etcetera.

I am sure that there are many specific cases of long term contributors at high levels, but overall I would expect that it is difficult to keep up intense levels of contribution over an extended time frame. The result of this would be a pretty heavy churn in the development team. Furthermore, given the organic nature of volunteer skills, there is no guarantee that as a communications specialist leaves a new communications specialist will arrive to take their place.

From my position, the entry of paid contributors is a necessary and useful step. These people have a salary incentive for working on the software. Their development time is not something that fights for the limited personal time left over after work and sleep are subtracted. If, say, IBM employs a TCP/IP specialist and that person leaves, then IBM will do a search to ensure that the position is filled with someone of comparable skills.

As a corporate buyer, I would feel better about purchasing software with paid contributors. I would feel more comfortable about the ongoing viability of such a model, and I would worry about the potential for a purely volunteer (spare time) ecosystem to sustain itself. When the paying entities represent a broad range of companies, I would feel comfortable about the sustainability even if one organization lost interest.

Finally, I would not be so concerned about the 'influence' of the corporations paying the wages. As another commenter noted, yes corporations do act in their self-interest, but that self-interest is not evil. Anyone can be evil, whether they are a corporate decision maker or an independent individual. If I was a stockholder of a contributing company, I would hope that dollars were spent with the intention of making the open-source software more useful to my customers. I'm not sure how efforts to make software more useful harms the software core.

I do understand that some corporate development may not be useful to the core. For example, some of IBM's development might enable corresponding features in DB2. Clearly this code would not be useful to the core. Clearly, then some sponsored development will be core orientated and other will be less useful to the general population.

The bottom line is that I think that economic incentives and support systems are not automatically bad things.

Linux and corporatization

Anonymous's picture

I just came to this discussion and find it quite compelling, to say the least. I have two comments.

First, I think it would benefit everyone who does not remember, to study the history of Unix when it "went corporate," but in a different way, all during the 1980s. This was when Unix had already become a phenomenon, corporations bought source licenses from AT&T, hired C programmers, and had a field day rewriting Unix into various and incompatible *nices, all locked down under closed copyrights. The result was the near destruction of what had been a thriving development community not hitherto driven by commercial considerations (even to the point that Bell Labs' own lawyers could not fully take control). Peter Salus's "The History of Unix" is a must read to understand the rise and fall of commercial AT&T Unix. I honestly believe that Linus, by his simple act of releasing a home-brew Unix (Linux 0.01) into the wild (Internet), saved the Unix social software development model and took it to new heights and to new generations of developers today. This is something that no corporation, no matter how big or powerful, could either start or stop. That big corporations have gotten onto the Linux bandwagon means, to me, that they have learned a big lesson about the value of community.

Second, my larger problem is not so much about who is writing Linux, but who is using it, and for what. I think most people who worked on early precommercial Unix, and developed Linux into what it became, let us say, up until that point in time in which the bulk of the code came from programmers working in corporations, typically have a highly developed social sense. They have long railed against all kinds of tyrannies, from stupid corporate masters making unhealthy social decisions (Microsoft! Windows! Word! MSN!) to "fascist" sysadmins who seem to derive pleasure from repressing hapless local users and honest but curious programming people, to the universal drive toward invasive and threatening personal tracking tools such as RF chips. It is precisely this vibrant and rebellious mentality that gave rise to Unix in the first place. (Remember: prior to Unix, what did we have? OS-360. Enough said.)

Unix, and now Linux, exist precisely as a sort of anarcho-artistic reaction against the ongoing repression of peaceful people by powerful institutional interests determined to reap profits at any cost. Think of the Spanish Civil War without thinking of the Lincoln Brigades, or of Guernica--the city or the painting by Picasso. It's just impossible. Well, Linux is our Lincoln Brigade, but this time, we may have won. Maybe. (I note with a certain dark pleasure that Linux has been accused of being a "communist" phenomenon more than once--just as Franco's propaganda said of the Spanish Republicans.)

So back to the question of what Linux is being used for. If Hitler had had Linux, his Stukas, his V-1s and V-2s, and his jet fighters could all have been far more lethal than the noisy propaganda weapons that they mostly were. Guernica would have been taken out by a single hit. If Franco had had Linux, would he have used cruise missiles against Barcelona? You bet. You can see where I'm going with this...

My concern is that the Linux development community has given a free and powerful weapon to all sides in present and future economic, political, and military conflicts, and now seem to be showing a certain indifference to the outcome. If you give an F-16 to a peasant farming family of ten, and another to the military forces who are attacking them, they are not equal in their ability to use or defend against such a weapon. By making such a gift, you are coming down solidly on the side of repression. (This too has a historical precedent in the Non-Intervention Pact of August 2, 1936, which helped Franco much more than it helped the Republicans, while providing the Allied powers a fig leaf of neutrality.) I don't think that's where most Linux guiders and developers want to be, and I would ask them to think it over carefully now that Linux is being deployed as embedded software in a vast variety of military weaponry.

What would you have us do?

Doc Searls's picture

You make a number of well-put points.

Now, however, the horses of code are out of the proverbial barn. Are you suggesting that we all stop writing code that's useful to anybody, and make it useful only to peace-loving peoples?

Can you clarify?

Thanks.

Doc Searls is Senior Editor of Linux Journal

Stop writing code? No, hell

Anonymous's picture

Stop writing code? No, hell no. That would only put us back under the tender mercies of M$.

Make it useful only to peace-loving peoples? No. Well, that would be nice, but of course our legal systems will probably not permit it.

My purpose in writing was to get or keep the conversation going. It is a social problem, not a code problem, and no single software developer can really do anything about it acting alone. But a group of programmers who are thinking along the lines I am could certainly get together and issue some sort of manifesto...

More from Tom Slee

Doc Searls's picture

Didn't want Tom's follow-up post fall through the cracks.

Doc Searls is Senior Editor of Linux Journal

Article: Linux controlled by corporate

W. Anderson's picture

The article lists many references to corporations providing paid development to the Linux "kernel", but lists the effort as the Linux" operating System" (OS). Since the kernel is completely worthless by itself in an OS framework, there should be more discussion or report on how this kernel development effort has or will effect the rest of the GNU/Linux OS.

Personally I have numerous suspicions about the true motivations and intent of the Linux Foundation - if one goes by ridiculous and stupid statements of it's president, since he acts totally as a lackey for corporate interest to the detriment and against interests of GNU/Linux developers and general users. I hope I am wrong, but I doubt it.

Without covering this perspective, the article has little value.

W. Anderson
wanderson@nac.net

Which statements?

Doc Searls's picture

W,

You say, if one goes by ridiculous and stupid statements of it's president, since he acts totally as a lackey for corporate interests....

So, what statements? Can you point to some?

Thanks,

Doc

Doc Searls is Senior Editor of Linux Journal

Should we suppose that corporate influence is a bad thing?

Thomas G's picture

I detect a tone of presumed dislike for corporations that underlies much of this discourse. Sure, there are corporations who seem to be bad actors that make decisions that are disadvantageous to certain user populations. But I think this is an aspect of particular corporate cultures specifically and not a feature of corporations in general. The corporate economic model has really been an engine for the development of computer technology in its present form. Do you believe computing would have advanced so far and become so cheap and broadly accessible if there had not been corporate, economic motives driving its advancement in the first place? If it had remained purely in the domain of academic institutions and hobbyists, I don't think anyone can reasonably say we would be where we are now.

I would just like to frame the discussion in a way that does not assume a priori that the corporate system is evil and driven by greed. A corporation is, like a person, driven by self interest. In that sense, it does not have any particular moral importance apart from its specific actions. Are there corporations I dislike? Sure. But I am glad we live in a society where a group of like-minded individuals can start a corporation to pursue their shared economic interests. So, in that sense I find myself in agreement with many of the quotations that observe that "corporate" or "professional" involvement in Linux development is not a bad thing, and probably, on balance a good thing.

corporate masters? Yes

Anonymous's picture

Nick Carr has it right.

"The shift in Linux kernel development from unpaid to paid labor, from volunteers to employees, suggests that the Net doesn't necessarily weaken the hand of central management or repeal all the old truths about business organization."

The question is though, does

Anonymous's picture

The question is though, does this matter? Corporate development increases the speed Linux adapts to new hardware and increases support for existing components.

True, IBM aren't going to give a rats ass about Nvidia graphics cards but, just how many ThinkPad's are out in the wild? Intel provide Intel support... how many Intel based machines are in the wild?

It's a good thing. All those (C) statements in dmesg is what we need to increase Linux penetration.

Isn't it a good thing that

Clair's picture

Isn't it a good thing that there are companies who are making sure that there are developers paid for their efforts in kernel development? But it doesn't totally mean that every Linux distribution project is corporate-funded all the time. It happened that the kernel development is becoming like that but it's probably because of practical considerations that it became like that.

In some ways, I really do think that there are decisions that companies make sometimes so that they could improve their market share, the support they give to their hardware's users, etc. etc and in the end things work out.

some complain that...

Tomasz Chmielewski's picture

Some complain that the centre of gravity of Linux kernel development is placed in the server world, and not in the desktop world...

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState