Could Microsoft's Photosynth Have Been Free Software?

Photosynth is one of the most exciting programs I've seen in a long time. It takes a group of photos, typically of a single geographical location, but possibly taken at different times by different people, analyses them for similarities, and then stitches then together into a smooth-flowing, pseudo-3D panorama. It's really great. Just two problems. One: it won't run on GNU/Linux; and two: it's from Microsoft, and so is unlikely ever to do so.

My question is this: Why didn't the free software community come up with Photosynth first?

Ten years ago, or maybe even five, that would have been a ridiculous question for most people. Free software was widely regarded as many things – wacky, cool, generous, unusual, fun – but hardly innovative. Indeed, one of the commonest accusations against open source was that it was “chasing tail-lights” as Microsoft's Jim Allchin famously put it in the first Halloween Document. Today, that is no longer a tenable position.

Just look at the contrasting receptions accorded to Microsoft Vista and Ubuntu's Hardy Heron. Where the former has been a rather pathetic damp squib, with people actively avoiding it in favour of XP, the mounting excitement in the days leading up to the launch of Hardy was almost palpable (even if there were a few grumbles afterwards about disappointed expectations). Quite simply, GNU/Linux (and the Apple Mac) are now where people expect new and exciting things.

It could be argued that for all its popularity, Ubuntu is hardly innovative. But the same could not be said about Firefox. It may have caught on initially by offering what was effectively a better Internet Explorer – faster and more secure – but with the arrival of version 3, it is clear that Firefox is now setting the pace. That includes obvious elements like the “AwesomeBar” - which, despite my extreme scepticism in the face of this immodest name, I have indeed found to be truly awesome - as well as a host of more subtle improvements in terms of speed and memory management. Moreover, the Firefox team are already aiming to do better thanks to things like TraceMonkey:

an evolution of Firefox's SpiderMonkey JavaScript engine for Firefox 3.1 that uses a new kind of Just-In-Time (JIT) compiler to boost JS performance by an order of magnitude or more.

If you want another example of where open source is innovating in ways closed source can only dream about, I offer Apache for your perusal. What began as a quick hack around some code that was being neglected has turned into not only the backbone of the public Web – despite Microsoft's desperate efforts to knock it off its perch – but also an entire system for nurturing and managing open source projects. This latter aspect, crystallised in the Apache Software Foundation, may ultimately prove even more important that the original Apache code, which one day will doubtless be superseded. Coming up with a lightweight model of governance for software - the “Apache Way” - is something that will live on.

Still not convinced? Take a look at Blender. Most people know this as a really neat 3D drawing program, but recently it has moved far beyond that. It has been used to create high-quality open source animations – all of the data files are freely available - and is currently being applied to develop an open source game. In other words, Blender is no longer simply a program, but an entire software ecosystem of interlocking capabilities.

Against this background of continuing and accelerating innovation, my question is therefore why Photosynth came from Microsoft, and whether it could have been open source instead. To answer that question, it's helpful to understand how exactly Photosynth was created. Here is Microsoft's description of what happened:

In 2006 Microsoft acquired small, Seattle-area startup Seadragon, whose technology is capable of delivering a buttery smooth experience browsing massive quantities of visual information over the Internet. It is all the detail you want, exactly when you want it, with predictable performance regardless of the amount of data—from megapixels to gigapixels.

The same year, from the groundbreaking research of Noah Snavely (UW), Steve Seitz (UW), and Richard Szeliski (Microsoft Research), a prototype called ‘photo tourism’ was born. The idea was simple: given a few dozen or few hundred photos of a place, is there enough information to reconstruct a 3D model of that place? The advanced computer vision techniques pioneered in pursuit of this goal form the basis of the synther.

Together these incredible tools are the foundation that makes Photosynth work. The synther requires large amounts of visual data to generate its 3D environments, and Seadragon technology makes it possible.

Seeing the promise in the product, Microsoft Live Labs built a small startup team to incubate the Photosynth project. Collaborating with teams around Microsoft, including Virtual Earth, Microsoft Research, Windows Live, and others, they have been hard at work making Photosynth more than just a prototype, creating an experience that anyone can enjoy and where anyone can create something amazing...

From this it's clear that Photosynth is the result of Microsoft having lots of money to buy young companies with innovative technologies, invest in advanced, “pure” research, and then put together an internal startup. So how can free software hope to compete?

The innovative technologies part shouldn't be a problem. Free software is about trying stuff out, following hunches and generally having fun with code: all of these feed directly into the creation of new and exciting software.

The “pure”, speculative research side is more tricky – not because such “blue-sky” work doesn't happen outside deep-pocketed companies like Microsoft, but because increasingly there is a pressure on academics in universities to make money from their discoveries. This is part of a larger problem – the commercialisation of science – that adversely affects free software in other ways through things like patents.

One way around this is for companies that proclaim themselves to be friendly to open source – IBM, Sun etc. - to make more of their research freely available to the open source world. In doing so, they would be recognising that it may be in their long-term interests not to commercialise such research in the short term, but to allow the community to build on it. Another tack would be for academics to stand up for themselves – and their research – by refusing to acquiesce in the mindless exploitation of their work for short-term financial rewards without due consideration being given to alternatives, like working with the free software community to produce longer-term and broader social benefits (easier said than done, admittedly.)

The final phase noted by Microsoft – the creation of internal startups – is perhaps the most problematic, because of the inherently fragmented, decentralised nature of the open source world. Until recently, there have been no natural homes for such teams trying to turn new ideas into new tools. But once more, I think that Firefox has shown the way. Mozila has managed to create a structure that allows development work to be supported just like in traditional software companies, but without losing all of the extra benefits of being based on open code supported by an open community. It's not hard to imagine a more ambitious Mozilla creating small startup teams in the future to drive the kind of innovative work that has led to Photosynth.

Whether within Mozilla, the Apache Software Foundation, or some entirely new entity, perhaps funded by open source companies and supporters specifically for this purpose, the free software world needs to find a way to move beyond its present, undeniable achievements, and start to match and surpass Microsoft in what is perhaps the last area where the Redmond behemoth can plausibly claim to be ahead. If it doesn't, then open source will always be chasing those tail-lights.

Glyn Moody writes about open source at opendotdotdot.


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Open Source Alternative version 0.2 (Pixelstruct)

gnubyexample's picture

There is an open source project now to attempt to do this sort of 3D meshing of images which can be found here:
Pixelstruct on Freshmeat
Pixelstruct home
Pixelstruct 0.2 tar.gz

The link includes a video demonstration using pictures of Notre Dame

In order to enhance this early release, and seriously compete with the well funded venture microsoft is sponsoring, the community will have to figure out how best to support what looks like a promising project.

Just a further comment on breeder communities for open source projects:

"Until recently, there have been no natural homes for such teams trying to turn new ideas into new tools."

Just another possibility to give it some awareness:

Launchpad ( ) is a software collaboration platform that provides:

* Bug tracking
* Code hosting using Bazaar
* Code reviews
* Ubuntu package building and hosting
* Translations
* Mailing lists
* Answer tracking and FAQs
* Specification tracking

Launchpad host 14,527 projects including the MySQL branch project Maria

It's not competing with photosynth

JEIhrig's picture

Make3D is not competing with photosynth. It's research that allows depth estimation from a single image. Using multiple images is the easiest way to estimate depth from images. It is MUCH more challenging to get 3D depth from a single image. Carnege Melon uses vanishing points to determine depth from a single still image. But this relies on the image contaning man made objects such as buildings.

Make3D can generalize based on a trained algorithm using machine learning to extract depth from even unstructured data such as trees. The same algorithm was used to allow a remote controlled car to navigate an unstructured environment.(Only assumptions for Make3D are: ground is flat, sky is up)

Photosynth is much more mature than Make3D, but I believe that an algorithm that uses less data to achieve results will have the higher potential in the long run.

OOps, that was supposed to

JEIhrig's picture

OOps, that was supposed to attach as a reply to another comment.

The original project has released code

Albert the Moose's picture

I've not had (and won't have) time to play, but it's located at:


David Roberts's picture

I've also created an opensource tool for visualizing the output produced by Bundler. The homepage is here.

Hmmm it seems like most

Anonymous's picture

Hmmm it seems like most linux developers can hash out lots of great industry standard code, but when it comes to projects that are on the forefront of research. only a few people in the world can easily do them. Most of these people will probably not be foss developers. The exception to that, that i've seen is software by scientists that can potentially help other scientists a great deal. Some examples are JImage, Octave, and Gap. I doubt something like photosynth would help the machine vision community in any way.

I definetly have faith that we can do it, but i don't have a lot of faith that we will be reading archaic journals not related to our work simply for the small occasional goldmine of software potential.

All of what you see that

Anonymous's picture

All of what you see that involves advanced algorithms or image processing starts in academia. Hence MIT's much older project.

If the OSS community wants to stay ahead of the curve (and Microsoft), there's a lot to be said about reading academic publications.

It's usually enough if you at least get a copy of conference proceedings; there are a lot of people who have the same interests as us that referee those papers and would like to see them shared with others.


BobCFC's picture

I remember seeing early alphas of this technology in the MS Research videos on their Channel9 pages a few years ago.

At the time the guy who developed this stuff was just a small company, not much more than an advanced Phd project and in the interview he said they were in the process of converting the engine from OpenGL to DirectX since they had been bought by Microsoft, and the demos at the time were still in OpenGL.

He was marvelling at the new office space they had etc, it's not really home grown Microsoft stuff at all they just bought it while it was nascent.

If you ask me, Photosynth

Anonymous's picture

If you ask me, Photosynth doesn't look any different than Make3D: -- which is open source. And dates back to at least 2005.

Photosynth has roots in GNUI/Linux... before Microsoft Ruined It

Roy Schestowitz's picture

Photosynth ran on and was demonstrated on GNU/Linux before Microsoft /acquired/ it (sorry... 'in-no-vated' it). You can still find the videos in YouTube.


Glyn Moody's picture

In that case, the question becomes why it couldn't continue to be developed on that platform - in other words, what was missing: maybe just money for the move to a startup group....

Yes, really.

Jeroen Van Goey's picture

See this comment by Kevin Chiu:
Hi. I worked on this project. The University of Washington’s contribution to the project (what you see at was developed on linux. The online demo app was programmed on windows and os x using Java.

Video where you can see the Linux titlebar.

Thanks for that

Glyn Moody's picture

That's really fascinating - and sad, too, considering what happened to it. As I say, the real question becomes *why* it started out on GNU/Linux, and finished up on Windows only....

Make 3D is very clever...

Glyn Moody's picture

...but doing something quite different. As the Web site says:

"Make3D converts your single picture into a 3-D model, completely automatically."

Photosynth takes *many* pictures, shot from different angles, and creates a composite landscape from them. Potentially, it could create very extensive navigable spaces in a way that Make 3D cannot.

Maybe "different" in the

Anonymous's picture

Maybe "different" in the sense that it is more clever. The fact that the automated converter works from one picture instead of many isn't a failure. The Make3D web app may only work in one way, but that certainly doesn't mean the algorithms are restricted; if Make3D can "learn depth" from a single image, it could do it from a sequence shot from different angles, too.

And who is to say what Photosynth "potentially" could do and what Make3D cannot "potentially" do? If we're going to talk "potentially"s, then we must compare what Photosynth can "potentially" do to what Make3D can "potentially" do -- not compare Photosynth's "potential" to the here-and-now-only of the Make3D reference implementation.

I'm not criticising Make 3D

Glyn Moody's picture

But there's no way it can put in visual information that isn't there - for example, beyond the picture that it is working on. Photosynth uses multiple images, and therefore can extend the visual field in any direction, dependent only on what other pictures are available.

And my use of "potentially" was not in the sense "one day it might", but in the sense that it has the power, now, to do this. Sorry if it wasn't clear.