Mediated Reality: University of Toronto RWM Project

A solution to this problem may be obtained through something I call “Mediated Reality”. Mediated Reality (MR) differs from virtual reality (or augmented reality) in the sense that it allows us to filter out things we do not wish to have thrust upon us against our will. This capability is implicit in the notion of self-determination and mastery over one's own destiny. Just as a Sony Walkman allows us to drown out Muzak with our own choice of music, MR allows us to implement a “visual filter”. I will now describe how MR works. Later, we will see the importance of a good software basis for MR and why Linux was selected as the operating system for the apparatus of the invention (WearComp) upon which MR is based.

Figure 1

To understand how the reality mediator works, imagine first a device called a “Lightspace Analyzer” (see Figure 1). The Lightspace Analyzer is a hypothetical “lightspace glass” that absorbs and quantifies incoming light—it is completely opaque. It provides a numerical description (e.g., it turns light into numbers). It is not necessarily flat; the analyzer is drawn curved in the figure to emphasize this point.

Figure 2

Imagine also a “Lightspace Synthesizer” (see Figure 2). The Lightspace Synthesizer turns an input stream of numbers into corresponding rays of light.

Figure 3

Suppose we connect the output of the Lightspace Analyzer to the input of the Lightspace Synthesizer (see Figure 3). We now have an illusory transparency.

Moreover, suppose we could bring the Lightspace Analyzer glass into direct contact with the Lightspace Synthesizer glass. Placing the two back to back would create a collinear illusory transparency, in which any emergent ray of virtual light would be collinear with the incoming ray of real light that gave rise to it. (See Figure 4.)

Figure 4

Now, a natural question to ask is, why all this effort in making a simple illusion of transparency when we could just as easily purchase a small piece of clear glass? The answer is that we have the ability to modify our perception of visual reality by inserting a WearComp between the Lightspace Analyzer and the Lightspace Synthesizer. (See Figure 5.)

Figure 5

In practice, there are other more practical embodiments of this invention than the one described above, but the basic principle is the same. Some practical examples are described further elsewhere in the literature (see Proceedings of IEEE ISWC98, “WearCam, the Wearable Camera”, by Steve Mann, pages 124-131). The result is a computational means of altering one's visual perception of reality.

WearComp has the potential to make all the world virtual as well as real; moreover, the potential is there to create a modified perception of visual reality. Such a computer-mediated reality can not only augment, but also diminish or otherwise alter the perception of reality.

Why would one want to do this? Why would anyone buy a pair of sunglasses that made them see less?

An example might be when we are driving and trying to concentrate on the road. Sunglasses that not only diminish the glare of the sun's rays but also filter out distracting billboards could help us see the road better, and therefore drive more safely.

Figure 6. Sunglasses for Filtering

Moreover, Mediated Reality can help us reclaim solitude in personal spaces. By wearing special sunglasses in which a visual filter is implemented (see Figure 6), it is possible to filter out offensive advertising.

Figure 7. Top: Convention Hall Reality. Second from top through bottom: Convention Hall Mediated Reality

Lightspace entering the analysis side of the glass manifests itself as an input image sequence where it is absorbed and quantified by the special sunglasses. Figure 7 shows Convention Hall as it truly is; Figure 8 shows its transformation with visual filtering.

Recall that the sunglasses are totally opaque except for the fact that the WearComp copies the input side to the output side, with possible modification. In the case of an offensive advertisement, the modification could take the form of replacing the advertisement with a more calming image of a waterfall.