Marco presented the talk first during TED 2012.

And here is a view backstage:
Click here to view the embedded video.

For additional information on the project please refer to the documentation on onformative.com

TEAM
Created and produced by Marco Tempest
in cooperation with onformative and checksum5

Special thanks to David Britland for his amazing script, Michael Ricar for the music and sound design. Tebjan Halm for additional vvvv programming. Enrico Viola for the relentless OF sandboxing and prototyping.
Jason Saragih for sharing his FaceTracker library. Tobias Beckwith for the voice cameos, Sherry and Matt for being amazed and to Steve Cohen for bringing Jean Robert-Houdin to life one last time.

This is backstage: (video removed upon request by client)

For addidional information and fancy imagery please visit:
Documentation on m-box.de (with video)
Documentation on artcom.de

img, #cubbies-overlay{ -moz-transition-property: margin, box-shadow, z-index; -moz-transition-duration: 0.1s; -webkit-transition-property: margin, box-shadow, z-index; -webkit-transition-duration: 0.1s; } .cubbies-selected{ z-index: 9999; box-shadow: 3px 3px 8px -1px blue !important; cursor: pointer !important; margin: -3px 3px 3px -3px; } .cubbies-selected:active{ box-shadow: 2px 2px 5px -1px darkblue !important; margin: -1px 1px 1px -1px; } #cubbies-overlay{ position: fixed; z-index: 9999; bottom: 30px; left: 30px; box-shadow: 0 2px 3px rgba(0,0,0,0.8); border: none; } #cubbies-overlay:hover{ box-shadow: 0 2px 3px rgb(0,0,0); }

–>

[..] The participants there will experience the Lufthansa brand with all their senses and become brand ambassadors. The main target groups for the Brand Academy are managers and service staff on board and ground. via: Lufthansa Investor Relations

Part of the Brand Academy is a nearly 360° surround projection inside the replication of an airport tower replacing all beveled windows by screens. For that projection space whitevoid designed an interactive 3d flight schedule viewer based on real flight data.

All flight connections of Lufthansa and its partner air lines can be explored one by one, symbolized by either their static flight nets or by their actual scheduled flights showing plane positions at any choosable simulation speed or for the current time. Possible displayed air lines include Lufthansa, German Wings, Swiss Air and the Star Alliance. A 3d mouse allows the viewer to navigate in a google-earth like fashion at any time of the simulation. Additional features include a helper button for proper camera pitch depending on the distance to the earth and a panic button driving you home from any position.

The project was realized by a vvvv boygroup with 3 clients controlling 6 screens covering 180°.

The vvvv server also did some precomputations for a fitting audio experience. Near and visible planes and towers had to be filtered out and were sent via OSC commands to the audio engine.

The highly detailed simulation & the pure mass of flights together with the diversity of possible viewing angles onto the scene demanded adding new features to vvvv, such as better text rendering in 3d, better performance of big spreads / more performant caching mechanisms for already computed data and the possibility to draw billboards and onesided shapes.

Since vvvv beta 21 everyone can make profit of these new features.

Team

Creative Director: Christopher Bauder
VVVV Programming: Sebastian Gregor, joreg
3D Modelling: Christian Guder, Jakob Sabra
Audio Engine / Sound Design: Robert Henke

Additional Information
Documentation on whitevoid.com
Documentation on monolake.de

Animation

All the animations during the 10 minutes (except 15 seconds of prerendered video) were generated in realtime following a timeline and triggered by the visitors interactions. All scenes were previously designed by 2d and 3d artists that exported textures and 3d models for later use within vvvv.

photo: Rainer Kohlberger

Interaction

As expos are typically highly frequented the installation was designed so that it could be experienced by about 50 people at once. 10 cameras hanging from the ceiling oversaw the whole floor and a special video tracking software developed by brainsalt.net communicated with the vvvv boygroup to tell it when and where interaction took place.

photo: Rainer Kohlberger

Highlight of the show was a scene that featured an interactive water surface filling the whole floor. By walking around or simply waving their hands people were able to disturb the water surrounding them, in a quite natural way.

In another scene buds appeared on the floor and by touching them people let them grow to large roots that were realized using an L-System algorithm. Here is a first sketch of the growing root:

The next video shows a sketch of people walking around in a swirl of particles that were always avoiding them. The particles were later replaced by nicely shaded triangular geometries, as the photo below shows.

Team

Director: Marc Tamschick
Concept: Marc Tamschick together with GPD S.A. Amaya Gonzales, Boris Micka
Visual Artists: Judith Holzer, Martin Retschitzegger, Claudia Rohrmoser, Marc Tamschick
3D Artists: Wanja Gökler, Martin Retschitzegger, Markus Egerter
VVVV Artist: Rainer Kohlberger
VVVV Programming & Consulting: Sebastian Gregor, Joreg
Interaction & Image Hardware: Gerd Trautner
Music & Sounddesign: Marc Lingk

Addidional Information:
Documentation on brainsalt.net

All prerendered videos were realized by m-box and tamschick media+space. Music was composed by idee und klang.

Warping & Softedge

Despite the size of the project, money constraints prohibited the use of an automatic projector calibration system. Therefore warping and Softedge had to be realised manually which obviously cannot be perfect. Another thing intensifying the problem at the orverlapping projected regions (as you can see in the video above) are the different colors some of the projectors, that also were calibrated manually.

Generative Content

Due to the modular nature of vvvv, now that the basic setup is running it is possible to add different generative contents for special occasions. Here an example of a simple ambient mode projection for evening events:

The dramaturgy of this mode can be easily adjusted via remote control, resulting in an animation that ranges from a very calm and reduced sligthly twisted spiral to a turbulent stream with local and global impacts.

photo: whitevoid

The multitouch surface was realized using the well known FTIR technique. One PC generated and projected the 3 consecutive images of 1400×1050 each. A second PC had two firewire cameras attached that saw the whole screen from the back, detected touches and sent that information to the first PC.

photo: whitevoid

The bubbles were lit by four point lights floating around in the water. By touching the surface you could catch their interest, so that at least one of them would come and look at what you are doing. As a special visual effect the whole surface behaved like a fluid, causing waves to expand from whereever touched with ones fingers.

During Development

These are screenshots from the development process of the bubble stream:

SPIN

The spherical projection interface has a diameter of 3 meters and can be entered through a closeable hole on one of its poles. The big sphere rests on 9 bowling balls and thus can be moved in all directions by a person walking inside, similarly to a hamster inside its wheel. From the outside a seamless projection covers the sphere which gives the user a 360° of freedom experience in a virtual world.

photo by time’s up

Even though the structure of the sphere can be seen from inside, it is easy to focus ones eyes on the virtual environment rather than on the spheres skin thus gaining a relatively good sense of full immersion.

Jacket

Before entering the sphere the visitor had to put on one of the crash-test-jackets. Those were equipped with speakers and a microphone to have contact to the outside world and hear the worlds sounds. Further the jacket carried pulse and breath sensors whose data were used as parameters in the virtual world.

Worlds

In all of the labyrinths it was the goal to find a glowing sphere which marked the exit and served as the entrance to one of the other levels.

Besides the very unique experience of being able to explore a virtual world by actually just walking around, some adjustments of the physical laws had to be made to match the experience of being able to walk in every direction, but not being able to step up or fall down. Since we wanted to allow the player to also climb objects we built a physics engine that followed the physics behavior insects may experience. When walking onto a bevel the world around you seemed to rotate, so that the bevel became the normal floor. From that perspective the former floor looks like a bevel. In fact we couldn’t decide yet if it’s really the world that rotated, or that it was the players perspective that rotated inside a static world…

Tube World

In this world you needed to find your way through a 3-dimensional tube labyrinth. In a tube you see two ends to which you can steer to. However when you steered to the walls or when you were imprecise in your walk to one of the ends the world around you seemed to rotate, since you were actually also climbing the bevels of the walls in a way. In that way a good sense of direction was necessary to remember where exactly you wanted to go when reaching one of the crossings.

The data of the breath-sensor was used to control the behavior of doors blocking ones way inside the tubes. Facing such a door one had to take and hold a deep breath in order to be able to pass it.

Relative World

In the Relative World we explored the basic contradiction between the explorable virtual architecture of a building and another coordinate system which reflects, that within SPIN you actually only walked to sides.
So some gates to other levels were hidden in a global coordinate system and others were distributed in the architecture of the building which would rotate each time you hit a wall to become a floor. The arrows were hints that there is an alternative, pervasive world that is worth to explore. The arrows didn’t rotate when the architecture did and simply pointed to one of the gates. That way it was easy to find that gate when focusing on that system and forgetting about the tumbling architecture around you.

Paper World

Once again you were confronted with a labyrinth where you obviously need to follow some paths. The paper world managed to stabilize your trip by rotating accordingly. However it was also possible to fall from a paper by crossing its edges. In that case you were placed back to your last save position.
By breathing you blew paper around you so that it made ripples.
The basic idea for the paper world was also derived from the kind of abstract, mathematical idea of one-sided surfaces. So some surfaces were only visible and walkable from one side, which made it sometimes hard to find the right path to a gate at the horizon.

The 4 projections surrounding the sphere were generated from only one pc with two graphic cards. Image-Warping was realized via 2-pass rendering as the vvvv-screenshot below shows two of the four distorted renderer outputs in its upper right corner:

The video below shows a timelapse recording of setting up SPIN, which takes about 4 hours.
[See post to watch QuickTime movie]

Additional Information:
SPIN documentation on timesup.org
related: egosoccer - a game concept for SPIN.

The bank building with its different types of balkonies was a very good target as the architecture would cast many shadows onto itself.

As the virtual light ball flying in front of the building was also interacting with the architecture, it was sometimes caught in a balkony causing a kind of magic light situation.

On the lower part of the building a screen was hung to have a flat surface where players see their shadows interacting with the ball. A camera was pointed at that screen from the opposite side of the street to analyze the shadows that would influence the physics of the ball.

A 3d geometry representation of the facade was created and projected to fit exactly onto the architecture. The images below show the 3d geometry rendered in wireframe mode during setup.

The setup needed a lot of manual calibration to get both real and virtual shadows connected to people, architecture and virtual light source. Here are some impressions of this process:

Additional Information:
Documentation on hoepfel.net

Main Club

The dancefloor has a base area in the form of a triangle, with entrances on each of the 3 corners. Every wall is being lit by 7 projectors, all together having a boygroup of 21 pcs + a server behind them, that generates the continuous images surrounding the whole space.

Membrane

One of the quirks of the setup is the projection surface, which is not a plain wall, but a membrane mainly consisting of holes. Through those holes a second layer of the membrain can be seen shifted back by some centimeters.

Knowing the exact geometries of the whole membrane the system is able to project precisely onto the membranes surface or into its holes. Also the “cocoons”, rest-areas integrated into the walls, can be addressed individually by the vj.

Roomjockey Interfaces

In order for one person to control all the visuals at once, several special interfaces had to be created that communicate with the main rendering pcs (the boygroup). Additionally it was requested that video and light could be controlled by one guy, hence the name “roomjockey”. Via touchscreens the roomjockey now can change patterns, colors, textures, videos and geometries of the current scene. Animation parameters can be played and recorded (for loop-playback) in realtime and predefined sets can be or triggered to change the whole scene at once.

A vvvv-patch on the boygroup-server showing the preview rendering of one of the 3 walls:

A short pan around the club during setup showing first-test renderings on the membrane:

Additional Information:
Documentation on meso.net

360°

Entering the ball you could step up to its equator to find yourself within a 360° surrounding projection. All projected content was rendered in real time. Via different interfaces like touchscreens and trackballs visitors were able to browse soccer related informations, video clips, play a quiz or send messages to the wall. The projection was realized with a 10 projector setup and was our second major boygroup.

photo: Wolfgang Stahl

Touch

In addition to the large projection there where 3 touchmonitors for the little ones to reach. They featured lovely 3d monsters animated by fiftyeight that would react to the kids touches.

photo: Wolfgang Stahl

Additional Information:
Documentation on meso.net