As a result of the ENSAL residency, I met the Atou dance company, led by Japanese choreographer Anan Atoyama. Anan had a show in production involving video but was interested in integrating some generative/interactive elements. She approached the Theoriz Crew team about the possibility of augmenting the show given the short time and budget. Since our ENSAL collaboration had gone well and the tracking problem they were facing was so similar to the work that I was already doing, I agreed to participate in the project in exchange for having a "live" context to test my work.
The result was SHiNMu (a portmanteau from the Japanese meaning "deep dreams"), a 50 minute 9 part dance piece exploring dreams, dreaming, horror and solitude.
 |
| "Colorscript" provided by AToU, used by the visuals and lighting team to grade the colors over time |
This was a truly a collaborative project. I was directly responsible for a portion of the tracking software and for hardening the backend used to process that data, as well as for operating the interaction and video board during performance. the other half of the tracking pipeline was TSPS system augmented as previously described by David Chanel and myself. The visuals were produced by a graphics team composed of Alexane Douzet Sebastien Vallart, and the generative Unity3D visuals were created by ECAL student Marc Dubois and the music by composer Aurélien Marion Welsh. There were a number of fantastic stage technicians working with us on the install and the lights and of course the dance piece itself and overall direction came from Anan and Marc.
The main contribution from my PhD work was Merge, a software designed to merge the data from multiple depth camera sources (in this case two Asus Xtion's mounted above the stage). The result of this was sent via OSC to Unity3D game engine, which was used to generate the visuals we projected onto the stage (using Millumin to handle the mapping and stitching of three independent projectors).
 |
| Screenshot of Merge software, showing three cameras as input. System outputs a single coordinate system. |
Merge's main job is to collate the data from multiple independent networked data sources and handle rotation, scaling and positioning of the cameras. The software speaks the TSPS protocol and can therefore be used directly with TSPS (as we did in this show). Also, a drop-in replacement for TSPS which better suits the system is currently in development.
Work with AToU dance company "SHiNMu (Deep Dreams)" from Andrew Sempere on Vimeo.
