Desire Paths of November

For the month of November I lived and worked in Zürich, a guest of the Institute for Computer Music and Sound Technology (ICST) at Zürich University of the Arts (ZHdK). (Thanks to everyone, but most especially Daniel Bisig, who was a gracious and accommodating host.)

This was an intense and productive period. In addition to work on my own project, I attended three conferences: STS Turns Aesthetic, The Diagrammatic Practice of the Micropolitical –

the Spatio‐temporal Expression of Play between Power, Knowledge and the Aesthetics of Existence and Post-Digital Cultures (the latter was actually in early December in Lausanne, but right after I got back) and had numerous formal and informal meetings with folks working in design, arts and in-between.

I rather like Zürich.

Sticking to my production schedule, I have at this point been involved in heavy development work on my system for more than one year. There are still a few technical hurdles to overcome, but 

I am making a concerted effort to turn my work away from technical development and towards the creation of interventions in public space. 

The November period ended with a public event entitled Stage Digital II - The Making of Atmosphere

(this was the second instantiation of the Stage Digital event which I participated in last year). 

I began with the desire path concept which I began working on this fall. The idea was to fold the ReOSC recording tool and the visualization component (previously composed with Unity3d) into one piece of software which would form the basis of a tool to collect and visualize the movement of people in space over time.

I had hoped to have a much longer working period to develop the concept (and particularly the aesthetics) much further, but the month was full of good and productive distractions and in the end I managed only the most basic implementation of the desire lines system. Still, what I have sets the stage for the work I want to do in the near future: I now have a tracking system and a basic spacial-memory; A way that I can augment architectural space with memory and recall. 

The presentation at Stage Ditial II also allowed me a chance to show this system to general public and, perhaps more importantly for my work, to students and faculty from ZhDK, ETH Zürich and EPFL working in theatre, scenography, performance and architecture and interdisciplinary research.

Finally, from the beginning one of the main design principles of my system has been flexibility with the data. I have built everything around OSC (Open Sound Control) and Syphon in order to permit the processing of the data by nearly any system. Although it was only an early test, I was pleased that it took about 10 minutes to configure the sharing of data wirelessly from my tracking installation to the ICST Dodecahedron. Within 30 minutes, Daniel had integrated that data into the behavior of several of the actors in their Max/MSP swarm simulation (for more on the dodecahedron, which is not my project, check out this blog post).

Artistic Research?

As a feral academic from the US living in Europe, I occasionally stumble into pockets of language or concept that are at once familiar and foreign. None have been as recently persistent in my universe as the phrase Artistic Research.

I am not, at the moment, attempting to suggest that anything is or isn't artistic or research, nor am I collecting examples (although I always love to see interesting work). I'm mostly just curious how widespread this idea is. I had not encountered it as a named category of work before I moved to Switzerland. 

As far as I can tell, Artistic Research refers collectively to two closely paired notions: 1. That research can be undertaken using methodologies which were not traditionally recognized by established academic research communities and that 2. Knowledge itself can be acquired, understood, described and contained in ways which are not traditionally recognized by established academic research communities.

From a research perspective it seems to be primarily about acknowledging methods and strategies previously ignored. To enable a lecture to also be a performance, or to permit painting, drawing or other non-verbal methods to exist alongside writing and reading. To attempt to elevate these "methods of understanding" to augment or replace the traditional research-object (IE a written document).

From an artistic perspective it seems to be primarily about adding a layer of methodology to "tighten" your work. IE to describe a research question prior to executing the work as an experiment. Artists do this anyway, but this is a kind of formalization of the process in an effort to make the outcome more generalizable. 

Across Europe, one major impetus for this seems to be the Bologna Accords, which mandated that all institutions  of higher learning in the  47 member countries fund and support research. With this event, we find a number of schools (in particular art, design and theatre schools) struggling to come to terms with what constitutes research and how this can become part of their practice.

A few more datapoints:

• I gave this talk on artistic research at the design school HEAD (Haute Ecole d'Art et de Design, Geneva) in the late spring. it was my own attempt to map the phrasing onto my practice.

• The grant I am working under (SINLAB) is part of the SINERGIA project and is hosted by the theatre school HETSR (Haute Ecole de Théâtre de Suisse Romande). In part  they provided us a home as a tactical way of integrating research into their curriculum (it is a struggle.)

• ZHdK offers "alternative" PhDs in collaboration with art schools which allow you to submit for final consideration either: A formal dissertation, a dissertation with an art object as a case study, an art object exclusively.

• Norway seems to be ahead of the curve, and has (among other things) hosted the Sensuous Knoweldge conference for nearly a decade. This conference is explicitly focused on the acquisition and retention of knowledge angle of AR. 

• The Journal of Artistic Research was created to archive this sort of work  (in part with support from the same individuals who architected the ZhDK PhD program). There is a companion tool for the creation of AR called the Research Catalogue.

• I recently attended the STS Turns Aesthetic conference hosted by ETH Zurich, which was largely about exploring this from an STS perspective: what happens when STS "turns" aesthetic, how can this particular practice absorb/observe/engage in artistic modes of exploration.

• ZHdK has been hosting an annual "performative practice" (link in German)  workshop

• There are a number of things happening which are not explicitly labeled Artistic Research but fall firmly into this camp, such as Knutpunkt,  an international conference about Nordic-style LARP which has been running for the last four years. 

Finally, it is worth noting (In as non-cynical a view as possible), AR also seems to be a strategy for acquiring funding, as it makes research "sexier" and makes art "more rigorous." This is acknowledged, but among practitioners there also seems to be a true belief that there is something new here.

I am trying to figure out if this is a widespread thing, a silly semantic exercise or an idea whose time has come (or all of the above.)

Atou SHiNMu: Interactive Dance

As a result of the ENSAL residency, I met the Atou dance company, led by Japanese choreographer Anan Atoyama. Anan had a show in production involving video but was interested in integrating some generative/interactive elements. She approached the Theoriz Crew team about the possibility of augmenting the show given the short time and budget. Since our ENSAL collaboration had gone well and the tracking problem they were facing was so similar to the work that I was already doing, I agreed to participate in the project in exchange for having a "live" context to test my work.

The result was SHiNMu (a portmanteau from the Japanese meaning "deep dreams"), a 50 minute 9 part dance piece exploring dreams, dreaming, horror and solitude. 

"Colorscript" provided by AToU, used by the visuals and lighting team to grade the colors over time

This was a truly a collaborative project. I was directly responsible for a portion of the tracking software and for hardening the backend used to process that data, as well as for operating the interaction and video board during performance.  the other half of the tracking pipeline was TSPS system augmented as previously described by David Chanel and myself. The visuals were produced by a graphics team composed of Alexane Douzet Sebastien Vallart, and the generative Unity3D visuals were created by ECAL student Marc Dubois and the music by composer Aurélien Marion Welsh. There were a number of fantastic stage technicians working with us on the install and the lights and of course the dance piece itself and overall direction came from Anan and Marc.

The main contribution from my PhD work was Merge, a software designed to merge the data from multiple depth camera sources (in this case two Asus Xtion's mounted above the stage). The result of this was sent via OSC to Unity3D game engine, which was used to generate the visuals we projected onto the stage (using Millumin to handle the mapping and stitching of three independent projectors). 

Screenshot of Merge software, showing three cameras as input. System outputs a single coordinate system.

Merge's main job is to collate the data from multiple independent networked data sources and handle rotation, scaling and positioning of the cameras. The software speaks the TSPS protocol and can therefore be used directly with TSPS (as we did in this show). Also, a drop-in replacement for TSPS which better suits the system is currently in development.

Work with AToU dance company "SHiNMu (Deep Dreams)" from Andrew Sempere on Vimeo.

ENSAL Residency and TSPS

This post is about ENSAL residency (Sept 1-7, 2013) and the two weeks prior

After the initial CYBORGAME rehearsals, I spent several weeks tightening the software and fixing various bugs. I also had an opportunity to spend a week working with David Chanel (of Theorize Crew) who was working at the time in the EPFL LTS2 lab. Among other things they are responsible for a system which tracks commuter patterns in the Lausanne Train station by stitching together the data from more than 60 depth cameras.

David and I discussed the overall tracking problem that I've been tackling (covering a stage-sized area by stitching depth data) and he convinced me to change my approach. 

Until CYBORGAME I had been planning to push all of the point cloud data from each remote Kinect through the network. In fact I did this (and it works pretty well, especially when compressed) but the compression introduces a bit of time overhead and more importantly limits the capability of your average network to 4-6 simultaneous point clouds (if you are lucky and assuming a closed, wired gigabit network).

The Lausanne train station system is far less precise than what I need for stage applications, but it collates the data from  many more (approximately 60) ASUS Xtions with little overhead. 

The approach is to offload the scene processing to embedded computers (or in my case, to the Mac Minis) which are connected to each Kinect. Software on these machines is responsible for collecting the point cloud data, but also analyzing it and running background subtraction, cleaning the noise, and ultimately sending a simple dataset which consists only of points of interest. 

As a starting point, David and I spent a week working on TSPS, which is an openFrameworks based system capable of doing visual tracking and scene analysis at fairly high frame rate. It also has the advantage of being  (theoretically) cross-platform.

TSPS showing one of our modifications - the addition of depth camera support, clipping and projection

TSPS is under active development, and I'm sure this is outdated as I type it, but it was built for and around optical tracking (IE webcams). It does an excellent job of this, but for my purposes it was not robust enough and lacked a lot of the support for the depth cameras (Kinect and ASUS) that we are using. 

David and I spent the week building out TSPS to include the features we thought we needed for the tracking problem. Following this, I accepted an invitation for a one-week residency at École Nationale Supérieure d’Architecture de Lyon (ENSAL) to work with a team on three installations they had proposed inside the main ENSAL building. For this project there were six of us (a mix of developers, musicians and visual effects folks). We split into groups of two and presented the result at the end of the week. 

In addition to testing the TSPS based system, I wanted to begin shifting course away from purely technical development and towards the architectural ideas which sent me on this course in the first place. I have in mind some notions of architectural memory

To that end I used the week to develop ReOSC, an OSC logger and playback tool that we can use in conjunction with a tracking system to record the movement in a space over time. The idea is to collect data that comprises the "memory" of a space. On a purely instrumental level this can be used to determine usage patterns, etc, but I am more interested in the conceptual: Invisible lines of desire and the ways we traverse space. Can a building dream? Can a space mourn the loss of an occupant?

ReOSC: Tool for recording and playback of OSC data

ENSAL has a basilica layout, with a large central corridor ending in a multistory circular rotunda that houses the program offices. This main "street" connects all the labs and classrooms on the ground floor and provides access to the exists and stairways to the upper level. It can be observed from a second story balcony and, during the day, is brightly lit by a glass ceiling that runs the length of the building.

The main hallway of the ENSAL building where the residency took place

In addition to developing ReOSC, for the residency I created a very simple projection/interaction system which tracked people moving down the hallway and provided them with a colorful contrail describing their trajectory. This simple interaction allowed me the time and space to work on the tracking/recording system and activated the space with an element of pleasant surprise. In some cases individuals traversing the hallway ignored the installation completely (still providing a small light show to incidental observers). In other cases, individuals spent a good deal of time playing with the light, making shapes and experimenting with interactions between each other. My favorite interaction was an older man who I was later told was a retired Swiss judge. He spent 20 or 30 minutes dancing with the system.

Although it provided a good base to work from, after this week I concluded that TSPS is not up to the job after all, due mostly to the user interface code. In theory I could re-write this, but it seems unwise to do battle against the framework at its core. Instead we are decomposing the problem and working on platform-native code that should be more stable for stage applications.

MIDI Mapping

I decided to integrate the Korg NanoKontrol2 and NanoPad midi controllers into my software for Cyborgame. Along the way I also mapped the Vestax Spin (mostly because I wanted something to work on while I waited for the Korgs to arrive).

Maybe someone else has done this, or maybe it's even published somewhere, but I appreciated the chance to learn the guts of MIDI, and if it helps you, you can have it here in PDF form.

CYBORGAME - Pre Production

Remember : " If I get out of it, I will leave stronger."

DARK

F AI N T LI G H T

A dark room.

Shaped like a cube.
The ground is about two hundred meters square.
The ceiling isn't visible.
No clear entry.
No exit.
No windows.
Cut in a faint light, we can make out the silhouette of the Character from the Book.
Some details stand out in him, a fresh femininity (once a man - but today a hybrid mix of a woman, part animal, and machine - when we speak of her, we call her "she." When we think of her, we think of a Don Quixote).
We think at first that she is naked.
But with another glance, we notice that she is wearing an armor of some kind. Delicate cybernetic skin.
She is alone.

- opening from CYBORGAME by Gildas Milin -

For the last year or so I've been working on software whose main intent is to be able to stitch the data from multiple Kinect cameras together so that I can track people over a large area. My other requirements were that this be compiled, native software (for portability, speed and stability) and be Mac based. I am using this software as the basis for my PhD work on interactive scenography and architecture.

I've been working in the context of a theatre school, which has a periodic visiting artist program. (Our first  (7/23: correction: second. First was Pablo Ventura) visiting artist, Mark Coniglio, is, as I write this, in the US working on a Kinect based piece called SWARM. Having seen the early work and as an admirer of his work Loop Diver, it's going to be great).

Our latest artist in residence is the French director and playwright Gildas Milin, who is working on a piece called CYBORGAME or 33 BATTLES or THE FICTIONAL CHARACTER (a Hybrid Female Mutant Don Quixote).

The play follows the story of the "fictional character" played by Julie, who wakes up in an isolation cell confused and disoriented, imprisoned by "the military" and a second character (The Colonel) who is in turns a guide, tormentor, alter-ego and captor. The Colonel is played by Vassar, lead singer of the French pop/rockabilly band Sons of Nusku, (all of the music in the show is from their album which is available on Spotify if you want a listen).

Gildas' style is intense and process oriented: he has a particular system for the cast and crew to be involved in every step of the production, from install to teardown to performance to group meals and foursquare games. A typical rehearsal day runs from 11 to 11. Musicians become actors, actors become theatre techs, software becomes performer and along the way elements compose and re-compose to form the scenography.

Shortly after I met Gildas he asked me to participate in the rehearsals by "just showing up and working, whatever you normally do."

At the time what I was "normally doing" was long hours of coding image and pointcloud processing routines, so I began attending the rehearsals and pointing my networked Kinects at the cast. I recorded more than 10 gigabytes of raw pointcloud data from various angles and worked on better compression for the networked data (I'm still having compression issues, incidentally - I'm using lz4 which is ok but could be better if you have any ideas).

Simultaneously, I began work with my colleague Selena Savic on a project to create "disembodied text" (essentially a large scale teleprompter) in order to layer the actual text of the play into the performance space.

Selena did a tremendous amount of work on materials selection: identifying and testing the right combination of materials and their position on stage to give the desired effect without too much excess light or distortion. Selena also wrote a text-mining script to pull relevant strings out of the play script in realtime, while I worked in parallel on some custom software to allow us to play with the display of text.

Throughout this process Gildas would periodically look over my shoulder and mention an effect or a visual he particularly liked. We took a break, I worked on tightening the software and then we regrouped. Over the last two weeks we've been engaged in an intense full-scale rehearsal. What we have now are multiple projections on a scrim cube surrounding a rotating stage. During the performance I live-mix the feed from several Kinects, the recorded material I've collected and various color, light and text effects. There are four projectors and six screens. 

The result of this is that my tracking and text display system has, somewhat accidentally, vectored off into a performance/VJ setup for generating live and recorded motion graphics for CYBORGAME. In short, my tracking system has become the atmosphere generator for the show, sketching the scenography live, in light, across the skin of the actors and the surfaces of the theatre.

What is remarkable to me is that, as initially conceived, this tool was never meant to be seen by an audience. Most of the initial "aesthetic" decisions were purely functional or accidental glitches. For example, rotating a cube with the openGL clear screen turned off makes for a lovely retro-spirograph effect. This is now built into the software.

Over the last couple of weeks I've been adding a lot of additional performance features (including Space Navigator and MIDI controller support) mostly to make my own life easier during mixing.

Finally, I'm looking into using my system alongside "real" performance software (like QLab) to make things even easier to manage, and I haven't forgotten my main target, which is still fast scalable tracking software.

I've been up to my eyeballs in technical issues, but there are a large number of interesting artistic/aesthetic questions around this work (some of which I hope to address during a month-long workshop residency in Zurich in November at ZhDK on Atmosphere and the Aesthetics of Tracking, and some I hope to address in the context of the Feral Research project) but for now it's an awful lot of fun just to be part of a functional project and to let my work breathe a bit. 

The show is in pre-production now. We just finished an intense 2 week rehearsal. In September there will be another intense 5 days of culminating in a small public performance (do let me know if you're in Lausanne and interested). 

And then?... We shall see.