I designed and built an interactive art installation in collobaration with Chris Mueller. In adition to the interactive art we also wanted to research large-scale motion control. The result was a large-scale music sequencer which uses the Microsoft Kinect to gather all input. Predictably, our research showed that different types of motion control interactions are good for different purposes.
The concept was for people to control a giant music sequencer through motion. Users create musical "nodes" to place around the virtual environment; their 3D coordinates would determine tone and timing. Additionally, we would use this as an opportunity to research different methods of large-scale motion control. We would call the project Modal Nodes
Pose vs. Gestures
One aspect of testing was to see the differences between gesture-based (movement over time) and pose-based (static) interactions. We set up user testing sessions which varied poses and gestures for different interactions. Through observation and retrospective interviews, we found a few key differences.
- Higher affordance
- Natural connection to the resulting action
- Lower success rate (users performs gesture incorrectly)
- Low affordance
- Easily performed
- Highly repeatable
We decided to use a combination of poses and gestures to leverage their strengths depending on the situation. Broad, easily performed gestures would be used for interactions that didn't require great precision, and poses would be used for quick, repeatable actions.
Affordance and Discoverability
Being a public installation, we wanted it to be usable without instruction. So, we needed to make sure that the interactions we chose were both intuitive and easy to perform. One way we accomplish this is by having the environment react as a user begins to complete an action.
For example, when users begin the process of creating a node, the normally ambient particles move away from the background and swarm around a user’s hands – the closer a user is to completing the action, the tighter and faster the particles swarmed (as shown in an exaggerated form below). When the action is complete, the particles turn into a node.
Sound For All
Most people are terrible musicians (including myself). If you get a bunch of people in front of a screen where their movement creates sound, they are going to explore and make a bunch of noise. Then multiply that by n people and you get an aweful experience for everyone. We needed to design for this type of environment.
Imagine the resulting noise like a waveform - plotting the tone on the y-axis over time. If you are moving between sharp, distinct tones, you'll have sharp edges in that waveform. While the tones will be more distinct, it has a greater chance to sound chaotic because the y value is fluctuating wildly. If you smooth those edges and stretch it out, then the resulting noise will blend together and become less abrasive and more ambient.
We approached this problem by first limiting the range of tones that users could create (reducing the possible delta). We then added a long, pulsing echo effect to blend the notes together (reducing the sharpness at each vertex). The result is an output that is expressive enough for users to see variation but is constrained enough that it doesn't punish experimentation.