Overview
The Sand Noise Device, or SND, is being developed with the intent of creating an intuitive and tactile method for controlling and interacting with a generative electronic music system. The SND makes use of an original hardware and software system to measure the topography of a sandscape as well as the location and color of several tangible objects. Users are able to reconfigure the sand
scape and change the placement and color of these objects as a means of influencing various aspects of the generative music system. This project is the culmination of a Research and Development Thesis for the California State University East Bay Multimedia Masters program. Description
The SND is both a complex generative music system as well as a novel and intuitive interface for influencing and interacting with this system. The interface consists of a table height box filled with sand, a Microsoft Kinect (which provides an RGB camera and depth sensing capabilities), an overhead projector, and several internally lit tangible objects. The projector is used to provide visual feedback. Sound is provided by a multi-channel loudspeaker system arranged around the box. The various parameters that determine the functionality of the SND’s generative music system are influenced by the user’s interactions with the sandscape and provided tangible objects. By manipulating the sand and objects the user is able to influence various aspects of the generative music system. The audio aesthetic of the SND tends more towards the textural than the melodic. Traditional western music standards (e.g. tonal harmony, 12TET, and homophony) are not given much emphasis in the design of the generative music system. The SND is intended to produce music more in line with something like Silver Apples of the Moon than Switched on Bach. Physical Description
The SND is designed to provide the user with a completely immersive experience. The sandbox is built so that the average adult user can reach almost across the entire surface. It is at table height and allows for a comfortable stance for interacting with the sand. This setup is enclosed inside a darkened gazebo tent so that outside sounds and lights do not interfere with the user's experience. Four speakers are arranged in the corners of the gazebo that point inwards and down towards the user standing in the center. The sandbox is filled with 3-4 inches of sand to give the user the ability to dig down into the sand and create large holes and tall mounds. This allows for interesting sandscapes and many options for audio control. A Microsoft Kinect and a video projector are mounted above the sandbox and pointed down on top of the sand. The projector provides visual feedback as well as a constant and controllable light source. The Kinect is positioned so that it can easily and accurately read the topography of the sand and the colors of the various tangible objects. The tent is custom designed to mount all of the electronics and wiring so that it is not perceived by the user. The four sides of the tent will have retractable curtains that, when dropped down, will help to block out ambient light and external sounds immersing the user in the experience. Operation
The sound generation system of the SND is based on the motion and position of various virtual objects which digitally interact and collide with each other in virtual space. The behavior of these virtual objects is dependant on the topography of the sandscape as well as the locations and colors of the various tangible objects. Visual representations of the virtual objects are projected onto the sandscape to provide the user with a clear view of the nature of these interactions. By adding, removing, or relocating objects as well as varying the topography of the sandscape, the user is able to physically interact with the virtual objects and in turn vary the operation of the sound generation system. This method of interaction is designed in such a way as to avoid a one to one relationship between a specific physical action and sonic parameter. The SND is not meant to be played like a traditional electronic musical instrument such as the Theremin, Ondes Martenot, Minimoog, or even the less traditional Reactable, so the user is provided no direct parametric control. Rather, they are provided with the means to physically interact with a dynamic virtual system. Tangible Objects and Sandfield Topography
Each tangible object contains an RGB LED and a method for switching between a set of five different colors. When one of these objects is placed into the sandfield its location and color are tracked by the Kinect and a corresponding virtual object is created. This is indicated to the user by the projection of an appropriately colored ring around the tangible object. As the tangible object is moved by the user so is its virtual counterpart. If the object is removed from the sandfield its corresponding virtual object is also removed. The Kinect’s depth sensor continuously tracks changes to the topography of the sandscape. This topographical data determines aspects of the behavior of several of the virtual objects. A software filtering process is used to ignore anything (such as a user’s limbs) that falls outside of the sandfield but still within the view of the sensor. Virtual Objects
There are three prototypical types of virtual objects: Waves, particles, and static objects. The properties, interactions, and collisions of these virtual objects are the primary mechanisms of the SND’s generative music system. Waves are circular objects that radiate outwards from a specified point. Their primary purpose is to collide with other virtual objects. The parameters that define a wave are its location, color, speed, intensity, and duration. The values of these parameters determine what happens when a wave collides with another virtual object. The color determines the base collision type while the other parameters determine the low level details of the collision event. Waves are represented visually as thin, colored, unfilled circles. Particles are small objects that travel in a path around the virtual space. They are the most complicated type of virtual object. The parameters that define a particle are its location, color, velocity, angle, and intensity. Each of the various types of particles moves in a different manner. They can move in straight lines, follow the topography of the sand, spiral outwards, etc. Unlike the other virtual object types, particles have a Z value based on the height of the sandfield at their current location. This is one of the primary ways the user is able to use the sandfield’s topography as data for sonification. Static Objects are the virtual representations of the tangible objects described above. The primary functions of a static object is to act as a stationary entity for particles and waves to collide with and as a point of origin for the creation of new particles and waves. In addition, static objects can exert influence on the paths of particles. Other than altering the topography of the sandscape, interacting with static objects is the only means provided to the user for influencing the functioning of the music system. Just as with their tangible counterparts, the important parameters of static objects are their location and color. They are represented visually as thick, colored, unfilled circles. New particles and waves are created when either a particle or wave collides with a static object. The type of object created is dependent upon the nature of the colliding objects. Because this is a dynamic system particles and waves have limited lifespans. Waves fade out after a certain amount of time and particles are destroyed when they collide with another object or other conditions have been met. This keeps the system from chaotically growing out of control. Since waves and particles are created by collisions and they have limited lifespans it is possible for the system to reach a state where there are no particles r waves present. To counteract this, a ring of particles are generated at the center of the sand field every ten seconds. These particles act as an initial impulse to trigger the start of the virtual object system. The SND’s generative music system operates by sonifying several of the virtual object parameters described above as well as derived parameters such as distance between an object and certain points on the sandfield. In addition, collisions between objects result in special audio events whose content is determined by the parameters of the colliding objects. Technology
The SND’s software system is being created almost entirely with open source tools. The logic, sensor reading, and graphical components are being written in C++ using openFrameworks, an open source creative coding toolkit. OpenFrameworks provides an interface to the Microsoft Kinect as well as an implementation of the industry standard openCV computer vision library. All of the audio is synthesized using Pure Data, a graphical programming language with an emphasis on real time audio DSP. The Open Sound Control protocol is being used to facilitate communication between openFrameworks and Pure Data.
