SoundAffects is an interactive installation in the form of a wall with embedded sensors, cameras, and light components, installed on street level at 13th Street and 5th Avenue, New York City. The wall “listens” to the active sounds of the city that surrounds it– taxis, voices, footsteps, honking horns, jackhammers– as well as ambient stimuli created by the environment, such as rain drops, brightness, color, temperature and radio wave activity. The wall translates this real-time sensory input into a unique generative musical and light composition, which listeners can experience live at the wall itself and online at soundaffectsnyc.com.

Since its founding in 1896, Parsons The New School for Design has been one of the world’s leaders in educating the next generation of artists and designers. Their academic program was among the first in the industry to teach a holistic design process, centered not just around aesthetics but around entire systems – systems which sometimes need to be re-thought from the ground up. Art & design schools are in the unique cultural position of constantly re-imagining the perception of our changing world and then communicating that exploration process to others.

New York City is a notoriously difficult place to engage people who are moving from A to B. The constant movement, the crowded sidewalks, the traffic congestion, and the high ambient noise level all contribute to a high level of stress and a lack of engagement with surroundings. But what if there was a different way to see that environment? What if the stimuli of the New York City street could be transformed into an alluring and charismatic aesthetic experience? How might that change the perception of our cities?

Over the course of the two-week experiment, everything from passing taxicabs to evening rain contributed to the unique audiovisual fingerprint of a moment in time in that location. Dancing children, angry drivers, even clusters of balloons in the breeze were recorded and translated in real-time as part of the SoundAffects experience. On the website, users could view events as-they-happened as well as scroll back on the GUI timeline to see and hear how the environment affected the composition.

Mono, the creative agency who developed the core concept for Parsons, came to Tellart for the project’s design and technology. One of our objectives for this event was to make the technology invisible and let the beauty of the interaction take center stage. Passersby are hooked by the object; they can see the direct manipulation and feel the magic of discovering a complex, organic musical composition embedded in something on the street. As they gain interest, they are compelled to dive deeper into the collected data– and this is where the web naturally takes over. The ways to access the data become manifold: there is the music and the visualization; the timeline view that allows users to move backward and forward in time; the camera that provides the “wall’s eye view” and therefore a window onto the activity that is affecting the composition. An array of live “experiments” augmented the data and provided touchpoints that illustrate the effects of each stimulus and confirm relationship between the sensor data and the generative composition.

Part of Parsons’ intent for this project was to stimulate interest in the school using its physical location. How can an interactive street-based installation effectively advertise the experimental and cutting-edge spirit of academia at Parsons? Moreover, the implied messaging was as important as the direct: how can this project show respect for the intelligence of its users, rather than simply coming across as so much gimmicky technology stuck into a billboard?

Mono came to Tellart knowing that our specialty is in connecting physical objects to the web in meaningful forms. The sensory stimulation of a street in New York City is just that – sensory stimulation. Sound, light, precipitation, wireless networks, traffic and pedestrian activity are all sources of data that can be displayed and transformed in fascinating ways. Moreover, these are all technologies that allow passersby to interact with an object before they even realize that’s what they’re doing. Seeing the real-time cause-and-effect of environmental stimuli on a piece of architecture is an excellent way to hook people’s interest. Therefore, Tellart’s challenge was to design an object and experience that invited that type of interest.

SoundAffects employed many different types of technology which all had to work together in perfect synchronization for a cohesive live experience to be presented. Industrial sensors embedded in the wall were operated with the help of Arduino microcontrollers. Multiple cameras recorded live video, both for display on the website and to mine for color data. The software experience was enabled by HTML5, JavaScript, Max/MSP/Jitter, Ableton Live, and various other tools.

One difficulty in this type of composition process is finding ways that unpredictable stimuli can control an experience without simply creating chaos. A main goal of the experiment was to create serenity out of disorder – a task that is easier said than done. Extensive experimentation had to be done at each stage of the process. How should the sound of construction relate to the yellow of a taxicab? How does the composition change with the weather and time of day? How can subtle stimuli (raindrops) compete with obvious stimuli (car alarms), or should it even try? Additionally, in keeping with this strategy, each individual sensor had to be finely tuned to pick up the full range of the stimuli it was meant to detect. Tellart has several expert musicians on staff, who wrote the musical composition algorithm that connected all of these stimuli in a natural and compelling way.

SoundAffects generated value on several different levels. On the surface level, SoundAffects accomplished its goal of engaging the public with the location & identity of Parsons, thereby increasing visibility and interest in the institution. That visibility was quantifiable, both in terms of enormous website traffic and in the form of the millions of external media impressions that were generated– including features in publications such as Wired Magazine, The Boston Globe, Communication Arts, Creativity Online, and many more. What began with a simple advertising brief was transformed into something that brought the public delight and a deeper engagement with (and understanding of) their surroundings.

On a deeper level, the project accomplished a goal that runs through many of Tellart's projects: using the web to augment and bring magic to everyday occurrences, and to facilitate a global interest in a decidedly local experience. It translated raw data into an ambient, emotionally rich experience. SoundAffects hooked interest and encouraged interaction purely through its magical “ah ha!” features, and then made a multitude of layers of information available to be continuously peeled back. One of the most amazing qualities of the modern web is how it enables people to dive deeper into information, learning and experiencing more along the way. Transforming that quality into the realm of physical space is one of the most fascinating design challenges of our time.