Unnatural Nature: Immersive Sonification

and Interactive Eco-Art

Eloysa Zelada, School of Music, Theatre & Dance

Collaborators: Dr. Anil Camci, Associate Professor, PAT, SMTD; Matthew Dunlap, Alumni, Music Technology, NYU

“Unnatural Nature” is an immersive installation project blending sonification and interactive art to convey climate change data emotionally. Participants engage with touch-sensitive plants, shaping a unique musical performance and fostering a connection with the environment.

The project’s success includes a well-received public presentation and international recognition at the International Conference of Auditory Displays.

Future enhancements aim to refine sound design, explore projection mapping, and collaborate with ecologists for real-time plant data. The project envisions continued expansion, reaching diverse spaces to inspire reflection and action on the environmental crisis.

I am delighted to present the evolution of the “Unnatural Nature” project, an immersive and interactive installation born from the exploration initiated during my master’s thesis in the Performing Arts Technology program. Now, as a new PhD student in the same department, my enthusiasm for advancing the realms of sonification, interactive art, and their profound connection with environmental issues and ecology has only deepened.

In this proposal, I aim to articulate the significant progress made thus far and request additional funding to further enhance and present this project at upcoming events, with a notable inclusion at an outdoor event in Ann Arbor during the summer of 2024.

Current Project Overview: The existing project structure provides a two-part immersive experience designed to elevate awareness surrounding the urgent issue of climate change and our individual roles in addressing it. The first segment employs sonification to convey scientific data emotionally, thereby making intricate information accessible and impactful for a broader audience.

In the second phase, participants are encouraged to interact with physical plants equipped with touch sensors, influencing the sonic and visual responses of the installation. The real plants become conduits for a unique musical performance, fostering a connection between participants and the environment while prompting contemplation of our relationship with the natural world.

The project’s initial public presentation on April 6, 2023, at the Davis Technology Studio in the Moore Building, was attended by around 50 people. The feedback was overwhelmingly positive, with attendees describing the experience as powerful, moving, and impactful. Many highlighted the installation’s inspirational message, viewing it as a metaphor for collective action, emphasizing that collaboration can lead to more significant impact and positive change.

In addition to the public presentation, a paper co-authored with my advisor, Dr. Anil Camci, for the International Conference of Auditory Displays held in Norrköping, Sweden, in July of the same year, provided an opportunity to share the project with experts in the field of sonification and sound from around the world.

The positive feedback received at the conference has fueled my motivation to explore new facets of the project and expand its goals, with a specific focus on user experience and increased collaboration with experts in the field of ecology.

Extensions and New Areas of Exploration

Fine-Tuning of Sensors: Collaborating with Matthew Dunlap, a now alumni from NYU, a longstanding collaborator in this project, our aim is to fine-tune the system for outdoor adaptability. The focus is on designing a protective mechanism for the Mac Mini, our main network system, using a Pelican computer case.

This adaptation is crucial, considering the outdoor installation at a festival in Ann Arbor, where a high density of people will be in attendance. The goal is to enhance the project’s technical implementation, making it more portable and adaptable to various locations beyond the high-tech setup available at the Davis Studio.

Scientific Collaboration: To further refine sound design, I plan to collaborate with an ecologist to capture real-time data from plants’ electrical signals. This data-driven approach will provide a more realistic experience, offering profound insights into the internal world of plants and their direct relevance to environmental issues.

User Studies: In addition to evaluating sonification effectiveness through user studies, I aim to conduct a more comprehensive study. This will involve control groups, visualization versus sonification comparisons, and a long-term analysis of the effects of sonification on the understanding of climate change data. The goal is to create a robust design that captures the intricacies of user interaction with the installation.

Projection Mapping: Exploring projection mapping to intensify the immersive experience, I aim to map projections onto a spherical object representing the Earth. Seeking collaboration with a Stamps student proficient in projection mapping, this addition will not only enhance the visual aspect of the installation but also provide more cohesiveness to the overall design.

Project Timeline and Funding Needs: The funded portion of the project’s completion aligns with the outdoor event in Ann Arbor during the summer of 2024. In addition, I plan to have additional public demonstrations at other spaces and venues such as Video studio at the Duderstadt Center, Matthei Botanical Gardens, among others, between Winter 2024 – Winter 2025.

The requested support primarily covers:

Obtaining Software Licenses: The initial exploration and trials of the project utilized free versions and trials of software such as Max MSP and TouchDesigner. With the expiration of the Max MSP trial, it remains the main software for receiving, parsing, and analyzing data from the sensors.

Additionally, all sound design and triggers for visual cues are executed within this software. The TouchDesigner free version, however, has limitations, especially regarding resolution and frame rate capabilities, essential for projection mapping.

Purchasing the full versions of both Max MSP and TouchDesigner is imperative for the project’s continued growth and functionality. In addition, the purchase of Resolume, a software tested in its free version, is essential for the final video rendering of the visual design. Its capabilities are particularly crucial for incorporating projection mapping into the project.

Technical Enhancements: To ensure the project’s portability and adaptability, technical enhancements include the purchase of a waterproof Pelican case to protect the Mac Mini and additional sensors as backups for the outdoor installation. Two projectors are also necessary for testing the implementation mapping and increasing flexibility for presenting in various spaces that may lack additional equipment.

User Study Costs: Funding is required for expenses related to the setup of user studies, including printing, designing, and distributing necessary material. Compensation to participants will also be provided to attract a more significant and diverse group of participants.

Results Presentation and Publishing Opportunities: Results from the initial project have already been published in the International Conference of Auditory Displays, with a second paper in progress for the Personal and Ubiquitous Computing Journal by Springer. The high-quality presentation and rendering necessary for both the event and potential publications are reliant on the mentioned software tools.

Beyond academic publications, future plans include presenting at conferences such as ICAD, AudioMostly, ICCM (International Conference of Computer Music), NIME (New Interfaces of Musical Expression), or ARTECH. Additionally, once the project is ready and flexible for easy setup in various spaces, efforts will be made to reach out to different venues and spaces to expand its reach and impact.

In conclusion, the “Unnatural Nature” project has already demonstrated its potential to captivate audiences and contribute meaningfully to the discourse on climate change awareness. The proposed enhancements, collaborations, and funding support will not only ensure the success of the project at the outdoor event in Ann Arbor (A2 Summer Festival) but also facilitate future presentations in diverse spaces.

I extend my gratitude for the support Arts Engine has provided thus far and eagerly anticipate continued progress and achievements with “Unnatural Nature.”