Deep Drawing centers on the creative use of AI in intermedia performance. Using the live audio input captured from the human performer’s drawing/writing, the AI performer generates a prediction of what the drawing looks like. Students will be involved in the design and application of the machine learning model that takes signals from a table-top wooden box outfitted with four contact mics and creates video output.
The AI performer translates the nuances and variations in the audio into corresponding visual movements and marks on a projected digital canvas. The real drawing and the predictive drawing are mixed to highlight the successes and failures of the machine. Students are encouraged to interrogate the ethical and practical use of AI in artistic creation.
The team will experience designing a ML model from scratch, specifically 1) real-time generation of pen positions from audio input feeding a drawing application, and 2) experimental image generation using both audio and video training data of the human performer drawing. Convolutional recurrent neural networks (CRNNs) are used to extract local spatial information and their trajectories from real-time audio features. Generative adversarial networks (GANs) are used to augment the custom and relatively small training dataset of the human performer’s drawing videos.
Students apply to a specific role on team as follows:
Machine Learning (2 Students)
Preferred Skills: Practical experience with multiple approaches to machine learning, such as supervised / unsupervised learning, reinforcement learning, dimensionality reduction, neural networks, deep learning; experience with real-time and/or systems
Likely Majors/Minors: ARCH, CE, CS, DATA, ECE, EE, ME, ROB, SI
Signal Processing (1 Student)
Preferred Skills: A background in digital signal processing, manipulation and cleaning of audio data, experience with DAWs and Python audio libraries such as Librosa.
Likely Majors/Minors: CE, CS, DATA, ECE, EE, FTVM, PAT, ROB, SI
Faculty Project Lead
Julie Zhu is a composer, artist, and carillonist. Her work is conceptual and transdisciplinary, operating on an expansive definition of algorithm. Creative and ethical use of AI and machine learning in the arts is one of her research interests and the focus of her Presidential Postdoctoral Fellowship at the University of Michigan.
As an advocate for intermedia composition, Zhu collaborates with artists and musicians globally. The results of these collaborations have been exhibited at and performed in studios and residencies throughout Europe, North America, and Asia, but namely Carnegie Hall (NYC), Herbst Theatre (San Francisco), IRCAM (Paris, France), digitIZMir (Izmir, Turkey), Tetramatyka Festival (Lviv, Ukraine), Sansusī (Latvia), Chicago Home Theater Festival, Miami Design District, College Band Director’s National Association, ICMC (Shenzhen, China), among others. Notable commissions came from Radio France, GMEM, San Francisco Contemporary Music Players, and the University of Chicago Carillon.
Many of Zhu’s pieces are characterized by constraints: whether it is putting a percussionist (or herself) in a wooden box, drawing circles that are amplified through a speaker dome, or hiding the harpsichordist and keyboard in a cardboard house with live video, or developing the just-intonated harmonic architecture for the specific rotation, orbit, and angle of Mercury, Venus, Earth, and Mars in a string quartet. For Zhu, art is best as an anonymous gesture and as a trap.
As a carillonist, Zhu regularly performs on the Burton and Lurie Tower at the University of Michigan and concertizes in the summer. During her time as a visual artist in New York City, she was the resident carillonneur at Saint Thomas Church Fifth Avenue.
Students: 3
Likely Majors/Minors: ARCH, CE, CS, DATA, ECE, EE, FTVM, ME, PAT, ROB, SI
Meeting Details: Hybrid. Stearns 123, Wednesday or Friday afternoon.
Application: Consider including a link to your portfolio or other websites in the personal statement portion of your application to share work you would like considered as part of your submission.
Summer Opportunity: Summer research fellowships may be available for qualifying students.
Citizenship Requirements: This project is open to all students on campus.
IP/NDA: Students who successfully match to this project team will be required to sign an Intellectual Property (IP) Agreement prior to participation.
Course Substitutions: CoE Honors