Augmented Touch: A Mounting Adapter for Oculus Touch Controllers that Enables New Hyperreal Instruments

NIME 2022 · 2022-06-16

In this paper, we discuss our ongoing work to leverage virtual reality and digital fabrication to investigate sensory mappings across the visual, auditory, and haptic modalities in VR, and how such mappings can affect musical expression in this medium. Specifically, we introduce a custom adapter for the Oculus Touch controller that allows it to be augmented with physical parts that can be tracked, visualized, and sonified in VR. This way, a VR instrument can be made to have a physical manifestation that facilitates additional forms of tactile feedback besides those offered by the Touch controller, enabling new forms of musical interaction. We then discuss a case study, where we use the adapter to implement a new VR instrument that integrates the repelling force between neodymium magnets into the controllers. This allows us to imbue the virtual instrument, which is inherently devoid of tactility, with haptic feedback––an essential affordance of many musical instruments.

A method for analyzing room modal response using auralization

Applied Acoustics · 2022 · 1 citations

• Computer Science
• Acoustics
• Computer Science

Capturing kinetic wave demonstrations for sound control

2020

• Computer Science
• Computer Science
• Acoustics

In musical acoustics, wave propagation, reflection, phase inversion, and boundary conditions can be hard to conceptualize. Physical kinetic wave demonstrations offer visible and tangible experiences of wave behavior and facilitate active learning. We implement such kinetic demonstrations, a long spring and a Shive machine, using contemporary fabrication techniques. Furthermore, we employ motion capture (MoCap) technology to transform these kinetic assemblies into audio controllers. Time-varying coordinates of Mo-Cap markers integrated into the assemblies are mapped to audio parameters, closing a multi-sensory loop where visual analogues of acoustic phenomena are in turn used to control digital audio. The project leads to a pedagogical practice where fabrication and sensing technologies are used to reconstitute demonstrations for the eye as controllers for the ear.

Additive manufacturing in musical acoustics

The Journal of the Acoustical Society of America · 2020-10-01

Additive Manufacturing (AM) affords the production of complex geometries for acoustic research. The rapid materialization of parametric models facilitates the practical comparison of related sounding or sound-filtering objects. This accelerated fabrication cycle continues our field's tradition of establishing and demonstrating results through making, with applications in structural acoustics (metamaterials), architectural acoustics (diffuser design) and musical acoustics (instrument design). This talk focuses on applications of AM in musical acoustics: the replication of known instrument geometries, the precise transformation of those geometries in order to test predictions, and explorations of the musicality of neighboring forms. Results, limitations and future prospects are considered.

Recreating a Rare Instrument Using VR and Fabrication: A Hyperreal Instrument Case Study

HAL (Le Centre pour la Communication Scientifique Directe) · 2020 · 3 citations

• Computer Science
• Computer Science
• Human–computer interaction

International audience

Acoustics education through digital fabrication

The Journal of the Acoustical Society of America · 2019-10-01

The integration of makerspaces in academic settings presents an opportunity to apply principles of musical acoustics to the rapid production of sounding objects. Digital fabrication accelerates the design cycle and alters what can be accomplished in the scope of such educational labs. Acoustic principles motivate parametric 3-D designs using open-source CAD software. These models can then be materialized through digital fabrication to empirically test numerical predictions; they also serve as prototypes for new instrument design and sound art. This paper reviews such a course offered at the University of Michigan in the Performing Arts Technology program. Machine and software requirements are discussed as well as methods to combine acoustic problem-sets with contemporary fabrication methods.

Mending Bells and Closing Belfries with Faust

Humanities Commons CORE (Modern Language Association / Columbia University) · 2019-01-01

Finite Element Analyses (FEA) was used to predict the resonant modes of the Tsar Kolokol, a 200-ton fractured bell that sits outside the Kremlin in Moscow. Frequency and displacement data informed a physical model implemented in the Faust programming language (Functional Audio Stream). The authors hosted a concert for Tsar bell and carillon with the generous support of Meyer Sound and a University of Michigan bicentennial grant. In the concert, the simulated Tsar bell was triggered by the keyboard and perceptually fused with the bourdon of the Baird Carillon on the University of Michigan campus in Ann Arbor.

Hyperreal Instruments: Bridging VR and Digital Fabrication to Facilitate New Forms of Musical Expression

Leonardo Music Journal · 2019-09-20 · 11 citations

Virtual Reality (VR) and digital fabrication technologies today are ushering in a new wave of opportunities in instrument design; the marriage of these two domains, seemingly at odds with each other, can bring impossible instruments to life. In this article, the authors first sample such instruments throughout history. The authors also look at how technology has facilitated the materialization of impossible instruments from the twentieth century on. They then discuss the bridging of VR and fabrication as a new frontier in instrument design, where synthetic sounds can be used to condition an equally synthetic sensory scaffolding upon which the time-varying spectra can be interactively anchored: The result is new instruments that can defy our sense of audiovisual reality while satisfying our proprioceptive and haptic expectations. The authors report on their ongoing work as well as their projections of how emerging technologies in VR and fabrication will shape the design of new musical interfaces.