## Announcements - User Research marks and feedback out later this week - [Gitlab Template available for final project](https://gitlab.comp.anu.edu.au/comp3900/2026/comp3900-2026-A3-final-project) - Final project due: **2026-11-05 23:59 AEDT** - Remember: main deliverable in the assignment is your presentation video `project-presentation.mp4` - Demo at end of lecture on [recording a video in Powerpoint](https://support.microsoft.com/en-au/office/record-a-slide-show-with-narration-and-slide-timings-0b9502c6-5f6c-40ae-b1e7-e47d8741161c#OfficeVersion=2013%2C_2016&officeversion=newer_versions). ## Plan for the class - conceptualising expressive interactions - drawing interactions - music interactions - dance interactions - installed interactions - playful interactions - human-AI creative interactions - takeaways - recording your project video in PowerPoint # Conceptualising Expressive Interactions why? expressive, artistic experiences can be drivers for HCI.
First head-mounted AR display system (Sutherland, 1968).
The Sound of One Hand (1992) a VR performance piece: Jaron Lanier improvising on virtual instruments with a DataGlove (Lanier, 1992, 1993). Video at Moogfest 2016.
## Supporting Creativity
Movement towards studying tools to **support creativity**. > there is a move from routine work and productive concerns to human and creative ones.” ([Edmonds, 2018](#ref-edmonds-art-of-interaction:2018)) Expression and creativity let us get *inside* the process of an interaction. People are *interested* in expressive experiences, leading to critique and understanding.
Iamascope as shown in Edmonds (2018)
## Principles for Creativity Support Tools (CSTs) Shneiderman ([2007](#ref-shneiderman-creativity-support-tools:2007)) posits principles for developing CSTs
1. Support exploration. 2. Low threshold, high ceiling, and wide walls. 3. Support many paths and many styles. 4. Support collaboration. 5. Support open interchange. 6. Make it as simple as possible—and maybe even simpler.
7. Choose black boxes carefully. 8. Invent things that you would want to use yourself. 9. Balance user suggestions with observation and participatory processes. 10. Iterate, iterate—then iterate again. 11. Design for designers. 12. Evaluate your tools.
## Artists as Power Users Week 8 said to raise a research question until it “covers many people”. So why study a handful of experts? Because the chain runs the other way: 1. **Artist spec is the hardest spec.** “There are three levels of design: standard spec, military spec and artist spec… the third, artist spec, is the hardest (and most important)” ([Buxton, 1997](#ref-buxton-artists-luthier:1997)). Artists push a tool until it breaks. 2. **So artists are “creative power users”** (Linda Candy in Shneiderman ([2007](#ref-shneiderman-creativity-support-tools:2007))): lead users who show us the **boundaries** of an interaction. 3. **Findings at the boundary transfer.** What an expert needs from a mapping or an instrument tells us what creativity support tools need for “everyday creativity” ([Edmonds, 2018](#ref-edmonds-art-of-interaction:2018)). The sample is narrow on purpose. The claim is about what the **interface** can support, not about the population of artists. A tool that meets artist spec meets standard spec; the reverse does not hold. ## Research Questions for Expressive Interfaces - **Which gap?** Usually a **capability gap** (“what are we not able to *do* yet?”) rather than a knowledge gap ([Hornbæk et al., 2025, Section 45.2](#ref-hornbaek-intro-hci:2025)). - **Level of abstraction.** Ask about the *interaction*, not the artist. “What gesture-to-sound mappings let a performer control timbre in improvisation?” affects everyone who builds a mapping. - **Refutable.** The performer can find the mapping unplayable, or prefer the “worse” model ([Fiebrink et al., 2011](#ref-fiebrink-human-model-eval:2011)), or abandon the instrument entirely ([Morreale & McPherson, 2017](#ref-morreale-design-for-longevity:2017)). If your question can’t come out that way, it isn’t a question. - **Scope.** On week 8’s pros and cons this is the uncontrolled side. You trade generalised results for insight at the boundary. Say so in your discussion. Why this work matters: the boundary is where the next everyday interface comes from. Lanier was improvising on virtual instruments with a glove in 1992; consumer VR took another 25 years. ## When the Artist is You: First-Person Methods
Much expressive-interface research is people studying their own practice. HCI calls these **first-person methods**: “qualitative research approaches that turn to the researcher as the subject of inquiry” ([Desjardins et al., 2021](#ref-desjardins-first-person-methods:2021)). - **Autoethnography:** your own experience of a phenomenon is the data, written with reflexivity and a personal voice. - **Autobiographical design:** build a system for yourself as a user with a genuine need, then refine it through your own use over time. - **Soma design:** the designer’s lived, moving body is the design material. “There is no short-cut, no simplified wireframes” ([Höök et al., 2018](#ref-hook-first-person-soma:2018)).
PhaseRings came from your lecturer’s own practice; the *cube* AR instrument (coming up) used autobiographical design.
**Careful:** first-person is not “I liked it”. The editors ask for authenticity, ethics and rigour: documentation, sustained use, and saying plainly what a sample of one can claim. It shows what is *possible*, not what is *typical*.
## What is an expressive interaction?
Mapping sensed **gestures** to an expressive output that is fed back to the user. - **gestures:** the use of motions by the limbs or body as a means of expression - can be unintentional, control, or ancillary gestures - from non-human actors (e.g., the movement of leaves on a branch of a tree) - *“any sort of motion, that may be understood as an expression of something”* The interaction itself is expressive, and the output is an expression as well. We consult Composing Interactions ([Baalman, 2022](#ref-baalman2022composing)) as a resource.
Sensing movement and touch to create music, Atau Tanaka performing in 2010 (Tanaka, 2010)
## Mapping from Gesture to Output
Why is mapping an important consideration? > Considering a performer performing making gestures on a stage, which gestures effect changes in the output medium of sound, which can be heard by the performer and audience in the real, physical environment.
Two tinted circles joined by a loop. Environment on the left holds performer, audience and other factors; media on the right holds video, audio, light, mechatronics and other media. Mapping sits between them.
The question of mapping: the connection between a gesture in the environment and the output media, redrawn after Baalman (2022).
## Steps in Mappings
Baalman ([2022](#ref-baalman2022composing)) expands the mapping process into a cycle. - A gesture is performed in the environment; - This is captured by a sensor that translates the gesture is processed by an electronic circuit, often to digitalise it; - Next, the signal enters some sort of computational model that translates the data to parameters; - These parameters control an output medium such as sound, light, video, or mechatronics. how is output from one connected to input? what happens in each step?
Seven stages around a loop: gesture, sensor, electronic circuits, computational model, output model, output media, and the performer's perception, which leads back to the next gesture.
Steps in the mapping process, redrawn after Baalman (2022).
# Drawing Interaction ## Surface Drawing System Drawing with hands, moving, scaling, erasing (a pair of kitchen tongs) ([Schkolne et al., 2001](#ref-schkolne2001organicshape))
Hands define a stroke.
Kitchen tongs move a drawing.
A variety of shapes created with Surface Drawing (Schkolne et al., 2001).
## AirPens
- AirPens: Musical Doodling ([Frame et al., 2025](#ref-frame-airpens-doodling:2025)), an installation at NIME 2025. - Make music with mark-making. - Use IMU sensors to convert movement into sound; explore different mappings between movement and sound.
AirPen at NIME25.
## MicroJam
App for making short (5s) musical performances with a sketch ([video](https://youtu.be/HhI47-XzrtI) and [info](https://charlesmartin.au/projects/microjam/)). ([Martin & Torresen, 2020](#ref-martin-tiny-touchscreen:2020)) - Uses touch location and movement to create sound. - Replay performances by rewinding the sketch and viewing it again. - Social media interface: view other people’s drawing and add layers as a musical “reply” - Analysed 1600 tiny jams to understand musical drawing behaviour.
Analysing the kinds of drawings
Analysing the speed of swipes
# Music Interaction The [New Interface for Musical Expression (NIME)](https://nime.org) research community. - Research into musical instrument design that explores how technological innovation can enable new musical expression, enhance performer control and intimacy, and shape the musician-instrument relationship. - Digital Musical Instruments (DMIs): digital piano, drum pad. - Augmented instruments: magnetic resonator piano (grand piano -\> string instruments), pictured. - Novel instrument: [lady’s glove](https://sonami.net/portfolio/items/ladys-glove/), magnetic AI instrument *thales* ([Privato et al., 2023](#ref-privato2023thales)), percussive instrument [PhaseRings](https://charlesmartin.au/projects/phaserings/) ([C. Martin, 2018](#ref-martin2018phaserings)), AR instrument *cube* ([Y. Wang & Martin, 2022](#ref-wang-cubing-sound:2022)). ## Case Study: The Reactable (2003 to now)
Jordà’s group at Pompeu Fabra made a synthesiser into a tangible tabletop: pucks, a camera underneath, projected lines showing what feeds what ([Jordà, 2010](#ref-jorda2010reactable)). - **Prior work:** tangible tabletops ([Ishii & Ullmer, 1997](#ref-ishii1997tangible)) and a decade on collaborative visual instruments ([Jordà, 2003](#ref-jorda2003sonigraphical)). - **Had a theory:** learnable by novices, deep for experts, and shared between players - **Kept going:** iterated for years and toured, with the tracking software released as open source. - **Then got lucky:** demos “watched millions of times” as YouTube took off, Björk’s 2007 world tour.
The Reactable (Jordà, 2010). Source: YCAM.
Jordà’s retrospective, with an expert commentary by your lecturer: ([Jordà, 2017](#ref-jorda2017sonigraphical)).
## Case Study: Ocarina (2008)
Ge Wang had an iPhone: mic, multitouch, accelerometer. The *efficient* design is tap-a-note: instant, easy to learn, hard to get wrong. - **What he did:** blow into the mic to sound a note, cover four holes for pitch, tilt for vibrato. Harder, slower, and you will play wrong notes ([G. Wang, 2014](#ref-wang2014ocarina)). - **What happened:** ten million users (with its sequel) and the App Store Hall of Fame; the rationale came out in *Computer Music Journal* six years later.
**Careful:** everything that made Ocarina work would count as a defect in week 8! The effort and the errors were the *point*, and that’s this week’s topic.
Breath, four holes, tilt (G. Wang, 2014).
## PhaseRings: natural gestures on big touchscreens
How can we perform music on big touchscreens? - Rather than re-implement music production apps, looked at natural gestures in long-term artistic practice. - Lots of apps from 2011–2014 and lots of performances e.g., Martin ([2016](#ref-martin-phaserings:2016)) - Initially, sought to understand how percussionists would play touchscreens ([C. Martin et al., 2014](#ref-martin-exploring-percussive-gesture:2014)) - Then explored networked connections in ensemble performance ([C. Martin et al., 2015](#ref-martin-tracking-ensemble-performance:2015)) - Then compared how networked ensembles supported by the interface ([C. Martin et al., 2016](#ref-martin-intelligent-agents:2016))
PhaseRings played by visitors to a gallery workshop (2015)
## Hyperinstruments: non-guitar guitars
- When is a Guitar not a Guitar? ([Harrison et al., 2018](#ref-harrison2018guitar)): novel instruments and controllers resemble traditional instruments. - Four designs examining variation in form (held vs tabletop) and interaction (strings vs touch sensor) - **Guitarists** prefer the technical familiarity of a stringed instrument - **Non-musicians** prefer touch-interface: ease of use, cultural load of the guitar form
Four “guitar-like” instruments from Harrison et al. (2018). strings vs no strings, held vs tabletop.
## Authentic musical instruments for AR
*cube* system: authentic design for a head-mounted AR musical instrument ([Y. Wang & Martin, 2022](#ref-wang-cubing-sound:2022)) Autobiographical design ([Desjardins et al., 2021](#ref-desjardins-first-person-methods:2021)) to compare three interface candidates 1. **Physical interface:** slow accurate manipulation due to hand tracking. 2. **Spatial interaction:** emerged from bodily movement allowing ease of use. 3. **Flexible freehand interaction:** allow multiple notes to be played simultaneously, taking advantage of the full-hand tracking affordance in the AR headset.
Physical interface: a translucent cube in AR holding synthesiser sliders for two oscillators, a VCA, an ADSR envelope and a filter, operated by a tracked hand. Spatial interaction: two oscillator cubes; moving one vertically changes pitch, horizontally changes LFO pulse width, and resizing changes volume. Flexible freehand interaction: a ring of coloured note cubes played with the fingertips of both hands at once.
## Gesture to Sound Mappings in Music
- **Traditional approach:** decompose input actions, sound production and feedback ([Miranda & Wanderley, 2006](#ref-mirandawanderley2006dmi); [Wessel & Wright, 2002](#ref-wessel2002problems)). - **New sounds, music, and experiences:** expanded control over timbre and texture, richer and more diverse musical experience ([Magnusson, 2010](#ref-magnusson2010designconstrain); [Wessel & Wright, 2002](#ref-wessel2002problems)). - **Control intimacy:** mediated gesture and sound relation ([Wessel & Wright, 2002](#ref-wessel2002problems)), a limit of the machine-centred approach ([Pigrem & McPherson, 2018](#ref-pigrem2018we)). - **Current research:** finding better frameworks. Performer and instrument engage in a mutually influential relationship during the design process ([McPherson et al., 2024](#ref-mcpherson2024mappingnew)).
The traditional DMI model separates a musician’s input action captured by the controller, the mapping engine bridging the input interface and sound engine (Magnusson, 2010).
## Mapping by demonstration: the Wekinator
- The Wekinator ([Fiebrink et al., 2009](#ref-fiebrink-wekinator-nime:2009)) turns mapping to demonstration. Show the system an input and the output you want, train, play, and retrain when it feels wrong. - Evaluation by playing. Fiebrink’s CHI paper ([Fiebrink et al., 2011](#ref-fiebrink-human-model-eval:2011)) found practitioners judged models by how they felt under the hands, and often preferred a “worse” model. - The demo: a webcam, a coloured object or a face, a synth. From Fiebrink’s teaching rather than in the early papers ([Fiebrink, 2019](#ref-fiebrink-ml-education:2019)). - A regression from two camera coordinates to some synth params is simple to implement in 2026.
**Interactive machine learning** in one sentence: the human is in the training loop, and the loop is fast enough to feel like tuning an instrument.
## Does Anyone Still Play It?
The Reactable and Ocarina are the success stories. What happened to everything else? Morreale and McPherson wrote to the authors of every NIME 2010 to 2014 paper introducing an instrument; 70 of 97 replied ([Morreale & McPherson, 2017](#ref-morreale-design-for-longevity:2017)). Of 51 built for prolonged performance: - **47.1% not ready to perform on** - of those, two thirds needed more than a few hours’ work, and **15.7% no longer existed at all** Top reasons: “I don’t have the opportunity at the moment” (22), “I turned my attention to building other DMIs” (14).
Years after the paper (Morreale & McPherson, 2017).
## Easy Instruments Make the Choices for You
McPherson, Morreale and Harrison compared instruments from NIME, HCI, and Kickstarter ([McPherson et al., 2019](#ref-mcpherson-instruments-for-novices:2019)). - **Campaigns say:** 16 of 30 pitch ease of use. Play “any sound you can imagine”. - **Designs do:** 22 of 30 are MIDI controllers, made “easier” by limiting pitch space or quantising time, which decides in advance what you can play. - **Concept:** “easy” means someone else chose your constraints ([Magnusson, 2010](#ref-magnusson2010designconstrain)).
**The test:** longevity. Not “liked it for an hour” but “still playing it two years later”. Where does your project sit on that survey in 2028?
My bench. Which of these still work in 2028? (Charles, 2024)
# Dance Interaction From gestures to body (embodied) movements. Pictured: the CO/DA system ([Françoise et al., 2022](#ref-coda2022chi)). - Support real-time manipulation of continuous streams of the dancers’ motion data for interactive sound synthesis. - Enable novel dance improvisations through live coding. - Live coding: interactively programming musical or visual processes as performance. ## Co/da system
- Movements are measured using motion sensors, and the live coder processes motion signals to generate feedback in real-time. - Enable a multitude of feedback loops: sound feedback -\> movement improvisation -\> the coder alters the relationships between movement and sound. - Dynamic improvisation that stimulates novel movements’ exploration.
Co/Da’s mapping diagram (Françoise et al., 2022)
# Installed Interactions Putting expressive interactions into public places. Pictured: [Bellyhorn](https://dianneverdonk.com/bellyhorn/) by Dianne Verdonk. ## Dinosaur Choir
Dinosaur Choir: Adult Corythosaurus ([Brown & Gajewski, 2025](#ref-brown-dinosaur-choir:2025)) - Singing dinosaur skull musical instruments. - Experience dinosaur vocalisation: imagine dinosaur vocal anatomy from a bird syrinx, (vocal structure open question in paleontology) - Microphone for user, computational vocal model, sound resonates through a 3D printed replica of the dinosaur’s nasal cavities and skull - Change the pitch and timbre of the vocalisation by changing the shape of the mouths, like trumpet player
Dinosaur Choir at NIME25.
## Illumicube (in Canberra!)
[official website](https://www.arts.act.gov.au/public-art/i/illumicube) - Interactive sculpture in Canberra by Kerry Simpson (1988) - Glass and sound (now movement) activated lighting - Location: Ainslie Avenue, Canberra
**Careful:** playful installed interaction can lead to unwanted behaviour! Noise from folks exiting civic pubs!
Illumicube (link) CC BY-NC 2.0
# Playful Interaction Pictured: the Phox Ears listening helmet by Kleinberger et al. ([2015](#ref-kleinberger-phoxears:2015))
**Think:** can silly or playful ideas turn into interesting interactions? Can we use play to examine more serious HCI concepts?
It’s fun to make the world more fun. ## Breath Controlled Amusement Ride
> Can an amusement ride be controlled by breath? - Inspired by robotic technologies for control of individual seats on rollercoasters and other thrill rides. - the [Broncomatic](https://www.somabotics.ac.uk/research-article/broncomatic/) is a bucking bronco game (horse riding; you try to stay on!) - The twist is it’s controlled by the user’s breath - the ride kicks you around and if you lose control it kicks harder (until you fall off) “Breath Control of Amusement Rides” ([Marshall et al., 2011](#ref-marshall2011chi))
Falling off the Broncomatic (Marshall et al., 2011)
## More on the Broncomatic
- A straightforward mapping: the rider’s breathing to the horizontal rotation of the ride. - Inhale: spins clockwise; exhale: spins anti-clockwise. - Breathing speed controls rotation speed: fast breathing faster spin; holding breath stops spinning. - Difficulty levels. - The program is a game in which the player scores more points the more that they breathe: a physical challenge vs reward dynamic. - More breathing for faster ride but harder to stay on. To score high, you must breathe more, but this makes the ride tougher.
Falling off the Broncomatic (Marshall et al., 2011)
# Human-AI Creative Interaction There has been interest in incorporating AI into creative interaction since computing began. Recent work often focuses on current genAI breakthroughs, e.g. [Autolume visual generator system](https://www.metacreation.net/projects/autolume-automating-live-music-visualisation-technical-report?rq=autolume) and [Reprising Elements performance (2023)](https://arshiasobhan.com/works/reprising-elements) — pictured. ## Why introduce AI into expressive interaction?
Computational creativity helps create new ideas in three ways ([Boden, 1998](#ref-boden1998computational)) - Produce novel combinations of familiar ideas; - Explore the potential of conceptual spaces; - Make transformations that enable the generation of previously impossible ideas. Creativity and technology: a sociotechnological perspective ([Bown, 2021](#ref-bown2021beyondcreative)). - The social nature of human behaviour. - Artistic behaviour is social in nature (is it?)
Beyond the Creative Species Making Machines That Make Art and Music. (Bown, 2021) open-access link
## Cobbie: co-creative robots
Cobbie system ([Lin et al., 2020](#ref-lin2020cocreativerobots)) - *Motivation*: Co-creative partner can reason about user’s intention and stably present novel ideas with the user initiative, which are hindered in human team due to social loafing or a resolute partner otherwise. - Take turns to draw ideas, give the dominant position to the user, and use movements and sound feedback for communication. - Three human-robot interactions: your turn, pause and draw again, progressing with feedback
Co-creating with Cobbie: a user sketches on paper, the robot generates an idea from the sketch, draws it on the paper, and the user continues from the robot's lines.
## Holographic dancing ghost
- Co-creative public dancer ([Long et al., 2019](#ref-long2019lumi); [Trajkova et al., 2023](#ref-trajkova2023holographicai), [2024](#ref-trajkova2024dance)) - Explore the design of the modular AI agent to creatively collaborate with a dancer. - A Kinect motion capture device to detect the user’s motion, visualised as a virtual shadow on a projection screen. - The humanoid agent shadow dances by analysing the user’s movement and responding with a movement that it deems to be similar in terms of parameters such as energy, tempo, or size. - Study results showed in-the-moment influences, self, partner, environment([Trajkova et al., 2024](#ref-trajkova2024dance)).
A dancer on a dark stage improvising beside a large projected mannequin-like AI dance partner. Three renderings of the AI dance partner: a green mannequin dancing with two people, a sparkling purple figure, and a glowing gradient figure.
# Takeaways - **Mapping is the interaction.** Gesture, sensor, model, output: every step is designable ([Baalman, 2022](#ref-baalman2022composing)). - **Artist spec is the hardest spec.** Experts show us the boundary, and boundary findings transfer ([Buxton, 1997](#ref-buxton-artists-luthier:1997)). - **Effort can be the point.** Ocarina’s breath and the Broncomatic’s bucking could be defects but actually make the design work. - **Evaluate by playing.** People judge mappings by feel, and may prefer the “worse” model ([Fiebrink et al., 2011](#ref-fiebrink-human-model-eval:2011)). - **The real test is longevity.** Nearly half of new instruments are unplayable a few years on ([Morreale & McPherson, 2017](#ref-morreale-design-for-longevity:2017)). - **AI partners ask:** who has the initiative? That’s next week.
**Think:** which of these would your final project’s evaluation let you claim?
# Recording a video in Powerpoint
For the final project you need to record and upload a presentation video. The specification is: - must be no longer than 5 minutes + 10% (300 + 30 seconds) - must be no larger than 1920 x 1080 pixels - must be no larger than 100MB - must be narrated with your voice - must show video of you speaking
**Do:** You can do this easily with [Powerpoint](https://support.microsoft.com/en-au/office/record-a-slide-show-with-narration-and-slide-timings-0b9502c6-5f6c-40ae-b1e7-e47d8741161c#OfficeVersion=2013%2C_2016&officeversion=newer_versions), so let’s give it a try.
recording a presentation video in powerpoint
## Questions: Who has a question?
**Who has a question?** - I can take *catchbox* question up until 2:55 - For after class questions: meet me outside the classroom at the bar (for 30 minutes) - Feel free to ask about **any aspect of the course** - Also feel free to ask about **any aspect of computing at ANU**! I may not be able to help, but I can listen.
Meet you at the bar for questions. 🍸🥤🫖☕️ Unfortunately no drinks served! 🙃
# References
Baalman, M. (2022). *Composing interactions: An artist’s guide to building expressive interactive systems*. V2_Publishing.
Boden, M. A. (1998). Creativity and artificial intelligence. *Artificial Intelligence*, *103*(1), 347–356.
Bown, O. (2021). *Beyond the creative species: Making machines that make art and music*. The MIT Press.
Brown, C., & Gajewski, C. (2025). Dinosaur choir: Adult corythosaurus. In S. Rose, J. Mulder, & N. Carroll (Eds.), *Proceedings of the international conference on new interfaces for musical expression* (pp. 138–141).
Buxton, B. (1997). Artists and the art of the luthier. *SIGGRAPH Comput. Graph.*, *31*(1), 10–11.
Desjardins, A., Tomico, O., Lucero, A., Cecchinato, M. E., & Neustaedter, C. (2021). Introduction to the special issue on first-person methods in HCI. *ACM Transactions on Computer-Human Interaction*, *28*(6), 1–12.
Edmonds, E. (2018). *The art of interaction: What HCI can learn from interactive art*. Morgan & Claypool Publishers.
Fiebrink, R. (2019). Machine learning education for artists, musicians, and other creative practitioners. *ACM Transactions on Computing Education*, *19*(4), 1–32.
Fiebrink, R., Cook, P. R., & Trueman, D. (2011). Human model evaluation in interactive supervised learning. *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems*, 147–156.
Fiebrink, R., Trueman, D., & Cook, P. R. (2009). A meta-instrument for interactive, on-the-fly machine learning. *Proceedings of the International Conference on New Interfaces for Musical Expression*.
Frame, C., O’Hara, S., Trolland, S., & Ilsar, A. (2025). AirPens: Musical doodling. In S. Rose, J. Mulder, & N. Carroll (Eds.), *Proceedings of the international conference on new interfaces for musical expression* (pp. 65–67).
Françoise, J., Fdili Alaoui, S., & Candau, Y. (2022). CO/DA: Live-coding movement-sound interactions for dance improvisation. *Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems*.
Harrison, J., Jack, R. H., Morreale, F., & McPherson, A. P. (2018). When is a guitar not a guitar? Cultural form, input modality and expertise. In T. M. Luke Dahl Douglas Bowman (Ed.), *Proceedings of the international conference on new interfaces for musical expression* (pp. 299–304). Virginia Tech.
Höök, K., Caramiaux, B., Erkut, C., Forlizzi, J., Hajinejad, N., Haller, M., Hummels, C. C. M., Isbister, K., Jonsson, M., Khut, G., Loke, L., Lottridge, D., Marti, P., Melcer, E., Müller, F. F., Petersen, M. G., Schiphorst, T., Márquez Segura, E., Ståhl, A., … Tobiasson, H. (2018). Embracing first-person perspectives in soma-based design. *Informatics*, *5*(1).
Hornbæk, K., Kristensson, P. O., & Oulasvirta, A. (2025). *Introduction to human-computer interaction*. Oxford University Press.
Ishii, H., & Ullmer, B. (1997). Tangible bits: Towards seamless interfaces between people, bits and atoms. *Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems*, 234–241.
Jordà, S. (2003). Sonigraphical instruments: From FMOL to the reacTable\*. *Proceedings of the International Conference on New Interfaces for Musical Expression*, 70–76.
Jordà, S. (2010). The reactable: Tangible and tabletop music performance. In *CHI’10 extended abstracts on human factors in computing systems* (pp. 2989–2994).
Jordà, S. (2017). 2003: Sonigraphical instruments: From FMOL to the reacTable\*. In A. R. Jensenius & M. J. Lyons (Eds.), *A NIME reader: Fifteen years of new interfaces for musical expression* (pp. 89–106). Springer International Publishing.
Kleinberger, R., Dublon, G., Paradiso, J. A., & Machover, T. (2015). PHOX ears: A parabolic, head-mounted, orientable, eXtrasensory listening device. In E. Berdahl & J. Allison (Eds.), *Proceedings of the international conference on new interfaces for musical expression* (pp. 30–31). Louisiana State University.
Lanier, J. (1992). *The sound of one hand*. Virtual reality performance, SIGGRAPH ’92 Electronic Theater, Aerie Crown Theater, Chicago, 28–30 July 1992.
Lanier, J. (1993). Music from inside virtual reality: The sound of one hand. *Whole Earth Review*.
Lin, Y., Guo, J., Chen, Y., Yao, C., & Ying, F. (2020). It is your turn: Collaborative ideation with a co-creative robot through sketch. *Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems*, 1–14.
Long, D., Jacob, M., & Magerko, B. (2019). Designing co-creative AI for public spaces. *Proceedings of the 2019 Conference on Creativity and Cognition*, 271–284.
Magnusson, T. (2010). Designing constraints: Composing and performing with digital musical systems. *Computer Music Journal*, *34*(4), 62–73.
Marshall, J., Rowland, D., Rennick Egglestone, S., Benford, S., Walker, B., & McAuley, D. (2011). Breath control of amusement rides. *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems*, 73–82.
Martin, C. (2018). *PhaseRings* (Version v1.3) \[Computer software\]. Zenodo.
Martin, C. P. (2016). PhaseRings for iPad ensemble and ensemble director agent \[Musical Performance\]. *Musical Program of the International Conference on Auditory Display*, 232–233.
Martin, C. P., & Torresen, J. (2020). Data driven analysis of tiny touchscreen performance with MicroJam. *Computer Music Journal*, *43*(4), 41–57.
Martin, C., Gardner, H., & Swift, B. (2014). Exploring percussive gesture on iPads with ensemble metatone. *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems*, 1025–1028.
Martin, C., Gardner, H., & Swift, B. (2015). Tracking ensemble performance on touch-screens with gesture classification and transition matrices. In E. Berdahl & J. Allison (Eds.), *Proceedings of the international conference on new interfaces for musical expression* (pp. 359–364). Louisiana State University.
Martin, C., Gardner, H., Swift, B., & Martin, M. (2016). Intelligent agents and networked buttons improve free-improvised ensemble music-making on touch-screens. *Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems*, 2295–2306.
McPherson, A., Morreale, F., & Harrison, J. (2019). Musical instruments for novices: Comparing NIME, HCI and crowdfunding approaches. In *New directions in music and human-computer interaction* (pp. 179–212). Springer International Publishing.
McPherson, A., Morrison, L., Davison, M., & Wanderley, M. M. (2024). On mapping as a technoscientific practice in digital musical instruments. *Journal of New Music Research*, *53*(1-2), 110–125.
Miranda, E. R., & Wanderley, M. (2006). *New digital musical instruments: Control and interaction beyond the keyboard (computer music and digital audio series)*. A-R Editions, Inc.
Morreale, F., & McPherson, A. (2017). Design for longevity: Ongoing use of instruments from NIME 2010-14. *Proceedings of the International Conference on New Interfaces for Musical Expression*, 192–197.
Pigrem, J., & McPherson, A. P. (2018). Do we speak sensor? Cultural constraints of embodied interaction. In T. M. Luke Dahl Douglas Bowman (Ed.), *Proceedings of the international conference on new interfaces for musical expression* (pp. 382–385). Virginia Tech.
Privato, N., Magnusson, T., & Einarsson, E. T. (2023). Magnetic interactions as a somatosensory interface. In M. Ortiz & A. Marquez-Borbon (Eds.), *Proceedings of the international conference on new interfaces for musical expression* (pp. 387–393).
Schkolne, S., Pruett, M., & Schröder, P. (2001). Surface drawing: Creating organic 3D shapes with the hand and tangible tools. *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems*, 261–268.
Shneiderman, B. (2007). Creativity support tools: Accelerating discovery and innovation. *Commun. ACM*, *50*(12), 20–32.
Sutherland, I. E. (1968). A head-mounted three dimensional display. *Proceedings of the December 9-11, 1968, Fall Joint Computer Conference, Part i*, 757–764.
Tanaka, A. (2010). Mapping out instruments, affordances, and mobiles. *Proceedings of the International Conference on New Interfaces for Musical Expression*, 88–93.
Trajkova, M., Deshpande, M., Knowlton, A., Monden, C., Long, D., & Magerko, B. (2023). AI meets holographic pepper’s ghost: A co-creative public dance experience. *Companion Publication of the 2023 ACM Designing Interactive Systems Conference*, 274–278.
Trajkova, M., Long, D., Deshpande, M., Knowlton, A., & Magerko, B. (2024). Exploring collaborative movement improvisation towards the design of LuminAI—a co-creative AI dance partner. *Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems*.
Wang, G. (2014). Ocarina: Designing the iPhone’s magic flute. *Computer Music Journal*, *38*(2), 8–21.
Wang, Y., & Martin, C. P. (2022). Cubing Sound: Designing a NIME for Head-mounted Augmented Reality. *International Conference on New Interfaces for Musical Expression*.
Wessel, D., & Wright, M. (2002). Problems and prospects for intimate musical control of computers. *Comput. Music J.*, *26*(3), 11–22.