Movement towards studying tools to support creativity.

there is a move from routine work and productive concerns to human and creative ones.” (Edmonds, 2018)

Expression and creativity let us get inside the process of an interaction. People are interested in expressive experiences, leading to critique and understanding.

1. Support exploration.
2. Low threshold, high ceiling, and wide walls.
3. Support many paths and many styles.
4. Support collaboration.
5. Support open interchange.
6. Make it as simple as possible—and maybe even simpler.

7. Choose black boxes carefully.
8. Invent things that you would want to use yourself.
9. Balance user suggestions with observation and participatory processes.
10. Iterate, iterate—then iterate again.
11. Design for designers.
12. Evaluate your tools.

Much expressive-interface research is people studying their own practice. HCI calls these first-person methods: “qualitative research approaches that turn to the researcher as the subject of inquiry” (Desjardins et al., 2021).

• Autoethnography: your own experience of a phenomenon is the data, written with reflexivity and a personal voice.
• Autobiographical design: build a system for yourself as a user with a genuine need, then refine it through your own use over time.
• Soma design: the designer’s lived, moving body is the design material. “There is no short-cut, no simplified wireframes” (Höök et al., 2018).

Mapping sensed gestures to an expressive output that is fed back to the user.

• gestures: the use of motions by the limbs or body as a means of expression
• can be unintentional, control, or ancillary gestures
• from non-human actors (e.g., the movement of leaves on a branch of a tree)
• “any sort of motion, that may be understood as an expression of something”

The interaction itself is expressive, and the output is an expression as well. We consult Composing Interactions (Baalman, 2022) as a resource.

Why is mapping an important consideration?

Considering a performer performing making gestures on a stage, which gestures effect changes in the output medium of sound, which can be heard by the performer and audience in the real, physical environment.

Baalman (2022) expands the mapping process into a cycle.

• A gesture is performed in the environment;
• This is captured by a sensor that translates the gesture is processed by an electronic circuit, often to digitalise it;
• Next, the signal enters some sort of computational model that translates the data to parameters;
• These parameters control an output medium such as sound, light, video, or mechatronics.

how is output from one connected to input? what happens in each step?

• AirPens: Musical Doodling (Frame et al., 2025), an installation at NIME 2025.
• Make music with mark-making.
• Use IMU sensors to convert movement into sound; explore different mappings between movement and sound.

App for making short (5s) musical performances with a sketch (video and info). (Martin & Torresen, 2020)

• Uses touch location and movement to create sound.
• Replay performances by rewinding the sketch and viewing it again.
• Social media interface: view other people’s drawing and add layers as a musical “reply”
• Analysed 1600 tiny jams to understand musical drawing behaviour.

Jordà’s group at Pompeu Fabra made a synthesiser into a tangible tabletop: pucks, a camera underneath, projected lines showing what feeds what (Jordà, 2010).

• Prior work: tangible tabletops (Ishii & Ullmer, 1997) and a decade on collaborative visual instruments (Jordà, 2003).
• Had a theory: learnable by novices, deep for experts, and shared between players
• Kept going: iterated for years and toured, with the tracking software released as open source.
• Then got lucky: demos “watched millions of times” as YouTube took off, Björk’s 2007 world tour.

Jordà’s retrospective, with an expert commentary by your lecturer: (Jordà, 2017).

Ge Wang had an iPhone: mic, multitouch, accelerometer. The efficient design is tap-a-note: instant, easy to learn, hard to get wrong.

• What he did: blow into the mic to sound a note, cover four holes for pitch, tilt for vibrato. Harder, slower, and you will play wrong notes (G. Wang, 2014).
• What happened: ten million users (with its sequel) and the App Store Hall of Fame; the rationale came out in Computer Music Journal six years later.

How can we perform music on big touchscreens?

• Rather than re-implement music production apps, looked at natural gestures in long-term artistic practice.
• Lots of apps from 2011–2014 and lots of performances e.g., Martin (2016)
• Initially, sought to understand how percussionists would play touchscreens (C. Martin et al., 2014)
• Then explored networked connections in ensemble performance (C. Martin et al., 2015)
• Then compared how networked ensembles supported by the interface (C. Martin et al., 2016)

• When is a Guitar not a Guitar? (Harrison et al., 2018): novel instruments and controllers resemble traditional instruments.
• Four designs examining variation in form (held vs tabletop) and interaction (strings vs touch sensor)
• Guitarists prefer the technical familiarity of a stringed instrument
• Non-musicians prefer touch-interface: ease of use, cultural load of the guitar form

cube system: authentic design for a head-mounted AR musical instrument (Y. Wang & Martin, 2022)

Autobiographical design (Desjardins et al., 2021) to compare three interface candidates

1. Physical interface: slow accurate manipulation due to hand tracking.
2. Spatial interaction: emerged from bodily movement allowing ease of use.
3. Flexible freehand interaction: allow multiple notes to be played simultaneously, taking advantage of the full-hand tracking affordance in the AR headset.

• Traditional approach: decompose input actions, sound production and feedback (Miranda & Wanderley, 2006; Wessel & Wright, 2002).
• New sounds, music, and experiences: expanded control over timbre and texture, richer and more diverse musical experience (Magnusson, 2010; Wessel & Wright, 2002).
• Control intimacy: mediated gesture and sound relation (Wessel & Wright, 2002), a limit of the machine-centred approach (Pigrem & McPherson, 2018).
• Current research: finding better frameworks. Performer and instrument engage in a mutually influential relationship during the design process (McPherson et al., 2024).

• The Wekinator (Fiebrink et al., 2009) turns mapping to demonstration. Show the system an input and the output you want, train, play, and retrain when it feels wrong.
• Evaluation by playing. Fiebrink’s CHI paper (Fiebrink et al., 2011) found practitioners judged models by how they felt under the hands, and often preferred a “worse” model.
• The demo: a webcam, a coloured object or a face, a synth. From Fiebrink’s teaching rather than in the early papers (Fiebrink, 2019).
• A regression from two camera coordinates to some synth params is simple to implement in 2026.

The Reactable and Ocarina are the success stories. What happened to everything else?

Morreale and McPherson wrote to the authors of every NIME 2010 to 2014 paper introducing an instrument; 70 of 97 replied (Morreale & McPherson, 2017). Of 51 built for prolonged performance:

• 47.1% not ready to perform on
• of those, two thirds needed more than a few hours’ work, and 15.7% no longer existed at all

Top reasons: “I don’t have the opportunity at the moment” (22), “I turned my attention to building other DMIs” (14).

McPherson, Morreale and Harrison compared instruments from NIME, HCI, and Kickstarter (McPherson et al., 2019).

• Campaigns say: 16 of 30 pitch ease of use. Play “any sound you can imagine”.
• Designs do: 22 of 30 are MIDI controllers, made “easier” by limiting pitch space or quantising time, which decides in advance what you can play.
• Concept: “easy” means someone else chose your constraints (Magnusson, 2010).

• Movements are measured using motion sensors, and the live coder processes motion signals to generate feedback in real-time.
• Enable a multitude of feedback loops: sound feedback -> movement improvisation -> the coder alters the relationships between movement and sound.
• Dynamic improvisation that stimulates novel movements’ exploration.

Dinosaur Choir: Adult Corythosaurus (Brown & Gajewski, 2025)

• Singing dinosaur skull musical instruments.
• Experience dinosaur vocalisation: imagine dinosaur vocal anatomy from a bird syrinx, (vocal structure open question in paleontology)
• Microphone for user, computational vocal model, sound resonates through a 3D printed replica of the dinosaur’s nasal cavities and skull
• Change the pitch and timbre of the vocalisation by changing the shape of the mouths, like trumpet player

official website

• Interactive sculpture in Canberra by Kerry Simpson (1988)
• Glass and sound (now movement) activated lighting
• Location: Ainslie Avenue, Canberra

Can an amusement ride be controlled by breath?

• Inspired by robotic technologies for control of individual seats on rollercoasters and other thrill rides.
• the Broncomatic is a bucking bronco game (horse riding; you try to stay on!)
• The twist is it’s controlled by the user’s breath
• the ride kicks you around and if you lose control it kicks harder (until you fall off)

“Breath Control of Amusement Rides” (Marshall et al., 2011)

• A straightforward mapping: the rider’s breathing to the horizontal rotation of the ride.
  • Inhale: spins clockwise; exhale: spins anti-clockwise.
  • Breathing speed controls rotation speed: fast breathing faster spin; holding breath stops spinning.
  • Difficulty levels.
• The program is a game in which the player scores more points the more that they breathe: a physical challenge vs reward dynamic.
  • More breathing for faster ride but harder to stay on. To score high, you must breathe more, but this makes the ride tougher.

Computational creativity helps create new ideas in three ways (Boden, 1998)

• Produce novel combinations of familiar ideas;
• Explore the potential of conceptual spaces;
• Make transformations that enable the generation of previously impossible ideas.

Creativity and technology: a sociotechnological perspective (Bown, 2021).

• The social nature of human behaviour.
• Artistic behaviour is social in nature (is it?)

Cobbie system (Lin et al., 2020)

• Motivation: Co-creative partner can reason about user’s intention and stably present novel ideas with the user initiative, which are hindered in human team due to social loafing or a resolute partner otherwise.
• Take turns to draw ideas, give the dominant position to the user, and use movements and sound feedback for communication.
• Three human-robot interactions: your turn, pause and draw again, progressing with feedback

• Co-creative public dancer (Long et al., 2019; Trajkova et al., 2023, 2024)
• Explore the design of the modular AI agent to creatively collaborate with a dancer.
• A Kinect motion capture device to detect the user’s motion, visualised as a virtual shadow on a projection screen.
• The humanoid agent shadow dances by analysing the user’s movement and responding with a movement that it deems to be similar in terms of parameters such as energy, tempo, or size.
• Study results showed in-the-moment influences, self, partner, environment(Trajkova et al., 2024).

Who has a question?

• I can take catchbox question up until 2:55
• For after class questions: meet me outside the classroom at the bar (for 30 minutes)
• Feel free to ask about any aspect of the course
• Also feel free to ask about any aspect of computing at ANU! I may not be able to help, but I can listen.