Movement towards studying tools to support creativity.

there is a move from routine work and productive concerns to human and creative ones.” (Edmonds, 2018)

Expression and creativity let us get inside the process of an interaction. People are interested in expressive experiences, leading to critique and understanding.

1. Support exploration.
2. Low threshold, high ceiling, and wide walls.
3. Support many paths and many styles.
4. Support collaboration.
5. Support open interchange.
6. Make it as simple as possible—and maybe even simpler.

7. Choose black boxes carefully.
8. Invent things that you would want to use yourself.
9. Balance user suggestions with observation and participatory processes.
10. Iterate, iterate—then iterate again.
11. Design for designers.
12. Evaluate your tools.

Mapping sensed gestures to an expressive output that is fed back to the user.

• gestures: the use of motions by the limbs or body as a means of expression
• can be unintentional, control, or ancillary gestures
• from non-human actors (e.g.,the movement of a leaves on a branch of a tree)
• “any sort of motion, that may be understood as an expression of something”

The interaction itself is expressive, and the output is an expression as well. We consult Composing Interactions (Baalman, 2022) as a resource.

Why is mapping an important consideration?

Considering a performer performing making gestures on a stage, which gestures effect changes in the output medium of sound, which can be heard by the performer and audience in the real, physical environment.

Baalman (2022) expands the mapping process into a cycle.

• A gesture is performed in the environment;
• This is captured by a sensor that translates the gesture is processed by an electronic circuit, often to digitalise it;
• Next, the signal enters some sort of computational model that translates the data to parameters;
• These parameters control an output medium such as sound, light, video, or mechatronics.

how is output from one connected to input? what happens in each step?

• AirPens: Musical Doodling
• Make music with mark-making.
• Use IMU sensors to convert movement into sound; explore different mappings between movement and sound.

App for making short (5s) musical performances with a sketch (video and info). (Martin & Torresen, 2020)

• Uses touch location and movement to create sound.
• Replay performances by rewinding the sketch and viewing it again.
• Social media interface: view other people’s drawing and add layers as a musical “reply”
• Analysed 1600 tiny jams to understand musical drawing behaviour.

How can we perform music on big touchscreens?

• Rather than re-implement music production apps, looked at natural gestures in long-term artistic practice.
• Lots of apps from 2011–2014 and lots of performances e.g., Martin (2016)
• Initially, sought to understand how percussionists would touchscreens (C. Martin et al., 2014)
• Then explored networked connections in ensemble performance (C. Martin et al., 2015)
• Then compared how networked ensembles supported by the interface (C. Martin et al., 2016)

• When is a Guitar not a Guitar? (Harrison et al., 2018): novel instruments and controllers resemble traditional instruments.
• Four designs examining variation in form (held vs tabletop) and interaction (strings vs touch sensor)
• Guitarists prefer technical familiarity of stringed instrument (guitarists)
• Non-musicians prefer touch-interface: ease of use, cultural load of the guitar form

cube system: authentic design for a head-mounted AR musical instrument (Wang & Martin, 2022)

Autobiographical design (Desjardins et al., 2021) to compare three interfaces candidates

1. Physical interface: slow accurate manipulation due to hand tracking.
2. Spatial interaction: emerged from bodily movement allowing ease of use.
3. Flexible freehand interaction: allow multiple notes to be played simultaneously, taking advantage of the full-hand tracking affordance in the AR headset.

• Traditional approach: decompose input actions, sound production and feedback (Miranda & Wanderley, 2006; Wessel & Wright, 2002).
• New sounds, music, and experiences: expanded control over timbre and texture, richer and more diverse musical experience (Magnusson, 2010; Wessel & Wright, 2002).
• Control intimacy: mediated gesture and sound relation (Wessel & Wright, 2002), a limit on machine-centre approach(Pigrem & McPherson, 2018).
• Current research: finding better frameworks. Performer and instrument engage in a mutually influential relationship during the design process (McPherson et al., 2024).

• Movements are measured using motion sensors, and the live coder processes motion signals to generate feedback in real-time.
• Enable a multitude of feedback loops: sound feedback -> movement improvisation -> the coder alters the relationships between movement and sound.
• Dynamic improvisation that stimulates novel movements’ exploration.

Dinosaur Choir: Adult Corythosaurus

• Singing dinosaur skull musical instruments.
• Experience dinosaur vocalisation: imagine dinosaur vocal anatomy from a bird syrinx, (vocal structure open question in paleontology)
• Microphone for user, computational vocal model, sound resonates through a 3D printed replica of the dinosaur’s nasal cavities and skull
• Change the pitch and timbre of the vocalisation by changing the shape of the mouths, like trumpet player

official website

• Interactive sculpture in Canberra by Kerry Simpson (1988)
• Glass and sound (now movement) activated lighting
• Location: Ainslie Avenue, Canberra

Can an amusement ride be controlled by breath?

• Inspired by robotic technologies for control of individual seats on rollercoasters and other thrill rides.
• the Broncomatic is a bucking bronco game (horse riding; you try to stay on!)
• The twist is it’s controlled by the user’s breath
• the ride kicks you around and if you lose control it kicks harder (until you fall off)

“Breath Control of Amusement Rides” (Marshall et al., 2011)

• A straightforward mapping: the rider’s breathing to the horizontal rotation of the ride.
  • Inhale: spins clockwise; exhale: spins anti-clockwise.
  • Breathing speed controls rotation speed: fast breathing faster spin; holding breath stops spinning.
  • Difficulty levels.
• The program is a game in which the player scores more points the more that they breathe: a physical challenge vs reward dynamic.
  • More breathing for faster ride but harder to stay on. To score high, you must breathe more, but this makes the ride tougher.

Computational creativity helps create new ideas in three ways (Boden, 1998)

• Produce novel combinations of familiar ideas;
• Explore the potential of conceptual spaces;
• Make transformations that enable the generation of previously impossible ideas.

Creativity and technology: a sociotechnological perspective (Bown, 2021).

• The social nature of human behaviour.
• Artistic behaviour is social in nature (is it?)

Cobbie system (Lin et al., 2020)

• Motivation: Co-creative partner can reason about user’s intention and stably present novel ideas with the user initiative, which are hindered in human team due to social loafing or a resolute partner otherwise.
• Take turns to draw ideas, give the dominant position to the user, and use movements and sound feedback for communication.
• Three human-robot interactions: your turn, pause and draw again, progressing with feedback

• Co-creative public dancer (Long et al., 2019; Trajkova et al., 2023, 2024)
• Explore the design of the modular AI agent to creatively collaborate with a dancer.
• A Kinect motion capture device to detect the user’s motion, visualised as a virtual shadow on a projection screen.
• The humanoid agent shadow dances by analysing the user’s movement and responding with a movement that it deems to be similar in terms of parameters such as energy, tempo, or size.
• Study results showed in-the-moment influences, self, partner, environment(Trajkova et al., 2024).

Who has a question?

• I can take cathchbox question up until 2:55
• For after class questions: meet me outside the classroom at the bar (for 30 minutes)
• Feel free to ask about any aspect of the course
• Also feel free to ask about any aspect of computing at ANU! I may not be able to help, but I can listen.