• search engine for scholarly sources
• add more search terms to get more specific
• you can use the “time” selector on the left column to find recent work

• ACM Digital Library archives proceedings of ACM conferences and journals.
• not as good at searching, but will show you only peer-reviewed works

• metadata for a reference can be used with any citation format
• academics often use special tools for storing reference metadata
• computer scientists tend to use BibTeX format
• BibTeX is part of the venerable (La)TeX ecosystem originally developed by CS luminary Donald Knuth in the late 70s
• in LaTeX you can select citation format (e.g., ACM, IEEE, Chicago, Harvard, APA)

@inproceedings{adiwangsa-charades:2025,
  author = {Adiwangsa, Michelle and Bransky,
    Karla and Wood, Erika and Sweetser, Penny},
  title = {A Game of ChARades: Using Role-Playing
    and Mimicry with and without Tangible Objects
    to Ideate Immersive Augmented Reality
    Experiences},
  year = {2025},
  isbn = {9798400714863},
  publisher = {Association for Computing
    Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3715668.3736382},
  abstract = {Using tangible objects for
    immersive augmented reality (AR) ...},
  booktitle = {Companion Publication of the 2025
    ACM Designing Interactive Systems Conference},
  pages = {440–445},
  numpages = {6}
}

The entry for Adiwangsa et al. (2025), cited as [@adiwangsa-charades:2025] in Markdown or \cite{adiwangsa-charades:2025} in LaTeX.

Big reference libraries are part of academic work.

• Charles’ “main” .bib file has ~1300 entries (started in 2013, first year of Charles’ PhD!)
• references.bib for this course has about 100 entries.
• Charles uses BibDesk to manage the big file, but usually just VSCode for smaller bibtex libraries.
• Other popular reference managers are Zotero, Mendeley and Endnote.
• Opinion time: Bibdesk is nice because the database is just a text file.

• Not expecting you to use BibTeX (yet)
• Expecting you to list references in “ACM format”.
• ACM format is inspired by APA referencing
  • supports both numerical [1, 2] and author-date (Martin, 2020) referencing.
  • includes full names of authors and publication details for clarity
  • includes DOI as a full URI
• ACM page plain text examples for references to different types of source
• CSL version (Citation Style Language)
• BST version (BibTeX Style)

There are many interfaces, but only a handful of ways of interacting with them (Rogers et al., 2023):

• Instructing: issue commands
• Conversing: having a dialogue
• Manipulating: acting on objects
• Exploring: moving through a space
• Responding: being addressed by the system

• Command Line
• Graphical
• Multimedia
• Virtual reality
• Web
• Mobile
• Appliance
• Voice
• Pen
• Touch
• Touchless

• Haptic
• Multimodal
• Shareable
• Tangible
• Augmented reality
• Wearables
• Robots and drones
• Brain-computer
• Smart
• Shape-changing
• Holographic

What do these interfaces let you do, and how do you know?

• Affordance: a relationship between what a person can do and what a thing allows (not just a property of the interface) (Norman, 2013).
• Signifier: the perceivable cue that the affordance is there. Buttons look pressable. A prompt invites typing.
• Constraint: what the interface stops you doing. Constraints can give character (Magnusson, 2010).

Gaver (1991) organised affordances by whether you can perceive them:

Hidden: A shortcut you discover.

False: A button that’s too small to hit.

Let’s do it live on something easy…

• Type in commands (e.g., ls) or key combos (e.g., :q!)
• largely superseded, but useful for complex software (e.g., CAD), scripting, coding
• 1980s research on command form (abbreviations, familiar names), syntax and organisation found no universal optimum for naming (Shneiderman, 1992)

• Information represented within a graphical interface
• use of color, typography, and imagery (Mullet & Sano, 1995)
• interface features abbreviated as WIMP (Windows, Icons, Menus, Pointer)
• window management
  • enabling movement between windows and displays without distraction
  • keyboard shortcuts and task bars design; auto-fill in online forms
• consistent menu design and icon libraries

One interface combines different media (graphics, text, video, sound…) linked via interactivity

• better information presentation; developed for training, education and entertainment
• dynalinking: changes in one window directly update another (Rogers & Aldrich, 1996)
• design question: how do you get people to interact with all of the media, not just the video?

Interfaces can sit on a spectrum between fully virtual and fully real interaction (Milgram & Kishino, 1994).

• The big middle area includes “mixed reality” (MR) interfaces
• Augmented reality usually closer to “real” reality.
• eXtended reality (XR) is a more recent term.

Early AR

• blending of digital content with the physical world to create an enhanced real-world experience
• 1960s: Ivan Sutherland’s development of the first head-mounted three-dimensional display

• AR systems have evolved significantly, particularly displays and interaction models (Billinghurst et al., 2015; Speicher et al., 2019)
• displays: see-through, screen-based, projection-based.
• “spatial Computing”, another way of thinking about it, defined by Greenwold (2003)
  • “human interaction with a machine in which the machine retains and manipulates referents to real objects and spaces.”
• emphasising not only the augmentation of reality but also the meaningful interaction between digital and physical elements.

Emerged in 1970s with computer-generated graphical simulations.

Goal: to create user experiences that feel real when interacting with an artificial environment.

• stereoscopically displayed immersive image.
• variety of input devices
• used for:
  • training (e.g., pilots, medicine, first responders).
  • treatment (e.g, mental health)
  • entertainment.

• early websites: text and hyperlinks; then a shift to aesthetics and visual design
• breadcrumb nav: navigate without losing track
• now designed for phones, touch and infinite scrolling
• Keith Instone’s core questions: “Where am I? What’s here? Where can I go?” (Veen, 2000)
• most content is not read: “a billboard going by at 60 miles an hour” (Krug et al., 2014)

Smartphones, fitness trackers, smartwatches, large-sized tablets on the flight, educational tablets, etc.

• embedded sensors: movement, GPS, bio/fitness sensors.
• affordances led to novel and creative apps e.g.,
  • Ocarina (G. Wang, 2014)
  • contextual information access: scanning QR codes
• careful design of limited screen and control space
• touch target size crucial for touch small touch screens
• guidelines exist mobile interface design, e.g., (Babich, 2018).

• machines for everyday use in the home: washing machines, microwaves, fridges, toasters, blenders
• Cooper et al. (2014): appliance interfaces are transient; the interaction is short and infrequent, so nothing can depend on learning
• two design principles: simplicity and visibility
• compare the two knobs on the microwave with the twenty programs on the washing machine
• internet connections and remote apps: do they help the transient use case, or add a second, hidden interface?

• interact through spoken language: request information (flight times, weather) or issue commands (play music, set a timer)
• command-based or conversation-based; the system decides how human-like to pretend to be
• design choices: natural conversation vs directed dialogue, can the user interrupt (“barge-in”), synthesised voice or actor, gender, dialect, pronunciation
• dialogue structure, error handling and etiquette are still the core problems (Cohen et al., 2004)
• early systems earned a reputation for mishearing (still true?)
• almost no perceptible affordances: how do users learn what they can say?

• light pens, styluses, or scanners for drawing and writing
• write, draw, select, and move objects on a page or tablet.
• very early use in CRT displays with lightpen!
• touchscreen versions most common now
• another option: digital paper that scans what you write (IR sensor, special dot pattern on the paper)

Pens are great! We should do more with them.

Two pen-input handhelds, three years apart, with opposite answers.

• Newton MessagePad (1993): recognises your ordinary handwriting. The machine learns your style.
• Palm Pilot (1996): Graffiti, a one-stroke alphabet from Xerox PARC’s Unistrokes (Goldberg & Richardson, 1993). The user learns a new alphabet.

Newton MessagePad 100 and PalmPilot Professional. Photos: F. Winkelnkemper (CC BY-SA 4.0), Letdorf (CC BY-SA 3.0), Wikimedia Commons.

Apple’s sounds like the better idea, but it’s considered a failure.

• Newton misread handwriting often enough to become a joke: a week of Doonesbury (“Catching on?” became “Egg freckles”) (Trudeau, 1993) and the Simpsons’ “Eat up Martha”.
• Later models fixed the recognition, but the reputation didn’t recover and Apple killed the line in 1998.
• Graffiti: 86% after a minute with the chart, 97% after five minutes, still 97% a week later (MacKenzie & Zhang, 1997).

• common in kiosks, ATMs, checkouts, phones, tablets, computers.
• Now weird to find a non touchscreen.
• Single-touch respond to single taps.
• Multitouch supports multiple simultaneous touches and gestures like pinching, rotating
• Multitouch devices (e.g., smartphones, tablets, tabletops) enable intuitive interactions using one or both hands.
• Finger gestures enhance how users interact
• but there are limitations. How do you “undo” on an iPhone?

• vibration and forces (via actuators) give tactile feedback: phones, watches, game controllers, simulators, clothing
• skill training: the MusicJacket nudges a violinist’s bowing arm into position, and “novice players responded well to vibrotactile cues, adjusting their actions accordingly.”
• remote touch: actuators in clothing can simulate hugs or squeezes
• mid-air haptics: ultrasound creates buttons and sliders that can be felt but not seen (e.g., in-car controls)
• haptic exoskeletons: “Techno Trousers” (Rossiter, 2018) that stiffen or relax to assist movement
• perceptible to the hand, invisible to the eye: run the affordance questions on that!

• multiple input/output modalities (e.g., touch, sight, sound, speech) to enhance user interaction (Bouchet & Nigay, 2004);
• natural, flexible, and expressive interactions, similar the real world interaction experience (Oviatt et al., 2017).
• Common combinations: speech and gesture; eye-gaze and gesture; haptic and audio, pen input and speech (Dumas et al., 2009).
• Speech + vision processing is the most common (Deng & Huang, 2004)!
• most researched interaction modes are: speech, gesture, eye-gaze tracking.

Design guidelines, see (Oviatt et al., 2017).

Designed for multi-user interaction by co-located groups, unlike single-user PCs, laptops and phones: large wall displays, interactive tabletops.

• everyone can point, touch, and see the same content
• a shared point of reference improves group coordination and participation (Rogers et al., 2023)
• used in museums and galleries; supports group-based learning (Clegg et al., 2020)
• software shareables (Google Docs, Miro) move the same idea to remote collaboration
• design issues: display layout, balancing participation, and personal vs shared space

• physical objects (bricks, cubes, clay) linked to digital representations through embedded sensors (Fishkin, 2004; Ishii & Ullmer, 1997)
• manipulating the object triggers sound, light or animation, on the object or around it; or objects on a tabletop drive a simulation (Urp for urban planning)
• artefacts can be lifted, rearranged and handled directly, unlike anything on a screen
• the design problem is coupling: how does a physical action map to a digital effect, and where does the feedback appear? (Shaer et al., 2010; Ullmer et al., 2005)
• prototype with sticky notes and cardboard tokens first
• history and goals: Ullmer et al. (2022)

• Digital devices worn on the body – e.g., smartwatches, fitness trackers, smart glasses, and fashion tech.
• Early wearable computing enabled mobile recording and access to digital info.
• Advances in flexible displays, e-textiles, and physical computing (e.g., Arduino) have made wearables more practical and appealing.
• Items like jackets, jewelry, hats, and shoes have been designed to interact with digital info on the go.
• From convenience design focus to expressive and communicative functions.

Robots assist in manufacturing, hazardous environments, rescue, and remote exploration with remote controls using cameras and sensors.

Domestic robots help with chores and support elderly or disabled people.

Pet robots like Paro provide companionship and reduce loneliness, especially for dementia patients.

Drones, once military and hobbyist tools, now serve in delivery, entertainment, agriculture, construction, and wildlife monitoring, offering real-time data and automated operations.

Three types where the design knowledge is still forming:

• Shape-changing: physical form as input and output (Alexander et al., 2018). Do moving rods beat a chart on a screen?
• Holographic: the illusion of a 3D person in the room (ABBA Voyage, 2022). How do you interact with a projection, and how do you keep it from being creepy?
• Brain-computer: electrodes on the scalp drive a cursor, robot or game; assistive uses (BrainGate) and playful ones (Brainball). Ethics of mind reading get sharper with implants (Neuralink).

More on each in the interface considerations page.

A natural user interface (NUI) is designed to allow people to interact with a computer in the same way that they interact with the physical world—using their voice, hands, and bodies.

• but how natural are NUIs? (the Newton was the “natural” one)

Don Norman (Norman, 2013) argues “natural” depends on a number of factors:

• how much learning is required,
• the complexity of the app or device’s interface,
• and whether accuracy and speed are needed.

A gesture may be worth a thousand words; other times a word is worth a thousand gestures.

Every interface sends one of six commands to the same engine, so the only thing that changes is the interaction.

• Pads (manipulating): tap a sound
• CLI (instructing): loop kick x...x...
• Camera (exploring): move a hand into a region

Who has a question?

• I can take catchbox question up until 2:55
• For after class questions: meet me outside the classroom at the bar (for 30 minutes)
• Feel free to ask about any aspect of the course
• Also feel free to ask about any aspect of computing at ANU! I may not be able to help, but I can listen.