Evaluation

Dr Charles Martin

Announcements

assignment 2 due next Tuesday (6 October — Monday is a public holiday)
remember to use your tutorial time for meeting research clusters and collecting data.
remember to follow the step-by-step guide.
go fork the repo!

Markdown Formatting Check: There is a CI/CD job that checks your markdown formatting using the markdownlint-cli tool. Syntax rules are listed here in our script, rules MD013 and MD041 are disabled. All other rules are active.

Plan for the class

research questions
about evaluation: from a question to a yardstick
types of evaluation
planning evaluations
evaluation by inspection, with a case study: the Hawaii false missile alert
activity: two ways to evaluate nanojam

Research Questions

For the final project you will have to choose a research question to explore.

This is a clear question (one sentence) that guides the design of your research project.

RQs have been called survival beacons because they should guide all aspects of our research plans.

How do we choose a research question and write it clearly?

Important skill for any research activity.

Four steps to write a research question

This framework inspired by Lennart Nacke, everybody’s favourite HCI writing coach on LinkedIn.

Outline a broad area of interest
Identify a problem that needs solving
Justify solving this problem
Write the question

To be clear, a research question starts with a question word (what, how, why, can, do, should) and ends with a question mark. It can just be one sentence.

Seems too easy… let’s try it together.

Lennart Nacke’s 4-step research question framework

A worked research question example

haptic wearable interfaces.
keeping focussed in complicated meetings
lack of awareness in meeting can lead to poor work performance and embarrassment
here we go:

What effects can a haptic wearable interface have on lack of awareness during meetings and later work performance?

Encodes the broad area, the problem, the justification, the context, etc.

Is the problem worth solving?

The four steps get you a well-formed question. They don’t tell you if it’s a question worth asking.

Two useful ways to find a gap (Hornbæk et al., 2025, Section 45.2):

Knowledge gap: what do we not understand yet, given our present understanding?
Capability gap: what are we not able to do yet?

Try working backwards: instead of starting with a problem and then asking who cares, start from what people need to understand or do better.

“Good research produces results that improve stakeholders’ capacity to solve important problems.”

Three ways a research question goes wrong

1. Wrong level of abstraction. Too specific and almost nobody is affected:

Why do aging users make errors in Microsoft Word 17.3 when sitting near a pool in Belize?

Raise it until it covers many people — but not so far that it can’t be answered in one study.

Three ways a research question goes wrong (2)

2. Confounded with a solution. This one has the method baked in before any thinking has happened:

What does the interview method tell us about children’s motivation to use tablets in schools?

Better — solution-neutral, so the method is still an open choice:

What motivations do children have for using tablets in schools?

Three ways a research question goes wrong (3)

3. Not refutable. A question that presupposes its own answer isn’t a question.

Very easy to do accidentally when you’re attached to a pet theory, or want to show your system beats the baseline. Your RQ should allow more than one answer — including one you won’t like.

And on novelty:

“Off-hand comments about a problem being novel are almost always wrong.” (Hornbæk et al., 2025, Section 45.2)

Nearly every HCI problem has precursors. Finding them is your job, not an optional extra.

Research Question Bingo

Interfaces:

haptic feedback gloves
AR/VR headset
e-textile clothing
voice assistant
gesture recognition
smart headphones
ambient light display
wearable plants
eye-tracking interface
multi-touch table

Problems to Solve:

family meal planning
language learning while commuting
caring for houseplants
focus during remote work
medication schedules
teaching kids about conservation
organising hobby collections
non-verbal communication
practicing music in small spaces
tracking community events

Activity: Write a Research Question

Do: Let’s write a research question!

Together, let’s Spin the wheels to decide on a broad area and a problem.

Then, decide on a “justification” and write a research question. Remember that the RQ should include the broad area, the problem, and the justification.

Use the poll everywhere link to suggest research questions and vote on the best ones.

Write for 2-3 minutes, vote for 1 minute, then let’s discuss.

Extension: Before you vote, run the candidates past the three failure modes: is the abstraction right, is a solution smuggled in, and is it refutable?

PollEverywhere link: https://pollev.com/charlesmarti205

About Evaluation

Where we are in the course, following Rogers et al. (2023):

Week 4: how to collect data (interviews, questionnaires, observation, think-aloud, logs)
Weeks 5 and 6: how to analyse it
This week: how to design the study — which setting, which kind of study, which variables, and whether you need users in the room at all

Same collection and analysis techniques as needs-finding. What changes is that now there is an artefact, and a question about it.

What is evaluation?

Evaluation: collecting and analysing data from user experiences with an artefact.
Goal: to improve the artefact’s design.
Addresses: functionality, usability, user experience
Appropriate for all different kinds of artefacts and prototypes.
Methods vary according to goals.

Evaluating iPad apps in 2013 (Martin et al., 2014)

Why is evaluation important?

Understanding people
- Users may not have the same experiences or perspectives as you do
- Different users use software differently
Understanding designs
- Proof that ideas work
- Understand limitations, affordances, applications

Business
- Invest in the right ideas
- Find problems to solve (before production, before next iteration, etc.)
Research
- Evidence for new interactive systems
- Empirical proof of hypotheses
- New knowledge to answer research questions

What should you evaluate/measure?

Does the design do what the users need and want?

Examples:

Game App Developers: Whether young adults find their game fun and engaging compared to other games
Government authority: Whether their online service is accessible to users with a disability
Children’s talking toy designers: Whether six-year-olds enjoy the voice, feel of the soft toy, and can use safely

Usability and Usability Goals

Six usability goals:

Effective to use (effectiveness)
Efficient to use (efficiency)
Safe to use (safety)
Having good utility (utility)
Easy to learn (learnability)
Easy to remember how to use (memorability)

(Rogers et al., 2023)

Where should you evaluate your design?

Depends on your evaluation goal!

Lab studies (controlled settings)
In-the-wild studies (natural settings)
Remote studies (online behaviour)

Formative vs Summative Evaluation

When you evaluate changes what the evaluation is for (Rogers et al., 2023, Section 14.2):

Formative evaluation:
- during design, checking the product still meets users’ needs
- from early sketches to a nearly finished design
- feeds the next iteration
Summative evaluation:
- assessing whether a finished product is successful
- against a target: a usability goal, a baseline, a standard

For the final project you will almost certainly be doing a formative evaluation of a prototype.

Formative vs Summative Evaluation NNGroup on YouTube

Overview: from a goal to a yardstick to a study

Set the yardstick (what counts as “good”) first, and the kind of study follows (Hornbæk et al., 2025, table 40.2).

Yardstick	The system should…	Kind of study
No critical usability problems	not make users err, get lost, or give up	think-aloud study
Comply with guidelines	match known traits of good systems	heuristic evaluation
Meet usability goals	hit quantitative targets	summative usability test
Compare favourably to X	beat a baseline on some measure	experiment
Compatibility with user practices	be adopted into everyday life	field study

No data-collection column: that was week 4. The yardstick picks the study; the study picks the techniques.

Example: three questions about one app

nanojam (week 7): three questions, three yardsticks, three studies.

Can a first-time user make a repeating pattern in the command line without help? No critical usability problems. Think-aloud in the lab, six people, log where they stall.
Is the pads interface faster to a first pattern than the command line? Compare favourably to X. Experiment, same participants on both, counterbalanced, time to first pattern.
Does anyone keep using it after the lecture? Compatibility with user practices. Deployment, usage logs, follow-up interview.

Each is the right study for its question.

Types of Evaluation

Controlled settings (e.g., Usability testing)

A controlled evaluation setting is not the normal place for using a technology or for the user to be.

Usability Testing

Measures: numbers or time (e.g., tasks completed, errors made, time taken)
Techniques: combined from week 4: think-aloud during set tasks, a questionnaire afterwards (SUS, NASA-TLX), then a short interview
Data: variety of data depending on the techniques (e.g., video, audio, facial expressions, key presses, verbal feedback)
Settings: lab + observation room, mobile usability kit, university classroom
Number of participants: 5-12 baseline but more is better
What is new this week: the tasks are set by you, and the setting is controlled by you

A controlled setting at ANU for testing music apps. Interaction data, timing, audio, video, surveys and interviews were recorded (Martin et al., 2016)

Usability Testing Example

Schaadhardt et al. (2021) Understanding Blind Screen-Reader Users’ Experiences of Digital Artboards — Schaadhardt et al. (2021) **Understanding Blind Screen-Reader Users’ Experiences of Digital Artboards**

Natural settings (e.g., Field studies)

Evaluating a technology or context of use in the normal setting for the user.

Field studies can:

Help identify opportunities for new technology
Establish the requirements for a new design
Facilitate the introduction of technology or inform deployment of existing technology in new contexts

Helps to establish ecological validity.

Field Studies

Goals:
- Understanding how people interact with technologies in “messy worlds”, how technologies will be integrated into contexts
- Studying use of existing technologies and impacts of introducing new ones
Techniques: mostly qualitative (observation, interviews, diaries, interaction logging) rather than set tasks and timings
Duration: No fixed length- can be seconds, months, years
Paying attention to: Use situations, problems/errors, distractions, patterns of behaviours
How does your presence and involvement shape engagement? Observation vs participant observation
Findings: thematic analysis, vignettes, narratives, critical incident analysis (weeks 5 and 6)

Field Studies Example

Co-Designing with Orangutans: Enhancing the Design of Enrichment for Animals (Sarah Webber, Marcus Carter, Wally Smith, and Frank Vetere) Proc. DIS ’20 (Webber et al., 2020)

Design objective 1: Develop a digital installation to provide enhanced, varied enrichment for orangutans at Melbourne Zoo

Opportunistic Evaluations

quick feedback about a design idea in the early design process
confirm whether it’s worth developing an idea into a prototype
informal and doesn’t require lots of time or resources
not a replacement for formal evaluation
care required with ethics in research (Hons, Master, PhD). Asking supervisors and colleagues for advice vs collecting data to establish findings.

E.g., designers ask colleagues for design feedback: Yichen Wang’s arMIDI system early design process with supervisor and colleagues (Wang et al., 2025).

Which methods to choose from?

The evaluation setting guides certain dimensions of developed artefacts.

Combinations of methods are used for a richer understanding. E.g., usability testing is combined with observations to identify usability problems and how users use the system.

Pros and Cons

Controlled settings allow hypotheses testing on specific features for generalised results.
Uncontrolled settings offer unexpected insights into perception and experience of new technologies in daily life and work.

Lab research on AR co-creative system (Wang et al., 2025)

Activity: Evaluating an interactive toy

Talk: 🗣️🎤⭐️ You’re all HCI researchers and we need to evaluate this interactive toy.

We need to choose:

how we will evaluate the toy?
in what environment?
what information do we need and why?
what research questions are being asked?

Talk for 2-3 minutes and then we will hear some answers.

Where and why will we evaluate this toy? (Photo by COSMOH LOVE on Unsplash)

Planning Evaluations

What do we need to keep in mind to plan evaluations?

Design and Conduct Issues

Reliability: how well it produces the same results on separate occasions under the same circumstances
Validity: whether the evaluation method measures what it intended to measure
Ecological validity: how the environment in which an evaluation is conducted influences or distorts results
Bias: occurs when the results are distorted
Scope: how much of the findings can be generalised

Ethical Issues

Risks: what are the risks to participants? (e.g., physical harm, reputational risk, distressing conversations, being identified etc)
- …and how are risks mitigated…
Benefits: what are the benefits to participants? (e.g., none, helping research, fun experience, getting paid, course credit, etc)
Consent: how is informed consent established? (e.g., a participant information sheet and a written form)
Data: how is data stored and who has access to it? what will happen to it over time?

Universities have processes to approve the ethical aspects of research that collects data from humans following established rules (National Health and Medical Research Council (NHMRC) et al., 2025).

We don’t go deeply into research ethics in this course but the four issues above are the core ones.

Developing an evaluation plan

Evaluation Goal/Aims
Participants
Setting
Data to collect
Methods
Ethical Considerations and Consent
Data capture, recording, storage
Analysis method
Output(s) of evaluation process

Prepare what you need before the first participant arrives

The plan is a document. This is the checklist for the room (or the Zoom call).

a place for the participant and for you: tables, chairs, power
the prototype, working, in the state the tasks assume
printed instructions and task list, so everyone hears the same thing
data capture ready and tested: video, audio, screen, logs
consent form and information sheet, signed before recording

Then pilot it (week 4). Most study failures are “the laptop died” or “we didn’t press record”.

Yichen Wang’s human-AI musical collaboration research study setup at School of Music.

Experimental Variables

Independent variable: the condition the researcher controls.
Dependent variable: the outcome we are measuring.
Independent vars in HCI: different interfaces, input devices, software, colours, computer type
Dependent vars in HCI: efficiency, accuracy, subjective satisfaction, ease of learning, physical/cognitive demands

Variables shape your study

Tasks: completing specific tasks, or freely using a technology?
Interfaces: just using one interface, or comparing two (or more!) different styles

Feels like we doing chemistry or biology… (Photo by CHUTTERSNAP on Unsplash)

From a research question to a hypothesis

A hypothesis is a prediction: it says what you will change, what you will measure, and which way the result will go.

RQ: Is the pads interface faster to a first pattern than the command line?

H1: Participants reach a first repeating pattern faster with pads than with the command line.

The RQ allows more than one answer. The hypothesis picks one and dares the data to disagree.
A hypothesis can only be dismissed, never proven: null and alternative guide the test.
Dismissing one needs a controlled design and significance testing (week 10).

Not every RQ should become a hypothesis. nanojam questions 1 and 3 are better off without one.

Experiment Design

Which participants test which conditions?

Different-participant design: each participant sees one condition.
Same-participant design: everybody sees each condition.
Matched-participant design: matched groups of participants with a shared trait put into each group

Balanced Ordering is important to counter learning effects.
Design choices affects validity and reliability
Data collection: think back to week 4 lecture, but often includes task performance, completion time, errors, subjective satisfaction etc.

Focus is on the experiment not the design!! (Photo by Girl with red hat on Unsplash)

Table of Experimental Designs

Design	Advantages	Disadvantages
Different participants (between-participants design)	- No order effects	- Requires many participants - Individual differences can affect results - Random assignment helps minimize differences
Same participants (within-participants design)	- Eliminates individual differences between conditions	- Requires counterbalancing - Risk of order effects (e.g., learning or fatigue)
Matched participants (pair-wise design)	- No order effects - Reduces impact of individual differences	- Time-consuming to find matched pairs - May miss other influential variables

In-the-Wild Studies

Natural setting, minimal control over participants
reflecting real-world use unpredictable and complex
Ethical and practical challenges are greater, e.g., participant consent, privacy, equipment issues, and environmental factors.

Reveal insights about actual use and long-term integration that lab studies often miss.

Overview of performance-led research in the wild. (Benford et al., 2013)

People are complicated

HCI is hard. To do a study, you usually need to:

Plan the study
Find participants
Manage communication with them
Figure out what to do if they don’t show up
Managing a study requires some social skills! It’s hard work!

Is there any way to do evaluation without users?

Evaluation by Inspection

Skip the “users”! Just evaluate against established principles (heuristics) and standards.

Expert Evaluation

Conducted by designers and design “experts” rather than with end users
Inspection methods – expert role plays user
Heuristic evaluation: Researchers evaluate whether aspects design adhere to established usability principles (see over)
Cognitive walkthroughs: Simulating user reasoning and problem solving at each step in an interaction sequence (evidence, availability, accessibility of correct action)

Two more ways to evaluate with nobody in the room, covered after the case study:

Predictive models: compute the user’s performance instead of measuring it (Fitts’ law)
A/B testing: thousands of real users, none of whom know they are in a study. Not “without users” so much as without recruiting them.

Nielsen’s 10 Usability Heuristics

Visibility of system status: keep the user informed
Match between system and real world: system uses language and communication familiar to the user, information is natural and logical
User control and freedom: users make mistakes, there should be “emergency exits” to cancel and return quickly
Consistency and standards: users should not wonder whether words, situations or actions mean the same thing, follow conventions
Error prevention: eliminate error-prone conditions, or check with user before they occur

Recognition rather than recall: make elements, actions, and options visible
Flexibility and efficienty of use: shortcuts to speed up for experts, allow tailored experiences
Aesthetic and minimal design: less is more, no unnecessary information
Help users recognise, diagnose and recover from errors: error messages need plain language, and suggest solutions
Help and documentation: best if explanation is not needed, if it is, make it good

Activity: Inspect This Interface

An internal web app a state emergency agency uses to send public alerts. The officer picks a template, then confirms.

Do: You’re the expert evaluator. Log every violation of Nielsen’s 10 you can see, numbered by heuristic. 3 minutes, then we compare lists.

Decide what’s wrong before you find out what happened.

Reconstruction of the alert origination screen, from the FCC report’s description (Public Safety and Homeland Security Bureau, 2018).

Case Study: Hawaii, 13 January 2018

8:07am, mid-drill: the officer took the drill call for a real one and sent the live template. Same prompt for both, no preview.
“THIS IS NOT A DRILL” reached 1.4 million phones. No correction template existed; the correction took 38 minutes (Public Safety and Homeland Security Bureau, 2018).

The test that would have caught it: the one you just ran. Heuristics 5 and 9, one evaluator, one hour.

The fix, from Appendix B of the FCC report: drill relabelled, a “False Alarm” template added.

Cheapest first, never cheapest only

Inspection has a high false-negative rate, and different evaluators find markedly different problems. It “should not be trusted for complex or safety-critical systems” (Hornbæk et al., 2025, Section 40.3).

Hawaii’s alert system was safety-critical. The heuristic pass finds the problems on the face of the screen; it does not find the ones that only appear when a tired officer is mid-drill at 8am.

In Hornbæk’s full yardstick table, “comply with guidelines” maps to one method. “Safety” is the row marked multiple.

Shneiderman’s Eight Golden Rules of Design

Strive for consistency
Seek universal usability
Offer informative feedback
Design dialogs to yield closure
Prevent errors
Permit easy reversal of actions
Keep users in control
Reduce short-term memory load

Activity: Two ways to evaluate nanojam

Do (3 minutes): open https://charlesmartin.au/nanojam/#ui.interface=cli on your own device. You are the expert evaluator. Log every violation of Nielsen’s 10 you can find in the command line interface, numbered by heuristic.

Watch (3 minutes): one volunteer, projected, thinks aloud (week 4) while trying to make a repeating pattern on the pads interface, no help from me.

Then compare: what did the inspection find that the user never hit? What did the user hit that no heuristic covers?

Think: which of the three research questions from earlier does each of these studies actually answer?

A/B Testing

An online controlled experiment: users randomly get A (control) or B (new design), usually without knowing. Between-participants, at web scale; week 10 has the statistics.
A/A test first: same page to both groups, so any difference is a setup bug. Office Online’s failed because only the treatment was redirected; the delay alone cost page views (Kohavi et al., 2009).
Or don’t test, and find out later: Office Online swapped a yes/no feedback question for five stars, lost 80% of responses, and took eight months to notice.

Bing search ad: control (top) and treatment with a second line of ad text moved into the title (bottom) (Kohavi et al., 2020). More in week 10.

Predictive Models

Estimate user performance without needing real users, using formulas to assess task efficiency — useful in early design stages or when testing with users is difficult.

Fitts’ Law (Fitts, 1954):

predicts how long it takes to point at a target based on its size and distance
helps designers optimize button placement, size, and spacing on screens and devices.
applications: input methods (e.g., touch, gaze, tilt), mobile and VR, simulating interactions for users with motor impairments

Fitt’s Law from Sketchplanations (CC-BY 4.0)

Evaluation after deployment: adoption, use, and non-use

The last yardstick in the table, compatibility with user practices, can only be measured after the thing is out in the world. A whole literature asks what “success” means then:

Adoption/Appropriation/Design-in-use (Ehn, 2008)
Technology acceptance (Davis, 1989)
Non-use (Satchell & Dourish, 2009)
Technology habitation (Soro et al., 2016)
Technology individuation (Ambe et al., 2017)

Questions: Who has a question?

Who has a question?

I can take cathchbox question up until 2:55
For after class questions: meet me outside the classroom at the bar (for 30 minutes)
Feel free to ask about any aspect of the course
Also feel free to ask about any aspect of computing at ANU! I may not be able to help, but I can listen.

Meet you at the bar for questions. 🍸🥤🫖☕️ Unfortunately no drinks served! 🙃 — Meet you *at the bar* for questions. 🍸🥤🫖☕️ Unfortunately no drinks served! 🙃

References

Ambe, A. H., Brereton, M., Soro, A., & Roe, P. (2017). Technology individuation: The foibles of augmented everyday objects. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 6632–6644. https://doi.org/10.1145/3025453.3025770

Benford, S., Greenhalgh, C., Crabtree, A., Flintham, M., Walker, B., Marshall, J., Koleva, B., Rennick Egglestone, S., Giannachi, G., Adams, M., Tandavanitj, N., & Row Farr, J. (2013). Performance-led research in the wild. ACM Trans. Comput.-Hum. Interact., 20(3). https://doi.org/10.1145/2491500.2491502

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008

Ehn, P. (2008). Participation in design things. Proceedings of the Tenth Anniversary Conference on Participatory Design 2008, 92–101.

Fitts, P. M. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 47(6), 381.

Hornbæk, K., Kristensson, P. O., & Oulasvirta, A. (2025). Introduction to human-computer interaction. Oxford University Press. https://doi.org/10.1093/oso/9780192864543.001.0001

Kohavi, R., Crook, T., Longbotham, R., Frasca, B., Henne, R., Ferres, J. L., & Melamed, T. (2009). Online experimentation at Microsoft. Third Workshop on Data Mining Case Studies and Practice Prize, KDD 2009. http://ai.stanford.edu/~ronnyk/ExPThinkWeek2009Public.pdf

Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. https://doi.org/10.1017/9781108653985

Martin, C., Gardner, H., & Swift, B. (2014). Exploring percussive gesture on iPads with ensemble metatone. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1025–1028. https://doi.org/10.1145/2556288.2557226

References (cont.)

Martin, C., Gardner, H., Swift, B., & Martin, M. (2016). Intelligent agents and networked buttons improve free-improvised ensemble music-making on touch-screens. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 2295–2306. https://doi.org/10.1145/2858036.2858269

National Health and Medical Research Council (NHMRC), Australian Research Council (ARC), & Universities Australia. (2025). The national statement on ethical conduct in human research (2025). https://www.nhmrc.gov.au/about-us/publications/national-statement-ethical-conduct-human-research-2025

Public Safety and Homeland Security Bureau. (2018). Report and recommendations: Hawaii emergency management agency January 13, 2018 false alert. Federal Communications Commission. https://docs.fcc.gov/public/attachments/DOC-350119A1.pdf

Rogers, Y., Sharp, H., & Preece, J. (2023). Interaction design: Beyond human-computer interaction, 6th edition. John Wiley & Sons, Inc. https://quicklink.anu.edu.au/kv9b

Satchell, C., & Dourish, P. (2009). Beyond the user: Use and non-use in HCI. Proceedings of the 21st Annual Conference of the Australian Computer-Human Interaction Special Interest Group: Design: Open 24/7, 9–16. https://doi.org/10.1145/1738826.1738829

Schaadhardt, A., Hiniker, A., & Wobbrock, J. O. (2021). Understanding blind screen-reader users’ experiences of digital artboards. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445242

Soro, A., Brereton, M., & Roe, P. (2016). Towards an analysis framework of technology habituation by older users. Proceedings of the 2016 ACM Conference on Designing Interactive Systems, 1021–1033. https://doi.org/10.1145/2901790.2901806

Wang, Y., Xi, M., Adcock, M., & Patrick Martin, C. (2025). Seeing the sound: Supporting musical collaboration with augmented reality. Proceedings of the 2025 Conference on Creativity and Cognition, 99–112. https://doi.org/10.1145/3698061.3726905

Webber, S., Carter, M., Smith, W., & Vetere, F. (2020). Co-designing with orangutans: Enhancing the design of enrichment for animals. Proceedings of the 2020 ACM Designing Interactive Systems Conference, 1713–1725. https://doi.org/10.1145/3357236.3395559