This framework inspired by Lennart Nacke, everybody’s favourite HCI writing coach on LinkedIn.

1. Outline a broad area of interest
2. Identify a problem that needs solving
3. Justify solving this problem
4. Write the question

To be clear, a research question starts with a question word (what, how, why, can, do, should) and ends with a question mark. It can just be one sentence.

Seems too easy… let’s try it together.

Interfaces:

1. haptic feedback gloves
2. AR/VR headset
3. e-textile clothing
4. voice assistant
5. gesture recognition
6. smart headphones
7. ambient light display
8. wearable plants
9. eye-tracking interface
10. multi-touch table

Problems to Solve:

1. family meal planning
2. language learning while commuting
3. caring for houseplants
4. focus during remote work
5. medication schedules
6. teaching kids about conservation
7. organising hobby collections
8. non-verbal communication
9. practicing music in small spaces
10. tracking community events

Let’s write a research question!

Together, let’s Spin the wheels to decide on a broad area and a problem.

Then, decide on a “justification” and write a research question.

Remember that the RQ should include the broad area, the problem, and the justification.

Use the poll everywhere link to suggest research questions and vote on the best ones.

Write for 2-3 minutes, vote for 1 minute, then let’s discuss.

• Evaluation: collecting and analysing data from user experiences with an artefact.

• Goal: to improve the artefact’s design.

• Addresses: functionality, usability, user experience

• Appropriate for all different kinds of artefacts and prototypes.

• Methods vary according to goals.

• Understanding people
  • Users may not have the same experiences or perspectives as you do
  • Different users use software differently
• Understanding designs
  • Proof that ideas work
  • Understand limitations, affordances, applications

• Business
  • Invest in the right ideas
  • Find problems to solve (before production, before next iteration, etc.)
• Research
  • Evidence for new interactive systems
  • Empirical proof of hypotheses
  • New knowledge to answer research questions

Does the design do what the users need and want?

Examples:

• Game App Developers: Whether young adults find their game fun and engaging compared to other games
• Government authority: Whether their online service is accessible to users with a disability
• Children’s talking toy designers: Whether six-year-olds enjoy the voice, feel of the soft toy, and can use safely

Six usability goals:

• Effective to use (effectiveness)
• Efficient to use (efficiency)
• Safe to use (safety)
• Having good utility (utility)
• Easy to learn (learnability)
• Easy to remember how to use (memorability)

(Rogers et al., 2023)

Depends on your evaluation goal!

• Lab studies (controlled settings)
• In-the-wild studies (natural settings)
• Remote studies (online behaviour)

Evaluation serves different purposes at different stages of the design process

• Formative evaluation:
  • Assessing whether a product continues to meet users’ needs during a design process
  • Early or late stages
• Summative evaluation:
  • Assessing whether a finished product is successful
  • Feeds into an iterative design process

• Measures: numbers or time (e.g., tasks completed, errors made, time taken)
• Methods: mixture of methods (e.g., think aloud, observation, interviews, questionnaires, data logging and analytics)
• Data: variety of data depending on the methods (e.g., video, audio, facial expressions, key presses, verbal feedback)
• Settings: lab + observation room, mobile usability kit, university classroom
• Number of participants: 5-12 baseline but more is better

Evaluating a technology or context of use in the normal setting for the user.

Field studies can:

• Help identify opportunities for new technology
• Establish the requirements for a new design
• Facilitate the introduction of technology or inform deployment of existing technology in new contexts

Helps to establish ecological validity.

• quick feedback about a design idea in the early design process
• confirm whether it’s worth developing an idea into a prototype
• informal and doesn’t require lots of time or resources
• not a replacement for formal evaluation
• care required with ethics in research (Hons, Master, PhD). Asking supervisors and colleagues for advice vs collecting data to establish findings.

E.g., designers ask colleagues for design feedback: Yichen Wang’s arMIDI system early design process with supervisor and colleagues (Wang et al., 2025).

The evaluation setting guides certain dimensions of developed artefacts.

• Combinations of methods are used for a richer understanding. E.g., usability testing is combined with observations to identify usability problems and how users use the system.

Pros and Cons

• Controlled settings allow hypotheses testing on specific features for generalised results.
• Uncontrolled settings offer unexpected insights into perception and experience of new technologies in daily life and work.

You’re all HCI researchers and we need to evaluate this interactive toy.

We need to choose:

• how we will evaluate the toy?
• in what environment?
• what information do we need and why?
• what research questions are being asked?

Talk for 2-3 minutes and then we will hear some answers 🗣️🎤⭐️

• Evaluation Goal/Aims
• Participants
• Setting
• Data to collect
• Methods
• Ethical Considerations and Consent
• Data capture, recording, storage
• Analysis method
• Output(s) of evaluation process

• tables, chairs
• places for participants and researchers
• Instructions to participants
• Details, equipment for completing tasks
• Data collection equipment: video, audio recording
• In-person / Remote
• Zoom (e.g., COVID), online studies

• Independent variable: the condition the researcher controls.
• Dependent variable: the outcome we are measuring.
• Independent vars in HCI: different interfaces, input devices, software, colours, computer type
• Dependent vars in HCI: efficiency, accuracy, subjective satisfaction, ease of learning, physical/cognitive demands

Variables shape your study

• Tasks: completing specific tasks, or freely using a technology?
• Interfaces: just using one interface, or comparing two (or more!) different styles

Which participants test which conditions?

1. Different-participant design: each participant sees one condition.
2. Same-participant design: everybody sees each condition.
3. Matched-participant design: matched groups of participants with a shared trait put into each group

• Balanced Ordering is important to counter learning effects.
• Design choices affects validity and reliability
• Data collection: think back to week 4 lecture, but often includes task performance, completion time, errors, subjective satisfaction etc.

• Natural setting, minimal control over participants
• reflecting real-world use unpredictable and complex
• Ethical and practical challenges are greater, e.g., participant consent, privacy, equipment issues, and environmental factors.

Reveal insights about actual use and long-term integration that lab studies often miss.

1. Visibility of system status: keep the user informed
2. Match between system and real world: system uses language and communication familiar to the user, information is natural and logical
3. User control and freedom: users make mistakes, there should be “emergency exits” to cancel and return quickly
4. Consistency and standards: users should not wonder whether words, situations or actions mean the same thing, follow conventions
5. Error prevention: eliminate error-prone conditions, or check with user before they occur

6. Recognition rather than recall: make elements, actions, and options visible
7. Flexibility and efficienty of use: shortcuts to speed up for experts, allow tailored experiences
8. Aesthetic and minimal design: less is more, no unnecessary information
9. Help users recognise, diagnose and recover from errors: error messages need plain language, and suggest solutions
10. Help and documentation: best if explanation is not needed, if it is, make it good

Budd (2007) introduces further heuristics focussed on web, here’s some from the list:

• Clarity: Make the system as clear, concise and meaningful as possible for the intended audience.
• Minimise unneccessary complexity and cognitive load: Make the system as simple as possible for people to accomplish their tasks.
• Provide context: Interfaces should provide people with a sense of context in time and space
• Promote a pleasurable and positive experience: people should be treated with respect and the design should be aesthetically pleasing and promote a pleasurable and rewarding experience

• Large-scale, online controlled experiment used to compare two designs (A = control, B = new design) by measuring user behavior (e.g., click rates), often without users knowing they are part of a study.
• Between-participants design, randomly assigning users to different versions and analyzing outcomes statistically to determine if observed differences are due to the design and not chance.
• Proper setup is critical — running an A/A test first ensures the testing infrastructure is sound, and careful design is needed to avoid misleading results, as shown in real-world examples like Microsoft Office 2007.

Estimate user performance without needing real users, using formulas to assess task efficiency — useful in early design stages or when testing with users is difficult.

Fitts’ Law (Fitts, 1954):

• predicts how long it takes to point at a target based on its size and distance
• helps designers optimize button placement, size, and spacing on screens and devices.
• applications: input methods (e.g., touch, gaze, tilt), mobile and VR, simulating interactions for users with motor impairments

• Adoption/Appropriation/Design-in-use (Ehn, 2008)
• Technology acceptance (Davis, 1989)
• Non-use (Satchell & Dourish, 2009)
• Technology habitation (Soro et al., 2016)
• Technology individuation (Ambe et al., 2017)