Get information about people, their behaviour and experiences with technology.

• remember: you are not the user! user research aims at design-neutral descriptions of real users (Hornbæk et al., 2025)
• What information and why?
• Depends on research problem and phase of research/design process, e.g.,
  • Comparing two alternative interfaces
  • Understanding a context of use
  • Measure time taken to complete a task
  • Discover how users interact with an existing system

Who are the participants? How many are needed?

• Small group of stakeholders
• Criteria for involvement (“plays video games regularly”)
• Random sampling from large population
• Convenience/volunteer sampling from those available
• Users with specific skills/needs
• Snowball sampling (participants help find more)
• Researchers make a justified choice
• Number of participants: most common in HCI is 12 (Caine, 2016)

A data provider gives us data. What do they get back?

• Conduct research openly, ethically, responsibly
• Informed consent
• Clear communication of benefits and risks
• Respect and acknowledgement
• Particular care for some groups, e.g., Aboriginal and Torres Strait Islander communities.
• Practical: building rapport and understanding (not just taking)

• data can have strings attached
• privacy
• personal information (including name!)
• sensitive information (embarrassing or harmful)
• storage requirements
• do we just use Google drive for everything?
• research needs a data management plan

• data triangulation: data is drawn from different sources, times, places, people, etc
• investigator triangulation: different researchers (observers, interviewers, etc)
• triangulation of theories: different theoretical frameworks
• methodological triangulation: different data gathering or research techniques
• this typology is a classic idea from sociology (Denzin, 2017): no single method is bias-free, so combine them
• but debated: does combining methods assume there is one “true” picture for them to converge on? (Blaikie, 1991)

A small initial study to help plan a larger study.

• Could involve limited number of participants
• Limited interface or study parameters
• Checks that expected data can be obtained
• Can be called a “formative study” (to form the goals of the main study)
• E.g., assess the game platform used in a larger study (Mohaddesi & Harteveld, 2020)

• Interviews make a lot of sense in HCI
• What should a system do? Just ask.
• How should it work? Just ask.
• Did it work well? Ask away.
• How would you change it? Ask.

• exploratory, similar to conversations
• go into depth on experiences
• questions are open: no expectation on the content of answers or subsequent questions
• probing: can you tell me more about …?
• benefit: generate rich, complex data
• limitation: time consuming to analyse

Could you tell me about your experience using the NewWidgetApp?

You mentioned that you enjoyed (feature X), why was that?

Can you explain more about what happened when you used (feature X)?

• questions are predetermined
• questions need to be short and clear
• questions can be closed (answers from specific options)
• typically, whole interview is scripted
• low skill to deliver
• fast to deliver
• useful when goals are clearly understood and questions and responses can be identified

What is your most used code editor: VSCode, nvim, emacs, notepad.exe, or something else?

How often do you push your assessments to Gitlab: every minute, every hour, every day, just once?

• features of both structured and unstructured
• typically: script with questions on main topics but discussion and probing follows each question
• intended to be somewhat replicable
• probe: neutral questions to gain more detail
• prompt: reminder of some part of the topic to gain specific information
• careful: prompting can preempt answers (bias!)

How are things going with the interaction experience?

Are there any [initial] impressions you want to share?

Can you describe the connections between your movements and the resulting sound?

(Reed, 2023)

• interview with multiple participants (could be called a group interview)
• facilitator prompts discussion
• group members can influence each other (good or bad?)
• good for talking to lots of people.
• raising diverse viewpoints
• facilitation: need to be careful
• groupthink: generally considered harmful

(I don’t find the “Focus Group” term super useful in my academic research—but they are widely used in industrial/government research.)

• usually have a script for introductions, consent, etc (read verbatim)
• listen more than talk
• respond with sympathy but without bias
• enjoy the experience
• interviewing is hard work!

• usually interviewing involves a combination of:
  • notes
  • audio
  • video
• different data have different issues and needs
  • notes: type up, handwriting (bias?)
  • audio: convert format, transcribe (suggest Aiko)
  • video: edit, store (large file sizes, anonymity)

• written method of gathering structured data
• ask questions of a subset of people to generalise to a wider population (Müller et al., 2014)
• sometimes called a “survey”, technically a survey is the whole study
• questionnaire or survey instrument is the paper form with questions
• questions can be open or closed
• often questionnaire used for demographic data
• standard survey instruments are often questionnaires

Questions might be something like:

What were the strengths of this course?

Please provide any suggestions about how this course could be improved?

• the researcher might have little control over how seriously these questions are taken.
• potential to gather rich data
• careful: use these only with really open concepts

• closed form questions have preset responses from which the respondent must select.
• unordered responses
• rating scale questions
• careful: closed form questions design should be neutral and include most likely or relevant responses

What is your favourite fruit (select one answer): plum, tomato, pineapple”

How interesting is this lecture? Select a number from 1 (interesting) to 10 (amazingly interesting).

• Likert scale questions are very common in questionnaires
• named after Rensis Likert (social scientist) (Likert, 1932)
• The question includes a statement, e.g.: “The workload was appropriate for this course”
• A number of levels of agreement are provided, e.g.: “strongly disagree, disagree, neutral, agree, strongly agree.”
• 5-points is common, 3, 7, 9 or a continuous slider is also possible.

• ratings of an object, concept, situation, etc
• the answer is a point between two opposite concepts
• e.g., describe your experience of using the (insert system here)
  • difficult to use — easy to use
  • boring — fun
  • slow to learn — fast to learn

1. I think that I would like to use this system frequently.
2. I found the system unnecessarily complex.
3. I thought the system was easy to use.
4. I think that I would need the support of a technical person to be able to use this system.
5. I found the various functions in this system were well integrated.

6. I thought there was too much inconsistency in this system.
7. I would imagine that most people would learn to use this system very quickly.
8. I found the system very cumbersome to use.
9. I felt very confident using the system.
10. I needed to learn a lot of things before I could get going with this system.

1. How mentally demanding was the task?
2. How physically demanding was the task?
3. How hurried or rushed was the pace of the task?
4. How successful were you in accomplishing what you were asked to do?
5. How hard did you have to work to accomplish your level of performance?
6. How insecure, discouraged, irritated, stressed and annoyed were you?

Worksheets provided! (Hart & Staveland, 1988). Extra questions weight the ratings, though most studies now skip weighting and use “Raw TLX” (Hart, 2006).

• Idea: measure how well a system can support creativity (Carroll & Latulipe, 2009; Cherry & Latulipe, 2014)
• creativity support tools: writing, visualisation, video editing, music tools etc
• creativity is a bit hard to define, but the CSI includes:
  • exploration, expressiveness, immersion, enjoyment, results worth effort, collaboration
• inspired by TLX, 2 questions per factor + 15 paired comparisons.

• It’s hard to write survey questions! Beginners may do better with a standard questionnaire.
• More questions means more analysis work—counter-productive without a way to aggregate them.
• Rating scale data isn’t continuous: use non-parametric significance tests—more powerful and consistent than t-tests at typical HCI sample sizes (Kaptein et al., 2010)—though statisticians still argue about this (Norman, 2010).
• The distribution matters: prefer box plots over mean and standard error charts.

• watching people for science
• observing system use in its normal context
  • e.g., join a tour group to observe a navigation app
  • e.g., watch a concert to observe music technology
• “in the wild”: everyday contexts, not the lab (Chamberlain et al., 2012)

Example framework from Rogers et al. (2023):

• The person: who is using the technology?
• The place: where are they using it?
• The thing: what are they doing with it?

• Literally “study of culture”
• immersion and participation in a research context.
• e.g., embed in a factory (talk to workers, perform tasks) to create a factory management system
• often applied to workplaces: factories, hospitals, etc
• study participants in their own context rather than in a lab/classroom
• its job in HCI is debated: “implications for design” vs “what people actually do” vs interpretation (Crabtree et al., 2009; Dourish, 2006; Suchman, 1987)

• observation in a lab-based setting
• could be possible to identify close details
• record all aspects of technology use
• record data, audio, video as well as observations

• it can be difficult to remember details from a long sequence of tasks.
• techniques like video cued recall can help participants provide a commentary on an experience.
• the idea is: you record video of the participant completing an an interaction and then have an open-ended discussion while watching it back.

• participants write about their experiences with a system regularly in a diary
• easy for the researcher at the collection stage
• takes continuous effort from the participants (reminders? structure?)
• relies on participant’s memory and subjective account
• video and photos can reinforce written accounts
• the capture medium (photos, audio, tangible objects) shapes what participants record and later recall (Carter & Mankoff, 2005)

• Interaction log: system data showing exactly what the participant did and when
• e.g., key presses, mouse moves, GUI events, sensor data
• time spent on actions or software
• unobtrusive and automatic, but lots of data: needs visualisation or analysis to develop findings
• scraping public sources (e.g., social media) can also be observation

• data gathering should adapt to different participants
• children: think and react differently to adults, e.g., adapt scales to visual representations (Putnam et al., 2020)
• people with disabilities: caregivers might be involved to adapt interview questions
• animals(!): hard to interpret behaviour, need to design interfaces to work without hurting animals (Mancini et al., 2017)

• expectations for remote data gathering changed completely during pandemic!
• can access participants in different countries, age groups, abilities, specific or expert users
• observation could be recorded and then analysed later
• best practices might include (Mastrianni et al., 2021):
  • run pilot tests before sessions
  • have backups in place in case of issues
  • inform participants of technical requirements
  • use retrospective questioning if think-aloud has issues

Who has a question?

• I can take cathchbox question up until 2:55
• For after class questions: meet me outside the classroom at the bar (for 30 minutes)
• Feel free to ask about any aspect of the course
• Also feel free to ask about any aspect of computing at ANU! I may not be able to help, but I can listen.