User Research

Dr Charles Martin

Researching some real users—your classmates! (Photo by UX Indonesia on Unsplash)

Outline

Due date: 2025-09-29 23:59 AEST
Mark weighting: 30%
Submission: submit your assignment according to the instructions below
Policies: Late submission is not permitted for this assessment. This is an individual assessment task.
Gitlab Template Repository: https://gitlab.cecs.anu.edu.au/comp3900/2025/comp3900-2025-user-research

Description

Assignment 2 involves conducting a small-scale user research study involving an existing interface or interactive system. The research challenge is as follows:

AI Interface Usability: Sam Altman just called and it turns out that he’s been reading about “usability” and “user experience”. He is very concerned that AI-integrated technology may not be as usable as previously thought. Sam needs the results of a pilot study pronto and you’re the HCI researcher to run it. Your challenge is choose one existing AI-integrated interface and run a user research study with 3–5 participants (your classmates). The research question you will investigate is:

“How do users’ mental model of the AI system align with the behaviour of the system and what usability issues arise from any misalignments?”

To address this question you will collect and analyse data and present research findings. Your study should involve at least one form of quantitative and one form of qualitative data and analysis.

Your study documentation will include research plan, analysis, results, and conclusions under the following headings:

Interface, Research Plan, and Conclusions: explain the interface you are studying and why you have chosen it, your study plan including data gathering strategies, analysis framework(s), and justification from HCI academic research, and summarise the overall conclusions from your data analysis.
Quantitative Results: present analysis of your quantitative data and your findings from this data, include excerpts from data, plots, tables, or other presentation approaches to help articulate your findings.
Qualitative Results: present analysis of your qualitative data and your findings from this data, include excerpts from data, plots, tables, or other presentation approaches to help articulate your findings.

You should include all collected data from your study in the materials folder of the GitLab template, and you may refer to it from your main document.

To accomplish this task, you will need to have a strong understanding of data gathering, quantitative analysis, and qualitative analysis as discussed in weeks 5-8 of the course. Your submission must demonstrate sophisticated engagement with these concepts.

Specification

Your submission must:

clearly explain the interface you are studying
include a clear study plan with justification from HCI academic research
include a clear summary of the overall conclusions derived from your data and analysis.
include data from 3–5 participants: all participants must be students in COMP3900/6390! (list them in the acknowledgements section)
include results based on quantitative and qualitative data and analysis techniques
include at least two references to external sources in ACM format
be written in markdown format with correct syntax and formatting
be a maximum of 1800 words + 10% acceptable buffer (i.e., no more than 1980 words)
must be uploaded to your fork of the comp3900-2025-user-research repository on Gitlab by the due date

N.B. to eliminate doubt: all participants in your study must be current students in COMP3900/6390. The best way to find and study your classmates is to attend all lectures and tutorials and participate in assignment planning activities.

Submission Process

Fork the template repository.
Follow appropriate data gathering and analysis processes to address the research challenge.
Upload research data and analysis scripts as well as other relevant materials in your fork of the template repository.
Write your study documentation following the template in the repository.
Don’t forget to include at least two references to external sources in ACM format in your documentation.

Notes

Here’s how to get started with the work in this assignment:

Choose an (one) AI-integrated interface. This could be an app, a website, software on a computer or built into a hardware device. It will need to be something that you can evaluate with your classmates (so they should know what it is). You will need to explain why you have chosen this interface, and how makes sense to study it in terms of the research question.
Create a study plan. Use your research skills to find examples of studies into human-AI interfaces (e.g., at CHI) and use them to help plan your study. Make sure your study plan is realistic.
Find participants – they must be current COMP3900/6390 students. Attend all classes and find 3–5 people in your tutorial who will participate in each others’ studies. Don’t leave this until the last minute!
Collect data with your participants. You could do this during a drop-in session. You will need to participate in other people’s studies as well ask them to participate in yours. (don’t make up the data).
Analyse your data. Use the quantitative and qualitative analysis techniques covered in classes.
Summaries your findings. Your overall conclusions should be supported by data and analysis.
Write up your project in the correct format in your fork of the gitlab repository. Make sure you are using correct markdown syntax and have included all data and analysis files in your repository.
Acknowledge your research participants by listing them in your acknowledgements section.

Here’s some general advice:

Don’t forget to ask for help or advice on the course forum: Edstem discussions
Generative AI is not banned in this course, but submitted assignments are expected to be primarily your own work. Any use of generative AI must be explained in your Acknowledgements section including the scope to which it is applied. Use of generative AI without appropriate explanation and referencing comes under the N category in the marking rubric.
You may use generative AI to help with aspects of data analysis (see lectures!) but you are responsible for the output and whether findings are justified or not, we may re-run analyses to check your data.

Marking Rubric

CRITERIA	HD	D	CR	P	N
Sophistication and clarity of the study plan, summary of conclusions. (10 marks)	Excellent to outstanding study plan and conclusions with consideration of HCI research methods that go beyond learning materials. Level of communication and referencing is excellent.	Very good study plan and conclusions applying HCI research methods that follows specification, but not beyond learning materials. Level of communication and referencing is excellent.	Study plan and conclusions following HCI research methods at level of learning materials. Specification was followed. Level of communication and referencing is good.	Some effort to follow the specification for a study plan and derive conclusions. May touch on HCI research methods. Level of communication and referencing may have some errors.	Study plan and conclusions may have little connection to HCI research methods or be below acceptable standards. May not follow the specification or contain serious errors in communication and referencing.
Sophistication of quantitative data collection, analysis and articulation of findings. (10 marks)	Excellent to outstanding quantitative data collection, analysis and articulation of findings demonstrating in-depth understanding of HCI approaches. Excellent adherence to the assessment format.	Very good quantitative data collection, analysis and articulation of findings demonstrating in-depth understanding of HCI approaches. Excellent adherence to the assessment format.	Good quantitative data collection, analysis and articulation of findings that may not show sophisticated understanding of HCI approaches. Good adherence to the assessment format.	Satisfactory quantitative data collection, analysis and articulation of findings that may have limited incorporation of HCI approaches. The adherence to the submission format may be poor. Data analysis may have some errors.	Quantitative data collection, analysis and articulation of findings that is below acceptable standards. May have very poor adherence to submission format. Data collection and analysis may contain serious errors.
Sophistication of qualitative data collection, analysis and articulation of findings. (10 marks)	Excellent to outstanding qualitative data collection, analysis and articulation of findings demonstrating in-depth understanding of HCI approaches. Excellent adherence to the assessment format.	Very good qualitative data collection, analysis and articulation of findings demonstrating in-depth understanding of HCI approaches. Excellent adherence to the assessment format.	Good qualitative data collection, analysis and articulation of findings that may not show sophisticated understanding of HCI approaches. Good adherence to the assessment format.	Satisfactory qualitative data collection, analysis and articulation of findings that may have limited incorporation of HCI approaches. The adherence to the submission format may be poor. Data analysis may have some errors.	Qualitative data collection, analysis and articulation of findings that is below acceptable standards. May have very poor adherence to submission format. Data collection and analysis may contain serious errors.

Note that serious errors in referencing, communication, data collection and analysis are aligned with the N category in the assessment rubric. Serious errors may include: references to non-existent sources, research findings or conclusions that rely on non-existent data or analyses, fabricated data, fabricated results from analysis processes.