This tutorial will guide you through the process of designing a research project. You will learn how to identify real-world problems, recognise research gaps, and frame a strong research question. These skills connect directly to your final project.
As a pre-class task, you will analyse an existing evaluation study. In the in-class tasks, you will collaboratively discuss evaluation analyses, develop research questions, and plan an evaluation, along with an overall reflection.
Choose an example of a evaluation study from either CHI or DIS (use the ACM Digital Library to search for relevant papers, or browse through the proceedings). Read through the study and answer the following questions:
The questions above describe what the researchers did. Reading critically means asking whether it holds up. Hornbæk, Kristensson and Oulasvirta (2025, Section 45.1) boil critical reading down to five aspects — and note that these map closely onto the criteria reviewers actually use at HCI conferences:
| Aspect | Ask yourself |
|---|---|
| Argumentation | Are the conclusions based on solid reasoning? What alternative arguments could explain them? |
| Evidence | What is the evidence for the claims, and where did it come from? How does it compare with earlier research? |
| Contribution | What is now known about HCI that was not known before? |
| Impact | Who benefits, and how? Could anyone be harmed in a way the authors did not acknowledge? |
| Significance | Weighing positive and negative effects on different stakeholders, how significant is the result — technically, economically, socially, ethically? |
Choose two of the five and write a sentence or two on each for your paper. (Two, not five — the forum post is still only 100–200 words.)
Being critical does not mean being dismissive: a paper can be excellent and still have limits worth naming. It is worth knowing that only about 5% of citations in HCI papers engage critically with what they cite (Hornbæk et al., 2025, Section 45.1) — most just pass claims along. That is how errors spread: “muscle memory” is not a thing psychology recognises, and the often-repeated claim that 93% of communication is nonverbal comes from a study of far narrower scope than the claim suggests.
In this task, you will work in groups using a shared document provided by your tutor. The analyses you complete here will serve as references for the later tasks this week, helping you create research questions and plan the subsequent evaluations.
Research questions have been called survival beacons because they should guide all aspects of our research plans. You shouldn’t start an evaluation process without a clear research question.
We will re-do the activity from the lecture in groups to develop RQs and set out a clear evaluation plan.
Remember the steps for writing a research question:
As a small group, Spin the wheels to randomly find a broad area and a problem.
Then, you will need to come up with a justification (no wheels for that…) and write the question. The RQ should include the broad area, the problem, and the justification. Remember that a research question starts with a question word (what, how, why, can, do, should) and ends with a question mark. It can just be one sentence.
Iterate on your question until you feel it will help you plan an evaluation.
Tips:
Create an evaluation plan based on this slide, focusing on the research questions you created. Your tutor will let you know whether to write your plan in a shared document or on paper.
The activity can be divided into two parts:
Work with your group to go through each aspect, using the lecture slide for guidance. Your goal is to create an evaluation plan that clearly addresses your research questions.
In this task, you will share your evaluation plans with the whole class and receive comments and insights from others. As a group, you will present your plan by walking through the aspects from the lecture slide and showing how it connects to your research questions.
You may use the format below as a guide (but you are not limited to it):