| framework | data | focus | outcomes | granularity |
|----|----|----|----|----|
| conversation analysis | spoken conversation recordings | process of conversations | how conversations are processed and progress | words or smaller |
| discourse analysis | speech or writing | how words convey meaning | implicit or hidden meanings in text | word, phrase |
| content analysis | written text, video, audio, images | how often something is featured or is spoken about | frequency of items in text | words to artefacts or people |
| interaction analysis | video of activities | interactions between people and artefacts | how knowledge and action are used in an activity | artifact, dialogue, gesture |
| grounded theory | empirical data of any kind | building theory from a phenomenon | theory grounded in data | varying levels |
| systems-based frameworks | large-scale and heterogeneous data | large scale systems of people and technology | organisational insights | macro, organisational level |
| **reflexive thematic analysis** (today’s focus) | any qualitative data, esp. interviews | patterns of meaning across a dataset | themes constructed by the researcher | flexible, dataset-wide |
# Interpreting and Presenting Findings
## Big Research Writing Tip: Cite your methods
- Applies to research projects in this class and at Honours, master, PhD and research-focussed workplace
- Work with supervisors/mentors/managers to choose methodology and analytical frameworks
- Read and understand the framework from (recent) scholarly sources (not just `nngroup.com`)
- Change and evolution is allowed, but understand that there is a wealth of example and established approaches
## Structured Notations
Specific interaction information can be represented in a formal/structured way when presented.
- presenting information through formal notations related to particular domains
- e.g., music applications might involve musical notation or symbolic data formats
- could be related to an analytical framework (e.g., conversation analysis)
- tradeoff between precision and flexibility; structured notation can be precise but potentially less accessible to a reader or limited in scope
## Using Stories
Context of use and examples of user experience can be seen as stories or narratives.
- Participants tell stories during data gathering which can illustrate research findings.
- Observations can be framed as stories
- Stories can be written or in the form of storyboards or videos.
- Can be used to support research findings and provide authenticity.
## Summarising Findings
Overall advice about findings…
- Multi-modal is often good: combine styles such as stories, plots, data excerpts, numerical tables
- Developing plots and visualisations is critical and hard work, just like crafting text. We spend hours getting it right in Python/R!
- Important to balance the weight of a **claimed finding** against supporting evidence; however, doesn’t mean that small studies are not useful.
- Reviewers hate over-generalisation: careful with terms like “most,” “all,” “majority,” and “none” without justification
- Statistical claims require care to avoid misleading the reader
# What even is knowledge anyway?
## What is knowledge?
By this stage, you could be excused for being a bit confused about qualitative research in HCI.
- Research is often defined as “knowledge creation”, but it’s not always clear *what* that knowledge is in HCI:
- Is the knowledge from an interview different to a measured interaction?
- Does our interpretation matter?
- Do the users have to be objective?
- E.g., if you create a new app, and then evaluate it, can your evaluation ever be objective? (remember you created the app in the first place!)
- Not a new question: *epistemology* is a (philosophy) discipline to understand knowledge.
## Epistemology
This may feel firmly off topic, but we need to surface some friction about *knowledge* to properly explain the different approaches in qualitative research.
- **Positivism/Post-positivism** an objective reality exists and can be known through systematic, generalisable methods. The “post” bit concedes observation is never perfectly objective
- **Interpretivism:** knowledge can be socially constructed and meaning made by people
- **Critical theory:** examine power structures and hidden inequalities
- **Constructivism/Constructionism:** knowledge is created and shaped by human experience and social interaction, including with the researcher
- **Pragmatism:** apply frameworks that work best to solve the problem (e.g., mixed methods research)
- **New-Materialism:** *things* can create meaning too with HCI defined by intra-action ([Barad, 2007](#ref-barad-meeting-universe:2007)) of things and people
## “Bring your own epistemology”? RTA’s flexibility has limits
- RTA is *theoretically flexible*: no single built-in theory — but “there’s no such thing as atheoretical TA” ([Braun & Clarke, 2022](#ref-braun-thematic-analysis:2022)): you choose a position and declare it
- the flexibility is bounded: RTA lives inside the *qualitative paradigm* — subjectivity is a resource (not “bias”), analysis can’t be “objective”, themes don’t emerge
- positivist-flavoured TA exists (codebooks, inter-rater reliability) — for Braun & Clarke that’s a *different method*, and unacknowledged slippage is “positivism creep”
- so the real requirement is **coherence**: make your claims, quality practices and epistemology match
- Braun & Clarke’s own lean is relativist–constructionist — *declared, not required* ([Braun & Clarke, 2022, p. p188](#ref-braun-thematic-analysis:2022)): the book models the positionality it asks of you
## Activity: What should this mean to you
- Some aspects of HCI, e.g., “user experience” aren’t well uncovered by (post-)positivist frameworks.
- Interpretivist/Constructionist stances more popular in qualitative HCI research.
- New-Materialism/Agential-realism ([Barad, 2007](#ref-barad-meeting-universe:2007)) is emerging in HCI as a relevant mode of inquiry
- *but* this has some implications in terms of how findings are described!
**Talk:** 🧠🗣️ Does any of this make sense? What kind of knowledge would you want to deal with?
Discuss with someone near you for 2 minutes, then let’s hear some answers.
# Thematic analysis
- Let’s get into some more detail how to do *thematic analysis*
- Remember that this is a group of techniques!
- I’ll introduce *a* way of doing (reflexive) TA, adapted from Braun & Clarke ([2022](#ref-braun-thematic-analysis:2022))
## Phases of thematic analysis
| B | I | N | G | O |
|----|----|----|----|----|
| Mentions inter-coder reliability | Implicitly (post-)positivist TA (not acknowledged) | More than 3 levels of themes | Mention of a lack of (statistical) generalisability | Messy mix of realism and constructionism |
| Unacknowledged social cognitions (e.g., attitude or body image) | Themes are thin - just single idea (a code) | Themes do not have a central organising concept | “Themes emerged” | Data collection stopped at “saturation” |
| Use of passive voice | No reflexivity | **Thematic Analysis** | Only Braun and Clarke 2006 cited | Mention of “bias” |
| Clarke spelled as Clark (no e) | More than 6 themes | No theory of language-treated as a window to truth | Themes are topic summaries | Very few participants quoted / over-quoting of one or more |
| Implicitly realist TA (not acknowledged) | Braun pronounced BRAWN (not Brown) | Mismatch between extracts and analytic claims | Use of a codebook | Data are just paraphrased without interpretation |
# Reflexivity
## What is reflexivity?
Alharthi, S. A., Alsaedi, O., Toups Dugas, P. O., Tanenbaum, T. J., & Hammer, J. (2018). Playing to wait: A taxonomy of idle games. *Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems*, 1–15.
Barad, K. (2007). *Meeting the universe halfway: Quantum physics and the entanglement of matter and meaning*. Duke University Press.
Blandford, A., & Furniss, D. (2006). DiCoT: A methodology for applying distributed cognition to the design of teamworking systems. In *Interactive systems. Design, specification, and verification* (pp. 26–38). Springer Berlin Heidelberg.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. *Qualitative Research in Psychology*, *3*(2), 77–101.
Braun, V., & Clarke, V. (2019). Reflecting on reflexive thematic analysis. *Qualitative Research in Sport, Exercise and Health*, *11*(4), 589–597.
Braun, V., & Clarke, V. (2022). *Thematic analysis: A practical guide*. Sage Publications.
Byrne, D. (2022). A worked example of braun and clarke’s approach to reflexive thematic analysis. *Quality & Quantity*, *56*(3), 1391–1412.
Corbin, J. M., & Strauss, A. (2014). *Basics of qualitative research: Techniques and procedures for developing grounded theory*. SAGE Publications.
DeCuir-Gunby, J. T., Marshall, P. L., & McCulloch, A. W. (2011). Developing and using a codebook for the analysis of interview data: An example from a professional development research project. *Field Methods*, *23*(2), 136–155.
Glaser, B. G., & Strauss, A. L. (1967). *The discovery of grounded theory: Strategies for qualitative research*. Aldine Publishing Company.
Jordan, B., & Henderson, A. (1995). Interaction analysis: Foundations and practice. *Journal of the Learning Sciences*, *4*(1), 39–103. [https://doi.org/10.1207/s15327809jls0401\\2](https://doi.org/10.1207/s15327809jls0401\_2)
King, A. (2008). In vivo coding. In L. M. Given (Ed.), *The SAGE encyclopedia of qualitative research methods*. SAGE Publications, Inc.
Klein, L. (2014). What do we actually mean by “sociotechnical”? On values, boundaries and the problems of language. *Applied Ergonomics*, *45*(2), 137–142.
Lazar, J., Feng, J. H., & Hochheiser, H. (2017). *Research methods in human-computer interaction* (Second). Morgan Kaufmann.
Liang, C. A., Munson, S. A., & Kientz, J. A. (2021). Embracing four tensions in human-computer interaction research with marginalized people. *ACM Trans. Comput.-Hum. Interact.*, *28*(2).
Ljungblad, S., Man, Y., Baytaş, M. A., Gamboa, M., Obaid, M., & Fjeld, M. (2021). What matters in professional drone pilots’ practice? An\ interview\ study\ to understand the complexity of their work and inform human-drone interaction research. *Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems*.
McDonald, N., Schoenebeck, S., & Forte, A. (2019). Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice. *Proceedings of the ACM on Human-Computer Interaction*, *3*(CSCW), 72:1–72:23.
Morriss, L. (2024). Themes do not emerge. An editor’s reflections on the use of braun and clarke’s thematic analysis. *Qualitative Social Work*, *23*(5), 745–749.
Olmos-Vega, F. M., Stalmeijer, R. E., Varpio, L., & Kahlke, R. (2023). A practical guide to reflexivity in qualitative research: AMEE guide no. 149. *Medical Teacher*, *45*(3), 241–251.
Piggott, T., Morris, C., & Lee-Poy, M. (2015). Preceptor engagement in distributed medical school campuses. *Canadian Medical Education Journal*, *6*(2), e21–e28.
Porcheron, M., Fischer, J. E., Reeves, S., & Sharples, S. (2018). Voice interfaces in everyday life. *Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems*, 1–12.
Rode, J. A. (2011). Reflexivity in digital anthropology. *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems*, 123–132.
Rogers, Y., Sharp, H., & Preece, J. (2023). *Interaction design: Beyond human-computer interaction, 6th edition*. John Wiley & Sons, Inc.
Terry, G., & Hayfield, N. (2021). *Essentials of thematic analysis.* American Psychological Association.
