Processes

• (data gathering)
• data cleaning
  • e.g., age = 999
• analysis and interpretation
  • different tasks, but often parallel
• presentation

Bias in Analysis and Interpretation

• bias: influence affecting objective judgement and decision making
• integrating new data with past experience (bias is normal!)
• conscious: we know about them and their effect on us
• unconscious: subtle effects that we don’t know about

Specific Biases

• familiarity bias: do what we know best
  • ignore unfamiliar methods
  • ignore unfamiliar data sources
• self-attribution bias: giving ourselves too much credit
  • falsely attribute improvements to our design and not external factors, user learning, etc
  • overlook alternative explanations
• confirmation bias: discarding information that contradicts existing belief
  • more on confirmation bias
  • ignore non-supporting evidence
  • analyse for confirmation, not for discovery
  • overlook alternative analyses and sources

Quantitative and Qualitative

• typical categorisation of data
• Quantitative data: form of numbers or easily translated into numbers
  • years experience
  • number of minutes to perform a task
• Qualitative data: words, images, sound
  • descriptions
  • interview transcripts
  • photos
• Some data can be represented as both
  • e.g., digital text, sound, images can be represented and analysed numerically (is this useful?)

Quantitative and Qualitative Methods

Quantitative Analysis: find magnitude, amounts or size of something and make rigorous comparisons

Qualitative Analysis: find the nature of things, themes, patterns, stories

Use and Misuse of Data

• It’s easy to misuse numerical data and the results of analyses
• This can particularly happen when transforming the type of data (e.g., agreement ratings to a numerical code)
• Qualitative data can also be misused, e.g., content of questions reported as a finding.

Use and Misuse of Data: what was wrong?

First Steps

• Interviews: transcribe (e.g., using Aiko), expand notes, enter closed questions into spreadsheets (treated as quantitative)
• Questionnaires: enter into spreadsheets, clean up data, filter into datasets
• Observations: expand notes, transcribe recordings, edit videos, synchronise and clean up interaction data

Centre of a set of data

what’s the average of a set of values?

• Mean: sum of the values divided by the number of entries.
• Median: order the set numerically and find the value in the middle (or if even number of entries, halfway between the two middle entries)
• Mode: the most common entry

Example: [2, 2, 3, 4, 873]

Example shows that outliers mess up mean, so median is often more useful.

Spread of data

how spread out is a set of values?

• Range: max - min
• Standard Deviation: the typical distance of a value from the mean
• Interquartile range: range of the middle 50% of the data

Similarly to central measures, interquartile range is robust against weird outliers

Normal vs non-normal distributions

The distribution of the data is how it is spread out and where it is bunched up.

• normal distribution: a.k.a. bell curve, Gaussian distribution, mean, median, mode are the same, and the data evenly falls either side of the mean
• skewed distributions: data with a weirdly long tail in either direction
• multi-modal distributions: data that seems to have multiple bumps

This matters because statistical tests often assume data is normal so findings might be misleading.

Looking at the data

First thing to do after loading it in. May not be the most helpful approach… but still important to check it’s not garbled and the columns make sense.

Descriptive Statistics

Plotting Data

Third thing to do when loading data

• Plot the data to see the distribution and compare data points
• Scatter plot: see all the data! good for checking outliers and comparing aspects of data
• Histogram: useful to check distribution
• Boxplot: useful to compare distributions clearly (more abstract) charles approved plot!
• Bar plots: really just show one value (mean), may not be useful (too abstract!)
• Line plot: useful for showing data over time, not distributions

If plots show something interesting then you can investigate.

More complex plots

You can get more plots into one plot. Good for surfacing contrasts or telling a story about the data graphically.

sns.set_theme(style="ticks", palette="Set2")
plt.figure(figsize=(10, 6))
sns.boxplot(data=survey_data, x='degree_program', 
            y='interactive_activities_likert', 
            hue='time_in_canberra', 
            medianprops={'linewidth': 2, 'color': 'black'})
plt.savefig('plots/fake_data_complex_boxplot.png', 
            bbox_inches='tight', dpi=300)
plt.show()

Comparing data and tests

• You can compare two sets of data by finding the difference between their centres or other descriptive statistics but is the difference meaningful?
• We use statistical significance to help ascertain meaningful differences that might be a research finding.
• A classic test is the t-test which compares the means of two sets of data, the output of a t-test can tell us how likely differences are to be random or significant (meaningful)

Significance test notes

• we will come back to this later…
• t-tests assume normality and can only compare simple situations
• other tests can be used on any non-normal data 📊 and complex datasets (e.g., multiple values from each participants, multiple experimental conditions)

Quantitative Analysis with Python

Lots of ways to do data analysis:

• Excel/spreadsheets can do this a bit, but inferior to coding approaches
• Special programming languages exist: SPSS (1968–), S (1976–), R (1993–), Python (1991) + pandas (2008), julia (2012)
• R is where the statisticians, social scientists and psychologists live, Python and Julia are more where computer science folks hangout.

In this class we’ll use Python, numpy, pandas, scipy, seaborn, and matplotlib as a default stack for data analysis (yes, libraries are a problem in python…)

Last chance: answer the questionnaire!

Do: Did you miss the questionnaire at the start of class? Answer it now! (1 minute)

• go to https://pollev.com/charlesmarti205 or scan the QR code
• your answers are about to become our live demo dataset…

Demo time: analysing and plotting data in Python

Real data is messy: the PollEverywhere export

HCI Lecture Interaction Questionnaire
I enjoy interactive activities in lectures.
Response,Via,Screen name,...,Created At
Agree,...,User: Web_06529,"",2025-08-17 22:44:30
Strongly Agree,...,User: Web_aa2e2,"",...
""
I usually attend lectures in person
Response,Via,Screen name,...,Created At
...

Step 0: set up Colab

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import scipy.stats as stats

from google.colab import files
upload = files.upload()   # choose the PollEV .csv export
data_file = list(upload.keys())[0]

• our default stack: pandas (tables), seaborn (plots), matplotlib (plot plumbing), scipy.stats (tests)
• files.upload() is Colab-only — no live data? the notebook has a fallback cell that downloads the 2025 responses instead

Step 1: split the file into its six tables

from io import StringIO

raw = open(data_file).read()
blocks = raw.split('\n""\n')             # split on the "" lines
blocks[0] = blocks[0].split("\n", 1)[1]  # drop the report title
frames = []
for block in blocks:
    question, table = block.split("\n", 1)  # question, then CSV
    df = pd.read_csv(StringIO(table))
    df["question"] = question.strip()
    frames.append(df)
# long format: one row per answer, tagged with its question
responses = pd.concat(frames, ignore_index=True)

Step 2: keep what we need, name it sensibly

responses = responses[["Response", "Screen name", "question"]]

short_names = {
    "I enjoy interactive activities in lectures.":   "enjoy_interactive",
    "I usually attend lectures in person":           "attend_in_person",
    "I usually watch lectures online":               "watch_online",
    "Select the interactive activities you prefer:": "preferred_activity",
    "What kind of degree program are you in?":       "degree_program",
    "How long have you lived in Canberra?":          "time_in_canberra",
}
responses["question"] = responses["question"].replace(short_names)
responses.describe()   # count / unique / top --- does it look right?

Step 3: pivot to a tidy table

survey_data = responses.pivot(index="Screen name",
                              columns="question",
                              values="Response").dropna()

Tidy data: one row per participant, one column per question.

• pivot turns the long “one row per answer” table into a wide one
• .dropna() keeps only participants who answered everything — careful, that’s an analysis choice, and you should report it

Step 4: turn the Likert answers into numbers

likert = ["Strongly Disagree", "Disagree", "Neutral",
          "Agree", "Strongly Agree"]
for col in ["enjoy_interactive", "attend_in_person", "watch_online"]:
    survey_data[col] = pd.Categorical(survey_data[col],
                                      categories=likert, ordered=True)
    survey_data[col] = survey_data[col].cat.codes + 1

• as plain strings, the answers sort alphabetically: Agree < Disagree < Neutral < ... 🙃
• an ordered categorical knows the real order; .cat.codes + 1 maps it to 1–5
• boxplots and stats need numeric data, not strings
• (should you take a mean of Likert data? statisticians argue about this — more in week 10)

Step 5: explore, one variable at a time

survey_data["degree_program"].value_counts()
survey_data["time_in_canberra"].value_counts()
survey_data.describe()

# distribution of one question
sns.set_theme(style="ticks", palette="Set2")
sns.histplot(data=survey_data,
             x="enjoy_interactive", bins=5)

Step 6: compare groups

sns.boxplot(data=survey_data,
            x="degree_program", y="enjoy_interactive",
            medianprops={"linewidth": 2, "color": "black"})

# add a third variable with hue
sns.boxplot(data=survey_data,
            x="degree_program", y="enjoy_interactive",
            hue="time_in_canberra",
            medianprops={"linewidth": 2, "color": "black"})

• boxplots compare distributions between groups at a glance
• hue= splits each box again by a second grouping — but it gets crowded fast…

Step 7: melt back to long format

survey_long = pd.melt(
    survey_data,
    id_vars=["degree_program", "time_in_canberra", "preferred_activity"],
    value_vars=["enjoy_interactive", "attend_in_person", "watch_online"],
    var_name="question", value_name="score")

sns.boxplot(data=survey_long, x="question", y="score",
            hue="degree_program",
            medianprops={"linewidth": 2, "color": "black"})

• melt is the inverse of the pivot in step 3: wide → long
• why bother? seaborn wants one column of scores to plot all three Likert questions side by side

Step 8: facet to really see the groups

g = sns.FacetGrid(survey_long, col="time_in_canberra")
g.map_dataframe(sns.boxplot, x="question", y="score",
                hue="degree_program",
                medianprops={"linewidth": 2, "color": "black"})
g.add_legend(title="Degree")

• faceting: one small plot per group, with shared axes — differences between groups jump out
• swap col= and hue= to ask a different question of the same data
• “small multiples” is a classic visualisation idea — every plotting library has a version of it

Step 9: is the difference meaningful?

undergrad = survey_data[survey_data["degree_program"] ==
    "Undergraduate student (Bachelor degree)"]["enjoy_interactive"]
postgrad = survey_data[survey_data["degree_program"] ==
    "Postgraduate student (Master degree)"]["enjoy_interactive"]

t_stat, p_value = stats.ttest_ind(undergrad, postgrad)

• the boxplots suggested a difference between groups — the t-test asks how likely that gap is under pure chance
• convention: p < 0.05 counts as significant
• caveats: small samples, Likert data isn’t really interval, t-tests assume normality — much more on this in week 10

Coding in Qualitative Analysis

• in qualitative research the word coding has nothing to do with programming
• coding: annotating data with key words or phrases that provide a means for analysis over a large corpus
• codes can be inductive (bottom up, from the data) or deductive (top-down start with a framework of codes) (Braun & Clarke, 2006)
• challenges: creating meaningful, non-overlapping codes that are clearly defined and determining granularity
• inter-rater reliability measures the clarity and reliability of the coding scheme rather than correctness of analysis (McDonald et al., 2019)
• can be done with a text editor or dedicated software (NVivo, MAXQDA)

Analysing Video Material

• initial viewing involves watching entire recording while writing high-level narrative and noting timestamps of interesting events
• chronological and video times used to index events
• data augmentation: video plus screen captures, interaction logs, transcriptions
• coding schemes applied systematically to ensure reliability and consistency
• team-based coding can address subjectivity, ambiguity, and potential errors
• expert knowledge can help with unclear codes and supports interpretation

Identifying Themes

• Many qualitative research approaches distill themes from collected data.
• code the data (apply labels) and then create higher level themes from codes.
• (Reflexive) Thematic Analysis (RTA) (Braun & Clarke, 2022) is a well-known and accessible methodology.

Typical process

1. Familiarise with the data
2. Coding (short labels, multiple rounds)
3. Generating initial themes
4. Developing, reviewing, and refining themes

What is a theme, and why does it matter?

A theme is a high level finding from qualitative analysis, but what that means can differ.

• Two kinds of theme: patterns of meaning (uncovering implicit meaning behind words) versus data summaries (summarising responses across participants) (Braun & Clarke, 2019)
• themes can be created organically from the data through interpretation, or follow predetermined frameworks and categories
• no one way is correct, but need to be deliberate in methodology in particular to be careful about:
  • whether themes are a summary or reflect hidden or implicit meaning
  • whether themes arise from data or come from a pre-determined framework
  • whether we want “correct” codes, or to ensure that codes are clearly defined and interpreted consistently (can use multiple researchers to help)

Affinity Diagram

• affinity diagrams used for organising large amounts of data and identifying themes and overall narratives, a core part of contextual design (Beyer & Holtzblatt, 1997)
• both digital (e.g., Miro) and physical (e.g., sticky notes) diagramming approaches
• not necessarily a lot to this: summarise the data on notes, and arrange to find relationships between them.
• my PhD/master/Honours students tend to do this with Miro
• See this resource for a guide.

Categorising Data

• deductive analysis applies pre-existing theoretical frameworks or categories to analyse data
• data is systematically coded to segment and categorise specific elements, allowing for pattern identification and quantitative analysis
• e.g.: take a specific categorisation scheme from previous research (interface problems, or design recommendations), apply to new situation
• taking a quantitative approach: counting occurrences of categories per participant, identifying specific situations or issues

Critical Incident Analysis (Flanagan, 1954)

• focus on significant behavioral incidents rather than general impressions
• identifying specific moments that are pivotal in either positive or negative ways
• makes data analysis more manageable and focused
• can be identified by
  • participants during retrospective discussions,
  • observers through real-time monitoring
  • through video analysis of recorded sessions
• more info: nngroup, usability bok

Questions: Who has a question?

Who has a question?

• I can take catchbox questions up until 2:55
• For after class questions: meet me outside the classroom at the bar (for 30 minutes)
• Feel free to ask about any aspect of the course
• Also feel free to ask about any aspect of computing at ANU! I may not be able to help, but I can listen.