- measures of central tendency: - describe where most data is clustered, shows the representative characteristic. - Mean, median, mode. - measures of spread: - *measuring variability*, tells us how much data values differ from the center. - Range, variance, std. - normal distribution defined by mean and standard deviation - Many statistical tests (e.g., t-tests, ANOVAs) assume normality - Can use tests or plots to check for normality - If not normal: consider data transformation or nonparametric tests

As an example: Is an height difference of 20cm meaningful? - If talking about adult people, probably yes. - If talking about mature Eucalyptus trees, probably no. - Means don’t account for data variance. So what do we do? - Significance testing - Compares explained variance (from independent variable) vs. unexplained variance (random/error).

A tall tree (Photo by gryffyn m on Unsplash)

- *p*-values are thrown around a lot in some scientific fields - *p* is the probability of obtaining a measurement by chance given the existing distribution of measurements. - low *p*-value = low probability difference occurred by chance: likely a real effect - low *p*-value is evidence supporting a hypothesis

**Think:** a typical cut-off for “significance” is $`p=0.05`$. Is this the best choice?

p-value diagram with a normal distribution (source)

For one week Facebook hid some positive posts from some users’ feeds, then measured the words those users wrote ([Kramer et al., 2014](#ref-kramer-emotional-contagion:2014)). Exactly as reported: - positive words in their own posts: $`B = -0.1\%`$, $`p < 0.001`$, $`d = 0.02`$ - negative words: $`B = +0.04\%`$, $`p = 0.007`$, $`d = 0.001`$

**Think:** you’re the reviewer. Is this a finding, or too small to mean anything? Decide before the next slide.

An employee proposed longer ad headlines: pull the first line of ad text up into the title. A few days’ work. - **The fork:** build it, or leave it? Program managers ranked it low. It sat for more than six months, until a different engineer noticed it was cheap to test ([Kohavi & Thomke, 2017](#ref-kohavi-online-experiments:2017)). - **What happened:** the A/B test showed revenue up 12%, over \$100M a year in the US alone. So large it tripped the “too good to be true” alarm that normally means a bug ([Kohavi et al., 2020](#ref-kohavi-trustworthy-experiments:2020)).

**Concept:** expert judgement ranked Bing’s best-ever revenue idea near the bottom. Only the A/B test found it, and the same test stops a bad idea shipping because somebody felt confident about it.

Control (top) and treatment (bottom) (Kohavi et al., 2020).

- Used in HCI to compare means between two conditions (e.g., menu selection times) - Include the t-value, degrees of freedom (df), and $`p`$-value, where df depends on participant numbers, and $`p`$ indicates the probability the result is due to chance. - `p < 0.05` (1-in-20 chance of a random result) typically taken as evidence supporting a hypothesis - smaller p-values (e.g., \<0.01) indicating stronger evidence - independent values: when comparing samples from different participants - paired values: comparing different observations of the same participants.

``` python from scipy.stats import ttest_ind, ttest_rel # independent values t-test t_stat, p_value = ttest_ind(group1, group2, equal_var=False) # observations of different participants print(f"t-statistic: {t_stat:.4f}, p-value: {p_value:.4f}") # paired-values t-test t_stat, p_value = ttest_rel(observation1, observation2) # different observations of the same participant. print(f"t-statistic: {t_stat:.4f}, p-value: {p_value:.4f}") ```

What if you have more than two groups to compare? (e.g., three+ interface variations?) What if you have more than one independent variable? (e.g., comparing the individual and combined effects of two separate aspects of an interface) Analysis of variance (ANOVA) enables these more complicated comparisons. An ANOVA’s output is a statistic called F so sometimes called an *F-test*.

Understand how variables relate to each other (e.g., is age or experience related to performance?) Correlation: - Measures the strength and direction of the linear relationship between two variables. - Most common method: Pearson’s $`r`$: range: -1.00 (negative) to 1.00 (positive). 0 indicates no linear relationship. Pearson’s $`r^2`$ (Coefficient of Determination) - Represents the shared variance between two variables. - Example: If $`r = 0.70`$, then $`r^2 = 0.49`$, meaning 49% of variance in one variable is explained by the other.

**Careful:** correlation is not causation!

Imagine an experiment measuring time spent in an online shopping app vs income.

Relationship between correlated variables and an intervening variable.

E.g., income vs. performance may be correlated due to an intervening variable (e.g., age) rather than directly related.

Helps to analyse categorical data: e.g., a yes/no choice.

**Think:** does this look random?

| Group | Yes | No | |-------|-----|-----| | A | 5 | 7 | | B | 11 | 1 | Results: - Chi-square statistic: 4.6875 - Degrees of freedom: 1 - *p*-value: 0.0304 ($`p < 0.05`$)

``` python data = { 'Group': ['A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B'], 'Answer': ['Y', 'Y', 'N', 'N', 'Y', 'N', 'Y', 'Y', 'N', 'Y', 'N', 'Y', 'Y', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N'] } df = pd.DataFrame(data) contingency_table = pd.crosstab(df['Group'], df['Answer']) print(contingency_table) chi2, p, dof, expected = chi2_contingency(contingency_table) print(f"Chi-square statistic: {chi2:.4f}") print(f"Degrees of freedom: {dof}") print(f"P-value: {p:.4f}") ```

Research question: > What effects will different machine learning models and feedback mechanisms have on simple improvised music performances? “Understanding Musical Predictions with an Embodied Interface for Musical Machine Learning” Martin et al. ([2020](#ref-martin-understanding-musical-predictions:2020))

The Embodied Musical Predictive Instrument (EMPI)

- 12 participants did a short improvisation with each ML model and with the motor turned on and off. - Six improvisation for each performer! - 3 by 2 design - Used quantitative data to compare the six experiences - Survey of 8 aspects of the performance - Measured length of improvisations

| X | Motor off | Motor on | |-------|-----------|----------| | Human | Human/Off | Human/On | | Synth | Synth/Off | Synth/On | | Noise | Noise/Off | Noise/On |

Delwiche, L. D., & Slaughter, S. J. (2019). *The little SAS book: A primer*. SAS institute.

Kohavi, R., Tang, D., & Xu, Y. (2020). *Trustworthy online controlled experiments: A practical guide to A/B testing*. Cambridge University Press.

Kohavi, R., & Thomke, S. (2017). The surprising power of online experiments. *Harvard Business Review*, *95*(5).

Kramer, A. D. I., Guillory, J. E., & Hancock, J. T. (2014). Experimental evidence of massive-scale emotional contagion through social networks. *Proceedings of the National Academy of Sciences*, *111*(24), 8788–8790.

Lazar, J., Feng, J. H., & Hochheiser, H. (2017). *Research methods in human-computer interaction* (Second). Morgan Kaufmann.

Martin, C. P., Glette, K., Nygaard, T. F., & Torresen, J. (2020). Understanding musical predictions with an embodied interface for musical machine learning. *Frontiers in Artificial Intelligence*, *3*, 1–13.

Verma, I. M. (2014). Editorial expression of concern: Experimental evidence of massive-scale emotional contagion through social networks. *Proceedings of the National Academy of Sciences*, *111*(29), 10779.

Wobbrock, J. O., Findlater, L., Gergle, D., & Higgins, J. J. (2011). The aligned rank transform for nonparametric factorial analyses using only anova procedures. *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems*, 143–146.