CS474: Human Computer Interaction - Experimentally Evaluating UX
Activity Goals
The goals of this activity are:- To design experiments to obtain informative feedback from diverse stakeholders to improve the user experience
Supplemental Reading
Feel free to visit these resources for supplemental background reading material.- Norman and Nielson - Gestural Interfaces: A Step Backward In Usability
- A/B Testing Example
- Complete Beginner's Guide to Design Research
- Norman Chapter 5
The Activity
Directions
Consider the activity models and answer the questions provided. First reflect on these questions on your own briefly, before discussing and comparing your thoughts with your group. Appoint one member of your group to discuss your findings with the class, and the rest of the group should help that member prepare their response. Answer each question individually from the activity, and compare with your group to prepare for our whole-class discussion. After class, think about the questions in the reflective prompt and respond to those individually in your notebook. Report out on areas of disagreement or items for which you and your group identified alternative approaches. Write down and report out questions you encountered along the way for group discussion.Model 1: Evaluating the User Experience
Questions
- Whose fault is it when a user fails to use a system properly?
- A tree test presents the user with the workflow steps, one step at a time, and allows the user to progress in a choose-your-adventure style. What kinds of deficiencies might this kind of study reveal?
- An A/B test evaluates if one call to action is more effective than another. How might you employ this in a software system to evaluate different user interface designs or workflows?
- What would likely happen in these studies if all the stakeholders in a medical application UX study had the same job title?
Embedded Code Environment
You can try out these code examples in a development environment of your choice! Note that some embedded projects have multiple source files; you can see those by clicking the appropriate file tab to open that file.Try It: Did the Redesign Help? Analyzing a UX Experiment
Observation tells you what goes wrong; experiments tell you whether your fix helps. This notebook analyzes a simulated A/B test of two checkout designs end-to-end: visualizing the distributions, running a t-test, computing an effect size (because “statistically significant” is not the same as “meaningful”), and extending to three designs with an ANOVA. It is directly reusable for the user experience study in your final project.
Click the badge to run the notebook in your browser with Google Colab (no installation required), or download the notebook to run it locally with Jupyter.
Explore Further
- Nielsen Norman Group - Why You Only Need to Test with 5 Users — the classic (and often misapplied) argument that small qualitative studies find most usability problems; contrast this with the sample sizes the notebook shows you need for statistical claims.
- Nielsen Norman Group - Quantitative vs. Qualitative UX Research — a short guide to choosing between counting and observing, and why mature UX practice does both.