Usability benchmarking

The problem: Reddit launched translation features but didn’t know how they were performing in terms of usability. I launched a survey in four key markers to measure satisfaction and establish a baseline that the team could use to measure improvements against. This was the first time that this specific experience had been measured.

Stakeholder alignment: Since this was the first time that the usability of the translation experience would be measured, I worked closely with stakeholders to tease out and document assumptions. The research helped the team paint a more complete problem of how translation tools were performing.

Impact: The research found that the lack of opt-in/opt-out features and signposting were making the experience super confusing to users. The team prioritized understandability features to orient users around when they were consuming translated content.

Study design:
Sample Size: N = 1879 [DE = 421, FR = 493, PT-BR = 693, EN = 272]

Statistical tests: We calculated confidence intervals (95%) for continuous (likert scale) and discrete data (user options).

• We treated likert scale data as continuous.

• We used 5-point scales with extreme polarity (e.g. Very easy, very hard) to clearly distinguish opposites.

• We applied closed-codes to all write-in answers.

We tested hypotheses by:

• Using a chi-square test of independence to test independence between two categorical variables (language and measurement).

• One-tailed T-test to measure user satisfaction against a benchmark.

• Paired-sample T-test to measure difference in means of variables.

• Two-sample T-test to measure difference in means between EN responses and other languages.