This month we’re looking at overcoming objections to user research, and convincing teams to playtest. You might assume that this isn’t necessary – that teams understand the value of playtesting or UX research. Although that’s somewhat true – you rarely hear outright objections to user research, it say “not right now”.
This can be caused by fears of exposing the game prematurely to critique, and misunderstanding how research can support creativity. In this issue we’ll look at some fears that our colleagues might have, and what we can do about them.
Get future games user research lessons direct to your inbox
Overcoming objections to user research
Here are some of the most common objections I encounter, and some counter-points to consider.
“We don’t have time to playtest”
Sometimes teams have the belief that playtesting will take a lot of time away from development. A robust study does take time to prepare. Writing a test script, recruiting the right participants, analysing the data to draw reliable data takes time. But this doesn’t mean that development has to stop.
With good communication about the state of upcoming builds and milestones, it’s possible to prepare the study in advance. You can agree and aim for a shared test date. Rather than the team having to wait for you to prepare the test after the build is ready, the test is launched as soon as the build is ready. Doing this well requires strong communication with our teams (but communication with our teams is always important).
The critical time to minimise is that between the test build being delivered and the researcher sharing some results. Researchers can adapt how they debrief findings, and optimise their analysis process, to minimise this wait.
“Playtesting is incompatible with creativity”
Teams sometimes mistake what we do with market research or focus testing which aim to discover what is popular.
This is not what user researchers do – we shouldn’t be trying to make creative decisions. Our studies are focused on testing how effectively games are meeting their creative goals. We learn whether players are having the experience the designers intended. Alongside this, observing player behaviour may inspire game designers to come up with new ideas – but the decisions about “what should this game be” are not ours to make.
When we’re working with teams, be very careful about explaining this, avoiding research objectives that fall into market research, and being unbiased about the creative direction. We should be helping teams define and reach their creative goals, rather than changing those goals ourselves.
“It’s not ready”
(thanks Dan Choates for inspiration for this one!)
This assumes that the right approach is one big playtest to evaluate every aspect of the game. That’s risky, as it doesn’t normally come together until near the end, when there’s no time to make changes.
To address this, teach teams about research objectives – defining what specifically they want to learn from a playtest. Base objectives on what parts they are currently working on. The parts that are not ready can be ignored or worked around, and we can get directed feedback on the elements that are ready.
Conversations with team members will help expose what they are working on, the riskiest decisions they have had to make recently, and inspire potential research objectives. I believe doing lots of small targetted tests leads to a better outcome than one big playtest.
“It’s not valid if it’s not quantitative” or “we want a survey”
Teams may not recognise the value of qualitative research, or may have a preferred research method that they trust more.
As researchers the first step we always have to take is to think about the research objectives. What do the team need to know currently? What is the best method of answering that question? Picking the right method will ensure we get reliable results from our study.
Sometimes a survey or quantitative method is correct. However we need to determine that from the research objectives, rather than blindly pick a method because the team request it.
One of the roles a user researcher has to do is educate their teams on the research methods that exist, and the strengths of each. Talk to your teams about the value of qualitative research. Explain how observation and interviews build a deep understanding of ‘why players do that’. Examples can help teams recognise how understanding a player’s behaviour and mental model is relevant to the game design decisions they have to make. (mental models = their understanding of how something works)
“Players don’t know what they want”
This comes from a misunderstanding of what user research does. Teams may assume we just aggregate player feedback and report that back to game designers.
This isn’t what good user research is. As we discussed when covering creativity, we’re not taking creative directions from players. Our focus is on whether the experience players are having matches the one we want them to have. Demonstrate to teams that we understand their creative vision. Show how we combine data from observation, interviews and perform thorough analysis to see if the vision is being met. This will build confidence that we’re not just reporting players’ opinions.
“We already understand our players”
When I counted last year, I had observed players for over 25,000 hours (admittedly some of that was watching more than one player at a time!). In all of those sessions, I have never run a study where I didn’t learn something unexpected about player behaviour, or the issues they encountered.
Expert reviews, peer feedback and heuristics can help improve games a bit – but I have never seen a test fail to reveal something new. Look for examples of how the big studios with big budgets handle playtesting – by the end of development, Uncharted 3 was running roughly one playtest a week. There must be something in it!
The most convincing tool user researchers have
The most convincing tool user researchers have for the value of playtesting is actually doing it. Exposing people to real players – allowing them to observe sessions, spot issues themselves, and involving them in the analysis is a fantastic tool for winning converts to the value of user research and playtests. Don’t work in isolation as a ‘black box’ service – be open, collaborative, and people will see the value themselves.
More on how to do this in future issues
Be ready for your career in games
Get my free ebook of career tips from researchers at top game studios such as EA, Blizzard, PlayStation, Ubisoft, Activision and more. Based on exclusive interviews, they reveal ten essential tips to kick-start your GUR career today.
Plus receive a monthly email with free games user research lessons and curated Games UX and UR jobs.
Sign up now to get the free e-book (no spam, just a nice email from me each month!)
The end of 2021
I think there won’t be a full issue next month, just a brief update (although maybe a secret bonus feature if everything comes together!).
Thanks for all your support over this year, subscribing to the newsletter and everyone who picked up the How To Be A Games User Researcher book. I have been really happy with the reception it had this year and everyone’s kind words. I hope everyone has a great end of 2021, and looking forward to continuing to work together to become better games user researchers in 2022.
Don’t miss future issues
Thanks for reading this issue. Get a monthly email with a games user research lesson, and entry level Games UX and UR jobs – sign up here.