This month we’re looking at overcoming objections to user research, and convincing teams to playtest. You might assume that this isn’t necessary – that teams understand the value of playtesting or UX research. Although that’s somewhat true – you rarely hear outright objections to user research, it say “not right now”.
This can be caused by fears of exposing the game prematurely to critique, and misunderstanding how research can support creativity. In this issue we’ll look at some fears that our colleagues might have, and what we can do about them.
Get future games user research lessons direct to your inbox
Overcoming objections to user research
Here are some of the most common objections I encounter, and some counter-points to consider.
“We don’t have time to playtest”
Sometimes teams have the belief that playtesting will take a lot of time away from development. A robust study does take time to prepare. Writing a test script, recruiting the right participants, analysing the data to draw reliable data takes time. But this doesn’t mean that development has to stop.
With good communication about the state of upcoming builds and milestones, it’s possible to prepare the study in advance. You can agree and aim for a shared test date. Rather than the team having to wait for you to prepare the test after the build is ready, the test is launched as soon as the build is ready. Doing this well requires strong communication with our teams (but communication with our teams is always important).
The critical time to minimise is that between the test build being delivered and the researcher sharing some results. Researchers can adapt how they debrief findings, and optimise their analysis process, to minimise this wait.
“Playtesting is incompatible with creativity”
Teams sometimes mistake what we do with market research or focus testing which aim to discover what is popular.
This is not what user researchers do – we shouldn’t be trying to make creative decisions. Our studies are focused on testing how effectively games are meeting their creative goals. We learn whether players are having the experience the designers intended. Alongside this, observing player behaviour may inspire game designers to come up with new ideas – but the decisions about “what should this game be” are not ours to make.
When we’re working with teams, be very careful about explaining this, avoiding research objectives that fall into market research, and being unbiased about the creative direction. We should be helping teams define and reach their creative goals, rather than changing those goals ourselves.
“It’s not ready”
(thanks Dan Choates for inspiration for this one!)
This assumes that the right approach is one big playtest to evaluate every aspect of the game. That’s risky, as it doesn’t normally come together until near the end, when there’s no time to make changes.
To address this, teach teams about research objectives – defining what specifically they want to learn from a playtest. Base objectives on what parts they are currently working on. The parts that are not ready can be ignored or worked around, and we can get directed feedback on the elements that are ready.
Conversations with team members will help expose what they are working on, the riskiest decisions they have had to make recently, and inspire potential research objectives. I believe doing lots of small targetted tests leads to a better outcome than one big playtest.
“It’s not valid if it’s not quantitative” or “we want a survey”
Teams may not recognise the value of qualitative research, or may have a preferred research method that they trust more.
As researchers the first step we always have to take is to think about the research objectives. What do the team need to know currently? What is the best method of answering that question? Picking the right method will ensure we get reliable results from our study.
Sometimes a survey or quantitative method is correct. However we need to determine that from the research objectives, rather than blindly pick a method because the team request it.
One of the roles a user researcher has to do is educate their teams on the research methods that exist, and the strengths of each. Talk to your teams about the value of qualitative research. Explain how observation and interviews build a deep understanding of ‘why players do that’. Examples can help teams recognise how understanding a player’s behaviour and mental model is relevant to the game design decisions they have to make. (mental models = their understanding of how something works)
“Players don’t know what they want”
This comes from a misunderstanding of what user research does. Teams may assume we just aggregate player feedback and report that back to game designers.
This isn’t what good user research is. As we discussed when covering creativity, we’re not taking creative directions from players. Our focus is on whether the experience players are having matches the one we want them to have. Demonstrate to teams that we understand their creative vision. Show how we combine data from observation, interviews and perform thorough analysis to see if the vision is being met. This will build confidence that we’re not just reporting players’ opinions.
“We already understand our players”
When I counted last year, I had observed players for over 25,000 hours (admittedly some of that was watching more than one player at a time!). In all of those sessions, I have never run a study where I didn’t learn something unexpected about player behaviour, or the issues they encountered.
Expert reviews, peer feedback and heuristics can help improve games a bit – but I have never seen a test fail to reveal something new. Look for examples of how the big studios with big budgets handle playtesting – by the end of development, Uncharted 3 was running roughly one playtest a week. There must be something in it!
The most convincing tool user researchers have
The most convincing tool user researchers have for the value of playtesting is actually doing it. Exposing people to real players – allowing them to observe sessions, spot issues themselves, and involving them in the analysis is a fantastic tool for winning converts to the value of user research and playtests. Don’t work in isolation as a ‘black box’ service – be open, collaborative, and people will see the value themselves.
More on how to do this in future issues.
What skills are employers looking for?
At the GRUX Online conference last week, Raphaël Leroy gave an incredibly valuable talk. He has analysed 100 games user research jobs posted over the last three years, and looked for the themes of what employers wanted from candidates.
You should check out the full slides (and watch the video of the talk when it is posted on the Games Research & UX Special Interest Group youtube channel later this month)
Two slides that particularly spoke to me were the skills that the job adverts requested:
And the methods mentioned in job adverts:
This analysis can inspire what skills future user researchers should focus on developing and demonstrating. It’s also great inspiration for future issues of this newsletter – going deep on themes such as communication, collaboration and planning studies. We’ve previously discussed how to debrief findings, run remote playtests, run quantitative studies, and set research objectives but in future issues we can deep dive on writing surveys, running playtests, interviews and heuristic analysis – and hit all the essential skills companies are looking for.
Thanks Raphaël for running the analysis and presenting your conclusions to the community!
Jason Schklar’s career journey, and advice for people joining the industry
This month I’ve been really lucky to talk to Jason Schklar, industry veteran and co-founder of UX Is Fine. Jason has worked in the industry for over two decades with both game publishers and development studios, including Microsoft Games, Big Huge Games, Zynga & Disney Mobile.
He shared his experience making the business case for research, how research informs iteration and advice for people looking to start a career in UX or games user research.
Get the tools needed to run efficient and reliable playtests
People who follow me on twitter have heard me speak about this recently. This year I’ve been particularly interested in how teams who don’t have dedicated user researchers run playtests and UX research, and what’s hard for them. I shared some of the lessons I learned at Develop Conference this year.
Next year I will be releasing a toolkit for people who want to run better playtests + UX research, but aren’t from the big games studios with budget to employ full-time researchers. The kit will give you the tools needed to recruit players, run moderated & unmoderated playtests, create reliable surveys, analyse data and apply all of the best practice from UX professionals, to improve the efficiency and accuracy of your user research + playtests.
This will be released in the first half of next year. I know from the chats I’ve had with subscribers that many of you are UX designers, producers, community managers, solo devs, QAs, and other people who just want to run better playtests (as opposed to people seeking a career in user research). Because you are special newsletter friends, I wanted to give you the opportunity to pick the toolkit up at the special discount before it goes on a wider release.
If ‘I have to run playtests, but it’s not my full time job’ sounds familiar, this toolkit is for you. You can pre-order the kit now – the price is currently a special discount price that will go up next year when I start to talk about the toolkit publically.
If you’re not sure if this is for you, you can also subscribe to the playtest toolkit mailing list at the bottom of the page here (but will miss this first discount!) – I’ll be releasing a lot more info last year, so you can decide closer to launch whether this solves problems you have.
If you want to be a Games UX researcher as your full-time career, this toolkit probably isn’t for you. Stick with our regular newsletters for more help next year!
Games user research jobs
Every month, I feature a selection of games user research jobs aimed at juniors, or people switching from another discipline. This month, quite a few research-adjacent jobs came up…
EA – Data Science Intern
Bungie – UX Design Intern for Destiny 2
Nexon – UX Research Coordinator
Common playtest mistakes
This month I’ve been invited to speak at PlayTestCloud’s Meet The Expert series of webinars.
The event is free, running on Friday 10th December. I’ll be talking with Elie Mourand about some of the most common playtesting mistakes I’ve seen teams fall foul of, and Elie will share his expertise based on managing projects at PlayTestCloud.
The sign-up link will be ready next week – follow me on twitter to avoid missing it!
The end of 2021
I think there won’t be a full issue next month, just a brief update (although maybe a secret bonus feature if everything comes together!).
Thanks for all your support over this year, subscribing to the newsletter and everyone who picked up the How To Be A Games User Researcher book. I have been really happy with the reception it had this year and everyone’s kind words. I hope everyone has a great end of 2021, and looking forward to continuing to work together to become better games user researchers in 2022.
Don’t miss future issues
Thanks for reading this issue. If you’re reading online, and want to get future issues directly to your inbox – sign up here.