For the first five years of my career, most of the research studies I ran were in person. It was much easier to bring people to us, rather than transport our technically complex setup to them.
This had started to change for me prior to the pandemic, and then exploded last year when we could no longer safely bring participants to us. Now remote games user research is an essential part of our toolbox. In this issue we explore how to do it, and some of the things to look out for.
Get future games user research lessons direct to your inbox
Running remote games user research studies
Remote games user research is when participants can take part at home, rather than coming to us. That’s always been the case for large-scale beta tests, which use analytics and surveys to gather feedback. It has traditionally been challenging for earlier stage studies, like a typical usability test.
There are some advantages from running usability studies remotely. Previously our studies were limited to ‘people who could travel to us in a reasonable timeframe’. Running these remotely allows us to overcome our geographical bias, and recruit people from around the world to take part. Remote research makes the logistics of accessibility testing much easier too – it allows us to find more people with access needs who can take part in studies, and doesn’t put the burden on them to travel to us.
However there have always been some difficulties with running these studies remotely. Many studios are very secretive about their games, and the risk of leaks is increased when we can’t keep an eye on everything that our participants are doing. There are also technical and logistical obstacles that are more difficult remotely. It’s much easier for me to install a prototype of a mobile phone game on my phone and give the phone to a participant, than to get them to install the prototype successfully on their own phone.
Many of the challenges have got easier as specialist tools have developed – we’ll look at some today.
How to do remote games user research
Live moderated research
Moderated research – where you are able to speak to the participant live and ask questions, is often the best way of gathering the most data possible from your playtest.
Many teams have been exploring using Parsec to allow remote players to stream access to builds on your own device, similar to how Google Stadia works. This solves some of the technical issues (because the game is running on your own device it doesn’t need to be set up on their computer). It also reduces some opportunities for leaking, so can be preferable to installing the game on the playtester’s computer. There is a video of how Parsec works on their website, as well as a case study of Ubisoft using it to distribute builds for a marketing event.
If your team are comfortable with participants installing the game on their own device, conference tools like Google Meet allow you to view the player’s screen, and talk to them live. It also has the built-in ability to record the session. Google Meet requires very little technical knowledge from the player, so is simple to set up – and it’s free!
Another alternative to consider is unmoderated research – asking players to play in their own time, and receiving a video recording. This has some downsides – it’s not possible to ask questions in reaction to the behaviour you are seeing in the session. However, it can often be logistically easier to organise.
Some teams use automated services like PlaytestCloud to do this. This allows teams to specify a ‘type’ of player, upload a build, and receive videos of people playing it. This eliminates many of the logistical challenges of distributing games and finding players and makes running unmoderated user research studies easier.
However not every team has the budget to outsource this (or are not working on a mobile game), and will have to recreate this setup manually. This can be done using tools like Obs or DScout (recommended by @eddiepearson) to capture videos from players, and combine that with a free survey created on Google Forms.
This kind of unmoderated research can allow us to see many more players than we could observe linearly. It can be very powerful for combining qualitative feedback (‘observing what they do’) with quantitative feedback from surveys.
Non Disclosure Agreements
The risk of leaks is difficult to avoid entirely. Non-disclosure agreements can warn participants and reduce the likelihood of them sharing confidential information. This template from the research agency ping-pong will allow you to create your own NDA, and includes a plan to get it signed remotely.
Some remote research tips
I asked others on LinkedIn for their tips about remote research.
Christian Ress from PlaytestCloud gave this great advice:
Some great tips for the study design of remote studies (and good advice to run a pilot. Pilots are always an essential part of my in-person research plan, but easy to forget when working remotely!).
For a different take to addressing the issues from the pandemic, Player Research spoke last year about how they had adapted their lab to be safe for in-person studies, and I know this is something other research teams are looking into.
Career Interview with Joe Florey – Senior User Researcher at PlayStation
Each issue, we speak to a new or experienced user researcher about how they started their career in games. Joe Florey is an experienced user researcher, working in PlayStation’s European user research team. In this exclusive interview, he explained how mentoring helped him join the industry, and his experience converting a psychology PhD into a games user research role.
It’d be impossible to talk about the games industry this month, without mentioning the allegations of sexual harassment at Activision Blizzard.
In the ‘how to be a games user researcher’ book, I covered some of the downsides of working in the games industry. In particular, we focused on the risk of toxic fans, and misogynistic abuse from members of the public. Something I entirely failed to cover was that this also occurs from colleagues within the industry, not just from fans. Although the lawsuit is against a single company, I believe this happens throughout the industry.
Other people are much better placed to comment than I am, but I wanted to share some twitter threads I’ve found helpful over the last month to consider my own actions, and how to create a more inclusive work culture.
I’ve been listening to a great new games user research podcast. GUR Café is a podcast with Lanie Dixon, Olivier De Maeyer, & Sébastien Lourties of Ubisoft Montreal. In their second episode, they cover how important communication skills are for user researchers, and some tips on how to work with new teams.
Entry level games user research jobs
Adams Greenwood-Ericksen shared these exciting roles on the GRUX discord, working on games such as Call of Duty, Skylanders, World of Warcraft, Overwatch, Diablo, Candy Crush, and Bubble Witch.
Hey folks! Activision User Research is hiring for 3 (!) user research moderators to work on-site in our Woodland Hills CA lab. Since these are entry level roles, we’re looking for folks with interest, education, and/or skills in game user research, but we don’t expect any significant prior experience in the field. Come learn the trade and grow with us!
People who follow me on twitter have heard about this already, but I’m excited to share a bit about a current project I’m working on for release next year.
Both while working on the book, and in conversations with developers since, I’ve become increasingly aware of an unmet need for many games studios. Our deep dives on a career in games user research are helpful to teams who can give time and attention to user research. But many teams don’t have the time or budget to dedicate a full-time role to understanding players and testing games.
I see a need for converting the best practice from games user research. Making it achievable for teams with no time or money, making their current playtesting practise more efficient and more useful. I think this will be a ‘toolkit’ for game teams to apply to improve their playtests.
My plan is to develop this toolkit in the open. I want to open it up to critique throughout development and improve the quality of the final thing. As a first step I’ve been interviewing game developers to understand how they playtest currently, and what is difficult. Here’s a short thread covering some of what I’ve learned. This has inspired some ideas about what to do to help!
I believe the audience for that project (‘game developers who want to playtest better’) is different to this newsletter (‘people who want to be games user researchers’). So I don’t intend to regularly cover updates for that project here. If you’d like to hear when it’s ready, stick your email here. (no spam, just a nice email from me sometime next year).
Don’t miss future issues
Thanks for reading this issue. If you’re reading online, and want to get future issues directly to your inbox – sign up here.
Have a great month!