For the first five years of my career, most of the research studies I ran was in person. It was much easier to bring people to us, rather than transport our technically complex setup to them.
This had started to change for me prior to the pandemic, and then exploded last year when we could no longer safely bring participants to us. Now remote games user research is an essential part of our toolbox. In this issue we explore how to do it, and some of the things to look out for.
Sign up to get future games user research lessons direct to your inbox with my monthly newsletter (at the end of the article).
Running remote games user research studies
Remote games user research is when participants can take part at home, rather than coming to us. That’s always been the case for large-scale beta tests, which use analytics and surveys to gather feedback. It has traditionally been challenging for earlier stage studies, like a typical usability test.
There are some advantages from running usability studies remotely. Previously our studies were limited to ‘people who could travel to us in a reasonable timeframe’. Running these remotely allows us to overcome our geographical bias, and recruit people from around the world to take part. Remote research makes the logistics of accessibility testing much easier too – it allows us to find more people with access needs who can take part in studies, and doesn’t put the burden on them to travel to us.
However there have always been some difficulties with running these studies remotely. Many studios are very secretive about their games, and the risk of leaks is increased when we can’t keep an eye on everything that our participants are doing. There are also technical and logistical obstacles that are more difficult remotely. It’s much easier for me to install a prototype of a mobile phone game on my phone and give the phone to a participant, than to get them to install the prototype successfully on their own phone.
Many of the challenges have got easier as specialist tools have developed – we’ll look at some today.
How to do remote games user research
Live moderated research
Moderated research – where you are able to speak to the participant live and ask questions, is often the best way of gathering the most data possible from your playtest.
Many teams have been exploring using Parsec to allow remote players to stream access to builds on your own device, similar to how Google Stadia works. This solves some of the technical issues (because the game is running on your own device it doesn’t need to be set up on their computer). It also reduces some opportunities for leaking, so can be preferable to installing the game on the playtester’s computer. There is a video of how Parsec works on their website, as well as a case study of Ubisoft using it to distribute builds for a marketing event.
If your team are comfortable with participants installing the game on their own device, conference tools like Google Meet allow you to view the player’s screen, and talk to them live. It also has the built-in ability to record the session. Google Meet requires very little technical knowledge from the player, so is simple to set up – and it’s free!
Another alternative to consider is unmoderated research – asking players to play in their own time, and receiving a video recording. This has some downsides – it’s not possible to ask questions in reaction to the behaviour you are seeing in the session. However, it can often be logistically easier to organise.
Some teams use automated services like PlaytestCloud to do this. This allows teams to specify a ‘type’ of player, upload a build, and receive videos of people playing it. This eliminates many of the logistical challenges of distributing games and finding players and makes running unmoderated user research studies easier.
However not every team has the budget to outsource this (or are not working on a mobile game), and will have to recreate this setup manually. This can be done using tools like Obs or DScout (recommended by @eddiepearson) to capture videos from players, and combine that with a free survey created on Google Forms.
This kind of unmoderated research can allow us to see many more players than we could observe linearly. It can be very powerful for combining qualitative feedback (‘observing what they do’) with quantitative feedback from surveys.
Non Disclosure Agreements
The risk of leaks is difficult to avoid entirely. Non-disclosure agreements can warn participants and reduce the likelihood of them sharing confidential information. This template from the research agency ping-pong will allow you to create your own NDA, and includes a plan to get it signed remotely.
Some remote research tips
I asked others on LinkedIn for their tips about remote research.
Christian Ress from PlaytestCloud gave this great advice:
Some great tips for the study design of remote studies (and good advice to run a pilot. Pilots are always an essential part of my in-person research plan, but easy to forget when working remotely!).
For a different take to addressing the issues from the pandemic, Player Research spoke last year about how they had adapted their lab to be safe for in-person studies, and I know this is something other research teams are looking into.
Start your games user research Career
Every month, get sent the latest articles on how to start a career in games user research, and hand-picked job opportunities.
Plus get two free e-books of career guidance from top games companies