Preparing a games user research study requires a lot of admin to make sure the right people turn up, the right version gets tested and that the study runs smoothly. Learn how to recruit participants, and prepare a great study.

Preparing a games user research study

Preparation before running a study will ensure the sessions run smoothly and successfully answer the research objectives. The preparation needed to run a games user research study is very similar to most types of user research study – with perhaps more complexity in the technical setup and code screening required, as we will see. 

Having earlier covered designing a study, in this section we’ll cover all of the main stages required to prepare for running the study, and highlight some of the challenges specific to researching video games. This includes:

• Finding research participants
• Understanding the game and the build
• The technology
• The space
• The paperwork
• Preparing the game team
• Piloting

Recruiting games research participants

Most research methods need users to take part in the study. This requires finding the type of people who would play the game once it’s released, inviting them to take part in the study, and then making sure they turn up.

This takes a lot of coordination and time, and it is sensible to start recruitment as soon as the study plan is complete. At this point, it will be clear how many users are needed and the session length. The amount of time recruitment takes varies based on the method used but can be between a few days and a few weeks.

Finding the right players

We previously covered how to run kick-off meetings. In this meeting agree who the participants of the study should be, which will prevent arguments later about the participants not being representative or suitable. Recruitment should be aligned with who the game is aimed at, and can be informed by data from other teams, such as the marketing department. Some sensible criteria to recruit on include:

• Experience with other games (e.g. This is a high budget first person shooter, so we’re looking for people who have played other big FPS releases from last year).
• Familiarity with previous games in the series (e.g. This is a sequel to a popular sports franchise, so we’re looking for people who played the last edition)
• Demographic information, to some extent – for example if it’s aimed at children. Recruiting based on age and gender amongst adults is not recommended because it risks reinforcing inaccurate stereotypes about who games are for, and furthers representation issues.

It’s important to spend time making sure that the study is looking for the right kind of players. Getting the wrong kind of players means that the study’s findings won’t be relevant to the real game design decisions being made. If a participant has never used dual-stick controls before they will have a bad time in a study looking at a first person shooter which assumes prior experience with those controls. This may be useful when designing tutorials, but isn’t relevant when testing other parts of the game, since the audience is meant to be experienced first person shooter players. 

Screening participants

To make sure that the participants are appropriate, it’s necessary to screen them before inviting them to take part in the study. Screening is the process of checking that they match the correct criteria for participation in that study. It’s reasonably common for people to sign up to take part in studies they are not suitable for, because of the money offered to take part, so checking they are suitable is essential to avoid mis-recruits. This can often take multiple stages:

• A questionnaire asking for information about what games they play, or checking other screening criteria, written so it’s not obvious which answer is the correct one to take part in the study. This questionnaire can also check their availability for the intended dates and times of the study.
• Some teams also phone participants prior to confirming their session. This conversation covers more in-depth questions to check they really have played the games they say they have. 

Having checked they are the right kind of person to take part in a study, people can then be booked in to the time and dates needed for the study.

Making sure people turn up

No-shows, when a participant fails to turn up, are expensive and embarrassing. Expensive because it wastes the time of the researcher and the designers or producers observing. Embarrassing, because everyone then has nothing to do, and are sat in an observation room twiddling their thumbs until the next session is scheduled to begin. To reduce the risk of people not turning up, there are three tactics to consider using.

When participants fail to turn up, this is an embarrassing and expensive error!

Firstly participants should be incentivised – given money, or something close to money such as vouchers, to pay them for taking part. This makes people more likely to bother to turn up, rather than deciding they can’t be bothered on the day. It also helps find more ‘normal’ users – people who will do it for free are likely to be either much bigger fans, or much more vocal critics, than an average player. Recruiting only unpaid participants will introduce a sampling bias in the kind of users taking part in your research sessions.

Secondly, consider recruiting one or more spare participants. This means booking one more participant than the study requires – either to wait around all day, or as an extra session at the end of the day. If people do fail to show up, the spare can be used to make up for the missing participant.

Send confirmed participants a calendar invite and phone them the day before to remind them the study is taking place, and the time and place it will occur. This will help avoid the session being accidentally forgotten, or misunderstandings about when and where they are meant to go.

Finding the right participants takes a lot of work and is a specialist skill. Many research teams either have a dedicated team member hired to handle this or outsource it to an external participant recruitment company. Recruiting participants is complex, and perhaps the most suitable thing to outsource if budget is available to do so.

Understand the game and the build

Identifying usability and experiential issues within a game requires understanding the game very well. This not only covers understanding the intention of the game – how it’s meant to work – but also the state of the build when it will be tested. This is sometimes called code screening.

Running studies prior to the game being finished will mean that the version being tested is incomplete. Also, because games are complex systems, it is very likely that there will be bugs within the version being tested. Often game teams will want to create a custom release (build) for the test. When preparing the study, researchers need to play this test version of the game enough to understand what it’s contents are, what the bugs are in it, and how to overcome any bugs or incomplete sections so that this can be handled while moderating a session. 

It’s also sensible to check the test build against the study’s objectives – is the required content in there that allows the objectives to be answered? If not it’ll require a negotiation with colleagues to decide whether to change the test objectives or provide an updated build. 

Understand the design intent

As well as understanding the version being tested, researchers also need to understand the goal of the game and the design intent behind it (hopefully the designers have thought about it!). The study design will require some observations to be made during the sessions (for example, where players get lost, or how many times they die). For each of these, understanding the design intent is necessary to help decide whether an issue exists. This can require a lot of conversations with the people who designed these features or scenarios to help uncover how they want the player to experience them, such as how many  times they want players to fail before solving a puzzle. It also might require designers to articulate and quantify their design intent, which they might not be particularly experienced with. Building close relationships with colleagues is once again particularly important to running successful studies.

Researchers need to understand the design intent, to earn trust

Time spent understanding the game, and the design decisions that have gone into it, will help a researcher run a successful analysis of the data that comes out of a study. This allows them to reliably identify the most important issues for the team. This preparation will also help researchers lead more useful discussions around fixing issues, informed by an understanding of what has occurred before, which will be explained later in this book.

Preparing the technology

Testing video games will inevitably require being comfortable with technology – perhaps more so than in other domains of software development. In addition to preparing the version of the game being tested, there is also technical setup required to record or stream what happens in the study.

Getting the build ready

How to prepare the build will depend on the hardware being used for testing – such as whether it’s a console, mobile or PC game. Regardless of the specific process, a researcher will need to install the build and test that it is working appropriately. Because the software is still in development and hasn’t gone through QA, there is a reasonable chance that technical problems will occur when installing the game into the test environment. Leave plenty of time to trial and troubleshoot this. 

There is a tradeoff that will need to be negotiated with the game team – often they will want to provide the final build as late as possible so they can continue to make changes, but the researcher will want enough time to install it and check that it works as intended. Giving a  deadline for the final build to be provided two days before testing starts can be a good compromise, giving the researcher enough time to react to problems. 

Setting up the research lab technology

A research study also requires some custom technology to allow the session to be observed or recorded. Many research teams have permanent spaces (research labs) that allow them to do this without having to set up the recording technology each time. Smaller research teams might have to put up with using a meeting room for running their studies. A second room will also be useful to set up an observation room – a dedicated space to encourage teams to view live. Offering live viewing increases colleagues’ engagement and understanding of what’s occurring in the sessions, and amplifies the impact of the studies. If a dedicated room isn’t possible, many tech setups allow the sessions to be streamed live to people’s desks.

Technical setups for recording video from the sessions that combine what happens in the game, with video and audio recorded in the room can be done reasonably cheaply, using a combination of screen sharing software, HDMI splitters and recording software. There are many guides online about how to build a user research lab that supports recording and streaming video for testing one participant at a time, and I covered some potential setups in my previous book Building User Research Teams. 

A difference between games user research and other industries is the need to host many players playing simultaneously, to support some of the methods explained previously. This greatly increases the technical complexity, and some advanced lab setups that support these studies have been covered by Seb Long in the Games User Research book, and at the #GamesUR Summit, the videos from which are available on YouTube on the GRUX SIG channel.

Preparing the space

As well as setting up the room to handle the technical requirements for running the study, it’s also important to think about the impact of the room on the participant’s experience, and the areas the participant moves through to reach that room. The room itself should be neutral and avoid intimidating or biasing the participant. Avoid an overly clinical aesthetic that makes people feel they are being watched in a laboratory, and avoid heavy brand marketing that may change people’s opinions about what they are playing. 

Ensuring that the participant’s experience is considered and curated throughout their interactions with the study will help create a more comfortable and natural environment. This might include giving them specific information about what to do on arrival, thinking about where they get taken when they arrive, and briefing any reception staff so that they handle the participants appropriately.

In addition to preparing the room that the participants will be in during a study, the space for observation should also be prepared. If the technical setup supports live streaming, it’s often sensible to book a space where observers can view it communally. This encourages discussion between members of the development team and allows members of the research team to sit in and help guide that discussion. Preparing this space can involve ensuring that the video stream can be seen by all using a projector or large screen, creating space where post-its can be captured, and providing refreshments to encourage attendance. Booking meeting rooms can often be difficult in busy offices, so try and reserve them with plenty of time in advance.

Preparing the paperwork

Running user research studies generates a lot of paperwork, and it’s easy to forget to make and print these in the lead up to a study. These documents can include:

• Pre-study information to be given to participants
• Information to be given to the building’s reception desk.
• Non-disclosure agreements for participants
• Consent forms for participants
• Any surveys or questionnaires to be distributed to participants
• A method to capture the notes during the study

Templates for many of these are available for free on the website for my previous book, Building User Research Teams but we’ll explain each in brief now.

Prepare participants before they arrive

Pre-study information for participants should be emailed in advance, and tell them when and where the study will occur, as well as other logistical information such as travel options, parking or what to do when they arrive. As discussed previously, no-shows can be both expensive and embarrassing for a researcher, and by making sure participants have the information to hand, and know what to expect, it will help reduce the chance of this happening. 

Creating a positive first impression

When participants arrive at the front-desk of the building, ensuring they have a positive experience will help reduce their anxiety and encourage more natural behaviour. It’s reasonably intimidating to go to a big corporate office, and making sure that any reception staff understand who they are and how to handle their arrival will make the experience less scary. To help with this, create a document for reception staff that describes who will be arriving and what actions to take on their arrival – who to call, and where to ask the participant to wait.

Consent and non disclosure

As covered in the first part of this book, secrecy is considered very important for many games with extensive marketing strategies. Create a non-disclosure agreement in collaboration with a legal team, and ask participants to sign it. This will help discourage leaks, and increase the studio’s confidence that running user research studies is safe. 

Running ethical user research requires the participants to understand what the study is about, what information will be gathered, and how their data will be stored and used. This is often handled by combining an in-person briefing from the moderator with a document that the participant can read, sign, and keep a copy of. Giving this information on a document with a verbal briefing helps ensure that the participant has understood and is giving informed consent – an essential ethical requirement. Prepare a document that explains the high-level goal of the study (without revealing too many details that may impact their behaviour), for example “We are interested in learning about your experience with the game to help improve it”. The document should also list the data that will be captured – e.g. audio recordings, video recordings, their survey responses. The document should also give instructions for how they can request a copy of their data or remove consent at a later point. 

Preparing questionnaires and surveys

There are two types of questionnaire that might be useful to prepare for a study, and these can be done on paper or on a computer using survey tools such as Qualtrics. Although the participants should have been screened before they were invited to participate, it can be useful to reconfirm their habits around what games they play at the start of a session. This can help identify mis-recruits where the wrong person has been invited, participants have lied, or someone other than the invited person has turned up. It can also make it simpler to use the data about their playing habits as part of the analysis, since they will have consented to that data being collected and used on the consent form. Questionnaires will also need to be prepared and distributed when the study design requires surveys either during gameplay (for example after every level), or at the end of the session.

How will notes be taken?

The last thing to prepare is the method of note-taking that will be used by any researchers working on the study. For structured studies, where the things being observed are all identifiable in advance, some teams like to use a spreadsheet for note-taking. For more unstructured studies, where the player has greater autonomy over what they do it’s not possible to anticipate in advance what feedback will be collected. In these situations mind maps can greatly speed up data collection and analysis. Some details on how to do this are also covered in my previous book, Building User Research Teams. 

Also don’t forget to print out all of the above paperwork, and the discussion guide created in the previous section, before the study starts!

Preparing the game team 

When running user research, it is important to ensure that the game team feels involved with the study. Their active participation increases the likelihood that the research study will impact their decision making, and justify the investment in running studies. As covered earlier, active engagement with the team to decide the research objectives, and working together to make sure the researcher understands the design intent is essential to running a useful study and interpreting the data correctly. 

One of the easiest ways to fail to get buy-in is by not involving them in the study being run. At the minimum, user researchers should be inviting them to view research sessions, and invite them to a debrief where the results are discussed. Sending these invites can be easily forgotten, and people have very full calendars, so invites should be sent early in study preparation. Immediately before the study, remind the game team that the study is occurring, what the objectives are, and how they can observe the sessions will help increase engagement.

With a more mature team and an experienced researcher, once a relationship has been established and their understanding of research has improved, more exciting collaborations can be explored – such as collaboratively analysing the data from a study to come up with the results together. 

Run a pilot test

Plenty of things can and do go wrong when running a study. These could include:

• The build crashes and players lose their saved game
• Bugs occur in the game that make progress impossible
• The recording device fails to record
• The microphone wasn’t on, and the interview doesn’t get picked up
• The consent forms weren’t printed out
• The survey has the wrong scale on it
• The task set to participants was misunderstood, and people played the wrong part of the game.

In order to reduce the chance that these issues disrupt the real study, it’s very important to run a pilot study. This is a practise run of the study, using a pretend participant (usually a colleague), pretending they are a real participant and running through the complete study. It’s tempting to skip bits during a pilot – e.g. not asking the fake participant to fill out the consent form, or playing less of the game than a real participant would. That can be necessary when time is short, but each skipped part increases the risk of not noticing a problem with the study until it’s too late.

Running the pilot the day before the study is due to begin means that the real test build can be used, avoiding the risk of bugs emerging between the pilot and the real test. The day before still gives enough time to react to most technical or study design issues that might emerge. 

More on Preparing a Games User Research Study.

In this section, we’ve touched on a lot of the tasks that a researcher will be doing to prepare to run a successful study. We’ve only scratched the surface, and there is a lot more work that can be done to describe and optimise these processes, as well as other considerations such as the secure handling of personal data. 
The processes required for running user research studies are covered in more depth in my previous book Building User Research Teams, which might be a helpful resource when establishing research at a games studio that hasn’t done it before.