Moderating playtests requires a structure to make sure that the correct topics get covered. Learn how to run a games user research study that generates the right data to answer game design questions.

A games user research session is not significantly different from other user research sessions. The overall structure is likely very similar, and could include:

• Introducing the study to the participant
• Screening and interviewing the participant
• Performing some tasks
• Interviewing the participant 
• Wrapping up the study

Although the specific contents of each step will be different based on the objectives and method of each study, there are some common themes when working with games that are worth covering. 

This section will briefly describe each part of a user research study, and introduce some of the things games user researchers should be considering as they follow their study plan and execute the study.

The start of the session

As described in the previous section, when a participant arrives at the building, both they and the front-desk should be briefed on what happens when they arrive – typically waiting somewhere out of the way to be collected by the researcher. Many studios also ask the participant to lock away any recording devices at this point – such as mobile phones – to reduce the potential for leaks.

When collecting the participant, remember that this isn’t a typical experience for them and they may be feeling more concerned or cautious than normal – so be kind! 

What to explain to the participant at the start

The study plan created previously contains prompts for the information that should be delivered to the participants at the start of a session, which can include:

• Introducing who you are
• Explaining what data will be collected during the study, how it will be stored, and asking participants to give consent for their data to be collected and processed.
• Explaining the non-disclosure agreement and asking participants to sign it.

Be aware of local laws around consent and the use of personal data. Although this is more lax in North America than in Europe, we still have an ethical duty to ensure that participants know what data is being collected, how it will be used, and how they can remove consent. The standards required by Europe’s General Data Protection Regulation (GDPR) are a good start for ethical research.

This is also an appropriate time to explain the non-disclosure agreement (NDA). Introduce what an NDA is, and check with participants that they understand they are agreeing to not talk about what they see or do in this session. Keeping secrets requires a degree of trust with participants – one way of achieving this is to describe to participants why it’s important to keep what they see secret and how that allows us to continue to run studies to improve the game by running sessions that gather their feedback. Ask participants to sign the NDA, and provide them with a copy.

Introducing the study

Having gathered appropriate consent, a researcher should introduce the study to ensure that the participant understands what is required. This can include:

• Outlining what will happen in the session, and giving participants the chance to ask any questions.
• Explaining your relationship to the game – if you didn’t make it, tell the participant so and that they can give honest (and brutal) feedback without hurting your feelings.
• Explain that you’re not testing them, you are testing the game and that they should say if anything seems more or less difficult than they would expect
• Explaining think-aloud (as described in the methods section earlier), if the study requires participants to talk out loud about their experience.

Templates for a standard version of this introduction are available for free on the website www.BuildingUserResearchTeams.com 

What happens in the session

The specifics of a user research session will depend entirely on the study design and is difficult to generalise. Perhaps the most common types of study a games user researcher will run are one to one usability sessions, and multi-seat playtests which rely on some observation supplemented by surveys to understand player behaviour and sentiment. 

A typical study structure

For many study designs, the researcher will want to start with some questions for the participant, either asked verbally or through a questionnaire, to check that they are the right kind of participant for this study. These could ask about their typical playing habits – how much they play, what they play. This allows the researcher to put findings about their behaviour into context.

Most study designs will be asking participants to perform the tasks described in the study design. These tasks can often be very broad, such as ‘play the game as you normally would at home.’. For more nuanced tasks you might want to give the participant a printed prompt that describes the task in case they forget (and ask them to verbally read it out to you to check they have understood it). 

Some tasks can have success or fail criteria – such as ‘Upgrade your jetpack.’ With these tasks, ask players to tell you when they believe they have completed it. This avoids accidentally confirming when it’s been done correctly, or that it hasn’t yet been completed yet through your moderation. Also when creating tasks, remember to avoid giving away information the player doesn’t already have – ‘Use the pause menu to upgrade your jetpack’ tells the player which menu allows them to upgrade their jetpack, and would fail to uncover if that was a usability issue.

It’s common for research sessions to then end with an interview to probe the participant’s understanding and thoughts about what they did in the game. This gives the researcher an opportunity to see if misunderstandings remain about how the game works, which points towards further usability issues.

Moderating sessions

Many study designs require a researcher to moderate the session – interacting with the participant directly to ask questions or hand out tasks. This takes practice to achieve the right balance between being open and friendly, without influencing the participant’s natural behaviour or thoughts.

Keep your own opinions private

Moderators need to be neutral throughout the session. They should avoid introducing new ideas into the discussion that haven’t been raised by the participant, and avoid confirming or dismissing the participant’s opinions. It’s critical that the moderator doesn’t introduce their own opinions – this will put social pressure on the participant to change their opinion or not share their true thoughts openly. Failing to be neutral will not only bias the data being captured from the participant, but also reduce the game team’s trust in the researcher’s objectivity and the quality of their conclusions – reducing the impact that  research studies will have.

Ask probing questions

However, a moderator cannot just sit in silence for the session, they have to actively engage with the participant to confirm and explain what is being observed. Because researchers cannot see into participants’ brains, it’s not possible to tell why players are behaving in the way they are, and it can be dangerous to assume. A player might head the wrong way, but without asking questions it’s impossible to tell whether this was intentional or not, and why they did it – essential information for designers deciding whether to fix an issue, and how to do so. This means that researchers have to confirm and supplement their observations with information from the player on why they acted that way.

To achieve this, moderation requires an arsenal of bland probing questions which elicit information without giving anything away. These can include…

• ‘What are you up to currently in the game?’
• ‘What are you thinking here?’
• ‘What’s your goal?’
• ‘How did you discover you had to do that?’

… with follow up questions to explore their response in appropriate depth, and using pauses, or repeating the participant’s statements back at them, to create the opportunity for the participant to share more.

How to know when you have probed enough 

A helpful technique to ensure that a researcher is getting the required amount of information from participants is to anticipate what information the game team will need to take action. This means not just observing what the player did, but asking enough questions to understand why they did it, what the player believes is happening, and what factors within the game influenced their understanding. It also means observing what the participant did next – such as were they able to recover from the issue. Keeping in mind what a game team will need to know can help a moderator assess whether they need to probe further, or keep silent. 

Another technique that moderators often use is pretending not to be an expert on how the game works. This will reduce the participant’s fear of looking stupid, and also can encourage players to share their thoughts and understanding. Being humble, rather than acting like an expert who is analysing the participant, will create a friendlier atmosphere and reveal more of the participant’s natural behaviour.

As covered in the method section, many user research studies involve having one moderator supervising multiple players at the same time. Although this involves less one to one interaction with participants, a lot of the same skills still apply – creating a safe space where players feel comfortable sharing their thoughts, and making appropriate interventions to ask questions where required. 

Capturing data during the session

Studies generate a lot of data. Some of this is collected automatically, such as analytics generated within the game or surveys players fill out. Other data requires a researcher to capture it, including observations of player behaviour or what is said during interviews. 

This note-taking task cannot be done accurately live while moderating sessions, but can be done by a second researcher, or by reviewing recordings of the session. When capturing notes, recording details of which participant this note relates to, and the time stamp for the event will make it easier to correlate notes with the recordings. Some software allows timestamps to be added automatically, or via a keyboard shortcut, and some teams create custom scripts to make this easier. Correlating the notes with the recording can be useful when the notes are incomplete, and make it simpler to create screenshots or videos to help communicate the issue. 

Note takers should capture everything

When note-taking research sessions, it’s good practice to capture everything that occurs – and save the task of deciding which notes are meaningful until analysis. Trying to note-take and evaluate the worth of each note simultaneously will lead to observations being missed which may later be relevant. It is safer to over-capture, and then make evaluating the notes a separate task as part of analysis. When capturing observations, also remember to capture what occurs after the incident being described, to help evaluate the priority of the issue. If a player gets lost and finds their way after 15 seconds, that is likely to be less important than an issue where the player is lost for 20 minutes. Capturing the resolution of each issue allows this kind of prioritisation to be done.

Mind maps can help

We’ve covered previously how time sensitive games development is, and there often isn’t enough time to rewatch all of the sessions in full. Capturing high-quality notes first time will help ensure the results are ready faster and make it easier for the results to be ready in time to influence design decisions.

As mentioned when preparing a study, mind maps can speed up the process of capturing notes, in a suitable format for speedy analysis. More on this is in the previous book Building User Research Teams.

Wrapping up the study

After all of the tasks have been completed and any questions required by the study design have been asked, the session can wrap up. It’s ethical to remind the participant of the consent covered at the beginning – what data has been captured, and what the participant can do if they want to retract consent. It’s also sensible to remind them of any non-disclosure agreement they signed.

The participant will want to know about the process for getting the agreed incentive – are they being given cash immediately or will money be transferred to them. Explain this to them, and answer any other questions they have, while keeping within the limits of your own non-disclosure agreement.

Having run as many sessions as is required by the study design, a lot of data has been generated and it’s time to start analysis.