Designing a games user research study

Answering research objectives requires selecting the right methods that will reveal players real behaviour. In this section learn how to design a study that answers usability and games UX questions reliably.

This is a section from the book ‘How to be a Games User Researcher’. Get the full book here.

Designing a games user research study

Having gathered research objectives, a user researcher will then create a study that can answer them. This involves:

  • Deciding the appropriate method or methods to use
  • Creating the tasks or prompts that the player will be given
  • Deciding how to capture data generated from the study
  • Deciding how to analyse the data generated from the study

The study then gets written down in a study plan (or discussion guide), which can be shared with the game team and used by the researcher to guide their sessions.

Deciding the right method

Following the kick-off, a user researcher will have a list of objectives that the study would like to learn, such as:

  • ‘Do players learn how to use the jetpack?’
  • ‘Can players identify the correct strategy for defeating this boss, and beat it?’
  • ‘Is the difficulty correct?’
  • ‘How does the jump feel?’
  • ‘Is this multiplayer map balanced?’
  • ‘Do players discover the secondary mode for the weapon?’
  • ‘Will players know where to go for their next objective?’
  • ‘Do players like the chase sequence?’

These then need to be matched to the right method to answer them. The next chapter goes into methods in more depth, but one way of grouping them is:

  • Behavioural Methods (which uncover what players are doing, whether players understand and can perform tasks as intended)
  • Experiential Methods (which explore what players think about the game)

There’s been a long-running debate in the research community about the correct way to describe these groupings, and so the language used will not be consistent across companies. Some synonyms for these are covered in the glossary at the back of the book.

Learning if players understand and can play the game

Some of the example objectives given above are focused on whether a player understands or can do something. These are:

  • ‘Do players learn how to use the jetpack?’
  • ‘Can players identify the correct strategy for defeating this boss, and beat it?’
  • ‘Do players discover the secondary mode for the weapon?’
  • ‘Do players know where to go for their next objective?’

These objectives would be answered by observing players behaviour and asking questions to reveal their understanding or the behaviour observed. This can then help explain why players are failing to learn or use the abilities that the designer wanted them to.

Balancing a game

Some of the objectives are focused on measuring the player’s experience and comparing it to a predetermined goal. 

These are: 

  • ‘Is the difficulty correct?’
  • ‘Is this multiplayer map balanced?’

These would be answered by recording what happens in the game – for example, how many times the player dies, where those deaths are, or which starting position causes teams to win multiplayer games more often. They would then be compared to the experience the designer intended – such as how many times did the designer want the player to fail on this level.

Discovering if players like a game

Some of the objectives are based around what a player thinks about the game. These are:

  • ‘How does the jump feel?’
  • ‘Do players like the chase sequence?’

These would be answered by discovering what a player thinks, understanding why they think that, and then measuring what percentage of people agree with that opinion. 

Categorising and priotising objectives

When planning a study, the user researcher will review each of the objectives in turn and decide what the appropriate approach would be to answer them. This is done by first categorising the type of research objective it is, and then deciding what method to use within that category. In a real-world games development environment, this requires a bit of pragmatism, and the ideal method might not be possible due to budget or time constraints. Remember that getting some quite good findings to a studio at the right time is much more useful than getting very good findings to the team too late. We look at some ways of speeding up the research process later in the book. 

Prioritising the research objectives will also help with this. Sometimes a study requires different methods that cannot all be ran reliably in a single study. When this occurs work with colleagues to decide which research objectives are more important, and choose the method that best supports those objectives. This will help decide how to assign limited researcher time across mixed-method studies.

Creating the tasks

Having picked the method, the next step is to define how the research objective will be answered. Each research objective will require:

  • The task or tasks that will be given to the player to create data
  • The way data will be collected

Sometimes the task will seem obvious. If the research objective is ‘can players complete the puzzle?’, an obvious task would be ‘get players to complete the puzzle.’. For some objectives, the task doesn’t even need to be explicitly given to the participant – in longer playthroughs they could be asked to play the game, and the researcher can be alert for the sections they are interested in without prompting the player. This creates more natural behaviour, which is a good thing.

For other research objectives, more specific tasks may be appropriate for the player, such as directing them towards completing specific parts of the game. There are some traps to watch out for when giving defined tasks to players during research studies. 

Avoid biasing players with the tasks

First, be careful about the wording of the task. The words used will impact a player’s behaviour, and could give them information they might not already have had. For example saying ‘please complete this puzzle’ would reveal to players that a puzzle exists. If their issue had been that they didn’t see the puzzle, or didn’t understand it was a puzzle, that issue would then be lost because the researcher has revealed the puzzle exists. Being very nondescript when giving instructions to players is sensible, starting with broad tasks such as ‘please play this section of the game,’, and carefully guiding further as required.

Another aspect to consider when designing the tasks is to recreate knowledge a player would have got from other sections of the game. If the study starts on level two, and misses the tutorials from level one, plenty of usability issues will be caused by missing the tutorials. This isn’t a fair test, and is not creating useful findings – in the final game, players would have experienced the tutorial, and so any issues caused by missing the tutorial can be dismissed immediately.

When writing tasks, make sure to consider what information a player would normally have by this point. Then give that information directly to players, so that the findings from a study are more representative of the real experience players would have. Often tutorials aren’t ready until late in development but can be recreated artificially in a study by the moderator teaching participants the game’s controls and features, or through written handouts. When learning the controls or mechanics aren’t the objective of the study, introducing them manually allows other issues to be discovered. 

Matching objectives to tasks

Each research objective will require one or more tasks. Run through each objective, and decide what those tasks will be. These can then be ordered in a way that will make sense for the player, so that the session flows in a logical order. These can then be documented as part of creating the study plan.

A list of research objectives in one column, with matching tasks in the second column
Each research objective will have at least one task associated with it.

One task can answer more than one research objective (and vice versa!).

Deciding what data to collect

Each task that the player performs will be generating data about their behaviour and opinions, such as:

  • What the player is doing in the game – where they go, what items they use, where they succeed and where they fail.
  • What the player’s opinion is, and why they are thinking that.

It is through capturing and understanding this data that the research objectives can be answered. When planning a study, it is necessary to anticipate what data will be generated, and how to collect it.

Measuring player behaviour

For each objective, think about how it could be measured. Some objectives are focused on performance or behaviour, and could be measured by observing what players are doing – are they able to complete the section? Do they know where to go? Do they get stuck for a long time? Observations can be performed either by the researcher, or from in-game analytics.

Other objectives are focused around a player’s thoughts, such as do they enjoy the experience, does the player believe it’s too easy or too hard, etc. This data can be captured by getting players to articulate their thoughts or opinions – either through asking them questions verbally, or asking for ratings or comments on a survey.

Understand the design intent

It’s really important with all of these task measurements to make sure that the design intent is understood – what did the designer want the experience to be like. Depending on what the designer wants, dying three times on a level could mean it’s too easy, just right, or too hard. There’s no way to tell which is correct without understanding what the designer wanted to happen. It’s also useful to capture how this made players feel, to allow the designer to assess if their design idea creates the intended experience for players. As a researcher, for each of the things you intend to observe, be sure to define what a good or bad result would look like before the study begins.

Understanding the design intent is essential to discovering usability issues

Capturing enough detail to address issues

It’s also important to ensure that the data captured will give you enough detail to explain what you learned to the development team. As well as noting what occurred, also uncover and note why it occurred. This will require probing players to reveal the reason for their behaviour or opinion. Players may have failed to go the right way; without asking the right questions it won’t be clear whether this is because they failed to see where to go, failed to understand it was where they were meant to go, or whether they understood it was the correct route, but had decided not to go that way for other reasons.

Some of the observations that a researcher should make can be anticipated in the study plan.

A table with three columns - research objectives, matched to task, match to how it will be assessed
Decide how data for each research objective will be collected, and what will be assessed

Defining what data will be captured, and how it will be obtained, will help prepare for the analysis stage later.

Write down the plan

Having decided what tasks will be set to the player, and how the data will be collected, this can then be used to create the study plan (or discussion guide) which describes the study.

This document can be useful for multiple reasons. The first is that by creating the plan, it gives the researcher confidence that they have covered all of the objectives with appropriate tasks and that the study has no gaps.

Secondly, it can be used to guide the session itself, and the researcher can refer to it when running sessions to prompt them on the questions to ask. Lastly, it can be useful to show to others – note-takers or other researchers working on the study to encourage consistency between how the sessions run.

Create the study plan

This can be created as a word document, using Google Docs or Microsoft Word. Some sections that may be useful to include, for referring to in the session, are:

  • Recapping the research objectives
  • The times sessions are booked for
  • The running order of each session, for example:
    • Pre Interview
    • Task 1: Tutorial
    • Task 2: Introductory Level
    • Task 3: Car Chase Scene
    • Survey
    • Post Interview
  • The introductory information for participants – e.g. explaining consent, the study, what data will be collected. This can often be templated, and more information will come later in the book.
  • The tasks and prompts that will be given to participants, with the data that will be captured, using the format described above.
  • Scripts for pre- or post-session interviews

This might look something like this: 

A word document sectioned as a study plan, showing the pre-interview questions and the first task set to the player
An extract from a discussion guide. A free template for this is available on the website for my previous book Building User Research Teams, BuildingUserResearchTeams.com

With the study plan created, a researcher has a lot of admin ahead to ensure the study runs as planned. Before covering that, we will look in more depth at some potential methods that might be used in a games user research study to answer research objectives.

Next: Games user research methods

Get the full uncut ‘How to be a Games User Researcher‘ book from Amazon.