Throughout game development, teams have questions that require quantitative answers. “Do players enjoy this?”, “Is this too easy or too hard?”, “How do players rate the story”. These all require ratings, comparisons and benchmarking to reliably answer. However pre-launch, game development can often be too secretive to run these studies remotely, due to fears that footage, or the build itself will leak. 

Because of this, ‘multi-seat playtests’ are common in games user research – inviting ten or more players to your office to play the game simultaneously. Running multiple participants simultaneously allows us to see enough players to start to answer quantitative questions, while maintaining the security of a lab based setting. 

I’ve spent over a decade planning and running multi-seat playtests. In this guide we’ll cover practical guidance on how to run them yourself – from when they should be run, to setting up an appropriate space, and tips on how to moderate and analyse your data. 

Drop me an email if you’re running your own multi-seat playtest and still have questions – I appreciate it’s one of the more complex and stressful studies to set up.

What is a multi-seat playtest

A multi-seat playtest is when multiple players are invited in to play through an in-development version of your game. As with all user research, it should use properly screened players (matching your player definition), and be designed to answer specific research objectives that you’ve co-defined with the wider team.

What makes multi-seat playtests different from one-to-one playtests is that you are seeing more than one player at the same time. Running players through sessions in parallel makes the logistics of running the study much more efficient – twenty players can see ten hours of gameplay in just two days, whereas it’d take weeks to get those players through that same amount of gameplay with a one-to-one study.

However having multiple players going through the game simultaneously also changes the nature of moderation – you can not reliably watch people’s gameplay (and will generate too much footage to review in a reasonable timeframe). This will push the methods for data collection away from direct observation, and towards survey responses & interviews. We’ll cover more on the implications on data collection later.

When should you run a multi-seat playtest?

As a quantitative-focused method, multi-seat playtests are most appropriate when you want to measure something. In game development, measurement questions usually occur during the latter half of development, late in production or post production. 

Quantitative questions are often about balance – do players enjoy each level equally, are their any dips in their enjoyment, or difficulty spikes. (Read more about when to run different types of playtests). Before getting to this point, you have hopefully tested many of the systems (and how each system is taught) in isolation in a one-to-one setting, to discover usability issues. 

Another pragmatic reason to run a multi-seat playtest is when there is a lot of content to get through – as mentioned above, it’d take weeks to get ten players through a single player campaign otherwise. This might start to become relevant around the Alpha milestone, where there is enough content to require a larger test, but still have capacity to make changes to your development priorities based on what players are responding positively to. 

How many players do you need?

Picking the right number of players requires balancing the need to gather a lot of data, with pragmatic decisions about time and budget. Because we’re looking at quantitative data 

A large lab might have capacity for anywhere between ten and twenty players at one time, and it’s common to run multiple batches of players. This has to be balanced with ‘how long will it take each batch to get through enough gameplay to answer our research objectives’. 

This means I often end up landing on 20-40 players for these types of playtests, depending on the length of the game. I’ve written more on picking the right number of players here.

Collecting data from a multi-seat playtest

Because you’ll have multiple players playing simultaneously, close observation of player behaviour becomes impossible. Not only can you not watch each player individually, but as the number of players increases, the chance of technical issues multiplies, meaning you’ll spend a lot of your time restarting computers, fiddling with debug settings or fixing headphones. 

This means you’ll largely have to rely on self-reported data. Getting players to fill out regular surveys (e.g. after every level) is reasonably common, probing on players 

The exact questions should be inspired by your research objectives, but some common topics to ask about include:

• Players rating of the section of gameplay they just played
• Players ratings of difficulty
• Players rating of understanding/confusion
• Players rating of the controls

With associated free-text boxes, allowing players to explain why. 

You can read more about designing survey questions in this post. 

Surveys alone can be limited as a data-collection method. Players tire, and write short, vague and incomplete answers, and because you don’t look at the data until after the study is complete, you can fail to get enough detail to meaningfully understand and diagnose the issues raised.

This means you’ll also want to combine it with an actively moderated discussion. Ideally this would be 1:1 interviews (so you can give each individual player time and space to articulate their thoughts in a safe environment). However without lots of moderators present that can be difficult to manage (especially while other players are still playing and likely to encounter technical issues). 

This means it’s reasonably common to end the session with a group discussion. After capturing everyone’s individual sentiment with a final survey, getting all of the players together to have a final group discussion can give you a qualitative understanding of what players are thinking. When well moderated, this qualitative exploration will give you enough detail to fully explain the issues highlighted by surveys, and gather enough detail for your team to take confident action based on it. 

Setting up the space for a multi-seat playtest

Running a multi-seat study requires a lot of space. Each player will require an individual pod, including:

• A gaming device
• A recording method (such as a capture PC)
• A method of distributing surveys (either on paper, or on the capture PC)
• Headphones (otherwise people won’t be able to hear their own sound)
• A webcam (to capture player audio and footage)

Apart from requiring an appropriately high spec device to play the game, the rest of the set-up can be achieved at a reasonably low cost. Software such as obs are free methods of recording and mixing the game feed and webcam feed. Overtime this can be scaled to more reliable hardware based recording methods (I’ve used Epiphan products before for this). 

When setting up the physical space, think about where to place participants to minimise bias. If players can see other people’s screens, that can create pressure (when they realise they are behind the others), or reveal unwanted information (like seeing other players behaviour), which can unnaturally change players behaviour. Putting up dividers between each player can reduce unwanted information from being passed between players. 

You can rent lab space (and gaming PCs) for these kind of studies – googling ‘research labs’ for your local area can throw up a few options.

Running these sessions remotely can be difficult – not only as you’ll have to distribute the build to a wider group of players (while maintaining security and reducing leaks), but also using player’s own pcs will massively increase the number of technical issues they encounter (and reduce your ability to fix them). That might push you into doing an unmoderated version of this study, combined with 1:1 interviews or focus groups, rather than trying to run it as a live playtest remotely. 

Timings and the flow of a multi-seat playtest

Because you’ll have to manage multiple players at the same time, these sessions can take longer than you’d imagine – which will have an impact on how much gameplay you’ll get through. In a four hour session, expect players to see about three hours of the game, especially if you’re asking them to complete regular surveys during gameplay.

A typically flow could include:

• An introductory briefing
• Players playing through some gameplay. As players complete sections or levels, interim surveys get their feedback
• Players fill out a longer survey at the end of the session
• Players take part in a final roundtable discussion

There will be delays throughout the session. When you’re inviting a large number of players to attend at the same time, some will arrive late. You should make a decision before hand whether to start without them, and also whether latecomers will be admitted or sent away. If you do allow people to join late, you will have to re-run the introductory briefing for them, so plan how you’ll handle that. 

Players can spend a long time on their final survey, especially if they have a lot of free-text boxes, because it’s their only chance to give their own individual feedback. It’s very common that some players spend at least 30 minutes on even reasonably short surveys, so leave plenty of time.

Similarly, if you’re ending your session with a group discussion that will require at least thirty minutes also – and the time will fly by. 1:1 interviews would also help uncover individual player’s feedback, but require a lot of extra moderators to administer.

For longer sessions (anything over 2.5 hours for adults) you’ll likely want to have breaks for snacks or lunch. Players will naturally want to talk about the game (because it’s the only thing they have in common). Remind them not too, and seed the conversation with other topics (“what are you playing currently?”) to avoid accidental focus groups emerging!

Running multiplayer tests

Multi-seat tests open up the potential to run multiplayer tests. Largely the logistics of this are similar to running single player multi-seat tests, but with some additional logistics to consider:

When recruiting, think about how you’d expect people to play this game once launched. Is it meant to be played by existing groups of friends? Or is the intended experience to be played with strangers? Make your recruitment criteria match the intended player types – for example by recruiting existing groups of friends when required. Friendship groups can tend to copy each others answers during surveys, so be on the look out for this and ensure everyone is giving their own individual feedback in surveys during the session, rather than collaborating and creating group-think about their answers inside their team.

Multiplayer games often rely on dedicated servers. Based on experience, I can guarantee that these will go down at some point during your study. When preparing, make sure to find out in advance who to message when the server goes down (preferably an instant message rather than an email, as players will be waiting so it’s time sensitive). Send them a message in the weeks before warning them about the upcoming test (to avoid any planned downtime), and another message in the day before the test, to remind them. Also when piloting the test, check the multiplayer is connecting appropriately – firewalls can make it more difficult to test online games than singleplayer games.

Many multiplayer games are intended to be played with live voice chat. This will require some extra hardware and software preparation – getting all of the players microphones with their headphones, and setting up a voice chat server on Discord or Teamspeak before hand. You’ll want to control where your players sit, and put them in the correct voice chat servers before hand, to avoid having to do this live during the session.

In multiplayer sessions, you might want to sync the videos from different players later for analysis of highlight clips or – adding a timestamp to your video can help you synchronise across different players in your post-video edits.

What other questions do you have about running multiplayer playtests? Do drop me an email and let me know, and I’ll update this article!

Analysing multi-seat playtests

These playtests will generate a lot of data, including:

• Your observations of player behaviour
• Quantitative ratings from players surveys
• Qualitative comments from players surveys
• Qualitative opinions from focus groups or 1:1 interviews

Each of these are fine to handle in isolation – visualising the quantitative data, and performing thematic analysis on the qualitative data to find the meaning (you can spend an hour watching me analyse a playtest here), and will just take time to work though.

The larger challenge is likely to be ‘turning the data into a meaningful story’. One key ingredient for creating a compelling story is to start from your research objectives – remind yourself again of the questions that inspired you to run the test (“how do players feel about the difficulty”, “overall what is working and not working with the game”), and use your data to provide answers to these questions. 

Prioirtising your findings will also help you focus on the core messages – here’s a guide to prioritising your research findings – and remember that when it comes to impact, less can often be more. Telling a team the five or six most important things will go better than telling them a hundred issues, because people can’t handle that much raw information, or act upon it.

Get help running multi-seat playtests

Running these sessions is a core skill for games user researchers, but requires bravery to get started with, due to the size of these studies. As with all studies, a pilot session is especially important, because the consequence of the build failing or the room audio not being captured will be so much more wasteful than other research methods (due to the expense of recruiting so many participants per session).
If you’re looking for a multi-seat study for your own game, I run multi-seat studies, and train research teams in how to run studies themselves. And if you have any questions about running your own multi-seat studies, do get in touch.