When running a playtest or user research study, pretty early on you need to decide ‘how many playtesters’ do I need. Plucking a number out of the air feels wrong, and magic numbers float around game design communities of ‘the right number of playtesters’.

For teams who are lucky enough to have an active community, it’s tempting to send out a big survey to tens of thousands of players (and then regret it when receiving thousands of answers to sift through).

For other teams who have to actively recruit each individual playtester, it’s hard to know when you’ve done enough, and it’s time to stop. 

Personally, for each playest, I recommend…

• Six players when you’re trying to discover problems with your game
• Twelve players when trying to understand and define players
• One hundred players for quantitative measuring questions

That’s per playtest. A single playtest round will be tightly scoped, and games require multiple playtests throughout development, depending on their complexity (Destiny had over 150 playtests throughout development!)

In this article we’ll explain the reason behind these numbers, and help you make a confident choice about how many playtesters you need.

Before we go on: Get a single email each month about playtesting and UX in games, with a playtest guide just like this one. Sign up at the end of this article.

Where does the five number come from?

There’s a classic article from Jakob Nielsen where his team looked at previous usability tests, and counted how many problems they discovered per participant. 

They ended up with a graph like this:

The reason for stopping at five participants is that most of the usability issues have been found, and more participants don’t add a huge amount of new problems discovered. 

At that point, the sensible thing to do is stop, fix the problems we’ve found, and test again to discover a whole bunch of new problems. 

This guidance is widely cited and used by teams running usability tests. Often incorrectly. 

Applying this rule without considering whether it’s appropriate for what you want to learn is risky. Especially with games, which are often a lot more complex than the websites that Nielsen’s study was investigating.

So what is the right number for games?

Making pragmatic decisions

When working out the number of playtesters, there is a difference between the ‘perfect’ number that would be appropriate for academic studies and a pragmatic number.

We don’t have infinite time or money, and game development is fast-paced. If we spend too long getting answers, the opportunity has been missed, and the playtest will be pointless. So we need to make informed compromises.

The numbers below are a starting point for making a pragmatic comprise, but other researchers may choose different numbers. Consider how important the playtest is, the level of trust you have with your team, and how much time you have, to make a final decision on how many playtesters to use .

Also keep in mind what the needs of your game are – if it’s a 4v4 multiplayer game, you’ll need 8 people! (Thanks Ben Taels for the prompt!)

The right number of playtesters depends on what you want to learn

The number of playtesters depends on the type of thing you’re hoping to learn from your playtest (the “research objectives”)

First decide…

Is my playtest asking measuring questions 

• Am I trying to measure something, like “How do players rate <<this>>?”
• Am i finding which problems exist in my game?
• Am I trying to define my players (for example, to make personas)?

The right number of playtesters differ for each of these.

How many playtesters for a survey?

First, a short stats lesson on the ‘margin of error’.

We can’t survey everyone. For example if I wanted to survey gamers aged 18-35, I couldn’t email every 18-35 year-old gamer in the world. And if I could, many of them wouldn’t fill it out. So we only get responses from a smaller group. 

This is normal, it’s how surveys work, and is called sampling.

Because we’re not asking everyone in the world, we have to accept there’s a chance that our sample’s answers don’t match exactly what we’d get if we did ask everyone in the world. 

The more people we ask, the less chance we have of it being inaccurate. The ‘margin of error’ is how far off the ‘true’ answer we’re prepared for our answers to be.

So, how many players do we need to get to fill out our survey to get a (reasonably acceptable) 5% margin of error

Getting more than 400 people to fill out your survey just isn’t usually worth it – you’re getting a similar quality of conclusion, and end up with a lot more responses to process and work through.

However 400 survey responses isn’t practical for many teams – if you don’t have easy access to your players, it’ll be very hard to round up and encourage 400 people to fill our your survey.

In my opinion, 100 players for your survey is a good, practical number that gets ‘good enough answers’ for most game design questions.

It’s approximately a 10% margin of error which is acceptable for many applied research settings where lives aren’t at risk, and is an achievable number of people to find (here’s one way to get one hundred playtesters).

How many playtesters are needed to find problems with your game?

Finding problems, such as usability issues (things players don’t understand or can’t do) is a common playtest objective. 

This is where Nielsen’s graph is most relevant – relatively early we’ve seen most of the most prominent problems that exists. 

I personally recommend going for six playtesters, and then stopping and reflecting on what you’ve found so far. 

Why six? Because sometimes playtesters don’t turn up, and even with a no-show, you’ll still have five players.

It’s very likely at this point you’ve found some high priority issues are worth addressing. Fixing those issues, and then planning another playtest to see if they are fixed is more valuable than just putting more players through this playtest. 

(Remember, this is six playtesters per type of user. – if you’re testing kids AND adults, you’ll want six of each.)

How many playtesters are needed to define your players

With this kind of qualitative research,  we’re looking for the point when we’re just hearing repeats of stuff we’ve already been told (“saturation”).

Reaching saturation depends on how complicated the thing we’re trying to understand is. Understanding people to create personas is more complex than finding usability issues, so the number of people you need to speak to will be higher.

It’s not possible to anticipate in advance how many people you need to speak until you tip over into mainly hearing repeats – my recommendation is to start with 12 (per type of user), and then look back at what you’ve learned before deciding if more interviews are necessary.

One player is infinitely more than none. 

If you’re not able to reach the numbers listed in this post, because of time or budget pressures, that’s ok. Any playtesting is better than doing nothing. Seeing even a single player will reveal behaviour and issues that you didn’t anticipate, and broaden your understanding of the game – helping you make smarter design decisions.

If it’s only possible to get two or three playtesters through the door – do still run your playtest – Iteration is the secret weapon of game development, many small playtests will add up to a huge impact on the game.

Where can I find playtesters?

It’s essential that your playtesters represent your real players – what’s the point of getting feedback or behavior from people who are nothing like your target audience?

I’ve written a post about how to find first 100 playtesters.