Everyone runs playtests

User experience isn’t the sole responsibility of a single department. 

While the majority of our responses are self-identified “researchers” (perhaps because of our network!), we also see plenty of responses from roles not directly associated with research. 

There are Playtest Survey responses from producers, project managers, community managers, quality assurance testers, and even artists. 

Each of these fields (and more) impacts the user experience. And each of these fields can bring a unique, valid perspective on the research process.

Playtesting matters!

Playtesting is important! Over half of our responses agreed that playtesting (defined as any activity that gets player feedback on your game) is extremely important. 

Throughout this survey we’ll see various data on playtesting budgets, time, compensation, and other key playtesting attributes. 

But we all understand the importance and impact playtesting has on game development.

(…and teams that don’t have probably opted out of our survey…)

Playtesting is a Team Effort

Playtesting is a team effort. The disciplines that our respondents work with most closely to run playtest include Game Design, Production, and UX Designers/Researchers.

Cross-discipline communication is definitely a core skill for successful playtesting – the results will usually require teams to come together to prioritise and address the conclusions. Teams involved with overall game design, production, and hands-on creation of artifacts are the most commonly reported.

 The only common exception to this were indie projects where the entire team could be 2-3 people.

Playtesting is for all games

The vast majority of playtesting happens on ‘traditional’ console or PC games – often they are the games that have the budget to support playtesting, and user research can have a huge impact on development from ideation to launch. 

Mobile games offer unique opportunities to evolve post-launch, and impactful studies to optimize retention, and are the second most popular category from our responses.

When to playtest during development?

The impact of early research can resonate throughout the lifespan of a project. Whether it’s saving development costs or providing design peace of mind. 

Our results reflect this, with the majority of responses labeling their current project as pre-alpha, alpha, or beta. 

There are some responses on post-launch projects, suggesting playtesting (and research) maintains a seat at the table well into a game’s lifespan – particularly for mobile games, where post-launch optimisation can have a huge impact on retention and churn.

External Playtesting is only one part of assessing quality

Formal playtests, with external playtests, aren’t the only way of evaluating game design decisions. 

Internal playtests are still the most common method of getting feedback on games. Helping teams get the most value from these sessions and overcome internal bias is an exciting area for games user researchers to continue to explore. 

Alongside that, traditional research with external players continues to provide value throughout game development. 

Do players understand and enjoy games?

Data on player understanding and enjoyment are key playtesting metrics for survey respondents. This makes sense, as data on comprehension and attitudes are difficult to gather outside of hands-on methods like playtesting. These metrics are vital for ensuring a game’s fundamental design is received the way it was intended. This is especially important when this design is intentionally challenging to understand, and/or enjoyment is not explicitly “fun”. 

Respondents also report playtesting is useful for trialing new ideas. Playtests are a great, low-stakes opportunity to test new ideas or adjust preexisting designs. While game understanding and enjoyment are key metrics, this is the fine tuning data that can really help polish a title.

And, of course, any playtests can result in players encountering bugs. It’s important to do your due diligence to document these bugs to be solved, or at least to avoid/fix them while playtesting in the future.

Regular external playtests throughout development

When it comes to external playtests, most teams are running between two to five playtests a year – less than one every two months. 

Planning, running and analyzing a user research study can often take up to three weeks for dedicated user researchers, and this frequency creates opportunity for iteration between tests.

Participant recruitment needs to get better

We asked participants what they would want to change to make playtesting easier, and coded their answers.

By far the most popular responses were focused on participant recruitment.

Having a more controlled amount of playtesters that were more understanding of the game’s state, but also enough to give us the adequate feedback we need.”

“Getting the right sample. Having more resources to screen players and trust the screening .

Finding playtesters can often be difficult for teams with budget limitations. This guide to finding participants can help.

Other challenges raised included capacity limitations, such as the need for research ops, or more staff, and challenges with the depth or efficiency of analysis.

We need more people to analyze the videos, maybe then we can opt for more playtests.

All of these challenges will be familiar to stretched research teams, and many require investment and culture change to overcome.

How Long should a playtest be?

Although some teams are running studies of up to a week – the most common length for a playtest was between one and three hours.

That time is just about enough time to go beyond tutorials and explore the player’s experience in appropriate depth to find late-occuring issues. 

This is usually driven by the research objectives – how much do we need a player to see? To test a tutorial, that could be 15 minutes. To balance a full campaign – that might take a week!

How many players do we need?

Almost half of all teams run playtests with <=10 playtesters, while others are running huge tests, anywhere from 100 – 2000 playtesters. There’s no doubt that some of the “<=10” category are due to necessity or convenience. But it also emphasized that any amount of playtesting, with 2 or 2000 players, has value.

The data also suggests there are multiple cohorts that could divide & drill into The Playtest Survey data. What methods and tools do researchers with <10 player playtests use? What about the methodology of 1000+ person playtesters? How does playtesting and sharing results differ or match across these groups? These questions are outside the scope of this survey, but  further research on these topics could reveal insights helpful to small or scaling research teams.

If you’re wondering about your own playtest, you might find this guide on how many players to use helpful.

Finding players is hard!

If you’re not using a third-party agency or an in-house database, social media is your best friend for recruiting playtesters. There are many shared notes on playtester recruitment in social media marketing. We’d recommend looking into this guide from gamesuserresearch.com as a starter.

Also, don’t discount leveraging pre-existing fans (even for competitor games), networking, and friends & family for guerrilla playtesting – just stay aware of those biases!

But getting the right players is important

We asked this question, because qualitative interviews with game developers keep turning up teams who don’t prioritize participant recruitment. Time and budget constraints make it hard to put appropriate time and effort into recruiting. 

Although some teams take whoever they can get, the majority of teams believe they are putting in effort to find the most appropriate participants for their study.

A rewarding experience for players

Incentives help convince people to give you their time. Money is the most common way of incentivising players to take part in playtests.

Money talks, but it’s not the only solution. There are teams who are not in a position to provide financial compensation (money, vouchers, game codes, etc).

While our results show it’s common to not provide any compensation, even something as straightforward as in-game recognition or Discord access should be considered.

Playtest Methods

There’s little substitute for moderated testing and player interviews, which were mentioned in roughly 50% of all responses. It’s especially great to see remote testing placing so high as the world continues to explore workplace flexibility.

Despite this, the convenience and impact of surveys have them at the top of our reported methods. In fact, just below testing and interviews, we see in-game surveys and community feedback. 

It’s important to keep in mind the biases and limitations of these methods (as with any method). But the ability to craft a survey or feedback funnel and passively gain insights is too attractive for teams to ignore.

Match methods to objectives

For the vast majority of teams, the objectives are driving which method they use – as one developer put it:

“It depends on the research question and the stage of development. Smaller sample one-on-one usability sessions for understanding new mechanics, if people can navigate a level, tutorial, etc. and larger sample playtests  typically when the game is a bit more functional and we’re interested in attitudinal perspectives. “. 

However other teams are constrained by the tools they have available, budget or time barriers. 

Tools to prepare a playtest

Playtests have a large amount of admin overhead, to arrange the participants, get everyone on the team aligned with the study, and convincing people to come and view the sessions live. 

As a consequence work-standard suites are very common in preparing for playtests, for example spreadsheet software for participant management, word software for creating a discussion guide, or kick-off document, and Jira or Asana for task management.

Tools to analyze a playtest

Playtest action is happening in the spreadsheets.

Playtesting and user research often doesn’t require deep statistical knowledge, and so most other tools that provide opportunities for deeper analysis, such as R, or Tableu, are overkill (we used Google Sheets for this survey!) 

Tools to Report a playtest

Microsoft and Google suites dominated the results for sharing/presenting data. It’s hard to find an alternative solution to monoliths like Google Docs, Powerpoint, and Sheets/Excel.

But change might be coming! We see some popularity for wiki tools like Notion and Confluence. These tools do have some distinct advantages, especially when sharing content to playtesters or any external destination. One of our Playtest Survey interviews provided a perfect example of this. The developer was able to include important information for playtesters (Steam page, feedback form, build notes, recording software instructions, etc) in a single, sharable Notion page.

Overall many of these tools perform the same function, with minimal differences between them. While some are more popular, it’s important to find the option that works best for your workflows.

Playtesting can be hard

Playtesting can be hard. Some of the most common blockers are the time it takes to test, finding playtesters, and getting a testable build of the game. These will always need attention, but we recommend starting with this guide to finding playtesters to take part in your study.

Another common challenge is getting buy-in to convince stakeholders to invest in, and listen to the results from playtests. Advocacy and communication is a core skill for researchers – sometimes our most powerful tool is a cup of coffee and an informal chat. 

Some of the least blocking factors are understanding how to playtest, deciding on methods, and doing analysis – core research skills that we understand. If you do need to support with this, take a look at the games user research fundementals course.

More playtesting please

We heard that game developers would like to run more playtests than they currently do – creating the opportunity for more frequent iteration, and clarity on the vision and state of their game.

As we heard above, playtesting can be hard to prioritise, especially when builds are dodgy, players are hard to find, and everything else is on fire! We find that teams often get more value from many small moderated studies, rather than trying to run one big ‘perfect’ playtest – and encourage teams to be scrappy in their methods to unlock more frequent playtesting!

Playtests cost money

Over half of teams put some budget away for playtesting, which enables them to outsource some of the more difficult steps – most commonly finding players.

Budget can often hold back teams from playtesting, as we will see.

Others described the challenge of affording to run playtesting regularly, and are forced to use volunteer playtesters – increasing the time studies take (and the risk of sample bias!)

What does a playtest cost?

Budgets are an incredibly sensitive topic, and it’s one of the reasons we made all questions in The Playtest Survey optional.

So it’s no surprise that only 39% of respondents chose to reply to this question with a dollar amount (adding “nda” and other privacy-based responses brings this number to 53%).

Of those that did respond with a dollar amount, a $0 amount was just under half of all responses (45%). This number balloons to 69% if we include spend from $0 – $100! – very affordable playtesting, and money probably spent on incentivising playtesters

We do have a few standout responses in the $1,000 – $5,000 range – with even a few respondents reporting anywhere from $16,000 – $40,000, who are likely to have outsourced the entire study to an agency. 

We very much appreciate everyone who took the time to complete The Playtest Survey regardless of how they responded to this, or any other question. As a reminder, all data from The Playtest Survey is anonymous.

How does playtesting fit into a development budget

About 36% of respondents were comfortable sharing data on their playtesting budget relative to their game’s budget. This response rate is similar to the response rate for the “spend on a typical playtest” question (39%). All that is to say these results are inconclusive but still interesting.

Out of the five options in this question, we only saw responses for the bottom 3 (no budget, 1 – 5%, 5 – 10%). There were no responses for “I don’t know”.

Over 50% reported no playtesting budget. While not surprising, it can potentially speak to the state of playtesting. 

A lot of playtesting is done with littlebudget, emphasizing the need for scrappy, low/no-cost playtesting.

To learn how to best spend your budget on playtesting, read our article on budgeting for games user research.