The 2023 Playtest Survey

Understand the playtesting habits of the games industry.

In an industry first, we have called upon hundreds of game developers to share their thoughts, opinions and behaviour with playtesting.

Read on to learn how the games industry tackles playtesting, and learn how to improve your own playtests.

Introducing the 2023 Playtest Survey

From Steve Bromley and Jackson Herd

Playtesting is an essential part of game development – successful games need to be understandable, enjoyable and usable. UX research and playtesting is an important tool that game developers deploy throughout development to anticipate how players will experience their game, and make changes before it’s too late. 

But how do developers approach playtesting, and what does playtesting look like for most game studios? 

In this first of its kind survey, Steve Bromley and Jackson Herd invited over two hundred studios to share their playtesting practice – their approach, tools, and what’s difficult about playtesting. 

Our goal was to understand how playtesting really works with game teams of all sizes, including those without dedicated user researchers, and help push the games industry forward towards better games, and more reliable development approaches.

Thanks for taking the time to read – and learning with us about how teams are running high impact playtests throughout development.

 

A note on sample bias: This study was recruited via our games user research networks, and then outreach to every game team who has publicly recruited for a playtest on twitter in the last year. This will have an impact on responses – likely over-indexing on user researchers. A variety of incentives were offered to attract a broad variety of respondents to mitigate this. We’ve also previously run a lot of qualitative interviews with both user researchers + game developers about their playtesting approach, and the findings from this survey reflect the same themes, but be aware of potential sample bias! 

What do we mean by ‘playtests’

In qualitative interviews with game developers, it quickly became apparent that there is no consensus for what a playtest is. For some teams ‘playtests’ describes a specific user research method – a quantitative study, focused on learning ‘do players like the game’. For others it describes any sort of study with players, including ‘finding bugs’. For some it even includes “playing the game myself and seeing how it feels”.

For this study, we gave the definition of ‘any study that uses external players, regardless of research objective’. As you’ll see in the full results, this is used by game developers alongside a number of other methods for gathering data from players, and encompasses many objectives, from usability to appeal. Read on to learn more!

Everyone runs playtests

User experience isn’t the sole responsibility of a single department. 

While the majority of our responses are self-identified “researchers” (perhaps because of our network!), we also see plenty of responses from roles not directly associated with research. 

There are Playtest Survey responses from producers, project managers, community managers, quality assurance testers, and even artists. 

Each of these fields (and more) impacts the user experience. And each of these fields can bring a unique, valid perspective on the research process.

who took part in our survey

Playtesting matters!

Playtesting is important! Over half of our responses agreed that playtesting (defined as any activity that gets player feedback on your game) is extremely important. 

Throughout this survey we’ll see various data on playtesting budgets, time, compensation, and other key playtesting attributes. 

But we all understand the importance and impact playtesting has on game development.

(…and teams that don’t have probably opted out of our survey…)

Playtesting matters

What does Playtesting Look Like?

Playtesting will be relevant throughout game development.  In this first section, we look at which teams, genres, and development stages run the most playtests, and what we’re hoping to learn from them.

Playtesting is a Team Effort

Playtesting is a team effort. The disciplines that our respondents work with most closely to run playtest include Game Design, Production, and UX Designers/Researchers.

Cross-discipline communication is definitely a core skill for successful playtesting – the results will usually require teams to come together to prioritise and address the conclusions. Teams involved with overall game design, production, and hands-on creation of artifacts are the most commonly reported.

 The only common exception to this were indie projects where the entire team could be 2-3 people.

Who collaborates in a playtest?

Playtesting is for all games

The vast majority of playtesting happens on ‘traditional’ console or PC games – often they are the games that have the budget to support playtesting, and user research can have a huge impact on development from ideation to launch. 

Mobile games offer unique opportunities to evolve post-launch, and impactful studies to optimize retention, and are the second most popular category from our responses.

What type of games get playtested?

When to playtest during development?

The impact of early research can resonate throughout the lifespan of a project. Whether it’s saving development costs or providing design peace of mind. 

Our results reflect this, with the majority of responses labeling their current project as pre-alpha, alpha, or beta. 

There are some responses on post-launch projects, suggesting playtesting (and research) maintains a seat at the table well into a game’s lifespan – particularly for mobile games, where post-launch optimisation can have a huge impact on retention and churn.

playtesting throughout development

External Playtesting is only one part of assessing quality

Formal playtests, with external playtests, aren’t the only way of evaluating game design decisions. 

Internal playtests are still the most common method of getting feedback on games. Helping teams get the most value from these sessions and overcome internal bias is an exciting area for games user researchers to continue to explore. 

Alongside that, traditional research with external players continues to provide value throughout game development. 

different types of feedback

Do players understand and enjoy games?

Data on player understanding and enjoyment are key playtesting metrics for survey respondents. This makes sense, as data on comprehension and attitudes are difficult to gather outside of hands-on methods like playtesting. These metrics are vital for ensuring a game’s fundamental design is received the way it was intended. This is especially important when this design is intentionally challenging to understand, and/or enjoyment is not explicitly “fun”. 

Respondents also report playtesting is useful for trialing new ideas. Playtests are a great, low-stakes opportunity to test new ideas or adjust preexisting designs. While game understanding and enjoyment are key metrics, this is the fine tuning data that can really help polish a title.

And, of course, any playtests can result in players encountering bugs. It’s important to do your due diligence to document these bugs to be solved, or at least to avoid/fix them while playtesting in the future.

What is the point

Regular external playtests throughout development

When it comes to external playtests, most teams are running between two to five playtests a year – less than one every two months. 

Planning, running and analyzing a user research study can often take up to three weeks for dedicated user researchers, and this frequency creates opportunity for iteration between tests.

regular playtesting

Participant recruitment needs to get better

We asked participants what they would want to change to make playtesting easier, and coded their answers.

By far the most popular responses were focused on participant recruitment.

Having a more controlled amount of playtesters that were more understanding of the game’s state, but also enough to give us the adequate feedback we need.”


“Getting the right sample. Having more resources to screen players and trust the screening .

Finding playtesters can often be difficult for teams with budget limitations. This guide to finding participants can help.

Other challenges raised included capacity limitations, such as the need for research ops, or more staff, and challenges with the depth or efficiency of analysis.

We need more people to analyze the videos, maybe then we can opt for more playtests.

All of these challenges will be familiar to stretched research teams, and many require investment and culture change to overcome.

How to improve playtesting

How playtests are run

How do teams approach playtesting, and what are they learning from them?

How Long should a playtest be?

Although some teams are running studies of up to a week – the most common length for a playtest was between one and three hours.

That time is just about enough time to go beyond tutorials and explore the player’s experience in appropriate depth to find late-occuring issues. 

This is usually driven by the research objectives – how much do we need a player to see? To test a tutorial, that could be 15 minutes. To balance a full campaign – that might take a week!

How long is a playtest?

How many players do we need?

Almost half of all teams run playtests with <=10 playtesters, while others are running huge tests, anywhere from 100 – 2000 playtesters. There’s no doubt that some of the “<=10” category are due to necessity or convenience. But it also emphasized that any amount of playtesting, with 2 or 2000 players, has value.

The data also suggests there are multiple cohorts that could divide & drill into The Playtest Survey data. What methods and tools do researchers with <10 player playtests use? What about the methodology of 1000+ person playtesters? How does playtesting and sharing results differ or match across these groups? These questions are outside the scope of this survey, but  further research on these topics could reveal insights helpful to small or scaling research teams.

If you’re wondering about your own playtest, you might find this guide on how many players to use helpful.

Is ten players enough?

Finding players is hard!

If you’re not using a third-party agency or an in-house database, social media is your best friend for recruiting playtesters. There are many shared notes on playtester recruitment in social media marketing. We’d recommend looking into this guide from gamesuserresearch.com as a starter.

Also, don’t discount leveraging pre-existing fans (even for competitor games), networking, and friends & family for guerrilla playtesting – just stay aware of those biases!

Finding players is hard

But getting the right players is important

We asked this question, because qualitative interviews with game developers keep turning up teams who don’t prioritize participant recruitment. Time and budget constraints make it hard to put appropriate time and effort into recruiting. 

Although some teams take whoever they can get, the majority of teams believe they are putting in effort to find the most appropriate participants for their study.

Finding the right players

A rewarding experience for players

Incentives help convince people to give you their time. Money is the most common way of incentivising players to take part in playtests.

Money talks, but it’s not the only solution. There are teams who are not in a position to provide financial compensation (money, vouchers, game codes, etc).

While our results show it’s common to not provide any compensation, even something as straightforward as in-game recognition or Discord access should be considered.

Graph improved

Playtest Methods

There’s little substitute for moderated testing and player interviews, which were mentioned in roughly 50% of all responses. It’s especially great to see remote testing placing so high as the world continues to explore workplace flexibility.

Despite this, the convenience and impact of surveys have them at the top of our reported methods. In fact, just below testing and interviews, we see in-game surveys and community feedback. 

It’s important to keep in mind the biases and limitations of these methods (as with any method). But the ability to craft a survey or feedback funnel and passively gain insights is too attractive for teams to ignore.

Methods

Match methods to objectives

For the vast majority of teams, the objectives are driving which method they use – as one developer put it:

“It depends on the research question and the stage of development. Smaller sample one-on-one usability sessions for understanding new mechanics, if people can navigate a level, tutorial, etc. and larger sample playtests  typically when the game is a bit more functional and we’re interested in attitudinal perspectives. “. 

However other teams are constrained by the tools they have available, budget or time barriers. 

Match methods to objectives

Playtest tools

Tools can help speed up the process of planning, running and analysing playtests and user research studies. In this section we look at the most common tools used for games user research.

Tools to prepare a playtest

Playtests have a large amount of admin overhead, to arrange the participants, get everyone on the team aligned with the study, and convincing people to come and view the sessions live. 

As a consequence work-standard suites are very common in preparing for playtests, for example spreadsheet software for participant management, word software for creating a discussion guide, or kick-off document, and Jira or Asana for task management.

tools to prepare a playtest

Tools to run and record a playtest

There simply is no substitute for streaming and recording gameplay: OBS (Open Broadcaster Software) trumps every other response. It’s likely that OBS’ free pricing, open-source platform, and large user base make it useful to a broad range of scenarios.

Streaming

For capturing feedback, Google Forms and Qualtrics were tied in responses. There was also one response for the form builder service Tally, suggesting there are alternative and free solutions for researchers who want to avoid Google Workspace.

 

 

Surveys

We also saw multiple playtesting platforms. The most popular of these, PlaytestCloud and UserTesting, offer a suite of features to assist with gathering playtesters and processing results.Other teams are using Steam’s Playtest feature. Despite being platform specific, it’s a straightforward, pack-in choice if you’re already using Steam. Learn more about popular playtesting platforms on gamesuserresearch.com

 

One stop research solutions

Tools to analyze a playtest

Playtest action is happening in the spreadsheets.

Playtesting and user research often doesn’t require deep statistical knowledge, and so most other tools that provide opportunities for deeper analysis, such as R, or Tableu, are overkill (we used Google Sheets for this survey!) 

 

analyze a playtest

Tools to Report a playtest

Microsoft and Google suites dominated the results for sharing/presenting data. It’s hard to find an alternative solution to monoliths like Google Docs, Powerpoint, and Sheets/Excel.

But change might be coming! We see some popularity for wiki tools like Notion and Confluence. These tools do have some distinct advantages, especially when sharing content to playtesters or any external destination. One of our Playtest Survey interviews provided a perfect example of this. The developer was able to include important information for playtesters (Steam page, feedback form, build notes, recording software instructions, etc) in a single, sharable Notion page.

Overall many of these tools perform the same function, with minimal differences between them. While some are more popular, it’s important to find the option that works best for your workflows.

Sharing and presenting findings

Barriers to playtesting

What is stopping teams from doing more playtesting?

Playtesting can be hard

Playtesting can be hard. Some of the most common blockers are the time it takes to test, finding playtesters, and getting a testable build of the game. These will always need attention, but we recommend starting with this guide to finding playtesters to take part in your study.

Another common challenge is getting buy-in to convince stakeholders to invest in, and listen to the results from playtests. Advocacy and communication is a core skill for researchers – sometimes our most powerful tool is a cup of coffee and an informal chat. 

Some of the least blocking factors are understanding how to playtest, deciding on methods, and doing analysis – core research skills that we understand. If you do need to support with this, take a look at the games user research fundementals course.

Playtesting can be hard

More playtesting please

We heard that game developers would like to run more playtests than they currently do – creating the opportunity for more frequent iteration, and clarity on the vision and state of their game.

As we heard above, playtesting can be hard to prioritise, especially when builds are dodgy, players are hard to find, and everything else is on fire! We find that teams often get more value from many small moderated studies, rather than trying to run one big ‘perfect’ playtest – and encourage teams to be scrappy in their methods to unlock more frequent playtesting!

Quantity of playtests

A budget for playtests

How much does playtesting cost?

Playtests cost money

Over half of teams put some budget away for playtesting, which enables them to outsource some of the more difficult steps – most commonly finding players.

Budget can often hold back teams from playtesting, as we will see.

Others described the challenge of affording to run playtesting regularly, and are forced to use volunteer playtesters – increasing the time studies take (and the risk of sample bias!)

Playtest budget

What does a playtest cost?

Budgets are an incredibly sensitive topic, and it’s one of the reasons we made all questions in The Playtest Survey optional.

So it’s no surprise that only 39% of respondents chose to reply to this question with a dollar amount (adding “nda” and other privacy-based responses brings this number to 53%).

Of those that did respond with a dollar amount, a $0 amount was just under half of all responses (45%). This number balloons to 69% if we include spend from $0 – $100! – very affordable playtesting, and money probably spent on incentivising playtesters

We do have a few standout responses in the $1,000 – $5,000 range – with even a few respondents reporting anywhere from $16,000 – $40,000, who are likely to have outsourced the entire study to an agency. 

We very much appreciate everyone who took the time to complete The Playtest Survey regardless of how they responded to this, or any other question. As a reminder, all data from The Playtest Survey is anonymous.

What does a playtest cost?

How does playtesting fit into a development budget

About 36% of respondents were comfortable sharing data on their playtesting budget relative to their game’s budget. This response rate is similar to the response rate for the “spend on a typical playtest” question (39%). All that is to say these results are inconclusive but still interesting.

Out of the five options in this question, we only saw responses for the bottom 3 (no budget, 1 – 5%, 5 – 10%). There were no responses for “I don’t know”.

Over 50% reported no playtesting budget. While not surprising, it can potentially speak to the state of playtesting. 

A lot of playtesting is done with littlebudget, emphasizing the need for scrappy, low/no-cost playtesting.

To learn how to best spend your budget on playtesting, read our article on budgeting for games user research.

Budgetting for playtests

Continue learning

Where can you learn more about games user research and playtesting?

Where to learn about playtesting and games user research

Let’s start by recognising the sample bias. The survey was run on gamesuserresearch.com, which led to that being the most popular first option. 

Some of the ‘other’ options not mentioned include GDC videos,  PlaytestCloud’s Blog and John Hopson’s Secret Science of Games book.

A full list of resources mentioned is provided here:

updated list of resources

Playtesting Resources:

The resources that people submitted to our playtest survey.

 

Name Details
gamesuserresearch.com

by Steve Bromley

Games User Research Website 

How To Be A Games User Researcher Book

The Playtest Kit

IGDA – Games Research and User Experience SIG

(IGDA GRUX-SIG)

IGDA’s Games Research and UX Special Interest Group Website

Discord

YouTube Channel

Mentoring

Games User Research Book

by Anders Drachen, Pejman Mirza-Babaei, Lennart Nacke

Games User Research Book

ISBN-13: 978-0198794844

Game Developer Conference

(GDC)

YouTube Channel

GDC Vault

The Secret Science of Games

by John Hopson

Secret Science of Games Book

ISBN-13:  979-8987417812

The Beginners Guide to Playtesting Games

by Alexia Mandeville

Article

 

PlaytestCloud

From the PlaytestCloud Team

PlaytestCloud Blog

 

Antidote

From the Antidote Team

Antidote Blog

 

GUR café podcast

By Lanie Dixon

GUR Cafe

Available on spotify

Nielsen Norman Group

(NN/g)

10 Usability Heuristics for Games Article

Games User Research Intro

Video Game Design

The Art of Game Design: A Book of Lenses (3e)

by Jesse Schell

 

The Art of Game Design book

ISBN-13:  978-1138632059

Wrapping up

Thank you so much to all of the researchers and game developers who took part in this survey – we were overwhelmed by the response, and think it gives a great insight into current playtesting + user research practice in the games industry.

We also are sending a big thanks to Cait Goodale for her excellent support making the graphics for this report.

Do remember to share this with your teams, and happy playtesting!

– Steve Bromley and Jackson Herd, August 2023