Running playtests and user research is costly – not only the direct costs for recruiting participants (or outsourcing analysis), but also the indirect costs – the time and attention it takes away from other production priorities.

Despite the cost, not running playtests is even more expensive – saving up problems for later in development when changes will be more harder and most costly to make.

This means most studios end up spending up to 5% of their budget on planning and running playtests, and studios report seeing enormous return on the time invested into showing their game to players (God Of War’s Ed Dearien credited it to giving their game a huge advantage and was a core part of their success). 

The value of seeing real players interact with your game means it’s crucial to ensure that you are running playtests at the most impactful time, but the expense requires stopping promptly when we’ve learned enough. In this article, we’ll look at ‘how much is enough playtesting’ throughout development, and inspire when and how you should be doing your playtests.

User research informs decisions throughout development

Because the goal of user research is to inform important decisions, the most effective mindset for user research throughout game development is to identify risky decisions, and use data collected from playtests and games user research to de-risk them. 

The type of decisions that your team will be making changes throughout the production process, and so the type of playtests, and ‘when is enough’ will depend on the level of confidence you and your team have in the decisions they have to make.

I’ve previously shared this graphic, in my article on ‘when to playtest’

Which gives some inspiration about the type of studies you can run to help the team at each stage.

But what’s missing is ‘when should we stop’? When does your team know enough, and the return on investment of running further studies drop – allowing you to deprioritise playtesting. 

Playtests during concept and ideation

Your teams goal when prototyping during a concept or ideation stage is to come up with one or more ideas that your confident will make a compelling experience. 

Understanding players, and their reaction to your prototypes, will not only build internal (and funder) confidence in your idea, but also improve the quality of your decision making throughout the development process – having seen players experience is first hand, you will make better assumptions about their behaviour in the future. 

You’ll have done enough playtesting and user research in the concept phase, when you answer positively to the following questions:

Do you have some ideas that you feel good about?  

Have you seen players interact with your prototype, and that it’s creating the intended emotional reaction? This can be difficult to objectively measure in a study, so it will require you and your development team to see players first-hand, and build an internal consensus that this idea is worth pursuing. 

If you’re unsure whether players interacting with your prototype will create the desired reaction, this is a sign you’re not ready to leave the concept phase and need to spend longer with your prototypes before entering pre-production.

Does your whole team understand your players well enough that you can predict their behaviour?

Throughout development the development team will need to make hundreds of decisions – from big decisions about core mechanics, to small implementation decisions (‘what should this button say’). 

Most teams won’t have budget to test each and every decision, and will have to prioritise their future studies on only the most critical aspects of the game – relying on the expertise and knowledge of your team to cover the rest. This process is much easier if you’ve created a shared understanding within your team of who your players are, their game playing behaviour, and their context of play – introducing important design constraints to consider when making decisions. 

Before leaving the concept phase, make sure your team have a shared understanding of who their players are, inspired by interviews with genuine players from your target market. 

Playtests during pre-production

In pre-production, it’s common to combine the mechanics explored in a prototype to make sure it can hang together as a full experience – building a fuller, more representative experience of near-production quality and testing not just your ideas, but also your production pipeline and how you will make the full game. 

You’ll know you have done enough to understand players if you can give confident answers to the following questions.

Do you know ‘building a full game of this’ will engage players?

Having seen people play with each of your prototypes and ideas in isolation during concept testing, you’ll know that each idea works in isolation. However you haven’t yet seen how these ideas work together.

Before you leave pre-production, you and your team will want to have seen players genuinely interact with the core loop of the game, and be confident that this creates a compelling experience players want to stick with.  

Do you know what you will be building in production?

Production can be unfortunately chaotic, and teams are often forced to de-scope rapidly to hit hard deadlines. By seeing players experience your vertical slice, you will be able to confidently prioritise what’s important to retain in production, and where cuts can be made without compromising the core experience. 

You’ll know you’re ready to enter production when you have confidently prioritised where your limited development effort will be spent, based on a true understanding of players’ opinions and priorities. 

Playtests during production

During production, the team is creating the bulk of the content of the game, while trying to minimise scope creep or re-work. What exactly playtesting in production looks like depends on the genre, but typically includes the creation of scenarios that the player will encounter, and player onboarding. 

Production requires a lot of implementation decisions from the development team – how exactly should we communicate our vision to players, and will they take the right emotional and behavioural response from it. This creates huge potential for the value of playtesting, which should run iteratively until you can answer the following questions.

Do players understand how to use each of the systems in your game?

The onboarding for the game, and the teaching of the mechanics within it is hugely important for a successful player experience. However teaching is hard (players rarely read text, and opt out of anything that overtly looks like a tutorial). Because of this, teaching how to play the game becomes a significant challenge.

By running playtests in production focused on tutorials and the FTUE, you will mitigate this risk – and you shouldn’t be leaving production without confidence that players understand how to play your game. 

Is your game a complete experience, and do players enjoy it?

During production, the game will come together as a complete experience. Regularly putting players in front of this and seeing their end-to-end experience of the game will help regularly pinpoint what’s working, and where friction is emerging.

When you know that players enjoy the end to end experience of your game, you will be able to confidently leave production, and ready your game for release.

Playtests to run in post-production or post-launch

After production, the scope for changes is a lot more limited, however balancing and small changes will be possible in this post-production phase, and after launch for live service or mobile games. 

Especially with Early Access, Free-To-Play or subscription based games, retention becomes a core metric – can we keep people playing long term, to justify the cost of acquiring players. This is an ongoing process, but you will know you have run enough playtests when you can answer yes to the following questions.

Do players have a consistent experience, free from difficulty spikes?

Compelling games create a flow state, balancing challenge intentionally to keep players hooked. The level of difficulty can often be tweaked very late in development, by subtly altering the scenarios players encounter (for example adding or removing a few obstacles). 

Playtesting, and measuring the experience throughout, is essential to understanding where there is too much (or too little) friction, and is best achieved with a combination of quantitative measurement, to identify problem areas, and qualitative probing, to diagnose why the problem exists. Once your measurements come back showing a consistent player experience throughout the game, you’ll be confident you can stop playtesting.

Is your retention meeting best-in-class metrics? 

It won’t be possible to keep 100% of your players forever – but the number you do retain will have a huge impact on the financial success of your game. Benchmarking your retention stats against best in class for your genre will help your team recognise when you have achieved enough, and the value of playtesting diminishes.

This can be explored with a diary study – looking at players experience over time, to diagnose retention issues you have identified. 

Once you’re confident your game has achived best-in-class metrics, it’s safe to stop post-production & post-launch playtesting – at least until the team decide to make new content, and dive back into ideation! 

Constant communication within your team leads to high impact studies

Although this article can give a broad roadmap of ‘how much playtesting is enough’, remember that the core value of user research and playtesting is to enable informed decision making. This means the start of each study should be ‘what decisions are we currently making’ and ‘what data will help us make those decisions’.

Continual alignment and communication across your development team will ensure that your studies are aligned with the team’s genuine priorities, and lead to high impact studies that matter. 

To learn more about studies at each stage of production, claim your free copy of Playtest Plus today.