Are your playtests delivering what you need?
I’ve worked with hundreds of teams, and been exposed to teams working at each level of playtest maturity. A frequent question that comes up with teams I speak to is “how good are our playtests?”
This often comes with worries about low impact playtests – teams where no-one is listening to the results, or the findings come too late, or player feedback is not trusted. When tracing the issue back, flaws in their playtest process are often the root causes of low impact studies.
The data being this matrix is based on seeing how these teams are approaching playtesting, and intends to help you recognise ‘what good looks like’. This will start to identify gaps in your process and increase the impact of playtests (and let me know if you need help with that!).
By looking at your own current level in each of the six defined categories, you can identify where your team currently stands, and the most impactful places to improve your playtest process.
Note: In industry, studios frequently have different definitions of what ‘playtesting’ is. For some teams it describes any type of study where feedback is gathered about the game. For others it describes a specific method – usually a multi-seat playtest with many players taking part simultaneously. For this work I’ve gone with the broader description – anything that involves collecting data from genuine players playing your game.
Playtest Maturity Matrix
The PLAYTEST MATURITY Levels
There are four levels defined in this matrix, and six criteria that cover the most important areas playtests need to deliver to be useful and impactful. It’s likely that your team will be ahead in some areas, and behind in others – this tool is intended to help you work out where the most impactful areas to improve your playtesing are.
Level 1 – Emerging Playtesting
At level one, playtests (and other user research approaches) are not actively planned into the development process, and are forgotten about until the opportunity emerges. This impacts all stages of running a good study – recruitment is ad-hoc, objectives aren’t linked to the decisions the team are making, and analysis is informal or non-existent. Although it’s great to create opportunities to see how players react to your game, there are a lot of easy changes that make the most of these opportunities and help make better informed design decisions.
This level is often found in teams who have no history of running playtests or user research, or who are just the start of their journey – perhaps smaller teams who have historically had no budget for user research, or are too time stretched to give it any attention.
A lot of the smallest teams I encounter are at level one in some areas of their playtesting process, and just starting their journey – benefiting from products like the Playtest Kit to start to formalise their process. Changes at this stage are driven by producers or designers who have seen best practice elsewhere, and want to implement it in-house, to de-risk their development process.
Level 2 – Structured Playtesting
At level two, teams can execute a reliable games user research study (and are familiar with a lot of the best practices on how to run games user research playtests). This is often where teams who book a single playtest with us are at with their own internal processes which can have a huge impact, but they haven’t yet built this as a regular part of their workflow.
This is often the level I see publisher-based teams work at, functioning as an internal agency to take in study requests, execute on them, and deliver a report at the end. This process is simple to implement, but lacks the deep integration into the team’s decision making process required to have the most significant impact. Without being embedded in the team it can be difficult to surface opportunities at the right time, or follow through to ensure the results cut through other voices developers are hearing about their game.
Teams running studies internally at this level might have bought a license to a playtesting platform, and started on their journey – they are able to execute on a playtest, collect some data, and turn it into a report, but not yet achieved the cross-disciplinary engagement required to take it to the next level and make playtesting a guiding light for their development process.
Level 3 – Elite Playtesting
At level three, teams have started to identify the factors blocking progress – deep integration between the research process and the development team, and are taking steps to address this. This is usually achieved by bringing studies in-house, and embedding the research process with other disciplines.
At this level studios have a clear understanding of who their players are, have made a research roadmap to identify what their most risky development decisions are, and can deploy a full range of research tools in order to provide relevant and timely answers. Researchers are embedded within a team, and are working in a culture that supports evidence based decision making – leading to high impact studies.
This is often the level where teams are looking for consulting support, on helping them achieve this level for a title, or across their studio as a whole. Teams can also get here by ensuring they are running a reflective research practice – running a reflective session after each study to identify best practises, opportunities for improvement, and building that into a standardised research toolkit.
Level 4 – Industry Leading Playtesting
At the highest level, research becomes a team responsibility – each discipline’s expertise combines into one shared understanding of the state of the game, and the experience of players encountering it. Teams are making cross-disciplinary decisions, in a culture of transparency, and can handle a rolling roadmap of decision making and evidence capture in a continuous feedback loop.
When I speak to teams at this level, I’m very impressed that they have recognised the barrier for user research (and more broadly, making decisions based on evidence), isn’t just about the process of running studies, but instead requires cultural change. Creating an environment where research matters is a long term, and much harder, prospect than running the study itself – and requires an ongoing commitment, alongside clarity of vision with a shared understanding from all decision makers on ‘what are we trying to make’.
The Playtest Maturity Criteria
This playtest matrix covers six of the key areas to ensure playtests are effective and impactful.
- Research Objectives: Are you defining what you want to learn from each playtest?
- Participants: Are you finding people who genuinely represent your target audience?
- Methods: Are you picking the right data-collection method for what you need to know?
- Analysis: Do you have a structured process to interpret the data you are collecting?
- Inspiring Action: Are your playtest conclusions clearly mapped to team decisions?
- Integration with Development: Are you intentionally planning when to run your playtests?
You can find your own level in each of these criteria, to come up with an overall idea of where your team’s playtest maturity is.
How to level up your playtests.
The first step is to identify where you are currently in each of the six criteria . This can be done alone, but can also be a valuable group exercise (or speak to me about an external audit)
After working out your current level in each criteria, look for outliers and opportunities – where are you most behind, or most ahead. For many of the level one and level two processes, I have written articles about how to improve the quality of your playtesting, which can be a great first step. Higher level changes requires cultural change and so requires more bespoke work to understand how decision making happens in your organisation, and to create an approach designed for your unique challenges.
Although it’s tempting to aim straight for level four, you’ll find more success moving gradually up the ladder for each criteria individually – at higher levels ‘a new process’ won’t be enough to create lasting change, and making player-centred games is a long term process.
If you’re on this journey inside your organisation, join thousands of other game developer professionals and get my articles each month directly into your inbox, alongside my guide to the most essential playtests to run at each stage of development.
Integrate player insight throughout development
Every month, get sent a new guide on how to plan and run efficient high quality playtests to de-risk game development.
Plus your free early-access copy of ‘Playtest Plus’ – the essential guide to the most impactful playtests to run throughout development of your game
