Concept testing is one of the hardest types of study to get right. When a prototype exists you have something playable you can start to get genuine player data about their behaviour, and observe whether players are understanding and enjoying it. This is objective behavioural data on whether players are experiencing it in the way you hoped. However with a concept test, you typically don’t have anything playable, and are looking at one or more ideas for a game – often a pitch deck is all you have. 

The biggest trap with concept tests is asking players directly “would you play this”. Although players will give you an answer that sounds valid (“Yes, I love games in this genre” or “No, I don’t like the sound of it”), this answer is dangerous – it has very little relation to whether players will genuinely buy your game. Which is a shame, because ultimately that’s the question we’re trying to answer (before we spend three to five years building it!)

That doesn’t mean that it’s pointless to look at concepts with players – you are making big decisions during a concept phase and data will help inspire and inform your decisions. But we need to find a safe way of exploring your concept, and draw genuine data you can trust. In this article, I’ll give some guidance about how to do this. 

This is not market research

Market research can give broad answers about ‘does a game in this genre have the potential to be successful’, and can give advice on positioning. It is typically slightly removed from actual game production.

Our goal as user researchers is to inspire and inform design, development and production decisions. We understand that creativity is essential to great games, so our approach is to understand the team’s creative vision, identify what data would help you feel more confident about the decisions being made, and then provide it.

Concept testing is an interesting time for user research. Often the creative vision is still being defined at this point, so our studies will also provide raw inspiration for creativity, in addition to seeing how players react to the ideas so far. 

Players cant see the future

The reason why ‘will you buy this’ is a risky question is because players can’t anticipate their future behaviour – just like you will not be able to tell what films you’ll be watching in three years. Players are limited by time and budget, and won’t play games just because they like the idea – and they can’t tell us today what games will cut through. 

What players are very good at though is telling us the past – what have they played before in this genre, why did they buy it, and what was good or bad about it. Your concept should be a prompt for these more reliable discussions of previous behaviour, rather than future focused. 

Exploring past behaviour alongside your concept will not only reveal some of the risks or concerns you should be aware of with your concept, but will also help improve your literacy with your players – what do they think, how do they make decisions, how do they act – which will help you feel more confident making player-focused decisions throughout the rest of development. 

the right method for a concept test

Concept tests work best as a qualitative method – spending time talking 1:1 with each player to go beyond surface level ratings and opinions to understand their true behaviour and sentiment. 

You’ll get a lot more inspiration and data from interviewing ten players than from a survey of thousands.(and running this as individual interviews will avoid the group-think of focus groups.)  Knowing that 65% of people play your competitor games ‘for competition’ is too high-level to be useful for any genuine design decisions. Instead we want to uncover genuine stories about past good experiences and bad experiences with competition in games, and analyse ‘why that was good or bad’, in order to get anything but useless stats.

As always appropriate recruitment is incredibly important (because you need to trust the data represents your true players). Ignore demographics and focus on player behaviour – do they purchase & play games from your competitor set should be the primary drive for who you speak to (and remember to offer an appropriate incentive to take part, to move beyond the most vocal internet opinions and find true representatives of your real player base.) 

You’ll also get value from speaking to people who have lapsed or rejected previous games, to help to learn what to avoid (thanks Ben Lewis-Evans & Elizabeth Zelle for the prompt!)

You will hear divergent opinions during your interviews. Some teams might combine interviews with a wider survey after to see ‘how representative the opinions are’. However the true value is in knowing these opinions exist, so if you only have time for one – do the interviews! 

What should we show players in a concept test

Successful concept testing requires getting your vision of what the game is into the player’s brain so they can react appropriately. Players’ imagination will fill any gaps (and probably not with the same thing you’re intending to build). So we want to represent anything that doesn’t exist yet with something that best matches your design intent. 

Teams often have a lot of success using materials from their pitch deck – a combination of a sizzle reel that has been put together for pitching (combining any prototype footage with relevant examples from other games or movies) with a one page pitch, and then deeper descriptions of each pillar for the game. 

Text is incredibly open to interpretation, so where-ever possible supplementing it with images or video (including from other games) will help convey your vision to players as clearly as possible. 

What should you do in a concept test

Remember, the data we can trust most is players’ past behaviour, so we should always be asking them to frame their answers based on what they have done before. 

Start the session with a discussion about their habits with games from your competitor set. What have they purchased before and why? What have they liked and why? What have they disliked? This generates a lot of raw qualitative data that will help you channel your players better in the future.

Then we want to ‘show the thing’ – put them in front of the one pager or sizzle reel. The most useful discussions we can have at this point are “what’s interesting about this idea’ and ‘what concerns do you have about this idea’. Following up on the concerns, with questions about ‘how has that influenced your behaviour with previous games’ will help you understand what concerns would put players off your game, the potential impact of that (has it stopped them buying or caused them to abandon games before). This will give you a list of concerns you’ll want to mitigate during development. 

Following this format with each of the core pillars of ‘what games have done this well before for you, and why’ and ‘what games have done this poorly’, will further help your team’s decisions about how to implement the main pillars of their game concept. 

Dont just report what players say

The second biggest trap I see with concept tests is forgetting to do analysis of what players told you. To move beyond surface level impressions, we want to spend time thinking about ‘why did players say this’ (and later ‘what actions can we take to change that perception’), rather than taking every statement at face value. 

Concept tests are challenging but extremely valuable – they are the earliest possible opportunity to start to understand how your players think, and improve your ability to anticipate player behaviour throughout development. Decisions made during the concept phase will have huge ramifications on the success of your game – and better decisions now will help further milestones land on scope and budget.