Categories
Playtest Masterclass Research Skills

Top remote playtest platforms for unmoderated testing

Playtesting doesn’t just have to be in person, and online playtest platforms can take a lot of the hassle out of recruiting and organising people to take part in your playtest.

Finding genuine players, rather than just asking other game devs, friends or people at a conference vastly improves the quality of the playtest data you get back – and many of these tools make it convenient to access a wide range of player types.

This article introduces some of the most popular playtesting platforms, gives an overview of their features and shares my experience with each.

Before we go on: Get a single email each month about playtesting and UX in games, with a playtest guide just like this one.

What is a remote playtest platform

Online playtesting platforms offer a way to get playtest feedback on your game remotely. Most involve uploading a build of your game to their platform, writing some tasks or questions, and then they handle distributing those to players. 

You get back raw playtest data, such as videos of people playing your game, their survey responses, or analytics from your game build. 

This can save a bunch of time for finding playtesters, but comes at a cost – so can be out of reach for teams on a tight budget (here’s a free alternative method of recruiting players).

Not all platforms are equal though. They have different functionality, users, and specialities which means you should look beyond just ‘price’ as a distinguishing factor.

Top remote playtest platforms

PlaytestCloud

Playtest Cloud

Platforms: Mobile (iOS and Android)

Price: from $69 per participant, including video footage

Can they find participants?: Yes, included in the price

Can they help with the analysis as well?: Yes

I’m a big fan of PlaytestCloud, and they are trusted by many studios of all sizes for their mobile testing expertise.

They support a large variety of study designs, including multiplayer testing and longitudinal studies – and their price includes videos of each participant playing and survey responses. I’ve also seen a sneak peek of some of their upcoming features, and am excited about the potential for using their platform for more of my playtesting needs.

Their platform ensures that testing + game code is kept confidential, and they also offer support from in-house and external expert researchers to help design or analyse studies, for teams who are not confident designing robust playtests.

They also support testing with children + teenagers (while meeting ethical and legal compliance requirements) which is extremely valuable for teams who work on games for kids.

For teams who run regular playtesting, they offer a subscription package which brings the cost per participant down further, and also bespoke cheaper packages for indie teams. 

Find out more on the PlaytestCloud website. 

Antidote.gg

Antidote.gg

Platforms: PC, browser and mobile

Price: from $29.50 per participant, without video footage

Can they find participants?: Yes, included in the price

Can they help with the analysis as well?: Yes

Barcelona based Antidote are a growing provider of playtesting services, and support a wide variety of study designs including multiplayer, or longitudinal studies. They would currently be my top recommendation for testing a PC game, based on some lovely chats with their team. 

For an additional cost, they can provide gameplay footage, or support more complex participant recruits.

They also have an expert in-house team of researchers who can help design, run or analyse your study. 

Find out more on the Antidote.gg website

Playtesting.Games

Playtest.Games

Platforms: Mobile (Android only)

Price: from $39 per 15 minutes, including video footage

Can they find participants?: Yes, included in the price

Can they help with the analysis as well?: Yes

I recently met the team behind Playtesting.Games, who have been working on their playtest platform since 2019. Building on their history of building communities for gamers, their platform offers the ability to test in a single session, or multi-sessions over time, making it suitable for diary studies.

Uniquely, their platform also offers verification of players’ previous game history, by scanning the playtime of other apps – giving confidence to game developers who want to make sure they are getting legitimate players.

In the near future, they are adding the ability to record the player’s face via their phone’s camera. Playtesting.Games are also developing an SDK which allows their playtests to be triggered within a game engine, again offering options to target players once they reach specific milestones in the game with a survey or recorded playtest (and have an early version live now).

Find out more on Playtesting.Games website

Gotestify

GoTestify

Platforms: PC, iOS, Android

Price: Enquire for details

Can they find participants?: Yes

Can they help with the analysis as well?: No

Built by a team of former game developers, GoTestify supports a variety of study designs, including usability studies, multi-session testing and multiplayer tests.

Their platform promises a degree of automation to identify the most important moments automatically (although I haven’t seen this in action, so can’t verify!). This could potentially be very valuable for time-stretched teams, but I’ll hold off judgement until I’ve seen it’s impact on a real project.

They are relaunching their platform shortly, so we’ll learn a lot more soon.

Learn more about GoTestify

GameTester.gg

GameTester.gg

Platforms: PC, browser and mobile

Price: from approx $5 per participant, without gameplay footage

Can they find participants?: Yes, included in the price

Can they help with the analysis as well?: No

GameTester is focused around gathering quantitative data – both through surveys, and in game analytics. 

Surveys are only appropriate for some study designs, and the lack of video footage will make it unsuitable for many research objectives, but the price is very reasonable!

It’s not a platform I’ve used personally (although I will give them a go soon), but one to keep in mind if you are running a quant-focused study.

Learn more about gamestester.gg on their website.

Playcocola

PlayCoCola

Platforms: PC and browser

Price: from free, with gameplay footage included

Can they find participants?: Yes, for an additional cost

Can they help with the analysis as well?: No

Targetted directly at indie developers, Playcocola promises the full range of data collection, at an indie-friendly price. Their software allows playtesters to record their gameplay, spoken thoughts and written comments, and bundles that all up for developers to watch and analyse. For playtesters, it doesn’t require any special software installation or registration – making it easier to include players who are less confident with technology.

The default packages requires developers to source their own playtesters, but they also offer access to their playtest panel is possible at an additional cost.

Playcocola is reasonably new, but one to consider if already have a panel of research participants ready to go, (perhaps gathered through this guide!) or want to get started with playtesting with a small budget.

Learn more about playcocola on their site. 

Steam Playtest

SteamPlaytest

Platforms: PC

Price: Free, no gameplay footage or survey data

Can they find participants?: No

Can they help with the analysis as well?: No

If releasing a game on Steam, they also have an inbuilt way of distributing playtest builds, called ‘steam playtest’. This is free, but has several limitations that make it not suitable for professional quality user research.

First, it’s difficult to screen or control who takes part. Players sign up to a waiting list, and are let in at random based on the amount of empty slots you provide. This means that we have no insight into who is taking part in the playtests, no ability to screen players out, or to treat different segments of players differently.

Second, it’s extremely limited how you can get data back from your players. No tasks are given, no video footage is captured, and no surveys are distributed. To get data back from players will require either integrating telemetry, or to coordinate outside of Steam’s ecosystem – which is also challenging as developers have limited control over who can access their playtest. 

I’d recommend extreme caution using Steam Playtest – it encourages lax standards for your playtest around finding the right players, and capturing only convenient, rather than appropriate data. Those bad habits require expertise on how to avoid them biasing your conclusions.

Learn more about Steam Playtest

Don’t let the platform tail wag the dog…

Playtesting platforms make running some types of playtests convenient. But convenience isn’t the only thing we should consider when deciding the appropriate way to run a playtest.

The most common trap I see developers fall into is ‘deciding the research method before confirming what they want to know’. And playtest platforms encourage certain methods (usually unmoderated remote playtests supported by surveys). 

Always start by confirming ‘what do we need to learn from this playtest’ before even starting to think about ‘which method would be best to answer this’.

Sometimes that will be a method that a playtest platform supports. Other times, it might just be popping to the local coffee shop with a laptop and showing someone your game.


Need help with picking the right method, and running the right test for your game? The Playtest Kit is designed to help make running accurate playtests simple for time-stretched developers.

Continue growing your playtest skills

Found this helpful? We’ll be exploring all of these topics in more depth in the future. Get a single email each month about playtesting and UX in games, with an article just like this one.

Categories
Playtest Masterclass Research Skills

How many players do I need for a playtest

When running a playtest or user research study, pretty early on you need to decide ‘how many playtesters’ do I need. Plucking a number out of the air feels wrong, and magic numbers float around game design communities of ‘the right number of playtesters’.

For teams who are lucky enough to have an active community, it’s tempting to send out a big survey to tens of thousands of players (and then regret it when receiving thousands of answers to sift through).

For other teams who have to actively recruit each individual playtester, it’s hard to know when you’ve done enough, and it’s time to stop. 

Personally, for each playest, I recommend…

  • Six players when you’re trying to discover problems with your game
  • Twelve players when trying to understand and define players
  • One hundred players for quantitative measuring questions

That’s per playtest. A single playtest round will be tightly scoped, and games require multiple playtests throughout development, depending on their complexity (Destiny had over 150 playtests throughout development!)

How many playtesters to use
Find problems with my game - six
Define my players - twelve
Measure player opinions - one hundred

In this article we’ll explain the reason behind these numbers, and help you make a confident choice about how many playtesters you need.

Before we go on: Get a single email each month about playtesting and UX in games, with a playtest guide just like this one.

Where does the ‘five’ number come from?

There’s a classic article from Jakob Nielsen where his team looked at previous usability tests, and counted how many problems they discovered per participant. 

They ended up with a graph like this:

A graph showing how many usability issues are discovered per user.
At 5 users, around 80% of the issues have been found.

The reason for stopping at five participants is that most of the usability issues have been found, and more participants don’t add a huge amount of new problems discovered. 

At that point, the sensible thing to do is stop, fix the problems we’ve found, and test again to discover a whole bunch of new problems. 

This guidance is widely cited and used by teams running usability tests. Often incorrectly. 

Applying this rule without considering whether it’s appropriate for what you want to learn is risky. Especially with games, which are often a lot more complex than the websites that Nielsen’s study was investigating.

So what is the right number for games?

Making pragmatic decisions

When working out the number of playtesters, there is a difference between the ‘perfect’ number that would be appropriate for academic studies and a pragmatic number.

We don’t have infinite time or money, and game development is fast-paced. If we spend too long getting answers, the opportunity has been missed, and the playtest will be pointless. So we need to make informed compromises.

The numbers below are a starting point for making a pragmatic comprise, but other researchers may choose different numbers. Consider how important the playtest is, the level of trust you have with your team, and how much time you have, to make a final decision on how many playtesters to use .

Also keep in mind what the needs of your game are – if it’s a 4v4 multiplayer game, you’ll need 8 people! (Thanks Ben Taels for the prompt!)

The right number of playtesters depends on what you want to learn

The number of playtesters depends on the type of thing you’re hoping to learn from your playtest (the “research objectives”)

First decide…

Is my playtest asking measuring questions 

  • Am I trying to measure something, like “How do players rate <<this>>?”
  • Am i finding which problems exist in my game?
  • Am I trying to define my players (for example, to make personas)?

The right number of playtesters differ for each of these.

How many playtesters for a survey?

First, a short stats lesson on the ‘margin of error’.

We can’t survey everyone. For example if I wanted to survey gamers aged 18-35, I couldn’t email every 18-35 year-old gamer in the world. And if I could, many of them wouldn’t fill it out. So we only get responses from a smaller group. 

This is normal, it’s how surveys work, and is called sampling.

Because we’re not asking everyone in the world, we have to accept there’s a chance that our sample’s answers don’t match exactly what we’d get if we did ask everyone in the world. 

The more people we ask, the less chance we have of it being inaccurate. The ‘margin of error’ is how far off the ‘true’ answer we’re prepared for our answers to be.

So, how many players do we need to get to fill out our survey to get a (reasonably acceptable) 5% margin of error

A chart showing the number of players you need to fill out your survey.
For a population of 1,000 its 285
For a population of 10,000 its 385
For larger populations up to a million it's 400

Getting more than 400 people to fill out your survey just isn’t usually worth it – you’re getting a similar quality of conclusion, and end up with a lot more responses to process and work through.

However 400 survey responses isn’t practical for many teams – if you don’t have easy access to your players, it’ll be very hard to round up and encourage 400 people to fill our your survey.

In my opinion, 100 players for your survey is a good, practical number that gets ‘good enough answers’ for most game design questions.

It’s approximately a 10% margin of error which is acceptable for many applied research settings where lives aren’t at risk, and is an achievable number of people to find (here’s one way to get one hundred playtesters).

How many playtesters are needed to find problems with your game?

Finding problems, such as usability issues (things players don’t understand or can’t do) is a common playtest objective. 

This is where Nielsen’s graph is most relevant – relatively early we’ve seen most of the most prominent problems that exists. 

I personally recommend going for six playtesters, and then stopping and reflecting on what you’ve found so far. 

Why six? Because sometimes playtesters don’t turn up, and even with a no-show, you’ll still have five players.

It’s very likely at this point you’ve found some high priority issues are worth addressing. Fixing those issues, and then planning another playtest to see if they are fixed is more valuable than just putting more players through this playtest. 

(Remember, this is six playtesters per type of user. – if you’re testing kids AND adults, you’ll want six of each.)

How many playtesters are needed to define your players

With this kind of qualitative research,  we’re looking for the point when we’re just hearing repeats of stuff we’ve already been told (“saturation”).

Reaching saturation depends on how complicated the thing we’re trying to understand is. Understanding people to create personas is more complex than finding usability issues, so the number of people you need to speak to will be higher.

It’s not possible to anticipate in advance how many people you need to speak until you tip over into mainly hearing repeats – my recommendation is to start with 12 (per type of user), and then look back at what you’ve learned before deciding if more interviews are necessary.

One player is infinitely more than none. 

If you’re not able to reach the numbers listed in this post, because of time or budget pressures, that’s ok. Any playtesting better than doing nothing. Seeing even a single player will reveal behaviour and issues that you didn’t anticipate, and broaden your understanding of the game – helping you make smarter design decisions.

If it’s only possible to get two or three playtesters through the door – do still run your playtest – Iteration is the secret weapon of game development, many small playtests will add up to a huge impact on the game.

Where can I find playtesters?

It’s essential that your playtesters represent your real players – what’s the point of getting feedback or behavior from people who are nothing like your target audience?

I’ve written a post about how to find first 100 playtesters. Also if you sign up for the free playtest masterclass on playtestkit.com it guides you through the process, including templates for defining your players and writing the pitch. 

Continue growing your playtest skills

Found this helpful? We’ll be exploring all of these topics in more depth in the future. Get a single email each month about playtesting and UX in games, with an article just like this one.

Better playtesting today

If you’re interested in turbo-charging your playtesting, I’ve worked with game designers, producers, community managers, UX designers, QA managers and solo game devs to make The Playtest Kit.

It brings the expertise of over 25,000 playtest hours into one complete playtest toolkit – making playtesting accessible to game developers who have no time or money.

Learn more, and sign up for the free one week playtest masterclass at playtestkit.com

Categories
Newsletter Research Skills

How to make time for games UX research

Developing a game can be a project management disaster, lurching from crisis to crisis until we hit the launch date (and then diving straight into developing the DLC!)

For UX researchers or designers this can make it difficult to do our job ‘properly’. Our processes require time, and there never seems to be time to do proper research, explore design options, or think about what we’re doing. 

UX gets left as an afterthought.

Often UX practitioners aren’t given the space to do the job they were hired to do  – they know what they are meant to be doing, and have the UX skills required, but feel stuck or unable to do their job. And iteration becomes a pipe dream.

This is frustrating, and leads to disengagement or burnout. It doesn’t have to be that way.

By the end of this article, you will have a library of immediate and long-term tactics, to create the time for proper playtesting, research, UX design and iteration to happen. 

Game development is a crisis that lasts 2-5 years

Games are very complex software to develop.

Unlike apps, or websites, it’s often not possible to launch a ‘minimum viable product’, and the game has to be complete at launch. Plus marketing puts huge pressure on meeting strict deadlines. And the ‘fun of the game’ is often discovered on the way, so can’t be pre-defined! This adds up to something that is almost impossible to project manage.

These factors means game development often is rushed, and difficult to plan – leading to changing priorities, crunch, and panic everywhere. 

Panic makes proper UX difficult

UX research and design practices assume that we have time to think about and react to what we learn from users and playtesting. 

This requires coordination between teams (which often isn’t being done elsewhere), asking questions to understand the intended vision, and creating a cohesive and shared understanding of the game.

When you’re not given the time to do it properly, the impact of UX design and research is lost. UX designers end up just making wireframes (and I thought the point was “we’re not UI”). User research and playtesting becomes a checklist item, rather than real inspiration for design changes. It creates a culture of low quality, low impact work.

Not doing your job properly will make you sad

For you as an individual, this reduces job satisfaction, and can make it hard to get new roles – if you’ve just been doing low-quality work, how can you pass an interview at another company?

HOWEVER

Trying to fix a low-maturity UX environment is a pro-level skill, that will make you stand out as a senior practitioner. Doing the slow, frustrating work to understand the environment, and improve it is what will help you move up in expertise, get promotions, ace job interviews and grow your career. 

My favourite people are those who try and fix the environment, rather than just complaining about it. Here’s how we can do that.

Short-term fixes to get out of an emergency

A common situation in a panic-filled environment is an incoming, high-priority request which has to be done immediately. Due to poor planning and communication, suddenly a research study needs to be run today. 

Obviously this compromises ‘doing good research’. Without appropriate time or attention given to participant screening, using the appropriate method, or analysis, we do low-quality work. 

Understand the context for the request

Start with asking questions to understand the context. 

  • Why is this request urgent? 
  • What happened immediately before this request occurred?
  • Who is asking for it? 
  • What are they going to do with the results?

Often with exploration, it becomes clear that there isn’t time to react to the findings of a study, or that the request hasn’t been properly thought through. Sometimes with further questions, the request just goes away. That’s great to learn before you waste your time reacting to it (and extremely frustrating to learn after!).

🚀 Do this now: Prepare a list of context questions that you will ask about incoming work requests, and make a template.

Make pragmatic method decisions

Even with context, the reality is that sometimes fast research needs to be done. One tactic we have, but should use sparingly, is method selection. As researchers we’re always making conscious decisions about what is ‘the best’ method for each research objective. However to avoid running studies that are too late to have impact ‘the best’ should also take into account the time we have available – not just the most technically correct.

Making pragmatic decisions about methods can reduce the time that research studies can take – for example running an unmoderated study using a panel when there’s not enough time for a moderated study, guerilla recruitment methods, or running a usability review instead of a full study with users. 

This can extend into how we report. Instead of writing a full report, why not talk your team through your analysed notes? This will get us to debrief earlier and works well with an engaged team who has understood the context of the study.

We have to be careful when making pragmatic compromises. Doing lower-quality work, and compromising your processes or expertise will set expectations that this is normal, and can be done again. When making compromises, they need to be negotiated with the people making the requests, to ensure that they understand the impact of the compromise. When sharing findings, the compromises need to be reinforced with caveats in your deliverables and debriefs by explaining the limitations of the study, why the decision was made to compromise, and what a better study would look like.

🚀 Do this now: Note down the list of research methods you have available to you, and make a decision about what substitutions are appropriate.

Mid-term fixes – Planning more efficient research

The short-term fixes are immediate sticking plasters but will not improve the situation long term, and the same situation will keep recurring. We need to do something about that! 

Anticipate upcoming requests

Because user research & UX is often an afterthought, we need to be proactive to get as much notice about changing priorities and incoming potential work as possible. This requires speaking to people – usually producers & product managers who have the best visibility of the priorities of the game, in order to understand what’s important to them. 

Schedule regular check-ins with your producers if they aren’t already happening, and ask questions such as:

  • What’s important to you right now?
  • What are you most uncertain about?
  • What did you do as a result of our last study?

By having an earlier view of changing priorities, it can help create an appropriate roadmap for research + UX activities, planned out with enough time to execute properly. Conflicts can be negotiated within that timeline, and it can be updated as priorities change – but you will be creating more visibility, and time for you to do your role properly.

🚀 Do this now: Put in a regular meeting with your producer or PM to ask about priorities.

Make research more efficient

Researchers can also make their research process more efficient. The idea behind ‘research-ops’ is to recognise the parts of research that are outside the ‘data collection’ of actually speaking to users, and explore how to make them more effective and efficient. 

This can include creating a participant recruitment panel, but also the creation of templates and checklists to make the research process more efficient (This is what The Playtest Kit does). Optimising your analysis process with better tools (like learning how to master mindmaps) will also reduce the time it takes to get high-quality findings back from your research study. 

Making your research process more efficient will allow you to do higher quality work in less time, and fits better with hectic development schedules. 

🚀 Do this now: Decide which parts of your process would most benefit from being templated, and make your first template.

Re-surface old research findings

Sometimes the findings of previous studies can be useful again, reducing the pressure on running ‘new’ studies. 

Particularly when learning about user behaviour, many of the findings are relevant beyond when the study is first run, and can inform the team’s decision-making regularly. Making previous research discoverable will reduce the demand for new studie, and can be done with a research repository. (I covered this briefly in my book Building User Research Teams, and you can read more about research repositories here ).

🚀 Do this now: Create a spreadsheet to track the objectives and reports from previous research studies, and populate it for studies from the last six months.

Long-term: Change the culture of the organisation 

Longer time, we need to move the game team away from this short-term panicked decision-making. This is obviously much broader than what a researcher alone can accomplish. However we have skills as design and research professionals that can help culture change be more successful, and ultimately increase the impact our studies will have.

This can start with small questions – asking questions about the intended experience, and encouraging our colleagues to define their hypotheses so that we can test against them. This will help people start thinking about the impact on users when making decisions, and can feed into activities about defining the game’s vision. Debating and defining the intended experience for the game will reduce volatility, and create a more stable development environment.

Understand and map the current experience

As a researcher, you have the skillset required to understand why last-minute crises happen. It’s time to journey map!

In a calm(er) moment, to the team about a recent last-minute study or change you were asked to make. Ask questions to understand why that request came in, and follow up the decision-making chain to work out what occurred at each stage. 

Repeat the exercise for multiple instances until patterns start to emerge. Use your skills in mapping and visualising complex domains to show where problems emerge in our development process on a journey map. 

This journey map artefact can be a powerful shared tool for conversations around “why does this occur”, and “what’s the impact of not having enough time to do UX or user research”. It provides the evidence and information needed to make sensible decisions about how to fix development. 

A hypothetical journey map showing how a feature decision was made
A hypothetical journey map

UX and service designers have the skills to run structured activities around fixing the issues once identified, and this can be an important tool for creating cultural change.

🚀 Do this now: Pick a recent ‘panicked’ request that came in, and ask the person who brought it to you about where it came from. Write this down to start building data for your map.

Advocate for change

We’re also responsible for educating people who don’t know of better ways of working. No-one wants to work in a panicked work environment, with rapidly shifting priorities. However many non-UXers won’t recognise the relevance that user research and UX design can have to improving the environment. 

By understanding our users we can:

  • Have a consistent vision of the player experience we’re working towards
  • Spot things that aren’t working early, and pivot when there is still time
  • Make better decisions throughout development, as our baseline understanding of users increases
  • Identify the core pillars of the game, and emphasise them in gameplay and marketing.

We just need to help teams understand the development advantages to running research, and frame it in terms that they care about (often risk, money, development time). Education through research observation, show and tells, collaborative activities, and relationship building can all help achieve this. 

🚀 Do this now: Create a shortlist of people you need to convince about the value of research + UX process.

Recognise a hostile environment

As a sole researcher or UX designer, you cant change the culture of a company. If it’s not a two-way relationship, and the organisation isn’t reciprocating, or saying it recognises the need for change, you need to look after yourself.

Change takes time, but pushing for change in an environment where it’s not welcomed is a fast track to burn out. 

Recognise when the environment isn’t going to get better, and leave. 

Low maturity UX research environment - just walk out!
Career advice from Da Share Zone.

🚀 Do this now: Update your LinkedIn, so you are ready when it’s required.

Ignoring Games User Research, Playtesting and UX is saving up problems for later

UX research speeds up and improves the quality of the game development. But it’s also reasonably new, and we still need to help people understand how it achieves that, and demonstrate the success of it. 

This can be slow, hard work – and very frustrating in hectic game development environments. But by combining pragmatic compromises with long term fixes, we can help advance the maturity of the teams we work with, and ultimately do better, more fulfilling work.

If you need to create an iterative playtesting + games user research process, the Playtest Kit will help. Read more about the Playtest Kit here.

Continue growing your playtest skills

Found this helpful? We’ll be exploring all of these topics in more depth in the future. Get a single email each month about playtesting and UX in games, with an article just like this one.

Categories
Newsletter Research Skills

How to get games user research experience (before you have a job)

Applying for games user research roles can sometimes feel hopeless. There are hundreds of applications for each role, and yours gets lost in the pile. You never even hear back.

(and worse – what about those entry level roles which ask for two years experience?!) 

It’s the classic problem that everyone runs into at the start of their career. You need experience to show you can do the job, and get an interview. But you need the job to get experience. Sometimes it can feel like you’re stuck! 

Today we’re going to tackle how to get games experience before you get your first job. This will help you stand out amongst the other candidates, and increase the chance of getting interviews. Plus you get started actually doing user research, instead of just reading about it – which teaches a huge amount.

This is an updated version of my talk for the GRUX conference in 2021. You can watch the video of the original talk here. The updates cover a whole bunch of follow-up questions that have come up since such as:

  • Is it ethical to work for free?
  • How do I know when I have enough expertise to get started with this?
  • How do I know when I’ve done enough?
  • What skills should I emphasise when describing my work?

Read on to learn how to get games user experience without already working in the industry.

How to get Games UX Research Experience before you have the job

How to get games user research experience before you have the job

Junior games user research roles are extremely competitive. Every year a new batch of people finish studies, complete bootcamps, or decide to move on from another industry. People love games, and want to work in the industry. This means there are a lot of very qualified candidates, and it can be hard to stand out.

One tactic for standing out is experience. Demonstrating you can do games user research, and having a body of work you can show people, will distinguish you from other candidates, and make getting an interview more likely. 

The process we’ll cover is how to use usability reviews, and partner with real game devs, to get experience. This is better than reviewing a finished game, because it can lead to a real change in the game, and allow you to practice + demonstrate communication with game teams. 

This hits many of the skills game developers are looking for

You get to demonstrate all of these skills companies are looking for
Understanding game development
Identifying research objectives
Designing a study
Analysing data
Communicating findings
Working with game developers to make changes

Let’s begin.

What is a games usability review

A usability review (or ‘expert review’) is when one or more usability practitioners play through the game, and spot potential issues. (As an aside – I never refer to these as expert reviews, although it’s a common industry term. I don’t think going around calling yourself an expert is a way to make friends with game developers). 

The type of issues we’re looking for are usability issues – parts where the player won’t understand what they are meant to do, or what is happening, or how to accomplish their goal. Looking for usability issues is safer than trying to spot ‘user experience’ issues – whether players will like the game or not. We’re looking to anticipate their behaviour, not their opinions! 

As a research method, usability reviews are not used in industry that often. Running real tests with players creates objective data (“the player did or didn’t do this”), so there should be no debate about whether the issue is true or not. In contrast, because usability issues are based on your expertise, it can become a lot more difficult to convince people that the issues you’ve identified are real. However they are sometimes used before running a bigger test with real users, to spot some obvious problems. They are also commonly used as part of the application process when applying for games jobs, so a good thing to practice! 

They are also free to organise, which is why we’re going to do them today. 

You might be wondering “am I ready to start doing this?”. I started this process within my first few months of learning that games user research exists. The reviews have been lost to time, and were probably rubbish – but that’s normal. We need to start with doing this poorly, getting feedback and iterating, to get good. So, start today! 

Find a game that is in the right state

We need to find a real game to work on. Luckily, there are places where game developers hang out and ask for feedback – playtesting forums.

Some examples include:

Reddit’s r/playtesters, r/playtesting and r/destroymygame forums

TIGSource’s Playtesting forums

Itch.io’s playtesting forums

(of all of those, my recommendation is the r/playtesters forum is the most active and high quality). 

Normally, I don’t recommend using these playtesting forums to game developers. Because only other game developers hang out there, you get massive sampling bias (here’s a better method I recommend to small game teams). But for our purposes today, that doesn’t matter, as we’re just using them to look for games in the right state.

As you’ll see, game developers post their game on these forums looking for feedback. Most won’t be familiar with what ‘user research’ is, or what a ‘usability review’ is – we’ll deal with that in a minute. But the fact they are posting tells us they have a game, and it’s in a state where they are prepared to make changes to it – exactly what we’re after.


There are a lot of posts on these forums, and a lot of the games are low quality. We need to filter those down to being the best ones to run you review on. Some criteria I use are:

Is there time to make changes?

Read the most and look at the state of the game. Sometimes people mis-use those forums to promote a game post-launch. That’s no good for us – if the game is already out, they are unlikely to make real changes, regardless of what you find in your review. 

Can the team handle usability feedback?

Look at  what information they’ve provided about the game. Do they feel like they will continue development, and be able to react to the issues you discover? Or are they only requesting very specific types of feedback?

This is something we’ll explore more when we start talking to the developer – working out if they are open to, and will be able to react based on usability feedback you discover. 

(also a lot of the things on these forums will never make it to launch. Again that’s not necessarily a bad thing, but you should try and suss out how serious they are so that you can prioritise projects that have a chance of coming to fruition.)

Can I run this review?

Look at the genre of the game. Ask yourself, is this a genre I have played before and know the ‘format’ of. This will help us later identify what are real usability issues players will have, and not be distracted by issues that will not occur for ‘true’ players.

For example, If you’ve never played a real-time-strategy game before, you will find it challenging to identify and describe usability issues that occur within the genre.

Can I share what I learned?

From this usability review, we want to get a good case study that can be shared with potential employers. We’ll need to ask permission about including this project as a case study on your website or portfolio. That will allow you to be more specific with examples when talking about how you handled the usability review in interviews, or on a blog post.

I’ve never had an indie developer tell me they are not happy with the findings being shared (because they benefit from getting the feedback, and appreciate any attention being paid to their game), but it’s polite to check!

Can the team afford professional help?

User research is a professional skill, and deserves to be compensated as such. Agencies and consultants will be charging five figures to do this kind of review professionally. We want to avoid devaluaing our expertise, by not working for free. 

This means that we shouldn’t be doing this work for any company that can afford to pay a professional. If they are paying for professional QA, marketing, localisation, or other third party services – don’t do this for free. Either find a different team to work with, or come up with a fair price for your time. 

My ideal game team for this project is one who has no budget, because this then becomes a mutually beneficial partnership. You gain experience of running this kind of work, and a portfolio piece. They benefit from your growing UX expertise. 

When preparing this talk, I went through the process myself. Here’s the post I found for a team I wanted to approach – I thought it was a genre I understood and I could see they were still making changes to their game.

A picture of a game designer asking for beta testers for their puzzle game

I was still unsure if they could handle usability feedback, or would be comfortable with me sharing the results (especially as I wanted to use them for a conference talk!). But this was enough of a hopeful lead to get in touch with them.

Make the approach

Having found a potential lead it’s time to approach them. Remember that most game developers working at this level will not have encountered UX, usability reviews, or structured playtesting – they think they are just going to get general opinion feedback. We need to explain the difference between what they asked, and our proposal (which is great practice for advocating for games user research). 

You have to explain what it is you do:

Hi <<name>>

I'm <<name>>, and I run usability reviews of games which identify usability and UX issues that will prevent players from experiencing it the way you expect.

I saw your post looking for playtesters for <<game>>, and wondered if you would be interested in a usability review for it. I will identify usability problems, and send you a report covering the top issues. 

I wouldn't charge for the review, but I'd also be really interested in sharing this as a case study <<on my website/blog>>. This might create some extra publicity for your game, so hopefully a positive for us both.

Let me know if this sounds interesting and looking forward to chatting further.

Working with game developers is great experience to demonstrate when applying for game jobs later, and so building a good relationship with the developer is really important. The more you can do on this (such as asking their priorities for the game, understanding what they are worried about, and working out how to be helpful), the more useful this will be to reference in interviews and job applications later.

You will also want to feel out whether they will be a good partner. Do they feel open and interested in the process, and likely to be a good person to work with. Are they giving the impression they take game development seriously, or are they opening Unity for the first time and have vastly over-scoped their game. If you’re not getting good vibes, politely move on to the next developer. 

Running your review

Once you’ve set expectations, understood their priorities and have a copy of the game – you are ready to run your review.

Running good usability reviews is a whole different topic (and one we’ll cover in future newsletters, so sign up to make sure you don’t miss it). This talk from Seb Long is a good place to start learning in the meantime

My own approach when doing this was:

  • Work with the developer to understand their current priorities and focus. List those out
  • Then play through the game once, as a fresh player. Record the video, and audio of my ‘new player experience’. Write notes throughout.
  • Then do a second pass, looking at the game as a UX practitioner. Look at every element of the game, what I anticipate the design intent is, and how or why players might misunderstand.
  • Then watch back the new player video, take more notes, and try and notice ‘what the designer meant to the experience to be’ compared to what I actually experienced.

That generated a lot of raw material for my report. 

For extra credit, you could find some users and observe them playing. This will practise participant moderation (which is another of those core skills above that interviewers want t osee). But it does add a time commitment, potential cost (if you pay participants) and is more challenging to organise than a usability review.

Share what you discovered

After you have your raw data, you’ll want to write a report.  Here’s my free guide on how to write a report.

You can see the report I made when preparing this talk here

Making the report is good practice, as it’s a common way of sharing results when working in industry – and demonstrates your ability to communicate findings (another one of those core researcher skills!).

I strongly recommend that you show your report to someone else for feedback. This is critical for improving – getting feedback from peers, working out what people don’t understand, and finding better ways of communicating it. I’ve been doing this for over a decade, and I still ask peers to read my reports before sharing further.

(As an aside – are you interested in joining a small group of people learning games user research to share their work in progress, and give each other feedback? If so pop your name down here, and if enough people sign up we’ll make it happen).

After writing your report, and getting feedback, share it with the development team. Remember that communicating with developers is a core thing that will help you stand out from all the other candidates in interviews, so try and make this as ‘active’ as possible. If you can set up a 30 minute call to present your findings, that’s fantastic!

You should also follow up after sending the report. Ask them if the report helped, and what changes they are considering. This is great evidence to show the impact of your work, when interviewing for a games user research job later.

Make it into a portfolio piece

Now you’ve done some work, and have permission to share it, it’s time to do so. 

To share it with others, you want to tell the story – rather than just show the final report. This should include you explaining…

  • What you did
  • Why you did it
  • What compromises you had to make beyond the ‘perfect’ study, and why you made those compromises.

This would all make a great blog post, if you have a blog. We’ve previously covered how to make a portfolio, but some of the things to emphasise from this work will be the contact you had with the developer and the impact that your study had on changes to the game. That kind of experience very few other people will have, and should really help. 

This whole process does take time, but doing it just once or twice will put you in the top 10% of applicants for jobs. That doesn’t guarantee you will get through to an interview, but it does increase the chances.

Plus as an added benefit, you might even get a credit in the game. Here’s my credit, for the project I did to prepare for this talk:

Games User Research - Steve Bromley

Get started

So, time to get started by looking at those playtesting communities, and finding some good leads. Do tell me about your experiences running usability reviews, either on Twitter (@steve_bromley), or by dropping me an email.

(and did you spot the aside above about making a group of learners to share feedback on eachother’s work? Sign up if intrigued, and we’ll make it happen!)

Continue your games user research journey

Get my free ebook of career tips from researchers at top game studios such as EA, Blizzard, PlayStation, Ubisoft, Activision and more. Based on exclusive interviews, they reveal ten essential tips to kick-start your GUR career today.

Plus receive a monthly email with free games user research lessons and curated Games UX and UR jobs.

Sign up now to get the free e-book (no spam, just a nice email from me each month!)

Ten essential tips for new and aspiring games user reseachers ebook
Categories
Newsletter Research Skills

How to keep games secret in playtests

Techniques for avoiding leaks, and keeping games secret when running playtests and UX research.

We don’t want leaks from our playtests

When speaking to teams, I often see that they are worried about leaks. That fear can put them off the idea of playtesting.

With small teams and solo developers there is the worry that playtesting their game will give away their ideas, and someone might steal them. From the larger teams, there is the worry that leaks will ruin their marketing plans

Many games spend as much on marketing as they do on production (because marketing is such an important part of making a successful game). This means that leaks can be seriously disruptive to established marketing plans, and there is a lot of wariness around playtests. Compared to other industries games are particularly at risk of leaks, because players are very hungry for information, and there are very active player communities to share leaked information. Even the smallest game teams need to be conscious of what assets they are putting online, and who might see them.

This means that game teams, and senior management can be very hesitant about running playtests. 

This fear is well founded. From the thousands of hours of playtests I’ve run, I’ve had leaks occur twice, and neither was good. Even though they didn’t severely disrupt the game’s marketing or announcement they were embarrassing and had the potential to impact my relationship with the teams I worked with. With a bad enough leak, it could potentially make teams not want to playtest again.

However the risk of leaks can’t stop us from running playtests. The value of playtesting is too high, and needs to occur as early as possible in game development in order for us to have the right impact and fix problems at the right time. So, we need tactics to minimise leaks. 

In this post, we’ll look at techniques available to researchers, or game teams of all sizes, to minimise the risk of leaks from playtests. 

How to keep your games secret

Playtest without leaks

Start with the basics: Get participants to sign an NDA

A non-disclosure agreement (NDA) is a legal agreement to keep something secret. It’s typical to sign it when working on games projects (which makes portfolios difficult!), and equally typical to ask playtesters to sign them at the start of playtests. 

Templates for NDAs exist across the internet. Here’s a UK focused NDA template from UKIE (thanks Seb Long for the link). Most non-hobbyist development teams already have a friendly lawyer, who can put together a bespoke NDA for you. I’ve also included a link to several alternative NDAs in the Playtest Kit.

The risk with relying on an NDA is that they are very difficult to enforce. If a participant breaks the NDA and puts some screenshots or secrets on the internet, it costs money to get a lawyer to act on it, and might draw more attention to the leak. Plus at that point, it’s often too late – the leaked information is on the internet, and will be spreading immediately. 

Because enforcing them can be counter-productive, the point of NDAs is often more about the threat of taking action rather than a real intention to enforce it.

Best practice is to ask your participant to sign the non-disclosure agreement at the beginning of the session, and then let them take a copy with them. Remind them again that they signed an NDA at the end of the playtest. Making the process seem formal and official will hopefully increase players’ likelihood to take it seriously. 

Choose in-person research methods for the most secret games

Different research methods also have different levels of safety for leaks. At PlayStation, the vast majority of our research was in our research lab, so that we can keep an eye on players and prevent them from taking screenshots or information away from the session.  

Taking participants’ phones away (and asking them to put them in a locker) will also help prevent photos from being taken.

This also allows you to take time to explain and emphasise the NDA – running through it with the players to make sure they understand and making explicit that it is a legal document. This will increase adherence.

Make remote research as safe as possible

Lab-based, moderated research is the ‘safest’ method to prevent leaks, but even when running remote research, there are some steps that can help. Choosing to moderate the study, where the participant is talking with you live, will create fewer opportunities to capture leakable information.

Bob Tilford, now of RockStar, gave a great talk about how building rapport can help minimise leaks, which can be viewed here:

For unmoderated research, there is an increased risk that players are left with the game, and can screenshot it. It also prevents you building rapport with the participants, increasing the risk.

For many games, it may be sensible to stream the game to players, rather than sending them a build directly. Although not feasible for games which require extreme responsiveness, for slower games this means you don’t have to actually let players have un-monitored access to the game, and can be in control of when and where they can access the game.

Parsec is one of these tools that allow remote players to stream access to builds on your own device, similar to how Google Stadia works. There is a video of how Parsec works on their website, as well as a case study of Ubisoft using it to distribute builds for a marketing event.

Watermark the build

Watermarking is putting an overlay on the screen, with some identifying information, like in this picture:

A game screenshot. In the corner is a watermark which has a date, the name of the computer, and an IP address on.

This can be invisible to the player (see this thread on how it is done inside World Of Warcraft) but for our purposes, we want players to know that leaked information can be tracked back to them. Some sensible information to include can be the current time, date, and any identifiable information that can link the screenshot to a specific device. 

Showing that screenshots can be linked back to an individual increases the perceived risk of leaking information, and can help discourage leaks. This will however require some development time to implement, so isn’t practical for all teams.

Remove important assets from the playtest

When working with established IP, it can help to obscure the IP in the test build. This could look like changing the title screen or renaming the game in the build so that it doesn’t mention or show pictures from the brand. This is perhaps most relevant to simple mobile games, as replacing entire character models is unlikely to be possible in more complex games. 

Removing assets can be achieved either through creating a custom build, or avoiding including the final assets in any build until post-announcement. Replacing assets also requires development time, but can protect a game from leaks. 

And most importantly – screen your participants. 

Both times I’ve had issues from playtests, they could have been resolved by better screening. In one of the sessions, they were a journalist from a minor publication, who believed they had a scoop. In the other, two participants knew each other, and were able to collectively be brave enough, and then corroborate, leaked information to the games press.

Remember to check when recruiting, and verify during a pre-interview:

  • That they are not a journalist
  • That they are not a streamer/content creator
  • That they have no personal connection to other participants

There are very few actively malicious participants (I’ve seen thousands of players, and only had issues a few times) – but asking these questions, and removing people who fail them or can’t give convincing answers, will help identify the most at-risk sessions.

Building trust through consistent results.

Worries about leaks push teams to delay running external playtests. Instead many teams just run playtests with their colleagues as participants. But our internal colleagues are just so different from real players, that a lot of the value of playtesting gets lost. 

These techniques combined can lead to a robust playtest setup, with minimal risk of leaks. But getting permission to run playtests, or the confidence to run your own is a much more human problem. 

Building trust from management can take time. Looking at teams who have been successful at integrating playtesting in their development process, it can often take multiple games to achieve.  Showing value with late evaluative playtests can build trust, and on the next game playtesting can start earlier and have more impact.

A great example of how a relationship can change over time is the work done at Bungie, moving from a few studies in the year before release for Halo 2, to regular playtests across the four years prior to launch for Destiny.

A graph showing how many tests were run on Halo 2, 3 and Destiny.
On Halo 2, testing is mostly from a year before launch. Halo 3 - two years. Destiny has frequent tests throughout the four years before launch.

(The source is John Hopson’s 2015 GDC talk)

Building trust is a marathon, not a sprint – but being able to avoid embarrassing playtest leaks will help us get there! 

Continue growing your games UX skills

Every month I send a playtesting and games user research lesson, just like this one, to help game developers run better playtests, and people start a career in games user research. Sign up to get the next one direct to your inbox:

Better playtesting today

If you’re interested in turbo-charging your playtesting, I’ve worked with game designers, producers, community managers, UX designers, QA managers and solo game devs to make The Playtest Kit.

It brings the expertise of over 25,000 playtest hours into one complete playtest toolkit – making playtesting accessible to game developers who have no time or money.

Learn more, and sign up for the free one week playtest masterclass at playtestkit.com

Categories
Newsletter Research Skills

Find usability issues in games with playtests

Usability issues can ruin games.

When players can’t understand what they are meant to do, where they are meant to go, or what is happening, they get confused, bored and ultimately drop out. This impacts their opinion of the game, retention, and creates an experience that just isn’t fun.

User researchers can help.

One of the core skills for researchers, and one of the focus areas for the How To Be A Games User Researcher book is how to find and deal with usability issues. This helps make games that players understand and enjoy. 

Today we’re going to look at how to use playtests and UX research studies to find usability issues in games.

Get future games user research lessons direct to your inbox

Find games usability issues 

Find Games Usability Issues

Today we’re looking at usability issues. What we mean by usability is “can players do what they are expected to do”. 

That’s different to opinions about a game. “I don’t like that the weapons break in Zelda” is an opinion.  “I don’t understand why my weapons keep breaking” is a usability issue. Today we’re tackling the latter.

1. Start with understanding the design intent

Usability issues are ‘when the player can’t do what we expect them to do’. To spot that, we need to know ‘what do we expect them to do’.

This involves talking to our colleagues. Designers have an idea about how they expect the game to work, and what they think players should understand and do. You need to learn what their expectations are (the ‘design intent’).

Exactly who to speak to, and what to ask depends on what you’re testing – which is why our studies start with scoping around research objectives.  You may need to talk to combat designers, UI artists, level designers, producers, or others, But for an idea about the range of things we may need to understand look at Liz England’s ‘The Door Problem’. You need to know what the designer would say in answer to every one of those questions. 

This can be a lot of chatting.

But it will make the other steps a lot more effective! 

2. Create the right context to create realistic issues

Next comes creating the task. What are we actually going to ask the player to do. 

Some tasks can be very broad “Complete this level” or “play for thirty minutes”. Sometimes they can be very specific “Craft an upgrade to your horse armour” or “Place this object on the shelf”. Exactly what level to set your task depends on your research objectives, but my preference is to lean towards broader tasks where possible, because it allows unexpected issues to emerge.

If you are live in your playtest, you can be looser on the task. Because you are in the room, and watching them play, you can jump in with an improvised task at the right time. 

Designing tasks in UX tests can be tricky. Revealing too much information will artificially lead players to the solution. Revealing not enough information (such as missing tutorials) will create artificial situations that won’t occur in the final game. This needs a lot of thought to balance.

My preference is to be as undirected as possible, while meeting the research objectives. “Play through this bit” is fine, as long as they are exposed to the right content, any tutorials or pre-requisite content, and is easier when you are on hand to subtly guide players towards the right content.

3. Be quiet, watch and look out for deviations

Every time you intervene with players, you are introducing artificial elements to the playtest. Unless you intend to answer questions live for every player post-launch, you should resist answering questions, or revealing information in the session. 

Instead your attention should be focused on listening and watching. Observe what players are doing in the game, and when they do something unexpected, write it down. Looking back at your research objectives, and a good understanding of the design intent will help you recognise unexpected behaviour. 

Write down what you see (we’ll cover good note taking in a future issue). 

When necessary, hand out new tasks to direct players attention to the right areas. (but still be very careful that the words you use or instructions you give don’t reveal information that the player wouldn’t otherwise have)

4. Ask questions to understand what you’re seeing

Observed behaviour is only half of the story. In order to understand why that behaviour occurs, we need to know what’s happening in player’s heads.

This requires asking them questions. 

As described above, there is a massive risk that your questions reveal information to the player that they wouldn’t normally have – artificially influencing your playtest. Bland, non-leading questions are required – such as “what is happening now”. We wrote more about how to ask good questions in a previous issue. 

5. Use the impact to rate severity

It’s common to spot a lot of problems, and as covered in the How To Be A Games User Researcher book, researchers go through a structured analysis process to uncover and describe them all.

In order to focus the team’s attention, we need a process for identifying which problem is most pressing. We do this by anticipating what issues will be most disruptive to players when they play.

This extract from How To Be A Games User Researcher explains one approach for doing this:

A method I like is a four-point scale for issues:

  • Critical
  • High
  • Medium
  • Low

Each issue starts as ‘Low’. Then ask three questions about the issue. Each time the answer is yes, raise the severity up a level.

These questions are:

  • Is this something that the player needs to do to progress?
  • Did the moderator need to step in to help resolve the issue?
  • Once the player had overcome the issue, did they know how to avoid it when they encountered it again?

When running your playtest, noting “what happened to the player as a result of this problem”, gives you the raw information you need to do this analysis + rating process. We will cover analysis in more depth in future issues.

A chance to practice

I’m sharing a session of a player who is playing Pokemon Unite for the first time. During their play session, they encounter some usability issues with the game.

Watch an unmoderated usability test of Pokemon Unite

As you watch, think about:

  • The design intent (what did the game team expect players to understand or do)
  • The issues the players encounter
  • What questions you’d ask, if the session was live
  • What impact the issues had on the player’s experience

(Feel free to share what you spot with me on Twitter!) 

Be ready for a career in games user research

Get my free ebook of career tips from researchers at top game studios such as EA, Blizzard, PlayStation, Ubisoft, Activision and more. Based on exclusive interviews, they reveal ten essential tips to kick-start your GUR career today.

Plus receive a monthly email with free games user research lessons and curated Games UX and UR jobs.

Sign up now to get the free e-book (no spam, just a nice email from me each month!)

Ten essential tips for new and aspiring games user reseachers ebook

Don’t miss the next issue

Every month get an email from me with a free games user research lesson, and the latest entry-level games UX and UR jobs.

Categories
Newsletter Research Skills

Expert playtest moderation – ask unbiased questions

Moderating playtests is hard. The wrong question, or an inappropriate tone can derail your playtest. Accidentally upsetting the player, biasing their response, or revealing inappropriate information can severely reduce the value of the data you are collecting.

But one to one interaction with a player is also a core researcher skill, and one of the most impactful methods we have for understanding player behaviour.

As a skill moderation takes practice (I remember how poorly my first moderation went 🤦 ). This month we share some guidance on how to improve your moderation.

Get future games user research lessons direct to your inbox

Learn to moderate playtests

Prepare appropriately

Excellent moderation starts with preparation. Before you step into a room with a player, you will want to be clear on what your research objectives are, and have written a discussion guide to prepare for the session.

This will ensure that your session has structure, and that you know your questions will be relevant. Even if you have to free-style questions, having a reminder of the research objectives in front of you will help remind you of the objectives we’re trying to answer.

Here is a free template for a discussion guide from my first book, which can be adapted for your playtest.

Consider the space

We want players to be at ease when they arrive for their playtest. The room is an important part of creating a friendly atmosphere where players can give feedback freely.

Think about the room you are playtesting in – we want to avoid the space biasing player’s opinions or intimidating them. Remove the many trophies that say you are an excellent game designer, and the marketing posters to create a more realistic play space.

Also think about the playtester’s experience before they reach the playtest room. How will they know what to do when they arrive at the building? Will someone be there to meet them? Will they know what to say? Spending time thinking about and designing the experience of the playtester arriving at your building will help avoid them arriving unsure or upset, and create a more positive atmosphere for feedback.

Start the session off well

The introduction helps set the tone for the rest of the session. As Meta Quest’s Rich Ridlen explains:

Click through to read more great tips from Rich

This can be rehearsed, and again preparation helps. Use the template to remind yourself what to cover at the beginning, including explaining:

  • Your relationship to the game (“I didn’t make this”)
  • This is not a test (“We’re not testing you, we are testing the game”)
  • What is expected from them (“Today we’re going to play the game, and ask you some questions about it afterwards”)

Don’t help.

As a moderator, your presence is unnatural – when the game is released, you won’t be there to help. This means you have to be extremely careful not to introduce new information into the playtest that the game won’t introduce.

Don’t help players when they are stuck*. We need to learn what players would do without outside help, to recreate the authentic player experience and see whether they can overcome the issue.

*The exception to this rule is you can help if it’s something that isn’t in the game yet, but will be. As long as you don’t forget that this is untested because you helped, you can help players overcome unfinished parts of the game.

**The other exception is you can help if everyone understands and agrees it’s an issue. If one player has the problem, let it occur. If the whole team then understands and agrees that this is a problem that should be fixed, you can consider helping other players overcome it, so that you can see new issues.

Don’t confirm, deny or answer questions.

Part of not helping is not letting players know if they are doing the right or wrong thing. If they ask questions about how the game is meant to work, or if they are doing it right, you shouldn’t answer.

A good phrase is What would you do if I wasn’t here. Or turning their question back on them What do you think?

Ask questions, but be careful with what you ask.

So, if we’re unnatural, why are we in the room at all? Part of our role as a moderator is to ask questions to understand why players are acting the way they are. Just watching will show us what players do, but we have to ask questions to understand why.

Questions can be dangerous, as they can introduce information the player didn’t have. Asking How did you know that was the right way to go reveals that this was the right way to go.

Because of this, my favourite questions are usually very bland, and just get players explaining what is going on currently (which you can then follow up on): What is happening currently? or How did you realise that? get players talking, without revealing any information.

Also remember to be careful when you ask questions. As Camille explains:

https://twitter.com/rehe_m/status/1500896495637250052

Read more tips from Camille

Probe deeper

We need players to feel comfortable giving feedback. This requires creating a friendly atmosphere, but also listening to the points they raise.

When players say something, or give feedback, as a moderator we need to consider “is this enough information”. A moderator should always be thinking about what would level of detail would be useful to your team. This requires asking further questions to reveal enough detail that a designer can take action.

This game is too hard isn’t enough detail for a team to action. We need to ask follow up questions to ask what happened that made them believe the game is too hard, so that the team can fix it.

As Laure says this doesn’t always have to be asking a question, leaving space can also help reveal more information:

Read more tips from Laure

Focus on moderation

Moderation is a skill and takes attention. As a moderator we have to carefully watch players to make sure we’re asking timely relevant questions. As Francesca explains, it can’t be multitasked.

https://twitter.com/core2design/status/1501306315867623430

Note-taking is an equally valuable skill, which takes preparation to do correctly. As Donat describes:

Read more tips from Donat

Moderation is a skill that takes practice. We’ll cover more tips in a later issue.

Be ready for a career in games UX research

Get my free ebook of career tips from researchers at top game studios such as EA, Blizzard, PlayStation, Ubisoft, Activision and more. Based on exclusive interviews, they reveal ten essential tips to kick-start your GUR career today.

Plus receive a monthly email with free games user research lessons and curated Games UX and UR jobs.

Sign up now to get the free e-book (no spam, just a nice email from me each month!)

Ten essential tips for new and aspiring games user reseachers ebook

Support How To Be A Games User Researcher

Thanks for reading. If you like this and want to support, consider:

Have a great month.

Steve

Don’t miss the next issue

Every month, get an email with a free games UX research lesson, and the latest entry-level games UX and UR jobs from me.

Categories
Newsletter Research Skills

How to write a games user research report

This month, we explore what goes into writing a games user research report, and how to communicate research results effectively.

When a tree falls in the woods, does it make a sound? I’m unsure. But I’ve seen plenty of research studies make no sound because of how they were communicated.

The hard work that goes into planning and running the study is undermined by poor communication. Excellent reporting requires a balance of building trust, excellent storytelling and clear communication – today we’ll look at some tips on how to do that (and look at a real report!).

This month we also have an exclusive interview with Player Research’s Marco Alesci, Helen Johnson’s top videos for new researchers and top career tips from researchers at top companies like PlayStation, EA, Activision, Ubisoft and more. Read on also for new entry level games user research jobs.

Get future games user research lessons direct to your inbox

Write a games user research report

Write a research report

What is a user research report

A games user research report summarises the findings from a playtest or research study – they are the conclusion of all of the work defining objectives, writing tasks, moderating sessions and analysing data. 

Writing a formal report is only one way of sharing research findings with teams. Its strength is that it is a stand-alone document. A report should hopefully make sense to someone who didn’t see the research study, or to yourself in six months time when you’ve forgotten what you learned in that test. 

The importance of storytelling

A dull report that just lists findings will be easy to forget and will have low impact. With time researchers learn the value of sharing fewer findings, but making the ones they share more impactful. 

Having a narrative thread through the report is one way of achieving that. Here’s some tips from EA’s Jess Tompkins.

A report is not the only way to share findings. Depending on the relationship with the team, an email or conversation may be more appropriate. 

When deciding how to share findings, remember a report is also not the point of research. We’re running studies to improve games. At all times think about ‘what is the clearest way to communicate my findings with my team’, and use that to inform how you present your research findings – whether you are making a report or not. 

What is in the report

VidyaResearcher covers the benefits of keeping the structure for your report consistent: 

Making a template makes it easier to consistently structure your report. Here’s what’s typically included within the report:

A one-page summary

A report will have multiple audiences. Some people will need all the detail explaining exactly what was discovered and what caused the issues. This information is helpful for the people who have to design and implement fixes. This can include designers, artists and programmers working on the features described.

Other team members don’t need to know the specifics of each problem, and just need an overview of the scale of the issues and how much attention they will need – most commonly important execs, or producers who need to manage the schedule.

Many games user research reports start with a high level summary of the test, including the most significant findings – what the most important issues were, what areas of the game need the most attention, and the overall state of the game. A good model for thinking about the exec summary is “if someone asked me in a lift how that test went, what would I say”. 

Context

To increase the impact of the work, it’s helpful to explain why the study is relevant and important to what team’s need to know today. Here’s a tip from Tom Lorusso:

Again, effective storytelling is important to allow your findings to have the impact they deserve.

Grouping issues

Putting similar issues together makes it easier for teams to follow your findings. It allows teams to section the report, and share relevant sections with the right people to fix them. All of this makes communication of the points clearer, and increases the potential your report will be impactful. Your research objectives might inspire suitable ways to group your findings – whether that is by game mechanic (such as ‘issues with crafting’), by ‘type’ of issue (e.g. ‘issues with the UI)

Think about the ordering of the groups, and put them into a logical order. For linear games, that might be chronological – the order the player encounters them. Or you may consider ordering them by severity – the mechanics with the biggest issues first. 

Within each section, again consider how you are ordering the findings. Most severe first? Or chronological? Think about the experience of the reader, and pick an order that makes sense when read.

Each issue explained fully

Proper data collection and analysis should allow you to give appropriate detail to your findings. Describing ‘what is the issue’ isn’t enough by itself. We also need to give enough detail about the causes of the issue so that teams understand where their attention needs to be to fix the issue. Describing the impact on the player will also help teams prioritise, by anticipating the impact of fixing the issue on players.

All of this adds up to a cause/impact structure being common when reporting usability issues – where the intended experience is contrasted to what players actually experienced, such as on this slide: 

Control tutorials may be missed
Cause: The controls appear on screen for a limited time
Cause: The game doesn’t verify that the player has learned the control
Cause: It’s not explained that the tutorials are available on the pause menu
Cause: Some later puzzles require advanced controls to to progress, such as rotating cubes
Impact: Players may miss the tutorial, or pass it without learning the control
Impact: Players may have difficulty progressing before discovering the pause menu

This structure works for usability issues – but again isn’t the only way of reporting findings. With richer findings (such as describing truths about players), you should consider other ways of describing what you learned so that the key points are understood by the reader. 

Some teams also share recommendations or prompts to inspire solutions on their slides also. We’ll cover more about this in future lessons.

Across the whole report, clarity in writing is important. Some tips for writing clearly:

  • Have only one idea or finding per slide.
  • Use uncomplicated words that people use in normal everyday speech. Avoid jargon.
  • Use short sentences.
  • If you have to use a font-size smaller than… 16? you probably have too many words on your slide. If you’re not presenting findings, but trying to make a report that tells a story, aim for nothing under a font size of 30.
  • Pictures and diagrams can make your points clearer – but only if they are understandable. Overwhelming diagrams will quickly lose the audience.

Next steps

We want our research to lead to action – and a next steps section is one way to encourage that. If you intend to run a workshop to help teams apply your findings, you can suggest that as the next step. Or if teams are going to be addressing the issues themselves, you could suggest further research objectives to tackle, or remind teams to re-test to check the issues have been fixed successfully.

See a real research report

In my talk last year on ‘how to get games user research experience without having a job’, I partnered with Cat Floor Studio who kindly allowed me to share the report from our review together. 

It doesn’t incorporate all of the best practices discussed above (where is the storytelling!) and more work could be done on making the most important points obvious. However to see an example of what a report might look like, take a look at this usability review of The GoD Unit. 

Other reporting tips

Laure Duchamp shared her tips on LinkedIn

  • Think about your audiences, all of them. If your communication speaks to a variety of stakeholders, chances are it will have more reach and will be used for various purposes.
  • I usually divide it into parts and at the end of each part sum up key insights and recommendations. That way you have more rhythm, and people don’t have to wait until the end to get your point.
  • Make it more alive with images, gifs and videos clips! Varying inputs helps keep attention
  • Add all your « further data » (any complex graph or raw data) at the end, in a part you might not use when presenting. Having extra slides can be good for picky stakeholders who like to dig into the evidence, or just to add more context to your insights.
  • Your report is a material that has to stand and live on its own. It has to be reusable and shareable without you being there to talk it through.

We’ll explore other ways in which research findings can be shared in future issues, so stay tuned! 

Be ready for a career in games user research

Get my free ebook of career tips from researchers at top game studios such as EA, Blizzard, PlayStation, Ubisoft, Activision and more. Based on exclusive interviews, they reveal ten essential tips to kick-start your GUR career today.

Plus receive a monthly email with free games user research lessons and curated Games UX and UR jobs.

Sign up now to get the free e-book (no spam, just a nice email from me each month!)

Ten essential tips for new and aspiring games user reseachers ebook

Don’t miss the next issue

Get a monthly email with a games UX research lesson, and the latest entry level games UX and UR jobs.

Categories
Newsletter Research Skills

How to write a playtest survey

Survey design is one of those topics that looks simple, but it’s incredibly easy to make big errors that invalidate your results. They are particularly risky – unlike some other research methods, it’s hard to spot when your data is junk, and your conclusions are unreliable. 

Despite this, they are a very common research method, often used at the end of qualitative playtest studies, and an important skill for games user researchers to master.

In this issue we’ll look at the steps to designing a robust and reliable survey or questionnaire for your games user research study.

Get future games user research lessons direct to your inbox

Design Playtest Surveys

Designing surveys for games user research studies.

Decide what you want to learn

Before you start writing questions, you need to decide what you want to learn. These are your research objectives, and are different from the actual questions you will ask players. A research objective could be “is the difficulty of our game correct”, but you probably don’t want to ask that question directly to players in those words. 

Research objectives are best gathered by speaking to design colleagues, and understanding where the biggest risks for the game are & what do we need to learn to make good decisions. Our first #gamesUXchallenge this month gives you the opportunity to practice generating them.

Having identified research objectives, consider whether a survey is the best method for answering them. Surveys are good for measuring things, for example opinions (“Does this level look good or bad”) or behaviour (“How many times did it take to complete this level”). They are less suitable for deep qualitative feedback which would require observation and probing to answer (“Why did players take 20 attempts to complete this challenge”)

Remember – the research method should always be informed by the research objectives. Starting with a method (“today I’m going to run a survey”), will lead to low impact findings

Decide your sample

After deciding what you want to learn, think about who needs to answer your questions – who are your real players.

A common mistake I see online is sampling by convenience – e.g. getting whoever you can to fill out the survey. This is a waste of time. If you’re not confident that they are really your players, you’re gathering a lot of opinions and thoughts from people who would never buy your game. Pointless.

Some places to consider looking include forums for competitor games, using twitter hashtags and using facebook ads to recruit players. However any recruitment methods will introduce a bias into your participants that you should anticipate, and consider. Offering incentives – money for taking part – can help make your respondents more ‘typical’.

Putting careful thought into “who are the right people to ask this question to”, “how can I find them” and “how can I convince them to fill out my survey” are important for making sure your data is representative of your real players.

Write your questions

Each research objective should have one or more questions that try to find an answer. Some rough examples are here: 

A table. It shows research objectives in one column (how difficult a level or challenge is) and a matching survey question on the other side (how easy or difficult was this level)

Scales are a very common way of capturing answers to surveys and can include:

An overall numerical score – asking players to rate something out of 10.

A two-directional scale – with two conflicting ideas on each side of the scale. Make sure that they are opposites, or this one gets confusing. For example (“Very bad” to “very good”)

An agreement scale – asking players if they agree with a statement. (“Strongly disagree” -> “Strongly agree”)

When writing the text for your questions, keep the following points in mind:

  •  Will my player understand the question? Do they know what I’m asking about. Do I use language they are familiar with.
  •  Will they be able to remember the thing you are asking about? Are we asking about something that is recent enough for them to have a real opinion?
  • Will they be able to decide on a correct answer? Is this something they actually have an opinion on? And are they comfortable telling you the answer, or is it socially embarrassing to say?
  • Will they be able to enter the answer that matches their true feeling? Make sure your scale will allow them to give their true opinion. Asking them to rate from “neutral” to “good” is no help if they think it’s “bad”

A really powerful combination is combining rating questions with a open-ended question asking ‘why’ This does not replace interviews though, because a survey doesn’t allow the deep interrogation that is often required to really understand what players are thinking or doing.

Some tips for questions:

  • Avoid leading players by providing opposite statements in the question text e.g. instead of asking how difficult was the game, ask how easy or difficult was the game.
  • Bold key words in the question, so it’s easy to pick up the meaning from skimming it: How were the graphics in this level?
  • Keep question text short and direct, so they are easy to understand. Rate this level is clearer than Please provide a rating based on what you thought about this level
  • Be clear – use the same words that players do (interviews can help reveal this), and avoid jargon. What did you think about this character’s DPS won’t make sense to many players.
  • As with interviews, a good structure is to go from broad questions to direct questions. Ask ‘overall, how was the game‘ before asking specifics such as ‘how easy or difficult was using the jetpack’
  • Avoid yes/no questions as they are too closed and force players to give strong binary answers. Rating scales, or asking players to explain, allow more nuance to emerge.
  • Keep the survey short.
  • Never ask about future behaviour. Questions like will you buy this game have always been to correlate poorly with real behaviour, giving you unreliable conclusions. It’s much safer to ask about things they have previously done “What was the last game like this you bought” to get answers that are accurate

I’ve also gathered some excellent tips for survey design from the community. Find them all at the end of this section.

Pilot your survey

Ask a colleague or friend to answer your questions before you send it to players, and ask them what they think each question means. Doing this while sat with them, allows you to ask them questions about each question as they fill it out. This quickly reveals where they haven’t understood your question or the words you use.

Make your playtest survey for real

Use a survey tool to create your survey for real. Google Forms is free, and has some basic survey logic so can be appropriate for simpler surveys. 

More advanced tools, like SurveyMonkey or Qualtrics can be a good next step once you’ve reached the limit of what Google Forms can do.

Distribute your survey

Send your survey out to participants (or ask them to fill it in after coming in for a playtest). The best method of doing this is probably clear after doing the work to ‘decide your sample’ earlier.

Analyse the findings

Analysis is a whole topic in itself, but before we get started, we need to clean up our data.

I personally would always recommend exporting my data into a spreadsheet, rather than using the survey tool’s built in analysis tools, to help explore it. We looked in a previous issue at some tips for how to handle quantitative data

The data we have collected will usually have some issues with it. Some people will not have filled out the survey properly, and would have clicked through as fast as they can, or given some impossible answers, and we want to remove them from the data set. Read through the responses, review the start and end times to see how long people took on the survey, and look out for tell-tale signs such as ‘rating every question the same’ to indicate which participant data needs to be removed.

Analysis and finding meaning in the data is another skill to practice – but we’ll save that for a future newsletter deep-diving into analysis. 

Learn more about writing playtest surveys

When writing this post, I asked the community for their tips. I received so many that I had to spin them off into a separate post – read the expert tips for writing a playtest survey.

Elizabeth Zelle (who kindly submitted some of our expert tips), gave an excellent talk about survey design for games at a previous GRUX conference. Watch Elizabeth’s talk here

I also am a huge fan of the work of Erika Hall, and it would be remiss of me not to link her post about the dangers of surveys. Essential reading before you write your own.

Get ready for a career in Games User Research

Get my free ebook of career tips from researchers at top game studios such as EA, Blizzard, PlayStation, Ubisoft, Activision and more. Based on exclusive interviews, they reveal ten essential tips to kick-start your GUR career today.

Plus receive a monthly email with free games user research lessons and curated Games UX and UR jobs.

Sign up now to get the free e-book (no spam, just a nice email from me each month!)

Ten essential tips for new and aspiring games user reseachers ebook

Happy New Year

2022 is going to be a busy year for me, and so I’m really looking forward to sharing our games user research journey together. Here are some of the things on my mind currently…

Next month is the one year anniversary of the How To Be A Games User Researcher book, and I have been preparing a special gift ready to celebrate – look out for it in the next issue.

It’s also nearly a year of these email lessons. If they, or the book, have helped you get your first games job in the last year, do let me know – I’d love to hear your story!

I’m also sponsoring a few of the games user research and playtesting events this year – I’ll share more as they come around, and their excellent line-ups get announced.

Also early this year I’ll be releasing the playtest kit – aimed at helping developers run better playtests. It’s still at a pre-launch discounted price, so if you’re interested it’s a great time to pick it up now.

Thanks everyone. Do keep in touch on twitter, and I’ll see everyone for more games user research knowledge next month! 

Don’t miss future issues

Categories
Newsletter Research Skills

Convince teams to playtest

This month we’re looking at overcoming objections to user research, and convincing teams to playtest. You might assume that this isn’t necessary – that teams understand the value of playtesting or UX research. Although that’s somewhat true – you rarely hear outright objections to user research, it say “not right now”.

This can be caused by fears of exposing the game prematurely to critique, and misunderstanding how research can support creativity. In this issue we’ll look at some fears that our colleagues might have, and what we can do about them.

Get future games user research lessons direct to your inbox

Overcoming objections to user research

Convince Teams To Playtest

Here are some of the most common objections I encounter, and some counter-points to consider.

“We don’t have time to playtest”

Sometimes teams have the belief that playtesting will take a lot of time away from development. A robust study does take time to prepare. Writing a test script, recruiting the right participants, analysing the data to draw reliable data takes time. But this doesn’t mean that development has to stop.

With good communication about the state of upcoming builds and milestones, it’s possible to prepare the study in advance. You can agree and aim for a shared test date. Rather than the team having to wait for you to prepare the test after the build is ready, the test is launched as soon as the build is ready. Doing this well requires strong communication with our teams (but communication with our teams is always important). 

The critical time to minimise is that between the test build being delivered and the researcher sharing some results. Researchers can adapt how they debrief findings, and optimise their analysis process, to minimise this wait. 

“Playtesting is incompatible with creativity”

Teams sometimes mistake what we do with market research or focus testing which aim to discover what is popular. 

This is not what user researchers do – we shouldn’t be trying to make creative decisions. Our studies are focused on testing how effectively games are meeting their creative goals. We learn whether players are having the experience the designers intended. Alongside this, observing player behaviour may inspire game designers to come up with new ideas – but the decisions about “what should this game be” are not ours to make.

When we’re working with teams, be very careful about explaining this, avoiding research objectives that fall into market research, and being unbiased about the creative direction. We should be helping teams define and reach their creative goals, rather than changing those goals ourselves.

“It’s not ready” 

(thanks Dan Choates for inspiration for this one!)

This assumes that the right approach is one big playtest to evaluate every aspect of the game. That’s risky, as it doesn’t normally come together until near the end, when there’s no time to make changes. 

To address this, teach teams about research objectives – defining what specifically they want to learn from a playtest. Base objectives on what parts they are currently working on. The parts that are not ready can be ignored or worked around, and we can get directed feedback on the elements that are ready. 

Conversations with team members will help expose what they are working on, the riskiest decisions they have had to make recently, and inspire potential research objectives. I believe doing lots of small targetted tests leads to a better outcome than one big playtest.

“It’s not valid if it’s not quantitative” or “we want a survey”

(thanks migor)

Teams may not recognise the value of qualitative research, or may have a preferred research method that they trust more. 

As researchers the first step we always have to take is to think about the research objectives. What do the team need to know currently? What is the best method of answering that question? Picking the right method will ensure we get reliable results from our study.

Sometimes a survey or quantitative method is correct. However we need to determine that from the research objectives, rather than blindly pick a method because the team request it.

One of the roles a user researcher has to do is educate their teams on the research methods that exist, and the strengths of each. Talk to your teams about the value of qualitative research. Explain how observation and interviews build a deep understanding of ‘why players do that’. Examples can help teams recognise how understanding a player’s behaviour and mental model is relevant to the game design decisions they have to make.  (mental models = their understanding of how something works) 

“Players don’t know what they want”

(thanks GiGi!)

This comes from a misunderstanding of what user research does. Teams may assume we just aggregate player feedback and report that back to game designers.

This isn’t what good user research is. As we discussed when covering creativity, we’re not taking creative directions from players. Our focus is on whether the experience players are having matches the one we want them to have. Demonstrate to teams that we understand their creative vision. Show how we combine data from observation, interviews and perform thorough analysis to see if the vision is being met. This will build confidence that we’re not just reporting players’ opinions.

“We already understand our players”

When I counted last year, I had observed players for over 25,000 hours (admittedly some of that was watching more than one player at a time!). In all of those sessions, I have never run a study where I didn’t learn something unexpected about player behaviour, or the issues they encountered.

Expert reviews, peer feedback and heuristics can help improve games a bit – but I have never seen a test fail to reveal something new. Look for examples of how the big studios with big budgets handle playtesting – by the end of development, Uncharted 3 was running roughly one playtest a week. There must be something in it! 

The most convincing tool user researchers have

The most convincing tool user researchers have for the value of playtesting is actually doing it. Exposing people to real players – allowing them to observe sessions, spot issues themselves, and involving them in the analysis is a fantastic tool for winning converts to the value of user research and playtests. Don’t work in isolation as a ‘black box’ service – be open, collaborative, and people will see the value themselves. 

More on how to do this in future issues

Be ready for your career in games

Get my free ebook of career tips from researchers at top game studios such as EA, Blizzard, PlayStation, Ubisoft, Activision and more. Based on exclusive interviews, they reveal ten essential tips to kick-start your GUR career today.

Plus receive a monthly email with free games user research lessons and curated Games UX and UR jobs.

Sign up now to get the free e-book (no spam, just a nice email from me each month!)

Ten essential tips for new and aspiring games user reseachers ebook

The end of 2021

I think there won’t be a full issue next month, just a brief update (although maybe a secret bonus feature if everything comes together!). 

Thanks for all your support over this year, subscribing to the newsletter and everyone who picked up the How To Be A Games User Researcher book. I have been really happy with the reception it had this year and everyone’s kind words.  I hope everyone has a great end of 2021, and looking forward to continuing to work together to become better games user researchers in 2022.

Steve (follow me on twitter for more playtest + ux research chat)

Don’t miss future issues

Thanks for reading this issue. Get a monthly email with a games user research lesson, and entry level Games UX and UR jobs – sign up here.