Categories
Playtest Masterclass Research Skills

Top remote playtest platforms for unmoderated testing

Playtesting doesn’t just have to be in person, and online playtest platforms can take a lot of the hassle out of recruiting and organising people to take part in your playtest.

Finding genuine players, rather than just asking other game devs, friends or people at a conference vastly improves the quality of the playtest data you get back – and many of these tools make it convenient to access a wide range of player types.

This article introduces some of the most popular playtesting platforms, gives an overview of their features and shares my experience with each.

Before we go on: Get a single email each month about playtesting and UX in games, with a playtest guide just like this one.

What is a remote playtest platform

Online playtesting platforms offer a way to get playtest feedback on your game remotely. Most involve uploading a build of your game to their platform, writing some tasks or questions, and then they handle distributing those to players. 

You get back raw playtest data, such as videos of people playing your game, their survey responses, or analytics from your game build. 

This can save a bunch of time for finding playtesters, but comes at a cost – so can be out of reach for teams on a tight budget (here’s a free alternative method of recruiting players).

Not all platforms are equal though. They have different functionality, users, and specialities which means you should look beyond just ‘price’ as a distinguishing factor.

Top remote playtest platforms

PlaytestCloud

Playtest Cloud

Platforms: Mobile (iOS and Android)

Price: from $69 per participant, including video footage

Can they find participants?: Yes, included in the price

Can they help with the analysis as well?: Yes

I’m a big fan of PlaytestCloud, and they are trusted by many studios of all sizes for their mobile testing expertise.

They support a large variety of study designs, including multiplayer testing and longitudinal studies – and their price includes videos of each participant playing and survey responses. I’ve also seen a sneak peek of some of their upcoming features, and am excited about the potential for using their platform for more of my playtesting needs.

Their platform ensures that testing + game code is kept confidential, and they also offer support from in-house and external expert researchers to help design or analyse studies, for teams who are not confident designing robust playtests.

They also support testing with children + teenagers (while meeting ethical and legal compliance requirements) which is extremely valuable for teams who work on games for kids.

For teams who run regular playtesting, they offer a subscription package which brings the cost per participant down further, and also bespoke cheaper packages for indie teams. 

Find out more on the PlaytestCloud website. 

Antidote.gg

Antidote.gg

Platforms: PC, browser and mobile

Price: from $29.50 per participant, without video footage

Can they find participants?: Yes, included in the price

Can they help with the analysis as well?: Yes

Barcelona based Antidote are a growing provider of playtesting services, and support a wide variety of study designs including multiplayer, or longitudinal studies. They would currently be my top recommendation for testing a PC game, based on some lovely chats with their team. 

For an additional cost, they can provide gameplay footage, or support more complex participant recruits.

They also have an expert in-house team of researchers who can help design, run or analyse your study. 

Find out more on the Antidote.gg website

Playtesting.Games

Playtest.Games

Platforms: Mobile (Android only)

Price: from $39 per 15 minutes, including video footage

Can they find participants?: Yes, included in the price

Can they help with the analysis as well?: Yes

I recently met the team behind Playtesting.Games, who have been working on their playtest platform since 2019. Building on their history of building communities for gamers, their platform offers the ability to test in a single session, or multi-sessions over time, making it suitable for diary studies.

Uniquely, their platform also offers verification of players’ previous game history, by scanning the playtime of other apps – giving confidence to game developers who want to make sure they are getting legitimate players.

In the near future, they are adding the ability to record the player’s face via their phone’s camera. Playtesting.Games are also developing an SDK which allows their playtests to be triggered within a game engine, again offering options to target players once they reach specific milestones in the game with a survey or recorded playtest (and have an early version live now).

Find out more on Playtesting.Games website

Gotestify

GoTestify

Platforms: PC, iOS, Android

Price: Enquire for details

Can they find participants?: Yes

Can they help with the analysis as well?: No

Built by a team of former game developers, GoTestify supports a variety of study designs, including usability studies, multi-session testing and multiplayer tests.

Their platform promises a degree of automation to identify the most important moments automatically (although I haven’t seen this in action, so can’t verify!). This could potentially be very valuable for time-stretched teams, but I’ll hold off judgement until I’ve seen it’s impact on a real project.

They are relaunching their platform shortly, so we’ll learn a lot more soon.

Learn more about GoTestify

GameTester.gg

GameTester.gg

Platforms: PC, browser and mobile

Price: from approx $5 per participant, without gameplay footage

Can they find participants?: Yes, included in the price

Can they help with the analysis as well?: No

GameTester is focused around gathering quantitative data – both through surveys, and in game analytics. 

Surveys are only appropriate for some study designs, and the lack of video footage will make it unsuitable for many research objectives, but the price is very reasonable!

It’s not a platform I’ve used personally (although I will give them a go soon), but one to keep in mind if you are running a quant-focused study.

Learn more about gamestester.gg on their website.

Playcocola

PlayCoCola

Platforms: PC and browser

Price: from free, with gameplay footage included

Can they find participants?: Yes, for an additional cost

Can they help with the analysis as well?: No

Targetted directly at indie developers, Playcocola promises the full range of data collection, at an indie-friendly price. Their software allows playtesters to record their gameplay, spoken thoughts and written comments, and bundles that all up for developers to watch and analyse. For playtesters, it doesn’t require any special software installation or registration – making it easier to include players who are less confident with technology.

The default packages requires developers to source their own playtesters, but they also offer access to their playtest panel is possible at an additional cost.

Playcocola is reasonably new, but one to consider if already have a panel of research participants ready to go, (perhaps gathered through this guide!) or want to get started with playtesting with a small budget.

Learn more about playcocola on their site. 

Steam Playtest

SteamPlaytest

Platforms: PC

Price: Free, no gameplay footage or survey data

Can they find participants?: No

Can they help with the analysis as well?: No

If releasing a game on Steam, they also have an inbuilt way of distributing playtest builds, called ‘steam playtest’. This is free, but has several limitations that make it not suitable for professional quality user research.

First, it’s difficult to screen or control who takes part. Players sign up to a waiting list, and are let in at random based on the amount of empty slots you provide. This means that we have no insight into who is taking part in the playtests, no ability to screen players out, or to treat different segments of players differently.

Second, it’s extremely limited how you can get data back from your players. No tasks are given, no video footage is captured, and no surveys are distributed. To get data back from players will require either integrating telemetry, or to coordinate outside of Steam’s ecosystem – which is also challenging as developers have limited control over who can access their playtest. 

I’d recommend extreme caution using Steam Playtest – it encourages lax standards for your playtest around finding the right players, and capturing only convenient, rather than appropriate data. Those bad habits require expertise on how to avoid them biasing your conclusions.

Learn more about Steam Playtest

Don’t let the platform tail wag the dog…

Playtesting platforms make running some types of playtests convenient. But convenience isn’t the only thing we should consider when deciding the appropriate way to run a playtest.

The most common trap I see developers fall into is ‘deciding the research method before confirming what they want to know’. And playtest platforms encourage certain methods (usually unmoderated remote playtests supported by surveys). 

Always start by confirming ‘what do we need to learn from this playtest’ before even starting to think about ‘which method would be best to answer this’.

Sometimes that will be a method that a playtest platform supports. Other times, it might just be popping to the local coffee shop with a laptop and showing someone your game.


Need help with picking the right method, and running the right test for your game? The Playtest Kit is designed to help make running accurate playtests simple for time-stretched developers.

Continue growing your playtest skills

Found this helpful? We’ll be exploring all of these topics in more depth in the future. Get a single email each month about playtesting and UX in games, with an article just like this one.

Categories
Career Advice

17 Games UX Research Job Interview Questions To Prepare For

Preparing for interviews is an emotional rollercoaster. 

First there is the elation of finally hearing back from a games studio, and getting the interview.

Then the dread sets in – what are they going to ask, what do I need to do to prepare, and how can I make sure I don’t miss this chance to work at the company of my dreams?

In this post we’ll look at what questions you might be asked, and the themes interviewers will explore. Being ready for these will help youy to ace the interview.

(Having difficulty getting interviews? Read my previous post on networking to increase your chances of getting an interview).

If you’ve got the interview and you’re preparing for it – read on! 

Games UX Research Job INterview Questions

Interviewers want you to do your best

Interviews aren’t meant to be a trick. The interviewer wants you to be able to share all of your experience fully.Good interviewers will ask follow-up questions rather than letting your experience get missed or misunderstood accidentally. 

If you can get into the right mindset, treating the interview as a friendly conversation between peers, rather than an exam where you are being tested, will hopefully reduce nerves! 

What will the interview be like?

Sometimes there is a first screening round with a HR person, who will check that your background is vaguely appropriate. Any sort of industry or academic UX or research experience should be enough to proceed. 

The real test is the interview with the team – usually one or more senior researchers. 

At a junior level, their questions might be more hypothetical (“how would you do this”). At a senior level, they might be asking about specific times you have encountered these challenges and what you did. If you can give concrete examples, using something like the STAR method (Situation, Task, Action, Result), that will help give comprehensive answers and build confidence in your ability to do to the role.

Every games company is unique, and their approach and exact questions will differ. Here are some themes and example questions to prepare for…

Scenario based interview questions

A very common format for interview questions is to ask “what would you do”. These will be looking into how you approach problems. The interview is usually expecting you to ask more questions about the context, rather than diving straight into answering. Here are some examples: 

  1. A team have made a prototype for a new multiplayer party game, and would like to know if players enjoy it, to decide whether it’s worth taking further. What would you do?
  1. How would you handle a designer telling you they didn’t believe the research work you are presenting?
  1. A team has asked you to increase the sample size for your usability test. What do you do?
  1. Your team would like to know how to improve retention on their free-to-play mobile game. How would you approach this?
  1. What would you measure on a survey to see if players would buy a game on release?
  1. A PM asks you to soften the interpretation of data for an important review and says it’s on the plan to fix, how do you handle it? (Thanks @VidyaResearcher)

Knowledge based interview questions

Successful candidates will need to build confidence that they understand how to run research, how this intersects with game design, and that they are able to talk about games in an appropriate way with other game developers. Some questions ask directly about your game or research experience to explore those topics.

  1. How does games user research differ from other types of user research? 
  1. What is statistical significance, and when is it important?
  1. What’s a game you’ve played recently that has good usability?
  1. What can you do to minimise leaks from a playtest?
  1. When in game development should you be running user research, and what studies would you recommend? 

Opinions based interview questions

Some questions don’t have ‘correct’ answers, and are used to understand how you think. A good approach is to show you understand multiple perspectives on the topic, commit to an answer, and justify why you have committed to that answer. 

  1. What do you think is the hardest bit of being a UX researcher?
  1. What are your thoughts on personas?
  1. How do you make sure research findings are understood by the whole development team?
  1. How many playtesters do you need to test a game?
  1. What are your strengths and weaknesses as a researcher?
  1. What is a game you like, why do you like it? What is a game you don’t like, why do others like it? (Thanks @VidyaResearcher)

What comes after the interview

Email the interviewer to say thank you – it takes no time, and won’t hurt your chances. 

If you’re lucky enough to get through to another round, this can either be similar interviews with other domain experts (like an interview with non-researchers), or can be a take-home task, such as reviewing a game or designing a study. I cover more about what the process can look like in How To Be A Games User Researcher

Every month my free newsletter shares games user research job opportunities and advice for getting your first job in games, as well as exploring how to answer interview questions like these. Sign up and get the free e-book of career tips below.

Get my free ebook of career tips from researchers at top game studios such as EA, Blizzard, PlayStation, Ubisoft, Activision and more. Based on exclusive interviews, they reveal ten essential tips to kick-start your GUR career today.

Plus receive a monthly email with free games user research lessons and curated Games UX and UR jobs.

Sign up now to get the free e-book (no spam, just a nice email from me each month!)

Ten essential tips for new and aspiring games user reseachers ebook
Categories
Playtest Masterclass Research Skills

How many players do I need for a playtest

When running a playtest or user research study, pretty early on you need to decide ‘how many playtesters’ do I need. Plucking a number out of the air feels wrong, and magic numbers float around game design communities of ‘the right number of playtesters’.

For teams who are lucky enough to have an active community, it’s tempting to send out a big survey to tens of thousands of players (and then regret it when receiving thousands of answers to sift through).

For other teams who have to actively recruit each individual playtester, it’s hard to know when you’ve done enough, and it’s time to stop. 

Personally, for each playest, I recommend…

  • Six players when you’re trying to discover problems with your game
  • Twelve players when trying to understand and define players
  • One hundred players for quantitative measuring questions

That’s per playtest. A single playtest round will be tightly scoped, and games require multiple playtests throughout development, depending on their complexity (Destiny had over 150 playtests throughout development!)

How many playtesters to use
Find problems with my game - six
Define my players - twelve
Measure player opinions - one hundred

In this article we’ll explain the reason behind these numbers, and help you make a confident choice about how many playtesters you need.

Before we go on: Get a single email each month about playtesting and UX in games, with a playtest guide just like this one.

Where does the ‘five’ number come from?

There’s a classic article from Jakob Nielsen where his team looked at previous usability tests, and counted how many problems they discovered per participant. 

They ended up with a graph like this:

A graph showing how many usability issues are discovered per user.
At 5 users, around 80% of the issues have been found.

The reason for stopping at five participants is that most of the usability issues have been found, and more participants don’t add a huge amount of new problems discovered. 

At that point, the sensible thing to do is stop, fix the problems we’ve found, and test again to discover a whole bunch of new problems. 

This guidance is widely cited and used by teams running usability tests. Often incorrectly. 

Applying this rule without considering whether it’s appropriate for what you want to learn is risky. Especially with games, which are often a lot more complex than the websites that Nielsen’s study was investigating.

So what is the right number for games?

Making pragmatic decisions

When working out the number of playtesters, there is a difference between the ‘perfect’ number that would be appropriate for academic studies and a pragmatic number.

We don’t have infinite time or money, and game development is fast-paced. If we spend too long getting answers, the opportunity has been missed, and the playtest will be pointless. So we need to make informed compromises.

The numbers below are a starting point for making a pragmatic comprise, but other researchers may choose different numbers. Consider how important the playtest is, the level of trust you have with your team, and how much time you have, to make a final decision on how many playtesters to use .

Also keep in mind what the needs of your game are – if it’s a 4v4 multiplayer game, you’ll need 8 people! (Thanks Ben Taels for the prompt!)

The right number of playtesters depends on what you want to learn

The number of playtesters depends on the type of thing you’re hoping to learn from your playtest (the “research objectives”)

First decide…

Is my playtest asking measuring questions 

  • Am I trying to measure something, like “How do players rate <<this>>?”
  • Am i finding which problems exist in my game?
  • Am I trying to define my players (for example, to make personas)?

The right number of playtesters differ for each of these.

How many playtesters for a survey?

First, a short stats lesson on the ‘margin of error’.

We can’t survey everyone. For example if I wanted to survey gamers aged 18-35, I couldn’t email every 18-35 year-old gamer in the world. And if I could, many of them wouldn’t fill it out. So we only get responses from a smaller group. 

This is normal, it’s how surveys work, and is called sampling.

Because we’re not asking everyone in the world, we have to accept there’s a chance that our sample’s answers don’t match exactly what we’d get if we did ask everyone in the world. 

The more people we ask, the less chance we have of it being inaccurate. The ‘margin of error’ is how far off the ‘true’ answer we’re prepared for our answers to be.

So, how many players do we need to get to fill out our survey to get a (reasonably acceptable) 5% margin of error

A chart showing the number of players you need to fill out your survey.
For a population of 1,000 its 285
For a population of 10,000 its 385
For larger populations up to a million it's 400

Getting more than 400 people to fill out your survey just isn’t usually worth it – you’re getting a similar quality of conclusion, and end up with a lot more responses to process and work through.

However 400 survey responses isn’t practical for many teams – if you don’t have easy access to your players, it’ll be very hard to round up and encourage 400 people to fill our your survey.

In my opinion, 100 players for your survey is a good, practical number that gets ‘good enough answers’ for most game design questions.

It’s approximately a 10% margin of error which is acceptable for many applied research settings where lives aren’t at risk, and is an achievable number of people to find (here’s one way to get one hundred playtesters).

How many playtesters are needed to find problems with your game?

Finding problems, such as usability issues (things players don’t understand or can’t do) is a common playtest objective. 

This is where Nielsen’s graph is most relevant – relatively early we’ve seen most of the most prominent problems that exists. 

I personally recommend going for six playtesters, and then stopping and reflecting on what you’ve found so far. 

Why six? Because sometimes playtesters don’t turn up, and even with a no-show, you’ll still have five players.

It’s very likely at this point you’ve found some high priority issues are worth addressing. Fixing those issues, and then planning another playtest to see if they are fixed is more valuable than just putting more players through this playtest. 

(Remember, this is six playtesters per type of user. – if you’re testing kids AND adults, you’ll want six of each.)

How many playtesters are needed to define your players

With this kind of qualitative research,  we’re looking for the point when we’re just hearing repeats of stuff we’ve already been told (“saturation”).

Reaching saturation depends on how complicated the thing we’re trying to understand is. Understanding people to create personas is more complex than finding usability issues, so the number of people you need to speak to will be higher.

It’s not possible to anticipate in advance how many people you need to speak until you tip over into mainly hearing repeats – my recommendation is to start with 12 (per type of user), and then look back at what you’ve learned before deciding if more interviews are necessary.

One player is infinitely more than none. 

If you’re not able to reach the numbers listed in this post, because of time or budget pressures, that’s ok. Any playtesting better than doing nothing. Seeing even a single player will reveal behaviour and issues that you didn’t anticipate, and broaden your understanding of the game – helping you make smarter design decisions.

If it’s only possible to get two or three playtesters through the door – do still run your playtest – Iteration is the secret weapon of game development, many small playtests will add up to a huge impact on the game.

Where can I find playtesters?

It’s essential that your playtesters represent your real players – what’s the point of getting feedback or behavior from people who are nothing like your target audience?

I’ve written a post about how to find first 100 playtesters. Also if you sign up for the free playtest masterclass on playtestkit.com it guides you through the process, including templates for defining your players and writing the pitch. 

Continue growing your playtest skills

Found this helpful? We’ll be exploring all of these topics in more depth in the future. Get a single email each month about playtesting and UX in games, with an article just like this one.

Better playtesting today

If you’re interested in turbo-charging your playtesting, I’ve worked with game designers, producers, community managers, UX designers, QA managers and solo game devs to make The Playtest Kit.

It brings the expertise of over 25,000 playtest hours into one complete playtest toolkit – making playtesting accessible to game developers who have no time or money.

Learn more, and sign up for the free one week playtest masterclass at playtestkit.com

Categories
Newsletter Playtest Masterclass

How to run pop-up playtests

Playtesting is important, but it’s often hard to find the time. It takes a lot of effort to find playtesters, prepare the build, gather participants and analyse their feedback. This can be difficult to prioritise alongside everything else busy game developers have to do – marketing, social media, and… actually making the game. 

What if playtests could be quick, easy, and immediate? And you could get an immediate steer on whether your game is on the right track, and what the biggest problems might be.

Pop-up playtests offer a quick method for getting low-cost information about the player experience. They are most effective at highlighting the biggest issues with your game.

Spotting and fixing these obvious issues early gives you more opportunity to find more detailed and nuanced results with more formal playtests. 

However they also throw out a lot of the protection that more formal playtests have built it, and increase the risk of drawing misleading or unhelpful conclusions from your playtest. 

In this article you’ll learn how to make the most of pop-up playtests, how to avoid some of the traps that mislead game developers, and get the best possible data from your study. 

Pop-Up Games UX Research

Why do we need to run pop-up playtests?

Playtesting is hard to organise, and so it doesn’t happen enough. This means our games suffer – less playtesting means less iteration, discovering problems too late, and underwhelming releases. 

I’ve spent over 25,000 hours with players during playtests, and I’ve seen that a pop-up playtest is better than no playtest at all. Today, we’ll learn how to make the most of them.

What is a pop-up playtest?

A typical formal playtest involves finding the right players, designing tasks for them to do, gathering data and analysing it to inform game design decisions. Games user researchers (that’s me!) do this as a full-time job, and most game developers understand the value of running these formal playtests throughout the development of their game. 

Formal playtests take time to prepare and have a process behind them to ensure the results are reliable. This is what game user researchers do to a professional standard every day.

A pop-up playtest throws a lot of the process out – finding some people, putting them in front of your game, and seeing what happens. Minimal prep, minimal time, just instant feedback. And sometimes that’s all we have time for.

Just you, watching someone play your game.

And maybe asking them a question or two at the end.

Other names for this process are ‘guerilla’ research, or quick and dirty playtests. 

What tech do I need?

The tech for pop-up research can be very low-key – you just want to record what happens in the game, and what people are saying as they play. 

For a mobile game, you can do this with built in screen recording software, such as is built into iOS. Hit the button to start it recording, and then hand the phone to the player – easy. 

If you’re running playtests remotely over video conferencing software, Zoom (and others) have meeting recording built in – so you just need to get the player to share their screen, and start the meeting recording. 

If you’re testing on a PC or laptop, Obs (open broadcaster software) is a free tool that lets you record your screen and voice over – or windows game bar does it too. Start recording, hand your laptop to the player, and you’re away! 

🚀 Do this now: Decide if you’re running these playtests in person, or remotely, and get your tech setup together

What are the risks with pop-up playtests?

There’s a reason why playtesting (and “games user research”) is a discipline – it’s easy to get accidentally misled by feedback from players and draw wrong conclusions about your game. 

Some traps include:

  • Finding unrepresentative players, and balancing the game too easy or too hard for your real players. 
  • Focusing too much on what players say about the game, and not enough on their behaviour
  • Helping players too much, and biasing their experience artificially
  • Believing players who tell you “I’d buy this game”

Here are four principles to keep in mind when running playtests that are easy to incorporate and will improve the quality of your pop-up playtest. They will help you avoid those common traps! 

Four ways to make your pop-up playtests more effective 

Decide your playtest objectives

You might learn something if you just put your game in front of players and ask them “what do you think”. But you’ll get much more useful results if you think about what you want to learn from the playtest first.

Consider the most important things you need to learn to make your game a success, with prompt questions like “What parts of the game am I most uncertain about” or “What do I absolutely need players to understand about this game”.

Risk -> Objective -> Task

From that you can write a short list of objectives – things you want to learn from your playtest.

Some examples of objectives:

  • “Do players understand their goal from this level”
  • “Has the tutorial taught players how to use their special move”
  • “Will players recognise the correct strategy for this puzzle”

Use this list of objectives to decide which tasks to set players, which questions to ask them, and what to watch out for during the playtest. 

Some objectives are almost impossible to answer reliably with pop-up research. Complex questions around balancing the game, or measuring opinions require reliable quantitative methods, like surveys – and so will be a lot less suitable than “do players get stuck in this level”.

🚀 Do this now: Think about what are the riskiest or most important parts of your game. Write them out in a list, ready to inspire your playtest objectives.

Find the right players

It matters who takes part in your playtests. If they are game developers, their feedback is going to be very solution focused. If they are your friends they are going to be too kind. And if they don’t normally play this type of game their experience isn’t going to be typical of your real players.

Finding the right players takes time (here’s a guide on how to get started with finding playtesters). 

For your popup playtest make sure you’re consciously thinking about “how can I find people who are similar to my real players”.

Use that decide where to go and look for players – perhaps existing gaming communities or a local gaming cafe are a good start. Many communities of game players can be found on reddit (but be careful of game development communities – game developers are not your audience!)

Then consider “how will the type of people who playtest for me influence their feedback”.

Use this to filter the feedback you hear. Think about whether their background and the type of player they are would influence the type of feedback you’re getting, and use that to decide how much to prioritise what you’re hearing. This can often highlight interesting threads that should be explored with more rigour with more formal playtesting.

🚀 Do this now: Make a shortlist of places you could look for players, and for each consider “what influence will players from here have on their feedback”. Then pick the two or three most appropriate places to go and run your playtest.

Watch people play

It’s tempting to rely on surveys or feedback on a Discord channel to get data from playtests, because it feels convenient. But surveys rely on players self-reporting what happened to them, which misses a huge amount of valuable information. 

Players can’t tell you features they didn’t discover or didn’t understand. They can’t tell you what they missed. They can’t give you enough detail about why problems happened that you can fix the problem. 

To get the right level of detail, you need to watch people play.

Watching just two or three people play your game is often much more useful than a survey of hundreds of players. It gives a huge amount more rich information and put you in a much better position to actually make informed changes to your game, rather than trying to guess what players meant on a survey. 

🚀 Do this now: Go out into the places you shortlisted, and watch three people play your game! 

Analyse what you see and hear

After you watch people play your game, and speak to them about it, you’ll have a bunch of raw data to draw upon. It’s crucial to actually analyse this feedback, rather than act on their suggestions immediately. 

A good rule of thumb is to trust what you observed. If you saw a player do something, or saw that a player didn’t understand something, you know that is an objective fact, and it’s safe to take action to fix it. 

You have to be more careful with ‘feedback’ from players that they reported themselves. Their opinions about the game being too hard, or not enjoying certain bits are a clue that you should investigate that area more – but not necessarily trust their interpretation of what should be different. 

For example “This game is hard” might be fine if you want the game to be difficult. Or “You should add a new weapon” is a clue that something is wrong – but the fix might be completely different to adding a new weapon, and needs exploring.

Trust the player behaviour you see yourself, and use what players say as a clue to investigate further – but don’t act immediately on their feedback.

🚀 Do this now: Separate your playtest data into ‘observations’ and ‘feedback’. Prioritise working on issues flagged by the ‘observation’ data.

Get serious about playtesting

Pop-up playtesting is a good toe into the water of playtesting, and will start to show value immediately. But pretty soon you’ll realise you want to take this more seriously, and be ready to run in-person or remote playtests at the drop of a hat, write reliable surveys, and run high quality playtests throughout development. 

I’ve made the playtest kit for you. 

It is a repeatable playtest process, making it simple to run professional quality playtests, iterate your game, and make a game that players will love.

Learn more about the playtest kit, and sign up for the free email course to get started today gathering playtesters at playtestkit.com 

Categories
Newsletter Research Skills

How to make time for games UX research

Developing a game can be a project management disaster, lurching from crisis to crisis until we hit the launch date (and then diving straight into developing the DLC!)

For UX researchers or designers this can make it difficult to do our job ‘properly’. Our processes require time, and there never seems to be time to do proper research, explore design options, or think about what we’re doing. 

UX gets left as an afterthought.

Often UX practitioners aren’t given the space to do the job they were hired to do  – they know what they are meant to be doing, and have the UX skills required, but feel stuck or unable to do their job. And iteration becomes a pipe dream.

This is frustrating, and leads to disengagement or burnout. It doesn’t have to be that way.

By the end of this article, you will have a library of immediate and long-term tactics, to create the time for proper playtesting, research, UX design and iteration to happen. 

Game development is a crisis that lasts 2-5 years

Games are very complex software to develop.

Unlike apps, or websites, it’s often not possible to launch a ‘minimum viable product’, and the game has to be complete at launch. Plus marketing puts huge pressure on meeting strict deadlines. And the ‘fun of the game’ is often discovered on the way, so can’t be pre-defined! This adds up to something that is almost impossible to project manage.

These factors means game development often is rushed, and difficult to plan – leading to changing priorities, crunch, and panic everywhere. 

Panic makes proper UX difficult

UX research and design practices assume that we have time to think about and react to what we learn from users and playtesting. 

This requires coordination between teams (which often isn’t being done elsewhere), asking questions to understand the intended vision, and creating a cohesive and shared understanding of the game.

When you’re not given the time to do it properly, the impact of UX design and research is lost. UX designers end up just making wireframes (and I thought the point was “we’re not UI”). User research and playtesting becomes a checklist item, rather than real inspiration for design changes. It creates a culture of low quality, low impact work.

Not doing your job properly will make you sad

For you as an individual, this reduces job satisfaction, and can make it hard to get new roles – if you’ve just been doing low-quality work, how can you pass an interview at another company?

HOWEVER

Trying to fix a low-maturity UX environment is a pro-level skill, that will make you stand out as a senior practitioner. Doing the slow, frustrating work to understand the environment, and improve it is what will help you move up in expertise, get promotions, ace job interviews and grow your career. 

My favourite people are those who try and fix the environment, rather than just complaining about it. Here’s how we can do that.

Short-term fixes to get out of an emergency

A common situation in a panic-filled environment is an incoming, high-priority request which has to be done immediately. Due to poor planning and communication, suddenly a research study needs to be run today. 

Obviously this compromises ‘doing good research’. Without appropriate time or attention given to participant screening, using the appropriate method, or analysis, we do low-quality work. 

Understand the context for the request

Start with asking questions to understand the context. 

  • Why is this request urgent? 
  • What happened immediately before this request occurred?
  • Who is asking for it? 
  • What are they going to do with the results?

Often with exploration, it becomes clear that there isn’t time to react to the findings of a study, or that the request hasn’t been properly thought through. Sometimes with further questions, the request just goes away. That’s great to learn before you waste your time reacting to it (and extremely frustrating to learn after!).

🚀 Do this now: Prepare a list of context questions that you will ask about incoming work requests, and make a template.

Make pragmatic method decisions

Even with context, the reality is that sometimes fast research needs to be done. One tactic we have, but should use sparingly, is method selection. As researchers we’re always making conscious decisions about what is ‘the best’ method for each research objective. However to avoid running studies that are too late to have impact ‘the best’ should also take into account the time we have available – not just the most technically correct.

Making pragmatic decisions about methods can reduce the time that research studies can take – for example running an unmoderated study using a panel when there’s not enough time for a moderated study, guerilla recruitment methods, or running a usability review instead of a full study with users. 

This can extend into how we report. Instead of writing a full report, why not talk your team through your analysed notes? This will get us to debrief earlier and works well with an engaged team who has understood the context of the study.

We have to be careful when making pragmatic compromises. Doing lower-quality work, and compromising your processes or expertise will set expectations that this is normal, and can be done again. When making compromises, they need to be negotiated with the people making the requests, to ensure that they understand the impact of the compromise. When sharing findings, the compromises need to be reinforced with caveats in your deliverables and debriefs by explaining the limitations of the study, why the decision was made to compromise, and what a better study would look like.

🚀 Do this now: Note down the list of research methods you have available to you, and make a decision about what substitutions are appropriate.

Mid-term fixes – Planning more efficient research

The short-term fixes are immediate sticking plasters but will not improve the situation long term, and the same situation will keep recurring. We need to do something about that! 

Anticipate upcoming requests

Because user research & UX is often an afterthought, we need to be proactive to get as much notice about changing priorities and incoming potential work as possible. This requires speaking to people – usually producers & product managers who have the best visibility of the priorities of the game, in order to understand what’s important to them. 

Schedule regular check-ins with your producers if they aren’t already happening, and ask questions such as:

  • What’s important to you right now?
  • What are you most uncertain about?
  • What did you do as a result of our last study?

By having an earlier view of changing priorities, it can help create an appropriate roadmap for research + UX activities, planned out with enough time to execute properly. Conflicts can be negotiated within that timeline, and it can be updated as priorities change – but you will be creating more visibility, and time for you to do your role properly.

🚀 Do this now: Put in a regular meeting with your producer or PM to ask about priorities.

Make research more efficient

Researchers can also make their research process more efficient. The idea behind ‘research-ops’ is to recognise the parts of research that are outside the ‘data collection’ of actually speaking to users, and explore how to make them more effective and efficient. 

This can include creating a participant recruitment panel, but also the creation of templates and checklists to make the research process more efficient (This is what The Playtest Kit does). Optimising your analysis process with better tools (like learning how to master mindmaps) will also reduce the time it takes to get high-quality findings back from your research study. 

Making your research process more efficient will allow you to do higher quality work in less time, and fits better with hectic development schedules. 

🚀 Do this now: Decide which parts of your process would most benefit from being templated, and make your first template.

Re-surface old research findings

Sometimes the findings of previous studies can be useful again, reducing the pressure on running ‘new’ studies. 

Particularly when learning about user behaviour, many of the findings are relevant beyond when the study is first run, and can inform the team’s decision-making regularly. Making previous research discoverable will reduce the demand for new studie, and can be done with a research repository. (I covered this briefly in my book Building User Research Teams, and you can read more about research repositories here ).

🚀 Do this now: Create a spreadsheet to track the objectives and reports from previous research studies, and populate it for studies from the last six months.

Long-term: Change the culture of the organisation 

Longer time, we need to move the game team away from this short-term panicked decision-making. This is obviously much broader than what a researcher alone can accomplish. However we have skills as design and research professionals that can help culture change be more successful, and ultimately increase the impact our studies will have.

This can start with small questions – asking questions about the intended experience, and encouraging our colleagues to define their hypotheses so that we can test against them. This will help people start thinking about the impact on users when making decisions, and can feed into activities about defining the game’s vision. Debating and defining the intended experience for the game will reduce volatility, and create a more stable development environment.

Understand and map the current experience

As a researcher, you have the skillset required to understand why last-minute crises happen. It’s time to journey map!

In a calm(er) moment, to the team about a recent last-minute study or change you were asked to make. Ask questions to understand why that request came in, and follow up the decision-making chain to work out what occurred at each stage. 

Repeat the exercise for multiple instances until patterns start to emerge. Use your skills in mapping and visualising complex domains to show where problems emerge in our development process on a journey map. 

This journey map artefact can be a powerful shared tool for conversations around “why does this occur”, and “what’s the impact of not having enough time to do UX or user research”. It provides the evidence and information needed to make sensible decisions about how to fix development. 

A hypothetical journey map showing how a feature decision was made
A hypothetical journey map

UX and service designers have the skills to run structured activities around fixing the issues once identified, and this can be an important tool for creating cultural change.

🚀 Do this now: Pick a recent ‘panicked’ request that came in, and ask the person who brought it to you about where it came from. Write this down to start building data for your map.

Advocate for change

We’re also responsible for educating people who don’t know of better ways of working. No-one wants to work in a panicked work environment, with rapidly shifting priorities. However many non-UXers won’t recognise the relevance that user research and UX design can have to improving the environment. 

By understanding our users we can:

  • Have a consistent vision of the player experience we’re working towards
  • Spot things that aren’t working early, and pivot when there is still time
  • Make better decisions throughout development, as our baseline understanding of users increases
  • Identify the core pillars of the game, and emphasise them in gameplay and marketing.

We just need to help teams understand the development advantages to running research, and frame it in terms that they care about (often risk, money, development time). Education through research observation, show and tells, collaborative activities, and relationship building can all help achieve this. 

🚀 Do this now: Create a shortlist of people you need to convince about the value of research + UX process.

Recognise a hostile environment

As a sole researcher or UX designer, you cant change the culture of a company. If it’s not a two-way relationship, and the organisation isn’t reciprocating, or saying it recognises the need for change, you need to look after yourself.

Change takes time, but pushing for change in an environment where it’s not welcomed is a fast track to burn out. 

Recognise when the environment isn’t going to get better, and leave. 

Low maturity UX research environment - just walk out!
Career advice from Da Share Zone.

🚀 Do this now: Update your LinkedIn, so you are ready when it’s required.

Ignoring Games User Research, Playtesting and UX is saving up problems for later

UX research speeds up and improves the quality of the game development. But it’s also reasonably new, and we still need to help people understand how it achieves that, and demonstrate the success of it. 

This can be slow, hard work – and very frustrating in hectic game development environments. But by combining pragmatic compromises with long term fixes, we can help advance the maturity of the teams we work with, and ultimately do better, more fulfilling work.

If you need to create an iterative playtesting + games user research process, the Playtest Kit will help. Read more about the Playtest Kit here.

Continue growing your playtest skills

Found this helpful? We’ll be exploring all of these topics in more depth in the future. Get a single email each month about playtesting and UX in games, with an article just like this one.

Categories
Newsletter Research Skills

How to get games user research experience (before you have a job)

Applying for games user research roles can sometimes feel hopeless. There are hundreds of applications for each role, and yours gets lost in the pile. You never even hear back.

(and worse – what about those entry level roles which ask for two years experience?!) 

It’s the classic problem that everyone runs into at the start of their career. You need experience to show you can do the job, and get an interview. But you need the job to get experience. Sometimes it can feel like you’re stuck! 

Today we’re going to tackle how to get games experience before you get your first job. This will help you stand out amongst the other candidates, and increase the chance of getting interviews. Plus you get started actually doing user research, instead of just reading about it – which teaches a huge amount.

This is an updated version of my talk for the GRUX conference in 2021. You can watch the video of the original talk here. The updates cover a whole bunch of follow-up questions that have come up since such as:

  • Is it ethical to work for free?
  • How do I know when I have enough expertise to get started with this?
  • How do I know when I’ve done enough?
  • What skills should I emphasise when describing my work?

Read on to learn how to get games user experience without already working in the industry.

How to get Games UX Research Experience before you have the job

How to get games user research experience before you have the job

Junior games user research roles are extremely competitive. Every year a new batch of people finish studies, complete bootcamps, or decide to move on from another industry. People love games, and want to work in the industry. This means there are a lot of very qualified candidates, and it can be hard to stand out.

One tactic for standing out is experience. Demonstrating you can do games user research, and having a body of work you can show people, will distinguish you from other candidates, and make getting an interview more likely. 

The process we’ll cover is how to use usability reviews, and partner with real game devs, to get experience. This is better than reviewing a finished game, because it can lead to a real change in the game, and allow you to practice + demonstrate communication with game teams. 

This hits many of the skills game developers are looking for

You get to demonstrate all of these skills companies are looking for
Understanding game development
Identifying research objectives
Designing a study
Analysing data
Communicating findings
Working with game developers to make changes

Let’s begin.

What is a games usability review

A usability review (or ‘expert review’) is when one or more usability practitioners play through the game, and spot potential issues. (As an aside – I never refer to these as expert reviews, although it’s a common industry term. I don’t think going around calling yourself an expert is a way to make friends with game developers). 

The type of issues we’re looking for are usability issues – parts where the player won’t understand what they are meant to do, or what is happening, or how to accomplish their goal. Looking for usability issues is safer than trying to spot ‘user experience’ issues – whether players will like the game or not. We’re looking to anticipate their behaviour, not their opinions! 

As a research method, usability reviews are not used in industry that often. Running real tests with players creates objective data (“the player did or didn’t do this”), so there should be no debate about whether the issue is true or not. In contrast, because usability issues are based on your expertise, it can become a lot more difficult to convince people that the issues you’ve identified are real. However they are sometimes used before running a bigger test with real users, to spot some obvious problems. They are also commonly used as part of the application process when applying for games jobs, so a good thing to practice! 

They are also free to organise, which is why we’re going to do them today. 

You might be wondering “am I ready to start doing this?”. I started this process within my first few months of learning that games user research exists. The reviews have been lost to time, and were probably rubbish – but that’s normal. We need to start with doing this poorly, getting feedback and iterating, to get good. So, start today! 

Find a game that is in the right state

We need to find a real game to work on. Luckily, there are places where game developers hang out and ask for feedback – playtesting forums.

Some examples include:

Reddit’s r/playtesters, r/playtesting and r/destroymygame forums

TIGSource’s Playtesting forums

Itch.io’s playtesting forums

(of all of those, my recommendation is the r/playtesters forum is the most active and high quality). 

Normally, I don’t recommend using these playtesting forums to game developers. Because only other game developers hang out there, you get massive sampling bias (here’s a better method I recommend to small game teams). But for our purposes today, that doesn’t matter, as we’re just using them to look for games in the right state.

As you’ll see, game developers post their game on these forums looking for feedback. Most won’t be familiar with what ‘user research’ is, or what a ‘usability review’ is – we’ll deal with that in a minute. But the fact they are posting tells us they have a game, and it’s in a state where they are prepared to make changes to it – exactly what we’re after.


There are a lot of posts on these forums, and a lot of the games are low quality. We need to filter those down to being the best ones to run you review on. Some criteria I use are:

Is there time to make changes?

Read the most and look at the state of the game. Sometimes people mis-use those forums to promote a game post-launch. That’s no good for us – if the game is already out, they are unlikely to make real changes, regardless of what you find in your review. 

Can the team handle usability feedback?

Look at  what information they’ve provided about the game. Do they feel like they will continue development, and be able to react to the issues you discover? Or are they only requesting very specific types of feedback?

This is something we’ll explore more when we start talking to the developer – working out if they are open to, and will be able to react based on usability feedback you discover. 

(also a lot of the things on these forums will never make it to launch. Again that’s not necessarily a bad thing, but you should try and suss out how serious they are so that you can prioritise projects that have a chance of coming to fruition.)

Can I run this review?

Look at the genre of the game. Ask yourself, is this a genre I have played before and know the ‘format’ of. This will help us later identify what are real usability issues players will have, and not be distracted by issues that will not occur for ‘true’ players.

For example, If you’ve never played a real-time-strategy game before, you will find it challenging to identify and describe usability issues that occur within the genre.

Can I share what I learned?

From this usability review, we want to get a good case study that can be shared with potential employers. We’ll need to ask permission about including this project as a case study on your website or portfolio. That will allow you to be more specific with examples when talking about how you handled the usability review in interviews, or on a blog post.

I’ve never had an indie developer tell me they are not happy with the findings being shared (because they benefit from getting the feedback, and appreciate any attention being paid to their game), but it’s polite to check!

Can the team afford professional help?

User research is a professional skill, and deserves to be compensated as such. Agencies and consultants will be charging five figures to do this kind of review professionally. We want to avoid devaluaing our expertise, by not working for free. 

This means that we shouldn’t be doing this work for any company that can afford to pay a professional. If they are paying for professional QA, marketing, localisation, or other third party services – don’t do this for free. Either find a different team to work with, or come up with a fair price for your time. 

My ideal game team for this project is one who has no budget, because this then becomes a mutually beneficial partnership. You gain experience of running this kind of work, and a portfolio piece. They benefit from your growing UX expertise. 

When preparing this talk, I went through the process myself. Here’s the post I found for a team I wanted to approach – I thought it was a genre I understood and I could see they were still making changes to their game.

A picture of a game designer asking for beta testers for their puzzle game

I was still unsure if they could handle usability feedback, or would be comfortable with me sharing the results (especially as I wanted to use them for a conference talk!). But this was enough of a hopeful lead to get in touch with them.

Make the approach

Having found a potential lead it’s time to approach them. Remember that most game developers working at this level will not have encountered UX, usability reviews, or structured playtesting – they think they are just going to get general opinion feedback. We need to explain the difference between what they asked, and our proposal (which is great practice for advocating for games user research). 

You have to explain what it is you do:

Hi <<name>>

I'm <<name>>, and I run usability reviews of games which identify usability and UX issues that will prevent players from experiencing it the way you expect.

I saw your post looking for playtesters for <<game>>, and wondered if you would be interested in a usability review for it. I will identify usability problems, and send you a report covering the top issues. 

I wouldn't charge for the review, but I'd also be really interested in sharing this as a case study <<on my website/blog>>. This might create some extra publicity for your game, so hopefully a positive for us both.

Let me know if this sounds interesting and looking forward to chatting further.

Working with game developers is great experience to demonstrate when applying for game jobs later, and so building a good relationship with the developer is really important. The more you can do on this (such as asking their priorities for the game, understanding what they are worried about, and working out how to be helpful), the more useful this will be to reference in interviews and job applications later.

You will also want to feel out whether they will be a good partner. Do they feel open and interested in the process, and likely to be a good person to work with. Are they giving the impression they take game development seriously, or are they opening Unity for the first time and have vastly over-scoped their game. If you’re not getting good vibes, politely move on to the next developer. 

Running your review

Once you’ve set expectations, understood their priorities and have a copy of the game – you are ready to run your review.

Running good usability reviews is a whole different topic (and one we’ll cover in future newsletters, so sign up to make sure you don’t miss it). This talk from Seb Long is a good place to start learning in the meantime

My own approach when doing this was:

  • Work with the developer to understand their current priorities and focus. List those out
  • Then play through the game once, as a fresh player. Record the video, and audio of my ‘new player experience’. Write notes throughout.
  • Then do a second pass, looking at the game as a UX practitioner. Look at every element of the game, what I anticipate the design intent is, and how or why players might misunderstand.
  • Then watch back the new player video, take more notes, and try and notice ‘what the designer meant to the experience to be’ compared to what I actually experienced.

That generated a lot of raw material for my report. 

For extra credit, you could find some users and observe them playing. This will practise participant moderation (which is another of those core skills above that interviewers want t osee). But it does add a time commitment, potential cost (if you pay participants) and is more challenging to organise than a usability review.

Share what you discovered

After you have your raw data, you’ll want to write a report.  Here’s my free guide on how to write a report.

You can see the report I made when preparing this talk here

Making the report is good practice, as it’s a common way of sharing results when working in industry – and demonstrates your ability to communicate findings (another one of those core researcher skills!).

I strongly recommend that you show your report to someone else for feedback. This is critical for improving – getting feedback from peers, working out what people don’t understand, and finding better ways of communicating it. I’ve been doing this for over a decade, and I still ask peers to read my reports before sharing further.

(As an aside – are you interested in joining a small group of people learning games user research to share their work in progress, and give each other feedback? If so pop your name down here, and if enough people sign up we’ll make it happen).

After writing your report, and getting feedback, share it with the development team. Remember that communicating with developers is a core thing that will help you stand out from all the other candidates in interviews, so try and make this as ‘active’ as possible. If you can set up a 30 minute call to present your findings, that’s fantastic!

You should also follow up after sending the report. Ask them if the report helped, and what changes they are considering. This is great evidence to show the impact of your work, when interviewing for a games user research job later.

Make it into a portfolio piece

Now you’ve done some work, and have permission to share it, it’s time to do so. 

To share it with others, you want to tell the story – rather than just show the final report. This should include you explaining…

  • What you did
  • Why you did it
  • What compromises you had to make beyond the ‘perfect’ study, and why you made those compromises.

This would all make a great blog post, if you have a blog. We’ve previously covered how to make a portfolio, but some of the things to emphasise from this work will be the contact you had with the developer and the impact that your study had on changes to the game. That kind of experience very few other people will have, and should really help. 

This whole process does take time, but doing it just once or twice will put you in the top 10% of applicants for jobs. That doesn’t guarantee you will get through to an interview, but it does increase the chances.

Plus as an added benefit, you might even get a credit in the game. Here’s my credit, for the project I did to prepare for this talk:

Games User Research - Steve Bromley

Get started

So, time to get started by looking at those playtesting communities, and finding some good leads. Do tell me about your experiences running usability reviews, either on Twitter (@steve_bromley), or by dropping me an email.

(and did you spot the aside above about making a group of learners to share feedback on eachother’s work? Sign up if intrigued, and we’ll make it happen!)

Continue your games user research journey

Get my free ebook of career tips from researchers at top game studios such as EA, Blizzard, PlayStation, Ubisoft, Activision and more. Based on exclusive interviews, they reveal ten essential tips to kick-start your GUR career today.

Plus receive a monthly email with free games user research lessons and curated Games UX and UR jobs.

Sign up now to get the free e-book (no spam, just a nice email from me each month!)

Ten essential tips for new and aspiring games user reseachers ebook
Categories
Newsletter Research Skills

How to keep games secret in playtests

Techniques for avoiding leaks, and keeping games secret when running playtests and UX research.

We don’t want leaks from our playtests

When speaking to teams, I often see that they are worried about leaks. That fear can put them off the idea of playtesting.

With small teams and solo developers there is the worry that playtesting their game will give away their ideas, and someone might steal them. From the larger teams, there is the worry that leaks will ruin their marketing plans

Many games spend as much on marketing as they do on production (because marketing is such an important part of making a successful game). This means that leaks can be seriously disruptive to established marketing plans, and there is a lot of wariness around playtests. Compared to other industries games are particularly at risk of leaks, because players are very hungry for information, and there are very active player communities to share leaked information. Even the smallest game teams need to be conscious of what assets they are putting online, and who might see them.

This means that game teams, and senior management can be very hesitant about running playtests. 

This fear is well founded. From the thousands of hours of playtests I’ve run, I’ve had leaks occur twice, and neither was good. Even though they didn’t severely disrupt the game’s marketing or announcement they were embarrassing and had the potential to impact my relationship with the teams I worked with. With a bad enough leak, it could potentially make teams not want to playtest again.

However the risk of leaks can’t stop us from running playtests. The value of playtesting is too high, and needs to occur as early as possible in game development in order for us to have the right impact and fix problems at the right time. So, we need tactics to minimise leaks. 

In this post, we’ll look at techniques available to researchers, or game teams of all sizes, to minimise the risk of leaks from playtests. 

How to keep your games secret

Playtest without leaks

Start with the basics: Get participants to sign an NDA

A non-disclosure agreement (NDA) is a legal agreement to keep something secret. It’s typical to sign it when working on games projects (which makes portfolios difficult!), and equally typical to ask playtesters to sign them at the start of playtests. 

Templates for NDAs exist across the internet. Here’s a UK focused NDA template from UKIE (thanks Seb Long for the link). Most non-hobbyist development teams already have a friendly lawyer, who can put together a bespoke NDA for you. I’ve also included a link to several alternative NDAs in the Playtest Kit.

The risk with relying on an NDA is that they are very difficult to enforce. If a participant breaks the NDA and puts some screenshots or secrets on the internet, it costs money to get a lawyer to act on it, and might draw more attention to the leak. Plus at that point, it’s often too late – the leaked information is on the internet, and will be spreading immediately. 

Because enforcing them can be counter-productive, the point of NDAs is often more about the threat of taking action rather than a real intention to enforce it.

Best practice is to ask your participant to sign the non-disclosure agreement at the beginning of the session, and then let them take a copy with them. Remind them again that they signed an NDA at the end of the playtest. Making the process seem formal and official will hopefully increase players’ likelihood to take it seriously. 

Choose in-person research methods for the most secret games

Different research methods also have different levels of safety for leaks. At PlayStation, the vast majority of our research was in our research lab, so that we can keep an eye on players and prevent them from taking screenshots or information away from the session.  

Taking participants’ phones away (and asking them to put them in a locker) will also help prevent photos from being taken.

This also allows you to take time to explain and emphasise the NDA – running through it with the players to make sure they understand and making explicit that it is a legal document. This will increase adherence.

Make remote research as safe as possible

Lab-based, moderated research is the ‘safest’ method to prevent leaks, but even when running remote research, there are some steps that can help. Choosing to moderate the study, where the participant is talking with you live, will create fewer opportunities to capture leakable information.

Bob Tilford, now of RockStar, gave a great talk about how building rapport can help minimise leaks, which can be viewed here:

For unmoderated research, there is an increased risk that players are left with the game, and can screenshot it. It also prevents you building rapport with the participants, increasing the risk.

For many games, it may be sensible to stream the game to players, rather than sending them a build directly. Although not feasible for games which require extreme responsiveness, for slower games this means you don’t have to actually let players have un-monitored access to the game, and can be in control of when and where they can access the game.

Parsec is one of these tools that allow remote players to stream access to builds on your own device, similar to how Google Stadia works. There is a video of how Parsec works on their website, as well as a case study of Ubisoft using it to distribute builds for a marketing event.

Watermark the build

Watermarking is putting an overlay on the screen, with some identifying information, like in this picture:

A game screenshot. In the corner is a watermark which has a date, the name of the computer, and an IP address on.

This can be invisible to the player (see this thread on how it is done inside World Of Warcraft) but for our purposes, we want players to know that leaked information can be tracked back to them. Some sensible information to include can be the current time, date, and any identifiable information that can link the screenshot to a specific device. 

Showing that screenshots can be linked back to an individual increases the perceived risk of leaking information, and can help discourage leaks. This will however require some development time to implement, so isn’t practical for all teams.

Remove important assets from the playtest

When working with established IP, it can help to obscure the IP in the test build. This could look like changing the title screen or renaming the game in the build so that it doesn’t mention or show pictures from the brand. This is perhaps most relevant to simple mobile games, as replacing entire character models is unlikely to be possible in more complex games. 

Removing assets can be achieved either through creating a custom build, or avoiding including the final assets in any build until post-announcement. Replacing assets also requires development time, but can protect a game from leaks. 

And most importantly – screen your participants. 

Both times I’ve had issues from playtests, they could have been resolved by better screening. In one of the sessions, they were a journalist from a minor publication, who believed they had a scoop. In the other, two participants knew each other, and were able to collectively be brave enough, and then corroborate, leaked information to the games press.

Remember to check when recruiting, and verify during a pre-interview:

  • That they are not a journalist
  • That they are not a streamer/content creator
  • That they have no personal connection to other participants

There are very few actively malicious participants (I’ve seen thousands of players, and only had issues a few times) – but asking these questions, and removing people who fail them or can’t give convincing answers, will help identify the most at-risk sessions.

Building trust through consistent results.

Worries about leaks push teams to delay running external playtests. Instead many teams just run playtests with their colleagues as participants. But our internal colleagues are just so different from real players, that a lot of the value of playtesting gets lost. 

These techniques combined can lead to a robust playtest setup, with minimal risk of leaks. But getting permission to run playtests, or the confidence to run your own is a much more human problem. 

Building trust from management can take time. Looking at teams who have been successful at integrating playtesting in their development process, it can often take multiple games to achieve.  Showing value with late evaluative playtests can build trust, and on the next game playtesting can start earlier and have more impact.

A great example of how a relationship can change over time is the work done at Bungie, moving from a few studies in the year before release for Halo 2, to regular playtests across the four years prior to launch for Destiny.

A graph showing how many tests were run on Halo 2, 3 and Destiny.
On Halo 2, testing is mostly from a year before launch. Halo 3 - two years. Destiny has frequent tests throughout the four years before launch.

(The source is John Hopson’s 2015 GDC talk)

Building trust is a marathon, not a sprint – but being able to avoid embarrassing playtest leaks will help us get there! 

Continue growing your games UX skills

Every month I send a playtesting and games user research lesson, just like this one, to help game developers run better playtests, and people start a career in games user research. Sign up to get the next one direct to your inbox:

Better playtesting today

If you’re interested in turbo-charging your playtesting, I’ve worked with game designers, producers, community managers, UX designers, QA managers and solo game devs to make The Playtest Kit.

It brings the expertise of over 25,000 playtest hours into one complete playtest toolkit – making playtesting accessible to game developers who have no time or money.

Learn more, and sign up for the free one week playtest masterclass at playtestkit.com

Categories
Newsletter Research Skills

Find usability issues in games with playtests

Usability issues can ruin games.

When players can’t understand what they are meant to do, where they are meant to go, or what is happening, they get confused, bored and ultimately drop out. This impacts their opinion of the game, retention, and creates an experience that just isn’t fun.

User researchers can help.

One of the core skills for researchers, and one of the focus areas for the How To Be A Games User Researcher book is how to find and deal with usability issues. This helps make games that players understand and enjoy. 

Today we’re going to look at how to use playtests and UX research studies to find usability issues in games.

Get future games user research lessons direct to your inbox

Find games usability issues 

Find Games Usability Issues

Today we’re looking at usability issues. What we mean by usability is “can players do what they are expected to do”. 

That’s different to opinions about a game. “I don’t like that the weapons break in Zelda” is an opinion.  “I don’t understand why my weapons keep breaking” is a usability issue. Today we’re tackling the latter.

1. Start with understanding the design intent

Usability issues are ‘when the player can’t do what we expect them to do’. To spot that, we need to know ‘what do we expect them to do’.

This involves talking to our colleagues. Designers have an idea about how they expect the game to work, and what they think players should understand and do. You need to learn what their expectations are (the ‘design intent’).

Exactly who to speak to, and what to ask depends on what you’re testing – which is why our studies start with scoping around research objectives.  You may need to talk to combat designers, UI artists, level designers, producers, or others, But for an idea about the range of things we may need to understand look at Liz England’s ‘The Door Problem’. You need to know what the designer would say in answer to every one of those questions. 

This can be a lot of chatting.

But it will make the other steps a lot more effective! 

2. Create the right context to create realistic issues

Next comes creating the task. What are we actually going to ask the player to do. 

Some tasks can be very broad “Complete this level” or “play for thirty minutes”. Sometimes they can be very specific “Craft an upgrade to your horse armour” or “Place this object on the shelf”. Exactly what level to set your task depends on your research objectives, but my preference is to lean towards broader tasks where possible, because it allows unexpected issues to emerge.

If you are live in your playtest, you can be looser on the task. Because you are in the room, and watching them play, you can jump in with an improvised task at the right time. 

Designing tasks in UX tests can be tricky. Revealing too much information will artificially lead players to the solution. Revealing not enough information (such as missing tutorials) will create artificial situations that won’t occur in the final game. This needs a lot of thought to balance.

My preference is to be as undirected as possible, while meeting the research objectives. “Play through this bit” is fine, as long as they are exposed to the right content, any tutorials or pre-requisite content, and is easier when you are on hand to subtly guide players towards the right content.

3. Be quiet, watch and look out for deviations

Every time you intervene with players, you are introducing artificial elements to the playtest. Unless you intend to answer questions live for every player post-launch, you should resist answering questions, or revealing information in the session. 

Instead your attention should be focused on listening and watching. Observe what players are doing in the game, and when they do something unexpected, write it down. Looking back at your research objectives, and a good understanding of the design intent will help you recognise unexpected behaviour. 

Write down what you see (we’ll cover good note taking in a future issue). 

When necessary, hand out new tasks to direct players attention to the right areas. (but still be very careful that the words you use or instructions you give don’t reveal information that the player wouldn’t otherwise have)

4. Ask questions to understand what you’re seeing

Observed behaviour is only half of the story. In order to understand why that behaviour occurs, we need to know what’s happening in player’s heads.

This requires asking them questions. 

As described above, there is a massive risk that your questions reveal information to the player that they wouldn’t normally have – artificially influencing your playtest. Bland, non-leading questions are required – such as “what is happening now”. We wrote more about how to ask good questions in a previous issue. 

5. Use the impact to rate severity

It’s common to spot a lot of problems, and as covered in the How To Be A Games User Researcher book, researchers go through a structured analysis process to uncover and describe them all.

In order to focus the team’s attention, we need a process for identifying which problem is most pressing. We do this by anticipating what issues will be most disruptive to players when they play.

This extract from How To Be A Games User Researcher explains one approach for doing this:

A method I like is a four-point scale for issues:

  • Critical
  • High
  • Medium
  • Low

Each issue starts as ‘Low’. Then ask three questions about the issue. Each time the answer is yes, raise the severity up a level.

These questions are:

  • Is this something that the player needs to do to progress?
  • Did the moderator need to step in to help resolve the issue?
  • Once the player had overcome the issue, did they know how to avoid it when they encountered it again?

When running your playtest, noting “what happened to the player as a result of this problem”, gives you the raw information you need to do this analysis + rating process. We will cover analysis in more depth in future issues.

A chance to practice

I’m sharing a session of a player who is playing Pokemon Unite for the first time. During their play session, they encounter some usability issues with the game.

Watch an unmoderated usability test of Pokemon Unite

As you watch, think about:

  • The design intent (what did the game team expect players to understand or do)
  • The issues the players encounter
  • What questions you’d ask, if the session was live
  • What impact the issues had on the player’s experience

(Feel free to share what you spot with me on Twitter!) 

Be ready for a career in games user research

Get my free ebook of career tips from researchers at top game studios such as EA, Blizzard, PlayStation, Ubisoft, Activision and more. Based on exclusive interviews, they reveal ten essential tips to kick-start your GUR career today.

Plus receive a monthly email with free games user research lessons and curated Games UX and UR jobs.

Sign up now to get the free e-book (no spam, just a nice email from me each month!)

Ten essential tips for new and aspiring games user reseachers ebook

Don’t miss the next issue

Every month get an email from me with a free games user research lesson, and the latest entry-level games UX and UR jobs.

Categories
Career Advice Newsletter

Make a games user research portfolio

Many disciplines in games development demand portfolios. For artists, or UX Designers, they are almost essential. But how does that translate for games user researchers?

Making a portfolio can take a huge amount of time – and it’s difficult to know whether it’s going to have an impact on impressing employers, or getting your first games user research role.

In this issue we explore whether it’s worth making a games user research portfolio, and what a portfolio should include.

Get future games user research lessons direct to your inbox

Create a games user research portfolio

Build a games research portfolio

Do I need a games user research portfolio?

Need? No. Only once in my career have I been asked for a portfolio, and it was for a role where none of the people hiring me were researchers. When hiring researchers, I don’t ask for a portfolio.

I asked other hiring managers whether a portfolio was expected on Twitter, and the answer was broadly similar to my experience. Only 31% of hiring managers said they expected a portfolio.

Making a portfolio for games user research can be difficult for many experienced researchers – due to the secrecy around what we work on, and because our work can be less visual than other games disciplines where portfolios are more common.

However this doesn’t mean that it’s always a bad idea to make one. 

Should I make a portfolio anyway?

If you are already an established researcher, I wouldn’t recommend making a portfolio as a good use of your time. However for people looking to get their first games user research job, it can be very beneficial – even if it’s not required for applying for roles.

When advertising roles, hundreds of irrelevant applications get sent in (as they are for any ‘games’ job). Having a portfolio quickly establishes you as a sensible, realistic candidate who is someone genuinely applying to be a games user researcher, rather than just someone applying to every job in the games industry.

A challenge for hiring managers is differentiating between candidates when a lot of people apply with the same general experience and background. That’s even harder hiring for a junior role where there’s limited evidence available for them to assess your previous work. 

A portfolio is evidence that you care, and can help you stand out from the crowd.

Seb Long, from Player Research, described the benefit here:

What should my games user research portfolio include?

An ideal games user research candidate would show the following skills, so try and touch on all of them at some point in your portfolio:

  • Qualitative Research: Any experience designing and moderating qualitative research methods, such as interviews, usability testing, observation
  • Quantitative Research: Any experience with lighter quantitative research methods, such as survey design
  • Analysis: Note taking + data analysis experience
  • Presenting: Any experience presenting results to people
  • Collaboration: An ability to work with non-researchers + other disciplines
  • Understanding Games: An understanding of how games are made and what other game roles are
  • Defining participants: Any work on participant recruitment, screening and making sure you are testing with the right people
  • Community: Any involvement in the research community.

Obviously not all of this is possible without pre-existing experience, but a lot of it can be demonstrated through academia or personal projects (I covered a bit on this in my talk ‘how to get games user research experience without a job’)

How to make a games user research portfolio?

When deciding how to create a portfolio, prioritise making it low maintenance. Anything that takes a lot of time or effort to maintain will just not get done. A games user research portfolio could be as simple as some google slides or a powerpoint exported as a PDF. A wordpress blog can also be a suitable base for a portfolio (and has a lot of pre-existing themes that can be used)

There seems to be no consensus on what a research portfolio should look like.  My ideal format would include:

  • A paragraph on the situation
  • Details on the approach you took, and crucially why you took that approach
  • Evidence of impact – did anyone listen or do anything with what you learned? 

Here’s what one page from mine looked like many years back (with the text obscured, unfortunately). It picks one big example, and goes through the situation, approach and impact. Then it adds two smaller examples to show variety. I had similar pages for each of the bullet points listed above (qualitative research, quantitative research, presenting, etc…). 

An example portfolio. The header says 'qualitative research', and then a section for 'the problem', 'my approach', 'the big insights', and 'the outcome'. The text is obscured.

This approach was very text-heavy, and I would consider something more visual these days! 

Remember, a key skill for researchers is communication (because otherwise everyone ignores our findings). Make sure whichever format you chose is doing a good job of clearly communicating your experience. It doesn’t have to be pretty, but it does have to be clear. It will be an advert for you, and should build confidence that you are able to present information in a compelling way.

Keep in mind that you will be judged by the worse thing in your portfolio. A poor method choice, an unreliable conclusion or a research faux pas will be considered a warning sign. Some red flags could include choosing focus groups as a method (which might be appropriate for the right research objective, but would require some explaining) or too much emphasis on the design + development part of a project, rather than explaining the research.

To reduce the risk of including things in a portfolio that reflect poorly on you :

  • Find a peer to review it, don’t be defensive about negative feedback, and take time to understand why people got that impression from your portfolio.
  • Include less things in your portfolio. Your goal from the portfolio is to show ‘I am a UX researcher’, and that can be achieved with only two or three projects.
  • Make sure you explain your decisions. It’s ok to make ‘pragmatic’ rather than ‘perfect’ choices, but show that you did that consciously for good reasons.

Blogging is an alternative approach to a portfolio

Many established researchers don’t have portfolios, and never have. Instead, they used blog posts to demonstrate their interest and involvement in games user research. I believe this is just as effective as creating a portfolio for demonstrating user research experience. 

Some researchers kindly shared the blogs they worked on before they were in industry. Like my own blog, many of these posts no longer represent their opinions or ability – but are still helpful to show what evidence is appropriate for getting a first game in jobs. Many had stories that their blog posts were what got them the job. So consider blogging! 

Example games user research portfolios & blogs

Thanks to everyone who shared their current, or old portfolios + blogs. 

Whether it’s a formal portfolio, or a blog showing your thoughts on games usability, publicly sharing your work in games will help stand out from other applicants for games UX + user research roles.

Do you have a games user research blog, or have you made a portfolio? Send a link to me on twitter and I’ll include it in a future newsletter! 

More games user research career advice

Get my free ebook of career tips from researchers at top game studios such as EA, Blizzard, PlayStation, Ubisoft, Activision and more. Based on exclusive interviews, they reveal ten essential tips to kick-start your GUR career today.

Plus receive a monthly email with free games user research lessons and curated Games UX and UR jobs.

Sign up now to get the free e-book (no spam, just a nice email from me each month!)

Ten essential tips for new and aspiring games user reseachers ebook

Don’t miss the next issue

Categories
Newsletter Research Skills

Expert playtest moderation – ask unbiased questions

Moderating playtests is hard. The wrong question, or an inappropriate tone can derail your playtest. Accidentally upsetting the player, biasing their response, or revealing inappropriate information can severely reduce the value of the data you are collecting.

But one to one interaction with a player is also a core researcher skill, and one of the most impactful methods we have for understanding player behaviour.

As a skill moderation takes practice (I remember how poorly my first moderation went 🤦 ). This month we share some guidance on how to improve your moderation.

Get future games user research lessons direct to your inbox

Learn to moderate playtests

Prepare appropriately

Excellent moderation starts with preparation. Before you step into a room with a player, you will want to be clear on what your research objectives are, and have written a discussion guide to prepare for the session.

This will ensure that your session has structure, and that you know your questions will be relevant. Even if you have to free-style questions, having a reminder of the research objectives in front of you will help remind you of the objectives we’re trying to answer.

Here is a free template for a discussion guide from my first book, which can be adapted for your playtest.

Consider the space

We want players to be at ease when they arrive for their playtest. The room is an important part of creating a friendly atmosphere where players can give feedback freely.

Think about the room you are playtesting in – we want to avoid the space biasing player’s opinions or intimidating them. Remove the many trophies that say you are an excellent game designer, and the marketing posters to create a more realistic play space.

Also think about the playtester’s experience before they reach the playtest room. How will they know what to do when they arrive at the building? Will someone be there to meet them? Will they know what to say? Spending time thinking about and designing the experience of the playtester arriving at your building will help avoid them arriving unsure or upset, and create a more positive atmosphere for feedback.

Start the session off well

The introduction helps set the tone for the rest of the session. As Meta Quest’s Rich Ridlen explains:

Click through to read more great tips from Rich

This can be rehearsed, and again preparation helps. Use the template to remind yourself what to cover at the beginning, including explaining:

  • Your relationship to the game (“I didn’t make this”)
  • This is not a test (“We’re not testing you, we are testing the game”)
  • What is expected from them (“Today we’re going to play the game, and ask you some questions about it afterwards”)

Don’t help.

As a moderator, your presence is unnatural – when the game is released, you won’t be there to help. This means you have to be extremely careful not to introduce new information into the playtest that the game won’t introduce.

Don’t help players when they are stuck*. We need to learn what players would do without outside help, to recreate the authentic player experience and see whether they can overcome the issue.

*The exception to this rule is you can help if it’s something that isn’t in the game yet, but will be. As long as you don’t forget that this is untested because you helped, you can help players overcome unfinished parts of the game.

**The other exception is you can help if everyone understands and agrees it’s an issue. If one player has the problem, let it occur. If the whole team then understands and agrees that this is a problem that should be fixed, you can consider helping other players overcome it, so that you can see new issues.

Don’t confirm, deny or answer questions.

Part of not helping is not letting players know if they are doing the right or wrong thing. If they ask questions about how the game is meant to work, or if they are doing it right, you shouldn’t answer.

A good phrase is What would you do if I wasn’t here. Or turning their question back on them What do you think?

Ask questions, but be careful with what you ask.

So, if we’re unnatural, why are we in the room at all? Part of our role as a moderator is to ask questions to understand why players are acting the way they are. Just watching will show us what players do, but we have to ask questions to understand why.

Questions can be dangerous, as they can introduce information the player didn’t have. Asking How did you know that was the right way to go reveals that this was the right way to go.

Because of this, my favourite questions are usually very bland, and just get players explaining what is going on currently (which you can then follow up on): What is happening currently? or How did you realise that? get players talking, without revealing any information.

Also remember to be careful when you ask questions. As Camille explains:

https://twitter.com/rehe_m/status/1500896495637250052

Read more tips from Camille

Probe deeper

We need players to feel comfortable giving feedback. This requires creating a friendly atmosphere, but also listening to the points they raise.

When players say something, or give feedback, as a moderator we need to consider “is this enough information”. A moderator should always be thinking about what would level of detail would be useful to your team. This requires asking further questions to reveal enough detail that a designer can take action.

This game is too hard isn’t enough detail for a team to action. We need to ask follow up questions to ask what happened that made them believe the game is too hard, so that the team can fix it.

As Laure says this doesn’t always have to be asking a question, leaving space can also help reveal more information:

Read more tips from Laure

Focus on moderation

Moderation is a skill and takes attention. As a moderator we have to carefully watch players to make sure we’re asking timely relevant questions. As Francesca explains, it can’t be multitasked.

https://twitter.com/core2design/status/1501306315867623430

Note-taking is an equally valuable skill, which takes preparation to do correctly. As Donat describes:

Read more tips from Donat

Moderation is a skill that takes practice. We’ll cover more tips in a later issue.

Be ready for a career in games UX research

Get my free ebook of career tips from researchers at top game studios such as EA, Blizzard, PlayStation, Ubisoft, Activision and more. Based on exclusive interviews, they reveal ten essential tips to kick-start your GUR career today.

Plus receive a monthly email with free games user research lessons and curated Games UX and UR jobs.

Sign up now to get the free e-book (no spam, just a nice email from me each month!)

Ten essential tips for new and aspiring games user reseachers ebook

Support How To Be A Games User Researcher

Thanks for reading. If you like this and want to support, consider:

Have a great month.

Steve

Don’t miss the next issue

Every month, get an email with a free games UX research lesson, and the latest entry-level games UX and UR jobs from me.