Categories
Newsletter Research Skills

Sharing Games User Research Findings

User researchers find problems with games. But it’s usually someone else who has to fix them.

This means communication is one of the most important skills for a researcher to have – and sharing games user research results is critical to having an impact on the quality of games. 

In this issue, we’ll look at some approaches for sharing research findings, and share some tips from the pros! 

Get future games user research lessons direct to your inbox

Debriefing games user research results

Sharing games user research results accurately creates an impact

From our playtests, we learn how does the game currently differ from the design intent. Much more crucially, we also understand why it differs – what caused players to not understand, or to do the wrong thing. 

Understanding the why is essential to making the right decisions about how to fix the problems. Usually, it is someone else who actually makes the final decision on what to do to fix it – a game designer, art director, level designer, etc…

Full and accurate communication of the issues, their causes, and the impacts mean that our colleagues make the right choices. 

Making a traditional research report 

The most common way of sharing research findings is to write a report. I have an old example of one on my blog, which – although I would do some things differently today, is still a good idea of what a research report might look like. I’m also sharing a new, real, research report as part of an upcoming GRUX Online talk – look for that in a future issue. 

As you will see, the report runs through each of the issues in turn. They are grouped by topic. Matching your groups to ‘what discipline would be working on fixing this issue’ can make distributing the findings to the right person easier. 

For each issue it explains what the issue was, why it occurred, and what impact it had on the player – this information will help teams make the right fix and prioritise the issue appropriately. 

When writing reports, it’s important to write concisely, and use plain language – we’re trying to make sure we’re understood. Complicated words don’t impress anyone, and just confuse your readers! Here’s some tips on how to write better. 

I would always recommend presenting the report to the team live. People are much more likely to pay attention, will have the opportunity to ask questions if they don’t understand, and it just gets a whole lot more engagement. Although it can be tempting to send a report and say “let me know if there are other questions”, I suspect that this leads to reports being ignored, and reduces the chance the team will react to your findings. Ignored findings make running user research pointless, so let’s try and avoid that when we can.

A report isn’t right for everyone

It’s really easy to get into the trap of thinking ‘a user researchers job is to make research reports’. Our job is to make the game better, and that a research report might not be the best way of doing that. 

For some teams, a report isn’t appropriate and would be ignored – instead consider alternative methods of sharing findings:

  • An interactive workshop
  • An email covering the top 5 issues
  • A message on slack
  • A research insight database that they can interrogate
  • Access to the raw data 
  • A conversation 

When starting with a team, think about how much time they have, how they communicate currently, and use that to decide what is the right way to share the results so that they don’t get ignored.

Research has multiple audiences

As well as the team who asked for the study, there are some secondary audiences who will also be interested in aspects of your study. They might need a different method for the findings to be shared with them.

Some of those audiences can include:

  • Executives who don’t need to know the detail, but do need to know if there is a big problem
  • Other teams working on different projects which some of the findings might be relevant too
  • Yourself in 12 months time, trying to remember if this study is a topic you’ve done before or not. 

Many of these won’t have the time to read through all of the detail of your report – consider some of the other methods that might be more appropriate for sharing research findings with them.  

Some prompts to think about:

In interviews, you are likely to be asked about your experience sharing research findings. Some aspects to think about in your answers include:

  • How did you decide what was the right method of sharing the findings?
  • Did you understand the needs of the audience and use that to inform your decisions?
  • How did you evaluate if you had been understood correctly?

Some research results tips from the pros

I asked the community if they had any tips for a successful debrief. Here’s what they said. 

I like giving my results a hierarchicy that relates to the research problem. That way we circle them back to the research questions and it helps staying in scope. I also use a usability severity rating attached to insights so that it can have an impact on priorities in the debrief with designers :)

Laure described how prioritisation is important for making sure the results are relevant. A common mistake I see (and do myself) is trying to fit in all of the findings, and make them all prominent, when all the team really need is ‘what is the most important thing we need to do next’ – which Laure’s method addresses.

For a usability rating scale, I personally like using this severity scale described by userfocus – but other’s exist too! 

You should definitely make sure to know for whom you are writing your report. The vocabulary you use might be different based on who reads it. Also, the level of details might vary based on how much time readers are willing to dedicate to your report. It's also a good idea to have quick takeaways at the very beginning for people who don't have time or to pick up interest of others.

As Raphael says, understanding your audience is incredibly important. At the start of a project, make sure you know who will be interested in the results, so that you can tailor your delivery to them.

Playtesting at Develop Conference

A busy month ahead for me. I’ll be talking at Develop Conference next month sharing some lessons on running affordable playtesting for teams who can’t afford full-time research support. 

Steve Bromley Develop - Better Playtesting for Indie Developers

Do come and say hi if you’re attending! (and if it’s a topic you’re interested in, do sign up for updates on when I publish more resources to help indie teams)

Have a great month and good luck on your games user research journey!

Steve (follow me on Twitter for more games user research news throughout the month!)

Get a free games user research career guide

Get my free ebook of career tips from researchers at top game studios such as EA, Blizzard, PlayStation, Ubisoft, Activision and more. Based on exclusive interviews, they reveal ten essential tips to kick-start your GUR career today.

Plus receive a monthly email with free games user research lessons and curated early-career Games UX and UR jobs.

Sign up now to get the free e-book (no spam, just a nice email from me each month!)

Ten essential tips for new and aspiring games user reseachers ebook

Don’t miss future issues

Thanks for reading this issue. If you’re reading online, and want to get one email each month with a games user research lesson, and entry level games UX and UR jobs – sign up here.

Categories
Newsletter Research Skills

Remote games user research

For the first five years of my career, most of the research studies I ran were in person. It was much easier to bring people to us, rather than transport our technically complex setup to them.

This had started to change for me prior to the pandemic, and then exploded last year when we could no longer safely bring participants to us. Now remote games user research is an essential part of our toolbox. In this issue we explore how to do it, and some of the things to look out for.

Get future games user research lessons direct to your inbox

How to run games research remotely

Running remote games user research studies

Remote games user research is when participants can take part at home, rather than coming to us. That’s always been the case for large-scale beta tests, which use analytics and surveys to gather feedback. It has traditionally been challenging for earlier stage studies, like a typical usability test. 

There are some advantages from running usability studies remotely. Previously our studies were limited to ‘people who could travel to us in a reasonable timeframe’. Running these remotely allows us to overcome our geographical bias, and recruit people from around the world to take part. Remote research makes the logistics of accessibility testing much easier too – it allows us to find more people with access needs who can take part in studies, and doesn’t put the burden on them to travel to us.

However there have always been some difficulties with running these studies remotely. Many studios are very secretive about their games, and the risk of leaks is increased when we can’t keep an eye on everything that our participants are doing. There are also technical and logistical obstacles that are more difficult remotely. It’s much easier for me to install a prototype of a mobile phone game on my phone and give the phone to a participant, than to get them to install the prototype successfully on their own phone. 

Many of the challenges have got easier as specialist tools have developed – we’ll look at some today.

How to do remote games user research

Live moderated research

Moderated research – where you are able to speak to the participant live and ask questions, is often the best way of gathering the most data possible from your playtest. 

Many teams have been exploring using Parsec to allow remote players to stream access to builds on your own device, similar to how Google Stadia works. This solves some of the technical issues (because the game is running on your own device it doesn’t need to be set up on their computer). It also reduces some opportunities for leaking, so can be preferable to installing the game on the playtester’s computer.  There is a video of how Parsec works on their website, as well as a case study of Ubisoft using it to distribute builds for a marketing event.

If your team are comfortable with participants installing the game on their own device, conference tools like Google Meet allow you to view the player’s screen, and talk to them live. It also has the built-in ability to record the session. Google Meet requires very little technical knowledge from the player, so is simple to set up – and it’s free! 

Unmoderated research

Another alternative to consider is unmoderated research – asking players to play in their own time, and receiving a video recording. This has some downsides – it’s not possible to ask questions in reaction to the behaviour you are seeing in the session. However, it can often be logistically easier to organise.

Some teams use automated services like PlaytestCloud to do this. This allows teams to specify a ‘type’ of player, upload a build, and receive videos of people playing it. This eliminates many of the logistical challenges of distributing games and finding players and makes running unmoderated user research studies easier. 

However not every team has the budget to outsource this (or are not working on a mobile game), and will have to recreate this setup manually. This can be done using tools like Obs or DScout (recommended by @eddiepearson) to capture videos from players, and combine that with a free survey created on Google Forms.

This kind of unmoderated research can allow us to see many more players than we could observe linearly. It can be very powerful for combining qualitative feedback (‘observing what they do’) with quantitative feedback from surveys. 

Non Disclosure Agreements

The risk of leaks is difficult to avoid entirely. Non-disclosure agreements can warn participants and reduce the likelihood of them sharing confidential information. This template from the research agency ping-pong will allow you to create your own NDA, and includes a plan to get it signed remotely. 

Some remote research tips

I asked others on LinkedIn for their tips about remote research. 

Christian Ress from PlaytestCloud gave this great advice:

Some great tips for the study design of remote studies (and good advice to run a pilot. Pilots are always an essential part of my in-person research plan, but easy to forget when working remotely!).

For a different take to addressing the issues from the pandemic, Player Research spoke last year about how they had adapted their lab to be safe for in-person studies, and I know this is something other research teams are looking into. 

Get a free games user research career guide

Get my free ebook of career tips from researchers at top game studios such as EA, Blizzard, PlayStation, Ubisoft, Activision and more. Based on exclusive interviews, they reveal ten essential tips to kick-start your GUR career today.

Plus receive a monthly email with free games user research lessons and curated early-career Games UX and UR jobs.

Sign up now to get the free e-book (no spam, just a nice email from me each month!)

Ten essential tips for new and aspiring games user reseachers ebook

Don’t miss future issues

Thanks for reading this issue. If you’re reading online, and want to get future issues directly to your inbox – sign up here.

Have a great month! 

Categories
Newsletter Research Skills

Quantitative Research for new games user researchers

Let’s start with my confession. I have been a user researcher working with games for over ten years. I have run hundreds of studies, and overseen thousands of hours of playtests (over 25,000 player hours at last count!). And yet, I know very little about stats.

There are two quantitative research things I know how to do. Today, I will explain both.

Get future issues direct to your inbox

Quantitative Research for new Games User Researchers

Comparing two sets of numbers

The first thing is how to compare two sets of numbers. I was taught this by Cyril and Mirweis at PlayStation, and I am grateful to them both for teaching me the only stats I know – how to compare two sets of numbers. This is useful when comparing things such as ‘how many times did the player fail’ or ‘how long did it take people to complete this level’

This method is appropriate for when the data is numerical rather than categorical (or ordinal). Here’s a short explanation of what that means. 

When you have some numerical data, it’s quite common to want to compare it. This allows you to learn “is there a difference between these two things”, and then inspire conversations such as “do we want players to fail more times on this level than on the next one?”.

To do this, you want to find the average, and then work out some confidence intervals to anticipate whether the difference between them is real or whether it was potentially caused by not measuring enough people. 

So, after counting how many times people died on level 3, you can take an average – which looks like this. 

We can see that on average, players died around 2.5 times on level 3. 
We can then do the same thing for the next level.

(This is probably a good moment to mention there is a template that does the maths for you later in this post…)

Looking at the average for Level 4 shows us that people died on average more often on Level 4 than they did on Level 3. 


But we don’t know if this is because Level 4 causes more deaths, or just random chance that it occurred in this study.

To identify that bit, we calculate confidence intervals. Which looks like this…

And we can see that the confidence intervals (the uppey-downy bits) overlap. The top of Level 3 overlaps with the bottom of Level 4.

Level 5’s confidence intervals do not overlap with any of the other levels. If the confidence intervals don’t overlap, there is a real difference between them. It’s true that more people died, and will die, on level 5 than level 4.

This hopefully means that Level 5 harder – although you should watch people play to understand actually why the difference in deaths occurred.

If the confidence intervals don’t overlap, we can’t tell if there is a difference. This is the case for Level 3 + 4. This either means that the number of times people die are the same, or that we haven’t seen enough players to draw an appropriate conclusion.

(There are probably errors in the terminology above, but as I said, I know little about stats – I just know how to compare two sets of numbers).

I use this all of the time – to count and compare deaths, completion time, etc. I made a template that you can duplicate to see the formulas required, and to have a go at doing it yourself.

Go deeper on quantitative research

Beyond this one technique, I’ve found two other tools very helpful.

Adjusted wald calculators like this allow you to state your completion rate (e.g. 3 out of 10 people encountered this issue), and from that anticipate how many people in the real world would encounter the same issue (between 10% and 60% apparently).

And the book ‘Quantifying the User Experience’ which has lots of nice decision maps like these, which tells me what tools I should (and shouldn’t) be using … and includes a crash course in stats to explain how to do them! 

Picture from the book ‘Quantifying the user experience’
(Picture from the book ‘Quantifying the user experience’).

Avoid common quantitative research errors

The second thing I’ve learned is a collection of things not to do. By recognising some stats errors, it helps me know when I should seek out someone better with stats than me to help out. 

Avoiding common errors include:

  • Don’t do the kind of maths I described above on ordinal data (such as likert scales). People often do, and get away with it, but it’s somewhat inaccurate as you’re treating categories like they are numbers.
  • Think about the sampling bias you have created in your study, and don’t over-emphasise how representative your conclusions are
  • Don’t assume that because you are measuring what players say they think or do, you are actually measuring what they think or do.
  • Recognise that when you are limiting the options you allow people to select from, you are limiting the range of results you will get back, potentially distorting the truth.
  • Avoid dogmatic rules about sample sizes. There’s lots of rules out there that have become dogmatic (‘quant studies need 30 users’, ‘qual studies need 5 users’), and many people repeat them without understanding the reason behind them. Understand why those guidelines exist, think about what you are trying to learn, and make conscious decisions rather than following ‘rules’.

The job is not just ‘qualitative research’

I sometimes encounter the idea that user researchers are synonymous with qualitative research. I don’t think that is appropriate or correct. Even if you are more comfortable with qualitative research, you shouldn’t allow your skillset to determine the method you apply for answering research questions. 

Instead always lead with ‘what does the team want to know’, and then ‘what is the most appropriate way of discovering that’. If that method isn’t one you are comfortable with, use it as an opportunity to learn how to do a new thing, ask for help from the community, or bring in some help from someone who is comfortable with it. Our job is to “help the team make evidence-based decisions”, regardless of the methods we are most comfortable with.

What quantitative research skills should I be ready for in the job interview?

If you can answer the following questions, I would say you would be a stand-out candidate…

  • What is p-value?
  • How would you compare the difficulty between two levels? What would you measure, and how should that be interpreted?
  • How would you measure if players are enjoying a game?
  • How would you handle being asked ‘I think this study should have a larger sample size’?

You will notice that these questions are often not about ‘how do I do the stats’, but much more interested in ‘when is quantitative research appropriate, how should it be applied, how should I explain things to my colleagues, and what are the caveats for this kind of work’. Which I think is where the real challenge lies! 

Thanks, and hello…

Thanks for making it to the end, and particularly thank you to all of the new subscribers – we’ve had over 500 people sign up to this newsletter over the last few months. Many of the new readers joined this month, so welcome to all the new people starting their games user research careers.

I’ve written a book about how to be a games user researcher, do take a look if you haven’t already. If you had, I’d really appreciate a review on Amazon – it does have a huge impact on the book, and I value it very much.

As always, do email me or tweet me with feedback, questions, etc and I’ll see everyone next month! 

Don’t miss the next issue

Categories
Career Advice Newsletter

Networking to your first games user research job

Applying to your first games user research job is difficult. Despite putting hours into a CV or cover letter, you often don’t even get an interview, and never hear back. 

This is disheartening, and eventually leads to disillusionment, and giving up on your dreams to work in video games. 

The odds are stacked against you

I once read that 70% of jobs don’t make it to job sites – so most of the available opportunities you’re not even hearing about. 

And 80% of jobs are filled via personal and professional connections – again making it very hard to get your anonymous CV to be read.

This occurs not because hiring managers are bad people, or intentionally trying to give jobs to their friends. They are receiving hundreds of job applications every day, and that quickly becomes unmanageable. There are endless people out there who apply to every games job, regardless of their suitability for the role – and so hiring managers have a lot of noise to get through. 

This makes it very hard to quickly identify ‘this is a suitable person to interview’ vs ‘this is just someone who applies to all games jobs’ (or even ‘this is just someone who applies to all user research jobs’). 

Even very strong candidates will often be mis-identified, and not make it to interviews.

Increase the odds by making hiring managers’ jobs easier. 

Wouldn’t it be good if hiring managers always identified your relevance correctly, understood your expertise, and at least gave you an interview? 

In this article we’re going to look at effective networking. By the end of this article, you will understand how to get recognition in the industry (without awkward handshakes), in order to get more interviews.

Networking into your first games user research job

Networking doesn’t have to be awkward

People assume networking is going to conferences, interrupting a group of hiring managers talking together, and handing out your business cards. So awkward, intrusive and intimidating! 

Especially in an industry where a lot of people identify as introverted (👋) I couldn’t imagine anything worse – and I have often been in the situation where I don’t have anyone to talk to, don’t feel brave enough to approach strangers and so drink endless free coffee and wait…

These days, it is possible to build genuine connections virtually. We’ll look at some techniques for doing that, and some mistakes to avoid.

Getting the basics sorted

You need to create the impression that you want to work in games user research. LinkedIn is the most appropriate for professional contexts, but Twitter is where a lot of professional conversations happen already. You should consider having accounts for both.  

🚀 Do this now: Update or create your LinkedIn and Twitter profile, so that it reflects your interest in games user research

Be professional online

The industry is extremely small, and so always being professional and friendly is very important. This doesn’t mean you necessarily have to be stuffy + business focused in how you communicate online, but do look out for red flag behaviours such as abuse or discrimination.

Especially in shared spaces, like conferences or the GRUX discord – a lot of very experienced industry people are lurking, and unprofessional conduct will be noticed. 

🚀 Do this now: Review your previous interactions on Twitter or LinkedIn, and identify whether your online presence is professionally appropriate. Change how you conduct yourself online if not! 

Join in with existing conversations

Every week, there are discussions on the Games Research + UX Discord, aimed to encourage contributions at all levels.

As well as the GRUX discord, games user research conversations are happening elsewhere online – the GamesUR twitter account does a great job at collating some of them

You can find more people talking about games user research by searching on the hashtags #gamesUR or #GRUX, and join in the conversations. Reply to other people’s threads and ask questions. Interactions grow familiarity, and will help people recognise you as a person interested in the industry. 

If you need a list of people to follow, I put together a quick list of people who tweet about games user research here. (I will add more when the spam limit calms down…)

Don’t hold back on asking questions. I believe that questions reflect well on the person asking them (it’s better to ask than to remain ignorant), and a lot of researchers active on social media have time to explain their points better online!

🚀 Do this now: Join the GRUX discord, and look at what is being talked about. Join in with a conversation

Make genuine connections on the internet

At it’s heart, networking is making friends. Or at least – making acquaintences. Treating making friends as a transaction is transparent and icky. I’ve seen juniors act in this predatory way at conferences, very consciously seeking out the people who work at big companies, and rejecting those who are not. It’s not subtle, and not good, and will set you back….

You’ll need to reframe your goal as ‘making genuine connections with games user researchers on the internet’. That may eventually create new opportunities, or reduce the number of times accidental rejections occur, but those are side benefits rather than our main objective.

The easiest place to start is with your immediate peers – other people looking to start their own career in games user research. The people who were learning GUR when I started are still some of my most trusted and most closest industry friends, who I can (and have) asked to support me at different times in my career. That network will grow with you, and shouldn’t be overlooked.

Connections don’t even have to be about career things. Why not make a smaller community inside the games user research community for those people who are interested in roguelikes? Or cosy games? Or baking?

Making connections, first with your immediate peers, and then with the wider community is a long term investment in your future.

🚀 Do this now: Make a list of the people you identify as your peers, and send one of them a nice message! 

Doing useful things to create an ‘open’

The first approach to talk to a stranger can be very difficult. That’s the kind of awkward interaction that I want to avoid. If you are doing something useful, or involved in an activity together that approach becomes much easier. It gives you a reason to be talking and something to talk about.

People new to the Games Research and UX community assume you have to be on the steering committee to start a new initiative or to help out. This is not the case – the conferences need volunteers, and the community welcomes new initiatives being led by anyone, regardless of experience. I started the mentoring scheme and jobs board, and both of these can always do with more support. As can the other initiatives that people have started, such as the reading club.

Join the discord, see what is going on, and then volunteer to help out or start something new. They don’t just help your CV, but also start to create genuine connections by working together with other researchers. As an added benefit people will start to come to you and open conversations, so you no longer need to make approaches yourself – people will approach you! 

Doing useful things doesn’t have to be a ‘big initiative’. It could just be writing an article about a topic that will interest the community, or something you have tried and how it went – again it gives people a prompt to talk to you, and will lead to discussions. Lots of small efforts build up over time, and start to build a strong network of people you know, and who know you. 

🚀 Do this now: Think about your interests, and consider what initiative you could start, or help with – or what article you could write. Share your thoughts publicly to create the opportunity for people to engage.

Ask genuine questions

I have found the games research community to be very friendly, and willing to give up their time to people interested in understanding the discipline better. This has been evident through events like the office-hours initiative in the GRUX community. If you are not already part of the mentoring scheme, sign up for the opportunity to meet someone in the industry and learn about it. Many researchers respond positively to  LinkedIn messages, to answer questions.

Meet existing researchers, ask them about their role, how they got into the industry and show a genuine interest. It’s often very obvious if you’re just interested in a job, rather than treating every conversation as a learning opportunity. Leave everyone you speak to with a positive impression about you and your enthusiasm for games user research – It’s a very small industry, and so you will bump into the same people again and again.

To stand out even further – show you’ve done your homework. Find something you think will be interesting or relevant to them, and be helpful by sharing it with them. Particularly for researchers who are active on social media, there are a lot of clues about what aspects of research they focus on, or find interesting, and being genuinely helpful is a great way to start a relationship.

Overall, your goal is to not be a burden. People who start by asking me for an hour long meeting know that it’ll take me months to schedule that – whereas emails usually get a much quicker response. Make your ask proportional to how well you know each other! 

This will start to make connections with people outside of the job application process, and can inspire some great ideas about what you can do to get into the industry. This interview with Tyler Sterle shows how he approached these exact challenges as a student hoping to work in games. 

🚀 Do this now: Add me on Twitter, and start a conversation. I like to hear people’s perspectives on “what are the hardest bits of the journey to becoming a games user researcher”, if a prompt is helpful!

Networking reduces the chance that you’ll get missed

You won’t get your first job by networking. But by making genuine connections with other members of the industry, you will increase the chance that your application will be recognised and not accidentally rejected too early in the application process.

That’ll give you the chance to get an interview, and impress them with your deep research experience and understanding. 

Starting networking as an ‘in’ for any specific role is much too late – you need to start making friends in the industry well before any roles are advertised. But that’s ok. As you’re really making friends, and the benefits of making friends go well beyond just applying for jobs. 

What skills do you need to show with your interview? I asked a bunch of people who work in the industry already for their career tips. If you haven’t read this already, sign up to get it.👇

More career advice for getting started in games user research

Get my free ebook of career tips from researchers at top game studios such as EA, Blizzard, PlayStation, Ubisoft, Activision and more. Based on exclusive interviews, they reveal ten essential tips to kick-start your GUR career today.

Plus receive a monthly email with free games user research lessons and curated early-career Games UX and UR jobs.

Sign up now to get the free e-book (no spam, just a nice email from me each month!)

Ten essential tips for new and aspiring games user reseachers ebook
Categories
Career Advice Newsletter

Moving from academia into the games industry

Games user research and academia

Moving into the games industry from academia

It’s common that user researchers have academic experience. In this years games user research salary survey, 70% of people who took part had a postgraduate degree. 

Planning reliable playtest studies requires the application of scientific methods. A masters degree or PhD is one way of demonstrating that you are qualified to apply research methods. It is not the only way – plenty of great researchers don’t have academic experience. However postgraduate experience is an indication that you know how to pick an appropriate method, reduce bias and draw reliable conclusions.

I wanted to explore how to transition from academia to industry in this issue. My own experience was very fortunate. My postgraduate course gave me the opportunity to work on projects with real game studios which, supplemented by personal projects, helped me get my first job in games. 

Get future issues direct to your inbox

From my own personal experience, using academic study to get some experience in running in-person research is one of the biggest potential benefits of postgraduate study. We covered this in more depth in the HTBAGUR book. Jessica Tompkins, PhD and UX researcher at EA, help on this chapter was invaluable.

I also asked this question on twitter and LinkedIn, and Jessica gave some additional great points on LinkedIn:

Jess's comment about games user research. Linked above

Irena Pavlovic also raised the importance of pragmatism when performing research in industry.

Irena's comment about games user research. Linked above

On twitter, Natali, working as a user researcher at Ubisoft, raised the difference in pace between academia and industry, and that one of the challenges is adapting research methods, analysis and debriefing so that it finds answers when it’s still relevant. 

Natali's tweet about games user research - linked below

Link to Natali’s original tweet

Blake is a UX Researcher at XBox. He also highlighted how the change of environment impacts the goal of quantitative studies, and what we would consider an appropriate conclusion.

Blakes Tweet about Games User Research - linked below

Link to Blake’s original tweet  

Lots of great tips there from the community, and the need for speed comes up multiple times. In industry a study that provides findings too late is pointless, as the decision will have been made. The development team have already moved on to the next topic.

One way of speeding up analysis is through preparing templates for note-taking and analysis, and using mindmaps as a way to speed up analysis, as I covered in the Games User Research Summer Camp event last year.  We also explore this, and other aspects of moving from academia to industry in the book – but it’s a popular topic, so I know we’ll cover different aspects of the transition in more depth in the future.

See also my introduction to games user research for academics for more on this topic.

Get my free ebook of career tips from researchers at top game studios such as EA, Blizzard, PlayStation, Ubisoft, Activision and more. Based on exclusive interviews, they reveal ten essential tips to kick-start your GUR career today.

Plus receive a monthly email with free games user research lessons and curated early-career Games UX and UR jobs.

Sign up now to get the free e-book (no spam, just a nice email from me each month!)

Ten essential tips for new and aspiring games user reseachers ebook

Sign up to get more games user research career lessons

Categories
Career Advice Newsletter

How to convert user research experience into a games user research job

Transitioning into games from other user research sectors

This month, I wanted to go a bit deeper into how to take user research experience gained in other industries, and translate it into a games user research job. 

This is a reasonably common route that people use to join the industry, and one of the pieces of advice I give to people who are having difficulty joining the industry at a junior level is to look for a broader user research role. This helps develop very relevant experience you can bring into a games job application. 

This month’s interview is with Surabhi Mathur, a user researcher at EA who made the jump into games last year, who has some great advice based on her own experience. Before that though, let’s consider some of the differences between games user research and other sectors.

Get future issues direct into your mailbox:

Expect friction

In most other types of software, ‘difficulty’ is to be avoided. No-one wants it to be hard to order a book from an online shop. However for games, a game without difficulty isn’t (usually) fun. Work like Raph Koster’s A Theory Of Fun For Game Design and Mihaly Csikszentmihalyi’s Flow both put ‘overcoming challenges that were previously unachievable’ as a key part of creating compelling experiences. Games need friction.

Games user researchers need to be nuanced and put in more upfront work to understand the thing being tested. They need to understand the game well enough to know what difficulty the designer intended. This usually involves a lot of time working with designers to identify what the difficulty is meant to be and what players are meant to understand or be able to do before they should be allowed to progress. They need to moderate the session and capture notes in a way that recognises ‘intended’ challenge (that has been designed) and separates this from ‘unintended’ challenge (that is accidental). 

This also means that some of the typical things a user research team working outside of games (like ‘time to complete’), are not appropriate for games. These measures are not able to separate intended and unintended challenges.

Methodological constraints

Many games have big marketing budgets, and excited fans who care deeply about the games being made. This puts them at high risk of leaks, and the consequences of leaks can be very expensive for the studio. So, secrecy is extremely important.

This includes asking participants to sign non-disclosure agreements and finding ways of actively preventing recording, such as removing their phones.

This need for secrecy has a big impact on the methods available to us. For extremely secretive games, all research has to be lab-based. This is particularly difficult for quant research.  Other types of software can be distributed to users to use at their home, with unmoderated tools, surveys and analytics to capture their behaviour. With games, the huge risk of leaks means we can’t allow players to participate at home, and we need to bring them into a research lab. This leads to the need to build labs which can hold 10-40 players playing simultaneously

For a researcher coming from other industries, this requires them to adapt their study designs so that it works as a mass playtest and adapt their moderation so that they can control a room of many players simultaneously.

Research Objectives

Games are entertainment. Often your game team want to know some variation of ‘is this fun?’.

For clear usability issues, this is fine – we can frame usability problems as ‘barriers to fun’. But it also means working out a reliable method of measuring enjoyment or fun. One approach is using a combination of quant measures, benchmarking and qualitative data to explain any quant scores. Think about the challenges of measuring fun, and how to explain the limitations of measuring fun to a game team who will want to equate it with a review score. This will be excellent preparation for a researcher joining the games industry.

Converting your experience

If you are applying from another user research role, your experience of planning and running studies is extremely valuable. Strongly emphasise this in your application. Supplementing your experience with an understanding of how games are different to other software design projects, will make you a very strong candidate. 

Tom Lorusso is a Research Manager at Xbox, and also had some advice:

(Tom’s original tweet)

I think Tom’s point, about the implications of games being art, is interesting. There is not only the impact on ‘how much people care’ about what they are making. It also affects how user researchers describe their work. 

In other fields, it’s common to talk about how user research uncovers and fulfills the needs of it’s users. In games, we often describe that our role is to ensure the game design vision is being experienced correctly. We frame this around the vision of the designer, not the needs of the user. (At least, that’s what one of the posters says!).

I personally believe that this framing will change over time, as the maturity of research in the game design process grows. That’s a subject for another time. 

As Tom says, it’s definitely a high passion environment, which creates different dynamics to working in many corporate environments. 

How I got started in Games User Research

Surabhi Mathur is an Associate UX Researcher with EA Sports, working on FIFA. In this exclusive interview, I asked about her experience moving from user research at Uber, and the challenges of transitioning into the games industry

What was your journey into games user research?

As a UXR I look for opportunities that will help me grow, expand my skills and pose exciting challenges. A job posting by EA was one such opportunity that I took up. 

When I switched from tech to games user research, I didn’t expect GUR to be so different and distinct. Tech UXR has a big, cohesive community which allows a general understanding of how UXR functions in different businesses. Since GUR is largely absent from mainstream UXR conversations, I was oblivious to its uniqueness. 

Even though my journey began subconsciously, I quickly realised that GUR has its own tightnit network of highly qualified and passionate individuals. As soon as I joined( and even before), I started leveraging the information shared by this community and it helped rapidly ramp up. 

What did you find the most challenging step of getting a role working in games? 

I faced two key challenges when trying to enter games industry:

  1. Since games/GUR is unique in the way it functions, the industry prefers people with prior experience in the field. Experience in playing/interacting with games is the next best thing one can bank upon. Since I had neither, I started interacting with different games to get familiar with game design. I also read as much literature as possible on GUR, Game Design etc which helped me understand how I could use my previous experience to draw parallels with the GUR process. 
  2. GUR is more inclined towards quantitative research than tech UXR (subjective experience) and basic quant skills are tested during the interview process. To prepare for this, refreshed all my quant skills before the interview and took a few classes to learn basic statistical languages (R, Python).

Did your experience working in UX elsewhere help with getting a job in the games industry? 

Definitely! Working in Tech made me a versatile UXR with the ability to conduct research in any given circumstances, methods, timelines, stakeholders etc. These skills helped me during my interviews because I was able to think on my feet and be creative with my approach.

What would your top tip be for someone looking to become a games user researcher?

  1. Get in touch with the GUR community and leverage its resources to get started.
  2. Play the games you want to work on or of similar genre: you don’t have to play the whole thing but do experience it’s gameplay. 
  3. Focus on your quant skills. 

Fantastic advice from Surabhi. I personally found the book ‘Quantifying the User Experience’ a really helpful introduction into learning enough stats to get by in most user research roles, if this is a topic you don’t feel comfortable with.


If you’ve enjoyed this newsletter, there are two very helpful things you could do for me:

  1. Share it with people you think would find it helpful
  2. Leave a review for the book How To Be A Games User Researcher on Amazon.

I’d be extremely grateful if you decide to do either of these, as both are very helpful!

Steve (@steve_bromley)

Get the latest information on how to become a games user researcher direct to your inbox

Categories
Newsletter

Why do games user researchers start by defining research objectives?

Welcome to the second issue of the How To Be A Games User Researcher – and thanks for everyone’s kind words about issue one

  • Why is defining research objectives so important?
  • Represent games user research on Zoom
  • Elizabeth Zelle’s journey into games user research 
  • What interview questions should you be prepared for?
  • An entry level role with Warner Brothers Seattle

…and more! 

Get future issues direct into your mailbox

Every month we’ll be tackling topics important to people looking to develop their career in games, and run better playtests

Read on to learn how to become a games user researcher.

Games User Research Tips

Why do we start by defining objectives?

User research studies, or playtests, can give reliable answers to many useful game development decisions. They are strongest with questions around behaviour or understanding such as “what do we need to tell players”, “will players understand what they are meant to go” or “can players complete this challenge”. The right playtest can also answer harder questions, like “Do players like my game”, but this requires more skill to reliably answer.

The simplest playtests involve watching people play, and asking them questions. This reveals issues where they are not experiencing the game as intended, and help us identify what needs to be changed to improve their experience. 

However in order to design appropriate tasks, or pay attention to the right parts, we need to know what we want to learn from the study. Researchers call these the ‘objectives’ of the study.

Coming up with objectives is best done regularly throughout development by understanding what the team have been working on recently, and working with them to describe how they think players should experience it – what do we want players to understand, think and do. You can then use this information to decide what you need to measure, to check if they are experiencing it as intended.

I wrote more about how to define research objectives in the book, and as a free extract on the website. Read more here to learn how to come up with research objectives for your study.

Represent - Fight For Better Games

Educate people about games user research

Last month, I shared the posters and stickers created for the book by Chloe True which promote some key games user research principles.

Jess Tompkins (who also helped with the book!), suggested making them into Zoom backgrounds. So we did! 

Jess Tompkin's Tweet - Research Makes Games Better

Link to Jess’s tweet

Hannah Murphy's Tweet- Game Design Is Revealed By User Research

Link to Hannah’s original tweet

The zoom backgrounds are free to download from the website, alongside free printable posters to advocate for user research.

Get the free zoom backgrounds here

Do share your pictures of the backgrounds if you pick them up! 

Competition - Win Free Games

Congratulations to last month’s winners.

Last month we gave away two copies of ‘The GoD Unit’. Congratulations to Alex and Jel who each won a copy! 

It’s too late to enter, but if you’re feeling helpful I’d still love to hear about you on the survey link, so I can continue to tailor the newsletter contents

We’ll have a new competition soon! 

Career - Get Your First Job In Games User Research

How I got started in Games User Research

In the GRUX Discord, their regular QOTW feature asked last year how people got started in games user research. There were many great answers, including from Elizabeth Zelle, User Research Manager at Amazon Games. I caught up with Elizabeth – here’s her journey and a top tip for other people following in her footsteps! 

Elizabeth Zelle – User Research Manager at Amazon Games

Follow Elizabeth on Twitter

What was your journey into Games User Research?

In 2008 I applied to work in the QA department of Volition. At the time I had a number of friends from high school already working in QA there, and they encouraged me to apply. It was the middle of the recession and jobs in marine biology (what my actual degree was in) were few and far between, so there was nothing to lose. I got hired fairly easily, but I will admit that networking and name dropping my mom (who had worked at Volition years before) totally played a role there.

I was laid off a year later, but was in one of the first rounds of rehiring in 2010 when they started staffing up again. I was so high up the rehire list because I’d proven my skills, both in doing QA and in being a good team member, and people in the department specifically wanted me back, which felt good. 

I transitioned to User Research a little while later when I discovered it was a great fit for my skillsets and matched well with my interests. The only problem was that we didn’t have a researcher opening. So I approached the solo researcher at the company and told him that it looked like he had enough work that he needed an assistant. Luckily for me, he agreed!

My takeaway: Getting that first job was easy but also a lot of luck; getting rehired after the layoff was based on skill. Being detail-oriented, having strong note-taking skills, and being highly observant all were traits that helped me excel in QA and then transition over to UR (not surprising, since those are all skills that research work in college had hammered into me). 

My advice: Self-assess and identify which skills are your strengths, and then own them. Seek out the opportunities that you want and shoot your shot; don’t count yourself out before trying. Make friends; being someone that people value and want on their team will always benefit you.

What interview questions should you be prepared for?

Members of the GUR community discussed questions that could come up in games user research interviews. 

Callum Deery (@CallumDeery2 on twitter) shared some practice questions he’d gathered for UX applications.

Seb Long (@seb_long on twitter)  spoke to hiring managers about their interview questions for Games User Research jobs back in 2015.

Juney Dijkstra (find Juney on LinkedIn) shared some links she’d collated including:

Although not games specific, I’ve previously found this medium article helpful to think about the range of technical and situational questions a user researcher may be asked.

All of these links were originally shared on the #job_hunting_chat on the GRUX-DIG discord. Do join the community and continue the conversation! 

An entry level user research role

Warner Brothers are looking for two user research assistants for their Seattle team to help conduct research studies, including moderating studies and observing player behaviour. The role includes training but is ideal for people who have had previous experience running research studies with people, potentially gathered through a university degree.

Find more details, and apply here.

Help the Games User Research conference

The GUR conference is looking for volunteers for their 2021 summit. Find more details and sign up here.

Helping at the conferences is a great first step for getting involved in the community, so I’d recommend taking a look!

You made it!

British Pub - You Are Not Your Players

That’s it. Thanks for reading. For me personally, I’m continuing on the book tour (I’m popping up on many podcasts), and have some exciting game projects coming up in April so I know it’s going to be a good one! 

I’d love to chat about games user research, and what would be helpful for you in this newsletter – do email me by replying to this newsletter if you have any questions, or drop me a line on twitter!

If you’ve enjoyed this newsletter, there are two very helpful things you could do for me:

  1. Share it with people you think would find it helpful (there’s a share button just below)
  2. Leave a review for the book How To Be A Games User Researcher on Amazon.

I’d be extremely grateful if you decide to do either of these, as both are very helpful!

Have a great April! 

Steve (@steve_bromley)

Found this issue helpful? Sign up to receive the next issue here:

Categories
Newsletter

Test early and often

I will start with a thank you. The response to the launch of the book has been overwhelming. I’ve loved seeing everyone’s photos shared on Twitter and LinkedIn – and am really happy everyone’s finding it useful. There’s plenty more to come, so stay tuned. 

Let’s get stuck in! 

Get future issues direct into your mailbox

Every month we’ll be tackling topics important to people looking to develop their career in games, and run better playtests


Why do games user researchers always talk about ‘testing early and often’?

“It’s not yet ready to test” is a phrase I hear frequently from game teams. It is also a phrase I shouldn’t be hearing from teams who understand the value of playtesting. 

It’s very common to wait to playtest. Teams assume it’s only useful near the end of development. Sometimes developers know that bugs exist, which they believe will stop players from progressing. Other times developers hold back because the art isn’t finished yet, and they are worried people will think the game is ugly.

Although the game isn’t polished, testable bits of the core experience exist very early in development. And this creates a risk – the longer these go untested, the more ‘baked in’ they become, and the harder it becomes to change them. When problems are only discovered late in development, it becomes very expensive to change them. A lot of work needs to be thrown away and redone.

The safest approach is to expose the core mechanics to players early, from the first prototype. Watch people play, and ask questions to check if players are understanding the mechanics in the way you hoped. Check that they are able to use them as expected. You might even see interesting behaviour that can inspire new gameplay ideas.

I loved this Twitter thread from Katie Chironis, giving a concrete example of how this could work for Dark Souls, and how it’s possible to create and evaluate the core experience very early on.  

Moderated usability testing makes this easy

By sitting with players as they play, or streaming their session live, you can help them work around bugs, and ask them questions about the things you do care about such as their understanding of what’s happening – avoiding irrelevant comments about the unfinished assets. Let’s make sure our colleagues understand they can come to us from the very start of development – more on how to do that coming soon! 

How to run better usability reviews

Last year Seb Long, of Player Research fame, ran an exercise where he reviewed a whole bunch of usability ‘expert’ reviews from people at the start of their career and noticed some patterns in the errors. This featured at the games user research summit in 2019.

There is a huge wealth of valuable information here about the mistakes people make when running usability reviews, packed into 60 minutes. Do take a look! 


How I got started in Games UX

In the GRUX Discord, the regular QOTW feature asked how people got started in games user research. There were many great answers, including from Emma Varjo, UX Lead at FrozenByte games. I caught up with Emma – here’s her journey and a top tip for other people following in her footsteps! 

Emma Varjo – UX Lead at FrozenByte

Follow Emma on Twitter and LinkedIn

What was your journey to games user research and UX?

I didn’t know what I wanted to do after high school. I figured I like math, languages, and videogames, so I should study programming. I’d either end up working on games (dream) or making a good living (also acceptable), as long as it didn’t turn out I hated it.

I nearly quit during my first semester, but once I got it, I felt I could do anything the program threw at me. I did struggle with all new programs we needed to learn in order to do the work and felt it was dumb it was so hard to just get stuff done. Others told me it’s normal that you just have to learn the tools. I felt so happy when I discovered HCI and the people agreed with me! 

I tried to get internships in games studios, but thanks to a general economic downturn practically all studios in my city shut down and I could not afford to move cities for the summer and pay rents for two apartments so I could do an unpaid internship in games. Besides I could get paid nicely interning in other tech companies. 

Years later I was working in tech as a programmer and looked at games jobs again as there were more of them in the capital. Most were unpaid internships unless you had games experience. One AAA studio looked for a GUR person with “5+ years of experience in UR and at least one shipped game.” I did not apply, but in hindsight I really should’ve. 

Frozenbyte had all applicants do a task they’d have to submit with the application. For game design it was an expert evaluation and a user test of a demo of their game. Right up my alley! They replied months later saying it was one of the best applications they ever got, but couldn’t hire anyone at the time, but I should try again in the future. I took another job and forgot about it. A year later I was burning out in my new job and FB emailed me and asked if I was still interested. Heck yeah I was and here I am!

What’s your top tip for people looking to get started in games user research?

Empathy and communication skills are paramount. 

I was remembered a year after the initial conversation not only because I did a good evaluation and user test on their demo, but because I empathized with the designers reading my application and wrote my report on a way that it was easy for them to find where the issues were, how severe the issues were, as well as some ways to possibly fix them. (I was applying for a UXD position, so it was important to have – for our GUR that can be dangerous.) 

When you’re breaking in, and ever after too, it’s important to use your skills not only to do great research but also affect change based on it. 


Don’t miss an issue

Every month we’ll be tackling topics important to people looking to develop their career in games, and run better playtests