Expert tips for writing a playtest survey

Learn secrets from experts about how to write effective and useful playtest surveys.

Last updated:

When I mentioned I was going to be talking about playtest survey design, I reached out to the research community to hear their tips and best practice. 

I received such a lot of feedback and excellent tips, I’ve had to spin it into its own page. Here’s what we heard from the community

This list of tips was featured in our lesson on How To Design Surveys. Read the original post here.

Always pilot your surveys

Great topic and one that I needed a lot of help with when I started out building screeners!

Big one is to test your surveys, it’s always good to have colleagues or a small sample go through before you send it to the masses.

Another one which I find really useful is to have your hypothesis in a different tab or ideally in view while you’re writing out the questions. That way you can make sure you’re not veering off track while you’re making the questions

Read the original message from Jack Dunne

Oh I have so many! I'd probably start with challenging any misconceptions you might have that writing a survey is easy or quick...! It should take time and effort to plan and structure...but should be easy and quick to complete!

Ensure the questions you ask will actually give you the answers you need. Make sure there is no ambiguity in what you are asking (to the earlier point of testing, different interpretation of question wording can lead to getting a different response from what you are looking for.)

Read the original message from Katherine

Read the original message from Ashley

Lots of great points from Katherine, Ashley and Jack, but I think in particular the point about piloting the survey is very important. Treat it as a qualitative test where you ask someone to fill it out, and ask them what they think each question means – this can reveal significant misunderstandings between what you think you’re asking, and what you are actually asking. That’s really important to capture before you send it out to potentially thousands of people.

Keep your playtest survey brief

Read Tom and Eddie’s original tweets.

Most surveys are too long. Participant’s willingness to fill out a survey will drop rapidly after five minutes, and pushing past ten minutes introduces bias in the type of people who are completing your survey. This decreases the reliability of your results.

Start by agreeing with stakeholders ‘if we are going to learn one thing from this survey, what would it be’. Then use this to be brutal when eliminating other questions. Stakeholders will push to add more and more questions – your professional duty is to stand up to that, and ensure that you are getting results that we can trust.

Learn from survey design best practice

Read the MRS Survey Design Guide Gareth Lloyd recommended.

Read Gareth’s original tweets.

Look out for gaps in your questions

Read Gareth’s original tweet

See Ashley and John’s original tweets

Read John’s original tweets

It’s easy to forget how different we are from our players, and how the language we use (and the things we want them to talk about) might not be the same as what players want to express, or might

It’s really common to make simple errors, like forgetting to give a ‘never’ option, or forcing players to comment on something they have just told us they never do.

Piloting our surveys, and using broad final questions can help you diagnose problems in your questionnaire, and catch questions that you should have asked, but missed.

Make sure a survey is the right method

Read Elizabeth’s original tweets

Surveys can be messy and straddle qualitative and quantitative feedback. This requires care and attention, to avoid getting into trouble.

Elizabeth Zelle gave a fantastic talk for the Games User Research conference in 2019 about survey design – I highly recommend you watch the full thing here!

One idea per question

Read AlizĂ©e’s original tweet

AlizĂ©e describes how we need to be careful not to overload our questions, and make sure that we ask one thing per question – this avoids us getting muddy results that don’t represent reality because we haven’t given participants the ability to express their true sentiment or behaviour.

Pay attention to the playtest survey respondant experience

Read Court’s original tweet

Read Uncharted Horizon Entertainment’s original message.

Court and the Uncharted Horizon’s team’s points bring home the value of thinking how the participant will experience your survey when designing it. Paying careful attention to the words used, the survey order, and being respectful of their time is really important for getting honest + reliable results.

Support mobile

Read Daniel’s original tweet

Daniel makes excellent points, thinking about the context in which people will be filling in your survey. If they are going to be doing it on a mobile phone, make sure your surveys usable on a small screen (and remember to pilot this!)

Writing surveys looks easy, but it’s easy to make errors that invalidate your results. Thanks to everyone who shared their knowledge in this post.

What are your tips for writing a good survey? Tweet me at @steve_bromley and I’ll share them!

Ready to finally start your games user research career?

Every month, get sent the latest articles on how to start a career in game development, and find games user research jobs.

Plus get two free e-books of career guidance from top games companies

Free Games User Research Ebooks
Author image

Meet the author

Steve Bromley is an expert user researcher, who works with studios of all sizes to run playtests, and integrate user research into the game development process.

Learn more

One comment

Comments are closed.

Keep Exploring

Scope Studies For Impact

Scope your studies to increase research impact

How to scope games user research studies for impact – uncovering the context for decision making, and setting sensible objectives.

budget for games user research

How to budget for games user research

How much does user research cost, what to budget for user research, and how best to spend the budget you have to de-risk game development.

Master Games User Research

Free monthly new articles teaching playtesting & how to be a games user researcher. Join the most interesting conversations about games user research, discover job opportunities, and be introduced to new ways to think about game development.

Which best describes you?(Required)