When I mentioned I was going to be talking about playtest survey design, I reached out to the research community to hear their tips and best practice.
I received such a lot of feedback and excellent tips, I’ve had to spin it into its own page. Here’s what we heard from the community
This list of tips was featured in our lesson on How To Design Surveys. Read the original post here.
Always pilot your surveys
Read the original message from Jack Dunne
Read the original message from Katherine
Read the original message from Ashley
Lots of great points from Katherine, Ashley and Jack, but I think in particular the point about piloting the survey is very important. Treat it as a qualitative test where you ask someone to fill it out, and ask them what they think each question means – this can reveal significant misunderstandings between what you think you’re asking, and what you are actually asking. That’s really important to capture before you send it out to potentially thousands of people.
Keep your playtest survey brief
Read Tom and Eddie’s original tweets.
Most surveys are too long. Participant’s willingness to fill out a survey will drop rapidly after five minutes, and pushing past ten minutes introduces bias in the type of people who are completing your survey. This decreases the reliability of your results.
Start by agreeing with stakeholders ‘if we are going to learn one thing from this survey, what would it be’. Then use this to be brutal when eliminating other questions. Stakeholders will push to add more and more questions – your professional duty is to stand up to that, and ensure that you are getting results that we can trust.
Learn from survey design best practice
Read the MRS Survey Design Guide Gareth Lloyd recommended.
Read Gareth’s original tweets.
Look out for gaps in your questions
See Ashley and John’s original tweets
It’s easy to forget how different we are from our players, and how the language we use (and the things we want them to talk about) might not be the same as what players want to express, or might
It’s really common to make simple errors, like forgetting to give a ‘never’ option, or forcing players to comment on something they have just told us they never do.
Piloting our surveys, and using broad final questions can help you diagnose problems in your questionnaire, and catch questions that you should have asked, but missed.
Make sure a survey is the right method
Read Elizabeth’s original tweets
Surveys can be messy and straddle qualitative and quantitative feedback. This requires care and attention, to avoid getting into trouble.
Elizabeth Zelle gave a fantastic talk for the Games User Research conference in 2019 about survey design – I highly recommend you watch the full thing here!
One idea per question
AlizĂ©e describes how we need to be careful not to overload our questions, and make sure that we ask one thing per question – this avoids us getting muddy results that don’t represent reality because we haven’t given participants the ability to express their true sentiment or behaviour.
Pay attention to the playtest survey respondant experience
Read Uncharted Horizon Entertainment’s original message.
Court and the Uncharted Horizon’s team’s points bring home the value of thinking how the participant will experience your survey when designing it. Paying careful attention to the words used, the survey order, and being respectful of their time is really important for getting honest + reliable results.
Support mobile
Daniel makes excellent points, thinking about the context in which people will be filling in your survey. If they are going to be doing it on a mobile phone, make sure your surveys usable on a small screen (and remember to pilot this!)
Writing surveys looks easy, but it’s easy to make errors that invalidate your results. Thanks to everyone who shared their knowledge in this post.
What are your tips for writing a good survey? Tweet me at @steve_bromley and I’ll share them!
Ready to finally start your games user research career?
Every month, get sent the latest articles on how to start a career in game development, and find games user research jobs.
Plus get two free e-books of career guidance from top games companies
