Categories
Tips

Expert tips for writing a playtest survey

When I mentioned I was going to be talking about playtest survey design, I reached out to the research community to hear their tips and best practice. 

I received such a lot of feedback and excellent tips, I’ve had to spin it into its own page. Here’s what we heard from the community

This list of tips was featured in our lesson on How To Design Surveys. Read the original post here.

Always pilot your surveys

Great topic and one that I needed a lot of help with when I started out building screeners!

Big one is to test your surveys, it’s always good to have colleagues or a small sample go through before you send it to the masses.

Another one which I find really useful is to have your hypothesis in a different tab or ideally in view while you’re writing out the questions. That way you can make sure you’re not veering off track while you’re making the questions

Read the original message from Jack Dunne

Oh I have so many! I'd probably start with challenging any misconceptions you might have that writing a survey is easy or quick...! It should take time and effort to plan and structure...but should be easy and quick to complete!

Ensure the questions you ask will actually give you the answers you need. Make sure there is no ambiguity in what you are asking (to the earlier point of testing, different interpretation of question wording can lead to getting a different response from what you are looking for.)

Read the original message from Katherine

Read the original message from Ashley

Lots of great points from Katherine, Ashley and Jack, but I think in particular the point about piloting the survey is very important. Treat it as a qualitative test where you ask someone to fill it out, and ask them what they think each question means – this can reveal significant misunderstandings between what you think you’re asking, and what you are actually asking. That’s really important to capture before you send it out to potentially thousands of people.

Keep your playtest survey brief

Read Tom and Eddie’s original tweets.

Most surveys are too long. Participant’s willingness to fill out a survey will drop rapidly after five minutes, and pushing past ten minutes introduces bias in the type of people who are completing your survey. This decreases the reliability of your results.

Start by agreeing with stakeholders ‘if we are going to learn one thing from this survey, what would it be’. Then use this to be brutal when eliminating other questions. Stakeholders will push to add more and more questions – your professional duty is to stand up to that, and ensure that you are getting results that we can trust.

Learn from survey design best practice

Read the MRS Survey Design Guide Gareth Lloyd recommended.

Read Gareth’s original tweets.

Look out for gaps in your questions

Read Gareth’s original tweet

See Ashley and John’s original tweets

Read John’s original tweets

It’s easy to forget how different we are from our players, and how the language we use (and the things we want them to talk about) might not be the same as what players want to express, or might

It’s really common to make simple errors, like forgetting to give a ‘never’ option, or forcing players to comment on something they have just told us they never do.

Piloting our surveys, and using broad final questions can help you diagnose problems in your questionnaire, and catch questions that you should have asked, but missed.

Make sure a survey is the right method

Read Elizabeth’s original tweets

Surveys can be messy and straddle qualitative and quantitative feedback. This requires care and attention, to avoid getting into trouble.

Elizabeth Zelle gave a fantastic talk for the Games User Research conference in 2019 about survey design – I highly recommend you watch the full thing here!

One idea per question

Read AlizĂ©e’s original tweet

AlizĂ©e describes how we need to be careful not to overload our questions, and make sure that we ask one thing per question – this avoids us getting muddy results that don’t represent reality because we haven’t given participants the ability to express their true sentiment or behaviour.

Pay attention to the playtest survey respondant experience

Read Court’s original tweet

Read Uncharted Horizon Entertainment’s original message.

Court and the Uncharted Horizon’s team’s points bring home the value of thinking how the participant will experience your survey when designing it. Paying careful attention to the words used, the survey order, and being respectful of their time is really important for getting honest + reliable results.

Support mobile

Read Daniel’s original tweet

Daniel makes excellent points, thinking about the context in which people will be filling in your survey. If they are going to be doing it on a mobile phone, make sure your surveys usable on a small screen (and remember to pilot this!)

Writing surveys looks easy, but it’s easy to make errors that invalidate your results. Thanks to everyone who shared their knowledge in this post.

What are your tips for writing a good survey? Tweet me at @steve_bromley and I’ll share them!

Get monthly games user research lessons direct to your inbox

One reply on “Expert tips for writing a playtest survey”

Leave a Reply

Your email address will not be published.