Techniques for avoiding leaks, and keeping games secret when running playtests and UX research.

We dont want leaks from our playtests

When speaking to teams, I often see that they are worried about leaks. That fear can put them off the idea of playtesting.

With small teams and solo developers there is the worry that playtesting their game will give away their ideas, and someone might steal them. From the larger teams, there is the worry that leaks will ruin their marketing plans. 

Many games spend as much on marketing as they do on production (because marketing is such an important part of making a successful game). This means that leaks can be seriously disruptive to established marketing plans, and there is a lot of wariness around playtests. Compared to other industries games are particularly at risk of leaks, because players are very hungry for information, and there are very active player communities to share leaked information. Even the smallest game teams need to be conscious of what assets they are putting online, and who might see them.

This means that game teams, and senior management can be very hesitant about running playtests. 

This fear is well founded. From the thousands of hours of playtests I’ve run, I’ve had leaks occur twice, and neither was good. Even though they didn’t severely disrupt the game’s marketing or announcement they were embarrassing and had the potential to impact my relationship with the teams I worked with. With a bad enough leak, it could potentially make teams not want to playtest again.

However the risk of leaks can’t stop us from running playtests. The value of playtesting is too high, and needs to occur as early as possible in game development in order for us to have the right impact and fix problems at the right time. So, we need tactics to minimise leaks. 

In this post, we’ll look at techniques available to researchers, or game teams of all sizes, to minimise the risk of leaks from playtests. 

How to keep your games secret

Start with the basics: Get participants to sign an NDA

A non-disclosure agreement (NDA) is a legal agreement to keep something secret. It’s typical to sign it when working on games projects (which makes portfolios difficult!), and equally typical to ask playtesters to sign them at the start of playtests. 

Templates for NDAs exist across the internet. Here’s a UK focused NDA template from UKIE (thanks Seb Long for the link). Most non-hobbyist development teams already have a friendly lawyer, who can put together a bespoke NDA for you. I’ve also included a link to several alternative NDAs in the Playtest Kit.

The risk with relying on an NDA is that they are very difficult to enforce. If a participant breaks the NDA and puts some screenshots or secrets on the internet, it costs money to get a lawyer to act on it, and might draw more attention to the leak. Plus at that point, it’s often too late – the leaked information is on the internet, and will be spreading immediately. 

Because enforcing them can be counter-productive, the point of NDAs is often more about the threat of taking action rather than a real intention to enforce it.

Best practice is to ask your participant to sign the non-disclosure agreement at the beginning of the session, and then let them take a copy with them. Remind them again that they signed an NDA at the end of the playtest. Making the process seem formal and official will hopefully increase players’ likelihood to take it seriously. 

Choose in-person research methods for the most secret games

Different research methods also have different levels of safety for leaks. At PlayStation, the vast majority of our research was in our research lab, so that we can keep an eye on players and prevent them from taking screenshots or information away from the session.  

Taking participants’ phones away (and asking them to put them in a locker) will also help prevent photos from being taken.

This also allows you to take time to explain and emphasise the NDA – running through it with the players to make sure they understand and making explicit that it is a legal document. This will increase adherence.

Make remote research as safe as possible

Lab-based, moderated research is the ‘safest’ method to prevent leaks, but even when running remote research, there are some steps that can help. Choosing to moderate the study, where the participant is talking with you live, will create fewer opportunities to capture leakable information.

Bob Tilford, now of RockStar, gave a great talk about how building rapport can help minimise leaks, which can be viewed here:

For unmoderated research, there is an increased risk that players are left with the game, and can screenshot it. It also prevents you building rapport with the participants, increasing the risk.

For many games, it may be sensible to stream the game to players, rather than sending them a build directly. Although not feasible for games which require extreme responsiveness, for slower games this means you don’t have to actually let players have un-monitored access to the game, and can be in control of when and where they can access the game.

Parsec is one of these tools that allow remote players to stream access to builds on your own device, similar to how Google Stadia works. There is a video of how Parsec works on their website, as well as a case study of Ubisoft using it to distribute builds for a marketing event.

Watermark the build

Watermarking is putting an overlay on the screen, with some identifying information, like in this picture:

This can be invisible to the player (see this thread on how it is done inside World Of Warcraft) but for our purposes, we want players to know that leaked information can be tracked back to them. Some sensible information to include can be the current time, date, and any identifiable information that can link the screenshot to a specific device. 

Showing that screenshots can be linked back to an individual increases the perceived risk of leaking information, and can help discourage leaks. This will however require some development time to implement, so isn’t practical for all teams.

Remove important assets from the playtest

When working with established IP, it can help to obscure the IP in the test build. This could look like changing the title screen or renaming the game in the build so that it doesn’t mention or show pictures from the brand. This is perhaps most relevant to simple mobile games, as replacing entire character models is unlikely to be possible in more complex games. 

Removing assets can be achieved either through creating a custom build, or avoiding including the final assets in any build until post-announcement. Replacing assets also requires development time, but can protect a game from leaks. 

And most importantly – screen your participants. 

Both times I’ve had issues from playtests, they could have been resolved by better screening. In one of the sessions, they were a journalist from a minor publication, who believed they had a scoop. In the other, two participants knew each other, and were able to collectively be brave enough, and then corroborate, leaked information to the games press.

Remember to check when recruiting, and verify during a pre-interview:

• That they are not a journalist
• That they are not a streamer/content creator
• That they have no personal connection to other participants

There are very few actively malicious participants (I’ve seen thousands of players, and only had issues a few times) – but asking these questions, and removing people who fail them or can’t give convincing answers, will help identify the most at-risk sessions.

Building trust through consistent results.

Worries about leaks push teams to delay running external playtests. Instead many teams just run playtests with their colleagues as participants. But our internal colleagues are just so different from real players, that a lot of the value of playtesting gets lost. 

These techniques combined can lead to a robust playtest setup, with minimal risk of leaks. But getting permission to run playtests, or the confidence to run your own is a much more human problem. 

Building trust from management can take time. Looking at teams who have been successful at integrating playtesting in their development process, it can often take multiple games to achieve.  Showing value with late evaluative playtests can build trust, and on the next game playtesting can start earlier and have more impact.

A great example of how a relationship can change over time is the work done at Bungie, moving from a few studies in the year before release for Halo 2, to regular playtests across the four years prior to launch for Destiny.

(The source is John Hopson’s 2015 GDC talk)

Building trust is a marathon, not a sprint – but being able to avoid embarrassing playtest leaks will help us get there! 

Continue growing your games UX skills

Every month I send a playtesting and games user research lesson, just like this one, to help game developers run better playtests, and people start a career in games user research. Sign up to get the next one direct to your inbox at the bottom of the page.