The games business is notoriously secretive, so Sharkmob’s user testing facility in Malmo inhabiting an old bank vault might not come as a surprise. It has a suitably cool-sounding acronym too – REEF – Research End-User Experience Facility. But it’s not a top secret space for limited eyes only. Sharkmob’s user testing is for everyone who works at the development studio, and its observations are incorporated into everyday work. 

“We’re incredibly lucky to have this kind of facility,” says Brynley Gibson, executive producer at Sharkmob. “It’s a bit daunting when you first go in through this vault door, it can feel closed off, but once you’re inside it’s quite a nice space. It’s sort of relaxing. There’s no dark glass windows or anything like that, but everything is filmed inside. And we have a team of around three people – our user research experts who run tests each and every day.” 

REEF needs to be welcoming because Sharkmob tests a lot, using play testers, press and influencers, or non-games players if necessary. It even recruits using old-school fly posters pasted around Malmo – user testing needs you.

There’s nothing like sitting someone down and watching them play. You watch people’s mouths drop open where their assumptions are just destroyed. It’s really humbling

“We’ve tried to do all different types of testing and really, I think it’s fair to say, it’s an integral part of how we work. We test with real players, mostly every week,” reveals Gibson. “That can be just general – how is the game performing, basic metrics of movement and shooting, these kinds of things. Or it can be really drilling into something specific – a new set of levels or some new mechanics where we want to understand how well they’re working, how are people reacting, do they understand it?”

Opening up research for the whole team

While REEF is a controlled environment for gathering and reading data, it’s open to everyone from the studio, welcoming drop-in guests to make their own observations.

“We do have eye tracking and things like that but generally what people are watching is how they’re playing, and to see everyone playing together,” he says. “You can take your own notes, so if you’re a level designer you might just want to rock up and see how they are getting on in the world. And then we do interviews afterwards, the general and specific, and then that is shared with the whole team.”

Not everyone in the team has time – or needs to – absorb all of the data gathered by REEF, but because it’s integral to the studio, whether a deep-dive or just reassurance, user testing is just accepted as part of the Sharkmob way.

“Everyone can get that information and we try to do relatively regular updates from the insights team because we know not everyone’s going to read those documents,” details Gibson. “So as well as useful information that can help guide us, I think in game development you can be so heads down, so lost in your work, and in lieu of press before your game comes out, it’s another way of sanity checking what you’re doing.”

“There’s nothing like sitting someone down and watching them play,” he says. “Having the team watch people play the game secretly is eye opening, always, every single time. You watch people’s mouths drop open where their assumptions are just destroyed. It’s really humbling and you only have to do it once and then they’re like, ‘oh okay. That’s real people playing our game’.”

Balancing data with creativity

Even though product testing has existed in games for decades, there’s still some misconceptions around it, particularly when questioning the value of data. 

“It’s important for people to understand this doesn’t replace creativity, it’s really there as a guide,” says Gibson “You can validate and iterate by what you see. When we’re fine-tuning certain aspects of the game – let’s say onboarding – every button, every click, can mean someone stops or keeps on playing your game. 

“It’s just a very natural fit where you really need to test every single bolt in what you’re doing. Having it front and center, we talk about it very regularly, and we also have a data team so we have telemetry from internal tests and bigger public tests. That double pairing, it’s really useful to take it to team meetings as part of the creative conversation, as part of the production conversation. That’s how you can get people just used to it, like muscle memory.”

When we’re fine-tuning certain aspects of the game – let’s say onboarding – every button, every click, can mean someone stops or keeps on playing your game

Data gathered by REEF and Sharkmob is also used in wider conversations with studio owner Tencent, another organisation that puts heavy emphasis on data.

“Tencent is a very data-led organization so it’s useful for stakeholder management for us to have our own research and information to validate something we might be saying about the game,” says Gibson. “It helps with that conversation and obviously we receive reports and information from them so we share that too. It normalizes the language between us.”

Testing A live service game

Sharkmob is currently working on open-world extraction shooter Exoborne, and has expanded its user testing from the controlled environment of its bank vault to live play with tens of thousands of gamers. Both can give different results, and present different challenges – from the play environment, to the scale of testing, and assumptions based on genre. 

“It definitely can give different results but that doesn’t necessarily mean opposing,” says Gibson. “We do a daily survey – then three days, five days, seven days. You’ve got tens of thousands of people around the world. But how many people actually fill in that survey? 

“The person who doesn’t like your game, who churns, do they fill in the survey? Probably not. And actually that’s a person you want to know better. Did they stop playing because it’s bad, or for some specific reason? Or is it because they thought it was a different type of game? That kind of information is very valuable”. 

Some results from the live test can then be taken back and iterated upon in REEF, says Gibson, while others are just not replicable.

“There’s two things with Exoborne. Firstly, it’s open world, there’s a very large map. Secondly, it’s a live service game. Obviously a lot of the player experience is gained over time. So how do we record that? The data we get from REEF is not exactly the same as we’ll get from a large scale test. We can go deeper in REEF and ask more specific questions and we can see exactly what they’re doing. We’ve tried to simulate having someone coming back and play so we can better look at progression in the game,” he says.

“But the economy isn’t something we can really look at. Economy, we can sort-of simulate in Excel but really you need loads of people playing the game to see how a mass economy works, and that’s something we can’t really do in-house.

“I think that’s the biggest challenge; trying to simulate how people play at home as best as possible. Because there has been times where we thought something works and then we put it into the wild and it hasn’t. But then other things have matched up really nicely. Our latest tutorial and onboarding, we tested that vigorously, it looked good, we put it in the wild and it kind of tested similarly. You have to be aware of the different things you’re testing and maybe the kind of the strengths and weaknesses of that information the data will give you,” he adds.

Working with the press and content creators

As well as public and private testing with regular gamers, there’s another group that have proven useful for the development of Exoborne. Feedback from traditional press and content creators may go public in previews and hands-on video, but it’s also led to actionable changes behind the scenes.

“We do extra stuff with the press. They play and from the questions they ask you understand some of their feedback. And obviously when they publish a write-up you can see that as well,” says Gibson. 

“But we’ve also had content creators, privately and publicly, come to the studio to play the game. They are not a normal user, but they play a tonne of these games, they know them inside out, how they play is quite interesting,” he adds.

“The first closed test we did in early 2024, we had a bunch of influencers come to the studio and they played and we did interviews with them and we did Q&A. We had lunch with them, dinner with them, so we can really kind of talk about it openly, and there were things where they agreed and disagreed which is fine and totally normal. But the thing that they really hammered us about was the looting in the game. We thought it was okay but needed improvement – and they absolutely hated it. Unanimously. One of the guys said ‘I don’t mean to be rude but the looting is trash. It’s just trash’. And then they all slowly spoke up. 

“We were like ‘shit!’. I didn’t think they were rude. That’s real. We thought it needed improvement, but what we’ve done we now think is actually wrong. And that was something where we then redesigned the whole thing. It was an incredibly expensive decision and time consuming, but it was the right decision to literally rip it out and go again with it. Because it’s such a cool part of the game. We’re still fine tuning it but we’re now on the right track. It’s not taken lightly but it shows the power of those kinds of conversations where you are way off but you can fix it.

Making sense of the data

Having a lot of data, whether from in-house or mass testing, needs to be interpreted. Product testing is a true skill, a difficult one, and it requires a disciplined expert to make use of it, says Gibson.

The big tests give you big data, but the lab testing gives you finesse

“In-house you can really pull out subtle information and really dive down into something that you can’t do in the mass tests. The big tests give you big data and that is very useful for scale, but the lab testing gives you finesse and that’s vital,” he says. “You and I could look at the same report, if we’re looking at raw data, and we can definitely come up with different conclusions.

“That’s where you really need good experts who can interpret what they’re seeing and give you their analysis of it. We have user research experts. That is their bread and butter. They’re not coming up with designs, they’re not working on other parts of the game. They are very focused on their work and their work is to produce those reports.

“And then it’s up to us, the dev team, to work out what to do with our information. How do we solve that, with creative ideas? It’s an exciting part of the process. We make products. Yes, it’s a creative endeavor, it’s artistic, it’s craft. But we are in the business of selling video games.”