Tom O’Connor has always been an advocate for dedicated user testing. Having worked in senior production roles with the likes of Tencent and PlayStation, he’s seen its value and importance, helping companies scale their understanding and processes through the development lifecycle. Whether testing design or unique features, saving on costs, improving the final release, or tweaking games beyond the 2.0 iteration, O’Connor has helped publishers grow their processes with the ultimate end goal of perfecting product quality.

“It needs to be happening at the beginning, first of all. Right at the beginning when you want to steer concept tests,” says O’Connor. “I know that’s a bit of a different type of test, more of a marketing task, but when you work on something internally you’re so close, you just get so excited about something and you think it’s the next big thing.

“And also in the pre-production phase. Once you’ve got what you think is going to be a stand-out feature in your game then that’s something that you need to prove first. Get that into players’ hands as quickly as possible and as regularly as possible so you can iterate on it,” he adds.

You may think you have a unique feature – and you may well have – but you’re going to have to take a step back when assessing it. This is why external testing can be crucial, according to O’Connor. 

“I’ve been in situations where I hear ‘oh yeah we’re doing user tests, we’re getting all these students in and we’re watching them play the game’. But who’s watching them play the game? It’s the creative director. So they’re seeing what they want to see,” says O’Connor. “You’ve got to have somebody who’s completely unbiased. Someone who isn’t asking leading questions. They’re getting out of it what you’ve briefed them to try and find without putting words in people’s mouths.”

In-house testing can of course work, but it has to be with a dedicated team. According to O’Connor, a common misconception is that user testing is a glorified QA department. 

You’ve got to have somebody who’s completely unbiased. Someone who isn’t asking leading questions

“I know there’s a little bit of crossover with usability, but when it comes to proper qualitative feedback on core features, you’ve got to have the right people – and that’s a skill,” he says. “I’m always saying ‘don’t think that user research is the same as QA’. It’s not just about finding bugs and fixing blockers in games, it’s about getting stuff out of it that you might not have even thought about”.

Another common mistake is putting a different team of developers from the same studio on user testing duties, when there might already be a rivalry between departments, making feedback unreliable at best.

“So you’re going to test it with experienced developers where there’s already some kind of internal competition between projects, where people generally in my experience, actually want to be critical? They’re not the target player. I’m not saying that that type of feedback is redundant – it’s not – but you can’t rely on it”.

In O’Connor’s experience, getting the right play testers is crucial. Work on recruiting players who understand the genre, the game you’re trying to create, and how it stacks up against its inspiration and rivals, he says.

“Put your players into different tiers. If you’re testing something like a Souls-like, and it’s one of the first tests, your core feature set needs to be stand-out. If it’s all about the combat then you need to make sure that the players you’ve got have done an excessive amount of hours in the genre. They can talk about any of those other Souls games, and they don’t just talk positively about them. Even the best ones, like Elden Ring, they can pick it apart. They can say ‘oh you know it’s great, but…’. 

“There are a lot of other players that are more like followers. So they’ll play anything and they’ll say everything is amazing. You’ve got to make sure that you’ve got players who have done a thousand-plus hours and can prove it. Your recruitment company has to be doing their checks right”.

Unlike mobile user testing, where developers can use thousands of live players to iterate their games, console testing can be anything from a handful of players to upwards of 50. That might not seem like a lot in comparison, but studios should still take care when interpreting their results.

“When you do eventually get your consolidated report back, be very careful that there’s not one or two people in a participant group who are skewing things,” he says. “The raw data is important, you can go through and actually see whether or not there’s somebody in there that’s just kind of skewing results.

“Be aware of the different tiers and not grouping them all into one. Don’t group the feedback from the expert players who have played all of the latest Souls games for a thousand-plus hours with those who have dabbled in these types of games but haven’t necessarily completed them”.

O’Connor set up specific testing operations for Tencent in London, building slowly to a team that’s now used company-wide across all titles. And just like during his time at PlayStation, he built user testing into the development process from the off. Because if you wait to test, you miss opportunities to make actionable changes.

“Do not hold onto things until you think they’re ready,” he says. “Because you might be waiting for the wrong things. Get them to a point where a good play tester can look past all of the rough edges and still give really good quality feedback”.

Early and continuous testing also helps cushion some of the more negative feedback a development team will get from user testing. It’s important to get the whole development team behind the idea and understand the process of user testing, according to O’Connor. Even if the results may put a few noses out of joint initially, the results speak for themselves and help create a better game. 

“We look at the key milestones and work out where we think we will need to get good feedback from players and then we’ll tie a more significant research piece onto a key milestone. We’ll rely on the feedback as part of the agreement process,” he says.

“It becomes a significant internal goal where the whole team is aware that this is the first time people are going to play the game and you’re going to get feedback. You’re not going to like all of it but you’re going to need it to be able to feed that into the next milestone phase to improve the game.”  

This is the first time people are going to play the game and you’re going to get feedback. You’re not going to like all of it but you’re going to need it

“It’s part of the development process like anything else. Preparing that build to be able to run out of the studio is a lot of work. To get it to a stable enough point so it won’t crash. But also factor in the lead time for starting the test, ending the test, getting the report done, getting that debrief back to the development team, working out an action plan, putting that back into the development roadmap, and then testing again… if you don’t plan it you’re screwed, basically. Your game is not going to come out.”

There’s lessons to learn from big publishers, no matter the size of your business, says O’Connor. “They generally have a number of production gates. They get multiple expert groups within the company to feed back, and that also includes external partners. The feedback that comes from that gate depends on whether the game even proceeds or not. That’s how seriously companies take it.”

Placing gates earlier in your process can save time, work and money, says O’Connor. “They might come a year or two into development when you’ve already spent millions of pounds. You need to know very early on whether the game that you’re building has got a chance so having that feedback is critical”.

Another common mistake is waiting on features to be complete before testing them. “You can test a block running around a white box environment if it’s about a specific feature where the graphics aren’t important,” says O’Connor. 

User testing doesn’t have to be large scale to deliver useful results, argues O’Connor. Costs, time, and your player base can be kept manageable even if you’re a lean company.  

“The most important person is always going to be your customer. Get engaged with them,” he says. “Even with indie companies, there’s so many ways to engage with the community, whether it’s a Discord server, or speaking to people directly through community channels. 

“You haven’t got to release all of the details about your game, there’s still ways to talk to people. So get yourself a following from a community perspective as early as possible. You need that because you’re never going to have the marketing spend to get eyes on your game at launch, so start early. And work with them little and often”. 

“If costs are an issue you can advertise on your website, you can call out on social channels.  When you do find people or you think this person actually knows a lot about this genre, then find a way to keep on talking to them. Have them become ambassadors for you and your studio and your game, and help you steer the product. Don’t always think that it’s something that the publisher does. If you’re not self-publishing then generally the publisher will user test – but make sure that that’s not something that just happens at the end. 

“Even with your live game, keep on testing those patches and development in-house before release. Sometimes studios are just too slow turning it around. And then you’ve lost your audience, they’ve gone on to something else. And getting them back is expensive”.