Based in the heart of London, Fireshine Games is best known as a publisher of titles such as Core Keeper, A.I.L.A., and Shadows of Doubt, as well as working alongside UK publishers including Team 17 and Rebellion.

Fireshine has worked with user and play testing service providers before, but more recently decided to bring user-testing in-house. As it’s central to the publisher signing games, it made sense to have it as close to the publishing process as possible, where it can refine its own rigorous testing to help new releases from indie developers – and catch potential problems much earlier in the development process. 

“If you have a product that’s not connecting with people then you’re going to have a hard time when you get to market,” says Fireshine’s creative producer, Julian Bird. “So we changed the system of how we sign things to fit the process in many ways. It wasn’t uncommon a few years ago for us to get projects quite late,” he reveals. “We would get projects in alpha or pre-alpha and we found out very quickly that’s more of a case of tuning than it is being able to go ‘no, something is not connecting with people’.

“It’s something that we’ve spent a good few years iterating a process for,” he adds. “One of my responsibilities is making sure that when our games come out they hit a certain quality threshold. That they’re on target for where we want to be. And the only way that we can get something that can be used to validate that data is through careful user testing, being as open and being as indie-embracing as we can.”

It’s better to know it from ten slightly confused testers than from 300 angry steam reviewers

Now if a project is further in development but hasn’t been through a user research process, Fireshine will take a step back and put one in place before development continues. 

“Even if they are slightly more mature, we have the funds to reset the process,” he reveals. “We’re going to iterate another vertical slice build, and then that tends to be what we take to user tests. So two milestones of work – about six months worth of work – on either taking a prototype and putting it through our internal processes and putting it through our internal scrutiny. Or workshopping it with the developers back and forth, maybe some smaller behind the scenes tests, and then we bring it to user tests at the end of that six month period. We’ll use that to really steer where the rest of the project goes based on that initial reaction.”

Bringing Developers into the playtest Process

As well as bringing the user testing in-house, Fireshine brings its indie partners closer to the user testing experience too, as publisher and developer stand side-by-side to take in the feedback. Again, that’s a new way of working now Fireshine has created its internal processes.

“What we did with our previous clients was very much service-based. It was more that we would go away and find a partner and then they’ll come back with a report. That was a little bit sterile. A little bit corporate,” admits Bird.

“What we did with the new system – and it has partially benefited by the fact that we’re working with a lot of teams in Europe – is we said ‘alright we’re going to run this playtest and if we get to go see it in person, you can come see it in person’. We can bring the developers over to London, do that playtest in person, and we end up side-by-side with them seeing that data at the exact same time. We can go back and forth and say ‘this is what we’re thinking, what are you thinking?’ and you can share those notes. The second they see it for themselves in front of them, that spurs the action and it’s a much more collaborative process, it’s a much more useful process.”

For some indie developers, this initial reaction by potentially paying customers is crucial feedback. “A lot of the devs that we’ve worked with actually get quite excited about it. They’re not rock stars but it’s the equivalent of writing a song and then going out and playing it in a small pub venue in front of 20 people, just to see how it resonates and see how it lands. Because we work with a lot of smaller teams, we work with devs that are doing this in their bedroom, they’re doing this in the living room or whatever.”

Scaling Research to the team size

For Bird, it’s important to be that close to its developer partners, to show that Fireshine is as serious about the games it signs, no matter the size. “It’s good to show that you take their game seriously, it’s good to show that you take time,” he says. “Even though they’re ‘indie’, you treat that proposition of a game that they’re creating in the same way that you treat a new Super Mario game, a proper triple-A title. I think a lot of devs just appreciate that level of structure and service to it.”

Fireshine makes sure it scales testing to the game and the development team. “You have to really know what works for you,” he says. “When we started this we were looking at the GDC talks of PlayStation sharing their processes, but retrofitting that on an indie level is not always the easiest thing to do. So just be aware of your team, be aware of how you want to work and how you want to action feedback and there is something that will scale to your level,” he adds.

“We have a developer who has gone from managing a large studio and going from a 15 person team to working on his next title and going ‘I just want to work with me and my producer and my artist’. So that set up is much less based on a rigorous pipeline of AAA testing. It’s more about we’ll give you a space with a Discord or a couple of forms – a very, very light touch, very loose – where people can throw ideas at you and you’ll come back to them within an hour or two. That kind of scrappiness really works for him and for that set up,” he says.

“And I work with studios that don’t have that freedom, that if someone goes off and does this one thing for an hour that means ten other people are blocked. So with them it’s more about setting a checkpoint, setting this date for testing up to this point, and then we have an agreed period of time after that point that we’re going to action that. And anything that we don’t get to now, we’ll set a checkpoint further down the line for that. The good thing about all of this stuff is it is scalable, and we don’t really treat every game the same.”

With developers being so close to the player of an unfinished game, feedback can initially appear a little brutal. But Bird insists it’s a crucial and very useful part of the testing process, one that everyone can learn from.

Even though they’re ‘indie’, you treat that proposition of a game that they’re creating in the same way that you treat a new Super Mario game

“I always say it’s better to know it from ten slightly confused testers than from 300 angry steam reviewers,” he says. “You never can predict what people are going to say or what people are going to do in front of your game, and being open to that has changed how we approach projects. It changes how developers approach projects. Just be open to that because the customer is mostly right most of the time” he adds. “If it’s negative it can be a great motivator, if it’s positive it can be a great motivator.”  

Finding The Right Players

Breaking down the barrier between developer and player is at the heart of user testing. Another strength of Fireshine’s in-house facilities is that it can select the right player for the right game.  

“We do a lot of pre-screening to make sure that the folks that we’re getting in are the core audience. I’ll match players to your competitor titles because they’re the people that you’re going to be selling to at the end of the day. 

Bird is critical of general user testing services – especially online platforms – that don’t scrutinise their testers, and would rather rely on what they simply define as ‘their best’. 

“I’m very critical of this stuff in general – they’re kind of ‘career testers’ or ‘trusted testers’. You might as well get a QA house to provide that kind of feedback because you’re getting that same level of engagement,” he says. “It’s not that they’re not useful, they are if you just want an absolute splash of mass appeal feedback. 

“But you want to be talking to the customer you’re going to sell to. You want to get as close to that as possible and I do feel you can’t just say in your recruitment ‘here is this game do you like it or not?’  If the testers know that the developers are going to be present then they’re just going to gas you up. You get the same kind of feedback from taking your games to EGX or Gamescom or a trade show.

“But if you get somebody who wants to like it but might need a little bit of convincing, they’re usually the ones where feedback is invaluable. Because they’re telling you all of the reasons why when this game is on the front page of Steam and it’s got a price tag, they’re looking at every little bullet point and putting them up against that price tag. And they say ‘well maybe not this week’. And then if it’s not this week, with how fast the industry goes and how much content comes out, if it’s not this week it’s probably a never. I would rather it’s sat in your backlog than on your wishlist.