User Research: Turning Design Vision into Player Reality

by danhon

User Research: Turning Design Vision into Player Reality
Jerome Hagen, Microsoft Game Studios

So thanks for coming – I work at MGS, we have a games user research group there that’s been in existence for the past 12 years, and I’ve been there for as long in various roles and we have around 40 total people either working in the user research group or supporting it, around 20-25 user researchers who focus on individual games.

What I’m going to talk about today is about how we use user research but what the principles are about what we do with the research and what’s important to think about when you get user research on your own products.

At its most basic level, the definition of user research is to improve the gameplay experience before release, by understanding how players will interact with and react to the game.

To give you a little sense of what this like first hand, I have a short video for you – an old video – a classic – by the huge CRT monitor and the player using a ‘joystick’ to control a game, which I don’t know if anyone’s done for the past ten years, but the game we were doing a study on was Midtown Madness, a PC racing game at the time, a game where you race through the city and take any route to get to a destination. This was the first itme they’d played this kind of game.

[Show video]

This guy is in a race for three minutes, his interpretation of what he’s supposed to do is follow all the traffic rules, ‘this is a cool tree’. ‘It’ll be more like a TV, like real – man, I was going to run that light…’ ‘this is just cool! I mean I’m driving, I’m goinna see if I can follow this car!’

‘The arrow is yellow, now it’s green again… Well, if the arrow was indicative of what I should do, it means I should run up the grass which means no sense whatsoever…’

So this is a situation with novice gamers new to a new kind of game. You’ll be amazed at how many things people have problems with even if they’re experienced with the genre and the games themselves.

So, I’m going to talk a bit about extended metaphor here. One thing about user research is that it’s an important partnership with design. Designer is Amelia Earheart here – designers are like the early days of aviation – you have little information, have to do lots of things yourself, have to rely on instinct to get to your destination. It’s difficult because you don’t have realtime feedback about how your game is received in the outside world. So with user research, being a designer can be a little bit more like this [glass cockpit] – tonnes of information in realtime in terms of how the outside world looks and what you need to know to react to it. Of course, it’s an amazingly overwhelming amount of information so one of the key factors about user research is that it’s a prioritisation and filter system so you know what to pay attention to.

So one of the the important things that I think is cool about user research is that it’s a data service feeding into creative endeavour and having that data and having that feedback allows designers to be much more creative and it allows creatives to take more risks and get feedback about what’s working and not working and to get to destinations that would not have been possible otherwise.

We’ve had user research in the past which is a lot of information that requires a lot more of combining your own data and we’re really getting to the bpoint where we can summarise things more clearly.

As I mentioned, I work at MGS, Games User Research Group. We interact directly with development partners in MGS, internal studios like Lionhead and Rare and external like Epic and Bungie. But clearly we’re a development service. We’re not a final check. It’s working together with them throughout development.

So this is a brief look at some of our labs. These here are usability labs, similar to the guy playing Midtown Madness. On the top level, there’s a living room lab and a smaller one for PC and single player play, ont he bottom is some of our observation sides that allow us both to record information and to have designers and other team members in the lab with us so we can discuss realtime and dig deeper.

This is one of our playtest labs, where we’re not getitng as much realtime feedback, but it’s really important as we’re getting in to balancing and large group feedback about what’s going on.

So with user research there are a bunch of different tools available and going into them will take several hours. MGS is not the only people doing games user research. There’s lots of other companies out there, there are different terms, Im not going to go into too much detail on them, but as we talk about specific examples, I’ll talk about some of the techniques we use and the information we gather.

So: getting into the principles behind user research.

1. More than anything else, it’s important for you to get designers on the teams to watch people playing first hand.

That’s the single biggest thing to understand what it’ll be like for players in the real world. This is one of the Halo designers watching a gamer play through the level he designed, which can be extremely illuminating and disappointing at times. As a designer, you don’t come in the box with the game.

Understanding how players will interact with and react with the game.

There’s lots of different kinds of players out there. It’s a bigger discussion what your primary, secondary audience is, but it’s important to think about who you expect to play your game and those people who you reach out to that you get those people represented and that you see what the experience is like for them.

There’s a big distribution of players out there. The main sources of feedback without additional research – the forums are the most vocal people, and not necessarily most representative, they can be a tiny distribution of the people you have.

The next part – understanding how players interact with and react to the game.

Interacting with – what they do and why they’re doing it. So verbal feedback as they’re doing it. There’s mayn different methods for understanding what players are doing and understanding the reasons why is the only thing that’s going to help you address the issues. You need to undrestand what trips people up and what blocks them. In general those are the big ones.

Example from Halo 3. I was talking about what people play and what people do and why they do it. In Halo we have instrumentation which is direct recording of player behaviour as they go through, that helps with understanding of what people do. This is a map of the very first level in Halo 3. Up in the top here is dots that reprsent a snapshot of where players were – this is the first encounter near the beginning of the game. The design intent is that players get throug, follow the pathway through and get to the end of the level. We take several snapshots of where players are at various points. We see further on, there’s still people further on, some people are moving further, a lot of people went back to the start area, and then here, is where – further on – lots of people get much further, but lots of people are staying or backtracking to the start. The question we had at the time – we knew what was going on, we knew where they were going, but not why they were doing so. Were they stuck or lost, exploring, trying different things? So in addition to just this direct recording of play data, we also had a questionnaire – a subjective one that pops up every 3 minutes – too easy / not sure what to do / not sure where to go / too hard / I would quit.

The dots represent when people gave that response on the map. These responses are for the whole leve. If we zoom in up here to the start area, we see that people were giving those responses – brown dots, some green dots, one blue. So the green dots, not sure what to do. We have brown response – most extreme response – I would quit. So we saw that players weren’t havin this experience that they wanted to. As I mentinoed each of those points on the map were in places where people had gotten to certain encounters and then backtracked all the way to the beginning. The ultimate solution was funnelling people through the level and where they were backtracking unintentionally, dropoffs that could not be climbed back up were introduced.

The last most frequent thing that was in responses there was not so much how players reacted to the game. What’s important is the kind of experience players have and where the major problems are.

Here’s an example from Crackdown, a game that had some different kinds of gameplay that people had not experienced. A superpowered agent fighting through a city to defeat gangs. You can jump higher over time and you have several different skills you develop and improve over time.

As I mentioned there’s several skills. The first one is agility, then explosives, then firearms, driving, then hand to hand combat.

Going back to design intent – it was that players would discover and try out all the skills and they would use whatever skill they most enjoyed to get through the game. There’s 21 gang bosses you have to defeat, none require the use of a specific skill, so players can essentially build up the skills they care about. And actually, fairly far out from release. we had a game where we removed brick walls and hurdles. And we could let them run free.

They explored skills they were most familiar with from other games – shooting and driving, which left out a lot of what was fun about the game and the different techinques that players have. The feedback at the time was that they’d seen shooting and driving was, they didn’t see what the big deal was, they didn’t know what this game brought. The skills develop over time. So players both use them over time to defeat enemies and in some cases get rewards for other things.

So there were two big changes that went in. 1 it was made clear the differences between skills and giving them experience of them, and making them advance much faster.

This is what we saw when people played through again. Instead of ‘I just ran and drove and shot, that was boring’ it was really clear that advancing the skills was coming up more strongly, and particularly with agility. ‘I like leveing up the most’. We asked peopel what they thought of the game every 20 minutes. We would see fun over time. At 20 and 40 minutes, epoeple were starting to get into the game. 60 minutes was where they sarted advancing the skills. Particularly with agility. We used those stats so that everyone would hit within the first 30 minutes in the demo and able to see the experience in a different situation. Helping people see what was unique about Crackdown.

One other short exapmle was from Shadowrun. MGS worked on this. Largely multiplayer game. It was one of where toward the end of development, there was a lot of feedback that they felt the weapons weren’t strong enough. Based on balancing the game. We could design a study with 3 different conditions. 1. As it was, 2. another where we added a shield where people showed when they got hit and a 3. where we added visual and audio feedback.

So, talked a lot about some heavy duty user research that used a lot of data. You might be wondering if this is somethign I should bother with. I would definitely say that it’s really important to get player feedback early. There are a few key things I talked about, if you do nothing else, then you must do these things.

1. Get designers, team members, watching people play the game.
2. Get players who are representative of the playing audience
3. Focus on the things that block people and trip them up.

I’m not just saying this as a thing that if you don’t have the resources we have, we use this principles all the time – I recently did an important study – we used an empty conference with a laptop and a webcam. We got things out early in development. It’s working with our develpoment partners – I encourage them even if not formal research to get new players playing the game. It helps the designers and helps me as a user resesarcher so we can deal with the really sticky stuff.

That’s not to diminish the amazing things that user researchers are doing. One of the biggest thing is paying attention to the player experience throughout development. Everyone on the team should be thinking about that, but having that feedback of seeing what players do, what they’ve done on that game, and as I mentioned, it’s really – you’re going to get a lot, an experienced researcher can really help out with prioritising and filtering that.