Beta & Feedback

Take a look at this screenshot. Notice anything about it?

There are a lot of numbers. Four numbers in the top left. Similar numbers on the bottom right. Some stuff in the middle. Three numbers in a health bar. Three more in the other health bar. Two numbers in icons in the middle of the screen. Another of those yellow circles with a number in it. Then a bunch of squares with both numbers and letters in them.

This is what we released, when we released our closed beta.

The funny thing isn't that players were confused. It's that we thought there was a chance they wouldn't be.

A funny thing happens when you play your own game for a while. You stop seeing what it is. There are a lot of reasons for this - you've been playing it since it was more broken, and so it seems fine, because it's way less broken than it used to be. You think it's simple, because you've been playing it since it was very simple, and since features take time to "come online", you started with something very simple, and it only very gradually got more complex. Now it's very complex, but you ramped up to it, so you don't see that.

There are many ways you can "test" a game & get "fresh eyes" on it. You can run tests at usertesting.com. It's genuinely fantastic resource. You can watch people as they play your game for the first time, and note all the places they get super confused. You can do playtests in person - informal ones over coffee, or more structured ones in a lab. The more you do, the more $ it costs. So there's always a balance.

It may seem very strange that we did very little playtesting before releasing our game to beta. Well, little external playtesting, at any rate. We did spend hundreds of hours playing the game internally, and we've had friends & family playing it for weeks. The big difference with Friends & Family is that if one of them is confused, they can come talk to you, or you can see it on their face as you play. Not so with beta testers - they get the explanation they get, and if they get confused, or don't like your game, chances are you'll never hear from them again.

But the flip side of that is that the best feedback you can get is from live players. The most honest, the most reflective of how someone will actually play your game. And here's the weird bit. It's okay for players to be confused at first. If they can see some of the potential in a game, they'll keep exploring, and work their way through the confusion. Part of releasing in a very rough state is figuring out whether there's something that a player sees that they think is worth fighting through the rough parts for.

Because as much as polish makes a great game great, polish can't save a dud, and passionate players will fight through a lack of polish if something about your game speaks to them.

So what have we gained through this beta?

  1. We know players are confused by the overwhelming amount of numbers on-screen. We've taken significant strides to fixing that, some of which is live in the Beta already, and some which will be live by tomorrow.
  2. We also have a more significant UI update coming, which will make the different potential actions you can take clearer and more distinct to address some early confusion players feel.
  3. We know that yes, some players have been able to figure it out, and that those players are engaged - sometimes extremely so!
  4. We've had some close games, some blowouts, and gotten our butts kicked by some of the beta players. Honestly, there's no better feeling than having a player out-strategize a developer. It's a clear sign something is going well.

The most important thing? It's not that we found out that players were confused. We knew that would be the case. But it's that players were confused in a different way than we thought. And the reason that's important is that without the beta player experiences & data, we'd have continued to improve things that weren't relevant to helping how players are actually getting confused.

This has already redirected/refocused our development process, and made priority #1 clearing up some very specific types of confusion. If that's all we get out of this beta, it'll be a success. But it's not by far all we're getting out of the beta. We've been playing a handful of passionate, engaged people. We've seen how fast people can take up the core strategy of the game. We've learned about focusing users' attention on a very small number of things that are important. We've learned that some "shorthand" we thought would be useful is way more confusing than helpful. We've learned that even though "power" players may want some things, if it makes things unclear for novice players, that's a problem.

There's a lot to take away from this - only a few days of real, live player data. But also, it's validated one thing - we can move fast. We can fix these problems. More, this is what we're good at, and it's how we're used to working.

I'm excited, because it's only getting better - way better - from here.