What GameDevs should learn from Nate Silver

One of the most critical points of Feeding the Iteration Machine is taking a numerical look at what your product is doing. You can’t truly iterate-to-improve your idea unless you know what’s wrong with it. As a game dev, you need to cut through the jargon and gut feelings and embrace big-data to develop a better game.

Viewing the future using numbers

Nate Silver recently gained fame after he accurately predicted 50/50 states for the 2012 US presidential election, (following his 2008 prediction of 49/50 states, where he gained less fame for some odd reason). This was a fantastic opportunity for big-data to get back into the light of the lives of common, everyday lives. It was amazing to see how Nate was able to use data prediction and modeling, a counter-intuitive process in a political system that is still driven by people following their gut feelings. And much like Moneyball, it’s apparent that the political system will never be the same; for those of us in Silicon valley, this was old news, but for most people, what Nate did was magic:

The reality is that big data rules the 21st century, and any business developer that’s not utilizing every statistic available is immediately at a disadvantage; you’re flying blind. That’s one of the great things about having your business be digital, you get input and feedback instantly.

This type of statistical analysis doesn't just extend to your product in the wild, but should also be used to track the information about your game in pre production too. Utilizing big-data in order to track your game’s frame rate, stability, and content-churn are all great predictors. We've seen this being used to great success for social casual games, and even hardcore FPS games. Directly, you shouldn't be asking if your game should track data, but rather what data should I track?

Big data checklist

Here’s a simple data-checklist on the things you should be tracking and how you should be responding to it.

What you should be tracking:

• Stability each day
• Track your crashes, against platform, region, map, callstacks etc.
• Track a user click-stream
• What are they clicking on the most, what aren’t they clicking on?
• How do you analyze it?
• Track a user purchase profile
• If you’re using a freemium style, it’s important you not expecting every user to monetize the same way. Create a profile for your users and react to them properly. use location
• How are users discovering your game
• Do you know how you’re acquiring users? is your marketing campaign working? Should you be spending advertising money on kotaku.com, or thebumb.com?
• Track player progress
• How many players make it to level 7? How many players buy item Y?

What you’ll need to use the data properly

• Automatically Generate daily reports
• Generate reports from your data each day.
• Make sure that you can easily access your reports. If it’s too difficult, then you’ll ignore it over time, or not get a chance to look the data the right way. You’ll ignore the signals before it’s too late.
• Availability to drill-down as needed
• Make sure that your data isn’t static, and that there’s ways to look at the information in different ways. It’s not beneficial if it takes an extra 3 days to track a new metric, or chart a graph plot against another value.
• A flexible backend
• Make sure your backend can easily accept new events, or discard old events.
• Make sure your backend can easily be duplicated / redundantly stored; You should be able to track all the data for your game since inception.
• Semi-regular “What does this mean” sessions
• It’s important to schedule semi-regular sessions with the stakeholders of the product to determine what the data means. After all, there’s lies, damn lies, and statistics, and it’s important to get multiple validated viewpoints on the data coming in before making any actions
• Ability to push builds quickly
• We’ve already discussed how important this is for the life of your game

Big data fixes failures

Failure is not an option, it’s a requirement. If you’re a game developer, and expecting to not fail, then you’re delusional. The best studios hide these failures from being commercial by moving the iteration process into something that’s not tied to their bottom line (internal testing, beta tests etc)

The truth is, that if you fail,and don’t know why then you’re going to keep failing, and success isn’t coming any time sooner. Too many game devs will fail on a product, and not know why; they end up blaming tangential things which may not be correlated. Once the failure has occurred, you should know why, and be able to fix it next time.

“Insanity is the act of doing the same thing but expecting a different result”

Make sure you know why you’re failing, so you can stop doing it.

~Main

You can find Colt McAnlis here: