Super Bowl MVP: Cam Newton, Peyton Manning, or 4th Down Bot?

Fifty yard line on American football field.

The Denver Broncos are set to square off against the Carolina Panthers in Super Bowl 50 this weekend, but had it not been for a few key plays, it could have been Tom Brady and the New England Patriots staring down the almost-undefeated National Football Conference champions.

Here’s the situation: it was fourth and one—Patriots’ ball on the Broncos’ 16-yard line—with 6:03 remaining in the fourth quarter of the American Football Conference championship game. The Patriots trailed by eight. Kicking a field goal would put the Broncos’ lead at only five points—a single touchdown—which the Patriots would later in the game go on to score—away from taking the lead.

With the ball only one yard away from converting into a first down, Pats’ coach Bill Belichick decided to go for it. It didn’t work. Tom Brady’s pass was caught by wide receiver Julian Edelman—but he was brought down a yard short of the first down. The Broncos took over possession, maintaining their eight-point lead.

Belichick’s decision to go for it, instead of the “safe” field goal option, is already the subject of intense debate. But there’s at least one thing that says Belichick made the right call—and it’s a robot.

More Than A Decade of NFL Data, Analyzed in Python

Created by data journalists at the New York Times, 4th Down Bot is a data science-powered robot that analyzes every fourth down play in the NFL, deciding in real-time if the coach should choose to punt the ball away, attempt a field goal, or “go for it.” The bot was originally conceived by the Times‘ Kevin Quealy, Josh Katz, and Brian Burke, but the current version—its brain at least—was built by Trey Causey, a data scientist with ChefSteps and operator of the sports analytics blog The Spread.

For those unfamiliar, in American Football you have four attempts—called downs—to move the ball 10 yards, or else the ball turns over to the other team. Most teams usually only use three of those four downs, leaving them with a decision on fourth down: Do I punt the ball away to try and make sure that my opponent is not too close to my end zone? Do I try to kick a field goal, if I’m close enough to their end zone? Or do I attempt to ‘go for it’—conduct a fourth play to try to get the rest of that 10 yards and convert into a first down, granting a new set of four downs?

“NFL coaches are very conservative,” Causey said. “They tend to punt or attempt a field goal on fourth down—not go for the fourth down conversion—even though there’s all this literature and economics and game theory that says it would maximize their chances of winning if they would go for it more often.”

Broncos vs. Patriots 2016 AFC Championship Game

Coaches and 4th Down Bot don’t always agree. Here’s its analysis of three plays from the 2016 AFC Championship game between the New England Patriots and the Denver Broncos.

The idea behind the 4th Down Bot was to make a thing that could tell you in real-time what the “win-maximizing” decision is, to both be useful as a football strategy as well as to educate people about data-informed decision making.

“This is a case where we have a lot of data and a really clear analysis of what it takes to win a game,” Causey said. “The data says: ‘here’s what you could be doing to win more games, as it turns out, and you’re not doing it.'”

The 4th Down Bot is written almost entirely in Python. The model it runs on was trained using scikit-learn, a regularized logistic regression, to analyze around 13 years of NFL play-by-play data. The NFL data is stored in CSVs in a way that’s meant to be uploaded to a relational database. Plays in one table, players in another table, fourth downs in a third table, punts in a fourth, and so on. Every play type discretely in its own file.

What the model actually does is relatively simple. For a given play (that is, for every play of the game), it calculates: “what is the probability that the team with the ball (on offense at the beginning of the given play) will win the football game.” Based on the choices made in each play of all that historic NFL data, the model then has a good idea of how every single play of a game has an effect on the eventual outcome of that game. The next step is the tricky part: serializing the model.

“The bot needs to be able to make requests in real time, as the games are going on,” Causey said. For that, Causey built an API using Flask, a simple Python framework for building web applications.

For each play—as it’s happening—the bot is sent a JSON object with all of the current (that is, upcoming, about to happen) play information. Current score, game time, time of possession, which team has the ball, and so on. The bot then sends back another JSON object that basically says: “here’s the win probability before the play, here’s all the different things that could happen on the play, here’s the break-even point—the point at which you should be indifferent between kicking at something (a field goal or a punt) or going for it on fourth down—and here’s the ‘optimal decision’ in this particular scenario.”

That JSON object is sent to a package written in Node.js containing information about kickers, stadiums, and weather conditions that calculates all those factors and makes a prediction about how likely it is that the kick (again, either a field goal or a punt) will be successful. It then calculates the chances of the team with the ball winning the game for each possible outcome of the play. The option that would result in the highest winning percentage chance (taking into account the chances of that play succeeding) is then bot-approved!

“All that happens in real time,” Causey said. “It has to be pretty fast, because one of the improvements we made recently was to make the bot tweet out what it would do in that scenario if it were the coach.” Now the bot makes and tweets out a prediction just before every play, then follows it up with a more in-depth analysis shortly after the fact. Everything is then hosted on the New York Times, which dynamically generates pages with a nice clear write-up of all the factors affecting the play.

Bots in the Box

On fourth and one with the Patriots’ ball on the Broncos’ 16-yard line, 4DB calculated that the Patriots had a 22 percent chance of winning the game before the play. If the Pats kicked a field goal—which it calculated would succeed 94 percent of the time—the winning percentage chance dropped to 21 percent. Remember, the Patriots trailed by eight, and a field goal would score only three points. Going for it, on the other hand, only carried a 68 percent chance of converting. But even so, choosing that option raised their chances to 23 percent: bot-approved. (A punt here would make no sense given their field position—the bot says doing so would drop their winning chances to 14 percent.)

Patriots coach Bill Belichick elected to go for it.

“The coach and I agree that going for it is the thing to do here, but you may be thinking it makes sense to attempt a field goal,” reads the 4th Down Bot’s analysis. “That would be the right call if you think the Patriots’ chances of converting on fourth down are less than 58 percent. But based on my analysis, I’d give the Patriots a 68 percent chance to get a first down here.”

As it turns out, Patriots quarterback Tom Brady passed to Julian Edelman, who caught the ball but was brought down for a loss of one yard. In other words, that 68 percent chance of conversion ended up failing. Obviously, the numbers are incredibly close, but the bot says the Pats made the right call.

Here's 4th Down Bot's analysis of the Patriots' 4th and 1 on the Broncos' 16 yard line.

Here’s 4th Down Bot’s analysis of the Patriots’ 4th and 1 on the Broncos’ 16 yard line.

So did Coach Belichick have a bot in the coaches box telling him what to do?

“The NFL has crazy rules that say you can’t use computers in the booth—where some of the coaches are—or on the sidelines,” Causey said. “The coaches can’t be in contact with anyone using a computer during a game either.”

Teams get around this by running through different scenarios in practice and memorizing the best decisions to make, as well as generating printouts of possible outcomes—there’s no rule against analog cheat-sheets.

But what if the rules were changed? Will we see the dawn of a robo-coach in football of the future? Probably not—football is a complex game with far more factors in play than simple play-calling. But some aspects of the game might start to see the influence of the machines.

“I think that the decision when to call a timeout is a really interesting question,” Causey said. “Once you have a win probability model, you can answer all sorts of interesting questions about the value of a timeout. What’s the added win probability of having more timeouts than your opponent? How much does it change your win probability to be able to stop the clock at various points in the game?”

Computers could certainly take the role of some booth analysts, as well. Currently there are analysts who sit in the booth watching the game, counting how many times each team runs the ball, how many times they pass, times they run to the left, to the right, and so on. Then, at halftime, they send their analysis to the coaches and say “hey, when they run to the right, this happens—we should make this adjustment.” Causey thinks there’s a lot of room for speeding up that process, getting those adjustment suggestions to coaches on the fly, rather than waiting until halftime.

Another aspect Causey thinks is ripe for automating is the process of reviewing game film to assess performance. “You have no idea how long people spend just watching film and tapes of previous games,” he said. “I think as things like deep learning and computer vision continue to improve, we can start building models out of that film and save a lot of person-hours as well as minimize human error.”

“We’re not anywhere close to that yet,” Causey said, “but that’s an area I’m really excited about.”

Want more data science tutorials and content? Subscribe to our data science newsletter.

[[formassembly formid=432325]]