Data Science

Can data science predict the stock market?

By Wendy Gittleson for Galvanize

Whether it’s targeted social media ads or Netflix predicting your next favorite binge, data science has gotten pretty darned good at predicting human behavior. What happens, though, when data scientists try to predict the behaviors of the world’s investors? Can data science predict the stock market?

What is Data Science?

For those unfamiliar with data science, it’s the science of gathering sets of data and boiling them down to useful information for companies and/or industries. Data science can be as simple as a sock company noticing sales are 30% more robust in colder states, resulting in more advertising in those states. It can also be as complicated as predicting pregnancy based on seemingly unrelated shopping patterns, such as the purchase of cotton balls and unscented soap. 

Data science helps advance and personalize healthcare. It helps manage traffic. It helps solve and predict crime. Data scientists even help build sports teams, and with mixed results, predict their success. If you think it’s a coincidence that the Vegas house has better odds than you, or that you win just enough to keep you gambling, it’s not. For that, you can credit (or blame) very savvy data scientists. 

With all that data science can do, it’s no surprise that Wall Street wants in on this highly specialized discipline, and they’re paying a lot of money in hopes of great returns. 

How Wall Street Hopes to Benefit from Data Science

Wall Street began hiring data scientists as early as the 1980s. The best and the brightest from our nation’s technology universities created models that outdid almost all expectations. After some initial success, though, they discovered some roadblocks. 

According to a recent Bloomberg Businessweek article, “most investors can’t beat the average, and every computer that momentarily finds a winning formula will soon face others trying to outwit it. But it turns out that investing is also simply harder than, say, predicting your next Amazon purchase.”

“It’s one of the most difficult problems in applied machine learning,” according to Ciamac Moallemi, a professor at Columbia Business School and a principal at Bourbaki LLC, in the Bloomberg Businessweek article.

While we humans take great pride in our individuality, our behaviors are actually much more predictable than the markets. An American paper manufacturer’s stock, for example, could take an unexpected dip over a sudden rise in interest rates somewhere across the planet. A hurricane or coup near a lumber farm could affect production for months. On the other hand, it’s pretty simple to forecast how, where, and when customers might make their next paper towel purchase.  

Another obstacle to market data science is the shockingly small amount of data.

According to Bloomberg, “The history of stock prices is relatively thin. Say you’re trying to predict how stocks will perform over a one-year horizon. Because we only have decent records back to 1900, there are only 118 nonoverlapping one-year periods to look at in the United States. Compare this with Facebook Inc., which has an endless trove of stuff to comb through—it processes 350 million pictures a day.”  

Alternative Data

Turning to data to predict stock performance is hardly new. Investors have long used data such as financial statements, sales figures, SEC filings, etc. to gauge a company’s overall health and investment-worthiness. Today, though, data scientists are turning to “alternative data,” or simply put, data sets that are less traditional, and quite often, outside the company’s control. 

Examples of alternative data include cell phone usage, credit card transactions, social media activity, product reviews, satellite technology, and news sources. Alternative data is nearly limitless. 

Study Says Yes, Data Science Can Predict the Stock Market

A recent MIT study combined the use of alternative data with traditional data, such as financial records, and the results showed that with the right data, computers outperformed humans by 57 percent.

“Alternative data are these weird, proxy signals to help track the underlying financials of a company,” says first author Michael Fleder, a postdoc in the Laboratory for Information and Decision Systems (LIDS), in an MIT News article. “We asked, ‘Can you combine these noisy signals with quarterly numbers to estimate the true financials of a company at high frequencies?’ Turns out the answer is yes.”

57 percent might not seem like a lot, but even a small advantage can net investors billions of dollars.

How Much do Wall Street Data Scientists Earn?

Wall Street data scientists make up to $150,000 per year, depending on skills and experience. Those who attend an immersive data science bootcamp or a data science bootcamp online can expect to earn even more.

What Does it Take to Become a Stock Data Scientist?

Most data scientists have a background in math. They should also learn a language like Python. While most employers ask for bachelor’s or master’s degrees, you can also learn the skills at three-month-long bootcamps.

In both the in-person and online bootcamps, you will learn data visualization, machine learning, statistical analysis, and natural language processing. 

0 comments on “Can data science predict the stock market?

Leave a Reply

Your email address will not be published. Required fields are marked *