Can AI Predict the Stock Market? Enter Large Language Models

We’ve all seen it—AI beating humans at chess, diagnosing illnesses, and now, predicting stock prices. But hold on, isn’t predicting stock prices the Holy Grail of financial markets? A team of researchers from Hamburg University, led by Frederic Voigt, is asking that very question. In their latest study, presented at the PETRA 2024 conference, they explore whether AI techniques from Natural Language Processing (NLP) can be repurposed to predict stock movements. Sounds a bit futuristic, right?

But, if you’re thinking, “Isn’t stock price prediction all about numbers? What does language have to do with it?” Well, Voigt’s team argues that stocks and sentences have a lot in common. Let’s dive in.

NLP and Stocks: An Unlikely Friendship

Voigt and his co-authors make an interesting point. When you think about it, stock prices are just like sentences. They follow sequences, they have patterns, and if you analyze the past, you can try to predict the next “word” or price movement. So, it’s not such a wild leap to use tools designed for understanding language to make sense of financial markets.

The idea is that models like Transformers, which power AI tools like ChatGPT, could be trained to recognize patterns in stock data much like they predict the next word in a sentence. Could the same AI that finishes your sentences also tell you when to buy or sell Tesla stock?

Stock2Vec and Transformers: AI’s Playbook for Wall Street

The researchers have a couple of neat tricks up their sleeves. They introduced models like Stock2Vec, an adaptation of Word2Vec, which, in plain terms, is an algorithm used in NLP to turn words into numbers. They’ve just replaced words with stock prices. The idea is that by embedding stock data into vectors (basically turning prices into data points in a multidimensional space), you can better understand and predict stock movements.

But they didn’t stop there. They also experimented with pre-trained Transformers—yep, the same kind that powers large language models like BERT and GPT. By training these models to predict stock movements, Voigt’s team wanted to see how well AI could forecast price trends based on historical data.

So, Does It Work?

Well, kind of. The paper presents some promising early results, especially from models like Stock2Vec and Stock Transformers. But the researchers are upfront about the limitations. They’re not trying to create a foolproof, market-beating AI (that would be the end of Wall Street as we know it!). Instead, their goal is more exploratory—can these tools, designed for language, be useful in analyzing financial data?

So far, it seems the answer is “yes,” but there’s a lot more research needed. They’ve only scratched the surface with quantitative models—methods that analyze stock prices using numbers alone. Down the road, they’re planning to explore multimodal approaches, bringing in things like news articles, social media chatter, and other text data to further boost prediction accuracy.

The Challenges of AI Stock Prediction

Voigt’s paper is clear: predicting the stock market is no easy feat. One of the biggest hurdles is Black Swan events—those unpredictable, world-changing events that no one sees coming (think the 2008 financial crash or the COVID-19 pandemic). AI models can get thrown off course by these outliers because, by nature, they rely on historical data. If something happens that the AI has never seen before, it can’t react properly.

The researchers acknowledge that AI models like LLMs can’t fully handle these events—at least not yet. But that doesn’t mean they can’t be useful. The goal isn’t necessarily to predict the unpredictable, but to better understand market trends under “normal” conditions.

What’s Next for AI and Stock Predictions?

The future is bright for this field of research. While Voigt’s team has focused on quantitative data (just the numbers), they’re looking to incorporate textual data—think financial news, company reports, and even Twitter. This method, known as fundamental analysis, has the potential to make predictions more robust by capturing not just what’s happening in the markets but why.

This mix of numerical and textual data is called a multimodal approach, and it could bring us closer to AI models that can better predict market movements by understanding the broader context.

Bottom Line: AI Won’t Replace Stock Analysts Just Yet

AI has some exciting potential in stock market prediction, but it’s not perfect. Voigt and his team are showing us the first steps—using cutting-edge AI models to analyze financial data in ways that were unimaginable a decade ago. But there’s still a long road ahead before these models can outsmart seasoned traders or hedge fund managers.

For now, AI is just another tool in the investor’s toolkit—one that’s evolving fast. The challenge will be finding the right balance between human insight and machine predictions. Voigt’s team is confident that as these AI models get more sophisticated, they’ll be a powerful asset in the financial world, but they’re clear that we’re not there yet.