Using Artificial Intelligence (AI) to predict Stock Prices

Question

Given a set of data very similar to the Motley Fool CAPS system, where individual users enter BUY and SELL recommendations on various equities. What I would like to do is show each recommendation and I guess some how rate (1-5) as to whether it was good predictor<5> (ie. correlation coefficient = 1) of the future stock price (or eps or whatever) or a horrible predictor (ie. correlation coefficient = -1) or somewhere in between.

Each recommendation is tagged to a particular user, so that can be tracked over time. I can also track market direction (bullish / bearish) based off of something like sp500 price. The components I think that would make sense in the model would be:

user
direction (long/short)
market direction
sector of stock

The thought is that some users are better in bull markets than bear (and vice versa), and some are better at shorts than longs- and then a combination the above. I can automatically tag the market direction and sector (based off the market at the time and the equity being recommended).

The thought is that I could present a series of screens and allow me to rank each individual recommendation by displaying available data absolute, market and sector out performance for a specific time period out. I would follow a detailed list for ranking the stocks so that the ranking is as objective as possible. My assumption is that a single user is right no more than 57% of the time - but who knows.

I could load the system and say "Lets rank the recommendation as a predictor of stock value 90 days forward"; and that would represent a very explicit set of rankings.

NOW here is the crux - I want to create some sort of machine learning algorithm that can identify patterns over a series of time so that as recommendations stream into the application we maintain a ranking of that stock (ie. similar to correlation coefficient) as to the likelihood of that recommendation (in addition to the past series of recommendations ) will affect the price.

Now here is the super crux. I have never taken an AI class / read an AI book / never mind specific to machine learning. So I cam looking for guidance - sample or description of a similar system I could adapt. Place to look for info or any general help. Or even push me in the right direction to get started...

My hope is to implement this with F# and be able to impress my friends with a new skill set in F# with an implementation of machine learning and potentially something (application / source) I can include in a tech portfolio or blog space;

Thank you for any advice in advance.

Solution 1

I have an MBA, and teach data mining at a top grad school.

The term project this year was to predict stock price movements automatically from news reports. One team had 70% accuracy, on a reasonably small sample, which ain't bad.

Regarding your question, a lot of companies have made a lot of money on pair trading (find a pair of assets that normally correlate, and buy/sell pair when they diverge). See the writings of Ed Thorpe, of Beat the Dealer. He's accessible and kinda funny, if not curmudgeonly. He ran a good hedge fund for a long time.

There is probably some room in using data mining to predict companies that will default (be unable to make debt payments) and shorting† them, and use the proceeds to buy shares in companies less likely to default. Look into survival analysis. Search Google Scholar for "predict distress" etc in finance journals.

Also, predicting companies that will lose value after an IPO (and shorting them. edit: Facebook!). There are known biases, in academic literature, that can be exploited.

Also, look into capital structure arbitrage. This is when the value of the stocks in a company suggest one valuation, but the value of the bonds or options suggest another value. Buy the cheap asset, short the expensive one.

Techniques include survival analysis, sequence analysis (Hidden Markov Models, Conditional Random Fields, Sequential Association Rules), and classification/regression.

And for the love of God, please read Fooled By Randomness by Taleb.

† shorting a stock usually involves calling your broker (that you have a good relationship with) and borrowing some shares of a company. Then you sell them to some poor bastard. Wait a while, hopefully the price has gone down, you buy some more of the shares and give them back to your broker.

Solution 2

My Advice to You:
There are several Machine Learning/Artificial Intelligence (ML/AI) branches out there:
http://www-formal.stanford.edu/jmc/whatisai/node2.html

I have only tried genetic programming, but in the "learning from experience" branch you will find neural nets. GP/GA and neural nets seem to be the most commonly explored methodologies for the purpose of stock market predictions, but if you do some data mining on Predict Wall Street, you might be able to utilize a Naive Bayes classifier to do what you're interested in doing.

Spend some time learning about the various ML/AI techniques, get a small data set and try to implement some of those algorithms. Each one will have its strengths and weaknesses, so I would recommend that you try to combine them using Naive Bays classifier (or something similar).

My Experience:
I'm working on the problem for my Masters Thesis so I'll pitch my results using Genetic Programming: www.twitter.com/darwins_finches

I started live trading with real money in 09/09/09.. yes, it was a magical day! I post the GP's predictions before the market opens (i.e. the timestamps on twitter) and I also place the orders before the market opens. The profit for this period has been around 25%, we've consistently beat the Buy & Hold strategy and we're also outperforming the S&P 500 with stocks that are under-performing it.

Some Resources:
Here are some resources that you might want to look into:

• Max Dama's blog: http://www.maxdama.com/search/label/Artificial%20Intelligence
• My blog: http://mlai-lirik.blogspot.com/
• AI Stock Market Forum: http://www.ai-stockmarketforum.com/
• Weka is a data mining tool with a collection of ML/AI algorithms: http://www.cs.waikato.ac.nz/ml/weka/

The Chatter:
The general consensus amongst "financial people" is that Artificial Intelligence is a voodoo science, you can't make a computer predict stock prices and you're sure to loose your money if you try doing it. None-the-less, the same people will tell you that just about the only way to make money on the stock market is to build and improve on your own trading strategy and follow it closely.

The idea of AI algorithms is not to build Chip and let him trade for you, but to automate the process of creating strategies.

Fun Facts:
RE: monkeys can pick better than most experts
Apparently rats are pretty good too!

Solution 3

I understand monkeys can pick better than most experts, so why not an AI? Just make it random and call it an "advanced simian Mersenne twister AI" or something.

Solution 4

Much more money is made by the sellers of "money-making" systems then by the users of those systems.

Instead of trying to predict the performance of companies over which you have no control, form a company yourself and fill some need by offering a product or service (yes, your product might be a stock-predicting program, but something a little less theoretical is probably a better idea). Work hard, and your company's own value will rise much quicker than any gambling you'd do on stocks. You'll also have plenty of opportunities to apply programming skills to the myriad of internal requirements your own company will have.

Solution 5

If you want to go down this long, dark, lonesome road of trying to pick stocks you may want to look into data mining techniques using advanced data mining software such as SPSS or SAS or one of the dozen others.

You'll probably want to use a combination or technical indicators and fundamental data. The data will more than likely be highly correlated so a feature reduction technique such as PCA will be needed to reduce the number of features.

Also keep in mind your data will constantly have to be updated, trimmed, shuffled around because market conditions will constantly be changing.

I've done research with this for a grad level class and basically I was somewhat successful at picking whether a stock would go up or down the next day but the number of stocks in my data set was fairly small (200) and it was over a very short time frame with consistent market conditions.

What I'm trying to say is what you want to code has been done in very advanced ways in software that already exists. You should be able to input your data into one of these programs and using either regression, or decision trees or clustering be able to do what you want to do.

View more solutions

Comments

[David Thornley]

Remember that there are a large number of highly intelligent and highly educated people who work very hard on stock market prediction, and apparently not doing better than relatively uneducated monkeys, so you may well not find any patterns to exploit. The project is an interesting idea, but you might want to look for a field where you're more confident of finding patterns.

[Kiril]

@David Algorithmic trading companies reported about $21 billion profits in 2008... if people really were not much better than relatively uneducated monkeys, then I have a bridge to sell you. There are patterns in trading: one pattern is that many people loose money on the market and their loss is a profit for somebody.

[akaphenom]

I have at my disposal a unique dataset that is not publiclly avaialble. I am not trying to figure out new applications of stochastic oscilators, or anything like that. It wasn't a question of should I (because I am going to) - but how to best go about it.

[Kiril]

@akaphenom See my answer for the various branches of ML/AI, read up on them and play around with them... there is no single right methodology or technique, it's really about what you find to be most comfortable for you and what brings you the best results. You can apply various classifiers, genetic algorithms, neural nets and many other techniques... all of them produce varying results and the approach of the winning team for the Netflix prize (which you may find applicable) was to aggregate all of the predictions.

[akaphenom]

I think this is more of a quesiton of me wanting the experience of building it or thinking through the system. When learning a new language (F#) it always is more fun to have a project to try and apply it to. And better yet - if I can acquire some knowledge on machine learning at the same time all the better. here exists (from a stocking picking contests) a whole bunch of dated recommendations (buy this / sell that). Each individual contributor either did fundamental analyis or picks stocks they are familiar with. My app should detect which relationships are important...

[akaphenom]

does anyody think this problem subjects itself to the supervised machine learnign domain? Or am that far off?

[BlueTrin]

where did you find your data ?

[Kiril]

@BlueTrin Yahoo Finance and the Cunningham T4 API: ctsfutures.com/t4_api.aspx

[BlueTrin]

I wanted to use Java and Meka because they are mature technologies, but the geek within me wants to use F# for the pure awesomeness of using something functional and new.

[BlueTrin]

Supervised machine learning would involve teaching him, since you do not seem to be a good trader yourself, it is more likely you want to code up a backtesting and use the profits and losses function (PL function) as a signal to tell the algorithm that it is doing well. Therefore it seems more to me that it is unsupervised learning ...

[usumoio]

Using the machines to pick the strategies and then working from there is the way to go.

[mjs]

how would one get hold of enough data to mine? What sources did your students use? Thanks.

[Neil McGuigan]

@momo Our university has access to lots of historical news stories. I helped them write scripts to grab news stories about a particular company and match that with stock market data.