Backtesting for Future Gains

Backtesting is the process of trying out an investment theory using existing data. Instead of testing out your investment scenario on data that is currently unfolding, which can take years to compile, you can use data that has already been compiled. This allows you to get an immediate analysis of how successful your model is in the real world. This process has its shortcomings because by using past data, you are analyzing how well your theory works in the past, not how well it will work in the future. This can cause problems when you go to apply your theory.

Backtesting Example

An analyst has a theory that generic mutual funds outperform brand name mutual funds. To test this theory over a five-year period, the analyst compiles data from 2005 to 2010. They track all performance of mutual funds over that five year period. Working in the past allows the analyst to complete their model in a day. If an analyst can show that generic funds outperform mutual funds, they may be able to demonstrate their value to a client. In this way, an analyst can close a sale and establish credibility.

Backtesting Benefits

Most analysts believe the business cycle is a predictable and repetitive model. Due to the predictability, they also believe they can analyze the past and establish strategies for the future. Backtesting is the perfect example of this theory. It allows analysts to point out previous trends and show how a particular model would have worked during that trend. In the future, if the trend should arise again, the analysts theory could be applied.

Backtesting Drawbacks

The bigges drawback is that past performance does not necessarily indicate future success for any economic theory. For example, one analyst could use the period of time between 2000 and 2005 to show the profitability of mortgage backed securities. Another analyst could then use the period of time between 2005 and 2010 to show the opposite. Since backtesting only uses a small window in history to test a theory, it often produces a result that is not in keeping with the modern market. If even one factor changes in the model, the result can be largely different.

When to Use Backtesting

Backtesting is most effective when it is applied over very large periods of time. For example, backtesting will show growth stocks, those which operate at a relatively low trading volume, typically outperform stocks that are commonly traded and well-known. Modeling can prove this is true over five-year periods, not just one period in time. In another example, backtesting can show bonds are less risky than stocks, or brand name mutual funds, and do not outperform the market. Keep in mind that backtesting does not work when the scenario is limited to a specific segment in time. The results may be skewed and making trades based on these results is not a good idea.