M3-Competition Final Results Reported in the International Journal of Forecasting

Final results from the M-3 forecasting competition have been reported by Professors Spiros Makridakis and Michele Hibon of the French business school INSEAD.  Designed to evaluate the accuracy of different forecasting methods, the competition was sponsored by the International Journal of Forecasting and is the largest and most comprehensive empirical forecasting study ever performed.  The most striking result was the performance of the fully-automated Forecast Pro, which significantly outperformed all of the other software approaches as well as 18 out of 19 academic teams.

The study compared the accuracy of 26 different approaches used to prepare 3,003 forecasts based on historic demand data.  The data were selected to cover as wide of a range of data types as possible (e.g., microeconomic, macroeconomic, industrial, financial and demographic) and included weekly, monthly, quarterly and annual series.  Each entrant was free to use any method he or she wished to prepare the forecasts.  Nineteen of the approaches were implemented by forecasting experts from academic institutions such as the Wharton School, Case Western Reserve, INSEAD and the Imperial College (London).  Approaches included techniques like exponential smoothing models, Box-Jenkins models, neural networks and rule-based approaches.  Human judgment and statistical expertise played a significant role in many of the approaches.  The remaining seven approaches were implemented using commercially available, fully automated forecasting packages—Forecast Pro, SmartForecasts, ForecastX, Autocast and Autobox (three variations).  The researchers evaluated the forecasts against the actual future values of the series.

The results surprised many who assumed that a computer program would not be able to outperform human experts.  The results did not surprise Dr. Robert Goodrich, the author of Forecast Pro and president of Business Forecast Systems.  According to Goodrich, “Over the last ten years, our clients have used Forecast Pro to forecast hundreds of millions of time series.  Whenever unusual forecasts are reported, we carefully analyze the situation and make improvements to our algorithms.  While it is difficult to fine-tune a subjective human decision-making process, it is fairly straightforward to fine-tune quantitative algorithms in a computer program.  The result is that Forecast Pro recognizes and responds appropriately to many more special circumstances than a human practitioner would have encountered and would be able to keep track of.”  Goodrich also noted that the M-3 competition is the second time that Forecast Pro has outperformed human practitioners and other forecasting packages in a formal study.  The first time was in a much smaller study conducted by Keith Ord and Sam Lowe of Penn State and published in the February, 1996 issue of The American Statistician.

Final results from the M-3 competition have been published in a special issue of the International Journal of Forecasting. 

Table adapted from Hibon M. and Makridakis S. [2000]. The M3 Competition: results, conclusions and implications. International Journal of Forecasting 16, 451-476.

Figures in bold represent first place finishes.

^* Entries submitted following publication of initial results (a considerable advantage).