We try to appreciate the quality of time series forecasts with different periodicities. This game gives each participant the opportunity to evaluate his forecasts comparing them with those given by the other players.
Actually three periodicities are retained (annual, quarterly and monthly) and each one corresponds to a tournament. The game is thus made of three tournaments, each participant being free to complete one, two or all tournaments.
After the opening date of the game, each player is able to download from this site three files containing the data, each one corresponding to one of the retained periodicities and filled with 12 series. Until the deadline, each participant can upload his forecasts and eventually make corrections on previous responses. At the closing date, the last forecasts uploaded on the site by a participant will be considered as his final answers.
According to periodicity of the data, a different number of forecasts must be completed:

1. Annual data : 4 forecasts
2. Quarterly data : 8 forecasts
3. Monthly data : 12 forecasts

Answers are relative to out-of-sample forecasts: the objective is to give predictions about realizations which follow the last observed value of each series. For example, for the tournament with quarterly data, if one variable is observed from time t=1 to time t=T then the forecasts must be given for time t=T+1 to t=T+8.
At the closing date, a ranking of the responses will be published for each tournament and an overall ranking will also be calculated for the players who participate in the three tournaments. For these rankings only participants who give all answers for a tournament will be considered, that is to say those who give forecasts about the twelve annual variables, or the twelve quarterly variables or the twelve monthly variables and about the 36 variables for the overall ranking.
We will ask the first three of these rankings to give a brief description of the methodology they used and this description will be published with the results at the end of the game. At that time, the complete data file containing observations initially masked and meaning of the various variables will also be downloadable.

Participants are people who have completed the registration form on this site. A participant can be a single person or a team. In the latter case, the inscription form authorizes the registration of the names of all team members. However, only one email address can be given by a member of the team who will then be our only correspondent.

Evaluation of the forecasts

Rankings will be based on the SMAPE (Symmetrical Mean Absolute Percentage Error). This statistics is often used to evaluate the performance of forecasts. For a given forecasting horizon, the formula for its computation is given below:

In which n is the number of variables in a tournament (here, n=12), At is the realization of one of these variables at time t, and Ft is the forecast for this date.
Other statistics could be calculated (MAPE, RMSE…)

BenchMarks

Forecasts given by the time series forecasting system FWS are used as references, allowing participants to compare their proposals with automatic forecasting algorithms implemented in this professional product. Consequently, we ask participants of the game not to use the free version of this software to build their own forecasts.

Game schedule

Registration and Login

The various stages of the game are the following ones:

February 15th, 2010 :

Opening of the registrations, authorization is given to download the data files by the participants and to upload or to correct their forecasts for every variable included in one of the 3 tournaments.

May 15th, 2010 :

Closing of the possibilities to fill or to correct forecasts already uploaded.

End of May:

Rankings of each tournament and the overall classification are published on the site of the game.

In these rankings, and only if they agree, the name of the three best players will be displayed. For other participants only the pseudo they have chosen will be published.

When you register, you are asked:

1. To give your name and first name,
2. To specify a nickname that will be specifically used in the publication of the rankings, allowing you to maintain your anonymity in these rankings,
3. To provide a valid email address which will be your identifier for each login and we can send information to (e.g. in case you forget your password),
4. To select a password: it is personal and certifies that the answers in your log entries are yours,
5. To indicate your curriculum/training (Master, University,…),
6. To prohibit or authorize the publication of your name and your formation: if you give your permission, they will appear in the results list if you are in the top 3 of one of the three tournaments or of the overall game,
7. A statement in which you agree not to seek to know the origin of data used in the game or use the realizations of the series put into play after the closing date if, by any chance, you happen to know it, and not use the free version of the product FWS Kresterion to build your own forecasts.