Accuracy Function

Question

I am doing a TS Analysis. What is the difference between these two accuracies:

fit<-auto.arima(tsdata)
fcast<-forecast(fit,6)
accuracy(fcast)             #### First Accuracy

fit<-auto.arima(tsdata)
fcast<-forecast(fit,6)
accuracy(fcast,actual values)  #### Second Accuracy

How does the accuracy function work when I don't specify the actual values in the accuracy function as in the first case.

Secondly what is the right approach to calculate accuracy?

Solution

In this answer I'm assuming you are using the function from the forecast package. The answer lies within accuracy's description:

Returns range of summary measures of the forecast accuracy. If x is provided, the function measures out-of-sample (test set) forecast accuracy based on x-f. If x is not provided, the function only produces in-sample (training set) accuracy measures of the forecasts based on f["x"]-fitted(f). All measures are defined and discussed in Hyndman and Koehler (2006).

In your case x being the second argument of the function. So, in short accuracy(fcst) provides an estimation of the prediction error, based on the training set.

For example: lets assume you have 12 months and predicting 6 ahead. Then if you use accuracy(fcst) you get the error of the model over the 12 months (only).

Now, let's assume x = real demand for the 6 months you are forecasting. And that you didn't use this data to build the Arima model. In this case accuracy(fcst, x) gives you the test set error, which is a better measure to what you will get in the future using this model (compared to the train set error).

The best practice is to use a test set error because this measure is less prone to bias (you will most likely get "better" prediction results on the training set then on a "hideout" test set, but these results will be a sort of "overfitting"). If you have a test set, you should use the test set as the second argument.

Comments

[Shivam Sarin]

Thanks a lot, Adi ...It helps a lot

[Adi Sarid]

Sure, no problem @ShivamSarin. If you find my answer helpful please mark it as the correct answer.... Thanks.