Estimation, the sample ACF
& model selection

The sample autocorrelation function

The theoretical ACF \(\rho(k)\) is unknown; we estimate it from the realisation. The sample autocorrelation at lag \(k\) is

$$\hat{\rho}_T(k)=\frac{\sum_{t=k+1}^{T}(y_t-\bar y)(y_{t-k}-\bar y)}{\sum_{t=1}^{T}(y_t-\bar y)^2}, \qquad \bar y=\frac{1}{T}\sum_{t=1}^{T}y_t,$$

the sample autocovariance at lag \(k\) divided by the sample variance. If \(Y_t\) is stationary, \(\hat\rho_T(k)\) is consistent for \(\rho(k)\); if \(Y_t\) is white noise it converges in probability to 0 at every lag. Plotting \(\hat\rho_T(k)\) against \(k\) produces the correlogram, the single most useful diagnostic in univariate time series.

The correlogram and its confidence bands

The reason the correlogram is so useful is that we know the sampling distribution of \(\hat\rho_T(k)\) under the null of no autocorrelation. If \(Y_t\) is white noise then, by a central limit theorem,

$$\sqrt{T}\,\hat\rho_T(k)\ \xrightarrow{\ d\ }\ N(0,1) \qquad \text{for each } k=1,2,\ldots$$

To test \(H_0:\rho(k)=0\) against \(H_A:\rho(k)\neq0\) we compare \(\sqrt{T}\,\hat\rho_T(k)\) with the standard normal. Since \(\Pr\{-1.96 < Z < 1.96\}=0.95\),

$$\Pr\left\{-1.96\le\sqrt{T}\,\hat\rho_T(k)\le1.96\right\}\approx0.95,$$

so we reject \(H_0\) at the 5% level whenever \(|\hat\rho_T(k)|>1.96/\sqrt{T}\). Those horizontal lines at \(\pm1.96/\sqrt{T}\) are the dotted bands drawn on every correlogram: a spike poking outside them is statistically significant autocorrelation at that lag.

The portmanteau (Q) test

Rather than test each lag separately we often want a joint test of \(H_0:\rho(1)=\cdots=\rho(L)=0\) against the alternative that at least one is non-zero. Under the null the scaled sample autocorrelations are asymptotically independent standard normals, so the sum of their squares is chi-squared:

$$Q=\sum_{j=1}^{L}\left(\sqrt{T}\,\hat\rho_T(j)\right)^2=T\sum_{j=1}^{L}\hat\rho_T(j)^2\ \xrightarrow{\ d\ }\ \chi^2_L.$$

We reject when \(Q\) exceeds the \(\chi^2_L\) critical value; software reports the associated p-value \(\Pr\{\chi^2_L>Q\}\) in the correlogram's "Prob" column. This portmanteau statistic (in its Box–Pierce and Ljung–Box forms) is the standard omnibus check for whether any predictable linear structure remains.

Estimating AR and ARMA models

An AR(p) is linear in observable regressors, so it is estimated by ordinary least squares: regress \(Y_t\) on a constant and \(Y_{t-1},\ldots,Y_{t-p}\). For the AR(1), \(Y_t=\alpha+\phi_1 Y_{t-1}+\varepsilon_t\), the OLS slope is

$$\hat\phi_1=\frac{\sum_{t=2}^{T}(y_t-\bar y)(y_{t-1}-\bar y_{-1})}{\sum_{t=2}^{T}(y_{t-1}-\bar y_{-1})^2}.$$

Provided \(|\phi_1|<1\) and \(\varepsilon_t\) is genuinely white noise, \(\hat\phi_1\) is consistent, \(\hat\phi_1\xrightarrow{p}\phi_1\), and asymptotically normal,

$$\sqrt{T}\,\frac{\hat\phi_1-\phi_1}{\mathrm{sd}(\hat\phi_1)}\ \xrightarrow{\ d\ }\ N(0,1),$$

so the usual \(t\)- and \(F\)-tests are valid in large samples. The key assumption is that the errors are serially uncorrelated: if they are not, \(Y_{t-1}\) becomes correlated with \(\varepsilon_t\) (through \(\varepsilon_{t-1}\)), OLS loses its exogeneity, and the model is misspecified.

A pure MA or mixed ARMA model cannot be estimated by linear regression, because the lagged errors \(\varepsilon_{t-1},\ldots\) are unobserved. Instead the errors are built up recursively from a starting value \(\varepsilon_0=0\) and the coefficients estimated by iterating a least-squares/maximum-likelihood scheme until successive estimates agree. The important practical point is that the resulting estimators are again asymptotically normal, so inference proceeds with the familiar \(t\)- and \(F\)-tests.

Choosing the order: AIC and SIC

Richer models always fit the sample better, so order selection trades goodness of fit against parsimony using an information criterion. With residual variance estimate \(\hat\sigma^2=\mathrm{RSS}/T^*\) and \(p+q+1\) estimated coefficients,

$$\mathrm{AIC}=\log\hat\sigma^2+\frac{2(p+q+1)}{T^*}, \qquad \mathrm{SIC}=\log\hat\sigma^2+\frac{(p+q+1)\log T^*}{T^*}.$$

Both add a penalty that grows with the number of parameters; the Schwarz criterion (SIC/BIC) penalises more heavily, since \(\log T^*>2\) once the sample is moderately large. We compute the criterion over a grid \(0\le p\le p_{\max},\ 0\le q\le q_{\max}\) and pick the minimiser. As a result SIC tends to choose more parsimonious models than AIC (\(p_{\mathrm{AIC}}\ge p_{\mathrm{SIC}}\)); if the true process is a finite AR, SIC selects its order consistently as \(T^*\to\infty\), whereas AIC may over-specify.

Testing the residuals for serial correlation

A well-specified dynamic model should leave white-noise residuals. Suppose we have fitted an AR(p) but the errors are actually AR(m); then the correct model is an AR(p+m). This motivates the Breusch–Godfrey (LM) test: estimate the model, save residuals \(e_t\), then run the auxiliary regression

$$y_t=\alpha+\phi_1 y_{t-1}+\cdots+\phi_p y_{t-p}+\psi_1 e_{t-1}+\cdots+\psi_m e_{t-m}+\varepsilon_t,$$

and test \(H_0:\psi_1=\cdots=\psi_m=0\) against the alternative that at least one \(\psi_j\neq0\). The test statistic is asymptotically \(\chi^2_m\) under the null (an \(F\)-version performs better in finite samples). A rule of thumb sets \(m=4\) for quarterly data and \(m=12\) for monthly data, so the test also catches residual seasonality. Note that the Durbin–Watson statistic is not valid once lagged dependent variables are present, so it should not be used here.

A worked example: UK GDP growth

Take quarterly UK GDP growth over roughly three decades, \(T=116\) observations. The lag-1 sample autocorrelation is \(\hat\rho_T(1)=0.582\). The correlogram bands are \(\pm1.96/\sqrt{116}=\pm0.182\); since \(0.582>0.182\) we comfortably reject \(\rho(1)=0\). The first few autocorrelations sit outside the bands and the joint Q-test of \(\rho(1)=\rho(2)=\rho(3)=0\) has a p-value of essentially zero, so growth is clearly not white noise. The positive, quickly-decaying pattern points to an AR(1).

Fitting candidates confirms it. An MA(2) has all coefficients significant, but a Breusch–Godfrey test to \(m=4\) finds significant residual serial correlation — the model is inadequate. An AR(1) has a highly significant constant and slope and leaves residuals with no significant serial correlation. Both AIC and SIC select the AR(1). This is the whole workflow in miniature: read the correlogram, estimate rival models, test the residuals, and let the information criteria break ties.

Check your understanding

A correlogram from \(T=100\) observations shows \(\hat\rho_T(1)=0.31\). Is it significant at 5%?

The 5% band is \(\pm1.96/\sqrt{100}=\pm0.196\). Since \(0.31>0.196\), the lag-1 autocorrelation lies outside the band, so we reject \(H_0:\rho(1)=0\) at the 5% level: there is significant first-order autocorrelation. Equivalently, \(\sqrt{100}\times0.31=3.1>1.96\).

Two models fit a series equally well, but model A has AIC \(-3.20\) / SIC \(-3.05\) and model B has AIC \(-3.24\) / SIC \(-2.98\). Which do you choose?

Choose the model with the smaller criterion. AIC prefers B (\(-3.24<-3.20\)) but SIC prefers A (\(-3.05<-2.98\)). This is the classic disagreement: B is larger and AIC's lighter penalty tolerates the extra parameters, while SIC's heavier \(\log T^*\) penalty favours the more parsimonious A. In practice check the residual diagnostics too — if A already leaves white-noise residuals, its parsimony makes it the safer choice.

Why can't the Durbin–Watson statistic be used to test AR(1) residuals?

Durbin–Watson assumes the regressors are non-stochastic and, crucially, that no lagged dependent variable appears among them. An AR(p) has \(Y_{t-1},\ldots\) as regressors, which violates that condition and biases the DW statistic toward 2 (i.e. toward "no autocorrelation"). The Breusch–Godfrey LM test remains valid with lagged dependent variables and is the correct tool.

Related notes

Downloads