ECON60052 · Lecture 4

Advanced panel data methods

Beyond two periods: the first-difference, fixed-effects and random-effects estimators, what each assumes, when they coincide, and how the Hausman and Mundlak tests choose between them. The fixed-effects within transformation is the key tool for removing unobserved heterogeneity.

With more than two periods there is a choice of estimators for the unobserved-effects model. All aim to deal with a time-constant unobservable $a_i$; they differ in what they assume about it and in their efficiency.

The unobserved-effects model

For a balanced panel with $T\ge2$, the single-regressor model is

$$y_{it}=a_i+\beta x_{it}+u_{it},\qquad i=1,\dots,N;\ t=1,\dots,T,$$

with $a_i$ the fixed effect and $u_{it}$ the idiosyncratic error. Three estimators follow: first difference (FD), fixed effects (FE) and random effects (RE).

First differencing when T > 2

FD is OLS on the differenced equation $\Delta y_{it}=\Delta \mathbf{x}_{it}\boldsymbol\beta+\Delta u_{it}$, losing the first period for each unit ($N(T-1)$ observations). When $T=2$ it reduces to the two-period estimator. The catch for $T\gt2$: for the differenced errors to be serially uncorrelated we need $u_{it}$ to be a random walk. You can test this by regressing $\Delta \hat u_{it}$ on $\hat u_{i,t-1}$,

$$\Delta u_{it}=\rho\,u_{i,t-1}+e_{it},$$

and testing $H_0:\rho=0$ with a Dickey–Fuller-type test (possible only if $T\ge3$).

The fixed-effects (within) estimator

Average the model over time for each unit, $\bar y_i=a_i+\beta \bar x_i+\bar u_i$, and subtract. Because $a_i$ is constant over $t$ it cancels:

$$\ddot y_{it}=\beta\,\ddot x_{it}+\ddot u_{it},\qquad \ddot y_{it}\equiv y_{it}-\bar y_i.$$

Only within-unit variation survives. OLS on the time-demeaned data is the FE, within or LSDV estimator:

$$\hat\beta_w=\frac{\sum_i\sum_t (x_{it}-\bar x_i)(y_{it}-\bar y_i)}{\sum_i\sum_t (x_{it}-\bar x_i)^2},\qquad \hat a_i=\bar y_i-\hat\beta_w\bar x_i.$$

Degrees of freedom. The demeaning implicitly estimates the $N$ fixed effects, so the correct residual degrees of freedom are $N(T-1)-k$, not $NT-k$. Standard errors from naive OLS on the demeaned data must be scaled by $\sqrt{(NT-k)/(N(T-1)-k)}\approx\sqrt{T/(T-1)}$; panel software does this automatically. Adding one dummy per individual (dropping the constant) gives numerically identical estimates — the LSDV equivalence.

FE's great strength is that it allows arbitrary correlation between $a_i$ and the regressors under strict exogeneity $\mathbb{E}(u_{it}\mid \mathbf{x}_i,a_i)=0$. Its cost: any time-constant regressor (gender, ethnicity) is swept away because its demeaned value is zero, and variables whose change is constant for everyone (age with a full set of year dummies) drop out through collinearity.

FE versus FD

$T=2$: FE and FD are identical.
$T\ge3$: they differ but are usually close. If the idiosyncratic errors are serially uncorrelated, neither dominates; if the differenced errors are serially uncorrelated (i.e. $u_{it}$ is a random walk), FD is efficient.
FD typically loses more data in unbalanced panels, so FE is often preferred there. If the panel is unbalanced for non-random reasons, both are biased.

The random-effects estimator

RE assumes the unobserved effect is uncorrelated with every regressor, $\operatorname{Cov}(\mathbf{x}_{it},a_i)=0$. Then pooled OLS is consistent but inefficient, because the composite error $v_{it}=a_i+u_{it}$ is serially correlated within a unit:

$$\operatorname{Corr}(v_{it},v_{is})=\frac{\sigma_a^2}{\sigma_a^2+\sigma_u^2}\quad (t\neq s).$$

The efficient response is GLS, implemented by quasi-demeaning: subtract only a fraction $\lambda$ of each unit's time-mean,

$$y_{it}-\lambda \bar y_i=(1-\lambda)\beta_0+\beta\,(x_{it}-\lambda \bar x_i)+(v_{it}-\lambda \bar v_i),\qquad \lambda=1-\sqrt{\frac{\sigma_u^2}{\sigma_u^2+T\sigma_a^2}}.$$

The transformation interpolates between the estimators you already know: $\lambda=0$ (when $\sigma_a^2=0$) gives pooled OLS, $\lambda=1$ gives FE, and $\lambda\to1$ as $T\to\infty$. Crucially, for any $\lambda\lt1$ the time-constant regressors survive, so RE can estimate the effect of gender, ethnicity or education — a genuine advantage when those are of interest.

Choosing between FE and RE: the Hausman test

Everything hinges on whether $a_i$ is correlated with the regressors. The canonical case against RE is wages and schooling: unobserved motivation is surely correlated with schooling, so RE is inconsistent while FE is not. The Hausman test compares the two coefficient vectors:

$$(\hat{\boldsymbol\beta}_w-\hat{\boldsymbol\beta}_g)'\big[\mathbf{V}(\hat{\boldsymbol\beta}_w)-\mathbf{V}(\hat{\boldsymbol\beta}_g)\big]^{-1}(\hat{\boldsymbol\beta}_w-\hat{\boldsymbol\beta}_g)\sim\chi^2(k).$$

Under $H_0$ (no correlation) RE is consistent and efficient while FE is consistent either way, so the two estimates should be close and the statistic small. A large value rejects RE in favour of FE.

Correlated random effects and Mundlak's trick

Model the fixed effect as linear in the unit means, $a_i=\alpha+\gamma\bar x_i+r_i$ with $r_i$ uncorrelated with the regressors, and substitute:

$$y_{it}=\alpha+\beta x_{it}+\gamma\bar x_i+r_i+u_{it}.$$

Running RE on this augmented equation reproduces the FE slope exactly (Mundlak, 1976), so testing $H_0:\gamma=0$ is an alternative Hausman test — with the correct degrees of freedom.

Worked example · panel wage regression (T = 2)

On a two-wave household panel of 2,145 individuals ($\log$ real hourly pay on region, age, marital status, occupation, union coverage and time interactions):

FE and FD coincide (because $T=2$) — e.g. coverage 0.1006, the male–year interaction $-0.043$. The time-constant male dummy is dropped for collinearity.
Pooled OLS recovers male (0.428) because it uses between-person variation, but is biased if $a_i$ is correlated with the regressors.
Random effects also estimates male (0.402), quasi-demeaning with $\rho\approx0.50$.
The Hausman test gives $\chi^2(7)=157.7$ (p < 0.001): reject RE, use FE. The gap is driven by manual (FE $-0.026$ vs RE $-0.250$) and covered — unobserved ability is correlated with holding a manual or covered job.
The Mundlak regression reaches the same verdict with the correct degrees of freedom ($\chi^2(6)=158.0$) and its within coefficients match FE exactly.

The cost of FE and FD

These estimators buy consistency with efficiency. A whole cross-section is consumed estimating the fixed effects; they rely entirely on "changers", so if a characteristic barely moves over time the estimate is imprecise; and if it never changes (gender, ethnicity), its effect cannot be estimated at all — which is why FE cannot study, for example, a pure gender wage gap. There is a genuine trade-off between the consistency of FE and the efficiency of RE, sharpened by the fact that differencing and demeaning worsen any measurement error.

Self-check questions

With T = 2, why are the FE and FD coefficient estimates identical?

When $T=2$, $\bar y_i=(y_{i1}+y_{i2})/2$, so the demeaned value $\ddot y_{i2}=y_{i2}-\bar y_i=\tfrac12(y_{i2}-y_{i1})=\tfrac12\Delta y_{i2}$ and $\ddot y_{i1}=-\tfrac12\Delta y_{i2}$. The same factor $\tfrac12$ multiplies both $\ddot y$ and $\ddot x$, so it cancels in the ratio $\hat\beta_w=\sum \ddot x\,\ddot y/\sum \ddot x^2$, leaving exactly the FD slope. The empirical example confirms it: every FE coefficient equals its FD counterpart, and male is dropped for collinearity in both.

Why is male omitted under FE but estimated (≈0.40–0.43) under pooled OLS and RE?

male is time-constant, so its demeaned value $\ddot m_{it}=0$ for everyone — FE sweeps it out entirely. Pooled OLS uses the raw (un-demeaned) data, so between-person variation in gender is available and its coefficient is estimated (0.428). RE only quasi-demeans, subtracting a fraction $\lambda\lt1$ of the mean, so time-constant regressors survive and male is estimated (0.402). Being able to keep fixed characteristics is a key practical advantage of RE.

The Hausman statistic is χ²(7) = 157.7 (p < 0.001). What do you conclude, and which regressors drive it?

Reject the null that $a_i$ is uncorrelated with the regressors. So the RE assumption fails, RE is inconsistent, and FE (consistent either way) should be used. Inspecting the FE–RE differences, the gap is dominated by manual (FE $-0.026$ vs RE $-0.250$) and covered: unobserved ability is correlated with being in a manual or covered job, exactly the correlation RE rules out. The Mundlak regression reaches the same conclusion with the correct degrees of freedom ($\chi^2(6)=158.0$).

Related notes

Lecture 3: Policy analysis with panel data — the two-period case and difference-in-differences.
Lecture 2: Heteroskedasticity & clustering — why panel errors need cluster-robust standard errors.
ECON60052 hub · Econometrics explained.

Download

Lecture 4 slides · Advanced panel data methods

First difference, fixed effects, random effects, the Hausman test and Mundlak's correlated-random-effects trick.

Download PDF

Studying this for a module?

I tutor economics, econometrics and maths at university and postgraduate level — from weekly problem sets and past exams to dissertation methods. Book a free 30–45 minute consultation to talk through exactly where you are.

Book consultation Pricing & how it works