Yule–Walker equations

Yule–Walker equations
Name	Yule–Walker equations
Field	Time series analysis, Statistics
Namedafter	Udny Yule, Gilbert Walker
Relatedconcepts	Autoregressive model, Partial autocorrelation function, Levinson recursion

Yule–Walker equations are a set of linear equations that relate the autocorrelation coefficients of a stationary process to the parameters of an autoregressive model. They provide a fundamental method for estimating the coefficients of an autoregressive process by solving a system derived from the Wiener–Khinchin theorem. These equations are central to time series analysis and spectral estimation, forming the theoretical basis for algorithms like the Levinson–Durbin algorithm.

Definition and formulation

For a stationary autoregressive model of order \( p \), denoted AR(\( p \)), the Yule–Walker equations are a system of \( p+1 \) equations. The system is typically written using the autocovariance function \( \gamma(\tau) \) or the autocorrelation function \( \rho(\tau) \). The core formulation expresses each autocorrelation at lag \( k \) as a linear combination of the AR parameters \( \phi_1, \ldots, \phi_p \) and the autocorrelations at previous lags, plus a term for the white noise variance \( \sigma^2 \). This structure directly arises from multiplying the AR equation by a lagged value of the process and taking expectations, leveraging properties of ergodicity and stationarity. The mathematical elegance of the formulation connects the model's parameters directly to the second-order moments of the observed data, a principle also seen in the Wold's theorem.

Derivation

The derivation begins with the standard definition of an AR(\( p \)) process, where the current value is a linear combination of \( p \) past values plus a white noise innovation. By multiplying both sides of this defining equation by \( X_{t-k} \) for \( k \ge 0 \) and applying the expected value operator, one obtains equations involving the autocovariance. Using the property that the innovation is uncorrelated with past values of the process for positive lags simplifies the system. For \( k = 0 \), the equation includes the variance of the innovation, leading to the complete Yule–Walker system. This derivation mirrors techniques used in linear prediction theory and is foundational for understanding spectral density estimation, as it relates the model's parameters to the power spectrum via the Wiener–Khinchin theorem.

Applications in autoregressive models

The primary application of the Yule–Walker equations is in fitting autoregressive models to observed time series data. By replacing the theoretical autocorrelations with their sample estimates, one obtains the Yule–Walker estimates of the AR parameters. This method is extensively used in fields like econometrics, signal processing, and geophysics. For instance, in speech processing, the equations underpin linear predictive coding, a key technique for audio compression. In climatology, they are used to model phenomena like the Southern Oscillation, originally studied by Gilbert Walker. The framework also facilitates the computation of the partial autocorrelation function, which provides a critical tool for model order selection in the Box–Jenkins method.

Estimation and algorithm

Estimation via the Yule–Walker equations involves solving a Toeplitz system of linear equations where the matrix elements are the sample autocorrelations. The Levinson–Durbin algorithm provides an efficient \( O(p^2) \) recursive solution to this system, exploiting the Toeplitz structure. This algorithm is numerically stable and simultaneously yields the partial autocorrelation coefficients, which are useful for order determination. The estimated parameters are consistent under the assumption of a true AR(\( p \)) process, a result supported by the ergodic theorem. However, the method of moments approach inherent in this estimation can be suboptimal compared to maximum likelihood estimation or the Burg's method, especially for short data records.

Properties and limitations

The Yule–Walker estimator yields a stationary AR model by construction, as the solution guarantees that the estimated polynomial has roots inside the unit circle. This property is desirable for forecasting and spectral analysis. The estimator is also asymptotically consistent under standard conditions. A significant limitation is that it can be biased for finite samples, particularly for processes with roots near the unit circle. The method's reliance on the sample autocorrelation function makes it sensitive to the choice of the biased estimator of autocorrelation, which can degrade performance. Furthermore, for processes that are not purely autoregressive, such as those with moving average components, the Yule–Walker estimates may be inefficient compared to procedures like those in the ARMA model framework.

Extensions and related equations

The Yule–Walker framework has been extended to more general models. For ARMA models, the equations generalize to the modified Yule–Walker equations, which involve both autoregressive and moving average parameters. In multivariate settings, the vector autoregression model employs a block version of these equations. Related concepts include the Prony's method for exponential fitting and the Pisarenko harmonic decomposition for sinusoidal retrieval. The equations are also connected to the theory of orthogonal polynomials on the unit circle and to the Schur algorithm in complex analysis. Their principles influence modern techniques in machine learning, such as Gaussian process regression for temporal data.

Category:Time series analysis Category:Equations Category:Statistical models