Laplace's approximation

Part of a series on
Bayesian statistics
Posterior = Likelihood × Prior ÷ Evidence
Background
Bayesian inference Bayesian probability Bayes' theorem Bernstein–von Mises theorem Coherence Cox's theorem Cromwell's rule Principle of indifference Principle of maximum entropy
Model building
Weak prior ... Strong prior Conjugate prior Linear regression Empirical Bayes Hierarchical model
Posterior approximation
Markov chain Monte Carlo Laplace's approximation Integrated nested Laplace approximations Variational inference Approximate Bayesian computation
Estimators
Bayesian estimator Credible interval Maximum a posteriori estimation
Evidence approximation
Evidence lower bound Nested sampling
Model evaluation
Bayes factor Model averaging Posterior predictive
Mathematics portal
v t e

Laplace's approximation provides an analytical expression for a posterior probability distribution by fitting a Gaussian distribution with a mean equal to the MAP solution and precision equal to the observed Fisher information.^[1][2] The approximation is justified by the Bernstein–von Mises theorem, which states that under regularity conditions the posterior converges to a Gaussian in large samples.^[3][4]

For example, a (possibly non-linear) regression or classification model with data set ${\displaystyle \{x_{n},y_{n}\}_{n=1,\ldots ,N}}$ comprising inputs ${\displaystyle x}$ and outputs ${\displaystyle y}$ has (unknown) parameter vector ${\displaystyle \theta }$ of length ${\displaystyle D}$. The likelihood is denoted ${\displaystyle p({\bf {y}}|{\bf {x}},\theta )}$ and the parameter prior ${\displaystyle p(\theta )}$. Suppose one wants to approximate the joint density of outputs and parameters ${\displaystyle p({\bf {y}},\theta |{\bf {x}})}$

${\displaystyle p({\bf {y}},\theta |{\bf {x}})\;=\;p({\bf {y}}|{\bf {x}},\theta )p(\theta |{\bf {x}})\;=\;p({\bf {y}}|{\bf {x}})p(\theta |{\bf {y}},{\bf {x}})\;\simeq \;{\tilde {q}}(\theta )\;=\;Zq(\theta ).}$

The joint is equal to the product of the likelihood and the prior and by Bayes' rule, equal to the product of the marginal likelihood ${\displaystyle p({\bf {y}}|{\bf {x}})}$ and posterior ${\displaystyle p(\theta |{\bf {y}},{\bf {x}})}$. Seen as a function of ${\displaystyle \theta }$ the joint is an un-normalised density. In Laplace's approximation we approximate the joint by an un-normalised Gaussian ${\displaystyle {\tilde {q}}(\theta )=Zq(\theta )}$, where we use ${\displaystyle q}$ to denote approximate density, ${\displaystyle {\tilde {q}}}$ for un-normalised density and ${\displaystyle Z}$ is a constant (independent of ${\displaystyle \theta }$). Since the marginal likelihood ${\displaystyle p({\bf {y}}|{\bf {x}})}$ doesn't depend on the parameter ${\displaystyle \theta }$ and the posterior ${\displaystyle p(\theta |{\bf {y}},{\bf {x}})}$ normalises over ${\displaystyle \theta }$ we can immediately identify them with ${\displaystyle Z}$ and ${\displaystyle q(\theta )}$ of our approximation, respectively. Laplace's approximation is

${\displaystyle p({\bf {y}},\theta |{\bf {x}})\;\simeq \;p({\bf {y}},{\hat {\theta }}|{\bf {x}})\exp {\big (}-{\tfrac {1}{2}}(\theta -{\hat {\theta }})^{\top }S^{-1}(\theta -{\hat {\theta }}){\big )}\;=\;{\tilde {q}}(\theta ),}$

where we have defined

${\displaystyle {\begin{aligned}{\hat {\theta }}&\;=\;\operatorname {argmax} _{\theta }\log p({\bf {y}},\theta |{\bf {x}}),\\S^{-1}&\;=\;-\left.\nabla _{\theta }\nabla _{\theta }\log p({\bf {y}},\theta |{\bf {x}})\right|_{\theta ={\hat {\theta }}},\end{aligned}}}$

where ${\displaystyle {\hat {\theta }}}$ is the location of a mode of the joint target density, also known as the maximum a posteriori or MAP point and ${\displaystyle S^{-1}}$ is the ${\displaystyle D\times D}$ positive definite matrix of second derivatives of the negative log joint target density at the mode ${\displaystyle \theta ={\hat {\theta }}}$. Thus, the Gaussian approximation matches the value and the curvature of the un-normalised target density at the mode. The value of ${\displaystyle {\hat {\theta }}}$ is usually found using a gradient based method, e.g. Newton's method. In summary, we have

${\displaystyle {\begin{aligned}q(\theta )&\;=\;{\cal {N}}(\theta |\mu ={\hat {\theta }},\Sigma =S),\\\log Z&\;=\;\log p({\bf {y}},{\hat {\theta }}|{\bf {x}})+{\tfrac {1}{2}}\log |S|+{\tfrac {D}{2}}\log(2\pi ),\end{aligned}}}$

for the approximate posterior over ${\displaystyle \theta }$ and the approximate log marginal likelihood respectively.^[5] In the special case of Bayesian linear regression with a Gaussian prior, the approximation is exact. The main weaknesses of Laplace's approximation are that it is symmetric around the mode and that it is very local: the entire approximation is derived from properties at a single point of the target density. Laplace's method is widely used and was pioneered in the context of neural networks by David MacKay,^[6] and for Gaussian processes by Williams and Barber.^[7]

References