Mathematical methods used in Bayesian inference and machine learning
For the method of approximation in quantum mechanics, see Variational method (quantum mechanics).
Part of a series on
Bayesian statistics
Posterior = Likelihood × Prior ÷ Evidence
Background
Bayesian inference
Bayesian probability
Bayes' theorem
Bernstein–von Mises theorem
Coherence
Cox's theorem
Cromwell's rule
Principle of indifference
Principle of maximum entropy
Model building
Weak prior ... Strong prior
Conjugate prior
Linear regression
Empirical Bayes
Hierarchical model
Posterior approximation
Markov chain Monte Carlo
Laplace's approximation
Integrated nested Laplace approximations
Variational inference
Approximate Bayesian computation
Estimators
Bayesian estimator
Credible interval
Maximum a posteriori estimation
Evidence approximation
Evidence lower bound
Nested sampling
Model evaluation
Bayes factor
Model averaging
Posterior predictive
Mathematics portal
v
t
e
Variational Bayesian methods are a family of techniques for approximating intractable integrals arising in Bayesian inference and machine learning. They are typically used in complex statistical models consisting of observed variables (usually termed "data") as well as unknown parameters and latent variables, with various sorts of relationships among the three types of random variables, as might be described by a graphical model. As typical in Bayesian inference, the parameters and latent variables are grouped together as "unobserved variables". Variational Bayesian methods are primarily used for two purposes:
To provide an analytical approximation to the posterior probability of the unobserved variables, in order to do statistical inference over these variables.
To derive a lower bound for the marginal likelihood (sometimes called the evidence) of the observed data (i.e. the marginal probability of the data given the model, with marginalization performed over unobserved variables). This is typically used for performing model selection, the general idea being that a higher marginal likelihood for a given model indicates a better fit of the data by that model and hence a greater probability that the model in question was the one that generated the data. (See also the Bayes factor article.)
In the former purpose (that of approximating a posterior probability), variational Bayes is an alternative to Monte Carlo sampling methods—particularly, Markov chain Monte Carlo methods such as Gibbs sampling—for taking a fully Bayesian approach to statistical inference over complex distributions that are difficult to evaluate directly or sample. In particular, whereas Monte Carlo techniques provide a numerical approximation to the exact posterior using a set of samples, variational Bayes provides a locally-optimal, exact analytical solution to an approximation of the posterior.
Variational Bayes can be seen as an extension of the expectation-maximization (EM) algorithm from maximum a posteriori estimation (MAP estimation) of the single most probable value of each parameter to fully Bayesian estimation which computes (an approximation to) the entire posterior distribution of the parameters and latent variables. As in EM, it finds a set of optimal parameter values, and it has the same alternating structure as does EM, based on a set of interlocked (mutually dependent) equations that cannot be solved analytically.
For many applications, variational Bayes produces solutions of comparable accuracy to Gibbs sampling at greater speed. However, deriving the set of equations used to update the parameters iteratively often requires a large amount of work compared with deriving the comparable Gibbs sampling equations. This is the case even for many models that are conceptually quite simple, as is demonstrated below in the case of a basic non-hierarchical model with only two parameters and no latent variables.
and 24 Related for: Variational Bayesian methods information
VariationalBayesianmethods are a family of techniques for approximating intractable integrals arising in Bayesian inference and machine learning. They...
graphical models and variationalBayesianmethods. In addition to being seen as an autoencoder neural network architecture, variational autoencoders can also...
In variationalBayesianmethods, the evidence lower bound (often abbreviated ELBO, also sometimes called the variational lower bound or negative variational...
Variational may refer to: Look up variational or variation in Wiktionary, the free dictionary. Calculus of variations, a field of mathematical analysis...
value of P ( B ) {\displaystyle P(B)} with methods such as Markov chain Monte Carlo or variationalBayesianmethods. The general set of statistical techniques...
target distribution. It differs from other Bayesian approximation approaches such as variationalBayesianmethods. More specifically, suppose we wish to approximate...
estimated from the data. This approach stands in contrast to standard Bayesianmethods, for which the prior distribution is fixed before any data are observed...
inference and a more embodied (enactive) view of the Bayesian brain. Using variationalBayesianmethods, it can be shown how internal models of the world...
approaches to artificial intelligence; it is formally related to variationalBayesianmethods and was originally introduced by Karl Friston as an explanation...
that minimize reconstruction loss. Variational autoencoders (VAEs) belong to the families of variationalBayesianmethods. Despite the architectural similarities...
problematic due to the Explaining Away problem raised by Judea Perl. VariationalBayesianmethods uses a surrogate posterior and blatantly disregard this complexity...
Bayesian inference (/ˈbeɪziən/ BAY-zee-ən or /ˈbeɪʒən/ BAY-zhən) is a method of statistical inference in which Bayes' theorem is used to update the probability...
needs to be either analytically or numerically approximated. In variationalBayesianmethods, the posterior probability is the probability of the parameters...
in research and applications of Bayesianmethods, mostly attributed to the discovery of Markov chain Monte Carlo methods and the consequent removal of many...
working on the efficient coding hypothesis, predictive coding and variationalBayesianmethods. The argument for reasoning about the information geometry on...
the image as words; one of the variations is called spatial latent Dirichlet allocation. VariationalBayesianmethods Pachinko allocation tf-idf Infer...
algorithm. MCMC methods are primarily used for calculating numerical approximations of multi-dimensional integrals, for example in Bayesian statistics, computational...
propagation, generalized belief propagation and variationalmethods. In order to fully specify the Bayesian network and thus fully represent the joint probability...
268-276 (2018) Griffiths et al. Constrained Bayesian Optimization for Automatic Chemical Design using Variational Autoencoders Chemical Science: 11, 577-586...
Global Information
