Continuous Bernoulli distribution information

Continuous Bernoulli distribution
	Probability density function
Notation
Parameters
Support
PDF	; where
CDF
Mean
Variance

In probability theory, statistics, and machine learning, the continuous Bernoulli distribution^[1]^[2]^[3] is a family of continuous probability distributions parameterized by a single shape parameter $\lambda \in (0,1)$ , defined on the unit interval $x\in [0,1]$ , by:

p(x|\lambda )\propto \lambda ^{x}(1-\lambda )^{1-x}.

The continuous Bernoulli distribution arises in deep learning and computer vision, specifically in the context of variational autoencoders,^[4]^[5] for modeling the pixel intensities of natural images. As such, it defines a proper probabilistic counterpart for the commonly used binary cross entropy loss, which is often applied to continuous, $[0,1]$ -valued data.^[6]^[7]^[8]^[9] This practice amounts to ignoring the normalizing constant of the continuous Bernoulli distribution, since the binary cross entropy loss only defines a true log-likelihood for discrete, $\{0,1\}$ -valued data.

The continuous Bernoulli also defines an exponential family of distributions. Writing $\eta =\log \left(\lambda /(1-\lambda )\right)$ for the natural parameter, the density can be rewritten in canonical form: $p(x|\eta )\propto \exp(\eta x)$ .

^ Loaiza-Ganem, G., & Cunningham, J. P. (2019). The continuous Bernoulli: fixing a pervasive error in variational autoencoders. In Advances in Neural Information Processing Systems (pp. 13266-13276).
^ PyTorch Distributions. https://pytorch.org/docs/stable/distributions.html#continuousbernoulli
^ Tensorflow Probability. https://www.tensorflow.org/probability/api_docs/python/tfp/edward2/ContinuousBernoulli Archived 2020-11-25 at the Wayback Machine
^ Kingma, D. P., & Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.
^ Kingma, D. P., & Welling, M. (2014, April). Stochastic gradient VB and the variational auto-encoder. In Second International Conference on Learning Representations, ICLR (Vol. 19).
^ Larsen, A. B. L., Sønderby, S. K., Larochelle, H., & Winther, O. (2016, June). Autoencoding beyond pixels using a learned similarity metric. In International conference on machine learning (pp. 1558-1566).
^ Jiang, Z., Zheng, Y., Tan, H., Tang, B., & Zhou, H. (2017, August). Variational deep embedding: an unsupervised and generative approach to clustering. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (pp. 1965-1972).
^ PyTorch VAE tutorial: https://github.com/pytorch/examples/tree/master/vae.
^ Keras VAE tutorial: https://blog.keras.io/building-autoencoders-in-keras.html.

[1] Loaiza-Ganem, G., & Cunningham, J. P. (2019). The continuous Bernoulli: fixing a pervasive error in variational autoencoders. In Advances in Neural Information Processing Systems (pp. 13266-13276).

[2] PyTorch Distributions. https://pytorch.org/docs/stable/distributions.html#continuousbernoulli

[3] Tensorflow Probability. https://www.tensorflow.org/probability/api_docs/python/tfp/edward2/ContinuousBernoulli Archived 2020-11-25 at the Wayback Machine

[4] Kingma, D. P., & Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.

[5] Kingma, D. P., & Welling, M. (2014, April). Stochastic gradient VB and the variational auto-encoder. In Second International Conference on Learning Representations, ICLR (Vol. 19).

[6] Larsen, A. B. L., Sønderby, S. K., Larochelle, H., & Winther, O. (2016, June). Autoencoding beyond pixels using a learned similarity metric. In International conference on machine learning (pp. 1558-1566).

[7] Jiang, Z., Zheng, Y., Tan, H., Tang, B., & Zhou, H. (2017, August). Variational deep embedding: an unsupervised and generative approach to clustering. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (pp. 1965-1972).

[8] PyTorch VAE tutorial: https://github.com/pytorch/examples/tree/master/vae.

[9] Keras VAE tutorial: https://blog.keras.io/building-autoencoders-in-keras.html.

Continuous Bernoulli distribution information

and 27 Related for: Continuous Bernoulli distribution information

Continuous Bernoulli distribution

Continuous uniform distribution

Bernoulli distribution

Beta distribution

Probability distribution

List of probability distributions

Binomial distribution

List of things named after members of the Bernoulli family

Exponential distribution

Bernoulli trial

Bernoulli process

Laplace distribution

Joint probability distribution

Multinomial distribution

Exponential family

Normal distribution

Logistic distribution

Dirichlet distribution

Bernoulli number

Cantor distribution

Continuous or discrete variable

Convolution of probability distributions

Pareto distribution

Markov chain

Skew normal distribution

Jacob Bernoulli

Geometric distribution

Probability density function
Notation	${\mathcal {CB}}(\lambda )$
Parameters	$\lambda \in (0,1)$
Support	$x\in [0,1]$
PDF	$C(\lambda )\lambda ^{x}(1-\lambda )^{1-x}\!$ where $C(\lambda )={\begin{cases}2&{\text{if }}\lambda ={\frac {1}{2}}\\{\frac {2\tanh ^{-1}(1-2\lambda )}{1-2\lambda }}&{\text{ otherwise}}\end{cases}}$
CDF	${\begin{cases}x&{\text{ if }}\lambda ={\frac {1}{2}}\\{\frac {\lambda ^{x}(1-\lambda )^{1-x}+\lambda -1}{2\lambda -1}}&{\text{ otherwise}}\end{cases}}\!$
Mean	$\operatorname {E} [X]={\begin{cases}{\frac {1}{2}}&{\text{ if }}\lambda ={\frac {1}{2}}\\{\frac {\lambda }{2\lambda -1}}+{\frac {1}{2\tanh ^{-1}(1-2\lambda )}}&{\text{ otherwise}}\end{cases}}\!$
Variance	$\operatorname {var} [X]={\begin{cases}{\frac {1}{12}}&{\text{ if }}\lambda ={\frac {1}{2}}\\-{\frac {(1-\lambda )\lambda }{(1-2\lambda )^{2}}}+{\frac {1}{(2\tanh ^{-1}(1-2\lambda ))^{2}}}&{\text{ otherwise}}\end{cases}}\!$