Methods of estimating differential entropy given some observations
In various science/engineering applications, such as independent component analysis,[1] image analysis,[2] genetic analysis,[3] speech recognition,[4] manifold learning,[5] and time delay estimation[6] it is useful to estimate the differential entropy of a system or process, given some observations.
The simplest and most common approach uses histogram-based estimation, but other approaches have been developed and used, each with its own benefits and drawbacks.[7] The main factor in choosing a method is often a trade-off between the bias and the variance of the estimate,[8] although the nature of the (suspected) distribution of the data may also be a factor,[7] as well as the sample size and the size of the alphabet of the probability distribution.[9]
^Dinh-Tuan Pham (2004) Fast algorithms for mutual information based independent component analysis. In Signal Processing. Volume 52, Issue 10, 2690–2700, doi:10.1109/TSP.2004.834398
^Chang, C.-I.; Du, Y.; Wang, J.; Guo, S.-M.; Thouin, P.D. (2006) Survey and comparative analysis of entropy and relative entropy thresholding techniques. In Vision, Image and Signal Processing, Volume 153, Issue 6, 837–850, doi:10.1049/ip-vis:20050032
^Martins, D. C. et al. (2008) Intrinsically Multivariate Predictive Genes. In Selected Topics in Signal Processing. Volume 2, Issue 3, 424–439, doi:10.1109/JSTSP.2008.923841
^Gue Jun Jung; Yung-Hwan Oh (2008) Information Distance-Based Subvector Clustering for ASR Parameter Quantization. In Signal Processing Letters, Volume 15, 209–212, doi:10.1109/LSP.2007.913132
^Costa, J.A.; Hero, A.O. (2004), Geodesic entropic graphs for dimension and entropy estimation in manifold learning. In Signal Processing, Volume 52, Issue 8, 2210–2221, doi:10.1109/TSP.2004.831130
^Benesty, J.; Yiteng Huang; Jingdong Chen (2007) Time Delay Estimation via Minimum Entropy. In Signal Processing Letters, Volume 14, Issue 3, March 2007 157–160 doi:10.1109/LSP.2006.884038
^ abJ. Beirlant, E. J. Dudewicz, L. Gyorfi, and E. C. van der Meulen (1997) Nonparametric entropy estimation: An overview. In International Journal of Mathematical and Statistical Sciences, Volume 6, pp. 17–
39.
^T. Schürmann, Bias analysis in entropy estimation. In J. Phys. A: Math. Gen, 37 (2004), pp. L295–L301. doi:10.1088/0305-4470/37/27/L02
^Pinchas A., Ben-Gal I., Painsky A. (2024) (2024). "A Comparative Analysis of Discrete Entropy Estimators for Large-Alphabet Problems" (PDF). Entropy. 26 (5). Entropy. 2024; 26(5):369. doi:10.3390/e26050369: 369. doi:10.3390/e26050369.{{cite journal}}: CS1 maint: multiple names: authors list (link) CS1 maint: numeric names: authors list (link)
and 22 Related for: Entropy estimation information
recognition, manifold learning, and time delay estimation it is useful to estimate the differential entropy of a system or process, given some observations...
Maximum entropy spectral estimation is a method of spectral density estimation. The goal is to improve the spectral quality based on the principle of maximum...
Differential entropy (also referred to as continuous entropy) is a concept in information theory that began as an attempt by Claude Shannon to extend...
in discrete and continuous density estimation. Similar to support vector machine estimators, the maximum entropy principle may require the solution to...
postulated Maximum spacing estimation: a related method that is more robust in many situations Maximum entropyestimation Method of moments (statistics):...
However, it usually requires more samples for accurate estimation. The probabilities in the entropy formula can be estimated using different approaches (binning...
Both of the reseedings reset the entropyestimation of the fast pool to zero, but the last one also sets the estimation of the slow pool to zero. The reseeding...
password entropy using newly available real-world data has demonstrated that the NIST scheme does not provide a valid metric for entropyestimation of human-selected...
maximum entropy Maximum entropy probability distribution Maximum entropy spectral estimation Maximum likelihood Maximum likelihood sequence estimation Maximum...
Maximum entropy thermodynamics Maximum entropy spectral estimation Principle of maximum entropy Maximum entropy probability distribution Maximum entropy classifier...
Information entropy Information theory Limiting density of discrete points Self-information Kullback–Leibler divergence Entropyestimation Dembo, Amir;...
Lake, Douglas E.; Moorman, J. Randall (January 2011). "Accurate estimation of entropy in very short physiological time series: the problem of atrial fibrillation...
Probability estimation for PPM, Original Source from archive.org. Schürmann, T.; Grassberger, P. (September 1996). "Entropyestimation of symbol sequences"...
Data binning Density estimation Kernel density estimation, a smoother but more complex method of density estimationEntropyestimation Freedman–Diaconis...
constant, and in Planck's law of black-body radiation and Boltzmann's entropy formula, and is used in calculating thermal noise in resistors. The Boltzmann...
statistical signal processing, the goal of spectral density estimation (SDE) or simply spectral estimation is to estimate the spectral density (also known as the...
In statistics and information theory, a maximum entropy probability distribution has entropy that is at least as great as that of all other members of...
be used to estimate the entropy of a stochastic process and called Neural Joint Entropy Estimator (NJEE). Such an estimation provides insights on the...
concepts involved in spectral estimation include autocorrelation, multi-D Fourier transform, mean square error and entropy. When it comes to multidimensional...
_{i=1}^{R}p_{i}\ln(p_{i})\right)} which is the exponential of the Shannon entropy calculated with natural logarithms (see above). In other domains, this...
the increased entropy of the gas phase overcomes the intermolecular forces. As a given quantity of matter always has a higher entropy in the gas phase...
Global Information