Entropy Estimators for Markovian Sequences: A Comparative Analysis
Entropy estimation is a fundamental problem in information theory that has applications in various fields, including physics, biology, and computer science. Estimating the entropy of discrete sequences can be challenging due to limited data and the lack of unbiased estimators. Most existing entropy...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-01-01
|
| Series: | Entropy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1099-4300/26/1/79 |
| _version_ | 1797344098237022208 |
|---|---|
| author | Juan De Gregorio David Sánchez Raúl Toral |
| author_facet | Juan De Gregorio David Sánchez Raúl Toral |
| author_sort | Juan De Gregorio |
| collection | DOAJ |
| description | Entropy estimation is a fundamental problem in information theory that has applications in various fields, including physics, biology, and computer science. Estimating the entropy of discrete sequences can be challenging due to limited data and the lack of unbiased estimators. Most existing entropy estimators are designed for sequences of independent events and their performances vary depending on the system being studied and the available data size. In this work, we compare different entropy estimators and their performance when applied to Markovian sequences. Specifically, we analyze both binary Markovian sequences and Markovian systems in the undersampled regime. We calculate the bias, standard deviation, and mean squared error for some of the most widely employed estimators. We discuss the limitations of entropy estimation as a function of the transition probabilities of the Markov processes and the sample size. Overall, this paper provides a comprehensive comparison of entropy estimators and their performance in estimating entropy for systems with memory, which can be useful for researchers and practitioners in various fields. |
| first_indexed | 2024-03-08T10:57:26Z |
| format | Article |
| id | doaj.art-a0060f6f98da406d8bb265438a555027 |
| institution | Directory Open Access Journal |
| issn | 1099-4300 |
| language | English |
| last_indexed | 2024-03-08T10:57:26Z |
| publishDate | 2024-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Entropy |
| spelling | doaj.art-a0060f6f98da406d8bb265438a5550272024-01-26T16:23:16ZengMDPI AGEntropy1099-43002024-01-012617910.3390/e26010079Entropy Estimators for Markovian Sequences: A Comparative AnalysisJuan De Gregorio0David Sánchez1Raúl Toral2Institute for Cross-Disciplinary Physics and Complex Systems IFISC (UIB-CSIC), Campus Universitat de les Illes Balears, E-07122 Palma de Mallorca, SpainInstitute for Cross-Disciplinary Physics and Complex Systems IFISC (UIB-CSIC), Campus Universitat de les Illes Balears, E-07122 Palma de Mallorca, SpainInstitute for Cross-Disciplinary Physics and Complex Systems IFISC (UIB-CSIC), Campus Universitat de les Illes Balears, E-07122 Palma de Mallorca, SpainEntropy estimation is a fundamental problem in information theory that has applications in various fields, including physics, biology, and computer science. Estimating the entropy of discrete sequences can be challenging due to limited data and the lack of unbiased estimators. Most existing entropy estimators are designed for sequences of independent events and their performances vary depending on the system being studied and the available data size. In this work, we compare different entropy estimators and their performance when applied to Markovian sequences. Specifically, we analyze both binary Markovian sequences and Markovian systems in the undersampled regime. We calculate the bias, standard deviation, and mean squared error for some of the most widely employed estimators. We discuss the limitations of entropy estimation as a function of the transition probabilities of the Markov processes and the sample size. Overall, this paper provides a comprehensive comparison of entropy estimators and their performance in estimating entropy for systems with memory, which can be useful for researchers and practitioners in various fields.https://www.mdpi.com/1099-4300/26/1/79Shannon entropyMarkovian systemsdata analysisestimators |
| spellingShingle | Juan De Gregorio David Sánchez Raúl Toral Entropy Estimators for Markovian Sequences: A Comparative Analysis Entropy Shannon entropy Markovian systems data analysis estimators |
| title | Entropy Estimators for Markovian Sequences: A Comparative Analysis |
| title_full | Entropy Estimators for Markovian Sequences: A Comparative Analysis |
| title_fullStr | Entropy Estimators for Markovian Sequences: A Comparative Analysis |
| title_full_unstemmed | Entropy Estimators for Markovian Sequences: A Comparative Analysis |
| title_short | Entropy Estimators for Markovian Sequences: A Comparative Analysis |
| title_sort | entropy estimators for markovian sequences a comparative analysis |
| topic | Shannon entropy Markovian systems data analysis estimators |
| url | https://www.mdpi.com/1099-4300/26/1/79 |
| work_keys_str_mv | AT juandegregorio entropyestimatorsformarkoviansequencesacomparativeanalysis AT davidsanchez entropyestimatorsformarkoviansequencesacomparativeanalysis AT raultoral entropyestimatorsformarkoviansequencesacomparativeanalysis |