Bias in Zipf’s law estimators
Abstract The prevailing maximum likelihood estimators for inferring power law models from rank-frequency data are biased. The source of this bias is an inappropriate likelihood function. The correct likelihood function is derived and shown to be computationally intractable. A more computationally ef...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2021-08-01
|
| Series: | Scientific Reports |
| Online Access: | https://doi.org/10.1038/s41598-021-96214-w |
| _version_ | 1818851579189526528 |
|---|---|
| author | Charlie Pilgrim Thomas T Hills |
| author_facet | Charlie Pilgrim Thomas T Hills |
| author_sort | Charlie Pilgrim |
| collection | DOAJ |
| description | Abstract The prevailing maximum likelihood estimators for inferring power law models from rank-frequency data are biased. The source of this bias is an inappropriate likelihood function. The correct likelihood function is derived and shown to be computationally intractable. A more computationally efficient method of approximate Bayesian computation (ABC) is explored. This method is shown to have less bias for data generated from idealised rank-frequency Zipfian distributions. However, the existing estimators and the ABC estimator described here assume that words are drawn from a simple probability distribution, while language is a much more complex process. We show that this false assumption leads to continued biases when applying any of these methods to natural language to estimate Zipf exponents. We recommend that researchers be aware of the bias when investigating power laws in rank-frequency data. |
| first_indexed | 2024-12-19T07:07:16Z |
| format | Article |
| id | doaj.art-9b29ac7b9d07447eaecece75d6538d88 |
| institution | Directory Open Access Journal |
| issn | 2045-2322 |
| language | English |
| last_indexed | 2024-12-19T07:07:16Z |
| publishDate | 2021-08-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj.art-9b29ac7b9d07447eaecece75d6538d882022-12-21T20:31:15ZengNature PortfolioScientific Reports2045-23222021-08-0111111110.1038/s41598-021-96214-wBias in Zipf’s law estimatorsCharlie Pilgrim0Thomas T Hills1Mathematics for Real-World Systems Centre for Doctoral Training, The University of WarwickDepartment of Psychology, The University of WarwickAbstract The prevailing maximum likelihood estimators for inferring power law models from rank-frequency data are biased. The source of this bias is an inappropriate likelihood function. The correct likelihood function is derived and shown to be computationally intractable. A more computationally efficient method of approximate Bayesian computation (ABC) is explored. This method is shown to have less bias for data generated from idealised rank-frequency Zipfian distributions. However, the existing estimators and the ABC estimator described here assume that words are drawn from a simple probability distribution, while language is a much more complex process. We show that this false assumption leads to continued biases when applying any of these methods to natural language to estimate Zipf exponents. We recommend that researchers be aware of the bias when investigating power laws in rank-frequency data.https://doi.org/10.1038/s41598-021-96214-w |
| spellingShingle | Charlie Pilgrim Thomas T Hills Bias in Zipf’s law estimators Scientific Reports |
| title | Bias in Zipf’s law estimators |
| title_full | Bias in Zipf’s law estimators |
| title_fullStr | Bias in Zipf’s law estimators |
| title_full_unstemmed | Bias in Zipf’s law estimators |
| title_short | Bias in Zipf’s law estimators |
| title_sort | bias in zipf s law estimators |
| url | https://doi.org/10.1038/s41598-021-96214-w |
| work_keys_str_mv | AT charliepilgrim biasinzipfslawestimators AT thomasthills biasinzipfslawestimators |