Lossless Transformations and Excess Risk Bounds in Statistical Inference
We study the excess minimum risk in statistical inference, defined as the difference between the minimum expected loss when estimating a random variable from an observed feature vector and the minimum expected loss when estimating the same random variable from a transformation (statistic) of the fea...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-09-01
|
| Series: | Entropy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1099-4300/25/10/1394 |
| _version_ | 1797573973473492992 |
|---|---|
| author | László Györfi Tamás Linder Harro Walk |
| author_facet | László Györfi Tamás Linder Harro Walk |
| author_sort | László Györfi |
| collection | DOAJ |
| description | We study the excess minimum risk in statistical inference, defined as the difference between the minimum expected loss when estimating a random variable from an observed feature vector and the minimum expected loss when estimating the same random variable from a transformation (statistic) of the feature vector. After characterizing lossless transformations, i.e., transformations for which the excess risk is zero for all loss functions, we construct a partitioning test statistic for the hypothesis that a given transformation is lossless, and we show that for i.i.d. data the test is strongly consistent. More generally, we develop information-theoretic upper bounds on the excess risk that uniformly hold over fairly general classes of loss functions. Based on these bounds, we introduce the notion of a <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mi>δ</mi></semantics></math></inline-formula>-lossless transformation and give sufficient conditions for a given transformation to be universally <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mi>δ</mi></semantics></math></inline-formula>-lossless. Applications to classification, nonparametric regression, portfolio strategies, information bottlenecks, and deep learning are also surveyed. |
| first_indexed | 2024-03-10T21:16:43Z |
| format | Article |
| id | doaj.art-84fe53f4fb854d1db56701148b3f5205 |
| institution | Directory Open Access Journal |
| issn | 1099-4300 |
| language | English |
| last_indexed | 2024-03-10T21:16:43Z |
| publishDate | 2023-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Entropy |
| spelling | doaj.art-84fe53f4fb854d1db56701148b3f52052023-11-19T16:24:16ZengMDPI AGEntropy1099-43002023-09-012510139410.3390/e25101394Lossless Transformations and Excess Risk Bounds in Statistical InferenceLászló Györfi0Tamás Linder1Harro Walk2Department of Computer Science and Information Theory, Budapest University of Technology and Economics, H-1111 Budapest, HungaryDepartment of Mathematics and Statistics, Queen’s University, Kingston, ON K7L 3N6, CanadaFachbereich Mathematik, Universität Stuttgart, 70569 Stuttgart, GermanyWe study the excess minimum risk in statistical inference, defined as the difference between the minimum expected loss when estimating a random variable from an observed feature vector and the minimum expected loss when estimating the same random variable from a transformation (statistic) of the feature vector. After characterizing lossless transformations, i.e., transformations for which the excess risk is zero for all loss functions, we construct a partitioning test statistic for the hypothesis that a given transformation is lossless, and we show that for i.i.d. data the test is strongly consistent. More generally, we develop information-theoretic upper bounds on the excess risk that uniformly hold over fairly general classes of loss functions. Based on these bounds, we introduce the notion of a <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mi>δ</mi></semantics></math></inline-formula>-lossless transformation and give sufficient conditions for a given transformation to be universally <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mi>δ</mi></semantics></math></inline-formula>-lossless. Applications to classification, nonparametric regression, portfolio strategies, information bottlenecks, and deep learning are also surveyed.https://www.mdpi.com/1099-4300/25/10/1394statistical inference with lossstrongly consistent testinformation-theoretic boundsclassificationregressionportfolio selection |
| spellingShingle | László Györfi Tamás Linder Harro Walk Lossless Transformations and Excess Risk Bounds in Statistical Inference Entropy statistical inference with loss strongly consistent test information-theoretic bounds classification regression portfolio selection |
| title | Lossless Transformations and Excess Risk Bounds in Statistical Inference |
| title_full | Lossless Transformations and Excess Risk Bounds in Statistical Inference |
| title_fullStr | Lossless Transformations and Excess Risk Bounds in Statistical Inference |
| title_full_unstemmed | Lossless Transformations and Excess Risk Bounds in Statistical Inference |
| title_short | Lossless Transformations and Excess Risk Bounds in Statistical Inference |
| title_sort | lossless transformations and excess risk bounds in statistical inference |
| topic | statistical inference with loss strongly consistent test information-theoretic bounds classification regression portfolio selection |
| url | https://www.mdpi.com/1099-4300/25/10/1394 |
| work_keys_str_mv | AT laszlogyorfi losslesstransformationsandexcessriskboundsinstatisticalinference AT tamaslinder losslesstransformationsandexcessriskboundsinstatisticalinference AT harrowalk losslesstransformationsandexcessriskboundsinstatisticalinference |