Discriminant Analysis under <i>f</i>-Divergence Measures
In statistical inference, the information-theoretic performance limits can often be expressed in terms of a statistical divergence between the underlying statistical models (e.g., in binary hypothesis testing, the error probability is related to the total variation distance between the statistical m...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-01-01
|
| Series: | Entropy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1099-4300/24/2/188 |
| _version_ | 1827655137096630272 |
|---|---|
| author | Anmol Dwivedi Sihui Wang Ali Tajer |
| author_facet | Anmol Dwivedi Sihui Wang Ali Tajer |
| author_sort | Anmol Dwivedi |
| collection | DOAJ |
| description | In statistical inference, the information-theoretic performance limits can often be expressed in terms of a statistical divergence between the underlying statistical models (e.g., in binary hypothesis testing, the error probability is related to the total variation distance between the statistical models). As the data dimension grows, computing the statistics involved in decision-making and the attendant performance limits (divergence measures) face complexity and stability challenges. Dimensionality reduction addresses these challenges at the expense of compromising the performance (the divergence reduces by the data-processing inequality). This paper considers linear dimensionality reduction such that the divergence between the models is maximally preserved. Specifically, this paper focuses on Gaussian models where we investigate discriminant analysis under five <i>f</i>-divergence measures (Kullback–Leibler, symmetrized Kullback–Leibler, Hellinger, total variation, and <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><msup><mi>χ</mi><mn>2</mn></msup></semantics></math></inline-formula>). We characterize the optimal design of the linear transformation of the data onto a lower-dimensional subspace for zero-mean Gaussian models and employ numerical algorithms to find the design for general Gaussian models with non-zero means. There are two key observations for zero-mean Gaussian models. First, projections are not necessarily along the largest modes of the covariance matrix of the data, and, in some situations, they can even be along the smallest modes. Secondly, under specific regimes, the optimal design of subspace projection is identical under all the <i>f</i>-divergence measures considered, rendering a degree of universality to the design, independent of the inference problem of interest. |
| first_indexed | 2024-03-09T22:02:31Z |
| format | Article |
| id | doaj.art-5f285c413ff74d2d8451add87546c2a3 |
| institution | Directory Open Access Journal |
| issn | 1099-4300 |
| language | English |
| last_indexed | 2024-03-09T22:02:31Z |
| publishDate | 2022-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Entropy |
| spelling | doaj.art-5f285c413ff74d2d8451add87546c2a32023-11-23T19:47:24ZengMDPI AGEntropy1099-43002022-01-0124218810.3390/e24020188Discriminant Analysis under <i>f</i>-Divergence MeasuresAnmol Dwivedi0Sihui Wang1Ali Tajer2Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY 12180, USADepartment of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY 12180, USADepartment of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY 12180, USAIn statistical inference, the information-theoretic performance limits can often be expressed in terms of a statistical divergence between the underlying statistical models (e.g., in binary hypothesis testing, the error probability is related to the total variation distance between the statistical models). As the data dimension grows, computing the statistics involved in decision-making and the attendant performance limits (divergence measures) face complexity and stability challenges. Dimensionality reduction addresses these challenges at the expense of compromising the performance (the divergence reduces by the data-processing inequality). This paper considers linear dimensionality reduction such that the divergence between the models is maximally preserved. Specifically, this paper focuses on Gaussian models where we investigate discriminant analysis under five <i>f</i>-divergence measures (Kullback–Leibler, symmetrized Kullback–Leibler, Hellinger, total variation, and <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><msup><mi>χ</mi><mn>2</mn></msup></semantics></math></inline-formula>). We characterize the optimal design of the linear transformation of the data onto a lower-dimensional subspace for zero-mean Gaussian models and employ numerical algorithms to find the design for general Gaussian models with non-zero means. There are two key observations for zero-mean Gaussian models. First, projections are not necessarily along the largest modes of the covariance matrix of the data, and, in some situations, they can even be along the smallest modes. Secondly, under specific regimes, the optimal design of subspace projection is identical under all the <i>f</i>-divergence measures considered, rendering a degree of universality to the design, independent of the inference problem of interest.https://www.mdpi.com/1099-4300/24/2/188dimensionality reductiondiscriminant analysis<i>f</i>-divergencestatistical inference |
| spellingShingle | Anmol Dwivedi Sihui Wang Ali Tajer Discriminant Analysis under <i>f</i>-Divergence Measures Entropy dimensionality reduction discriminant analysis <i>f</i>-divergence statistical inference |
| title | Discriminant Analysis under <i>f</i>-Divergence Measures |
| title_full | Discriminant Analysis under <i>f</i>-Divergence Measures |
| title_fullStr | Discriminant Analysis under <i>f</i>-Divergence Measures |
| title_full_unstemmed | Discriminant Analysis under <i>f</i>-Divergence Measures |
| title_short | Discriminant Analysis under <i>f</i>-Divergence Measures |
| title_sort | discriminant analysis under i f i divergence measures |
| topic | dimensionality reduction discriminant analysis <i>f</i>-divergence statistical inference |
| url | https://www.mdpi.com/1099-4300/24/2/188 |
| work_keys_str_mv | AT anmoldwivedi discriminantanalysisunderifidivergencemeasures AT sihuiwang discriminantanalysisunderifidivergencemeasures AT alitajer discriminantanalysisunderifidivergencemeasures |