A study of local approximations in information theory

Thesis: S.M., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2015.

Bibliographic Details
Main Author: Makur, Anuran
Other Authors: Lizhong Zheng.
Format: Thesis
Language:eng
Published: Massachusetts Institute of Technology 2015
Subjects:
Online Access:http://hdl.handle.net/1721.1/99789
_version_ 1826198084376330240
author Makur, Anuran
author2 Lizhong Zheng.
author_facet Lizhong Zheng.
Makur, Anuran
author_sort Makur, Anuran
collection MIT
description Thesis: S.M., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2015.
first_indexed 2024-09-23T10:58:36Z
format Thesis
id mit-1721.1/99789
institution Massachusetts Institute of Technology
language eng
last_indexed 2024-09-23T10:58:36Z
publishDate 2015
publisher Massachusetts Institute of Technology
record_format dspace
spelling mit-1721.1/997892019-04-10T08:56:19Z A study of local approximations in information theory Makur, Anuran Lizhong Zheng. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science. Electrical Engineering and Computer Science. Thesis: S.M., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2015. This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections. Cataloged from student-submitted PDF version of thesis. Includes bibliographical references (pages 171-173). The intractability of many information theoretic problems arises from the meaningful but nonlinear definition of Kullback-Leibler (KL) divergence between two probability distributions. Local information theory addresses this issue by assuming all distributions of interest are perturbations of certain reference distributions, and then approximating KL divergence with a squared weighted Euclidean distance, thereby linearizing such problems. We show that large classes of statistical divergence measures, such as f-divergences and Bregman divergences, can be approximated in an analogous manner to local metrics which are very similar in form. We then capture the cost of making local approximations of KL divergence instead of using its global value. This is achieved by appropriately bounding the tightness of the Data Processing Inequality in the local and global scenarios. This task turns out to be equivalent to bounding the chordal slope of the hypercontractivity ribbon at infinity and the Hirschfeld-Gebelein-Renyi maximal correlation with each other. We derive such bounds for the discrete and finite, as well as the Gaussian regimes. An application of the local approximation technique is in understanding the large deviation behavior of sources and channels. We elucidate a source-channel decomposition of the large deviation characteristics of i.i.d. sources going through discrete memoryless channels. This is used to derive an additive Gaussian noise channel model for the local perturbations of probability distributions. We next shift our focus to infinite alphabet channels instead of discrete and finite channels. On this front, existing literature has demonstrated that the singular vectors of additive white Gaussian noise channels are Hermite polynomials, and the singular vectors of Poisson channels are Laguerre polynomials. We characterize the set of infinite alphabet channels whose singular value decompositions produce singular vectors that are orthogonal polynomials by providing equivalent conditions on the conditional moments. In doing so, we also unveil the elegant relationship between certain natural exponential families with quadratic variance functions, their conjugate priors, and their corresponding orthogonal polynomial singular vectors. Finally, we propose various related directions for future research in the hope that our work will beget more research concerning local approximation methods in information theory. by Anuran Makur. S.M. 2015-11-09T19:13:17Z 2015-11-09T19:13:17Z 2015 2015 Thesis http://hdl.handle.net/1721.1/99789 927713010 eng M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission. http://dspace.mit.edu/handle/1721.1/7582 173 pages application/pdf Massachusetts Institute of Technology
spellingShingle Electrical Engineering and Computer Science.
Makur, Anuran
A study of local approximations in information theory
title A study of local approximations in information theory
title_full A study of local approximations in information theory
title_fullStr A study of local approximations in information theory
title_full_unstemmed A study of local approximations in information theory
title_short A study of local approximations in information theory
title_sort study of local approximations in information theory
topic Electrical Engineering and Computer Science.
url http://hdl.handle.net/1721.1/99789
work_keys_str_mv AT makuranuran astudyoflocalapproximationsininformationtheory
AT makuranuran studyoflocalapproximationsininformationtheory