Estimating Predictive Rate–Distortion Curves via Neural Variational Inference
The Predictive Rate−Distortion curve quantifies the trade-off between compressing information about the past of a stochastic process and predicting its future accurately. Existing estimation methods for this curve work by clustering finite sequences of observations or by utilizing analytic...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2019-06-01
|
| Series: | Entropy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1099-4300/21/7/640 |
| _version_ | 1811278116622958592 |
|---|---|
| author | Michael Hahn Richard Futrell |
| author_facet | Michael Hahn Richard Futrell |
| author_sort | Michael Hahn |
| collection | DOAJ |
| description | The Predictive Rate−Distortion curve quantifies the trade-off between compressing information about the past of a stochastic process and predicting its future accurately. Existing estimation methods for this curve work by clustering finite sequences of observations or by utilizing analytically known causal states. Neither type of approach scales to processes such as natural languages, which have large alphabets and long dependencies, and where the causal states are not known analytically. We describe Neural Predictive Rate−Distortion (NPRD), an estimation method that scales to such processes, leveraging the universal approximation capabilities of neural networks. Taking only time series data as input, the method computes a variational bound on the Predictive Rate−Distortion curve. We validate the method on processes where Predictive Rate−Distortion is analytically known. As an application, we provide bounds on the Predictive Rate−Distortion of natural language, improving on bounds provided by clustering sequences. Based on the results, we argue that the Predictive Rate−Distortion curve is more useful than the usual notion of statistical complexity for characterizing highly complex processes such as natural language. |
| first_indexed | 2024-04-13T00:28:54Z |
| format | Article |
| id | doaj.art-7dc8117aff2346a8a1604b360e451306 |
| institution | Directory Open Access Journal |
| issn | 1099-4300 |
| language | English |
| last_indexed | 2024-04-13T00:28:54Z |
| publishDate | 2019-06-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Entropy |
| spelling | doaj.art-7dc8117aff2346a8a1604b360e4513062022-12-22T03:10:31ZengMDPI AGEntropy1099-43002019-06-0121764010.3390/e21070640e21070640Estimating Predictive Rate–Distortion Curves via Neural Variational InferenceMichael Hahn0Richard Futrell1Department of Linguistics, Stanford University, Stanford, CA 94305, USADepartment of Language Science, University of California, Irvine, CA 92697, USAThe Predictive Rate−Distortion curve quantifies the trade-off between compressing information about the past of a stochastic process and predicting its future accurately. Existing estimation methods for this curve work by clustering finite sequences of observations or by utilizing analytically known causal states. Neither type of approach scales to processes such as natural languages, which have large alphabets and long dependencies, and where the causal states are not known analytically. We describe Neural Predictive Rate−Distortion (NPRD), an estimation method that scales to such processes, leveraging the universal approximation capabilities of neural networks. Taking only time series data as input, the method computes a variational bound on the Predictive Rate−Distortion curve. We validate the method on processes where Predictive Rate−Distortion is analytically known. As an application, we provide bounds on the Predictive Rate−Distortion of natural language, improving on bounds provided by clustering sequences. Based on the results, we argue that the Predictive Rate−Distortion curve is more useful than the usual notion of statistical complexity for characterizing highly complex processes such as natural language.https://www.mdpi.com/1099-4300/21/7/640Predictive Rate–Distortionnatural languageinformation bottleneckneural variational inference |
| spellingShingle | Michael Hahn Richard Futrell Estimating Predictive Rate–Distortion Curves via Neural Variational Inference Entropy Predictive Rate–Distortion natural language information bottleneck neural variational inference |
| title | Estimating Predictive Rate–Distortion Curves via Neural Variational Inference |
| title_full | Estimating Predictive Rate–Distortion Curves via Neural Variational Inference |
| title_fullStr | Estimating Predictive Rate–Distortion Curves via Neural Variational Inference |
| title_full_unstemmed | Estimating Predictive Rate–Distortion Curves via Neural Variational Inference |
| title_short | Estimating Predictive Rate–Distortion Curves via Neural Variational Inference |
| title_sort | estimating predictive rate distortion curves via neural variational inference |
| topic | Predictive Rate–Distortion natural language information bottleneck neural variational inference |
| url | https://www.mdpi.com/1099-4300/21/7/640 |
| work_keys_str_mv | AT michaelhahn estimatingpredictiveratedistortioncurvesvianeuralvariationalinference AT richardfutrell estimatingpredictiveratedistortioncurvesvianeuralvariationalinference |