Topics in Sparsity and Compression: From High dimensional statistics to Overparametrized Neural Networks
This thesis presents applications of sparsity in three different areas: covariance estimation in time-series data, linear regression with categorical variables, and neural network compression. In the first chapter, motivated by problems in computational finance, we consider a framework for jointl...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Published: |
Massachusetts Institute of Technology
2023
|
| Online Access: | https://hdl.handle.net/1721.1/151535 |
| _version_ | 1811081819678834688 |
|---|---|
| author | Benbaki, Riade |
| author2 | Mazumder, Rahul |
| author_facet | Mazumder, Rahul Benbaki, Riade |
| author_sort | Benbaki, Riade |
| collection | MIT |
| description | This thesis presents applications of sparsity in three different areas: covariance estimation in time-series data, linear regression with categorical variables, and neural network compression.
In the first chapter, motivated by problems in computational finance, we consider a framework for jointly learning time-varying covariance matrices under different structural assumptions (e.g., low-rank, sparsity or a combination of both). We propose novel algorithms for learning these covariance matrices simultaneously across all time blocks and show improved computational efficiency and performance across different tasks.
In the second chapter, we study the problem of linear regression with categorical variables, where every categorical variable can have a large number of levels. We seek to reduce or cluster the number of levels for statistical and interpretability reasons. To this end, we propose a new estimator and study its computational and statistical properties.
And in the third chapter, we explore the problem of pruning or sparsifying the weights of a neural network. Modern neural networks tend to have a large number of parameters, which makes their storage and deployment expensive, especially in resource-constrained environments. One solution to this is compressing the network by pruning or removing some parameters, while trying to maintain a similar level of performance compared to the dense network. To achieve this, we propose a new optimization-based pruning algorithm, and show how it leads to significantly better sparsity-accuracy trade-offs compared to existing pruning methods. |
| first_indexed | 2024-09-23T11:52:59Z |
| format | Thesis |
| id | mit-1721.1/151535 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T11:52:59Z |
| publishDate | 2023 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1515352023-08-01T03:50:22Z Topics in Sparsity and Compression: From High dimensional statistics to Overparametrized Neural Networks Benbaki, Riade Mazumder, Rahul Massachusetts Institute of Technology. Operations Research Center This thesis presents applications of sparsity in three different areas: covariance estimation in time-series data, linear regression with categorical variables, and neural network compression. In the first chapter, motivated by problems in computational finance, we consider a framework for jointly learning time-varying covariance matrices under different structural assumptions (e.g., low-rank, sparsity or a combination of both). We propose novel algorithms for learning these covariance matrices simultaneously across all time blocks and show improved computational efficiency and performance across different tasks. In the second chapter, we study the problem of linear regression with categorical variables, where every categorical variable can have a large number of levels. We seek to reduce or cluster the number of levels for statistical and interpretability reasons. To this end, we propose a new estimator and study its computational and statistical properties. And in the third chapter, we explore the problem of pruning or sparsifying the weights of a neural network. Modern neural networks tend to have a large number of parameters, which makes their storage and deployment expensive, especially in resource-constrained environments. One solution to this is compressing the network by pruning or removing some parameters, while trying to maintain a similar level of performance compared to the dense network. To achieve this, we propose a new optimization-based pruning algorithm, and show how it leads to significantly better sparsity-accuracy trade-offs compared to existing pruning methods. S.M. 2023-07-31T19:46:56Z 2023-07-31T19:46:56Z 2023-06 2023-07-13T16:01:48.238Z Thesis https://hdl.handle.net/1721.1/151535 Attribution 4.0 International (CC BY 4.0) Copyright retained by author(s) https://creativecommons.org/licenses/by/4.0/ application/pdf Massachusetts Institute of Technology |
| spellingShingle | Benbaki, Riade Topics in Sparsity and Compression: From High dimensional statistics to Overparametrized Neural Networks |
| title | Topics in Sparsity and Compression: From High dimensional
statistics to Overparametrized Neural Networks |
| title_full | Topics in Sparsity and Compression: From High dimensional
statistics to Overparametrized Neural Networks |
| title_fullStr | Topics in Sparsity and Compression: From High dimensional
statistics to Overparametrized Neural Networks |
| title_full_unstemmed | Topics in Sparsity and Compression: From High dimensional
statistics to Overparametrized Neural Networks |
| title_short | Topics in Sparsity and Compression: From High dimensional
statistics to Overparametrized Neural Networks |
| title_sort | topics in sparsity and compression from high dimensional statistics to overparametrized neural networks |
| url | https://hdl.handle.net/1721.1/151535 |
| work_keys_str_mv | AT benbakiriade topicsinsparsityandcompressionfromhighdimensionalstatisticstooverparametrizedneuralnetworks |