Improved Information-Theoretic Generalization Bounds for Distributed, Federated, and Iterative Learning
We consider information-theoretic bounds on the expected generalization error for statistical learning problems in a network setting. In this setting, there are <i>K</i> nodes, each with its own independent dataset, and the models from the <i>K</i> nodes have to be aggregated...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-08-01
|
| Series: | Entropy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1099-4300/24/9/1178 |
| _version_ | 1797488733212114944 |
|---|---|
| author | Leighton Pate Barnes Alex Dytso Harold Vincent Poor |
| author_facet | Leighton Pate Barnes Alex Dytso Harold Vincent Poor |
| author_sort | Leighton Pate Barnes |
| collection | DOAJ |
| description | We consider information-theoretic bounds on the expected generalization error for statistical learning problems in a network setting. In this setting, there are <i>K</i> nodes, each with its own independent dataset, and the models from the <i>K</i> nodes have to be aggregated into a final centralized model. We consider both simple averaging of the models as well as more complicated multi-round algorithms. We give upper bounds on the expected generalization error for a variety of problems, such as those with Bregman divergence or Lipschitz continuous losses, that demonstrate an improved dependence of <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>1</mn><mo>/</mo><mi>K</mi></mrow></semantics></math></inline-formula> on the number of nodes. These “per node” bounds are in terms of the mutual information between the training dataset and the trained weights at each node and are therefore useful in describing the generalization properties inherent to having communication or privacy constraints at each node. |
| first_indexed | 2024-03-10T00:06:33Z |
| format | Article |
| id | doaj.art-af8804968996447497480c03aa2d1a3e |
| institution | Directory Open Access Journal |
| issn | 1099-4300 |
| language | English |
| last_indexed | 2024-03-10T00:06:33Z |
| publishDate | 2022-08-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Entropy |
| spelling | doaj.art-af8804968996447497480c03aa2d1a3e2023-11-23T16:07:17ZengMDPI AGEntropy1099-43002022-08-01249117810.3390/e24091178Improved Information-Theoretic Generalization Bounds for Distributed, Federated, and Iterative LearningLeighton Pate Barnes0Alex Dytso1Harold Vincent Poor2Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08544, USADepartment of Electrical and Computer Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USADepartment of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08544, USAWe consider information-theoretic bounds on the expected generalization error for statistical learning problems in a network setting. In this setting, there are <i>K</i> nodes, each with its own independent dataset, and the models from the <i>K</i> nodes have to be aggregated into a final centralized model. We consider both simple averaging of the models as well as more complicated multi-round algorithms. We give upper bounds on the expected generalization error for a variety of problems, such as those with Bregman divergence or Lipschitz continuous losses, that demonstrate an improved dependence of <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>1</mn><mo>/</mo><mi>K</mi></mrow></semantics></math></inline-formula> on the number of nodes. These “per node” bounds are in terms of the mutual information between the training dataset and the trained weights at each node and are therefore useful in describing the generalization properties inherent to having communication or privacy constraints at each node.https://www.mdpi.com/1099-4300/24/9/1178generalization errorinformation-theoretic boundsdistribution and federated learning |
| spellingShingle | Leighton Pate Barnes Alex Dytso Harold Vincent Poor Improved Information-Theoretic Generalization Bounds for Distributed, Federated, and Iterative Learning Entropy generalization error information-theoretic bounds distribution and federated learning |
| title | Improved Information-Theoretic Generalization Bounds for Distributed, Federated, and Iterative Learning |
| title_full | Improved Information-Theoretic Generalization Bounds for Distributed, Federated, and Iterative Learning |
| title_fullStr | Improved Information-Theoretic Generalization Bounds for Distributed, Federated, and Iterative Learning |
| title_full_unstemmed | Improved Information-Theoretic Generalization Bounds for Distributed, Federated, and Iterative Learning |
| title_short | Improved Information-Theoretic Generalization Bounds for Distributed, Federated, and Iterative Learning |
| title_sort | improved information theoretic generalization bounds for distributed federated and iterative learning |
| topic | generalization error information-theoretic bounds distribution and federated learning |
| url | https://www.mdpi.com/1099-4300/24/9/1178 |
| work_keys_str_mv | AT leightonpatebarnes improvedinformationtheoreticgeneralizationboundsfordistributedfederatedanditerativelearning AT alexdytso improvedinformationtheoreticgeneralizationboundsfordistributedfederatedanditerativelearning AT haroldvincentpoor improvedinformationtheoreticgeneralizationboundsfordistributedfederatedanditerativelearning |