Optimizing Multi-Objective Federated Learning on Non-IID Data with Improved NSGA-III and Hierarchical Clustering
Federated learning (FL) can tackle the problem of data silos of asymmetric information and privacy leakage; however, it still has shortcomings, such as data heterogeneity, high communication cost and uneven distribution of performance. To overcome these issues and achieve parameter optimization of F...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-05-01
|
| Series: | Symmetry |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2073-8994/14/5/1070 |
| _version_ | 1797495000282431488 |
|---|---|
| author | Jialin Zhong Yahui Wu Wubin Ma Su Deng Haohao Zhou |
| author_facet | Jialin Zhong Yahui Wu Wubin Ma Su Deng Haohao Zhou |
| author_sort | Jialin Zhong |
| collection | DOAJ |
| description | Federated learning (FL) can tackle the problem of data silos of asymmetric information and privacy leakage; however, it still has shortcomings, such as data heterogeneity, high communication cost and uneven distribution of performance. To overcome these issues and achieve parameter optimization of FL on non-Independent Identically Distributed (non-IID) data, a multi-objective FL parameter optimization method based on hierarchical clustering and the third-generation non-dominated sorted genetic algorithm III (NSGA-III) algorithm is proposed, which aims to simultaneously minimize the global model error rate, global model accuracy distribution variance and communication cost. The introduction of a hierarchical clustering algorithm on non-IID data can accelerate convergence so that FL can employ an evolutionary algorithm with a low FL client participation ratio, reducing the overall communication cost of the NSGA-III algorithm. Meanwhile, the NSGA-III algorithm, with fast greedy initialization and a strategy of discarding low-quality individuals (named NSGA-III-FD), is proposed to improve the convergence efficiency and the quality of Pareto-optimal solutions. Under two non-IID data settings, the CNN experiments on both MNIST and CIFAR-10 datasets show that our approach can obtain better Pareto-optimal solutions than classical evolutionary algorithms, and the selected solutions with an optimized model can achieve better multi-objective equilibrium than the standard federated averaging (FedAvg) algorithm and the Clustering-based FedAvg algorithm. |
| first_indexed | 2024-03-10T01:43:15Z |
| format | Article |
| id | doaj.art-5975549949c441b29505d29c95f16845 |
| institution | Directory Open Access Journal |
| issn | 2073-8994 |
| language | English |
| last_indexed | 2024-03-10T01:43:15Z |
| publishDate | 2022-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Symmetry |
| spelling | doaj.art-5975549949c441b29505d29c95f168452023-11-23T13:20:56ZengMDPI AGSymmetry2073-89942022-05-01145107010.3390/sym14051070Optimizing Multi-Objective Federated Learning on Non-IID Data with Improved NSGA-III and Hierarchical ClusteringJialin Zhong0Yahui Wu1Wubin Ma2Su Deng3Haohao Zhou4Science and Technology on Information System Engineering Laboratory, National University of Defense Technology, Changsha 410073, ChinaScience and Technology on Information System Engineering Laboratory, National University of Defense Technology, Changsha 410073, ChinaScience and Technology on Information System Engineering Laboratory, National University of Defense Technology, Changsha 410073, ChinaScience and Technology on Information System Engineering Laboratory, National University of Defense Technology, Changsha 410073, ChinaScience and Technology on Information System Engineering Laboratory, National University of Defense Technology, Changsha 410073, ChinaFederated learning (FL) can tackle the problem of data silos of asymmetric information and privacy leakage; however, it still has shortcomings, such as data heterogeneity, high communication cost and uneven distribution of performance. To overcome these issues and achieve parameter optimization of FL on non-Independent Identically Distributed (non-IID) data, a multi-objective FL parameter optimization method based on hierarchical clustering and the third-generation non-dominated sorted genetic algorithm III (NSGA-III) algorithm is proposed, which aims to simultaneously minimize the global model error rate, global model accuracy distribution variance and communication cost. The introduction of a hierarchical clustering algorithm on non-IID data can accelerate convergence so that FL can employ an evolutionary algorithm with a low FL client participation ratio, reducing the overall communication cost of the NSGA-III algorithm. Meanwhile, the NSGA-III algorithm, with fast greedy initialization and a strategy of discarding low-quality individuals (named NSGA-III-FD), is proposed to improve the convergence efficiency and the quality of Pareto-optimal solutions. Under two non-IID data settings, the CNN experiments on both MNIST and CIFAR-10 datasets show that our approach can obtain better Pareto-optimal solutions than classical evolutionary algorithms, and the selected solutions with an optimized model can achieve better multi-objective equilibrium than the standard federated averaging (FedAvg) algorithm and the Clustering-based FedAvg algorithm.https://www.mdpi.com/2073-8994/14/5/1070federated learningmulti-objective optimizationNSGA-IIIparameter optimization |
| spellingShingle | Jialin Zhong Yahui Wu Wubin Ma Su Deng Haohao Zhou Optimizing Multi-Objective Federated Learning on Non-IID Data with Improved NSGA-III and Hierarchical Clustering Symmetry federated learning multi-objective optimization NSGA-III parameter optimization |
| title | Optimizing Multi-Objective Federated Learning on Non-IID Data with Improved NSGA-III and Hierarchical Clustering |
| title_full | Optimizing Multi-Objective Federated Learning on Non-IID Data with Improved NSGA-III and Hierarchical Clustering |
| title_fullStr | Optimizing Multi-Objective Federated Learning on Non-IID Data with Improved NSGA-III and Hierarchical Clustering |
| title_full_unstemmed | Optimizing Multi-Objective Federated Learning on Non-IID Data with Improved NSGA-III and Hierarchical Clustering |
| title_short | Optimizing Multi-Objective Federated Learning on Non-IID Data with Improved NSGA-III and Hierarchical Clustering |
| title_sort | optimizing multi objective federated learning on non iid data with improved nsga iii and hierarchical clustering |
| topic | federated learning multi-objective optimization NSGA-III parameter optimization |
| url | https://www.mdpi.com/2073-8994/14/5/1070 |
| work_keys_str_mv | AT jialinzhong optimizingmultiobjectivefederatedlearningonnoniiddatawithimprovednsgaiiiandhierarchicalclustering AT yahuiwu optimizingmultiobjectivefederatedlearningonnoniiddatawithimprovednsgaiiiandhierarchicalclustering AT wubinma optimizingmultiobjectivefederatedlearningonnoniiddatawithimprovednsgaiiiandhierarchicalclustering AT sudeng optimizingmultiobjectivefederatedlearningonnoniiddatawithimprovednsgaiiiandhierarchicalclustering AT haohaozhou optimizingmultiobjectivefederatedlearningonnoniiddatawithimprovednsgaiiiandhierarchicalclustering |