pFedKT: personalized federated learning with dual knowledge transfer
Federated learning (FL) has been widely studied as an emerging privacy-preserving machine learning paradigm for achieving multi-party collaborative model training on decentralized data. In practice, such data tend to follow non-independent and identically distributed (non-IID) data distributions. Th...
| Main Authors: | , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/180139 |
| _version_ | 1811693383689699328 |
|---|---|
| author | Yi, Liping Shi, Xiaorong Wang, Nan Wang, Gang Liu, Xiaoguang Shi, Zhuan Yu, Han |
| author2 | School of Computer Science and Engineering |
| author_facet | School of Computer Science and Engineering Yi, Liping Shi, Xiaorong Wang, Nan Wang, Gang Liu, Xiaoguang Shi, Zhuan Yu, Han |
| author_sort | Yi, Liping |
| collection | NTU |
| description | Federated learning (FL) has been widely studied as an emerging privacy-preserving machine learning paradigm for achieving multi-party collaborative model training on decentralized data. In practice, such data tend to follow non-independent and identically distributed (non-IID) data distributions. Thus, the performance of models obtained through vanilla horizontal FL tends to vary significantly across FL clients. To tackle this challenge, a new subfield of FL – personalized federated learning (PFL) – has emerged for producing personalized FL models that can perform well on diverse local datasets. Existing PFL approaches are limited in terms of effectively transferring knowledge among clients to improve model generalization while achieving good performance on diverse local datasets. To bridge this important gap, we propose the personalized Federated Knowledge Transfer (pFedKT) approach. It involves dual knowledge transfer: (1) transferring historical local knowledge to local models via local hypernetworks; and (2) transferring latest global knowledge to local models through contrastive learning. By fusing historical local knowledge and the latest global knowledge, the personalization and generalization of individual models for FL clients can be simultaneously enhanced. We provide theoretical analysis on the generalization and convergence of pFedKT. Extensive experiments on 3 real-world datasets demonstrate that pFedKT achieves 0.74%–1.62% higher test accuracy compared to 14 state-of-the-art baselines. |
| first_indexed | 2024-10-01T06:50:49Z |
| format | Journal Article |
| id | ntu-10356/180139 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T06:50:49Z |
| publishDate | 2024 |
| record_format | dspace |
| spelling | ntu-10356/1801392024-09-18T07:09:59Z pFedKT: personalized federated learning with dual knowledge transfer Yi, Liping Shi, Xiaorong Wang, Nan Wang, Gang Liu, Xiaoguang Shi, Zhuan Yu, Han School of Computer Science and Engineering Computer and Information Science Personalized federated learning Knowledge transfer Federated learning (FL) has been widely studied as an emerging privacy-preserving machine learning paradigm for achieving multi-party collaborative model training on decentralized data. In practice, such data tend to follow non-independent and identically distributed (non-IID) data distributions. Thus, the performance of models obtained through vanilla horizontal FL tends to vary significantly across FL clients. To tackle this challenge, a new subfield of FL – personalized federated learning (PFL) – has emerged for producing personalized FL models that can perform well on diverse local datasets. Existing PFL approaches are limited in terms of effectively transferring knowledge among clients to improve model generalization while achieving good performance on diverse local datasets. To bridge this important gap, we propose the personalized Federated Knowledge Transfer (pFedKT) approach. It involves dual knowledge transfer: (1) transferring historical local knowledge to local models via local hypernetworks; and (2) transferring latest global knowledge to local models through contrastive learning. By fusing historical local knowledge and the latest global knowledge, the personalization and generalization of individual models for FL clients can be simultaneously enhanced. We provide theoretical analysis on the generalization and convergence of pFedKT. Extensive experiments on 3 real-world datasets demonstrate that pFedKT achieves 0.74%–1.62% higher test accuracy compared to 14 state-of-the-art baselines. Agency for Science, Technology and Research (A*STAR) National Research Foundation (NRF) This research is supported in part by the National Science Foundation of China under Grant 62272252 and 62272253, the Key Research and Development Program of Guangdong under Grant 2021B01013 10002, and the Fundamental Research Funds for the Central Universities; the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Programme (AISG Award No: AISG2-RP-2020-019); the RIE 2020 Advanced Manufacturing and Engineering (AME) Programmatic Fund (No. A20G8b0102), Singapore. 2024-09-18T07:09:58Z 2024-09-18T07:09:58Z 2024 Journal Article Yi, L., Shi, X., Wang, N., Wang, G., Liu, X., Shi, Z. & Yu, H. (2024). pFedKT: personalized federated learning with dual knowledge transfer. Knowledge-Based Systems, 292, 111633-. https://dx.doi.org/10.1016/j.knosys.2024.111633 0950-7051 https://hdl.handle.net/10356/180139 10.1016/j.knosys.2024.111633 2-s2.0-85187792396 292 111633 en AISG2-RP-2020-019 A20G8b0102 Knowledge-Based Systems © 2024 Elsevier B.V. All rights reserved. |
| spellingShingle | Computer and Information Science Personalized federated learning Knowledge transfer Yi, Liping Shi, Xiaorong Wang, Nan Wang, Gang Liu, Xiaoguang Shi, Zhuan Yu, Han pFedKT: personalized federated learning with dual knowledge transfer |
| title | pFedKT: personalized federated learning with dual knowledge transfer |
| title_full | pFedKT: personalized federated learning with dual knowledge transfer |
| title_fullStr | pFedKT: personalized federated learning with dual knowledge transfer |
| title_full_unstemmed | pFedKT: personalized federated learning with dual knowledge transfer |
| title_short | pFedKT: personalized federated learning with dual knowledge transfer |
| title_sort | pfedkt personalized federated learning with dual knowledge transfer |
| topic | Computer and Information Science Personalized federated learning Knowledge transfer |
| url | https://hdl.handle.net/10356/180139 |
| work_keys_str_mv | AT yiliping pfedktpersonalizedfederatedlearningwithdualknowledgetransfer AT shixiaorong pfedktpersonalizedfederatedlearningwithdualknowledgetransfer AT wangnan pfedktpersonalizedfederatedlearningwithdualknowledgetransfer AT wanggang pfedktpersonalizedfederatedlearningwithdualknowledgetransfer AT liuxiaoguang pfedktpersonalizedfederatedlearningwithdualknowledgetransfer AT shizhuan pfedktpersonalizedfederatedlearningwithdualknowledgetransfer AT yuhan pfedktpersonalizedfederatedlearningwithdualknowledgetransfer |