A comparison of distributed machine learning methods for the support of “many labs” collaborations in computational modeling of decision making
Deep learning models are powerful tools for representing the complex learning processes and decision-making strategies used by humans. Such neural network models make fewer assumptions about the underlying mechanisms thus providing experimental flexibility in terms of applicability. However, this co...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2022-08-01
|
| Series: | Frontiers in Psychology |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fpsyg.2022.943198/full |
| _version_ | 1828355078334971904 |
|---|---|
| author | Lili Zhang Lili Zhang Himanshu Vashisht Andrey Totev Nam Trinh Tomas Ward Tomas Ward |
| author_facet | Lili Zhang Lili Zhang Himanshu Vashisht Andrey Totev Nam Trinh Tomas Ward Tomas Ward |
| author_sort | Lili Zhang |
| collection | DOAJ |
| description | Deep learning models are powerful tools for representing the complex learning processes and decision-making strategies used by humans. Such neural network models make fewer assumptions about the underlying mechanisms thus providing experimental flexibility in terms of applicability. However, this comes at the cost of involving a larger number of parameters requiring significantly more data for effective learning. This presents practical challenges given that most cognitive experiments involve relatively small numbers of subjects. Laboratory collaborations are a natural way to increase overall dataset size. However, data sharing barriers between laboratories as necessitated by data protection regulations encourage the search for alternative methods to enable collaborative data science. Distributed learning, especially federated learning (FL), which supports the preservation of data privacy, is a promising method for addressing this issue. To verify the reliability and feasibility of applying FL to train neural networks models used in the characterization of decision making, we conducted experiments on a real-world, many-labs data pool including experiment data-sets from ten independent studies. The performance of single models trained on single laboratory data-sets was poor. This unsurprising finding supports the need for laboratory collaboration to train more reliable models. To that end we evaluated four collaborative approaches. The first approach represents conventional centralized learning (CL-based) and is the optimal approach but requires complete sharing of data which we wish to avoid. The results however establish a benchmark for the other three approaches, federated learning (FL-based), incremental learning (IL-based), and cyclic incremental learning (CIL-based). We evaluate these approaches in terms of prediction accuracy and capacity to characterize human decision-making strategies. The FL-based model achieves performance most comparable to that of the CL-based model. This indicates that FL has value in scaling data science methods to data collected in computational modeling contexts when data sharing is not convenient, practical or permissible. |
| first_indexed | 2024-04-14T02:35:29Z |
| format | Article |
| id | doaj.art-5351e9fb15654c269ffddcdfb89d2de4 |
| institution | Directory Open Access Journal |
| issn | 1664-1078 |
| language | English |
| last_indexed | 2024-04-14T02:35:29Z |
| publishDate | 2022-08-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Psychology |
| spelling | doaj.art-5351e9fb15654c269ffddcdfb89d2de42022-12-22T02:17:26ZengFrontiers Media S.A.Frontiers in Psychology1664-10782022-08-011310.3389/fpsyg.2022.943198943198A comparison of distributed machine learning methods for the support of “many labs” collaborations in computational modeling of decision makingLili Zhang0Lili Zhang1Himanshu Vashisht2Andrey Totev3Nam Trinh4Tomas Ward5Tomas Ward6School of Computing, Dublin City University, Dublin, IrelandInsight Science Foundation Ireland Research Centre for Data Analytics, Dublin, IrelandIn the Wild Research Limited, Dublin, IrelandSchool of Computing, Dublin City University, Dublin, IrelandSchool of Computing, Dublin City University, Dublin, IrelandSchool of Computing, Dublin City University, Dublin, IrelandInsight Science Foundation Ireland Research Centre for Data Analytics, Dublin, IrelandDeep learning models are powerful tools for representing the complex learning processes and decision-making strategies used by humans. Such neural network models make fewer assumptions about the underlying mechanisms thus providing experimental flexibility in terms of applicability. However, this comes at the cost of involving a larger number of parameters requiring significantly more data for effective learning. This presents practical challenges given that most cognitive experiments involve relatively small numbers of subjects. Laboratory collaborations are a natural way to increase overall dataset size. However, data sharing barriers between laboratories as necessitated by data protection regulations encourage the search for alternative methods to enable collaborative data science. Distributed learning, especially federated learning (FL), which supports the preservation of data privacy, is a promising method for addressing this issue. To verify the reliability and feasibility of applying FL to train neural networks models used in the characterization of decision making, we conducted experiments on a real-world, many-labs data pool including experiment data-sets from ten independent studies. The performance of single models trained on single laboratory data-sets was poor. This unsurprising finding supports the need for laboratory collaboration to train more reliable models. To that end we evaluated four collaborative approaches. The first approach represents conventional centralized learning (CL-based) and is the optimal approach but requires complete sharing of data which we wish to avoid. The results however establish a benchmark for the other three approaches, federated learning (FL-based), incremental learning (IL-based), and cyclic incremental learning (CIL-based). We evaluate these approaches in terms of prediction accuracy and capacity to characterize human decision-making strategies. The FL-based model achieves performance most comparable to that of the CL-based model. This indicates that FL has value in scaling data science methods to data collected in computational modeling contexts when data sharing is not convenient, practical or permissible.https://www.frontiersin.org/articles/10.3389/fpsyg.2022.943198/fulldeep learningdecision-makingdistributed learningfederated learningdata privacy |
| spellingShingle | Lili Zhang Lili Zhang Himanshu Vashisht Andrey Totev Nam Trinh Tomas Ward Tomas Ward A comparison of distributed machine learning methods for the support of “many labs” collaborations in computational modeling of decision making Frontiers in Psychology deep learning decision-making distributed learning federated learning data privacy |
| title | A comparison of distributed machine learning methods for the support of “many labs” collaborations in computational modeling of decision making |
| title_full | A comparison of distributed machine learning methods for the support of “many labs” collaborations in computational modeling of decision making |
| title_fullStr | A comparison of distributed machine learning methods for the support of “many labs” collaborations in computational modeling of decision making |
| title_full_unstemmed | A comparison of distributed machine learning methods for the support of “many labs” collaborations in computational modeling of decision making |
| title_short | A comparison of distributed machine learning methods for the support of “many labs” collaborations in computational modeling of decision making |
| title_sort | comparison of distributed machine learning methods for the support of many labs collaborations in computational modeling of decision making |
| topic | deep learning decision-making distributed learning federated learning data privacy |
| url | https://www.frontiersin.org/articles/10.3389/fpsyg.2022.943198/full |
| work_keys_str_mv | AT lilizhang acomparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking AT lilizhang acomparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking AT himanshuvashisht acomparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking AT andreytotev acomparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking AT namtrinh acomparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking AT tomasward acomparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking AT tomasward acomparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking AT lilizhang comparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking AT lilizhang comparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking AT himanshuvashisht comparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking AT andreytotev comparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking AT namtrinh comparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking AT tomasward comparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking AT tomasward comparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking |