Multimodal Autoencoder: A Deep Learning Approach to Filling In Missing Sensor Data and Enabling Better Mood Prediction
© 2017 IEEE. To accomplish forecasting of mood in real-world situations, affective computing systems need to collect and learn from multimodal data collected over weeks or months of daily use. Such systems are likely to encounter frequent data loss, e.g. when a phone loses location access, or when a...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Institute of Electrical and Electronics Engineers (IEEE)
2021
|
| Online Access: | https://hdl.handle.net/1721.1/138088 |
| _version_ | 1811087946983407616 |
|---|---|
| author | Jaques, Natasha Taylor, Sara Sano, Akane Picard, Rosalind |
| author2 | Massachusetts Institute of Technology. Media Laboratory |
| author_facet | Massachusetts Institute of Technology. Media Laboratory Jaques, Natasha Taylor, Sara Sano, Akane Picard, Rosalind |
| author_sort | Jaques, Natasha |
| collection | MIT |
| description | © 2017 IEEE. To accomplish forecasting of mood in real-world situations, affective computing systems need to collect and learn from multimodal data collected over weeks or months of daily use. Such systems are likely to encounter frequent data loss, e.g. when a phone loses location access, or when a sensor is recharging. Lost data can handicap classifiers trained with all modalities present in the data. This paper describes a new technique for handling missing multimodal data using a specialized denoising autoencoder: The Multimodal Autoencoder (MMAE). Empirical results from over 200 participants and 5500 days of data demonstrate that the MMAE is able to predict the feature values from multiple missing modalities more accurately than reconstruction methods such as principal components analysis (PCA). We discuss several practical benefits of the MMAE's encoding and show that it can provide robust mood prediction even when up to three quarters of the data sources are lost. |
| first_indexed | 2024-09-23T13:54:20Z |
| format | Article |
| id | mit-1721.1/138088 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T13:54:20Z |
| publishDate | 2021 |
| publisher | Institute of Electrical and Electronics Engineers (IEEE) |
| record_format | dspace |
| spelling | mit-1721.1/1380882021-11-10T03:21:29Z Multimodal Autoencoder: A Deep Learning Approach to Filling In Missing Sensor Data and Enabling Better Mood Prediction Jaques, Natasha Taylor, Sara Sano, Akane Picard, Rosalind Massachusetts Institute of Technology. Media Laboratory © 2017 IEEE. To accomplish forecasting of mood in real-world situations, affective computing systems need to collect and learn from multimodal data collected over weeks or months of daily use. Such systems are likely to encounter frequent data loss, e.g. when a phone loses location access, or when a sensor is recharging. Lost data can handicap classifiers trained with all modalities present in the data. This paper describes a new technique for handling missing multimodal data using a specialized denoising autoencoder: The Multimodal Autoencoder (MMAE). Empirical results from over 200 participants and 5500 days of data demonstrate that the MMAE is able to predict the feature values from multiple missing modalities more accurately than reconstruction methods such as principal components analysis (PCA). We discuss several practical benefits of the MMAE's encoding and show that it can provide robust mood prediction even when up to three quarters of the data sources are lost. 2021-11-09T21:53:21Z 2021-11-09T21:53:21Z 2017-10 2019-08-02T11:22:22Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/138088 Jaques, Natasha, Taylor, Sara, Sano, Akane and Picard, Rosalind. 2017. "Multimodal Autoencoder: A Deep Learning Approach to Filling In Missing Sensor Data and Enabling Better Mood Prediction." en 10.1109/acii.2017.8273601 Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf Institute of Electrical and Electronics Engineers (IEEE) MIT web domain |
| spellingShingle | Jaques, Natasha Taylor, Sara Sano, Akane Picard, Rosalind Multimodal Autoencoder: A Deep Learning Approach to Filling In Missing Sensor Data and Enabling Better Mood Prediction |
| title | Multimodal Autoencoder: A Deep Learning Approach to Filling In Missing Sensor Data and Enabling Better Mood Prediction |
| title_full | Multimodal Autoencoder: A Deep Learning Approach to Filling In Missing Sensor Data and Enabling Better Mood Prediction |
| title_fullStr | Multimodal Autoencoder: A Deep Learning Approach to Filling In Missing Sensor Data and Enabling Better Mood Prediction |
| title_full_unstemmed | Multimodal Autoencoder: A Deep Learning Approach to Filling In Missing Sensor Data and Enabling Better Mood Prediction |
| title_short | Multimodal Autoencoder: A Deep Learning Approach to Filling In Missing Sensor Data and Enabling Better Mood Prediction |
| title_sort | multimodal autoencoder a deep learning approach to filling in missing sensor data and enabling better mood prediction |
| url | https://hdl.handle.net/1721.1/138088 |
| work_keys_str_mv | AT jaquesnatasha multimodalautoencoderadeeplearningapproachtofillinginmissingsensordataandenablingbettermoodprediction AT taylorsara multimodalautoencoderadeeplearningapproachtofillinginmissingsensordataandenablingbettermoodprediction AT sanoakane multimodalautoencoderadeeplearningapproachtofillinginmissingsensordataandenablingbettermoodprediction AT picardrosalind multimodalautoencoderadeeplearningapproachtofillinginmissingsensordataandenablingbettermoodprediction |