Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones
Audio-visual correlation is a common phenomenon in real life. In this article, aiming at analyzing the correlation between multiple colors and combined tones, we comprehensively used experimental methods and technologies such as experimental psychology methods, audio-visual information processing te...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2022-12-01
|
| Series: | Frontiers in Psychology |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fpsyg.2022.970219/full |
| _version_ | 1811185943063822336 |
|---|---|
| author | Shuang Wang Shuang Wang Shuang Wang Jingyu Liu Jingyu Liu Jingyu Liu Xuedan Lan Xuedan Lan Xuedan Lan Qihang Hu Qihang Hu Qihang Hu Jian Jiang Jingjing Zhang Jingjing Zhang Jingjing Zhang Jingjing Zhang |
| author_facet | Shuang Wang Shuang Wang Shuang Wang Jingyu Liu Jingyu Liu Jingyu Liu Xuedan Lan Xuedan Lan Xuedan Lan Qihang Hu Qihang Hu Qihang Hu Jian Jiang Jingjing Zhang Jingjing Zhang Jingjing Zhang Jingjing Zhang |
| author_sort | Shuang Wang |
| collection | DOAJ |
| description | Audio-visual correlation is a common phenomenon in real life. In this article, aiming at analyzing the correlation between multiple colors and combined tones, we comprehensively used experimental methods and technologies such as experimental psychology methods, audio-visual information processing technology, and machine learning algorithms to study the correlation mechanism between the multi-color perceptual attributes and the interval consonance attribute of musical sounds, so as to construct an audio-visual cross-modal matching models. Specifically, in the first, this article constructed the multi-color perceptual attribute dataset through the subjective evaluation experiment, namely “cold/warm,” “soft/hard,” “transparent/turbid,” “far/near,” “weak/strong,” pleasure, arousal, and dominance; and constructed the interval consonance attribute dataset based on calculating the audio objective parameters. Secondly, a subjective evaluation experiment of cross-modal matching was designed and carried out for analyzing the audio-visual correlation, so as to obtain the cross-modal matched and mismatched data between the audio-visual perceptual attributes. On this basis, through visual processing and correlation analysis of the matched and mismatched data, this article proved that there is a certain correlation between multicolor and combined tones from the perspective of perceptual attributes. Finally, this article used linear and non-linear machine learning algorithms to construct audio-visual cross-modal matching models, so as to realize the mutual prediction between the audio-visual perceptual attributes, and the highest prediction accuracy is up to 79.1%. The contributions of our research are: (1) The cross-modal matched and mismatched dataset can provide basic data support for audio-visual cross-modal research; (2) The constructed audio-visual cross-modal matching models can provide a theoretical basis for audio-visual interaction technology; (3) In addition, the research method of audio-visual cross-modal matching proposed in this article can provide new research ideas for related research. |
| first_indexed | 2024-04-11T13:37:54Z |
| format | Article |
| id | doaj.art-b8c4fe1296274406bf3149ebf778f9da |
| institution | Directory Open Access Journal |
| issn | 1664-1078 |
| language | English |
| last_indexed | 2024-04-11T13:37:54Z |
| publishDate | 2022-12-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Psychology |
| spelling | doaj.art-b8c4fe1296274406bf3149ebf778f9da2022-12-22T04:21:24ZengFrontiers Media S.A.Frontiers in Psychology1664-10782022-12-011310.3389/fpsyg.2022.970219970219Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tonesShuang Wang0Shuang Wang1Shuang Wang2Jingyu Liu3Jingyu Liu4Jingyu Liu5Xuedan Lan6Xuedan Lan7Xuedan Lan8Qihang Hu9Qihang Hu10Qihang Hu11Jian Jiang12Jingjing Zhang13Jingjing Zhang14Jingjing Zhang15Jingjing Zhang16State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing, ChinaKey Laboratory of Acoustic Visual Technology and Intelligent Control System, Ministry of Culture and Tourism, Communication University of China, Beijing, ChinaBeijing Key Laboratory of Modern Entertainment Technology, Communication University of China, Beijing, ChinaState Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing, ChinaKey Laboratory of Acoustic Visual Technology and Intelligent Control System, Ministry of Culture and Tourism, Communication University of China, Beijing, ChinaBeijing Key Laboratory of Modern Entertainment Technology, Communication University of China, Beijing, ChinaState Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing, ChinaKey Laboratory of Acoustic Visual Technology and Intelligent Control System, Ministry of Culture and Tourism, Communication University of China, Beijing, ChinaBeijing Key Laboratory of Modern Entertainment Technology, Communication University of China, Beijing, ChinaState Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing, ChinaKey Laboratory of Acoustic Visual Technology and Intelligent Control System, Ministry of Culture and Tourism, Communication University of China, Beijing, ChinaBeijing Key Laboratory of Modern Entertainment Technology, Communication University of China, Beijing, ChinaChina Digital Culture Group Co., Ltd, Beijing, ChinaState Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing, ChinaKey Laboratory of Acoustic Visual Technology and Intelligent Control System, Ministry of Culture and Tourism, Communication University of China, Beijing, ChinaBeijing Key Laboratory of Modern Entertainment Technology, Communication University of China, Beijing, ChinaSchool of Computer and Cyber Sciences, Communication University of China, Beijing, ChinaAudio-visual correlation is a common phenomenon in real life. In this article, aiming at analyzing the correlation between multiple colors and combined tones, we comprehensively used experimental methods and technologies such as experimental psychology methods, audio-visual information processing technology, and machine learning algorithms to study the correlation mechanism between the multi-color perceptual attributes and the interval consonance attribute of musical sounds, so as to construct an audio-visual cross-modal matching models. Specifically, in the first, this article constructed the multi-color perceptual attribute dataset through the subjective evaluation experiment, namely “cold/warm,” “soft/hard,” “transparent/turbid,” “far/near,” “weak/strong,” pleasure, arousal, and dominance; and constructed the interval consonance attribute dataset based on calculating the audio objective parameters. Secondly, a subjective evaluation experiment of cross-modal matching was designed and carried out for analyzing the audio-visual correlation, so as to obtain the cross-modal matched and mismatched data between the audio-visual perceptual attributes. On this basis, through visual processing and correlation analysis of the matched and mismatched data, this article proved that there is a certain correlation between multicolor and combined tones from the perspective of perceptual attributes. Finally, this article used linear and non-linear machine learning algorithms to construct audio-visual cross-modal matching models, so as to realize the mutual prediction between the audio-visual perceptual attributes, and the highest prediction accuracy is up to 79.1%. The contributions of our research are: (1) The cross-modal matched and mismatched dataset can provide basic data support for audio-visual cross-modal research; (2) The constructed audio-visual cross-modal matching models can provide a theoretical basis for audio-visual interaction technology; (3) In addition, the research method of audio-visual cross-modal matching proposed in this article can provide new research ideas for related research.https://www.frontiersin.org/articles/10.3389/fpsyg.2022.970219/fullmultiple colorscombined tonesmulti-color perceptioninterval consonanceaudio-visual cross-modal matching modelsubjective evaluation experiment |
| spellingShingle | Shuang Wang Shuang Wang Shuang Wang Jingyu Liu Jingyu Liu Jingyu Liu Xuedan Lan Xuedan Lan Xuedan Lan Qihang Hu Qihang Hu Qihang Hu Jian Jiang Jingjing Zhang Jingjing Zhang Jingjing Zhang Jingjing Zhang Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones Frontiers in Psychology multiple colors combined tones multi-color perception interval consonance audio-visual cross-modal matching model subjective evaluation experiment |
| title | Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones |
| title_full | Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones |
| title_fullStr | Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones |
| title_full_unstemmed | Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones |
| title_short | Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones |
| title_sort | cross modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones |
| topic | multiple colors combined tones multi-color perception interval consonance audio-visual cross-modal matching model subjective evaluation experiment |
| url | https://www.frontiersin.org/articles/10.3389/fpsyg.2022.970219/full |
| work_keys_str_mv | AT shuangwang crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones AT shuangwang crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones AT shuangwang crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones AT jingyuliu crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones AT jingyuliu crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones AT jingyuliu crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones AT xuedanlan crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones AT xuedanlan crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones AT xuedanlan crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones AT qihanghu crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones AT qihanghu crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones AT qihanghu crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones AT jianjiang crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones AT jingjingzhang crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones AT jingjingzhang crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones AT jingjingzhang crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones AT jingjingzhang crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones |