Exploring the Use of Contrastive Language-Image Pre-Training for Human Posture Classification: Insights from Yoga Pose Analysis by Andrzej D. Dobrzycki, Ana M. Bernardos, Luca Bergesio, Andrzej Pomirski, Daniel Sáez-Trigueros

“…Accurate human posture classification in images and videos is crucial for automated applications across various fields, including work safety, physical rehabilitation, sports training, or daily assisted living. Recently, multimodal learning methods, such as Contrastive Language-Image Pretraining (CLIP), have advanced significantly in jointly understanding images and text. …”
Get full text

Empowering digital pathology applications through explainable knowledge extraction tools by Stefano Marchesin, Fabio Giachelle, Niccolò Marini, Manfredo Atzori, Svetla Boytcheva, Genziana Buttafuoco, Francesco Ciompi, Giorgio Maria Di Nunzio, Filippo Fraggetta, Ornella Irrera, Henning Müller, Todor Primov, Simona Vatrano, Gianmaria Silvello

“…This work demonstrates the viability of unsupervised Natural Language Processing (NLP) techniques to extract critical information from cancer reports, opening opportunities such as data mining for knowledge extraction purposes, precision medicine applications, structured report creation, and multimodal learning.SKET is a practical and unsupervised approach to extracting knowledge from pathology reports, which opens up unprecedented opportunities to exploit textual and multimodal medical information in clinical practice. …”
Get full text

Research Advances in Argument Mining by LI Jiao, ZHAO Ruixue, XIAN Guojian, HUANG Yongwen, SUN Tan

“…Possible research directions include: 1) the use of LLMs in argument mining, because they exhibit significant benefits in downstream applications such as natural language processing and multimodal learning, and can also provide certain technical conditions for the generation of argument content; 2) the use of domain knowledge organization systems such as vocabulary, knowledge base and knowledge graph: with these systems, researchers can combine domain-specific argument mining models with rich knowledge structure, to strengthen semantic representation and organization improve the systematization and dig deeper into argument mining model research in the domain; 3) promoting the application research and practice of argument mining in more fields or across disciplines, and improving the retrieval and visualization of argument information, such as combining information retrieval methods with argument mining to build the next generation of argument search engines.…”
Get full text

A multistage multimodal deep learning model for disease severity assessment and early warnings of high-risk patients of COVID-19 by Zhuo Li, Zhuo Li, Ruiqing Xu, Yifei Shen, Jiannong Cao, Ben Wang, Ying Zhang, Shikang Li

“…Also, comparison tests show the advantage of multimodal learning. MMDL with multimodal inputs can beat any reduced model with single-modal inputs only. …”
Get full text

Understanding image-text relations and news values for multimodal news analysis by Gullal S. Cheema, Sherzod Hakimov, Eric Müller-Budack, Eric Müller-Budack, Christian Otto, John A. Bateman, Ralph Ewerth, Ralph Ewerth

“…We assess and discuss the elements of the framework with real-world examples and use cases, setting out research directions at the intersection of multimodal learning, multimodal analytics and computational social sciences that can benefit from our approach.…”
Get full text

Spatial audio and spatial audio-visual learning by He, Y

“…How can we design robust audio-visual multimodal learning framework in embodied settings where sound and vision are weakly associated?…”

Artificial intelligence-based methods for fusion of electronic health records and imaging data by Farida Mohsen, Hazrat Ali, Nady El Hajj, Zubair Shah

“…Specifically, early fusion was the most used technique in most applications for multimodal learning (22 out of 34 studies). We found that multimodality fusion models outperformed traditional single-modality models for the same task. …”
Get full text

Analysing domain-specific problem-solving processes within authentic computer-based learning and training environments by using eye-tracking: a scoping review by Christian W. Mayer, Andreas Rausch, Jürgen Seifried

“…Also, post-hoc performance predictions are being developed for future integration into multimodal learning analytics. In most cases, self-reporting is used as an additional measurement for data triangulation. …”
Get full text

Sentiment analysis using image, text and video by Chen, Qian

“…To mine the correlated and complementary depression patterns in multimodal learning, we consider a chained-fusion mechanism to jointly learn facial appearance and dynamics in a unified framework. …”
Get full text

100 years of anthropogenic impact causes changes in freshwater functional biodiversity by Niamh Eastwood, Jiarui Zhou, Romain Derelle, Mohamed Abou-Elwafa Abdallah, William A Stubbings, Yunlu Jia, Sarah E Crawford, Thomas A Davidson, John K Colbourne, Simon Creer, Holly Bik, Henner Hollert, Luisa Orsini

“…We apply explainable network models with multimodal learning to community-level functional biodiversity measured with multilocus metabarcoding, to establish correlations with biocides and climate change records. …”
Get full text

Rethinking vision transformer and masked autoencoder in multimodal face anti-spoofing by Yu, Zitong, Cai, Rizhao, Cui, Yawen, Liu, Xin, Hu, Yongjian, Kot, Alex Chichung

“…Recently, vision transformer (ViT) based multimodal learning methods have been proposed to improve the robustness of face anti-spoofing (FAS) systems. …”
Get full text

The Multimodal-Based Learning Improves Students' Reading Ability in Perspective of Systemic Functional Linguistics by Ridwin Purba, Herman Herman, Endang Fatmawati, Nanda Saputra, Yusniati N. Sabata

“…Based on this, the multimodality learning method can be used to improve the analytical literacy talents of junior high school students in classes with various backgrounds. …”
Get full text