Towards Automated Semantic Explainability of Multimedia Feature Graphs
Multimedia feature graphs are employed to represent features of images, video, audio, or text. Various techniques exist to extract such features from multimedia objects. In this paper, we describe the extension of such a feature graph to represent the <i>meaning</i> of such multimedia fe...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-12-01
|
| Series: | Information |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2078-2489/12/12/502 |
| _version_ | 1797503675864711168 |
|---|---|
| author | Stefan Wagenpfeil Paul Mc Kevitt Matthias Hemmje |
| author_facet | Stefan Wagenpfeil Paul Mc Kevitt Matthias Hemmje |
| author_sort | Stefan Wagenpfeil |
| collection | DOAJ |
| description | Multimedia feature graphs are employed to represent features of images, video, audio, or text. Various techniques exist to extract such features from multimedia objects. In this paper, we describe the extension of such a feature graph to represent the <i>meaning</i> of such multimedia features and introduce a formal context-free PS-grammar (Phrase Structure grammar) to automatically generate human-understandable natural language expressions based on such features. To achieve this, we define a semantic extension to syntactic multimedia feature graphs and introduce a set of production rules for phrases of natural language English expressions. This explainability, which is founded on a semantic model provides the opportunity to represent any multimedia feature in a human-readable and human-understandable form, which largely closes the gap between the technical representation of such features and their semantics. We show how this explainability can be formally defined and demonstrate the corresponding implementation based on our generic multimedia analysis framework. Furthermore, we show how this semantic extension can be employed to increase the effectiveness in precision and recall experiments. |
| first_indexed | 2024-03-10T03:54:07Z |
| format | Article |
| id | doaj.art-d6bb94e7fa974641afd0f0f63e8110a7 |
| institution | Directory Open Access Journal |
| issn | 2078-2489 |
| language | English |
| last_indexed | 2024-03-10T03:54:07Z |
| publishDate | 2021-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Information |
| spelling | doaj.art-d6bb94e7fa974641afd0f0f63e8110a72023-11-23T08:51:26ZengMDPI AGInformation2078-24892021-12-01121250210.3390/info12120502Towards Automated Semantic Explainability of Multimedia Feature GraphsStefan Wagenpfeil0Paul Mc Kevitt1Matthias Hemmje2Faculty of Mathematics and Computer Science, University of Hagen, D-58097 Hagen, GermanyAcademy for International Science & Research (AISR), Londonderry BT48 7ER, UKFaculty of Mathematics and Computer Science, University of Hagen, D-58097 Hagen, GermanyMultimedia feature graphs are employed to represent features of images, video, audio, or text. Various techniques exist to extract such features from multimedia objects. In this paper, we describe the extension of such a feature graph to represent the <i>meaning</i> of such multimedia features and introduce a formal context-free PS-grammar (Phrase Structure grammar) to automatically generate human-understandable natural language expressions based on such features. To achieve this, we define a semantic extension to syntactic multimedia feature graphs and introduce a set of production rules for phrases of natural language English expressions. This explainability, which is founded on a semantic model provides the opportunity to represent any multimedia feature in a human-readable and human-understandable form, which largely closes the gap between the technical representation of such features and their semantics. We show how this explainability can be formally defined and demonstrate the corresponding implementation based on our generic multimedia analysis framework. Furthermore, we show how this semantic extension can be employed to increase the effectiveness in precision and recall experiments.https://www.mdpi.com/2078-2489/12/12/502indexingretrievalexplainabilitysemanticmultimediafeature graph |
| spellingShingle | Stefan Wagenpfeil Paul Mc Kevitt Matthias Hemmje Towards Automated Semantic Explainability of Multimedia Feature Graphs Information indexing retrieval explainability semantic multimedia feature graph |
| title | Towards Automated Semantic Explainability of Multimedia Feature Graphs |
| title_full | Towards Automated Semantic Explainability of Multimedia Feature Graphs |
| title_fullStr | Towards Automated Semantic Explainability of Multimedia Feature Graphs |
| title_full_unstemmed | Towards Automated Semantic Explainability of Multimedia Feature Graphs |
| title_short | Towards Automated Semantic Explainability of Multimedia Feature Graphs |
| title_sort | towards automated semantic explainability of multimedia feature graphs |
| topic | indexing retrieval explainability semantic multimedia feature graph |
| url | https://www.mdpi.com/2078-2489/12/12/502 |
| work_keys_str_mv | AT stefanwagenpfeil towardsautomatedsemanticexplainabilityofmultimediafeaturegraphs AT paulmckevitt towardsautomatedsemanticexplainabilityofmultimediafeaturegraphs AT matthiashemmje towardsautomatedsemanticexplainabilityofmultimediafeaturegraphs |