Harnessing LLMs for Automated Video Content Analysis: An Exploratory Workflow of Short Videos on Depression
CSCW Companion ’24, November 9–13, 2024, San Jose, Costa Rica
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
ACM|Companion of the 2024 Computer-Supported Cooperative Work and Social Computing
2024
|
| Online Access: | https://hdl.handle.net/1721.1/157763 |
| _version_ | 1824458393713115136 |
|---|---|
| author | Liu, Jiaying (Lizzy) Wang, Yunlong Lyu, Yao Su, Yiheng Niu, Shuo Xu, Xuhai Zhang, Yan |
| author_facet | Liu, Jiaying (Lizzy) Wang, Yunlong Lyu, Yao Su, Yiheng Niu, Shuo Xu, Xuhai Zhang, Yan |
| author_sort | Liu, Jiaying (Lizzy) |
| collection | MIT |
| description | CSCW Companion ’24, November 9–13, 2024, San Jose, Costa Rica |
| first_indexed | 2025-02-19T04:25:11Z |
| format | Article |
| id | mit-1721.1/157763 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2025-02-19T04:25:11Z |
| publishDate | 2024 |
| publisher | ACM|Companion of the 2024 Computer-Supported Cooperative Work and Social Computing |
| record_format | dspace |
| spelling | mit-1721.1/1577632025-01-05T04:17:02Z Harnessing LLMs for Automated Video Content Analysis: An Exploratory Workflow of Short Videos on Depression Liu, Jiaying (Lizzy) Wang, Yunlong Lyu, Yao Su, Yiheng Niu, Shuo Xu, Xuhai Zhang, Yan CSCW Companion ’24, November 9–13, 2024, San Jose, Costa Rica Despite the growing interest in leveraging Large Language Models (LLMs) for content analysis, current studies have primarily focused on text-based content. In the present work, we explored the potential of LLMs in assisting video content analysis by conducting a case study that followed a new workflow of LLM-assisted multimodal content analysis. The workflow encompasses codebook design, prompt engineering, LLM processing, and human evaluation. We strategically crafted annotation prompts to get LLM Annotations in structured form and explanation prompts to generate LLM Explanations for a better understanding of LLM reasoning and transparency. To test LLM's video annotation capabilities, we analyzed 203 keyframes extracted from 25 YouTube short videos about depression. We compared the LLM Annotations with those of two human coders and found that LLM has higher accuracy in object and activity Annotations than emotion and genre Annotations. Moreover, we identified the potential and limitations of LLM's capabilities in annotating videos. Based on the findings, we explore opportunities and challenges for future research and improvements to the workflow. We also discuss ethical concerns surrounding future studies based on LLM-assisted video analysis. 2024-12-05T21:56:15Z 2024-12-05T21:56:15Z 2024-11-11 2024-12-01T08:48:28Z Article http://purl.org/eprint/type/ConferencePaper 979-8-4007-1114-5 https://hdl.handle.net/1721.1/157763 Liu, Jiaying (Lizzy), Wang, Yunlong, Lyu, Yao, Su, Yiheng, Niu, Shuo et al. 2024. "Harnessing LLMs for Automated Video Content Analysis: An Exploratory Workflow of Short Videos on Depression." PUBLISHER_CC en https://doi.org/10.1145/3678884.3681850 Creative Commons Attribution-NonCommercial-NoDerivs https://creativecommons.org/licenses/by-nc-nd/4.0/ The author(s) application/pdf ACM|Companion of the 2024 Computer-Supported Cooperative Work and Social Computing Association for Computing Machinery |
| spellingShingle | Liu, Jiaying (Lizzy) Wang, Yunlong Lyu, Yao Su, Yiheng Niu, Shuo Xu, Xuhai Zhang, Yan Harnessing LLMs for Automated Video Content Analysis: An Exploratory Workflow of Short Videos on Depression |
| title | Harnessing LLMs for Automated Video Content Analysis: An Exploratory Workflow of Short Videos on Depression |
| title_full | Harnessing LLMs for Automated Video Content Analysis: An Exploratory Workflow of Short Videos on Depression |
| title_fullStr | Harnessing LLMs for Automated Video Content Analysis: An Exploratory Workflow of Short Videos on Depression |
| title_full_unstemmed | Harnessing LLMs for Automated Video Content Analysis: An Exploratory Workflow of Short Videos on Depression |
| title_short | Harnessing LLMs for Automated Video Content Analysis: An Exploratory Workflow of Short Videos on Depression |
| title_sort | harnessing llms for automated video content analysis an exploratory workflow of short videos on depression |
| url | https://hdl.handle.net/1721.1/157763 |
| work_keys_str_mv | AT liujiayinglizzy harnessingllmsforautomatedvideocontentanalysisanexploratoryworkflowofshortvideosondepression AT wangyunlong harnessingllmsforautomatedvideocontentanalysisanexploratoryworkflowofshortvideosondepression AT lyuyao harnessingllmsforautomatedvideocontentanalysisanexploratoryworkflowofshortvideosondepression AT suyiheng harnessingllmsforautomatedvideocontentanalysisanexploratoryworkflowofshortvideosondepression AT niushuo harnessingllmsforautomatedvideocontentanalysisanexploratoryworkflowofshortvideosondepression AT xuxuhai harnessingllmsforautomatedvideocontentanalysisanexploratoryworkflowofshortvideosondepression AT zhangyan harnessingllmsforautomatedvideocontentanalysisanexploratoryworkflowofshortvideosondepression |