Multiattribute multitask transformer framework for vision-based structural health monitoring
Using deep learning (DL) to recognize building and infrastructure damage via images is becoming popular in vision-based structural health monitoring (SHM). However, many previous studies solely work on the existence of damage in the images and directly treat the problem as a single-attribute classif...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/174338 |
| _version_ | 1811694170529595392 |
|---|---|
| author | Gao, Yuqing Yang, Jianfei Qian, Hanjie Mosalam, Khalid M. |
| author2 | School of Electrical and Electronic Engineering |
| author_facet | School of Electrical and Electronic Engineering Gao, Yuqing Yang, Jianfei Qian, Hanjie Mosalam, Khalid M. |
| author_sort | Gao, Yuqing |
| collection | NTU |
| description | Using deep learning (DL) to recognize building and infrastructure damage via images is becoming popular in vision-based structural health monitoring (SHM). However, many previous studies solely work on the existence of damage in the images and directly treat the problem as a single-attribute classification or separately focus on finding the location or area of the damage as a localization or segmentation problem. Abundant information in the images from multiple sources and intertask relationships are not fully exploited. In this study, the vision-based SHM problem is first reformulated into a multiattribute multitask setting, where each image contains multiple labels to describe its characteristics. Subsequently, a general multiattribute multitask detection framework, namely ϕ-NeXt, is proposed, which introduces 10 benchmark tasks including classification, localization, and segmentation tasks. Accordingly, a large-scale data set containing 37,000 pairs of multilabeled images is established. To pursue better performance in all tasks, a novel hierarchical framework, namely multiattribute multitask transformer (MAMT2) is proposed, which integrates multitask transfer learning mechanisms and adopts a transformer-based network as the backbone. Finally, for benchmarking purposes, extensive experiments are conducted on all tasks and the performance of the proposed MAMT2 is compared with several classical DL models. The results demonstrate the superiority of the MAMT2 in all tasks, which reveals a great potential for practical applications and future studies in both structural engineering and computer vision. |
| first_indexed | 2024-10-01T07:03:19Z |
| format | Journal Article |
| id | ntu-10356/174338 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T07:03:19Z |
| publishDate | 2024 |
| record_format | dspace |
| spelling | ntu-10356/1743382024-03-29T15:40:31Z Multiattribute multitask transformer framework for vision-based structural health monitoring Gao, Yuqing Yang, Jianfei Qian, Hanjie Mosalam, Khalid M. School of Electrical and Electronic Engineering Engineering Transfer learning Vision based Using deep learning (DL) to recognize building and infrastructure damage via images is becoming popular in vision-based structural health monitoring (SHM). However, many previous studies solely work on the existence of damage in the images and directly treat the problem as a single-attribute classification or separately focus on finding the location or area of the damage as a localization or segmentation problem. Abundant information in the images from multiple sources and intertask relationships are not fully exploited. In this study, the vision-based SHM problem is first reformulated into a multiattribute multitask setting, where each image contains multiple labels to describe its characteristics. Subsequently, a general multiattribute multitask detection framework, namely ϕ-NeXt, is proposed, which introduces 10 benchmark tasks including classification, localization, and segmentation tasks. Accordingly, a large-scale data set containing 37,000 pairs of multilabeled images is established. To pursue better performance in all tasks, a novel hierarchical framework, namely multiattribute multitask transformer (MAMT2) is proposed, which integrates multitask transfer learning mechanisms and adopts a transformer-based network as the backbone. Finally, for benchmarking purposes, extensive experiments are conducted on all tasks and the performance of the proposed MAMT2 is compared with several classical DL models. The results demonstrate the superiority of the MAMT2 in all tasks, which reveals a great potential for practical applications and future studies in both structural engineering and computer vision. Published version This research received funding support from: (i) California Department of Transportation (Caltarns) for the “Bridge Rapid Assessment Center for Extreme Events (BRACE2)” project, Task Order 001 of the PEER-Bridge Program agreement 65A0774 to the Pacific Earthquake Engineering Research (PEER) Center, (ii) Artificial Intelligence Institute for Food Systems (AIFS), https://aifs.ucdavis.edu/, and (iii) Taisei Chair of Civil Engineering at the University of California, Berkeley. 2024-03-27T00:35:28Z 2024-03-27T00:35:28Z 2023 Journal Article Gao, Y., Yang, J., Qian, H. & Mosalam, K. M. (2023). Multiattribute multitask transformer framework for vision-based structural health monitoring. Computer-Aided Civil and Infrastructure Engineering, 38(17), 2358-2377. https://dx.doi.org/10.1111/mice.13067 1093-9687 https://hdl.handle.net/10356/174338 10.1111/mice.13067 2-s2.0-85164310471 17 38 2358 2377 en Computer-Aided Civil and Infrastructure Engineering © 2023 The Authors. Computer-Aided Civil and Infrastructure Engineering published by Wiley Periodicals LLC on behalf of Editor. This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. application/pdf |
| spellingShingle | Engineering Transfer learning Vision based Gao, Yuqing Yang, Jianfei Qian, Hanjie Mosalam, Khalid M. Multiattribute multitask transformer framework for vision-based structural health monitoring |
| title | Multiattribute multitask transformer framework for vision-based structural health monitoring |
| title_full | Multiattribute multitask transformer framework for vision-based structural health monitoring |
| title_fullStr | Multiattribute multitask transformer framework for vision-based structural health monitoring |
| title_full_unstemmed | Multiattribute multitask transformer framework for vision-based structural health monitoring |
| title_short | Multiattribute multitask transformer framework for vision-based structural health monitoring |
| title_sort | multiattribute multitask transformer framework for vision based structural health monitoring |
| topic | Engineering Transfer learning Vision based |
| url | https://hdl.handle.net/10356/174338 |
| work_keys_str_mv | AT gaoyuqing multiattributemultitasktransformerframeworkforvisionbasedstructuralhealthmonitoring AT yangjianfei multiattributemultitasktransformerframeworkforvisionbasedstructuralhealthmonitoring AT qianhanjie multiattributemultitasktransformerframeworkforvisionbasedstructuralhealthmonitoring AT mosalamkhalidm multiattributemultitasktransformerframeworkforvisionbasedstructuralhealthmonitoring |