Generalized self-cueing real-time attention scheduling with intermittent inspection and image resizing
Abstract This paper proposes a generalized self-cueing real-time attention scheduling framework for DNN-based visual machine perception pipelines on resource-limited embedded platforms. Self-cueing means we identify subframe-level regions of interest in a scene internally by exploitin...
| Main Authors: | , , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Springer US
2023
|
| Online Access: | https://hdl.handle.net/1721.1/150910 |
| _version_ | 1811075688120188928 |
|---|---|
| author | Liu, Shengzhong Fu, Xinzhe Hu, Yigong Wigness, Maggie David, Philip Yao, Shuochao Sha, Lui Abdelzaher, Tarek |
| author2 | Massachusetts Institute of Technology. Laboratory for Information and Decision Systems |
| author_facet | Massachusetts Institute of Technology. Laboratory for Information and Decision Systems Liu, Shengzhong Fu, Xinzhe Hu, Yigong Wigness, Maggie David, Philip Yao, Shuochao Sha, Lui Abdelzaher, Tarek |
| author_sort | Liu, Shengzhong |
| collection | MIT |
| description | Abstract
This paper proposes a generalized self-cueing real-time attention scheduling framework for DNN-based visual machine perception pipelines on resource-limited embedded platforms. Self-cueing means we identify subframe-level regions of interest in a scene internally by exploiting temporal correlations among successive video frames as opposed to externally via a cueing sensor. One limitation of our original self-cueing-and-inspection strategy (Liu et al. in Proceedings of the 28th IEEE real-time and embedded technology and applications symposium (RTAS), 2022b) lies in its lack of computational efficiency under high workloads, like busy traffic scenarios where a large number of objects are identified and separately inspected. We extend the conference publication by integrating image resizing with intermittent inspection and task batching in attention scheduling. The extension enhances the original algorithm by accelerating the processing of large objects by reducing their resolution at the cost of only a negligible degradation in accuracy, thereby achieving a higher overall object inspection throughput. After extracting partial regions around objects of interest, using an optical flow-based tracking algorithm, we allocate computation resources (i.e. DNN inspection) to them in a criticality-aware manner using a generalized batched proportional balancing algorithm (GBPB), to minimize a concept of generalized system uncertainty. It saves computational resources by inspecting low-priority regions intermittently at low frequencies and inspecting large objects at low resolutions. We implement the system on an NVIDIA Jetson Xavier platform and extensively evaluate its performance using a real-world driving dataset from Waymo. The proposed GBPB algorithm consistently outperforms the previous BPB algorithm that only uses intermittent inspection and a set of baselines. The performance gain of GBPB is larger in facing more significant resource constraints (i.e., lower sampling intervals or busy traffic scenarios) because its multi-dimensional scheduling strategy achieves better resource allocation of machine perception. |
| first_indexed | 2024-09-23T10:10:21Z |
| format | Article |
| id | mit-1721.1/150910 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T10:10:21Z |
| publishDate | 2023 |
| publisher | Springer US |
| record_format | dspace |
| spelling | mit-1721.1/1509102024-01-29T19:00:48Z Generalized self-cueing real-time attention scheduling with intermittent inspection and image resizing Liu, Shengzhong Fu, Xinzhe Hu, Yigong Wigness, Maggie David, Philip Yao, Shuochao Sha, Lui Abdelzaher, Tarek Massachusetts Institute of Technology. Laboratory for Information and Decision Systems Massachusetts Institute of Technology. Department of Aeronautics and Astronautics Statistics and Data Science Center (Massachusetts Institute of Technology) Abstract This paper proposes a generalized self-cueing real-time attention scheduling framework for DNN-based visual machine perception pipelines on resource-limited embedded platforms. Self-cueing means we identify subframe-level regions of interest in a scene internally by exploiting temporal correlations among successive video frames as opposed to externally via a cueing sensor. One limitation of our original self-cueing-and-inspection strategy (Liu et al. in Proceedings of the 28th IEEE real-time and embedded technology and applications symposium (RTAS), 2022b) lies in its lack of computational efficiency under high workloads, like busy traffic scenarios where a large number of objects are identified and separately inspected. We extend the conference publication by integrating image resizing with intermittent inspection and task batching in attention scheduling. The extension enhances the original algorithm by accelerating the processing of large objects by reducing their resolution at the cost of only a negligible degradation in accuracy, thereby achieving a higher overall object inspection throughput. After extracting partial regions around objects of interest, using an optical flow-based tracking algorithm, we allocate computation resources (i.e. DNN inspection) to them in a criticality-aware manner using a generalized batched proportional balancing algorithm (GBPB), to minimize a concept of generalized system uncertainty. It saves computational resources by inspecting low-priority regions intermittently at low frequencies and inspecting large objects at low resolutions. We implement the system on an NVIDIA Jetson Xavier platform and extensively evaluate its performance using a real-world driving dataset from Waymo. The proposed GBPB algorithm consistently outperforms the previous BPB algorithm that only uses intermittent inspection and a set of baselines. The performance gain of GBPB is larger in facing more significant resource constraints (i.e., lower sampling intervals or busy traffic scenarios) because its multi-dimensional scheduling strategy achieves better resource allocation of machine perception. 2023-06-15T17:03:19Z 2023-06-15T17:03:19Z 2023-06-08 2023-06-13T03:26:53Z Article http://purl.org/eprint/type/JournalArticle https://hdl.handle.net/1721.1/150910 Liu, Shengzhong, Fu, Xinzhe, Hu, Yigong, Wigness, Maggie, David, Philip et al. 2023. "Generalized self-cueing real-time attention scheduling with intermittent inspection and image resizing." en https://doi.org/10.1007/s11241-023-09396-z Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature application/pdf Springer US Springer US |
| spellingShingle | Liu, Shengzhong Fu, Xinzhe Hu, Yigong Wigness, Maggie David, Philip Yao, Shuochao Sha, Lui Abdelzaher, Tarek Generalized self-cueing real-time attention scheduling with intermittent inspection and image resizing |
| title | Generalized self-cueing real-time attention scheduling with intermittent inspection and image resizing |
| title_full | Generalized self-cueing real-time attention scheduling with intermittent inspection and image resizing |
| title_fullStr | Generalized self-cueing real-time attention scheduling with intermittent inspection and image resizing |
| title_full_unstemmed | Generalized self-cueing real-time attention scheduling with intermittent inspection and image resizing |
| title_short | Generalized self-cueing real-time attention scheduling with intermittent inspection and image resizing |
| title_sort | generalized self cueing real time attention scheduling with intermittent inspection and image resizing |
| url | https://hdl.handle.net/1721.1/150910 |
| work_keys_str_mv | AT liushengzhong generalizedselfcueingrealtimeattentionschedulingwithintermittentinspectionandimageresizing AT fuxinzhe generalizedselfcueingrealtimeattentionschedulingwithintermittentinspectionandimageresizing AT huyigong generalizedselfcueingrealtimeattentionschedulingwithintermittentinspectionandimageresizing AT wignessmaggie generalizedselfcueingrealtimeattentionschedulingwithintermittentinspectionandimageresizing AT davidphilip generalizedselfcueingrealtimeattentionschedulingwithintermittentinspectionandimageresizing AT yaoshuochao generalizedselfcueingrealtimeattentionschedulingwithintermittentinspectionandimageresizing AT shalui generalizedselfcueingrealtimeattentionschedulingwithintermittentinspectionandimageresizing AT abdelzahertarek generalizedselfcueingrealtimeattentionschedulingwithintermittentinspectionandimageresizing |