BP-YOLO: A Real-Time Product Detection and Shopping Behaviors Recognition Model for Intelligent Unmanned Vending Machine
Intelligent unmanned vending machines (UVMs) based on machine vision have attracted great attention in the unmanned retail industry. However, due to the complexity of practical application scenarios and environments, the existing vision-based intelligent UVMs face challenges related to missed-detect...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2024-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10418907/ |
| _version_ | 1797316067342680064 |
|---|---|
| author | Jingxiang Li Fuquan Tang Chao Zhu Shiwei He Shujin Zhang Yu Su |
| author_facet | Jingxiang Li Fuquan Tang Chao Zhu Shiwei He Shujin Zhang Yu Su |
| author_sort | Jingxiang Li |
| collection | DOAJ |
| description | Intelligent unmanned vending machines (UVMs) based on machine vision have attracted great attention in the unmanned retail industry. However, due to the complexity of practical application scenarios and environments, the existing vision-based intelligent UVMs face challenges related to missed-detection and mis-detection of product, and require costly physical components such as the infrared radio frequency sensors to capture shopping behaviors. In this study, we propose a BP-YOLO, the real-time model that integrates optimized YOLOv7 and BlazePose for product detection and shopping behaviors recognition. BP-YOLO can accurately detect the products purchased by consumers and their shopping behaviors in complex scenarios. To address the problems of missed-detection and mis-detection, we introduce the 3D attention mechanism SimAM and the deformable ConvNets v2 (DCNv2) to recombine and optimize the one-stage object detection model YOLOv7. This method reduces the interference of the invalid information in complex scenarios by adaptively weighting each channel and 3D spatial features, focuses on feature information in a sparse space, and minimizes the loss of feature information during the transmission process based on multi-scale feature extraction and fusion. To recognize and judge the shopping behaviors of consumers, we track the hand and arm key points of consumers using the pose estimation model BlazePose. Using the mAP@[0.5:0.95] as the evaluation metric for product detection, the experimental results on a customized product dataset show that BP-YOLO achieves an average accuracy of 96.17% for all product categories detection; the average success rate of consumer shopping recognition reaches 92%, 98%, and 94.7% under three light and noise intensity, respectively. Therefore, our BP-YOLO model for intelligent UVMs has effectiveness in commercial deployment. |
| first_indexed | 2024-03-08T03:12:52Z |
| format | Article |
| id | doaj.art-9112efbd2e1444acb194196d07066d1e |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-03-08T03:12:52Z |
| publishDate | 2024-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-9112efbd2e1444acb194196d07066d1e2024-02-13T00:01:23ZengIEEEIEEE Access2169-35362024-01-0112210382105110.1109/ACCESS.2024.336167510418907BP-YOLO: A Real-Time Product Detection and Shopping Behaviors Recognition Model for Intelligent Unmanned Vending MachineJingxiang Li0https://orcid.org/0009-0001-9526-1523Fuquan Tang1Chao Zhu2https://orcid.org/0000-0002-9998-098XShiwei He3Shujin Zhang4Yu Su5School of Geomatics, Xi’an University of Science and Technology, Xi’an, ChinaSchool of Geomatics, Xi’an University of Science and Technology, Xi’an, ChinaSchool of Geomatics, Xi’an University of Science and Technology, Xi’an, ChinaCourant Institute of Mathematical Science, New York University, New York, NY, USASchool of Mechanical and Electronic Engineering, Northwest A&F University, Xianyang, ChinaSchool of Geomatics, Xi’an University of Science and Technology, Xi’an, ChinaIntelligent unmanned vending machines (UVMs) based on machine vision have attracted great attention in the unmanned retail industry. However, due to the complexity of practical application scenarios and environments, the existing vision-based intelligent UVMs face challenges related to missed-detection and mis-detection of product, and require costly physical components such as the infrared radio frequency sensors to capture shopping behaviors. In this study, we propose a BP-YOLO, the real-time model that integrates optimized YOLOv7 and BlazePose for product detection and shopping behaviors recognition. BP-YOLO can accurately detect the products purchased by consumers and their shopping behaviors in complex scenarios. To address the problems of missed-detection and mis-detection, we introduce the 3D attention mechanism SimAM and the deformable ConvNets v2 (DCNv2) to recombine and optimize the one-stage object detection model YOLOv7. This method reduces the interference of the invalid information in complex scenarios by adaptively weighting each channel and 3D spatial features, focuses on feature information in a sparse space, and minimizes the loss of feature information during the transmission process based on multi-scale feature extraction and fusion. To recognize and judge the shopping behaviors of consumers, we track the hand and arm key points of consumers using the pose estimation model BlazePose. Using the mAP@[0.5:0.95] as the evaluation metric for product detection, the experimental results on a customized product dataset show that BP-YOLO achieves an average accuracy of 96.17% for all product categories detection; the average success rate of consumer shopping recognition reaches 92%, 98%, and 94.7% under three light and noise intensity, respectively. Therefore, our BP-YOLO model for intelligent UVMs has effectiveness in commercial deployment.https://ieeexplore.ieee.org/document/10418907/BP-YOLOBlazePoseproduct detectionshopping behaviors recognitionunmanned vending machines |
| spellingShingle | Jingxiang Li Fuquan Tang Chao Zhu Shiwei He Shujin Zhang Yu Su BP-YOLO: A Real-Time Product Detection and Shopping Behaviors Recognition Model for Intelligent Unmanned Vending Machine IEEE Access BP-YOLO BlazePose product detection shopping behaviors recognition unmanned vending machines |
| title | BP-YOLO: A Real-Time Product Detection and Shopping Behaviors Recognition Model for Intelligent Unmanned Vending Machine |
| title_full | BP-YOLO: A Real-Time Product Detection and Shopping Behaviors Recognition Model for Intelligent Unmanned Vending Machine |
| title_fullStr | BP-YOLO: A Real-Time Product Detection and Shopping Behaviors Recognition Model for Intelligent Unmanned Vending Machine |
| title_full_unstemmed | BP-YOLO: A Real-Time Product Detection and Shopping Behaviors Recognition Model for Intelligent Unmanned Vending Machine |
| title_short | BP-YOLO: A Real-Time Product Detection and Shopping Behaviors Recognition Model for Intelligent Unmanned Vending Machine |
| title_sort | bp yolo a real time product detection and shopping behaviors recognition model for intelligent unmanned vending machine |
| topic | BP-YOLO BlazePose product detection shopping behaviors recognition unmanned vending machines |
| url | https://ieeexplore.ieee.org/document/10418907/ |
| work_keys_str_mv | AT jingxiangli bpyoloarealtimeproductdetectionandshoppingbehaviorsrecognitionmodelforintelligentunmannedvendingmachine AT fuquantang bpyoloarealtimeproductdetectionandshoppingbehaviorsrecognitionmodelforintelligentunmannedvendingmachine AT chaozhu bpyoloarealtimeproductdetectionandshoppingbehaviorsrecognitionmodelforintelligentunmannedvendingmachine AT shiweihe bpyoloarealtimeproductdetectionandshoppingbehaviorsrecognitionmodelforintelligentunmannedvendingmachine AT shujinzhang bpyoloarealtimeproductdetectionandshoppingbehaviorsrecognitionmodelforintelligentunmannedvendingmachine AT yusu bpyoloarealtimeproductdetectionandshoppingbehaviorsrecognitionmodelforintelligentunmannedvendingmachine |