Amount Estimation Method for Food Intake Based on Color and Depth Images through Deep Learning
In this paper, we propose an amount estimation method for food intake based on both color and depth images. Two pairs of color and depth images are captured pre- and post-meals. The pre- and post-meal color images are employed to detect food types and food existence regions using Mask R-CNN. The pos...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-03-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/24/7/2044 |
| _version_ | 1797212041457434624 |
|---|---|
| author | Dong-seok Lee Soon-kak Kwon |
| author_facet | Dong-seok Lee Soon-kak Kwon |
| author_sort | Dong-seok Lee |
| collection | DOAJ |
| description | In this paper, we propose an amount estimation method for food intake based on both color and depth images. Two pairs of color and depth images are captured pre- and post-meals. The pre- and post-meal color images are employed to detect food types and food existence regions using Mask R-CNN. The post-meal color image is spatially transformed to match the food region locations between the pre- and post-meal color images. The same transformation is also performed on the post-meal depth image. The pixel values of the post-meal depth image are compensated to reflect 3D position changes caused by the image transformation. In both the pre- and post-meal depth images, a space volume for each food region is calculated by dividing the space between the food surfaces and the camera into multiple tetrahedra. The food intake amounts are estimated as the difference in space volumes calculated from the pre- and post-meal depth images. From the simulation results, we verify that the proposed method estimates the food intake amount with an error of up to 2.2%. |
| first_indexed | 2024-04-24T10:36:05Z |
| format | Article |
| id | doaj.art-c57d0dcbfdc2499296025753d763cf28 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-04-24T10:36:05Z |
| publishDate | 2024-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-c57d0dcbfdc2499296025753d763cf282024-04-12T13:26:04ZengMDPI AGSensors1424-82202024-03-01247204410.3390/s24072044Amount Estimation Method for Food Intake Based on Color and Depth Images through Deep LearningDong-seok Lee0Soon-kak Kwon1AI Grand ICT Center, Dong-Eui University, Busan 47340, Republic of KoreaDepartment of Computer Software Engineering, Dong-Eui University, Busan 47340, Republic of KoreaIn this paper, we propose an amount estimation method for food intake based on both color and depth images. Two pairs of color and depth images are captured pre- and post-meals. The pre- and post-meal color images are employed to detect food types and food existence regions using Mask R-CNN. The post-meal color image is spatially transformed to match the food region locations between the pre- and post-meal color images. The same transformation is also performed on the post-meal depth image. The pixel values of the post-meal depth image are compensated to reflect 3D position changes caused by the image transformation. In both the pre- and post-meal depth images, a space volume for each food region is calculated by dividing the space between the food surfaces and the camera into multiple tetrahedra. The food intake amounts are estimated as the difference in space volumes calculated from the pre- and post-meal depth images. From the simulation results, we verify that the proposed method estimates the food intake amount with an error of up to 2.2%.https://www.mdpi.com/1424-8220/24/7/2044food intake amount estimationvolume estimationRGB-D imageobject detectiondeep learning |
| spellingShingle | Dong-seok Lee Soon-kak Kwon Amount Estimation Method for Food Intake Based on Color and Depth Images through Deep Learning Sensors food intake amount estimation volume estimation RGB-D image object detection deep learning |
| title | Amount Estimation Method for Food Intake Based on Color and Depth Images through Deep Learning |
| title_full | Amount Estimation Method for Food Intake Based on Color and Depth Images through Deep Learning |
| title_fullStr | Amount Estimation Method for Food Intake Based on Color and Depth Images through Deep Learning |
| title_full_unstemmed | Amount Estimation Method for Food Intake Based on Color and Depth Images through Deep Learning |
| title_short | Amount Estimation Method for Food Intake Based on Color and Depth Images through Deep Learning |
| title_sort | amount estimation method for food intake based on color and depth images through deep learning |
| topic | food intake amount estimation volume estimation RGB-D image object detection deep learning |
| url | https://www.mdpi.com/1424-8220/24/7/2044 |
| work_keys_str_mv | AT dongseoklee amountestimationmethodforfoodintakebasedoncoloranddepthimagesthroughdeeplearning AT soonkakkwon amountestimationmethodforfoodintakebasedoncoloranddepthimagesthroughdeeplearning |