More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch
For humans, the process of grasping an object relies heavily on rich tactile feedback. Most recent robotic grasping work, however, has been based only on visual input, and thus cannot easily benefit from feedback after initiating contact. In this letter, we investigate how a robot can learn to use t...
| Main Authors: | , , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Institute of Electrical and Electronics Engineers (IEEE)
2020
|
| Online Access: | https://hdl.handle.net/1721.1/126806 |
| _version_ | 1811088519772241920 |
|---|---|
| author | Calandra, Roberto Owens, Andrew Jayaraman, Dinesh Lin, Justin Yuan, Wenzhen Malik, Jitendra Adelson, Edward H Levine, Sergey |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Calandra, Roberto Owens, Andrew Jayaraman, Dinesh Lin, Justin Yuan, Wenzhen Malik, Jitendra Adelson, Edward H Levine, Sergey |
| author_sort | Calandra, Roberto |
| collection | MIT |
| description | For humans, the process of grasping an object relies heavily on rich tactile feedback. Most recent robotic grasping work, however, has been based only on visual input, and thus cannot easily benefit from feedback after initiating contact. In this letter, we investigate how a robot can learn to use tactile information to iteratively and efficiently adjust its grasp. To this end, we propose an end-to-end action-conditional model that learns regrasping policies from raw visuo-tactile data. This model - a deep, multimodal convolutional network - predicts the outcome of a candidate grasp adjustment, and then executes a grasp by iteratively selecting the most promising actions. Our approach requires neither calibration of the tactile sensors nor any analytical modeling of contact forces, thus reducing the engineering effort required to obtain efficient grasping policies. We train our model with data from about 6450 grasping trials on a two-finger gripper equipped with GelSight high-resolution tactile sensors on each finger. Across extensive experiments, our approach outperforms a variety of baselines at 1) estimating grasp adjustment outcomes, 2) selecting efficient grasp adjustments for quick grasping, and 3) reducing the amount of force applied at the fingers, while maintaining competitive performance. Finally, we study the choices made by our model and show that it has successfully acquired useful and interpretable grasping behaviors. |
| first_indexed | 2024-09-23T14:03:27Z |
| format | Article |
| id | mit-1721.1/126806 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T14:03:27Z |
| publishDate | 2020 |
| publisher | Institute of Electrical and Electronics Engineers (IEEE) |
| record_format | dspace |
| spelling | mit-1721.1/1268062022-09-28T18:02:07Z More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch Calandra, Roberto Owens, Andrew Jayaraman, Dinesh Lin, Justin Yuan, Wenzhen Malik, Jitendra Adelson, Edward H Levine, Sergey Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science For humans, the process of grasping an object relies heavily on rich tactile feedback. Most recent robotic grasping work, however, has been based only on visual input, and thus cannot easily benefit from feedback after initiating contact. In this letter, we investigate how a robot can learn to use tactile information to iteratively and efficiently adjust its grasp. To this end, we propose an end-to-end action-conditional model that learns regrasping policies from raw visuo-tactile data. This model - a deep, multimodal convolutional network - predicts the outcome of a candidate grasp adjustment, and then executes a grasp by iteratively selecting the most promising actions. Our approach requires neither calibration of the tactile sensors nor any analytical modeling of contact forces, thus reducing the engineering effort required to obtain efficient grasping policies. We train our model with data from about 6450 grasping trials on a two-finger gripper equipped with GelSight high-resolution tactile sensors on each finger. Across extensive experiments, our approach outperforms a variety of baselines at 1) estimating grasp adjustment outcomes, 2) selecting efficient grasp adjustments for quick grasping, and 3) reducing the amount of force applied at the fingers, while maintaining competitive performance. Finally, we study the choices made by our model and show that it has successfully acquired useful and interpretable grasping behaviors. 2020-08-25T19:21:29Z 2020-08-25T19:21:29Z 2018-07 2019-09-27T17:11:34Z Article http://purl.org/eprint/type/JournalArticle 2377-3766 2377-3774 https://hdl.handle.net/1721.1/126806 Calandra, Roberto et al. "More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch." IEEE Robotics and Automation Letters 3, 4 (October 2018): 3300 - 3307 © 2016 IEEE en http://dx.doi.org/10.1109/lra.2018.2852779 IEEE Robotics and Automation Letters Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf Institute of Electrical and Electronics Engineers (IEEE) arXiv |
| spellingShingle | Calandra, Roberto Owens, Andrew Jayaraman, Dinesh Lin, Justin Yuan, Wenzhen Malik, Jitendra Adelson, Edward H Levine, Sergey More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch |
| title | More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch |
| title_full | More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch |
| title_fullStr | More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch |
| title_full_unstemmed | More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch |
| title_short | More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch |
| title_sort | more than a feeling learning to grasp and regrasp using vision and touch |
| url | https://hdl.handle.net/1721.1/126806 |
| work_keys_str_mv | AT calandraroberto morethanafeelinglearningtograspandregraspusingvisionandtouch AT owensandrew morethanafeelinglearningtograspandregraspusingvisionandtouch AT jayaramandinesh morethanafeelinglearningtograspandregraspusingvisionandtouch AT linjustin morethanafeelinglearningtograspandregraspusingvisionandtouch AT yuanwenzhen morethanafeelinglearningtograspandregraspusingvisionandtouch AT malikjitendra morethanafeelinglearningtograspandregraspusingvisionandtouch AT adelsonedwardh morethanafeelinglearningtograspandregraspusingvisionandtouch AT levinesergey morethanafeelinglearningtograspandregraspusingvisionandtouch |