Learning to Infer Graphics Programs from Hand-Drawn Images
© 2018 Curran Associates Inc.All rights reserved. We introduce a model that learns to convert simple hand drawings into graphics programs written in a subset of LAT E X. The model combines techniques from deep learning and program synthesis. We learn a convolutional neural network that proposes pla...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
2021
|
| Online Access: | https://hdl.handle.net/1721.1/137831 |
| _version_ | 1826193571250700288 |
|---|---|
| author | Ellis, Kevin Ritchie, Daniel Solar-Lezama, Armando Tenenbaum, Joshua B. |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Ellis, Kevin Ritchie, Daniel Solar-Lezama, Armando Tenenbaum, Joshua B. |
| author_sort | Ellis, Kevin |
| collection | MIT |
| description | © 2018 Curran Associates Inc.All rights reserved. We introduce a model that learns to convert simple hand drawings into graphics programs written in a subset of LAT E X. The model combines techniques from deep learning and program synthesis. We learn a convolutional neural network that proposes plausible drawing primitives that explain an image. These drawing primitives are a specification (spec) of what the graphics program needs to draw. We learn a model that uses program synthesis techniques to recover a graphics program from that spec. These programs have constructs like variable bindings, iterative loops, or simple kinds of conditionals. With a graphics program in hand, we can correct errors made by the deep network and extrapolate drawings. |
| first_indexed | 2024-09-23T09:41:12Z |
| format | Article |
| id | mit-1721.1/137831 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T09:41:12Z |
| publishDate | 2021 |
| record_format | dspace |
| spelling | mit-1721.1/1378312021-11-09T03:34:35Z Learning to Infer Graphics Programs from Hand-Drawn Images Ellis, Kevin Ritchie, Daniel Solar-Lezama, Armando Tenenbaum, Joshua B. Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory © 2018 Curran Associates Inc.All rights reserved. We introduce a model that learns to convert simple hand drawings into graphics programs written in a subset of LAT E X. The model combines techniques from deep learning and program synthesis. We learn a convolutional neural network that proposes plausible drawing primitives that explain an image. These drawing primitives are a specification (spec) of what the graphics program needs to draw. We learn a model that uses program synthesis techniques to recover a graphics program from that spec. These programs have constructs like variable bindings, iterative loops, or simple kinds of conditionals. With a graphics program in hand, we can correct errors made by the deep network and extrapolate drawings. 2021-11-08T21:02:24Z 2021-11-08T21:02:24Z 2018 2019-07-10T13:27:12Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/137831 Ellis, Kevin, Ritchie, Daniel, Solar-Lezama, Armando and Tenenbaum, Joshua B. 2018. "Learning to Infer Graphics Programs from Hand-Drawn Images." en https://papers.nips.cc/paper/7845-learning-to-infer-graphics-programs-from-hand-drawn-images Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. application/pdf Neural Information Processing Systems (NIPS) |
| spellingShingle | Ellis, Kevin Ritchie, Daniel Solar-Lezama, Armando Tenenbaum, Joshua B. Learning to Infer Graphics Programs from Hand-Drawn Images |
| title | Learning to Infer Graphics Programs from Hand-Drawn Images |
| title_full | Learning to Infer Graphics Programs from Hand-Drawn Images |
| title_fullStr | Learning to Infer Graphics Programs from Hand-Drawn Images |
| title_full_unstemmed | Learning to Infer Graphics Programs from Hand-Drawn Images |
| title_short | Learning to Infer Graphics Programs from Hand-Drawn Images |
| title_sort | learning to infer graphics programs from hand drawn images |
| url | https://hdl.handle.net/1721.1/137831 |
| work_keys_str_mv | AT elliskevin learningtoinfergraphicsprogramsfromhanddrawnimages AT ritchiedaniel learningtoinfergraphicsprogramsfromhanddrawnimages AT solarlezamaarmando learningtoinfergraphicsprogramsfromhanddrawnimages AT tenenbaumjoshuab learningtoinfergraphicsprogramsfromhanddrawnimages |