Getting to the point: index sets and parallelism-preserving autodiff for pointful array programming
<jats:p>We present a novel programming language design that attempts to combine the clarity and safety of high-level functional languages with the efficiency and parallelism of low-level numerical languages. We treat arrays as eagerly-memoized functions on typed index sets, allowing abstract f...
| Main Authors: | , , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Association for Computing Machinery (ACM)
2022
|
| Online Access: | https://hdl.handle.net/1721.1/143845 |
| _version_ | 1811090369046118400 |
|---|---|
| author | Paszke, Adam Johnson, Daniel D Duvenaud, David Vytiniotis, Dimitrios Radul, Alexey Johnson, Matthew J Ragan-Kelley, Jonathan Maclaurin, Dougal |
| author2 | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science |
| author_facet | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Paszke, Adam Johnson, Daniel D Duvenaud, David Vytiniotis, Dimitrios Radul, Alexey Johnson, Matthew J Ragan-Kelley, Jonathan Maclaurin, Dougal |
| author_sort | Paszke, Adam |
| collection | MIT |
| description | <jats:p>We present a novel programming language design that attempts to combine the clarity and safety of high-level functional languages with the efficiency and parallelism of low-level numerical languages. We treat arrays as eagerly-memoized functions on typed index sets, allowing abstract function manipulations, such as currying, to work on arrays. In contrast to composing primitive bulk-array operations, we argue for an explicit nested indexing style that mirrors application of functions to arguments. We also introduce a fine-grained typed effects system which affords concise and automatically-parallelized in-place updates. Specifically, an associative accumulation effect allows reverse-mode automatic differentiation of in-place updates in a way that preserves parallelism. Empirically, we benchmark against the Futhark array programming language, and demonstrate that aggressive inlining and type-driven compilation allows array programs to be written in an expressive, "pointful" style with little performance penalty.</jats:p> |
| first_indexed | 2024-09-23T14:44:30Z |
| format | Article |
| id | mit-1721.1/143845 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T14:44:30Z |
| publishDate | 2022 |
| publisher | Association for Computing Machinery (ACM) |
| record_format | dspace |
| spelling | mit-1721.1/1438452023-07-28T17:47:27Z Getting to the point: index sets and parallelism-preserving autodiff for pointful array programming Paszke, Adam Johnson, Daniel D Duvenaud, David Vytiniotis, Dimitrios Radul, Alexey Johnson, Matthew J Ragan-Kelley, Jonathan Maclaurin, Dougal Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory <jats:p>We present a novel programming language design that attempts to combine the clarity and safety of high-level functional languages with the efficiency and parallelism of low-level numerical languages. We treat arrays as eagerly-memoized functions on typed index sets, allowing abstract function manipulations, such as currying, to work on arrays. In contrast to composing primitive bulk-array operations, we argue for an explicit nested indexing style that mirrors application of functions to arguments. We also introduce a fine-grained typed effects system which affords concise and automatically-parallelized in-place updates. Specifically, an associative accumulation effect allows reverse-mode automatic differentiation of in-place updates in a way that preserves parallelism. Empirically, we benchmark against the Futhark array programming language, and demonstrate that aggressive inlining and type-driven compilation allows array programs to be written in an expressive, "pointful" style with little performance penalty.</jats:p> 2022-07-19T12:39:43Z 2022-07-19T12:39:43Z 2021 2022-07-19T12:36:03Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/143845 Paszke, Adam, Johnson, Daniel D, Duvenaud, David, Vytiniotis, Dimitrios, Radul, Alexey et al. 2021. "Getting to the point: index sets and parallelism-preserving autodiff for pointful array programming." Proceedings of the ACM on Programming Languages, 5 (ICFP). en 10.1145/3473593 Proceedings of the ACM on Programming Languages Creative Commons Attribution 4.0 International license https://creativecommons.org/licenses/by/4.0/ application/pdf Association for Computing Machinery (ACM) ACM |
| spellingShingle | Paszke, Adam Johnson, Daniel D Duvenaud, David Vytiniotis, Dimitrios Radul, Alexey Johnson, Matthew J Ragan-Kelley, Jonathan Maclaurin, Dougal Getting to the point: index sets and parallelism-preserving autodiff for pointful array programming |
| title | Getting to the point: index sets and parallelism-preserving autodiff for pointful array programming |
| title_full | Getting to the point: index sets and parallelism-preserving autodiff for pointful array programming |
| title_fullStr | Getting to the point: index sets and parallelism-preserving autodiff for pointful array programming |
| title_full_unstemmed | Getting to the point: index sets and parallelism-preserving autodiff for pointful array programming |
| title_short | Getting to the point: index sets and parallelism-preserving autodiff for pointful array programming |
| title_sort | getting to the point index sets and parallelism preserving autodiff for pointful array programming |
| url | https://hdl.handle.net/1721.1/143845 |
| work_keys_str_mv | AT paszkeadam gettingtothepointindexsetsandparallelismpreservingautodiffforpointfularrayprogramming AT johnsondanield gettingtothepointindexsetsandparallelismpreservingautodiffforpointfularrayprogramming AT duvenauddavid gettingtothepointindexsetsandparallelismpreservingautodiffforpointfularrayprogramming AT vytiniotisdimitrios gettingtothepointindexsetsandparallelismpreservingautodiffforpointfularrayprogramming AT radulalexey gettingtothepointindexsetsandparallelismpreservingautodiffforpointfularrayprogramming AT johnsonmatthewj gettingtothepointindexsetsandparallelismpreservingautodiffforpointfularrayprogramming AT ragankelleyjonathan gettingtothepointindexsetsandparallelismpreservingautodiffforpointfularrayprogramming AT maclaurindougal gettingtothepointindexsetsandparallelismpreservingautodiffforpointfularrayprogramming |