Neural scaling of deep chemical models
Massive scale, in terms of both data availability and computation, enables important breakthroughs in key application areas of deep learning such as natural language processing and computer vision. There is emerging evidence that scale may be a key ingredient in scientific deep learning, but the imp...
| Main Authors: | , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Springer Science and Business Media LLC
2025
|
| Online Access: | https://hdl.handle.net/1721.1/158195 |
| _version_ | 1824458152219770880 |
|---|---|
| author | Frey, Nathan C Soklaski, Ryan Axelrod, Simon Samsi, Siddharth Gómez-Bombarelli, Rafael Coley, Connor W Gadepally, Vijay |
| author2 | Lincoln Laboratory |
| author_facet | Lincoln Laboratory Frey, Nathan C Soklaski, Ryan Axelrod, Simon Samsi, Siddharth Gómez-Bombarelli, Rafael Coley, Connor W Gadepally, Vijay |
| author_sort | Frey, Nathan C |
| collection | MIT |
| description | Massive scale, in terms of both data availability and computation, enables important breakthroughs in key application areas of deep learning such as natural language processing and computer vision. There is emerging evidence that scale may be a key ingredient in scientific deep learning, but the importance of physical priors in scientific domains makes the strategies and benefits of scaling uncertain. Here we investigate neural-scaling behaviour in large chemical models by varying model and dataset sizes over many orders of magnitude, studying models with over one billion parameters, pre-trained on datasets of up to ten million datapoints. We consider large language models for generative chemistry and graph neural networks for machine-learned interatomic potentials. We investigate the interplay between physical priors and scale and discover empirical neural-scaling relations for language models in chemistry with a scaling exponent of 0.17 for the largest dataset size considered, and a scaling exponent of 0.26 for equivariant graph neural network interatomic potentials. |
| first_indexed | 2025-02-19T04:21:21Z |
| format | Article |
| id | mit-1721.1/158195 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2025-02-19T04:21:21Z |
| publishDate | 2025 |
| publisher | Springer Science and Business Media LLC |
| record_format | dspace |
| spelling | mit-1721.1/1581952025-02-11T21:05:34Z Neural scaling of deep chemical models Frey, Nathan C Soklaski, Ryan Axelrod, Simon Samsi, Siddharth Gómez-Bombarelli, Rafael Coley, Connor W Gadepally, Vijay Lincoln Laboratory Massachusetts Institute of Technology. Department of Materials Science and Engineering Massachusetts Institute of Technology. Department of Chemical Engineering Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Massive scale, in terms of both data availability and computation, enables important breakthroughs in key application areas of deep learning such as natural language processing and computer vision. There is emerging evidence that scale may be a key ingredient in scientific deep learning, but the importance of physical priors in scientific domains makes the strategies and benefits of scaling uncertain. Here we investigate neural-scaling behaviour in large chemical models by varying model and dataset sizes over many orders of magnitude, studying models with over one billion parameters, pre-trained on datasets of up to ten million datapoints. We consider large language models for generative chemistry and graph neural networks for machine-learned interatomic potentials. We investigate the interplay between physical priors and scale and discover empirical neural-scaling relations for language models in chemistry with a scaling exponent of 0.17 for the largest dataset size considered, and a scaling exponent of 0.26 for equivariant graph neural network interatomic potentials. 2025-02-11T21:05:33Z 2025-02-11T21:05:33Z 2023 2025-02-11T20:58:27Z Article http://purl.org/eprint/type/JournalArticle https://hdl.handle.net/1721.1/158195 Frey, N.C., Soklaski, R., Axelrod, S. et al. Neural scaling of deep chemical models. Nat Mach Intell 5, 1297–1305 (2023). en 10.1038/s42256-023-00740-3 Nature Machine Intelligence Creative Commons Attribution https://creativecommons.org/licenses/by/4.0/ application/pdf Springer Science and Business Media LLC Springer Science and Business Media LLC |
| spellingShingle | Frey, Nathan C Soklaski, Ryan Axelrod, Simon Samsi, Siddharth Gómez-Bombarelli, Rafael Coley, Connor W Gadepally, Vijay Neural scaling of deep chemical models |
| title | Neural scaling of deep chemical models |
| title_full | Neural scaling of deep chemical models |
| title_fullStr | Neural scaling of deep chemical models |
| title_full_unstemmed | Neural scaling of deep chemical models |
| title_short | Neural scaling of deep chemical models |
| title_sort | neural scaling of deep chemical models |
| url | https://hdl.handle.net/1721.1/158195 |
| work_keys_str_mv | AT freynathanc neuralscalingofdeepchemicalmodels AT soklaskiryan neuralscalingofdeepchemicalmodels AT axelrodsimon neuralscalingofdeepchemicalmodels AT samsisiddharth neuralscalingofdeepchemicalmodels AT gomezbombarellirafael neuralscalingofdeepchemicalmodels AT coleyconnorw neuralscalingofdeepchemicalmodels AT gadepallyvijay neuralscalingofdeepchemicalmodels |