Gradient matching for domain generalization
Machine learning systems typically assume that the distributions of training and test sets match closely. However, a critical requirement of such systems in the real world is their ability to generalize to unseen domains. Here, we propose an inter-domain gradient matching objective that targets doma...
| Váldodahkkit: | , , , , , , |
|---|---|
| Materiálatiipa: | Conference item |
| Giella: | English |
| Almmustuhtton: |
OpenReview
2022
|
| _version_ | 1826312991657689088 |
|---|---|
| author | Shi, Y Seely, J Torr, PHS Siddharth, N Hannun, A Usunier, N Synnaeve, G |
| author_facet | Shi, Y Seely, J Torr, PHS Siddharth, N Hannun, A Usunier, N Synnaeve, G |
| author_sort | Shi, Y |
| collection | OXFORD |
| description | Machine learning systems typically assume that the distributions of training and test sets match closely. However, a critical requirement of such systems in the real world is their ability to generalize to unseen domains. Here, we propose an inter-domain gradient matching objective that targets domain generalization by maximizing the inner product between gradients from different domains. Since direct optimization of the gradient inner product can be computationally prohibitive - it requires computation of second-order derivatives - we derive a simpler first-order algorithm named Fish that approximates its optimization. We perform experiments on the WILDS benchmark, which captures distribution shift in the real world, as well as the DOMAINBED benchmark that focuses more on synthetic-to-real transfer. Our method produces competitive results on both benchmarks, demonstrating its effectiveness across a wide range of domain generalization tasks. Code is available at https://github.com/YugeTen/fish. |
| first_indexed | 2024-09-25T04:03:54Z |
| format | Conference item |
| id | oxford-uuid:ab143b26-fa2d-4b29-bf80-928e39dfdcb4 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-09-25T04:03:54Z |
| publishDate | 2022 |
| publisher | OpenReview |
| record_format | dspace |
| spelling | oxford-uuid:ab143b26-fa2d-4b29-bf80-928e39dfdcb42024-05-16T15:57:37ZGradient matching for domain generalizationConference itemhttp://purl.org/coar/resource_type/c_6670uuid:ab143b26-fa2d-4b29-bf80-928e39dfdcb4EnglishSymplectic ElementsOpenReview2022Shi, YSeely, JTorr, PHSSiddharth, NHannun, AUsunier, NSynnaeve, GMachine learning systems typically assume that the distributions of training and test sets match closely. However, a critical requirement of such systems in the real world is their ability to generalize to unseen domains. Here, we propose an inter-domain gradient matching objective that targets domain generalization by maximizing the inner product between gradients from different domains. Since direct optimization of the gradient inner product can be computationally prohibitive - it requires computation of second-order derivatives - we derive a simpler first-order algorithm named Fish that approximates its optimization. We perform experiments on the WILDS benchmark, which captures distribution shift in the real world, as well as the DOMAINBED benchmark that focuses more on synthetic-to-real transfer. Our method produces competitive results on both benchmarks, demonstrating its effectiveness across a wide range of domain generalization tasks. Code is available at https://github.com/YugeTen/fish. |
| spellingShingle | Shi, Y Seely, J Torr, PHS Siddharth, N Hannun, A Usunier, N Synnaeve, G Gradient matching for domain generalization |
| title | Gradient matching for domain generalization |
| title_full | Gradient matching for domain generalization |
| title_fullStr | Gradient matching for domain generalization |
| title_full_unstemmed | Gradient matching for domain generalization |
| title_short | Gradient matching for domain generalization |
| title_sort | gradient matching for domain generalization |
| work_keys_str_mv | AT shiy gradientmatchingfordomaingeneralization AT seelyj gradientmatchingfordomaingeneralization AT torrphs gradientmatchingfordomaingeneralization AT siddharthn gradientmatchingfordomaingeneralization AT hannuna gradientmatchingfordomaingeneralization AT usuniern gradientmatchingfordomaingeneralization AT synnaeveg gradientmatchingfordomaingeneralization |