Using mixup as a regularizer can surprisingly improve accuracy and out-of-distribution robustness
We show that the effectiveness of the well celebrated Mixup can be further improved if instead of using it as the sole learning objective, it is utilized as an additional regularizer to the standard cross-entropy loss. This simple change not only improves accuracy but also significantly improves the...
| Main Authors: | , , , , |
|---|---|
| Format: | Conference item |
| Language: | English |
| Published: |
Curran Associates, Inc
2023
|
| _version_ | 1826310474128424960 |
|---|---|
| author | Pinto, F Yang, H Lim, SN Torr, PHS Dokania, PK |
| author_facet | Pinto, F Yang, H Lim, SN Torr, PHS Dokania, PK |
| author_sort | Pinto, F |
| collection | OXFORD |
| description | We show that the effectiveness of the well celebrated Mixup can be further improved if instead of using it as the sole learning objective, it is utilized as an additional regularizer to the standard cross-entropy loss. This simple change not only improves accuracy but also significantly improves the quality of the predictive uncertainty estimation of Mixup in most cases under various forms of covariate shifts and out-of-distribution detection experiments. In fact, we observe that Mixup otherwise yields much degraded performance on detecting out-of-distribution samples possibly, as we show empirically, due to its tendency to learn models exhibiting high-entropy throughout; making it difficult to differentiate in-distribution samples from out-of-distribution ones. To show the efficacy of our approach (RegMixup), we provide thorough analyses and experiments on vision datasets (ImageNet & CIFAR-10/100) and compare it with a suite of recent approaches for reliable uncertainty estimation. |
| first_indexed | 2024-03-07T07:52:28Z |
| format | Conference item |
| id | oxford-uuid:19149256-9b9b-4f01-8262-e315f25b584e |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-07T07:52:28Z |
| publishDate | 2023 |
| publisher | Curran Associates, Inc |
| record_format | dspace |
| spelling | oxford-uuid:19149256-9b9b-4f01-8262-e315f25b584e2023-07-27T15:14:32ZUsing mixup as a regularizer can surprisingly improve accuracy and out-of-distribution robustnessConference itemhttp://purl.org/coar/resource_type/c_5794uuid:19149256-9b9b-4f01-8262-e315f25b584eEnglishSymplectic ElementsCurran Associates, Inc2023Pinto, FYang, HLim, SNTorr, PHSDokania, PKWe show that the effectiveness of the well celebrated Mixup can be further improved if instead of using it as the sole learning objective, it is utilized as an additional regularizer to the standard cross-entropy loss. This simple change not only improves accuracy but also significantly improves the quality of the predictive uncertainty estimation of Mixup in most cases under various forms of covariate shifts and out-of-distribution detection experiments. In fact, we observe that Mixup otherwise yields much degraded performance on detecting out-of-distribution samples possibly, as we show empirically, due to its tendency to learn models exhibiting high-entropy throughout; making it difficult to differentiate in-distribution samples from out-of-distribution ones. To show the efficacy of our approach (RegMixup), we provide thorough analyses and experiments on vision datasets (ImageNet & CIFAR-10/100) and compare it with a suite of recent approaches for reliable uncertainty estimation. |
| spellingShingle | Pinto, F Yang, H Lim, SN Torr, PHS Dokania, PK Using mixup as a regularizer can surprisingly improve accuracy and out-of-distribution robustness |
| title | Using mixup as a regularizer can surprisingly improve accuracy and out-of-distribution robustness |
| title_full | Using mixup as a regularizer can surprisingly improve accuracy and out-of-distribution robustness |
| title_fullStr | Using mixup as a regularizer can surprisingly improve accuracy and out-of-distribution robustness |
| title_full_unstemmed | Using mixup as a regularizer can surprisingly improve accuracy and out-of-distribution robustness |
| title_short | Using mixup as a regularizer can surprisingly improve accuracy and out-of-distribution robustness |
| title_sort | using mixup as a regularizer can surprisingly improve accuracy and out of distribution robustness |
| work_keys_str_mv | AT pintof usingmixupasaregularizercansurprisinglyimproveaccuracyandoutofdistributionrobustness AT yangh usingmixupasaregularizercansurprisinglyimproveaccuracyandoutofdistributionrobustness AT limsn usingmixupasaregularizercansurprisinglyimproveaccuracyandoutofdistributionrobustness AT torrphs usingmixupasaregularizercansurprisinglyimproveaccuracyandoutofdistributionrobustness AT dokaniapk usingmixupasaregularizercansurprisinglyimproveaccuracyandoutofdistributionrobustness |