On Autoencoders and Score Matching for Energy Based Models
We consider estimation methods for the class of continuous-data energy based models (EBMs). Our main result shows that estimating the parameters of an EBM using score matching when the conditional distribution over the visible units is Gaussian corresponds to training a particular form of regularize...
| Main Authors: | , , , , |
|---|---|
| Format: | Conference item |
| Published: |
ACM
2011
|
| _version_ | 1826301897577857024 |
|---|---|
| author | Swersky, K Ranzato, M Buchman, D Marlin, B Freitas, N |
| author_facet | Swersky, K Ranzato, M Buchman, D Marlin, B Freitas, N |
| author_sort | Swersky, K |
| collection | OXFORD |
| description | We consider estimation methods for the class of continuous-data energy based models (EBMs). Our main result shows that estimating the parameters of an EBM using score matching when the conditional distribution over the visible units is Gaussian corresponds to training a particular form of regularized autoencoder. We show how different Gaussian EBMs lead to different autoencoder architectures, providing deep links between these two families of models. We compare the score matching estimator for the mPoT model, a particular Gaussian EBM, to several other training methods on a variety of tasks including image denoising and unsupervised feature extraction. We show that the regularization function induced by score matching leads to superior classification performance relative to a standard autoencoder. We also show that score matching yields classification results that are indistinguishable from better-known stochastic approximation maximum likelihood estimators. |
| first_indexed | 2024-03-07T05:39:21Z |
| format | Conference item |
| id | oxford-uuid:e5048031-d75a-4db9-bbf9-b7784875292f |
| institution | University of Oxford |
| last_indexed | 2024-03-07T05:39:21Z |
| publishDate | 2011 |
| publisher | ACM |
| record_format | dspace |
| spelling | oxford-uuid:e5048031-d75a-4db9-bbf9-b7784875292f2022-03-27T10:20:57ZOn Autoencoders and Score Matching for Energy Based ModelsConference itemhttp://purl.org/coar/resource_type/c_5794uuid:e5048031-d75a-4db9-bbf9-b7784875292fDepartment of Computer ScienceACM2011Swersky, KRanzato, MBuchman, DMarlin, BFreitas, NWe consider estimation methods for the class of continuous-data energy based models (EBMs). Our main result shows that estimating the parameters of an EBM using score matching when the conditional distribution over the visible units is Gaussian corresponds to training a particular form of regularized autoencoder. We show how different Gaussian EBMs lead to different autoencoder architectures, providing deep links between these two families of models. We compare the score matching estimator for the mPoT model, a particular Gaussian EBM, to several other training methods on a variety of tasks including image denoising and unsupervised feature extraction. We show that the regularization function induced by score matching leads to superior classification performance relative to a standard autoencoder. We also show that score matching yields classification results that are indistinguishable from better-known stochastic approximation maximum likelihood estimators. |
| spellingShingle | Swersky, K Ranzato, M Buchman, D Marlin, B Freitas, N On Autoencoders and Score Matching for Energy Based Models |
| title | On Autoencoders and Score Matching for Energy Based Models |
| title_full | On Autoencoders and Score Matching for Energy Based Models |
| title_fullStr | On Autoencoders and Score Matching for Energy Based Models |
| title_full_unstemmed | On Autoencoders and Score Matching for Energy Based Models |
| title_short | On Autoencoders and Score Matching for Energy Based Models |
| title_sort | on autoencoders and score matching for energy based models |
| work_keys_str_mv | AT swerskyk onautoencodersandscorematchingforenergybasedmodels AT ranzatom onautoencodersandscorematchingforenergybasedmodels AT buchmand onautoencodersandscorematchingforenergybasedmodels AT marlinb onautoencodersandscorematchingforenergybasedmodels AT freitasn onautoencodersandscorematchingforenergybasedmodels |