On the Dirichlet Prior and Bayesian Regularization
A common objective in learning a model from data is to recover its network structure, while the model parameters are of minor interest. For example, we may wish to recover regulatory networks from high-throughput data sources. In this paper we examine how Bayesian regularization using a Dirichle...
| Main Authors: | , |
|---|---|
| Language: | en_US |
| Published: |
2004
|
| Subjects: | |
| Online Access: | http://hdl.handle.net/1721.1/6702 |
| _version_ | 1826211145722101760 |
|---|---|
| author | Steck, Harald Jaakkola, Tommi S. |
| author_facet | Steck, Harald Jaakkola, Tommi S. |
| author_sort | Steck, Harald |
| collection | MIT |
| description | A common objective in learning a model from data is to recover its network structure, while the model parameters are of minor interest. For example, we may wish to recover regulatory networks from high-throughput data sources. In this paper we examine how Bayesian regularization using a Dirichlet prior over the model parameters affects the learned model structure in a domain with discrete variables. Surprisingly, a weak prior in the sense of smaller equivalent sample size leads to a strong regularization of the model structure (sparse graph) given a sufficiently large data set. In particular, the empty graph is obtained in the limit of a vanishing strength of prior belief. This is diametrically opposite to what one may expect in this limit, namely the complete graph from an (unregularized) maximum likelihood estimate. Since the prior affects the parameters as expected, the prior strength balances a "trade-off" between regularizing the parameters or the structure of the model. We demonstrate the benefits of optimizing this trade-off in the sense of predictive accuracy. |
| first_indexed | 2024-09-23T15:01:18Z |
| id | mit-1721.1/6702 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T15:01:18Z |
| publishDate | 2004 |
| record_format | dspace |
| spelling | mit-1721.1/67022019-04-12T08:31:51Z On the Dirichlet Prior and Bayesian Regularization Steck, Harald Jaakkola, Tommi S. AI Regularization Dirichlet Prior A common objective in learning a model from data is to recover its network structure, while the model parameters are of minor interest. For example, we may wish to recover regulatory networks from high-throughput data sources. In this paper we examine how Bayesian regularization using a Dirichlet prior over the model parameters affects the learned model structure in a domain with discrete variables. Surprisingly, a weak prior in the sense of smaller equivalent sample size leads to a strong regularization of the model structure (sparse graph) given a sufficiently large data set. In particular, the empty graph is obtained in the limit of a vanishing strength of prior belief. This is diametrically opposite to what one may expect in this limit, namely the complete graph from an (unregularized) maximum likelihood estimate. Since the prior affects the parameters as expected, the prior strength balances a "trade-off" between regularizing the parameters or the structure of the model. We demonstrate the benefits of optimizing this trade-off in the sense of predictive accuracy. 2004-10-08T20:38:20Z 2004-10-08T20:38:20Z 2002-09-01 AIM-2002-014 http://hdl.handle.net/1721.1/6702 en_US AIM-2002-014 11 p. 3152389 bytes 1414851 bytes application/postscript application/pdf application/postscript application/pdf |
| spellingShingle | AI Regularization Dirichlet Prior Steck, Harald Jaakkola, Tommi S. On the Dirichlet Prior and Bayesian Regularization |
| title | On the Dirichlet Prior and Bayesian Regularization |
| title_full | On the Dirichlet Prior and Bayesian Regularization |
| title_fullStr | On the Dirichlet Prior and Bayesian Regularization |
| title_full_unstemmed | On the Dirichlet Prior and Bayesian Regularization |
| title_short | On the Dirichlet Prior and Bayesian Regularization |
| title_sort | on the dirichlet prior and bayesian regularization |
| topic | AI Regularization Dirichlet Prior |
| url | http://hdl.handle.net/1721.1/6702 |
| work_keys_str_mv | AT steckharald onthedirichletpriorandbayesianregularization AT jaakkolatommis onthedirichletpriorandbayesianregularization |