Minimising the Kullback–Leibler Divergence for Model Selection in Distributed Nonlinear Systems
The Kullback–Leibler (KL) divergence is a fundamental measure of information geometry that is used in a variety of contexts in artificial intelligence. We show that, when system dynamics are given by distributed nonlinear systems, this measure can be decomposed as a function of two information-theor...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2018-01-01
|
| Series: | Entropy |
| Subjects: | |
| Online Access: | http://www.mdpi.com/1099-4300/20/2/51 |
| _version_ | 1811308405697019904 |
|---|---|
| author | Oliver M. Cliff Mikhail Prokopenko Robert Fitch |
| author_facet | Oliver M. Cliff Mikhail Prokopenko Robert Fitch |
| author_sort | Oliver M. Cliff |
| collection | DOAJ |
| description | The Kullback–Leibler (KL) divergence is a fundamental measure of information geometry that is used in a variety of contexts in artificial intelligence. We show that, when system dynamics are given by distributed nonlinear systems, this measure can be decomposed as a function of two information-theoretic measures, transfer entropy and stochastic interaction. More specifically, these measures are applicable when selecting a candidate model for a distributed system, where individual subsystems are coupled via latent variables and observed through a filter. We represent this model as a directed acyclic graph (DAG) that characterises the unidirectional coupling between subsystems. Standard approaches to structure learning are not applicable in this framework due to the hidden variables; however, we can exploit the properties of certain dynamical systems to formulate exact methods based on differential topology. We approach the problem by using reconstruction theorems to derive an analytical expression for the KL divergence of a candidate DAG from the observed dataset. Using this result, we present a scoring function based on transfer entropy to be used as a subroutine in a structure learning algorithm. We then demonstrate its use in recovering the structure of coupled Lorenz and Rössler systems. |
| first_indexed | 2024-04-13T09:22:35Z |
| format | Article |
| id | doaj.art-cbc9ff52490c4c7e851ebc849d101a5a |
| institution | Directory Open Access Journal |
| issn | 1099-4300 |
| language | English |
| last_indexed | 2024-04-13T09:22:35Z |
| publishDate | 2018-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Entropy |
| spelling | doaj.art-cbc9ff52490c4c7e851ebc849d101a5a2022-12-22T02:52:33ZengMDPI AGEntropy1099-43002018-01-012025110.3390/e20020051e20020051Minimising the Kullback–Leibler Divergence for Model Selection in Distributed Nonlinear SystemsOliver M. Cliff0Mikhail Prokopenko1Robert Fitch2Australian Centre for Field Robotics, The University of Sydney, Sydney NSW 2006, AustraliaComplex Systems Research Group, The University of Sydney, Sydney NSW 2006, AustraliaAustralian Centre for Field Robotics, The University of Sydney, Sydney NSW 2006, AustraliaThe Kullback–Leibler (KL) divergence is a fundamental measure of information geometry that is used in a variety of contexts in artificial intelligence. We show that, when system dynamics are given by distributed nonlinear systems, this measure can be decomposed as a function of two information-theoretic measures, transfer entropy and stochastic interaction. More specifically, these measures are applicable when selecting a candidate model for a distributed system, where individual subsystems are coupled via latent variables and observed through a filter. We represent this model as a directed acyclic graph (DAG) that characterises the unidirectional coupling between subsystems. Standard approaches to structure learning are not applicable in this framework due to the hidden variables; however, we can exploit the properties of certain dynamical systems to formulate exact methods based on differential topology. We approach the problem by using reconstruction theorems to derive an analytical expression for the KL divergence of a candidate DAG from the observed dataset. Using this result, we present a scoring function based on transfer entropy to be used as a subroutine in a structure learning algorithm. We then demonstrate its use in recovering the structure of coupled Lorenz and Rössler systems.http://www.mdpi.com/1099-4300/20/2/51Kullback–Leibler divergencemodel selectioninformation theorytransfer entropystochastic interactionnonlinear systemscomplex networksstate space reconstruction |
| spellingShingle | Oliver M. Cliff Mikhail Prokopenko Robert Fitch Minimising the Kullback–Leibler Divergence for Model Selection in Distributed Nonlinear Systems Entropy Kullback–Leibler divergence model selection information theory transfer entropy stochastic interaction nonlinear systems complex networks state space reconstruction |
| title | Minimising the Kullback–Leibler Divergence for Model Selection in Distributed Nonlinear Systems |
| title_full | Minimising the Kullback–Leibler Divergence for Model Selection in Distributed Nonlinear Systems |
| title_fullStr | Minimising the Kullback–Leibler Divergence for Model Selection in Distributed Nonlinear Systems |
| title_full_unstemmed | Minimising the Kullback–Leibler Divergence for Model Selection in Distributed Nonlinear Systems |
| title_short | Minimising the Kullback–Leibler Divergence for Model Selection in Distributed Nonlinear Systems |
| title_sort | minimising the kullback leibler divergence for model selection in distributed nonlinear systems |
| topic | Kullback–Leibler divergence model selection information theory transfer entropy stochastic interaction nonlinear systems complex networks state space reconstruction |
| url | http://www.mdpi.com/1099-4300/20/2/51 |
| work_keys_str_mv | AT olivermcliff minimisingthekullbackleiblerdivergenceformodelselectionindistributednonlinearsystems AT mikhailprokopenko minimisingthekullbackleiblerdivergenceformodelselectionindistributednonlinearsystems AT robertfitch minimisingthekullbackleiblerdivergenceformodelselectionindistributednonlinearsystems |