Neural response interpretation through the lens of critical pathways
Is critical input information encoded in specific sparse pathways within the neural network? In this work, we discuss the problem of identifying these critical pathways and subsequently leverage them for interpreting the network's response to an input. The pruning objective - selecting the smal...
| Main Authors: | , , , , , |
|---|---|
| Format: | Conference item |
| Language: | English |
| Published: |
IEEE
2021
|
| _version_ | 1797107430066225152 |
|---|---|
| author | Khakzar, A Baselizadeh, S Khanduja, S Rupprecht, C Kim, ST Navab, N |
| author_facet | Khakzar, A Baselizadeh, S Khanduja, S Rupprecht, C Kim, ST Navab, N |
| author_sort | Khakzar, A |
| collection | OXFORD |
| description | Is critical input information encoded in specific sparse pathways within the neural network? In this work, we discuss the problem of identifying these critical pathways and subsequently leverage them for interpreting the network's response to an input. The pruning objective - selecting the smallest group of neurons for which the response remains equivalent to the original network - has been previously proposed for identifying critical pathways. We demonstrate that sparse pathways derived from pruning do not necessarily encode critical input information. To ensure sparse pathways include critical fragments of the encoded input information, we propose pathway selection via neurons' contribution to the response. We proceed to explain how critical pathways can reveal critical input features. We prove that pathways selected via neuron contribution are locally linear (in an `2-ball), a property that we use for proposing a feature attribution method: “pathway gradient”. We validate our interpretation method using mainstream evaluation experiments. The validation of pathway gradient interpretation method further confirms that selected pathways using neuron contributions correspond to critical input features. The code1 2 is publicly available. |
| first_indexed | 2024-03-07T07:14:27Z |
| format | Conference item |
| id | oxford-uuid:da576938-f95e-45ed-a3d4-0e76f7766322 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-07T07:14:27Z |
| publishDate | 2021 |
| publisher | IEEE |
| record_format | dspace |
| spelling | oxford-uuid:da576938-f95e-45ed-a3d4-0e76f77663222022-07-26T15:18:58ZNeural response interpretation through the lens of critical pathwaysConference itemhttp://purl.org/coar/resource_type/c_5794uuid:da576938-f95e-45ed-a3d4-0e76f7766322EnglishSymplectic ElementsIEEE2021Khakzar, ABaselizadeh, SKhanduja, SRupprecht, CKim, STNavab, NIs critical input information encoded in specific sparse pathways within the neural network? In this work, we discuss the problem of identifying these critical pathways and subsequently leverage them for interpreting the network's response to an input. The pruning objective - selecting the smallest group of neurons for which the response remains equivalent to the original network - has been previously proposed for identifying critical pathways. We demonstrate that sparse pathways derived from pruning do not necessarily encode critical input information. To ensure sparse pathways include critical fragments of the encoded input information, we propose pathway selection via neurons' contribution to the response. We proceed to explain how critical pathways can reveal critical input features. We prove that pathways selected via neuron contribution are locally linear (in an `2-ball), a property that we use for proposing a feature attribution method: “pathway gradient”. We validate our interpretation method using mainstream evaluation experiments. The validation of pathway gradient interpretation method further confirms that selected pathways using neuron contributions correspond to critical input features. The code1 2 is publicly available. |
| spellingShingle | Khakzar, A Baselizadeh, S Khanduja, S Rupprecht, C Kim, ST Navab, N Neural response interpretation through the lens of critical pathways |
| title | Neural response interpretation through the lens of critical pathways |
| title_full | Neural response interpretation through the lens of critical pathways |
| title_fullStr | Neural response interpretation through the lens of critical pathways |
| title_full_unstemmed | Neural response interpretation through the lens of critical pathways |
| title_short | Neural response interpretation through the lens of critical pathways |
| title_sort | neural response interpretation through the lens of critical pathways |
| work_keys_str_mv | AT khakzara neuralresponseinterpretationthroughthelensofcriticalpathways AT baselizadehs neuralresponseinterpretationthroughthelensofcriticalpathways AT khandujas neuralresponseinterpretationthroughthelensofcriticalpathways AT rupprechtc neuralresponseinterpretationthroughthelensofcriticalpathways AT kimst neuralresponseinterpretationthroughthelensofcriticalpathways AT navabn neuralresponseinterpretationthroughthelensofcriticalpathways |