HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment
We introduce HyperLex—a data set and evaluation resource that quantifies the extent of the semantic category membership, that is, type-of relation, also known as hyponymy–hypernymy or lexical entailment (LE) relation between 2,616 concept pairs. Cognitive psychology research has established that typ...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The MIT Press
2017-09-01
|
| Series: | Computational Linguistics |
| Online Access: | http://dx.doi.org/10.1162/coli_a_00301 |
| _version_ | 1827916661622046720 |
|---|---|
| author | Ivan Vulić Daniela Gerz Douwe Kiela Felix Hill Anna Korhonen |
| author_facet | Ivan Vulić Daniela Gerz Douwe Kiela Felix Hill Anna Korhonen |
| author_sort | Ivan Vulić |
| collection | DOAJ |
| description | We introduce HyperLex—a data set and evaluation resource that quantifies the extent of the semantic category membership, that is, type-of relation, also known as hyponymy–hypernymy or lexical entailment (LE) relation between 2,616 concept pairs. Cognitive psychology research has established that typicality and category/class membership are computed in human semantic memory as a gradual rather than binary relation. Nevertheless, most NLP research and existing large-scale inventories of concept category membership (WordNet, DBPedia, etc.) treat category membership and LE as binary. To address this, we asked hundreds of native English speakers to indicate typicality and strength of category membership between a diverse range of concept pairs on a crowdsourcing platform. Our results confirm that category membership and LE are indeed more gradual than binary. We then compare these human judgments with the predictions of automatic systems, which
reveals a huge gap between human performance and state-of-the-art LE, distributional and representation learning models, and substantial differences between the models themselves. We discuss a pathway for improving semantic models to overcome this discrepancy, and indicate future application areas for improved graded LE systems. |
| first_indexed | 2024-03-13T03:17:21Z |
| format | Article |
| id | doaj.art-b36087faaf3046d581a6e3c24f71093d |
| institution | Directory Open Access Journal |
| issn | 1530-9312 |
| language | English |
| last_indexed | 2024-03-13T03:17:21Z |
| publishDate | 2017-09-01 |
| publisher | The MIT Press |
| record_format | Article |
| series | Computational Linguistics |
| spelling | doaj.art-b36087faaf3046d581a6e3c24f71093d2023-06-25T14:50:05ZengThe MIT PressComputational Linguistics1530-93122017-09-0143410.1162/coli_a_00301HyperLex: A Large-Scale Evaluation of Graded Lexical EntailmentIvan VulićDaniela GerzDouwe KielaFelix HillAnna KorhonenWe introduce HyperLex—a data set and evaluation resource that quantifies the extent of the semantic category membership, that is, type-of relation, also known as hyponymy–hypernymy or lexical entailment (LE) relation between 2,616 concept pairs. Cognitive psychology research has established that typicality and category/class membership are computed in human semantic memory as a gradual rather than binary relation. Nevertheless, most NLP research and existing large-scale inventories of concept category membership (WordNet, DBPedia, etc.) treat category membership and LE as binary. To address this, we asked hundreds of native English speakers to indicate typicality and strength of category membership between a diverse range of concept pairs on a crowdsourcing platform. Our results confirm that category membership and LE are indeed more gradual than binary. We then compare these human judgments with the predictions of automatic systems, which reveals a huge gap between human performance and state-of-the-art LE, distributional and representation learning models, and substantial differences between the models themselves. We discuss a pathway for improving semantic models to overcome this discrepancy, and indicate future application areas for improved graded LE systems.http://dx.doi.org/10.1162/coli_a_00301 |
| spellingShingle | Ivan Vulić Daniela Gerz Douwe Kiela Felix Hill Anna Korhonen HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment Computational Linguistics |
| title | HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment |
| title_full | HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment |
| title_fullStr | HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment |
| title_full_unstemmed | HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment |
| title_short | HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment |
| title_sort | hyperlex a large scale evaluation of graded lexical entailment |
| url | http://dx.doi.org/10.1162/coli_a_00301 |
| work_keys_str_mv | AT ivanvulic hyperlexalargescaleevaluationofgradedlexicalentailment AT danielagerz hyperlexalargescaleevaluationofgradedlexicalentailment AT douwekiela hyperlexalargescaleevaluationofgradedlexicalentailment AT felixhill hyperlexalargescaleevaluationofgradedlexicalentailment AT annakorhonen hyperlexalargescaleevaluationofgradedlexicalentailment |