Extending the evaluation of Genia Event task toward knowledge base construction and comparison to Gene Regulation Ontology task
Background: The third edition of the BioNLP Shared Task was held with the grand theme "knowledge base construction (KB)". The Genia Event (GE) task was re-designed and implemented in light of this theme. For its final report, the participating systems were evaluated from a perspective of a...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2016
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/81489 http://hdl.handle.net/10220/40825 |
| _version_ | 1811683690733895680 |
|---|---|
| author | Kim, Jin-Dong Kim, Jung-jae Han, Xu Rebholz-Schuhmann, Dietrich |
| author2 | School of Computer Science and Engineering |
| author_facet | School of Computer Science and Engineering Kim, Jin-Dong Kim, Jung-jae Han, Xu Rebholz-Schuhmann, Dietrich |
| author_sort | Kim, Jin-Dong |
| collection | NTU |
| description | Background: The third edition of the BioNLP Shared Task was held with the grand theme "knowledge base construction (KB)". The Genia Event (GE) task was re-designed and implemented in light of this theme. For its final report, the participating systems were evaluated from a perspective of annotation. To further explore the grand theme, we extended the evaluation from a perspective of KB construction. Also, the Gene Regulation Ontology (GRO) task was newly introduced in the third edition. The final evaluation of the participating systems resulted in relatively low performance. The reason was attributed to the large size and complex semantic representation of the ontology. To investigate potential benefits of resource exchange between the presumably similar tasks, we measured the overlap between the datasets of the two tasks, and tested whether the dataset for one task can be used to enhance performance on the other. Results: We report an extended evaluation on all the participating systems in the GE task, incoporating a KB perspective. For the evaluation, the final submission of each participant was converted to RDF statements, and evaluated using 8 queries that were formulated in SPARQL. The results suggest that the evaluation may be concluded differently between the two different perspectives, annotation vs. KB. We also provide a comparison of the GE and GRO tasks by converting their datasets into each other's format. More than 90% of the GE data could be converted into the GRO task format, while only half of the GRO data could be mapped to the GE task format. The imbalance in conversion indicates that the GRO is a comprehensive extension of the GE task ontology. We further used the converted GRO data as additional training data for the GE task, which helped improve GE task participant system performance. However, the converted GE data did not help GRO task participants, due to overfitting and the ontology gap. |
| first_indexed | 2024-10-01T04:16:45Z |
| format | Journal Article |
| id | ntu-10356/81489 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T04:16:45Z |
| publishDate | 2016 |
| record_format | dspace |
| spelling | ntu-10356/814892022-02-16T16:27:34Z Extending the evaluation of Genia Event task toward knowledge base construction and comparison to Gene Regulation Ontology task Kim, Jin-Dong Kim, Jung-jae Han, Xu Rebholz-Schuhmann, Dietrich School of Computer Science and Engineering text mining knowledge base semantic web resource description framework shared task evaluation bionlp information extraction Background: The third edition of the BioNLP Shared Task was held with the grand theme "knowledge base construction (KB)". The Genia Event (GE) task was re-designed and implemented in light of this theme. For its final report, the participating systems were evaluated from a perspective of annotation. To further explore the grand theme, we extended the evaluation from a perspective of KB construction. Also, the Gene Regulation Ontology (GRO) task was newly introduced in the third edition. The final evaluation of the participating systems resulted in relatively low performance. The reason was attributed to the large size and complex semantic representation of the ontology. To investigate potential benefits of resource exchange between the presumably similar tasks, we measured the overlap between the datasets of the two tasks, and tested whether the dataset for one task can be used to enhance performance on the other. Results: We report an extended evaluation on all the participating systems in the GE task, incoporating a KB perspective. For the evaluation, the final submission of each participant was converted to RDF statements, and evaluated using 8 queries that were formulated in SPARQL. The results suggest that the evaluation may be concluded differently between the two different perspectives, annotation vs. KB. We also provide a comparison of the GE and GRO tasks by converting their datasets into each other's format. More than 90% of the GE data could be converted into the GRO task format, while only half of the GRO data could be mapped to the GE task format. The imbalance in conversion indicates that the GRO is a comprehensive extension of the GE task ontology. We further used the converted GRO data as additional training data for the GE task, which helped improve GE task participant system performance. However, the converted GE data did not help GRO task participants, due to overfitting and the ontology gap. Published version 2016-06-28T09:20:37Z 2019-12-06T14:32:06Z 2016-06-28T09:20:37Z 2019-12-06T14:32:06Z 2015 Journal Article Kim, J.-D., Kim, J.-J., Han, X., & Rebholz-Schuhmann, D. (2015). Extending the evaluation of Genia Event task toward knowledge base construction and comparison to Gene Regulation Ontology task. BMC Bioinformatics, 16(Suppl 10), S3-. 1471-2105 https://hdl.handle.net/10356/81489 http://hdl.handle.net/10220/40825 10.1186/1471-2105-16-S10-S3 26202680 en BMC Bioinformatics © 2015 Kim et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. 13 p. application/pdf |
| spellingShingle | text mining knowledge base semantic web resource description framework shared task evaluation bionlp information extraction Kim, Jin-Dong Kim, Jung-jae Han, Xu Rebholz-Schuhmann, Dietrich Extending the evaluation of Genia Event task toward knowledge base construction and comparison to Gene Regulation Ontology task |
| title | Extending the evaluation of Genia Event task toward knowledge base construction and comparison to Gene Regulation Ontology task |
| title_full | Extending the evaluation of Genia Event task toward knowledge base construction and comparison to Gene Regulation Ontology task |
| title_fullStr | Extending the evaluation of Genia Event task toward knowledge base construction and comparison to Gene Regulation Ontology task |
| title_full_unstemmed | Extending the evaluation of Genia Event task toward knowledge base construction and comparison to Gene Regulation Ontology task |
| title_short | Extending the evaluation of Genia Event task toward knowledge base construction and comparison to Gene Regulation Ontology task |
| title_sort | extending the evaluation of genia event task toward knowledge base construction and comparison to gene regulation ontology task |
| topic | text mining knowledge base semantic web resource description framework shared task evaluation bionlp information extraction |
| url | https://hdl.handle.net/10356/81489 http://hdl.handle.net/10220/40825 |
| work_keys_str_mv | AT kimjindong extendingtheevaluationofgeniaeventtasktowardknowledgebaseconstructionandcomparisontogeneregulationontologytask AT kimjungjae extendingtheevaluationofgeniaeventtasktowardknowledgebaseconstructionandcomparisontogeneregulationontologytask AT hanxu extendingtheevaluationofgeniaeventtasktowardknowledgebaseconstructionandcomparisontogeneregulationontologytask AT rebholzschuhmanndietrich extendingtheevaluationofgeniaeventtasktowardknowledgebaseconstructionandcomparisontogeneregulationontologytask |