Assessing the effects of hyperparameters on knowledge graph embedding quality
Abstract Embedding knowledge graphs into low-dimensional spaces is a popular method for applying approaches, such as link prediction or node classification, to these databases. This embedding process is very costly in terms of both computational time and space. Part of the reason for this is the opt...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SpringerOpen
2023-05-01
|
| Series: | Journal of Big Data |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s40537-023-00732-5 |
| _version_ | 1797832011151310848 |
|---|---|
| author | Oliver Lloyd Yi Liu Tom R. Gaunt |
| author_facet | Oliver Lloyd Yi Liu Tom R. Gaunt |
| author_sort | Oliver Lloyd |
| collection | DOAJ |
| description | Abstract Embedding knowledge graphs into low-dimensional spaces is a popular method for applying approaches, such as link prediction or node classification, to these databases. This embedding process is very costly in terms of both computational time and space. Part of the reason for this is the optimisation of hyperparameters, which involves repeatedly sampling, by random, guided, or brute-force selection, from a large hyperparameter space and testing the resulting embeddings for their quality. However, not all hyperparameters in this search space will be equally important. In fact, with prior knowledge of the relative importance of the hyperparameters, some could be eliminated from the search altogether without significantly impacting the overall quality of the outputted embeddings. To this end, we ran a Sobol sensitivity analysis to evaluate the effects of tuning different hyperparameters on the variance of embedding quality. This was achieved by performing thousands of embedding trials, each time measuring the quality of embeddings produced by different hyperparameter configurations. We regressed the embedding quality on those hyperparameter configurations, using this model to generate Sobol sensitivity indices for each of the hyperparameters. By evaluating the correlation between Sobol indices, we find substantial variability in the hyperparameter sensitivities between knowledge graphs with differing dataset characteristics as the probable cause of these inconsistencies. As an additional contribution of this work we identify several relations in the UMLS knowledge graph that may cause data leakage via inverse relations, and derive and present UMLS-43, a leakage-robust variant of that graph. |
| first_indexed | 2024-04-09T14:01:58Z |
| format | Article |
| id | doaj.art-35f08b07db924547890c9cba1a8ac9a6 |
| institution | Directory Open Access Journal |
| issn | 2196-1115 |
| language | English |
| last_indexed | 2024-04-09T14:01:58Z |
| publishDate | 2023-05-01 |
| publisher | SpringerOpen |
| record_format | Article |
| series | Journal of Big Data |
| spelling | doaj.art-35f08b07db924547890c9cba1a8ac9a62023-05-07T11:15:36ZengSpringerOpenJournal of Big Data2196-11152023-05-0110111510.1186/s40537-023-00732-5Assessing the effects of hyperparameters on knowledge graph embedding qualityOliver Lloyd0Yi Liu1Tom R. Gaunt2MRC Integrative Epidemiology Unit, Bristol Medical School, University of BristolMRC Integrative Epidemiology Unit, Bristol Medical School, University of BristolMRC Integrative Epidemiology Unit, Bristol Medical School, University of BristolAbstract Embedding knowledge graphs into low-dimensional spaces is a popular method for applying approaches, such as link prediction or node classification, to these databases. This embedding process is very costly in terms of both computational time and space. Part of the reason for this is the optimisation of hyperparameters, which involves repeatedly sampling, by random, guided, or brute-force selection, from a large hyperparameter space and testing the resulting embeddings for their quality. However, not all hyperparameters in this search space will be equally important. In fact, with prior knowledge of the relative importance of the hyperparameters, some could be eliminated from the search altogether without significantly impacting the overall quality of the outputted embeddings. To this end, we ran a Sobol sensitivity analysis to evaluate the effects of tuning different hyperparameters on the variance of embedding quality. This was achieved by performing thousands of embedding trials, each time measuring the quality of embeddings produced by different hyperparameter configurations. We regressed the embedding quality on those hyperparameter configurations, using this model to generate Sobol sensitivity indices for each of the hyperparameters. By evaluating the correlation between Sobol indices, we find substantial variability in the hyperparameter sensitivities between knowledge graphs with differing dataset characteristics as the probable cause of these inconsistencies. As an additional contribution of this work we identify several relations in the UMLS knowledge graph that may cause data leakage via inverse relations, and derive and present UMLS-43, a leakage-robust variant of that graph.https://doi.org/10.1186/s40537-023-00732-5Knowledge graphEmbeddingSensitivity analysisHyperparameter tuning |
| spellingShingle | Oliver Lloyd Yi Liu Tom R. Gaunt Assessing the effects of hyperparameters on knowledge graph embedding quality Journal of Big Data Knowledge graph Embedding Sensitivity analysis Hyperparameter tuning |
| title | Assessing the effects of hyperparameters on knowledge graph embedding quality |
| title_full | Assessing the effects of hyperparameters on knowledge graph embedding quality |
| title_fullStr | Assessing the effects of hyperparameters on knowledge graph embedding quality |
| title_full_unstemmed | Assessing the effects of hyperparameters on knowledge graph embedding quality |
| title_short | Assessing the effects of hyperparameters on knowledge graph embedding quality |
| title_sort | assessing the effects of hyperparameters on knowledge graph embedding quality |
| topic | Knowledge graph Embedding Sensitivity analysis Hyperparameter tuning |
| url | https://doi.org/10.1186/s40537-023-00732-5 |
| work_keys_str_mv | AT oliverlloyd assessingtheeffectsofhyperparametersonknowledgegraphembeddingquality AT yiliu assessingtheeffectsofhyperparametersonknowledgegraphembeddingquality AT tomrgaunt assessingtheeffectsofhyperparametersonknowledgegraphembeddingquality |