Performance of gender detection tools: a comparative study of name-to-gender inference services
Objective: To evaluate the performance of gender detection tools that allow the uploading of files (e.g., Excel or CSV files) containing first names, are usable by researchers without advanced computer skills, and are at least partially free of charge. Methods: The study was conducted using four phy...
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
University Library System, University of Pittsburgh
2021-10-01
|
| Series: | Journal of the Medical Library Association |
| Subjects: | |
| Online Access: | http://jmla.pitt.edu/ojs/jmla/article/view/1185 |
| _version_ | 1818983785584132096 |
|---|---|
| author | Paul Sebo |
| author_facet | Paul Sebo |
| author_sort | Paul Sebo |
| collection | DOAJ |
| description | Objective: To evaluate the performance of gender detection tools that allow the uploading of files (e.g., Excel or CSV files) containing first names, are usable by researchers without advanced computer skills, and are at least partially free of charge.
Methods: The study was conducted using four physician datasets (total number of physicians: 6,131; 50.3% female) from Switzerland, a multilingual country. Four gender detection tools met the inclusion criteria: three partially free (Gender API, NamSor, and genderize.io) and one completely free (Wiki-Gendersort). For each tool, we recorded the number of correct classifications (i.e., correct gender assigned to a name), misclassifications (i.e., wrong gender assigned to a name), and nonclassifications (i.e., no gender assigned). We computed three metrics: the proportion of misclassifications excluding nonclassifications (errorCodedWithoutNA), the proportion of nonclassifications (naCoded), and the proportion of misclassifications and nonclassifications (errorCoded).
Results: The proportion of misclassifications was low for all four gender detection tools (errorCodedWithoutNA between 1.5 and 2.2%). By contrast, the proportion of unrecognized names (naCoded) varied: 0% for NamSor, 0.3% for Gender API, 4.5% for Wiki-Gendersort, and 16.4% for genderize.io. Using errorCoded, which penalizes both types of error equally, we obtained the following results: Gender API 1.8%, NamSor 2.0%, Wiki-Gendersort 6.6%, and genderize.io 17.7%.
Conclusions: Gender API and NamSor were the most accurate tools. Genderize.io led to a high number of nonclassifications. Wiki-Gendersort may be a good compromise for researchers wishing to use a completely free tool. Other studies would be useful to evaluate the performance of these tools in other populations (e.g., Asian). |
| first_indexed | 2024-12-20T18:08:38Z |
| format | Article |
| id | doaj.art-4232e4e47ae744f2b0f12f9005c24607 |
| institution | Directory Open Access Journal |
| issn | 1536-5050 1558-9439 |
| language | English |
| last_indexed | 2024-12-20T18:08:38Z |
| publishDate | 2021-10-01 |
| publisher | University Library System, University of Pittsburgh |
| record_format | Article |
| series | Journal of the Medical Library Association |
| spelling | doaj.art-4232e4e47ae744f2b0f12f9005c246072022-12-21T19:30:31ZengUniversity Library System, University of PittsburghJournal of the Medical Library Association1536-50501558-94392021-10-01109310.5195/jmla.2021.1185598Performance of gender detection tools: a comparative study of name-to-gender inference servicesPaul SeboObjective: To evaluate the performance of gender detection tools that allow the uploading of files (e.g., Excel or CSV files) containing first names, are usable by researchers without advanced computer skills, and are at least partially free of charge. Methods: The study was conducted using four physician datasets (total number of physicians: 6,131; 50.3% female) from Switzerland, a multilingual country. Four gender detection tools met the inclusion criteria: three partially free (Gender API, NamSor, and genderize.io) and one completely free (Wiki-Gendersort). For each tool, we recorded the number of correct classifications (i.e., correct gender assigned to a name), misclassifications (i.e., wrong gender assigned to a name), and nonclassifications (i.e., no gender assigned). We computed three metrics: the proportion of misclassifications excluding nonclassifications (errorCodedWithoutNA), the proportion of nonclassifications (naCoded), and the proportion of misclassifications and nonclassifications (errorCoded). Results: The proportion of misclassifications was low for all four gender detection tools (errorCodedWithoutNA between 1.5 and 2.2%). By contrast, the proportion of unrecognized names (naCoded) varied: 0% for NamSor, 0.3% for Gender API, 4.5% for Wiki-Gendersort, and 16.4% for genderize.io. Using errorCoded, which penalizes both types of error equally, we obtained the following results: Gender API 1.8%, NamSor 2.0%, Wiki-Gendersort 6.6%, and genderize.io 17.7%. Conclusions: Gender API and NamSor were the most accurate tools. Genderize.io led to a high number of nonclassifications. Wiki-Gendersort may be a good compromise for researchers wishing to use a completely free tool. Other studies would be useful to evaluate the performance of these tools in other populations (e.g., Asian).http://jmla.pitt.edu/ojs/jmla/article/view/1185accuracygender detectionmisclassificationnamename-to-genderperformance |
| spellingShingle | Paul Sebo Performance of gender detection tools: a comparative study of name-to-gender inference services Journal of the Medical Library Association accuracy gender detection misclassification name name-to-gender performance |
| title | Performance of gender detection tools: a comparative study of name-to-gender inference services |
| title_full | Performance of gender detection tools: a comparative study of name-to-gender inference services |
| title_fullStr | Performance of gender detection tools: a comparative study of name-to-gender inference services |
| title_full_unstemmed | Performance of gender detection tools: a comparative study of name-to-gender inference services |
| title_short | Performance of gender detection tools: a comparative study of name-to-gender inference services |
| title_sort | performance of gender detection tools a comparative study of name to gender inference services |
| topic | accuracy gender detection misclassification name name-to-gender performance |
| url | http://jmla.pitt.edu/ojs/jmla/article/view/1185 |
| work_keys_str_mv | AT paulsebo performanceofgenderdetectiontoolsacomparativestudyofnametogenderinferenceservices |