Method to assess the trustworthiness of machine coding at scale
Physics education researchers are interested in using the tools of machine learning and natural language processing to make quantitative claims from natural language and text data, such as open-ended responses to survey questions. The aspiration is that this form of machine coding may be more effici...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
American Physical Society
2024-03-01
|
| Series: | Physical Review Physics Education Research |
| Online Access: | http://doi.org/10.1103/PhysRevPhysEducRes.20.010113 |
| _version_ | 1797271657700655104 |
|---|---|
| author | Rebeckah K. Fussell Emily M. Stump N. G. Holmes |
| author_facet | Rebeckah K. Fussell Emily M. Stump N. G. Holmes |
| author_sort | Rebeckah K. Fussell |
| collection | DOAJ |
| description | Physics education researchers are interested in using the tools of machine learning and natural language processing to make quantitative claims from natural language and text data, such as open-ended responses to survey questions. The aspiration is that this form of machine coding may be more efficient and consistent than human coding, allowing much larger and broader datasets to be analyzed than is practical with human coders. Existing work that uses these tools, however, does not investigate norms that allow for trustworthy quantitative claims without full reliance on cross-checking with human coding, which defeats the purpose of using these automated tools. Here we propose a four-part method for making such claims with supervised natural language processing: evaluating a trained model, calculating statistical uncertainty, calculating systematic uncertainty from the trained algorithm, and calculating systematic uncertainty from novel data sources. We provide evidence for this method using data from two distinct short response survey questions with two distinct coding schemes. We also provide a real-world example of using these practices to machine code a dataset unseen by human coders. We offer recommendations to guide physics education researchers who may use machine-coding methods in the future. |
| first_indexed | 2024-03-07T14:09:37Z |
| format | Article |
| id | doaj.art-40815cb1dca849c0b5d38d1b7a9d80db |
| institution | Directory Open Access Journal |
| issn | 2469-9896 |
| language | English |
| last_indexed | 2024-03-07T14:09:37Z |
| publishDate | 2024-03-01 |
| publisher | American Physical Society |
| record_format | Article |
| series | Physical Review Physics Education Research |
| spelling | doaj.art-40815cb1dca849c0b5d38d1b7a9d80db2024-03-06T17:24:36ZengAmerican Physical SocietyPhysical Review Physics Education Research2469-98962024-03-0120101011310.1103/PhysRevPhysEducRes.20.010113Method to assess the trustworthiness of machine coding at scaleRebeckah K. FussellEmily M. StumpN. G. HolmesPhysics education researchers are interested in using the tools of machine learning and natural language processing to make quantitative claims from natural language and text data, such as open-ended responses to survey questions. The aspiration is that this form of machine coding may be more efficient and consistent than human coding, allowing much larger and broader datasets to be analyzed than is practical with human coders. Existing work that uses these tools, however, does not investigate norms that allow for trustworthy quantitative claims without full reliance on cross-checking with human coding, which defeats the purpose of using these automated tools. Here we propose a four-part method for making such claims with supervised natural language processing: evaluating a trained model, calculating statistical uncertainty, calculating systematic uncertainty from the trained algorithm, and calculating systematic uncertainty from novel data sources. We provide evidence for this method using data from two distinct short response survey questions with two distinct coding schemes. We also provide a real-world example of using these practices to machine code a dataset unseen by human coders. We offer recommendations to guide physics education researchers who may use machine-coding methods in the future.http://doi.org/10.1103/PhysRevPhysEducRes.20.010113 |
| spellingShingle | Rebeckah K. Fussell Emily M. Stump N. G. Holmes Method to assess the trustworthiness of machine coding at scale Physical Review Physics Education Research |
| title | Method to assess the trustworthiness of machine coding at scale |
| title_full | Method to assess the trustworthiness of machine coding at scale |
| title_fullStr | Method to assess the trustworthiness of machine coding at scale |
| title_full_unstemmed | Method to assess the trustworthiness of machine coding at scale |
| title_short | Method to assess the trustworthiness of machine coding at scale |
| title_sort | method to assess the trustworthiness of machine coding at scale |
| url | http://doi.org/10.1103/PhysRevPhysEducRes.20.010113 |
| work_keys_str_mv | AT rebeckahkfussell methodtoassessthetrustworthinessofmachinecodingatscale AT emilymstump methodtoassessthetrustworthinessofmachinecodingatscale AT ngholmes methodtoassessthetrustworthinessofmachinecodingatscale |