Probabilistic Models for Competence Assessment in Education
Probabilistic models of competence assessment join the benefits of automation with human judgment. We start this paper by replicating two preexisting probabilistic models of peer assessment (<inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-02-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/12/5/2368 |
| _version_ | 1797475760067313664 |
|---|---|
| author | Alejandra López de Aberasturi Gómez Jordi Sabater-Mir Carles Sierra |
| author_facet | Alejandra López de Aberasturi Gómez Jordi Sabater-Mir Carles Sierra |
| author_sort | Alejandra López de Aberasturi Gómez |
| collection | DOAJ |
| description | Probabilistic models of competence assessment join the benefits of automation with human judgment. We start this paper by replicating two preexisting probabilistic models of peer assessment (<inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><msub><mi>G</mi><mn>1</mn></msub></mrow></semantics></math></inline-formula>-bias and PAAS). Despite the use that both make of probability theory, the approach of these models is radically different. While <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><msub><mi>G</mi><mn>1</mn></msub></mrow></semantics></math></inline-formula>-bias is purely Bayesian, PAAS models the evaluation process in a classroom as a multiagent system, where each actor relies on the judgment of others as long as their opinions coincide. To reconcile the benefits of Bayesian inference with the concept of trust posed in PAAS, we propose a third peer evaluation model that considers the correlations between any pair of peers who have evaluated someone in common: <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><mi>G</mi></mrow></semantics></math></inline-formula>-bivariate. The rest of the paper is devoted to a comparison with synthetic data from these three models. We show that <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><msub><mi>G</mi><mn>1</mn></msub></mrow></semantics></math></inline-formula>-bias produces predictions with lower root mean squared error (RMSE) than <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><mi>G</mi></mrow></semantics></math></inline-formula>-bivariate. However, both models display similar behaviors when assessing how to choose the next assignment to be graded by a peer, with an “<inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>R</mi><mi>M</mi><mi>S</mi><mi>E</mi></mrow></semantics></math></inline-formula> decreasing policy” reporting better results than a random policy. Fair comparisons among the three models show that <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><msub><mi>G</mi><mn>1</mn></msub></mrow></semantics></math></inline-formula>-bias makes the lowest error in situations of scarce ground truths. Nevertheless, once nearly <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>20</mn><mo>%</mo></mrow></semantics></math></inline-formula> of the teacher’s assessments are introduced, PAAS sometimes exceeds the quality of <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><msub><mi>G</mi><mn>1</mn></msub></mrow></semantics></math></inline-formula>-bias’ predictions by following an entropy minimization heuristic. <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><mi>G</mi></mrow></semantics></math></inline-formula>-bivariate, our new proposal to reconcile PAAS’ trust-based approach with <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><msub><mi>G</mi><mn>1</mn></msub></mrow></semantics></math></inline-formula>-bias’ theoretical background, obtains a similar percentage of error values to those of the original models. Future work includes applying the models to real experimental data and exploring new heuristics to determine which teacher’s grade should be obtained next to minimize the overall error. |
| first_indexed | 2024-03-09T20:48:22Z |
| format | Article |
| id | doaj.art-33bfee2af29f46f5b7a9d7709632340b |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-09T20:48:22Z |
| publishDate | 2022-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-33bfee2af29f46f5b7a9d7709632340b2023-11-23T22:39:38ZengMDPI AGApplied Sciences2076-34172022-02-01125236810.3390/app12052368Probabilistic Models for Competence Assessment in EducationAlejandra López de Aberasturi Gómez0Jordi Sabater-Mir1Carles Sierra2Artificial Intelligence Research Institute (IIIA-CSIC), 08193 Barcelona, SpainArtificial Intelligence Research Institute (IIIA-CSIC), 08193 Barcelona, SpainArtificial Intelligence Research Institute (IIIA-CSIC), 08193 Barcelona, SpainProbabilistic models of competence assessment join the benefits of automation with human judgment. We start this paper by replicating two preexisting probabilistic models of peer assessment (<inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><msub><mi>G</mi><mn>1</mn></msub></mrow></semantics></math></inline-formula>-bias and PAAS). Despite the use that both make of probability theory, the approach of these models is radically different. While <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><msub><mi>G</mi><mn>1</mn></msub></mrow></semantics></math></inline-formula>-bias is purely Bayesian, PAAS models the evaluation process in a classroom as a multiagent system, where each actor relies on the judgment of others as long as their opinions coincide. To reconcile the benefits of Bayesian inference with the concept of trust posed in PAAS, we propose a third peer evaluation model that considers the correlations between any pair of peers who have evaluated someone in common: <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><mi>G</mi></mrow></semantics></math></inline-formula>-bivariate. The rest of the paper is devoted to a comparison with synthetic data from these three models. We show that <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><msub><mi>G</mi><mn>1</mn></msub></mrow></semantics></math></inline-formula>-bias produces predictions with lower root mean squared error (RMSE) than <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><mi>G</mi></mrow></semantics></math></inline-formula>-bivariate. However, both models display similar behaviors when assessing how to choose the next assignment to be graded by a peer, with an “<inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>R</mi><mi>M</mi><mi>S</mi><mi>E</mi></mrow></semantics></math></inline-formula> decreasing policy” reporting better results than a random policy. Fair comparisons among the three models show that <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><msub><mi>G</mi><mn>1</mn></msub></mrow></semantics></math></inline-formula>-bias makes the lowest error in situations of scarce ground truths. Nevertheless, once nearly <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>20</mn><mo>%</mo></mrow></semantics></math></inline-formula> of the teacher’s assessments are introduced, PAAS sometimes exceeds the quality of <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><msub><mi>G</mi><mn>1</mn></msub></mrow></semantics></math></inline-formula>-bias’ predictions by following an entropy minimization heuristic. <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><mi>G</mi></mrow></semantics></math></inline-formula>-bivariate, our new proposal to reconcile PAAS’ trust-based approach with <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><msub><mi>G</mi><mn>1</mn></msub></mrow></semantics></math></inline-formula>-bias’ theoretical background, obtains a similar percentage of error values to those of the original models. Future work includes applying the models to real experimental data and exploring new heuristics to determine which teacher’s grade should be obtained next to minimize the overall error.https://www.mdpi.com/2076-3417/12/5/2368peer assessmentmultiagent systemprobabilistic modelcomparative analysisBayesian network |
| spellingShingle | Alejandra López de Aberasturi Gómez Jordi Sabater-Mir Carles Sierra Probabilistic Models for Competence Assessment in Education Applied Sciences peer assessment multiagent system probabilistic model comparative analysis Bayesian network |
| title | Probabilistic Models for Competence Assessment in Education |
| title_full | Probabilistic Models for Competence Assessment in Education |
| title_fullStr | Probabilistic Models for Competence Assessment in Education |
| title_full_unstemmed | Probabilistic Models for Competence Assessment in Education |
| title_short | Probabilistic Models for Competence Assessment in Education |
| title_sort | probabilistic models for competence assessment in education |
| topic | peer assessment multiagent system probabilistic model comparative analysis Bayesian network |
| url | https://www.mdpi.com/2076-3417/12/5/2368 |
| work_keys_str_mv | AT alejandralopezdeaberasturigomez probabilisticmodelsforcompetenceassessmentineducation AT jordisabatermir probabilisticmodelsforcompetenceassessmentineducation AT carlessierra probabilisticmodelsforcompetenceassessmentineducation |