Table 6 Relative Frequencies and Results of a Wilcoxon Test for Paired Samples Between the AI and Human Ratings (N = 1080)
From: Characteristics of learning tasks in accounting textbooks: an AI assisted analysis
Human coding | AI coding | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
Construct, category level | 0 | 1 | 2 | 0 | 1 | 2 | Method | z-Value | p-value | Effect size r |
Process orientation | 85.9% | 9.8% | 4.3% | 60.1% | 27.2% | 12.7% | NN | − 15.6 | 0 | − 0.475 |
Description of social processes | 92.0% | 6.7% | 1.3% | 81.0% | 16.7% | 2.3% | NN | − 9.27 | 0 | − 0.282 |
Description of goods and services | 92.3% | 7.7% | 86.6% | 13.4% | NN | − 6.54 | 0 | − 0.199 | ||
Description of cash flows | 90.1% | 9.9% | 83.5% | 16.5% | NN | − 7.89 | 0 | − 0.240 | ||
Document orientation | 94.5% | 0.0% | 5.5% | 94.7% | 0.0% | 5.3% | RF | 1.41 | .346 | 0.043 |
Identification of problems | 53.9% | 45.1% | 1.0% | 61.5% | 37.6% | 0.9% | RF | 8.9 | 0 | 0.271 |
Obtaining missing information | 69.4% | 19.3% | 11.3% | 55.9% | 27.5% | 16.6% | NN | − 8.29 | 0 | − 0.252 |
Assessing the relevance of information | 98.4% | 1.6% | 100% | 0.0% | NN | 4.12 | 0 | 0.125 | ||
Translate | 70.5% | 29.5% | 79.3% | 20.7% | RF | 9.75 | 0 | 0.297 | ||
Operate | 38.0% | 62.0% | 43.2% | 56.8% | RF | 7.18 | 0 | 0.218 | ||
Interpret | 79.4% | 20.6% | 74.3% | 25.7% | NN | − 4.04 | 0 | − 0.123 | ||
Interpret: liquidity aspects | 97.2% | 2.8% | 93.3% | 6.7% | NN | − 6.19 | 0 | − 0.188 | ||
Interpret: profitability aspects | 91.3% | 8.7% | 80.0% | 20.0% | NN | − 10 | 0 | − 0.304 | ||
Validate | 99.8% | 0.2% | 99.8% | 0.0% | RF | –/– | –/– | –/– | ||