Characteristics of learning tasks in accounting textbooks: an AI assisted analysis

Empirical Research in Vocational Education and Training

Table 3 Inter-coder reliability measured by Krippendorff’s alpha

Construct	Human rating						Training of AI
Construct	K = 5, N = 56			K = 3, N = 56			I = 100
	Initial analysis			Validation analysis			Neural Net		Random Forest
	%^a	%^b	α	%^a	%^b	α	%^b	α	%^b	α
Process orientation	91	52	0.095	98	68	0.457	85	0.397	87	0.110
Description of social processes	82	75	0.089	98	71	0.285	93	0.460	94	0.383
Description of cash flows	94	48	0.409	95	84	0.579	90	0.394	92	0.310
Description of goods and services	94	58	0.529	96	60	0.441	92	0.410	94	0.350
Document orientation	89	76	0.483	95	95	0.709	96	0.383	96	0.474
Identification of problems	94	46	0.393	93	54	0.445	71	0.439	78	0.554
Obtaining missing information	74	43	0.287	91	75	0.732	72	0.432	76	0.303
Assessing the relevance of information	100	83	0.057	100	65	0.166	99	0.153	99	-0.006
Translate	100	21	0.064	100	38	0.157	67	0.220	75	0.245
Operate	100	60	0.618	100	91	0.872	83	0.646	88	0.742
Interpret	100	65	0.543	100	61	0.382	78	0.339	83	0.280
Interpret: liquidity aspects	100	87	0.325	100	96	-0.009	97	0.236	97	0.142
Interpret: profitability aspects	100	89	0.504	100	93	0.407	92	0.507	93	0.341
Validate	100	85	0.070	100	88	0.258	100	0.980	100	1.000

K Number of raters, N number of analyzed tasks, I iterations of performance assessment
^aEstimation with Tolerance 1
^bEstimation with Tolerance 0