Analysing domain-specific problem-solving processes within authentic computer-based learning and training environments by using eye-tracking: a scoping review

Abstract

Recently, many studies have been published on the use of eye-tracking to analyse complex problem-solving processes within authentic computer-based learning and training environments. This scoping review aims to provide a systematic report of the current state-of-the-art for related papers. Specifically, this work offers a scoping review of studies that analyse problem-solving processes by using eye-tracking (alongside additional process data such as log files, think aloud, facial expression recognition algorithms, or psychophysiological measures) within authentic technology-based learning and training environments for professional and vocational education and training (VET). A total of 12 studies were identified. The most commonly calculated measures in eye-tracking research are position measures, and these are almost exclusively position duration measures such as the proportion of fixation times or total dwell times. Count measures are also mostly related to the number or proportion of fixations and dwells. Movement measures are rarely computed and usually refer to saccade directions or a scan path. Also, latency and distance measures are almost never calculated. Eye-tracking data is most often analysed for group comparisons between experts vs. novices or high vs. low-performing groups by using common statistical methods such as t-test, (M)ANOVA, or non-parametric Mann–Whitney-U. Visual attention patterns in problem-solving are examined with heat map analyses, lag sequential analyses, and clustering. Recently, linear mixed-effects models have been applied to account for between and within-subjects differences. Also, post-hoc performance predictions are being developed for future integration into multimodal learning analytics. In most cases, self-reporting is used as an additional measurement for data triangulation. In addition to eye-tracking, log files and facial expression recognition algorithms are also used. Few studies use shimmer devices to detect electrodermal activity or practice concurrent thinking aloud. Overall, Haider and Frensch’s (1996, 1999) “information reduction hypothesis” is supported by many studies in the sample. High performers showed a higher visual accuracy, and visual attention was more focused on relevant areas, as seen by fewer fixation counts and higher fixation duration. Low performers showed significantly fewer fixation durations or substantially longer fixation durations and less selective visual attention. Performance is related to prior knowledge and differences in cognitive load. Eye-tracking, (in combination with other data sources) may be a valid method for further research on problem-solving processes in computer-based simulations, may help identify different patterns of problem-solving processes between performance groups, and may hold additional potential for individual learning support.

Introduction

In educational research, collecting behavioural data is becoming increasingly important to learn more about cognitive and metacognitive processes during learning and instruction. Eye-tracking, a method for analysing the course of gaze, is increasingly used in educational research to improve the instructional design in computer-based learning environments and multimedia learning, to understand and promote the development of expertise, and to visualize the eye movements of experts (Jarodzka et al. 2017). Systematic reviews and meta-analyses of eye-tracking studies are available for various domains (e.g., medical education: Ashraf et al. 2018; mathematics: Strohmaier et al. 2020). Similarly, reviews on multimedia learning and instructional design were conducted (e.g., Alemdag and Cagiltay 2018: multimedia learning; Yang et al. 2018: instructional design of e-learning). However, little research has been done on vocational education and training (VET). This is especially true when more complex vocational tasks are the focus. Therefore, the paper at hand provides an overview of studies that have analysed domain-specific problem-solving processes by using eye-tracking (next to additional online data such as log files or psychophysiological measures) within authentic computer-based learning and training environments in professional training or vocational education and training. The review of the current state of research is conducted as a scoping review. Scoping reviews (Arksey and O’Malley 2005) are considered a useful approach to examining the design and conduct of research on a particular topic (for the key features of the scoping review approach see Munn et al. 2018). Similar to systematic reviews, scoping reviews are transparent and replicable by following a rigorous study search and selection process. Because fewer studies were identified that followed the same study design, we focused on a scoping review.

While systematic reviews aim to answer a specific question, a scoping review identifies, reports, and discusses a broader perspective on a given topic (Arksey and O’Malley 2005; Munn et al. 2018; Van Ostaeyen et al. 2022), such as analysing domain-specific problem-solving in computer-based simulations by using eye-tracking.

Computer-based learning environments (CBLE) refer to a broad range of technologies to support learning and instruction (Lajoie and Naismith 2012). This review includes studies that used eye-tracking as the primary method for collecting behavioural data in computer simulations and serious games. To control for different levels of immersion and interactivity, we excluded close-to-reality simulations, such as in situ studies and realistic simulator training (e.g., flight simulation including a full cockpit, virtual reality welding simulation, nursing practice simulation with mannequins or actors) and similarly, we excluded studies that examined cross-sectoral problem-solving testing (e.g., Raven Matrices, Tower of Hanoi). Also, we excluded studies on collaborative problem-solving, as these studies often focus on the phenomena of joint visual attention. This rigorous procedure resulted in a sample of studies, all of which analysed domain-specific problem-solving in computer-based simulations (see Fig. 1).

Fig. 1

The scope of this review (in bold) is illustrated within a continuum between cross-sectoral and domain-specific problem-solving simulations (pictures CC0-licensed)

The scoping review at hand aims to provide a systematic report of the current state-of-the-art in an emerging research field. It contains a variety of eye-tracking and process data measures in a broad range of domain-specific problem-solving tasks. Eye movements can be analysed with various measurements and should be carefully collected and interpreted (Holmqvist et al. 2011). In multimedia learning research, there is a wide range of empirical research methods. Online process tracking techniques such as eye tracking can be combined with other common measurement methods to draw better inferences (Jarodzka 2021). Thus, we address the following questions:

RQ1: Which eye-tracking measures and additional behavioural measurements were used and how were they analysed?

RQ2: What are the main findings of online data measures in relation to solving complex problems in computer-based simulations in VET?

The paper is structured in five sections. First, a brief overview of the theoretical background on problem-solving, computer-based simulations, and eye-tracking is given. Second, the methodological approach of this review (identification, screening, eligibility, and inclusion) is presented. Third, the main findings are reported. Finally, the results are discussed, and the limitations of this work and implications for future research are considered.

Theoretical background

Problem-solving

Problems arise when someone has a goal but lacks the knowledge of how to achieve it (Duncker 1945). Various problem types exist. Jonassen (2000) provides a taxonomy of 11 problems from well-structured such as algorithmic problems, to ill-structured such as dilemmas, in which the more ill-structured problem types may encompass more structured problems. Following Dörner (1987; Funke 2012), complex problems include various interconnected variables, multiple and conflicting goals, a lack of transparency, and dynamic development. It is difficult to accurately capture the scope of requirements to solve a particular problem. In educational contexts, problem difficulty is often assessed ex post facto based on solution rates (Jonassen and Hung 2008). Furthermore, it is challenging to define the difficulty or complexity of a problem a priori because the subjective perception of a problem varies with prior knowledge and experience in the respective domain (Dörner 1997). Similarly, Mayer and Wittrock (2006) distinguish routine and non-routine aspects of a problem, with a routine problem defined as a problem “for which the problem solver already possesses a ready-made solution procedure” (p. 288). In the context of competence assessment, Williamson et al. (2006) attempt to objectively define a task as complex if (a) the problem solver has to undergo multiple, non-trivial, domain-relevant steps and/or cognitive processes, (b) multiple features of task performance are captured, (c) task performance is relatively unconstrained, and (d) evaluations of task solutions recognise the interdependence of task features and aspects of performance.

Solving problems requires cognitive, metacognitive, and non-cognitive processes (Frensch and Funke 1995; Mayer 1998; Jonassen 2000). Weinert (2001) defines such competencies as a combination of “[…] intellectual abilities, content-specific knowledge, cognitive skills, domain-specific strategies, routines and subroutines, motivational tendencies, volitional control systems, personal value orientations, and social behaviours” (p. 51). Similarly, Fischer and Neubert (2015) define problem-solving competence as a multidimensional construct that includes knowledge, skills, abilities, and other components (KSAO), with ‘other components’ referring to non-cognitive facets such as frustration tolerance and a positive attitude in particular. Following Mayer and Wittrock (2006), problem-solving is preferably related to a specific domain instead of general heuristics (domain-specific principle), most likely restricted to a certain problem and not widely transferable to other problems (near-transfer principle), and should be integrated into teaching as guided problem-solving tasks to foster learning (knowledge integrating principle). Therefore, in order to promote problem-solving skills, problem-oriented tasks should be embedded as authentic, domain-specific scenarios in VET.

Problem-solving in computer-based simulations

Authentic domain-specific problems are so-called ‘metaproblems’ (Jonassen 2000), a combination of many problem variations and types that are connected over a single domain. Metaproblems can be illustrated within computer-based simulations to replicate real-world tasks in a safe environment for training and learning purposes while providing an authentic and dynamic simulation-based learning scenario that changes either with decisions (interactions), with time, or both (Dörner and Funke 2017). Such open-ended environments emphasize learner-centred activities, setting authentic tasks for learners, and providing them with authentic tools (Hannafin 1995; Clarebout et al. 2009). The experimental learning opportunity addresses the cognitive, motivational, affective, psychomotor, and social aspects of learning (Breckwoldt et al. 2014). Early research on problem-solving was conducted within computer-simulated microworlds (Brehmer and Dörner 1993). Nowadays, especially in the field of vocational education and training, there is a large number of domain-specific, authentic computer-based simulations to promote competence development in general and domain-specific problem-solving competence in particular (Beck et al. 2016; Rausch et al. 2016). However, most research in VET focuses on outcomes (competence assessment, learning performance, etc.) and not on the processes that precede these outcomes (Abele 2018). Therefore, it seems worthwhile to highlight the methodological advantages of process data channels such as eye-tracking.

Eye-tracking, eye movements, and eye-tracking in problem-solving

Eye-tracking is a technology used as a research method for recording eye behaviour such as pupil dilation, blinking, and especially eye movements, as an indicator of visual attention when processing information (Holmqvist et al. 2011; Duchowski 2017; Holmqvist and Andersson 2017). Eye-tracking in computer-based simulations might help to make inferences about cognitive and metacognitive processes during learning (van Gog et al. 2009; van Gog and Jarodzka 2013).

Eye movements reflect top-down (goal-driven or endogenous) and bottom-up (stimulus-driven or exogenous) visual attention (Rayner 1998; Theeuwes 2010; Orquin and Mueller Loose 2013). Bottom-up control depends on stimulus features, such as visual saliency, i.e., the subjective quality of a stimulus that grabs visual attention (contrast, colour, movements). Top-down control depends on observer features, such as expertise, prior knowledge, tasks, etc. Also, individual eye features need to be considered. The most common types of eye movement events are fixations and saccades, where a fixation refers to the state when the eyes remain still (e.g., a stop during reading) while a saccade refers to the motion of the eyes between fixations (Holmqvist and Andersson 2017). According to the ‘eye-mind hypothesis’, fixations should be a proxy of cognitive processing: “the eye remains fixated on a word as long as the word is being processed” (Just and Carpenter 1980). This influential assumption (originally related to reading research, but also tested beyond reading) has been challenged several times (Underwood and Everatt 1992; Anderson et al. 2004), and today there is a consensus that visual attention somewhat precedes gaze and that overt and covert attention can differ (Holmqvist and Andersson 2017).

Eye-movement data can reveal differences in visual attention to areas of interest (AOI) during problem-solving processes. Following Haider and Frensch’s ‘information reduction hypothesis’ (1996, 1999), deliberate practice helps students to learn to ignore redundant information and focus more on relevant information. Thus, especially experts have learned to distinguish relevant from irrelevant task-information through practice (Lee and Anderson 2001) and make use of efficient cognitive strategies through experience (van Merriënboer 2013). A meta-analysis examining the effects of expertise on visual comprehension conducted by Gegenfurtner and colleagues (2011) supported that experts have shorter fixation durations, more fixations on relevant areas, and fewer fixations on irrelevant areas than novices. Experts also showed selective attention through parafoveal processing (unattended locations of the visual field) indicated by longer saccades and shorter times to first fixation (Gegenfurtner et al. 2011). Additionally, research on eye-movement modelling examples (EMME) indicates that EMME might help to guide novices’ visual attention. EMME illustrates the visual processing behaviour of experts carefully performing a task by recordings their eye movements. A meta-analysis on EMME shows significant effects of eye-tracking measures such as time to first fixation and fixation duration on novice learners' performance outcomes in terms of learning outcomes and problem-solving (Xie et al. 2021).

Despite the unquestioned potential of the eye-tracking approach, conclusions from eye-tracking data must be drawn very carefully. While there are a variety of eye movement measures, taxonomies, and interpretations, it is important to emphasize the theoretical assumptions as well as the domain and task-specific characteristics that underlie a research objective, and to our knowledge, there is currently no cross-domain taxonomy. However, a categorization based on operational definitions including a variety of eye-tracking measures can be used (see Holmqvist and Andersson 2017; Part III on paradigms and measures). We would like to point out some aspects concerning the ambiguity of eye-tracking measurements. For example, if a participant shows a higher fixation duration for relevant information, this may indicate reasoning, but also confusion or mind-wandering (‘staring into space). Therefore, eye-tracking should be triangulated with offline and online data channels (which might induce other challenges to a research design), such as self-reports and self-assessments (which often suffer from several biases, Andrade 2019), log files (which are often restricted to binary representations and suffers ambiguous interpretability, Goldhammer et al. 2014), which do not capture relevant off-screen behaviour (Maddox et al. 2018), as well as retrospective (delayed report) or concurrent (disturbing) think aloud (Gegenfurtner and Seppänen 2013), and psychophysiological measures like heart rates which might be hard to interpret (Wu et al. 2014). However, the combination of multimodal data channels within advanced learning technologies is on the rise (Gabriel et al. 2022). Thus, the following review will also take additional behavioural measures into account to underline the potential of multimodal methods in combination with eye-tracking.

Methodology

Identification—Search strategy

A search and selection process in line with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA 2020; see Liberati et al. 2009; Page et al. 2021) was conducted (Fig. 2). The search was based on common educational research databases (Fraenkel et al. 2019): Web of Science, PsycInfo, and Education Resources Information Center (ERIC). We used the following four major search terms: (1) eye-tracking AND (2) problem-solving AND (3) training and learning AND (4) computer-based simulation. Eye-Tracking studies were identified by searching for eye-tracking OR eye* OR gaze OR fixation OR saccade. Problem-solving tasks were identified by searching for problem-solving OR problem* OR decision-making OR decision* OR choice OR domain-specific* OR complex task. The domains of professional training or education were addressed by education* OR vocation* OR apprentice* OR training OR program OR workshop OR workplace. Finally, the search targets computer-based simulations and thus includes computer OR simulation OR virtual. The database search yielded 1,061 records. An additional check of reference lists for prominent eye-tracking reviews and meta-analysis was conducted (Gegenfurtner et al. 2011: a meta-analysis on expertise differences in visual comprehension; Lai et al. 2013: a review on eye-tracking studies in learning; Orquin and Mueller Loose 2013, review of attentional shifts in decision tasks). Additionally, Google Scholar was manually searched with varieties of search strings. All findings were combined in a list, and duplicates were identified automatically. An additional manual check was performed. The search process resulted in 914 records for subsequent selection.

Fig. 2

PRISMA flow chart

Screening—Screening abstracts and titles

This review used CADIMA, a free web tool supporting the systematic review process (see Kohl et al. 2018). The selection criteria were applied at the title and abstract levels. Table 1 lists the key features, inclusion criteria, and exclusion examples.

Table 1 Key features, inclusion criteria, and exclusion examples

We tested the inclusion criteria with a consistency check between two independent researchers to examine the inter-reviewer agreement and to overcome researcher bias. A second researcher screened randomly selected abstracts (n = 45; > 5% of all potential records). The first check showed that the strength of agreement for screening (either inclusion, exclusion, or unsure) was k = 0.60 and considered as moderate (Landis and Koch 1977). A kappa value of at least k ≥ 0.6 is recommended to continue (Orwin and Vevea 2009; Higgins et al. 2019). Further screening was revised by examining differences and resolving disagreements through discussion and debate by two reviewers, resulting in overall intercoder reliability of k = 0.89, which can be interpreted (Landis and Koch 1977) as an almost perfect agreement (k ≥ 0.81). Given the satisfying kappa value and for the sake of resource efficiency, the process proceeded with a single-person screening. A total of 914 records (after the elimination of duplicates), were screened. Overall, 776 records were excluded from this study.

Eligibility—Screening full texts

No papers were excluded for reasons of inaccessibility. If a paper was not available through institutional access, we kindly asked via ResearchGate^® to provide the paper. A total of 138 records remained for screening at the full-text level. Overall, 126 full-text articles were excluded from the study. Owing to the high number of excluded articles, we provide some examples. We excluded studies that (1) did not conduct a complex problem-solving task or (2) were not constructed within an interactive environment.

(1): We excluded studies that did not match a domain-specific problem-oriented approach or interactive design. For example, we excluded a hypermedia-based learning environment that fostered self-regulated learning by using metacognitive pedagogical agents. The subjects learned about the human circulatory system for 90 min. However, a problem-solving task was not in the scope of this study (Taub and Azevedo 2019). Studies based on a fixed stimulus without further interactions (mostly graphs or animations) were excluded. For example, we excluded a multiple-choice science study conducted by Tsai et al. (2011), which examines a science problem represented by four images in a web browser.

(2): We excluded studies on highly interactive learning environments for different reasons. A training simulator often integrates motor-sensitive skills. Training simulators contain realistic switches, knobs, levers, and typical instruments, e.g., flying simulations for pilot training (Schriver et al. 2008), and offshore drilling simulation (Naqvi et al. 2020). Participants train in dangerous scenarios to become familiar with safety processes in realistic but risk-free simulations and to develop routines. However, the comparability to computer-based learning simulations may not be given, because these training environments also involve motor skills and quick reactions in critical situations. Similarly, we excluded studies that included dummies or actors (O’Meara et al. 2015) and in situ experiments (Esau and Fletcher 2018; Vrzakova et al. 2020). Finally, we excluded papers that were not related to the field of professional training or VET. On this basis, a total of only twelve articles could be included in the analysis.

Inclusion—Objectives of analysis and results

Included studies were analysed based on the domain, sample, task, performance measure, eye-tracking devices, measurements calculated, other behavioural measurements collected, analysis techniques, and main findings.

Results

Descriptive

All studies were published between 2005 and 2022, and seven of the studies were published in the last five years (Fig. 3). Most studies were conducted in the United States (6), followed by the Netherlands (3), Germany (1), Israel (1), and Taiwan (1). The sample size ranged between 7 and 70 participants (M = 36.8, SD = 24.7). The underlying domains are mainly related to the fields of science, technology, engineering, and mathematics (STEM), especially science (Taub et al. 2017; Emerson et al. 2020; Cloude et al. 2020), engineering (van Gog et al. 2005a; Gomes et al. 2013; Abele 2018), but also the healthcare sector (Lee et al. 2019, 2020; Dubovi 2022) is represented (Table 2).

Fig. 3

Included studies per 5-years interval

Table 2 Overview of domains

Usually, students and/or professionals were involved in the studies. Domain-specific complex problem-solving simulations include a broad variety of tasks and findings (see Appendix Tables 4 and 5 for an overview). Typical tasks include problems like finding the cause of a mysterious disease outbreak (Cloude et al. 2020), fixing malfunctions in electrical circuits (van Gog et al. 2005a), or stabilizing a virtual patient by applying a medical routine (Lee et al. 2019). Performance was most often assessed manually (e.g., by counting the errors solved during troubleshooting), and also by examining pre-post-test scores (e.g., assessing content knowledge on microbiology), or through an automated log file analysis (e.g., by calculating a completion score for the underlying task).

Eye-tracking, measures, and analyses techniques

To answer RQ1, we analysed the different eye-tracking measures that were calculated and the most common analysis techniques that were applied. Further, we examined which additional behavioural measurements were collected next to eye-tracking data. Non-intrusive remote (also screen-based) eye-tracking devices were used in most of the studies (see Table 3 for an overview). Remote setups are typical for experimental lab studies, in contrast to eye-tracking glasses for field studies (e.g., Rosengrant et al. 2021) or virtual reality headsets (e.g., Torres et al. 2017). Currently, webcam-based eye-tracking is examined as a low-cost alternative next to remote eye-tracking devices (Wisiecka et al. 2022).

Table 3 Overview of eye-tracking setup, calculated measures, analysis techniques, and other collected measures

Following a functionally operational taxonomy for eye-tracking measures (Holmqvist et al. 2011; Holmqvist and Andersson 2017), four types of measures are distinguished in this review: (1) movement measures: including movement measures of direction, amplitudes, duration, velocity, acceleration, shape, sequences and transitions, and scan path comparison measures; ibid., p.439 ff., (2) position measures: including basic positions, measures of position dispersion, similarity, duration, and dilation; ibid., p. 499 ff., (3) count measures: including a variety of countable entities such as saccades, smooth pursuits, blinks, fixations, dwells, AOIs, transitions, and more; ibid., p. 574 ff., as well as (4) latency and distance measures: including the latency of a saccade, pupil dilation latency, eye-mouse distances, and more; ibid., p. 579 ff.

Movement measures refer to the direction of eye movements and scan paths and are infrequently calculated. In total, three studies examined three different movement measures. Gomes et al. (2013) analysed the direction of saccadic movements (next to position measures) to examine eye movement patterns among high and low performers by applying a common unsupervised machine learning clustering (k-means) using trigrams of eye movements. Kang and Landry (2014) conducted a qualitative scan path analysis to examine whether novices in air traffic control follow a professional scan behaviour after being treated with expert scan path examples. Lee et al. (2019) calculated (next to several positions and count measures) the transition rates between AOIs to examine differences among experts and novices in their performance by applying t-test/Mann–Whitney-U and MANOVA.

Position measures are the most common measures used and refer to the positions where participants look. In total, position measures are calculated 15 times (in 11 of the 12 studies). Abele et al. (2017) measured the total fixation duration on relevant AOIs to analyse differences among performance groups by conducting a nonparametric Mann–Whitney-U test. Similar, Sohn et al. (2005) calculated fixation times to determine group differences. Cloude et al. (2020) calculated proportions of total fixation times to predict performance differences by applying stepwise simple and multiple linear regression models. Similarly, Emerson et al. (2020) integrated positional gaze data next to students’ behavioural traces (such as gameplay behaviour and facial action units) to predict performance and interest groups by running several logistic regression models with different feature compositions. Lee et al. (2020) examined the effects of pausing on the cognitive load within a medical serious game simulation by extracting pupil diameter and applying linear mixed effect models. A multi-level modelling approach was also applied by Taub et al. (2017) including proportions of fixation duration (next to interaction behaviour) to examine differences among performance while accounting for between and within-subject variances (random effects).

Count measures refer to the number and proportions of countable gaze behaviour and are frequently present. Count measures were calculated six times (3 studies) and were most often examined next to similar position measures. Tsai et al. (2016) calculated percentages of fixation counts (next to position measures) to analyse flow experience and visual attention among high and low-performing groups by applying the Mann–Whitney-U test. Also, AOI sequences were used for lag sequential analysis to examine different patterns of visual attention. Van Gog et al. (2005a, b) examined the number of fixations (next to position measures) to examine expertise-related differences by applying Mann–Whitney U, the Friedman test with Nemenyi post-hoc analysis as well as a qualitative data analysis by matching verbal and gaze data.

Finally, latency and distance measures refer to time delays and space distance across eye movements and other points (e.g., mouse cursor). Dubovi (2022) examine latency and distance measures (next to position and count measures) by calculating time to the first fixation and applying ANOVA and linear mixed effect models for group and individual differences as well as regression analysis for performance predictions.

Eye-tracking data can be very ambiguous and is dependent on individual characteristics. To overcome this challenge, researchers are increasingly examining other behavioural data measurements in conjunction with eye-tracking data (Dewan et al. 2019) which can be used for data triangulation. Offline and online measures are frequently used next to eye-tracking (see Table 3). Log files (Gomes et al. 2013; Lee et al. 2019, 2020; Emerson et al. 2020; Cloude et al. 2020) are often collected within computer-based simulations and result in additional and complementary insights into participants' behaviour through mouse clicks and keyboard strokes. Similarly, concurrent think-aloud can help to interpret eye-tracking data through the constant (or retrospective) verbalisation of participants' thoughts (van Gog et al. 2005a). Facial expression recognition (FER) algorithms analyse the expression (Emerson et al. 2020; Dubovi 2022) of anger, disgust, fear, happiness, sadness, surprise, and underlying facial action units (mostly based on the Facial Action Coding System (FACS); Ekman and Friesen 1976). Electrodermal activity (EDA) measures skin conduction as a proxy of psychological or physiological arousal (Dubovi 2022). Additionally, self-report questionnaires are used to measure subjective perceptions (Tsai et al. 2016; Lee et al. 2019, 2020; Emerson et al. 2020; Dubovi 2022).

Main findings

To answer RQ2, we group the main findings related to complex problem-solving in computer-based simulations based on process data measurements.

Most studies analyse differences across performance groups such as high and low performers or expert-novice comparisons and related patterns. Results show strong support for the information-reduction hypothesis following Haider and Frensch (1996; 1999). According to the information reduction hypothesis of Haider and Frensch (1996, 1999), deliberate practice helps learners to ignore redundant information and to focus more on relevant information. Thus, especially experts have learned through practice to distinguish relevant from irrelevant task information (Lee and Anderson 2001) and to use efficient cognitive strategies due to prior experiences (van Merriënboer 2013).

High performers or experts show a longer total fixation time and fewer fixations (Abele et al. 2017), higher proportions of dwell time to total time (with a large effect), a higher ratio of fixation count to total fixation counts (medium effect), and longer fixation duration (large effect) on critically relevant information (Lee et al. 2019). High performers spend more time in a ‘problem orientation’ and ‘action evaluate and next action decision’ phase, they spend more fixations on fault-related components, show shorter mean fixation durations in an ‘orientation’ as well as longer mean fixation durations during a ‘formulation’ phase (van Gog et al. 2005a, b). It is also reported that through practice, less time is spent on relevant and irrelevant areas (Sohn et al. 2005). Similarly, novices performed better (made fewer false alerts), perceived an expert scan path as useful for their training, and tended to follow a professional expert scan pattern after treatment with an expert scan path (showed a circular movement across the air traffic control screen) (Kang and Landry 2014). A combination of a shorter time to first fixation, fewer clicks, more unique fixations, and longer durations per fixation was found for the high-performance cluster (Gomes et al. 2013), while shorter durations for first fixations might indicate higher attentional readiness and indicates more time spent on reasoning before action. A longer time for the first fixation, a higher number of clicks and short fixation durations might indicate a lack of focus on the strategy or a lack of reasoning (trial and error) before action (Gomes et al. 2013).

In line with the results for experts and high performers, low performers or novices show shorter fixation times, more attention to similar but irrelevant AOIs, and lower visual accuracy. Low performers spend a higher proportion of time gathering information and less time generating hypotheses (Cloude et al. 2020). Low performers or novices show shorter or substantially longer fixation time (a behaviour that might indicate confusion) (Abele et al. 2017). The visual attention was spent on similar medicines, indicating processing difficulties through more fixation counts and dwells (Dubovi 2022). The low comprehension group showed higher mental effort (Tsai et al. 2016) and paid more attention to graphic information (while a high comprehension group spent less on the graphical and more attention on the textual information) examined by qualitative heatmap analysis (Tsai et al. 2016).

Individual differences are related to prior knowledge and differences in cognitive load demands. Lower prior knowledge positively moderated the relation between interaction and fixation on gathering information in a serious game, while a negative relation for higher prior knowledge was found (Cloude et al. 2020). Less successful participants tended to get stuck in messages (cues) and an out-of-screen gaze while successful participants tended to transfer the knowledge and might use an out-of-screen gaze for pausing or reasoning (Tsai et al. 2016) to reduce cognitive load demands. Allowing pauses in a medical simulation increases the performance and cognitive load, regardless of whether pauses were taken or not. During pauses, the cognitive load was lower than during the simulation. When pauses were available, taking those pauses did not further benefit cognitive load or performance (Lee et al. 2020). Pupillometry might be a valid measure of the cognitive load next to self-reports (Lee et al. 2020).

Other online and offline process measures for behavioural data shed further light on differences in gaze behaviour when solving problems in computer-based simulations. Self-reports showed that a higher flow time distortion was associated with more fixations on the main task while lower flow time distortion was associated with fixations on the message prompts (Tsai et al. 2016). No significant changes in self-reported affective states over time were reported, while a higher level of presence was related to more visual attention to the relevant medicine (Dubovi 2022). Facial expression recognition (FER) shows no significant impact of joy expression on post-tests, but frequent anger expressions were associated with lower post-test scores and positive emotions were related to inducing blinks (Dubovi 2022). EDA shows a significant correlation between EDA peaks and blinks, but not with participants’ emotional engagement (Dubovi 2022). Eye-tracking data helps to supplement and contextualize log files (Cloude et al. 2020). Experts show higher levels of systematicity (indicated by the HMM score obtained through a log file analysis) (Lee et al. 2019), and a negative effect between the number of books and performance, as well as for the frequency of books and performance were found. The best performance was associated with reading fewer books but higher frequencies per book, emphasizing a quality reading strategy (fewer books more often) compared to a quantity reading (more books) strategy. Also, no unique association between proportions of fixations on book content or book concept matrix with individual submission attempts were found, but a significant interaction effect between both emphasizes the collection and combination of multichannel data. Low proportions of fixations on book content and concept matrices were related to high performance (Taub et al. 2017). Finally, concurrent think-aloud verbal data for high expertise participants show a predictive behaviour while low expertise participants' verbal data show no orientation and an unstructured initial testing approach.

Behavioural data might be further used for performance prediction within multimodal learning analytics. Gaze as a feature (unimodal) or gameplay, and face as a multimodal feature approach yields an accuracy of .67 for prediction among three performance groups, but adding more modalities comes at the cost of noise, so feature selection must be done carefully to avoid overfitting (Emerson et al. 2020). Also, gameplay and face (multimodal) yield .59 accuracy for prediction among three interest level groups (Emerson et al. 2020). The emotional and cognitive engagement measured via multimodal metrics explained 51% of post-test learning achievements (Dubovi 2022). Interestingly, the blink rate is negatively associated with post-test scores and shows significantly lower rates during the actual problem (Dubovi 2022). Significant associations between performance and the multimodal predictors as well as for the interaction term were found (Taub et al. 2017). The highest performance was related to a higher frequency of books, fewer books, and lower proportions of fixations on book content or concept matrix (Taub et al. 2017). Overall, multimodal data channels are very promising for further progress toward individualized learning analytics approaches (Cloude et al. 2020).

Discussion

This scoping review aimed to analyse the current state of eye-tracking research on domain-specific complex problem-solving in authentic tasks within interactive computer-based simulations. A total of twelve studies from a wide range of vocational education and professional training domains were found.

The most commonly calculated measures are position measures, and these are almost exclusively position duration measures such as the proportion of fixation time or total dwell time. Count measures are also mostly related to the number or proportion of fixations and dwell times. Surprisingly, movement measures are rarely computed and usually refer to saccade directions or scan path. Heatmaps or scan paths are often qualitatively compared. There is a lack of quantitative approaches for measuring time patterns or similarity measurements for a scan path as stated by Holmqvist et al. (2011, 2017). Also, latency and distance measures are almost never calculated. This indicates that the potential to shed further light on complex problem-solving in computer-based simulation might not yet be fully exhausted by calculating other than the standard count and position duration eye-tracking measurements. The much broader variety of potential eye-tracking measures (concerning the underlying specific research questions) should be taken into account (Holmqvist et al. 2011; Holmqvist and Andersson 2017). For example, cognitive load might be measured by considering saccadic peak velocity (Di Stasi et al. 2011), time to first fixations might be an indicator for visual attention to cues and hints in serious games (Conati et al. 2013), and saccade paths (Wu et al. 2014) might be calculated for further insights into behavioural differences and performance predictions.

To analyse eye-tracking data, group comparisons between experts and novices or high-performing and low-performing groups are often computed using common statistical methods such as t-test, (M)ANOVA, or non-parametric Mann–Whitney-U. Patterns between groups are examined with heat maps and lag sequential analyses, by mostly using discrete behaviour codes, or common k-means for clustering purposes. Recently, an increasing number of researchers have focused on individual differences in addition to the group level to account for random effects by applying linear mixed-effects models. This is relevant for eye-tracking research since eye movement data can vary between and within participants over time. The emphasis on the application of mixed-effects models in reading research to analyse eye-tracking data (Catrysse et al. 2018) shows further potential for generalizing results of between-group comparisons while accounting for within-subject variances, and additionally, increases the power of statistical analyses (compared to common approaches such as ANOVA) when conducted for lower aggregated levels (Baayen et al. 2008; Quené and van den Bergh 2008; Catrysse et al. 2018). Finally, post-hoc performance predictions are the first attempts to develop multimodal learning analytics. However, these performance predictions are often performed as subsequent machine learning regression analyses, most of the reported accuracy scores seem to be currently not suited for practical implementation and are currently not integrated within the computer-based simulations for real-time assessments. Research out of the field of decision support systems (Causse et al. 2019) shows promising results to improve performance support.

Similarly, using multimodal data channels seems promising for educational purposes by integrating eye-tracking into systems of multimodal learning analytics (Cloude et al. 2022). Insights from additional questionnaires, think-aloud protocols, log files, and other psychophysiological measures have proven valuable next to eye-tracking data. Eye-tracking data combined with log file analysis and think-aloud protocols might be useful to validate each other and reveal further information about problem-solving processes (van Gog et al. 2005a, 2005b; Stieff et al. 2011). Interestingly, Taub et al. (2017) found significant effects for the interaction term between gaze and log file data. They emphasize the use of multimodal data by stating “that our most significant results were those that included online trace data from both log files and eye tracking” (Taub et al. 2017, p. 651). In many cases, self-reporting is used as an additional measurement for data triangulation. In addition to eye-tracking, log files and facial expression recognition algorithms are also used. However, few studies use shimmer devices to detect electrodermal activity or practice concurrent thinking aloud. Studies on psychophysiological measures have shown to be valid indicators for problem-solving performance-related measures such as stress (Kärner et al. 2018). Self-reports and log files are useful tools for data triangulation. However, sometimes changes in affective state are not consciously perceived and reported through self-reports but can be measured through facial expression recognition algorithms, as reported by Dubovi (2022). Also, log files show higher systematicity for expert behaviour through HMM scores introduced by Lee et al. (2019). To obtain these systematicity scores, a rigorous task analysis must be performed before computation. Overall, despite the rise of multimodal approaches, the recognition of facial expressions using algorithms, measuring electrodermal activity using shimmer devices, and concurrent (or retrospective, for a comparison see van Gog et al. 2005b) thinking aloud are rare in this sample and data synchronisation remains a challenging aspect of research when data is not collected within a single software, which is not always possible (e.g., when log files of educational data are protected on a separate and secured server).

According to the “information reduction hypothesis” of Haider and Frensch (1996, 1999), deliberate practice helps learners to ignore redundant information and to focus more on relevant information. Thus, experts (and high performers) have learned through practice to distinguish relevant from irrelevant task information (Lee and Anderson 2001) and to use efficient cognitive strategies through experience (van Merriënboer 2013). This is also indicated by many studies in the sample. Similarly, low-performers or novices show shorter fixation times, more attention to similar but irrelevant AOIs, and lower visual accuracy. Performance in computer-based simulations and problem-solving seems to be moderated by prior knowledge, which positively influences the interaction between simulations and information fixation. Lower prior knowledge relates to lower performance and more fiddling around (trial and error strategy). The effects of pausing in simulations (for the medical field) were found to increase performance, whether those pauses were taken or not. Also, successful problem solvers tended to take knowledge from cues when they were given, which could be related to the use of an out-of-screen gaze to pause or think, while unsuccessful participants got stuck in a loop between reading cues and an out-of-screen gaze.

Recently, some work has been done on the post hoc analysis of multimodal features such as eye-tracking and facial expression recognition data as well as log data for performance prediction. Interestingly, Emerson et al. (2020) state that that using more features for prediction comes at the cost of integrating more noise into the prediction, sometimes making a model’s performance worse by overfitting. Some regression-based approaches seem promising and could explain up to 67% of the total variance. Nevertheless, it is difficult to determine machine learning performance by metrics such as accuracy. Especially for the more typical unbalanced datasets, other evaluation metrics such as F1 scores (the harmonic mean of precision and recall) are typically reported in machine learning research.

A major limitation of any literature review is publication bias. By addressing more than one database and a broad search term, as well as additional reference checks, we attempted to challenge publication bias appropriately. Despite these efforts, it is still possible that there is literature available but was not found. Though a review must aim to be all-inclusive, it may not always be possible. A single researcher performed most of the selection procedures. However, acceptable kappa values were calculated for a small number of studies between the two independent coders. Generalisability is not provided over a broad range of domains and tasks. There are shortcomings in the representation of countries and samples. Studies from Western countries are mainly represented within this sample. Also, we want to underline that high performers and experts are not the same (performance-based vs. criteria-based selection). A major shortcoming of this review is the limited number of studies analysed. Thus, we stick to a narrative scoping review but can give no information about the overall statistical effect due to sample restrictions and the heterogeneity in terms of study designs and dependent variables.

Future research might conduct a more systematic review and meta-analysis, particularly on the relationship between performance differences and eye movement measures. So far, within this specific subfield of interest, not enough studies were conducted and published to further examine such relationships. Thus, one advantage of this review is that we show the diversity of eye-tracking as a data collection method as well as different analysis techniques to foster eye-tracking research for VET domains, where computer-based simulations gain increasing relevance for education. This review supports eye-tracking as a data collection method for studying behavioural patterns in learning processes. Further studies should collect experiences during problem-solving processes (Rausch et al. 2019) and learning-related emotions by examining affective states through facial expression recognition (Munshi et al. 2020). Finally, there is a general research gap for eye-tracking studies and behavioural analysis in vocational education and training, and more precisely a vast lack of studies in the field of business education.

Appendix

See Tables 4, 5

Table 4 Descriptive overview of included studies

Table 5 Overview of research questions/hypotheses and main findings