Multidisciplinary digital competencies of pre-service vocational teachers

Abstract

Developments of Industry 4.0 require a set of multidisciplinary digital competencies for future vocational teachers, consisting of specific knowledge, motivational aspects, cognitive abilities and skills to fulfill the demands of digitally interconnected work situations. The competence model that is adapted from future work scenarios of vocational apprentices in Industry 4.0 includes attitudes towards digitization and handling of digital devices, information literacy, application of digital security standards, virtual collaboration, digital problem solving as well as a demonstration of reflective judgment of one’s actions in an interconnected and digital environment. Structural equation modeling was used to assess N = 205 pre-service vocational teachers between 18 and 35 years of age. The findings indicate the relationship of the proposed dimensions, measured through external- and self-assessments validate the proposed structure of the multidisciplinary digital competencies. However, attitude towards digitization can predict the self-efficacy of the relevant Multidisciplinary Digital Competencies but not the actual achievement in an external assessed scenario. Nevertheless, this study confirms that self-assessed multidisciplinary digital competencies can predict achievement in an external and qualitative-assessed competence test. Fit indices show an acceptable model conception, the reliability and construct validity of the model were confirmed. Findings suggest that the attitude towards digitization and the application of digital security standards are important, whereas the ability to solve digital problems seems to have a weak relation to the general multidisciplinary digital competencies of pre-service vocational teachers.

Introduction

The current digital changes in the industrial world of work are often referred to as the “fourth industrial revolution” or Industry 4.0 (Acatech 2016; Mertens et al. 2017). While originally this was a German particularity, research on the topic is constantly growing in many different fields worldwide and is no longer limited to German industry (Belinski et al. 2020; Liao et al. 2017). In this paper, Industry 4.0 is understood as the horizontally and vertically interconnected digitization of entire industrial value chains. This includes the real-time data exchange between customers, employees, objects and systems via cyber-physical systems (CPS). This turns industrial value chains into value-adding networks (Gebhardt et al. 2015; Kagermann et al. 2013). Within these value-adding network, smart products, which carry operational data for their own individual building plans (Weyer et al. 2015), communicate with self-organized and decentralized cyber-physical production systems (CPPS). The resulting smart production lines enable batch size 1 production with maximum cost efficiency (Wilbers 2017). This interconnection facilitates adaptation to spontaneous changes in the environment (Hecklau et al. 2016). A particularity of working in Industry 4.0 is that the contents of work change due to the interconnection, the real-time transmission of data due to CPS and the increasing automation of production.

Even if most studies on Industry 4.0 are still visionary or conceptual in nature (Veile et al. 2019), publications already focus on the resulting and necessary changes of occupational structures and work activities as well as additional requirements for employees, i.e., digital competencies (Gronau et al. 2017; Hecklau et al. 2016; Hummel et al. 2015; Tisch and Metternich 2017), such as procurement of information and holistic thinking ability (Spöttl et al. 2016). There are already numerous studies on digital competencies (Carretero et al. 2017; Ferrari, 2013; Fraillon et al. 2013, 2019; Ilomäki et al. 2016) and twenty-first century skills (Ananiadou and Claro, 2009; Siddiq et al. 2016; van Laar et al. 2017, 2020). However, few studies have been conducted on the digital competencies of technical vocational students (Seufert 2020; Spöttl et al. 2016; Tenberg and Pittich 2017). These often have one thing in common: in the mostly explorative approaches, the pool of experts is very broad and stakeholders who are not directly involved in the training are often interviewed. This somewhat dilutes the statements of the respective studies (Hambach et al. 2017; Spath et al. 2013; Ziegler and Tenberg 2020). Based on two previous exploratory studies which focused on German corporate instructors and (technical) vocational teachers regarding required multidisciplinary digital competencies for future technical vocational students, Roll and Ifenthaler (2020a) suggest that multidisciplinary digital competencies consist of specific knowledge, motivational aspects, cognitive abilities and skills, which technical vocational students in Germany require in order to be prepared for the upcoming changes of Industry 4.0. The multidisciplinarity identified in these two exploratory studies is based on the findings of Tenberg (2020) and Walker et al. (2016) that, due to the interconnectedness of Industry 4.0, it is important in the various training professions to possess multidimensional digital and multidisciplinary skills and abilities that are not purely professional or purely technological to in order to face multidisciplinary problems. Accordingly, multidisciplinary digital competencies should be understood from a dispositional perspective in which they are the sum of various motivational and cognitive competence dimensions (Blömeke et al. 2015).

Teaching all kinds of Industry 4.0-related competencies in the twenty-first century requires correspondingly competent teachers (Instefjord and Munthe 2017; Maderick et al. 2015; Rubach and Lazarides 2019) and therefore a change in the competencies of technical vocational students has implications for all educational stakeholders in the vocational and educational training (VET) system (Sloane 2019). For example, school development will change because teachers must ensure that lessons are developed in such a way that the current state of digitization is discussed and the corresponding digital competencies are developed alongside current subject-related ones (Seufert et al. 2018).

In addition to the subject-related changes that Industry 4.0 brings to technical and commercial vocational school teachers, new adjustments of multidisciplinary perspectives are also highly relevant for general vocational teaching (Kutscha 2017). Technical vocational students must also be prepared in schools for the multidisciplinary challenges of Industry 4.0 (Wittmann and Weyland 2020). In addition to subject-related competencies, research is increasingly showing that all kinds of digital competencies such as dealing with IT security (Sîmandl et al. 2017) or Information Literacy (Scherer et al. 2017) are becoming important in the vocational classroom (Seufert 2020). However, there is a need to address this issue, because the training of vocational school teachers is not systematically prepared for developing such competencies, and there is no empirical evidence describing the level of multidisciplinary digital competencies of vocational teachers (Gössling et al. 2020; Tenberg 2020).

Self-assessment is a legitimate instrument to obtain information about the multidisciplinary digital competencies of pre-service vocational teachers in a resource-efficient manner (Calvani et al. 2008). But multiple or single choice questions are not fitting for every dimension of a multidisciplinary digital competencies framework (Calvani et al. 2008). Self-assessment in general can only measure competence indirectly (Meritt et al. 2005) and studies show that especially with regard to computer-related competencies, individuals tend to overestimate their skills and abilities (Ihme and Senkbeil 2017). However, self-assessment can determine self-efficacy by asking for the participants’ own subjective assessment of whether they are capable of dealing with specific situations (Bandura 1982).

Another measurement technique requires observation of a large group over a longer time (Calvani et al. 2008) or a qualitative approach via semi-structured interviews (Lundkvist and Gustavsson 2018). In contrast to Maderick et al. (2015) and their objective assessment via multiple choice questions, this study provides a qualitative approach to assessing pre-service vocational teachers’ multidisciplinary digital competencies. Guzmán-Simón et al. (2017) recommend a qualitative measurement approach to provide further insights into obtained quantitative data. Accordingly, the focus of the study is to investigate the paths of the various competence dimensions of multidisciplinary digital competencies identified in an exploratory study (Roll and Ifenthaler 2020a) among pre-service vocational school teachers through self- and external qualitative assessment. Specifically, this study has three aims: (1) to validate the proposed structure of the multidisciplinary digital competencies; (2) to examine the influence of attitude towards digitization on self- and externally assessed multidisciplinary digital competencies; and (3) to investigate the prediction of the externally and qualitatively assessed multidisciplinary digital competencies through self-assessment of multidisciplinary digital competencies.

Theoretical framework

Given the numerous concepts of competencies (e.g. Ferrari 2013; Ilomäki et al. 2011), literacies (e.g. Fraillon et al. 2013; Meyers et al. 2013; Pettersson 2017) and 21st century skills (van Laar et al. 2017) in the digital context, the variety of meanings can be irritating (Ilomäki et al. 2016; Pettersson 2017; Weinert 2001). A specific research stream involves the models dealing with the integration of technology in the classroom that examine when, why, how and with what quality teachers integrate digital technologies into their lessons – for example, the will, skill, (access to technological) tool model, which the authors expanded by a pedagogical dimension in 2016 (Knezek and Christensen 2016). It shows how teachers’ attitudes towards digital technologies, skills and technological equipment determine whether or not they integrate digital technologies into their lessons (Petko 2012). Another model for examining pedagogical, content and technological knowledge is the TPACK model by Mishra and Köhler (Schmid et al. 2020; Tondeur et al. 2020). It can be used as a basis, but taking into consideration that the present model does not explicitly ask for content knowledge, but for multidisciplinary competencies. Furthermore, this study is not intended to ask whether and how pre-service teachers have a pedagogical understanding of how to teach this interdisciplinary and technical content (Mishra and Koehler 2006).

The dispositional sum of multidisciplinary digital competencies, which is presented in the following, can be located in the competence dimension of technological knowledge in the TPACK model (Koehler et al. 2014). In the following, however, the focus is on whether pre-service vocational teachers also have the multidisciplinary digital competencies that are required of technical vocational students in Industry 4.0. Multidisciplinary digital competencies are thought of as competencies that orientate themselves on the changing work environments brought about by the implementation of Industry 4.0 (Gebhardt et al. 2015; Ifenthaler 2018; Sommer 2015; Veile et al. 2019). Multidisciplinary digital competencies specifically address the “necessary and sufficient conditions” of “tasks, goal and success criteria” (Weinert 2001, p. 51) related to Industry 4.0.

Multidisciplinary digital competencies, however, do not stand in contradiction with a domain-specific focus (Weinert 2001). Domain-specific skills, abilities and knowledge are crucial for solving complex problems within a specific domain (Weinert 2001). The focus of multidisciplinary digital competencies lies in the demanding problems of an interconnected Industry 4.0 work environment. Therefore, multidisciplinary digital competencies affect all professions that have to deal with Industry 4.0, regardless of their main discipline. Besides, if domain is described as a “universe of tasks and responses” (Shavelson 2010, p. 46) the universe of Industry 4.0 and its specific digital tasks marks multidisciplinary digital competencies as a domain-specific competence model. Vocational teachers require knowledge, abilities and skills in their specific subject domains (Rausch and Wuttke 2016). Considering the change of work requirements through Industry 4.0 (Ifenthaler 2018) pre-service vocational teachers should have digital competencies that are not only bound to their subject but are multidisciplinary (Roll and Ifenthaler 2020b). Based on the statements of corporate instructors, multidisciplinary digital competencies combine specific dimensions of several digital competencies or digital literacy frameworks to ensure that an individual has the willingness, abilities and skills to behave adequately, individually and socially responsibly in the digital context of professional, social and private situations (Roll and Ifenthaler 2020a, b). Figure 1 shows the seven dimensions of multidisciplinary digital competencies: (1) attitude towards digitization, (2) handling of digital devices, (3) Information Literacy, (4) application of digital security, (5) collaboration due to digital communication, (6) solving of digital problems and (7) reflection on the interconnected and digital environment.

Attitude towards digitization

The interviewed corporate instructors in Roll and Ifenthaler (2020a) emphasized a positive attitude towards learning and working with digital devices as essential for all kinds of work in Industry 4.0 (Roll and Ifenthaler 2020a). Ferrari (2012) indicates that an integration of attitude is the key difference between a digital literacy framework and digital competencies models. Weinert (2001) describes attitude as the motivational, volitional and social willingness to act. For example, a positive attitude towards digitization can foster the self-efficacy beliefs of pre-service vocational teachers for technology integration in the classroom (Farjon et al. 2019; Lee and Lee 2014; van Braak et al. 2004). Knezek and Christensen (2016) identified willingness as the greatest predictor in their will, skill, tool model. Therefore, attitude towards digitization (AD) is a predictor for this suggested theoretical framework of multidisciplinary digital competencies.

Handling of digital devices

Pre-service vocational teachers who want to integrate digital devices in the classroom need to know how to deal with the associated daily challenges (Koehler et al. 2014). Ilomäki et al. (2016) state that “technology-oriented terms describing general competences are diminishing in research papers” (Ilomäki et al. 2016, p. 668), but the handling of digital devices (HD) is still fundamental to models of digital competencies (Selwyn and Husen 2010) due to the growing number of portable digital devices (Delcker et al. 2016; Martin and Ertzberger 2013) This competence dimension includes both the handling of physical devices and the efficient use of corresponding software (Johnson et al. 2006; Roll and Ifenthaler 2020a). The term “handling” emphasizes the action-oriented and practical use of digital devices (Calvani et al. 2012). Therefore handling of digital devices includes skills such as “basic computer operations, email, Internet, word processing programmes and presentation programmes” from “technology proficiency” (Mah and Ifenthaler 2018, p. 122). But handling of digital devices does not involve programming skills (Fraillon et al. 2013).

Information literacy

Based on the concept of the future “knowledge worker” (Tenberg and Pittich 2017), it is important from a company trainer’s perspective to have a certain degree of Information Literacy (IL) (Roll and Ifenthaler 2020a). Information Literacy refers to accessing, analysing, evaluating and adequately communicating information (Fraillon et al. 2013, 2014, 2019). In contrast to media literacy, Information Literacy focuses on the procedural knowledge of managing information from static texts rather than understanding accordingly edited information (Fraillon et al. 2013, p. 17). Due to the large amount of information available, and to the fact that it can be incorrect, it is increasingly challenging for an individual to evaluate the authenticity, reliability and validity of information (Bundy 2004). That is why using information responsibly and safely is part of Information Literacy (Fraillon et al. 2019) and part of the multidisciplinary digital competencies framework.

Application of digital security

In addition to Information Literacy, the application of digital security (DS) is usually a main dimension of digital competencies frameworks (Carretero et al. 2017; Ferrari 2013; Vuorikari et al. 2016). Sommer (2015) identified the mishandling of data security issues as a major problem in Industry 4.0, especially for small and medium-sized enterprises (SME). Corporate instructors also added that this refers not just to the work, but also to the carefree private presence on the Internet of young people (Roll and Ifenthaler 2020a). It is becoming increasingly important to learn about application of digital security in school to develop adequate skills in this competence dimension (Fraillon et al. 2019; Sîmandl et al. 2017; Šimandl and Vaníček 2017). As a result, employees and teachers should know how to apply digital security measures (Šimandl and Vaníček 2017). Application of digital security deals, for example, with the impact of malware, the simple securing of digital devices and networks, creation of safe passwords, identity theft, risks of digital communication (e.g. phishing emails), and sharing private and work-related information (Sîmandl et al. 2017).

Virtual collaboration due to digital communication

The more devices are integrated in daily routine, the more popular virtual collaboration (CL) with digital communication devices becomes (Carretero et al. 2017; Ferrari 2013). Adequate digital communication is often directly linked to the manner and rules of virtual collaboration. Therefore collaboration implies skills in communicating via digital devices, exchanging information and negotiating with mutual respect (van Laar et al. 2017). This affects the choice of suitable communication tools for specific situations (private vs. work-related situations) and an appropriate verbal diction towards achieving a common goal. Corporate instructors, interviewed by Roll and Ifenthaler (2020a), claimed that vocational teachers should integrate virtual communication habits in the classroom in order to develop their students’ multidisciplinary digital competencies.

Solving of digital problems

Interconnectivity through Industry 4.0 often helps in decision making processes through “generating, collecting, and processing required information” (Abdel-Basset et al. 2019, p. 2). However, with growing interconnectivity the complexity increases (Arnold et al. 2017). Complex problem solving is described as crucial for the 21th century (Eseryel et al. 2011). The corporate instructors added that it is certainly not a new requirement calling for better problem solving skills, but in an increasingly networked world, such skills are of great importance (Roll and Ifenthaler 2020a). As a result, in order to be digitally competent teachers need skills in digital problem solving (PS) within the digital and interconnected context (Grzybowska and Łupicka 2017; Müller et al. 2018). Therefore, structuring and planning a strategy to solve digital problems is required. In addition, it also requires Information Literacy skills, such as comparing, evaluating and selecting information from the current problem (Grzybowska and Łupicka 2017).

Self-reflecting on the digital environment

The digital environment is becoming increasingly complex due to Industry 4.0 and the interconnection of CPS in private, work and educational situations (Arnold et al. 2017). It is crucial to understand the consequences of ones’ own digital actions and to self-reflect about one’s actions in the interconnected and digital environment (RF) (Roll and Ifenthaler 2020a). Reflection is systematic and ensures a continuity of learning (Lin et al. 2014). Therefore reflection within the interconnected and digital context of Industry 4.0 affects the individual attitude towards digitization (Ferrari 2012) and the development of multidisciplinary digital competencies in general. Chen, Kinshuk, Wei and Liu (2011) argue that reflection skills are crucial for gathering and evaluating new information. Following Dewey (1910), Rodgers (2002) interprets reflection as a meaning-making process that encourages a deeper understanding of the respective content and its consequences. As a result, RF includes the ability to reflect on one’s own actions within an interconnected world. This includes actions in private situations, such as sharing personal photos, but also affects the individual’s workplace. However, RF implies an understanding of the consequences of the specific working steps within a supply chain network.

Research questions and hypotheses

The three aims of this study are (1) to validate the proposed structure of the multidisciplinary digital competencies, (2) to examine the influence of attitude towards digitization on self- and externally assessed multidisciplinary digital competencies, and (3) to validate the prediction of the external and qualitative multidisciplinary digital competencies assessment through the self-assessment of multidisciplinary digital competencies. The first research objective focuses on the proposed structure and related model fit of the multidisciplinary digital competencies model of Roll and Ifenthaler (2020a). To develop students’ multidisciplinary digital competencies, vocational teachers must also possess the corresponding multidisciplinary digital competencies (Maderick et al. 2015). Therefore, the first research question aims to validate the influences of variables shown in Fig. 1. Accordingly, it is assumed that the theoretical dimensional structure of multidisciplinary digital competencies can be confirmed in this study (Hypothesis 1).

Fig. 1

Dimensions of multidisciplinary digital competencies

The second research objective focuses on the effect of attitude towards digitization on multidisciplinary digital competencies for pre-service vocational teachers (Petko 2012). The literature indicated a significant relationship between attitude towards digitization and self-assessed digital competencies (Bunz et al. 2007; Lee and Lee 2014; Pamuk and Peker 2009; Scherer et al. 2017; Wu and Tsai 2006). Yerdelen-Damar et al. (2017) found that pre-service teachers’ attitudes towards the use of digital devices had a direct influence (β = 0.20) on their self-efficacy beliefs in terms of the technological pedagogical content knowledge (TPACK-S). Other studies have confirmed a relationship between attitude towards digitization and self-efficacy (Prior et al. 2016), but focused less on the relationship between attitude towards digitization and actual performance. In a study by Aesaert et al. (2015), the authors neglect the relationship between these two variables. Bunz et al. (2007), meanwhile, found an influence of attitude on self-efficacy, but no relationship between attitude and performance. While Aesaert et al. (2015) clearly had younger participants in their study, the participants examined by Bunz et al. (2007) were first-year university students. In the following, based on the advanced studies and experience of the participants, the authors assume that the attitudes towards digitization also have an influence on multidisciplinary digital competencies. Accordingly, it is expected that attitude towards digitization (AD) influences the self-assessed multidisciplinary digital competencies (Hypothesis 2a) because attitude can be a predictor of self-assessed competencies (Yerdelen-Damar et al. 2017). And since attitude toward digitization is a driver for the use of digital media in education (Rubach and Lazarides 2019), it is expected that attitude towards digitization (AD) influences the external and qualitatively assessed multidisciplinary digital competencies (Hypothesis 2b).

The third research objective focuses on how self-assessment may predict the score of the external and qualitative assessment of multidisciplinary digital competencies. This would be indicated by an effect of self-assessed multidisciplinary digital competencies (SAMDC) on the achievement in QAMDC. The meta study by Multon et al. (1991) shows that self-assessed competencies correlate with actual performance. In addition, Hatlevik et al. (2015) found that self-efficacy predicts actual digital competencies. Due to the central assumptions of self-efficacy and performance, the third research question investigates the prediction on the actual performance of multidisciplinary digital competencies, externally and qualitatively assessed, through the self-assessment of multidisciplinary digital competencies (Bandura 1986; Pajares 1996) of pre-service vocational teachers. If pedagogical content knowledge positively influences the quality of teaching (Backfisch et al. 2020), then one can expect that the self-assessed multidisciplinary digital competencies, as specific advanced technological knowledge of the TPACK model, influence the quality of the actually displayed multidisciplinary digital competencies. Hence, following the findings of Hatlevik et al. (2015), it is assumed that self-assessed digital multidisciplinary competencies (SAMDC) can predict achievement in externally and qualitatively assessed multidisciplinary digital competencies (QAMDC) positively (Hypothesis 3).

Method

Participants

The participants of this study were N = 222 students of business and economic education at a European university. Seventeen were deleted from the dataset because they rushed through the online instruments. The critical threshold of minimum time needed to answer all questions truthfully was set at 25 min before the study (M = 40.48; SD = 9.02). Participants were between 18 and 35 years old (M = 22.78; SD = 2.89; 64.9% female; 74.1% undergraduates). At N = 205, the rule of thumb for critical CN = 200 is just exceeded. Exceeding CN indicates that its particular structure equation model could adequately reproduce an observed covariance structure (Bagozzi and Yi 2012; Hoelter 1981; Kline 2015). Of these 205 participants, 48 (23.49%) completed vocational training themselves prior to their studies to become vocational teachers. While business and economics are the major subjects within their study programme, the pre-service vocational teachers also have to choose a second subject to subsequently teach in schools (see Table 1). They usually choose their second subject in the fifth semester of their bachelor course, thus explaining why, at the time of this study, 58.5% had not yet chosen a subject. Sixty-one students were in the second semester, three in their third, and fifty-six participants stated that they were currently in the fourth semester and therefore had not had the chance to choose a second subject.

Table 1 Minor subjects of participating pre-service vocational teachers

Instruments and procedure

In order to measure the declarative and procedural knowledge of the competence dimensions, measuring instruments were developed to measure the respective constructs through self-assessment as well as through scaling of responses to scenario-based tasks (Blömeke et al. 2015). The participants were invited to a computer room, where they received instructions via a presentation and a link to a website, which led to the two instruments—first a self-assessment questionnaire, and then a survey with open questions. Figure 2 shows the second order measurement model including the two instruments SAMDC and QAMDC and how the dimensions of multidisciplinary digital competencies influence them.

Fig. 2

Measurement model of self- and externally and qualitatively assessed multidisciplinary digital competencies

Self-Assessed multidisciplinary digital competencies (SAMDC)

SAMDC is an instrument for measuring the self-efficacy of the several dimensions of multidisciplinary digital competencies. Therefore, it contains seven latent variables, which are presented in Table 2. Each of these consists of four items, which were measured through self-assessed five-point Likert scales. Within SAMDC the participants had the options “strongly agree”, “agree”, “neutral”, “disagree” or “strongly disagree” with regard to the given statements. The reliability and the descriptive statistics are shown in Table 2. Cronbach’s alpha indicates that the internal consistency is acceptable (α ≥ 0.7) for all dimensions used within the model except self-reflection (Bagozzi and Yi 2012), Cronbach’s alpha of RF was not improved by removing any items. However, because SAMDC measures the individual’s own judgement on its multidisciplinary digital competencies, the construction of the latent variable SAMDC made the integration of self-reflection obsolete, because SAMDC already reflects the judgment as a self-efficacy variable (Bandura 1986). This follows Roll and Ifenthaler's (2020b) exploratory study, in which vocational teachers ranked self-reflection as less important than the other multidisciplinary digital competencies. The self-assessed constructs are marked in the following with an “S” in front of the actual construct abbreviation (for example, the self-assessment for the competence dimension handling of digital devices (HD) is called SHD).

Table 2 Summary of self-assessed scales

Externally and qualitatively assessed multidisciplinary digital competencies (QAMDC)

The dimensions of multidisciplinary digital competencies were rated through qualified and trained researchers following criteria of qualitative content analyses. Attitude towards digitization and RF were not integrated into external and qualitative assessment of multidisciplinary digital competencies, because self-assessment seemed to be an adequate method of evaluating these two dimensions (Grant et al. 2002; Richter et al. 2000). QAMDC is designed as a fictive general work scenario. Within this problem setting, participants are asked to imagine that they work in a small to medium sized enterprise in the production sector, which is financially limited but it is under pressure to digitize processes. The participants are employees for the administration and have several tasks to deal with, such as procurement, human resources and marketing. Their supervisor has asked for a presentation about “Industry 4.0 and opportunities” for the firm. This scenario was divided into several questions and tasks, which were adapted from van Deursen and Van Dijk (2010) and created on the base of the particular examples of the DigComp 2.1 framework by Carretero et al. (2017):

• QHD—including what basic and advanced software would be helpful to solve the specific tasks (QBasc and QAdvc),

• QIL—explaining the strategy of searching for, structuring (QIorg) and evaluating information from the internet (QIeva),

• QDS—questioning how to handle upcoming security threats within the scenario (QShnd),

• QCL—questioning how you would collaborate via digital devices with

  a) your new supervisor and

  b) and old friend of yours, in terms of communication tools and rules (QCL).

• QPS—explaining how to deal with upcoming routine/well-structured problems, such as a sudden dysfunctional Internet connection (QProu), and writing down strategies for solving further complex ill-structured problem settings (QPcrt) (Seel et al. 2009).

QAMDC is based on a qualitative research approach, where participants had to answer open-ended survey questions (Hsieh and Shannon 2005) that were directly related to the given scenario. Responses to these questions were clustered by three qualified raters into a five-point Likert scale. The criteria to assess the responses were pre-tested and defined within a workshop. The constructs evaluated in this way are marked in the following with a “Q” before the actual construct abbreviation (for example, the self-assessment for the competence handling of digital devices (HD) is called QHD). Table 3 shows the interrater reliability and a summary of the assessed tasks. The interrater reliability, calculated via the Intraclass correlation (ICC3,1), demonstrated the two-way mixed consistency (Shrout and Fleiss 1979) of the three raters. Therefore, the scores in QAMDC can be compared with the self-assessed scores of the latent variables in SAMDC.

Table 3 Intraclass correlation (ICC3.1) and summary of externally and qualitatively assessed competence dimensions

Analytic strategy

To validate hypothesis 1, structural equation modelling (SEM) was used to evaluate the relations of the dimensions in SAMDC and QAMDC. SEM can test direct effects between constructs. This was used to validate Hypothesis 2a and Hypothesis 2b, because the applied SEM also contains the regression analysis for the influence of attitude towards digitization on SAMDC and QAMDC. The measurement model integrates the internal consistency within the dimensions of SAMDC and the assessments of the three qualified raters of QAMDC. By integrating all covariations and influences of the relevant dimensions, the SEM used shows the direct effect of SAMDC on QAMDC (Hypothesis 3). Due to the lack of normal distribution (Table 2), for the final SEM (with SAMDC and QAMDC) the robust maximum likelihood estimator (MLR) was used and adjusted through the Yuan-Bentler correction. Due to the settings of the applied self-programmed online survey tools, participants could not finish SAMDC and QAMDC if boxes had been left blank. Therefore, there was no missing data to deal with. To analyse the model of Fig. 2, the statistics software R (version 3.6.1), R-Studio (version 1.1.463) and the R-package lavaan (version 0.6–7) were used (Rosseel 2019; Steinmetz 2015).

Results

Confirmation of the dimensional structure (Hypothesis 1)

An adequate model fit is essential to confirm that the model properly represents the data (Hooper et al. 2008). The SEM of Fig. 2 shows a fit of χ² (846, N = 205) = 1105.378, p = 0.000, Comparative Fit Index (CFI) = 0.938, Tucker Lewis Index (TLI) = 0.934, Root Mean Square Error of Approximation (RMSEA) = 0.039 and Standardized Root Mean Square Residual (SRMR) = 0.071. According to Bagozzi and Yi (2012), one can reduce the stringent cut off rules that CFI and TLI are ≥ 0.95 to CFI ≥ 0.93 and TLI ≥ 0.92 if SRMR ≤ 0.07. Considering the unsatisfactory internal consistency of RF (see Table 2) and the fact that SAMDC measures the individual’s judgment of their multidisciplinary digital competencies (Bandura 1986), it was decided to modify the model and not integrate RF (Grant et al. 2002) any further. Overall, the structural equation model now shows a good fit of χ² (689, N = 205) = 863.001, p < 0.001, CFI = 0.956, TLI = 0.952, RMSEA = 0.035, SRMR = 0.068. To examine the relations within the model in Fig. 3, Table 4 shows the path estimates of the structural model. Hypothesis 1 is accepted because the significant path estimates and the fit indices confirm the theoretical dimensional structure of multidisciplinary digital competencies.

Fig. 3

Structural model and influence of attitude towards digitization on the dimensional structure of self- and externally and qualitatively assessed multidisciplinary digital competencies (p < .01)

Table 4 Path estimates of the structural model

Effects of attitude towards digitization on self- and externally and qualitatively assessed multidisciplinary digital competencies

Hypothesis 2a explores the relationship between pre-service vocational teachers’ attitudes towards digitization and their self-assessed of multidisciplinary digital competencies, which was analysed through structural equation modelling. Figure 3 shows a medium standard regression weight of attitude towards digitization to SAMDC and Table 4 confirms its significance at a more conservative level (β = 0.5, p < 0.000). Therefore, hypothesis 2a is accepted.

Hypothesis 2b investigates the effects of pre-service vocational teachers’ attitudes towards digitization and their achievement in QAMDC, which was also analysed using structural equation modelling. Figure 3 and Table 4 present a small but not significant standardized regression weight of attitude towards digitization towards QAMDC (β = 0.24, p = 0.174). Hypothesis 2b is rejected because of its level of significance of p > 0.05.

Prediction of external and qualitative assessment of multidisciplinary digital competencies through self-assessment

Hypothesis 3 examines whether the self-assessed digital multidisciplinary competencies (SAMDC) can positively predict achievement in externally and qualitatively assessed multidisciplinary digital competencies (QAMDC). As can be seen in Fig. 3 and Table 4, SAMDC significantly predicts the achievement in QAMDC (β = 0.55, p = 0.015). Therefore, Hypothesis 3 is accepted because p < 0.05.

When exploring dependencies in a structural equation model one has to be aware of the adverse effects of multicollinearity (Mansfield and Helms 1982). To test the model on multicollinearity the variance inflation factors (VIF) of each predicting variable were calculated (Mansfield and Helms 1982). A maximum of 1.53 of the VIFs meet the—surely debatable (O’Brien 2007; Thompson et al. 2017)—rule of thumb that they must be less than 4 (O’Brien 2007).

Discussion

Based on the fact that no significant differences regarding SAMDC and QAMDC were found between pre-service vocational teachers with and without vocational training, it can be assumed that the following results might also be interesting for the implementation of multidisciplinary digital competencies in the dual vocational education and training system.

To explore the misfit of SEM, fit indices are essential, and the results show that all fit indices support the primary hypothesis, whether the model can be validated or not (Cheung and Rensvold 2002; Marsh et al. 2005). The decision about whether the SEM fits is based on four widely known fit indices, which provide an insight in the model’s ability to reproduce an input covariance matrix (Bagozzi and Yi 2012; Hooper et al. 2008; Taasoobshirazi and Wang 2016). This study is based on the usual “goodness of fit” indices (GFIs), namely the Confirmatory Fit Index (CFI) and the Tucker-Lewis Index (TLI) (Hooper et al. 2008; Hoyle and Panter 1995; Taasoobshirazi and Wang 2016). Applying clear cut-off criteria, CFI and TLI should be ≥ 0.95 (Hu and Bentler 1998; Rigdon 1996). These severe values (Bagozzi and Yi 2012) were achieved after excluding RF from the SEM. Absolute fit indices gauge a “badness of fit” (BFI), which means that a value of zero would indicate an optimal fit (Hoyle and Panter 1995). An absolute fit index is the Root Mean Square Error of Approximation (RMSEA), which depends on N. Considering N = 205 in this study, the RMSEA should be less than 0.06 (Hu and Bentler 1998, 1999). Another absolute fit index is the Standardized Root Mean Square Residual (SRMR) (Taasoobshirazi and Wang 2016), which should not be higher than 0.08 (Hooper et al. 2008; Hu and Bentler 1998, 1999). By excluding RF, the absolute fit indices met these cut off criteria and the model fit became acceptable. Due to the good fit of the model there was no need to apply modification indices and change the model any further (MacCallum et al. 1992).

The estimates in Table 4 that relate to QAMDC are less strong than those that are associated with SAMDC, but the relative path estimates are similar. The estimates of the used latent variables to QAMDC are small to medium and not as highly significant as the self-assessed latent variables. The standardized path coefficients between SAMDC to SDS and SCL are quite high in Fig. 3. The dimension application of digital security with SDS and QDS has the greatest impact on SAMDC or QAMDC. SCL and QCL have the second largest estimates. The constructs for handling of digital devices are in third position and Information Literacy in fourth. The externally and qualitatively measured variable of Information Literacy (QIL) has a weak estimate on QAMDC, while SIL has a medium path estimate on SAMDC. Solving of digital problems seems to have the weakest bound to multidisciplinary digital competencies. In conclusion, the model shows an acceptable fit and Table 4 provides mostly significant path coefficients.

The second hypothesis examined the influence of the attitude towards digitization on the participants’ multidisciplinary digital competencies. The results show that the attitude towards digitization influences the self-assessed multidisciplinary digital competencies of pre-service vocational teachers. This significance confirms the finding of Yerdelen-Damar et al. (2017) that pre-service teachers’ attitudes towards technology use had an effect on their technological competence. The hypothesized assumption (Hypothesis 2b) that the attitude towards digitization would also directly influence the external and qualitative assessment of pre-service vocational teachers’ multidisciplinary digital competencies was not verified by this study. However, this supports the findings of Aesaert et al. (2015), who found no relationship “between pupils’ ICT attitudes and their actual ICT competence” (Aesaert et al. 2015, p. 67). Whether ICT is implemented in teaching seems to depend on the general mindset of teachers (Hermans et al. 2008; Tearle 2003). The findings confirm the conclusion of Bunz et al. (2007) that there is a direct relationship not between computer anxiety and actual computer-email-web-fluency, but between attitudes and self-perception of the participants. As the data shows, SAMDC could be a good and significant predictor for actual QAMDC. In other words, this study confirms that ICT self-efficacy correlates positively with the achievement in ICT competence tests (Fraillon et al. 2014; Hatlevik et al. 2015). Table 5 shows the latent correlation between the constructs of Fig. 2. The instruments significantly correlate at an almost medium level and positively with each other (r = 0.48, p < 0.0001), which is a significant and higher correlation than Ihme and Senkbeil found in 2017 (r = 0.22, p < 0.01). This is not surprising because, according to Hargittai and Hinnant (2008), digital competencies usually increase with the level of education, and while Ihme and Senkbeil (2017) focused on adolescents, the present study focuses exclusively on pre-service teachers in their bachelor’s or master’s programme. Table 5 shows the latent correlations of the corresponding (self-assessed and externally assessed) competence dimensions of the multidisciplinary digital competencies. While the dimensions handling of digital devices, Information Literacy, and application of digital security show weak but significant correlations, no significant correlation between QCL and SCL was found for collaboration. A minimal correlation was found within the dimension problem solving. This is not surprising, because even though the latent constructs aimed at the same content, they measure different realities, since the open questions in QAMDC were designed specifically for a fictive scenario.

Table 5 Summary of correlation between the constructs

Limitations

Studies which aim to measure digital skills are often limited in their definitions, small sample sizes or methods of data collection (van Deursen and van Dijk 2009). Firstly, the sample was limited to pre-service vocational teachers, who studied business and economic education. Consequently, the findings should not be generalized, although the sample size was adequate for applying SEM. Secondly, the self-assessments of pre-service vocational teachers should be treated with caution (Aesaert et al. 2017; Ihme and Senkbeil 2017). For this reason, the externally and qualitatively evaluated statements were added as QAMDC. An explanation for the weak to medium path coefficients could be the invalid evaluation of the three raters. However, Table 3 shows an acceptable interrater reliability. Therefore, the low performance of the QAMDC model does not result in multicollinearity or falsifying the items through different ratings of the open-ended questions.

Conclusions

Based on the explorative studies of Roll and Ifenthaler (2020a, b) on multidisciplinary digital competencies, this study can be seen as a specific conceptual addition to the TK dimension of the TPACK model (Koehler et al. 2014) in the multidisciplinary context of the dual vocational education and training system, because the origin of the multidisciplinary digital competencies dimensions focused on technical vocational students as a target group. As mentioned at the beginning of this paper, teachers should also have these digital competencies in order to be prepared for teaching in Industry 4.0. The results validate the conception of the named dimensions in multidisciplinary digital competencies, with the exception of the construct of reflecting on interconnected and digital environments. The path estimates of SAMDC and QAMDC are slightly different, but this can give pedagogically worthwhile insights into the dimensions that most influence the multidisciplinary digital competencies of pre-service vocational teachers. This could help to foster specific competence development of pre-service vocational teachers within their curricula (Ertmer 2005; Mishra and Koehler 2006). In addition to the education of vocational teachers, the curricula of further education of in-service vocational teachers could also benefit from such studies by specifically focusing on the development of such multidisciplinary digital competencies in training units (Seufert et al. 2018).

Furthermore, the findings indicate that attitude towards digitization has a large effect on the multidisciplinary digital competencies of pre-service vocational teachers (Ifenthaler and Schweinbenz 2013). To integrate digital devices in the classroom, vocational teachers’ self-efficacy in this context should be improved during their pre-service training. Looking at the low self-assessment in SHD and comparing it with the higher values in QHD or QAMDC, one notices that most students underestimate their handling of digital devices (Aesaert et al. 2017; Bunz et al. 2007; Dunning et al. 2003). Therefore, digital devices should be increasingly integrated into the training of pre-service vocational teachers to avoid an underestimation and boost their self-efficacy in handling digital devices (Brevik et al. 2019). In particular, a systematic integration of multidisciplinary and digital competencies into the curriculum of vocational teacher education would be of great benefit (Tenberg, 2016, 2020).

In the following studies, a critical reflection on the fictitious scenario and the wording of the tasks is required. Even if measuring digital skills via self-assessment is a resource-saving method, it does not provide accurate evidence of digital competence (van Deursen and Van Dijk 2010). The aim of this study was to test if the approach of a formative external and qualitative assessment of open-ended questions could be a resource-saving alternative to modelling complex scenarios via programming specific dashboards (Rausch 2017). Certainly, this is not valid if you want to use it as high-stakes testing. Self-assessment is not really suitable here. The partial convergence of the results with the existing research literature at least indicates that the applied approach is suitable, even if the instrument still needs to be optimized and tested on a larger sample size. This study provides an overview of the general structure of multidisciplinary digital competencies; however, a further investigation of each dimension would be desirable. Furthermore, it would be of interest to apply QAMDC to validate the multidisciplinary digital competencies of technical vocational students, exploring their readiness for Industry 4.0.