Learning Factories 4.0 in technical vocational schools: can they foster competence development?

Abstract

Learning Factories 4.0 are thought to prepare vocational students for the challenges of Industry 4.0. The implementation of those interconnected Learning Factories 4.0 at technical vocational schools may promote the development of subject-related technical competencies as well as multidisciplinary digital competencies. Still, research is scarce with regard to the development of competencies supported through Learning Factories 4.0 in technical vocational schools. Hence this research focusses on subject-related technical and multidisciplinary digital competencies of technical vocational students change due to different levels of Learning Factory 4.0 interaction over time. Three subsequent competence tests with N = 63 technical vocational students were conducted. Findings indicate the benefits of integrating Learning Factories 4.0 for developing subject-related competencies in technical vocational schools. However, the study could not identify a positive development of multidisciplinary digital competencies. The findings of this study can help educators to further develop learning environments with support from Learning Factories 4.0 as well as preparing their learners for the demanding competencies of Industry 4.0.

Introduction

Industry 4.0 is the vision of a horizontally and vertically interconnected digitization of entire industrial value chains (Stecken et al. 2019; Veile et al. 2019). This is based on the real-time data exchange between customers, employees, objects and production via cyber-physical systems (Lee et al. 2015). Current studies focusing on Industry 4.0 indicate that it will induce changes in occupational structures and work activities as well as additional requirements for employees (Groß et al. 2017; Liboni et al. 2019). These changes also affect the dual vocational and education training system in Germany. It is necessary to prepare technical vocational students with the competencies needed to cope with the challenges of Industry 4.0 (Gebhardt et al. 2015; Pfeiffer 2015). Especially students in technical vocational education and training will be confronted by the future requirements and impacts on their work (Delcker and Ifenthaler 2021; Gebhardt et al. 2015).

To qualify technical vocational students as future skilled workers in authentic and interconnected working environments, technical vocational schools installed so called Learning Factories 4.0. Learning Factories 4.0 simulate an Industry 4.0 production line as part of a learning environment (Scheid 2018). Initial research indicates that such interconnected model-like smart factories can foster competence development among students (Bauernhansl et al. 2018; Hummel et al. 2015). In addition, research suggests that Learning Factories 4.0 can help to develop not only technical, but other Industry 4.0-relevant competencies like information literacy, problem solving, or collaboration (Balve and Ebert 2019; Ifenthaler 2018; Tisch et al. 2013). However, current studies largely focus on learners at the university level (Abele et al. 2015; Balve and Ebert 2019; Belinski et al. 2020; Enke et al. 2018; Liebrecht et al. 2017; Müller-Frommeyer et al. 2017). Studies about the impact of Learning Factories 4.0 in the context of technical vocational educational training are scarce (Roll and Ifenthaler 2020b; Scheid 2018). Therefore, this study aims to investigate competence development through Learning Factories 4.0 at technical vocational schools.

Theoretical background

It is indispensable for technical vocational students to develop the competencies needed to work and participate in Industry 4.0 (Scheid 2018; Stettes 2018; Zinn 2014). Such competencies include the handling of digital devices and software, adequate Information Literacy, the application of digital security, the ability to collaborate digitally, and the ability to solve digital problems (Roll and Ifenthaler 2020b). Learning Factories 4.0 could support the development of these competencies.

Learning Factories 4.0

Abele (2016) defines a Learning Factory 4.0 on behalf of the International Academy for Production Engineering as a learning environment including four distinguishing characteristics:

• processes that are authentic, include multiple stations, and comprise technical as well as organizational aspects

• a setting that is changeable and resembles a real value chain

• a physical product being manufactured

• a didactical concept that comprises formal, informal and non-formal learning, enabled by the actions of the trainees in an on-site learning approach (Abele 2016, p. 1).

The interconnection of a Learning Factory 4.0, which is the fundamental idea of Industry 4.0, is based on cyber-physical production systems (CPPS). CPPS “are collaborating computational entities which are in intensive connection with the surrounding physical world and its on-going processes, providing and using, at the same time, data-accessing and data-processing services available on the internet” (Monostori 2014, p. 9) and enable an entire Industry 4.0 production line. The CPPS are connected to Manufacturing Execution Systems (MES) so that the current state of each item being manufactured could be tracked by the learners. Because a fitting instructional design is crucial for competence development within Learning Factories 4.0 (Tisch et al. 2013), most didactical concepts of Learning Factories 4.0 in technical vocational schools are not only based on learning with CPPS, but also have several separate, specific modular basic components in a foundation laboratory. As can be seen in Fig. 1 in this laboratory, the individual workstations (or learning modules) from the holistic production chain are set up separately, so that the learners can learn at these workstations before going to the large holistic plant (Scheid 2018).

Fig. 1

Exemplary Learning Factory at the technical vocational school in Offenburg (Badische Zeitung 2016)

As a consequence of this, the separated modules of the foundation laboratories are more often integrated into daily teaching than the complex CPPS (Scheid 2018). Due to the complexity of CPPS, students can develop a necessary understanding of the consequences of such interconnected processes (Abele et al. 2015; Scheid 2018; Tisch and Metternich 2017). The separate modules in the foundation laboratory are similar or equivalent to the components of the CPPS, but here they stand on their own. As can be seen in Fig. 1 they are sometimes moveable, they can be put together to simulate the product transfer from one module to another. The foundation laboratory and the CPPS are equipped with the newest technology (Scheid 2017, 2018). This allows technical vocational students to learn basic technical content with modern technology that is not physically linked to other production components. These modules are intended to prepare learners for the complex tasks and problems at the holistic CPPS of the Learning Factories 4.0 (Scheid 2018). Basing on ideas of the constructive alignment approach (Biggs 1996) universities often integrate Learning Factories 4.0 through project-based learning, which is organizationally difficult to implement at technical vocational schools due to organizational challenges, because of the often rigid timetables (Scheid 2018). And even in higher education it remains a challenge to develop and implement adequate didactical-methodological approaches in Learning Factories 4.0 in order to develop competencies (Pittich et al. 2020). To further support these developments and to expand the technology-rich learning environment, the use of augmented reality (AR) and, in some cases, virtual reality (VR) is a useful addition to the classroom (Scheid 2018).

In any case, the literature indicates professional and methodological competence development through Learning Factories 4.0 in higher education (Kreimeier et al. 2014; Müller-Frommeyer et al. 2017). There, action-based learning seems to have positive effects on subject-related competence development, improves the transfer of knowledge and motivates learners, because of the realistic workplace scenario (Lanza et al. 2016; Nickolaus 2019). These high-tech learning environments are particularly well-suited for developing problem solving skills (Abele et al. 2019). These considerations on competence development should be partially transferable from the tertiary education sector to technical vocational schools (Scheid 2018). To summarize, Learning Factories 4.0 can still be seen as “laboratories for developing methods of competence development for specific value adding systems” (Groß et al. 2017, p. 297).

Competence development through Learning Factories 4.0

Industry 4.0 may bring changes in occupational learning culture (Belinski et al. 2020; Ifenthaler 2018). In order to efficient competence acquisition and development streams of self-directed, action-based and hands-on learning (Belinski et al. 2020; Hummel et al. 2015), should be intentionally integrated by teachers in their instructional designs for conducive learning with Learning Factories 4.0 (Lanza et al. 2016; Liebrecht et al. 2017). Industrial processes are becoming more complex due to real-time interconnectivity, which means an overlap of several field of operations within a value added network to finalize a product (Gebhardt et al. 2015). Different fields of operations have to deal with each other and employees have to understand content from other disciplines to create synergies between the specific departments within and outside the enterprise (Gebhardt et al. 2015; Liboni et al. 2019). This melting of several fields of operations is also relevant for technical vocational students (Scheid 2018). An orientation for how tointegrate Industry 4.0-related topics to technical vocational students is provided by the Ministry of Education, Youth and Sports of Baden-Wuerttemberg (Löhr-Zeidler et al. 2016). This guideline consists of six different thematic scenarios in which Industry 4.0 is relevant for the technical vocational training. The different scenarios include Industry 4.0 content on (1) production development and production planning, (2) flexible manufacturing, (3) integration of manufacturing execution systems, (4) service and maintenance, (5) energy management, and (6) network connectivity and data security. The scenarios, in turn, are each subdivided into three different requirement areas. For each requirement area, the associated technical subject-related competencies are listed in the respective description of the scenarios. For example, the Industry 4.0-related contents of vocational training mechatronics technician include in scenario 1 the requirement area 1, in scenario 2 the requirement area 2, in scenario 3 again requirement area 1, in scenarios 4, 5 and 6 in each case requirement area 2. This suggests that mechatronics technicians will already have many points of contact with Industry 4.0 content in the scenarios. Based on the taxonomic wording, the respective difficulty can then be taken from here.

As Learning Factories 4.0 are didactic simulations of smart factories, they should help learners to develop the needed competencies (Abele et al. 2017; Hummel et al. 2015). Developing technical vocational students’ competencies, either related to the professional and technical disciplines or to the digital competencies which are necessary for several disciplines, is the aim of the Learning Factories 4.0 (Scheid 2018). Tisch et al. (2013) already wrote in 2013 that state-of-the-art technology is essential for Learning Factories 4.0 to provide up-to-date and adequate training for Industry 4.0 should enable students to work with its technology, which is usually the state of the art in 2020.

The Learning Factories 4.0 in technical vocational schools are usually designed to foster those subject-related technical competencies (STC) in all relevant modern production technologies like automation technology, electrical engineering, mechatronics and so on (Scheid 2018). Anyway, technical competencies are not sufficient for preparing technical vocational students for Industry 4.0 (Lanza et al. 2016). According to Gebhardt et al. (2015), in order to solve occupational tasks, which demand knowledge and skills in more than just one discipline, technical vocational schools need to integrate several other competencies that are related to general digitization rather than specifically subject-related. The literature provides many suggestions for non-subject related multidisciplinary digital competencies which university students should properly develop for Industry 4.0 with Learning Factories 4.0 (Bauernhansl et al. 2018; Enke et al. 2018; Pittich et al. 2020; Tisch et al. 2016). Unfortunately, similar studies for the technical vocational educational training are scarce. Scheid (2018) mentions several competencies relating to technical vocational educational training based on relevant studies in higher education. He also emphasizes the difficulty of comparing the high level of university students to the much more basic competence level of technical vocational students. There are also hardly any multidisciplinary digital competencies are specifically mentioned in the guidline of the Ministry of Education, Youth and Sports of Baden-Wuerttemberg (Löhr-Zeidler et al. 2016). This research gap was minimized through a recently conducted qualitative interview study (Roll and Ifenthaler 2020a). In that study, interviews were conducted with the responsible corporate instructors of several companies to investigate the question of which non-subject-related competencies technical vocational students should possess to be prepared for Industry 4.0. Based on the work of Abele et al. (2015), Hummel et al. (2015) and Tisch et al. (2016), semi-structured interview guidelines were created. These were supplemented by the literature review of Ilomäki et al. (2016), the competence dimensions of the DigComp 2.0 project by Carretero et al. (2017) and Vuorikari et al. (2016) and the concept of Information Literacy (Fraillon et al. 2014). The result of this exploratory study is a set of multidisciplinary digital competencies:

• The attitude towards digitization (AD), which involves the motivational, volitional and social willingness to act (Weinert 2001) within the digital, technical and vocational context (Abele et al. 2017).

• The handling of digital devices (HD) as well as software in general. This emphasizes the action-orientated knowledge of the efficiency of devices (Johnson et al. 2006; Selwyn and Husen 2010).

• The correct usage of copyright (CU) issues (Fraillon et al. 2019) as part of

• Information Literacy (IL), which includes gathering, processing and evaluating online information (Fraillon et al. 2014).

• Careful application of digital security (DS) standards is a major topic for corporate instructors within an interconnected world. Application of digital security involves adequate and cautious behaviour to comply with (corporate) digital security standards (Carretero et al. 2017; Sîmandl et al. 2017).

• An appropriate virtual collaboration (CL), which basically includes common rules to follow when exchanging information or negotiating via digital devices (Carretero et al. 2017; van Laar et al. 2017) when working with experts from others fields, as well as in the student’s private life.

• Solving problems within the context of interconnectivity is crucial. Therefore problem solving (PS) contains the skills, expertise and choice of suited methods to solve problems in a structured manner (Abele et al. 2015).

• The self-reflection (RF) on one’s own digital actions within an interconnected world ensures the continuity of learning (Lin et al. 2014), which is not only about consequences at work, but about gaining a deeper understanding of the content and its consequences (Dewey 1910; Rodgers 2002) in private life, too.

These multidisciplinary digital competencies should be understood as action-oriented competence dimensions, which are not just work related but would fit in every young person’s mindset of everyday life (Roll and Ifenthaler 2020a). For this reason, multidisciplinary digital competencies are defined as a combination of willingness, abilities and individual skills that enable the individual to act adequately and socially responsibly in the digital context of professional, social and private situations (Roll and Ifenthaler 2020a, p. 193). It is reasonable to assume that in an environment in which these competencies will be needed later on, they will develop under didactic guidance.

Hypotheses

This study aims to validate the development of subject-related technical competencies within a discipline and non-subject-related, multidisciplinary digital competencies through Learning Factories 4.0. Previous research indicates that the integration of Learning Factories 4.0 in vocational learning environments may support the competence development of technical vocational students (Lanza et al. 2016; Liebrecht et al. 2017; Tisch et al. 2016). For example, in a pre-posttest design, Aymans et al. (2018) found a significant development of self-assessed computer-related competencies in a group which was learning with a Learning Factory 4.0. Hence, the first research objective of this study focuses on this development of multidisciplinary digital competencies supported through Learning Factories 4.0 (Roll and Ifenthaler 2020b; Scheid 2018). Accordingly, it is assumed that the higher the level of interaction (LOI) of technical vocational students with a Learning Factory 4.0, the higher the level of their multidisciplinary digital competencies (MDC) will develop over time (Hypothesis 1). The level of interaction means (LOI), reflects how much the learners actually worked with the learning factory or the individual modules within the videotaped lessons.

Reining et al. (2019) have stated that university students who learned with a Learning Factory 4.0 discussed professional competencies significantly more than a control group. In this video-study professional competence was understood as a combination of “technical knowledge, knowledge of science and mechanics, application of knowledge, analystical thinking, measuring energy and presentation skills” (Reining et al. 2019, p. 2). The control group learned the respective content through a normal seminar and without working with the Learning Factory 4.0. Hence, the second research objective emphasizes the development of subject-related technical competencies supported by Learning Factories 4.0 (Abele et al. 2015; Hummel et al. 2015; Scheid 2018). It is expected that the higher the level of interaction (LOI) of technical vocational students with a Learning Factory 4.0, the higher their level of subject-related technical competencies (STC) will develop over time (Hypothesis 2).

Method

In the following sections, the participants of the study are briefly described, before the survey instruments are presented and the procedure is discussed in more detail.

Participants

Technical vocational schools were asked to join this study to explore the research objectives. The Ministry of Education, Youth and Sports in Baden-Wuerttemberg, Germany, helped with the acquisition by providing incentives for participating schools. Conditions for receiving incentives included providing the researchers’ access to classes of electrician vocational students in their second year of training with a typical performance level. The topic of the examined lessons had to be “an introduction into sensor technology”. The participants of this study were N = 71 electrician vocational students learning in four comparable classes from four technical vocational schools. Data for eight students was deleted from the dataset (2 of class 1, 2 of class 3 and 4 of class 4) because they did not participate at all three required subsequent tests. Students were between 18 and 37 years old (M = 20.48; SD = 3.04). All participants were in their second year of training for several different, but similar electrician professions. The different electrical professions are standardly being instructed together and only in the later specific contents they are being taught separately. Therefore, despite later different degrees such as electronics technician for industrial engineering or electronics technician for building and energy technology, one can speak of comparable classes, especially in this topic here. Only five students were female (8%), which reflects the typical non-heterogenic population of technical vocational students (Kroll 2017; Statistic Office of Baden-Wuerttemberg 2019b). Demographics and class sizes are shown in Table 1. The total sample size is in line with average class size of technical vocational schools measured by the official Statistic Office of Baden-Wuerttemberg (2019a, b), which is 16.1 students per class.. All classes were described by their teachers as typical electricians’ classes with the typical heterogenic level of performance.

Table 1 Summary of participating classes and level of interaction with the Learning Factory 4.0

Design

In order to analyse differences in competence development supported through Learning Factories 4.0, this study uses a mixed repeated-measures design (Keselman et al. 1998). The researchers and participating teachers discussed several opportunities to measure competence development within this topic. Previous research used written examinations for competence evaluation linked to Learning Factories 4.0 (Abele et al. 2019; Liebrecht et al. 2017). Hence, written exams were identified as an economical way to measure and evaluate competence development within the setting of an on-going school year. Therefore, the complex competence constructs were measured through open questions including the specific settings (Abele et al. 2019). The researchers provided the part of the written exam focusing on multidisciplinary digital competencies. The teachers provided a pool of subject-related exam questions. These questions were expected to have the same level of difficulty as they would have in any exam within this topic.

Data collection

Data was collected between September 2019 and January 2020. The videos of the examined lessons show how much Learning Factories 4.0 are used within the lessons by the technical vocational students. Given the fact that the topic was taught in eight lessons à 45 min, a total of 360 min was controlled for each class for how many minutes the students worked and learned with the CPPS or some modules of the foundation laboratory. As mentioned before, both components belong to the didactical concept of Learning Factories 4.0 and therefore this study does not differentiate between them. Beside the control group, Table 1 shows the actual minutes of hands-on learning with the Learning Factories 4.0 in the viewed lessons. Hands-on means that the learners actually worked with the Learning Factory 4.0 or its modules, i.e. read out data or examined individual components or discussed them directly on the Learning Factory 4.0. Every action of the technical vocational students which dealt in some way with Learning Factories 4.0 was counted. The authors did not distinguish between the quality of each learning process on the Learning Factories 4.0 as long as the students solved hands-on and action-oriented tasks with or on the Learning Factories 4.0 (Abele et al. 2019; Cachay et al. 2012). Quartiles were calculated based on the time spans in Table 1. Consequently, class 3 and 4 form the group of the highest level of interaction with their Learning Factory 4.0, which means in sum N = 18 students learned on a high interaction level with the components of Learning Factory 4.0 and 24 students had a medium level of interaction. The control group included 21 students and had no interaction with a Learning Factory 4.0 during the lesson.

Instrument

The authors received the didactical concepts of the examined lessons from each teacher. On the basis on these lesson plans, a discussion was held on how to properly design the instruments to meet a fair level of students’ performance and which dimensions of multidisciplinary digital competencies would play a major role, a minor one or no role during the teaching of the topic. First the students’ prior knowledge in the subject and state of multidisciplinary digital competencies were measured in a pre-test (“T0” at time point 0; TP0). The subsequent measurement instrument “T1” followed immediately after the last lesson in the chosen topic (in time point 1; TP1). To prove competence development in the long-term (Ilomäki et al. 2016), the participants were given a third test “T2” 4 weeks (in time point 2; TP2) after TP1. The setting of TP2 4 weeks after TP1 had practical reasons, because all teachers and classes were available for a maximum of 4 weeks after TP1 was absolved. After the tests were completed, the answers were deductively rated and correspondingly graded with points (Abele et al. 2019). However, the graded points were transformed into Likert scales for further analysis. All these tests were done per paper and pencil and consisted of two parts. In the first part, the students had to answer open questions about their subject-related competencies. In the second part of the instrument they answered open questions intended to measure their multidisciplinary digital competencies. The students filled out the instruments with pseudonyms, so that they could not be identified by their teachers or the authors.

Subject-related technical competencies

As experts in their field, teachers provided a pool of open questions, because the tasks in the first part of the instrument should have a similar difficulty over all three time points. The test at TP1 was conducted as a regular short exam to obtain a more realistic evaluation of those competencies (Aymans et al. 2018). Consequently, the teachers graded all tasks of all three time points as they would usually do for an exam in this subject. The grading of the given answers was transformed into a five-point Likert scale. When students left the answer blank, teachers rated this as zero, whereas a complete and perfect answer was rated as a four. Table 2 shows the general summary of the subject-related technical competencies for each time point. Due to organizational aspects, each teacher graded only his/her own class. Therefore, Table 2 includes Cronbach’s alpha instead of an interrater reliability.

Table 2 Summary of subject-related technical competencies

Multidisciplinary digital competencies

Concerning hypothesis 2, the authors provided open-ended survey questions (Hsieh and Shannon 2005) to measure all dimensions of their multidisciplinary digital competencies. These were rated by three qualified and trained researchers following criteria of qualitative content analyses (Mayring 2015). After consultation with teachers, it was decided to integrate the following competence dimensions in order to measure if the announced multidisciplinary digital competencies are really fostered by the didactical concepts of the teachers. The competence dimension attitude towards digital devices was not integrated because the tests had to be shortened at the request of all participating teachers. The teachers claimed that Information Literacy and application of digital security are the most fitting and most important non-subject-related competencies in these lessons. The Learning Factories 4.0 also should improve the collaboration (CL) and problem solving (PS) abilities in the long term. Handling of digital devices (HD) would play a minor role within the topic. The questions for the competency dimensions of MDC were similar in taxonomy and wording across all three test time points, but not the same.The criteria to assess the responses to these competence dimensions were pre-tested and defined within a workshop. A summary is shown in Table 3, which also provides the interrater reliability by presenting the Intraclass correlation (ICC3,k), proving the two-way mixed consistency of the three raters (Shrout and Fleiss 1979) for each item at each time point.

Table 3 Intraclass correlation (ICC3,k) and summary of the rated items at several time points

Analytic strategy

Hypotheses 1 and 2 require identical independent variables, namely the level of interaction with Learning Factories 4.0 (LOI) and time, while subject-related technical competencies and multidisciplinary digital competencies are the dependent variables. To validate the differences in learning outcomes due to LOI between the three groups, with a normal distribution a mixed repeated-measure two-way multivariate analysis of variance (MANOVA) would have been the adequate analytic strategy. Nevertheless, the Shapiro–Wilk test showed a significant violation (p < 0.001) of the multivariate normality distribution of the dependent variables. In addition, Box’s M-test was statistically significant (p < 0.001); hence, the data also violated the assumption of homogeneity of the variance–covariance matrices. Therefore, a nonparametric procedure was used (Keselman et al. 1998). The non-parametric equivalent of a two-way multivariate analysis of variance is the Scheirer–Ray–Hare test (Dytham 2017), which is a derivation of the multivariate Kruskal–Wallis test (Scheirer et al. 1976). To analyse differences over time and different levels of interaction with Learning Factories 4.0, post-hoc analyses were conducted. Therefore, pairwise Wilcoxon tests and Wilcoxon test effect sizes were chosen as adequate procedures after using Scheirer–Ray–Hare tests (Sokal and Rohlf 2001). The statistics software R (version: 4.0.2), R-Studio (version 1.2.502) and the R-package rcompanion (version 2.3.25) were used for data analysis (Mangiafico 2020).

Results

Development of multidisciplinary digital competencies through different levels of interaction with a Learning Factory 4.0 over time

To evaluate the effect of different LOI over time on multidisciplinary digital competencies and their relevant competence dimensions, several Scheirer–Ray–Hare tests were performed. Table 4 shows the results, including the generalized Eta-square (Olejnik and Algina 2003), which provides comparable effect sizes for studies with repeated-measures design (Bakeman 2005). The data did not show any extreme outliers, which could have been a threat to validty. In general, an absolute correlation coefficient of > 0.7 among two or more competence diemensions could indicate the presence of multicollinearity but there is almost no multicollinearity, as assessed by Pearson correlation. As shown in Table 4, there was no significant interaction between LOI and time on multidisciplinary digital competencies (df = 4, SS = 10,091, H = 3.37, p = 0.497, η² = 0.018) and its competence dimensions, except for problem solving (PS; df = 4, SS = 28,812, H = 9.66, p = 0.047, η² = 0.051). Further, Table 4 shows that LOI has a significant impact on all competence dimensions of multidisciplinary digital competencies. The generalized η² shows medium to large effect sizes, based on Cohen’s benchmarks (Cohen 1988; Richardson 2011). The factor time had no significant effect on these competence dimensions, except for collaboration (df = 2, SS = 60,032, H = 20.47, p < 0.001, η² = 0.108).

Table 4 Summary of the Scheirer–Ray–Hare test and effect sizes

Even though the interaction effect of LOI and time was not significant, LOI had an impact on multidisciplinary digital competencies and their competence dimensions. Therefore, they were analysed pairwise via Wilcoxon post-hoc analyses. Table 5 shows the differences via pairwise comparisons at each time points for the significant LOI of Table 4. Adjusted p-values using the Bonferroni multiple testing correction method within post-hoc Wilcoxon tests were applied, and Wilcoxon r as a measure of effect size was chosen (Fritz et al. 2012). To measure a development of competencies over time, groups should not differ significantly in their achieved level of multidisciplinary digital competencies and subject-related technical competencies in the pre-test. This would indicate a comparable level of these competencies. Nevertheless Table 5 shows that the control group (with) differs significantly from the group of medium LOI in three competence dimensions of multidisciplinary digital competencies (copyright usage: Diff = − 0.647, p = 0.004, r = 0.455; application of digital security: Diff = − 0.815, p < 0.001, r = 0.5; problem solving: Diff = − 0.979, p = 0.009, r = 0.445). The control group showed no significant difference to the group with the highest LOI (except for the competence dimension application of digital security (Diff = − 0.512, p = 0.048, r = 0.319).

Table 5 Summary of the pairwise comparisons using Bonferroni–Holm-correction grouped by “Level of Interaction”

As can be seen in Table 5, handling of digital devices witnessed the only significant differences between LOI groups at TP2, when both groups with LOI differ significantly from the control group (no to medium: Diff = − 0.683, p = 0.019, r = 0.35; no to high: Diff = − 0.757, p = 0.017, r = 0.385) but not from each other (p = 0.481). In Table 5 the medium LOI group of copyright usage differs significantly from no in TP0 (Diff = − 0.647, p = 0.004, r = 0.455) and high LOI (Diff = 0.472, p = 0.002, r = 0.469), which both do not show any significant differences between them for this competence dimension at any time point (TP0: p = 0.66, TP1: p = 0.359, TP2: p = 0.109). For Information Literacy there were no significant differences at TP0 (no to medium LOI: p = 0.077; no to high LOI: p = 0.291; medium to high LOI: p = 0.291), but there were at TP1 and TP2 between the control group and the groups with LOI. There was no significant difference between the LOI groups in TP1 (p = 0.618) and TP2 (p = 0.532). The significantly different level of application of digital security in the pre-test between the control group and the groups with LOI (no to medium: Diff = − 0.82, p < 0.000, r = 0.5; no to high: Diff = − 0.51, p = 0.048, r = 0.32) was still significant at TP1 (no to medium: Diff = − 0.571, p = 0.007, r = 0.421; no to high: Diff = − 0.825, p = 0.002, r = 0.553) and TP2 (no to medium: Diff = − 0.772, p < 0.000, r = 0.63; no to high: Diff = − 0.965, p < 0.000, r = 0.750). The only relevant significant difference between LOI groups for collaboration can be found at TP1 between no to medium level (Diff = − 1.161, p < 0.000, r = 0.577) and no to high level of LOI (Diff = − 0.619, p = 0.017, r = 0.386).

Table 6 provides the results of the pairwise comparisons for each construct, which had a significant effect of the factor time, detected by the Scheirer–Ray–Hare tests in Table 4. Therefore, Table 6 is grouped by time points and shows that the control group had a significantly higher level of collaboration in the pre-test than at TP1 (Diff = 0.841, p = 0.012, r = 0.413), but no significant difference to TP2 (p = 0.808). The effect of time on collaboration was not significant for the medium LOI group (TP0 to TP1: p = 0.194; TP0 to TP2: p = 0.393; TP1 to TP2: p = 0.265). The group with the highest LOI did not have a significantly higher level from TP0 to TP1 (p = 0.138). In addition, this group had a significantly higher level in TP0 than in TP2 (Diff = − 0.667, p = 0.003, r = 0.536). In TP2 they even reached a significantly lower level than in TP1 (Diff = − 1.019, p = 0.003, r = 0.626).

Table 6 Summary of the pairwise comparisons using Bonferroni–Holm correction grouped by “time”

Table 5 shows significant differences for problem solving between the control group and the group of medium LOI over all three time points (TP0: Diff = − 0.979, p = 0.009, r = 0.445; TP1: Diff = − 0.881, p < 0.000, r = 0.580; TP2: Diff = − 0.665, p = 0.034, r = 0.342). Here the differences between the control group to high LOI were not significant at TP0 (p = 0.06) and TP1 (p = 0.322) but in TP2 (Diff = − 1.003, p = 0.002, r = 0.542). The medium to high LOI does differ significantly at TP1 (Diff = 0.704, p = 0.001, r = 0.523), but not at TP0 (p = 0.06) and TP2 (p = 0.147). Overall, Table 5 shows that the level of accumulated multidisciplinary digital competencies was significant different for all three groups over all time points with the exceptions at TP0 between the control group and high LOI (p = 0.099) and at TP2 between medium and high LOI (p = 0.889).

Development of subject-related technical competencies through different levels of interaction with a Learning Factory 4.0 over time

To evaluate the effect of different LOI with a Learning Factory 4.0 over time on subject-related technical competencies a Scheirer–Ray–Hare test was performed. The results are presented in Table 4. There was a statistically significant interaction impact of LOI over time with a medium effect size (df = 4, SS = 44,361, H = 14.88, p = 0.005, η² = 0.079). As can be seen in Table 4, the factor time had a significantly large effect on subject-related technical competencies (df = 2, SS = 99,461, H = 33.36, p < 0.000, η² = 0.177). Therefore, post-hoc analyses were conducted and the pairwise comparisons in Table 6 were grouped by time points. Table 5 proves that within the pre-test at time point 0 there were no significant differences between the three groups (no to medium LOI: p = 0.126; no to high LOI: p = 0.955; medium to high LOI: p = 0.055). As Table 6 shows, there were no significant differences in subject-related technical competencies for the control group over time (TP0 to TP1: p = 0.648; TP0 to TP2: p = 0.827; TP1 to TP2: p = 0.648). The students who had a medium LOI had a highly significant higher level of subject-related technical competencies in TP1 in comparison to the pre-test in TP0 (Diff = − 2.42; p < 0.000; r = 0.77). Their results in TP2 were also significantly higher than in TP0 (Diff = − 1.104; p < 0.000; r = 0.522) but lower than in TP1 (Diff = 1.313; p < 0.000; r = 0.599).

The group with the highest LOI reached a significantly higher level of subject-related technical competencies at TP1 than at TP0 (Diff = − 0.696; p = 0.002; r = 0.329), but the level at TP2 was not significantly different from the level of subject-related technical competencies at TP0 (Diff = − 0.133, p = 0.468, r = 0.026). The score for subject-related technical competencies at TP1 was significantly higher than at TP2 (Diff = 0.563; p = 0.024; r = 0.354).

Discussion

The following sections highlight the key findings, the implications and limitations of this study and present ideas for future research.

Key findings

To summarize the results, there was no significant interaction effect of LOI and time on multidisciplinary digital competencies. That time had no significant effect on multidisciplinary digital competencies, is in line with Ilomäki et al. (2016), who rather see digital competence development as a long-term or even a life-long (Ferrari 2012) story. The multivariate design of the Scheirer–Ray–Hare test provided significant effects of LOI in this study on multidisciplinary digital competencies. According to Cohen’s benchmarks (Cohen 1988), the effect sizes of LOI on multidisciplinary digital competencies and its competence dimensions, presented through a generalized eta squared (Bakeman 2005) in Table 4, are medium (HD, CL) or large (CU, IL, DS, PS, MDC). To interpret the particular effects of the level of interaction between these groups one has to look at the post-hoc test results in Tables 5 and 6. Looking at these pairwise comparisons, the group which had a medium LOI seems often to score higher than the group with the highest level of interaction in terms of multidisciplinary digital competencies and its dimensions. This started at the pre-test, with a multidisciplinary digital competencies level which was 0.295 points better (p = 0.039) than the group of students with the highest LOI. With a difference of 0.279 (p = 0.023), the multidisciplinary digital competencies level was almost the same at TP1. At TP2 there was no significant difference anymore between these two groups.

Grouping the students into different LOI groups had the effect that the level of multidisciplinary digital competencies was already different in TP0, which also explains that the Scheirer–Ray–Hare test shows significant results for the factor LOI. Table 5 supplements this and shows that in TP2 there was no significant difference between the medium and high LOI. Because Table 4 does not show an interaction effect of LOI and time, and the differences of multidisciplinary digital competencies within the pre-test were significant, Hypothesis 1 has to be rejected. Even if LOI had no significant effect on subject-related technical competencies, the interaction effect of LOI and time on subject-related technical competencies was significant and has a medium effect (Cohen 1988). The significant impact of time on subject-related technical competencies had a medium effect as well.

The pairwise comparisons in Table 6, grouped by the significant factor time of Table 4, showed that there was no significant change in subject-related technical competencies level over time within the control group. The students who had a medium LOI improved their level of subject-related technical competencies significantly but had their peak at TP1, immediately after the lesson was over. Their subject-related technical competencies decreased from TP1 to TP2 significantly, by about 1.313 points (p < 0.000), which is confirmed by a large effect size of r = 0.522. Their level of subject-related technical competencies at TP2 was still significantly higher (Diff = 1.104) than at TP0 (p < 0.000), which also had a large effect (r = 0.599). The students with the highest LOI also improved their subject-related technical competencies significantly from TP0 to TP1 (Diff = − 0.696, p = 0.002, r = 0.329), which was also their peak performance. At TP2 their subject-related technical competencies were 0.563 lower than at TP1 (p = 0.024, r = 0.354) and not significant in comparison to their achieved score at TP0, which also showed a negligible effect size.

Table 6 clearly demonstrates that using Learning Factories 4.0 within the lesson seems to have a positive impact on the learning outcome of STC. The control group showed no significant improvements over time, but both groups with LOI did at least at TP1. The fact that all groups showed their maximum of subject-related technical competencies at TP1 is not surprising, because T1 was conducted in the lesson after the topic was finished. So T1 measured the competencies, when they were as fresh in the students’ minds as they could be. To sum this up, Hypothesis 2 is accepted, because Table 4 shows a significant interaction effect of LOI and time on subject-related technical competencies. In addition, Table 6 shows that the control group had no significant differences between the three time points at all. The group with medium LOI improved their subject-related technical competencies also in the long term (at TP2), in contrast to the group with the highest LOI, which had their peak performance at TP1. Their level of subject-related technical competencies decreased from TP1 to TP2, when it was not significantly different from that at TP0. As a side effect, the Scheirer–Ray–Hare test also showed significant results of a small interaction effect of LOI and time on problem solving (df = 4, SS = 28,812, H = 9.66, p = 0.047, η² = 0.051). The shown difference in Table 5 supports the current literature, which argues that Learning Factories 4.0 could foster the development of this competence dimension (Abele et al. 2015; Cachay and Abele 2012; Tisch et al. 2016).

Implications

Even if these findings have to be interpreted with caution due to the limitations, they have implications for the stakeholders, like school authorities, teachers, students and researchers. As the focus of Learning Factories 4.0 is action-oriented competence development, Tisch et al. claimed in 2016 that there were “no pragmatic and reliable instruments to evaluate the development of intended competencies in Learning Factories” (Tisch et al. 2016, p. 1358). This study scientifically explored the opportunities of repeated measures design with open questions as a first step to provide such an instrument. The detected development of subject-related technical competencies indicates a comprehensive didactical concept, which is essential for effective competence development (Lanza et al. 2016), and which Tisch et al. (2013) saw as a crucial problem for the design of Learning Factories 4.0 at universities, and Scheid (2018) at vocational schools. The number of stakeholders who are involved with Learning Factories 4.0 in vocational teaching is increasing, because the Ministry of Economic Affairs, Work and Housing of Baden-Wuerttemberg provided funding to more than 37 technical vocational schools to implement Learning Factories 4.0 (Ministry for Economic Affairs, Work, and Housing 2017, 2018, 2019). This means that these findings are interesting arguments for the ministry, but also for technical vocational schools, which are interested in procuring Learning Factories 4.0. Stakeholders should have a strong interest in academic findings like this, which validate their arguments to install such expensive facilities (Wilbers 2017) to foster the development of technical competencies with the newest technology on the market to prepare the future shop-floor staff for Industry 4.0 (Scheid 2018).

Limitations

The first limitation is surely the competence evaluation using knowledge tests. The relation between competence and measured performance (Chomsky 1966) is scientifically recognized but still not precisely explicable (Tisch et al. 2016). According to Pittich et al. (2020), conceptual knowledge tasks can be a good predictor of competencies. Still, it is questionable to interpret the results of subject-related technical competencies, which are basically written answers evaluated by technical vocational teachers, and multidisciplinary digital competencies, which are written answers rated by experienced raters, as competencies. Though the competence evaluation with simulated problem scenarios (Abele et al. 2019) was not an adequate alternative in this study, due to its organizational consequences on the on-going school year and the resources it would have required. In addition, all competencies relating to digitization need to be observed in a long-term, problem-based and technology-rich scenario, where they can be developed (Ilomäki et al. 2016). Learning Factory 4.0 as a learning environment provided two of these criteria, because of its up-to date technology and problem-based didactical design (Abele et al. 2019; Hummel et al. 2015). To counteract the problem of long-term multidisciplinary digital competencies development, the tests were repeated with similar problem-based tasks 4 weeks after the content was learned. Within these 4 week it was not possible to observe the informal learning aspects within this study (Dehnbostel 2014).

Another limitation that should be considered is that the participants were in four classes from different schools. The Learning Factories 4.0 of these technical vocational schools are also not completely identical, just as Learning Factories 4.0 are rarely similar (Abele et al. 2019). In order to keep this limiting factor as low as possible, the participating teachers were consulted and after a discussion they confirmed that the concepts are comparable and can be implemented with these classes and with the respective Learning Factories 4.0 at these technical vocational schools. Due to the many different companies where the students worked, the resources would have not been enough to control for such variables, too. The above mentioned difficulties of this implementation research design may also be a reason for the decrease of reliability (see Table 2), as the teacher’s motivation and relatedness being involved in the demanding research study may have also been affected negatively. In order to overcome the complexity of intereactions in the given learning environment, further quasi-experimental studies (design experiments) could be conducted to reveal existing interactions and major influence which could not be carved out fully in the current classroom setting. Further, such experimental settings could also include a specific focus on the development of comptences over time, including several measurement points. However, such longitudinal perspectives would add further demanding testing to all involved stakeholders, which may have influence on the reliability and validity of the findings.

A related limitation is the art of grouping the technical vocational students into the level of interaction with the Learning Factories 4.0. This study did not differentiate by the quality of learning processes on the Learning Factories 4.0, as for example Reining et al. (2019) did by analysing the content of conversations. Even though competence dimensions were defined for multidisciplinary digital competencies (Roll and Ifenthaler 2020a), they still contain too many skills and abilities to measure a significant development through working with or without Learning Factories 4.0. Concentrating on just measuring one particular skill of a competence dimension would not have supported the idea of multidisciplinary digital competencies (Roll and Ifenthaler 2020a). At the university level, the learning tasks for Learning Factories 4.0 are openly designed and aim to avoid any predefined approaches (Hummel et al. 2015). While teachers in technical vocational schools also try to design their instructions similarly, they have to coach their learners much more than university students need to be (Scheid 2018). Therefore, one should use caution when comparing findings on vocational training students to university level learners (Müller-Frommeyer et al. 2017). Even if the teaching approaches, subjects and technical infrastructure might be similar, the cognitive level of learners is not (Scheid 2018).

Future research

Given the limitation of not differentiating by different learning actions with the Learning Factories 4.0, further video analysis needs to be done to investigate the individual learning more specifically. One option could be to integrate a content analysis on the “act4learning framework”, like Reining et al. (2019) did, or to develop specific coding guidelines to evaluate each action of the learners within the Learning Factories 4.0 context. To dig deeper into the application of constructive alignment (Biggs 1996), sustainable competence development and its iterative assessment within Learning Factories 4.0 needs further research, which may focus on formal and informal assessment of technical vocational students (Dehnbostel 2014). To validate these findings, a larger sample would be required. Stakeholders such as school authorities should be interested in developing the basis of this research further and providing more empirical findings to all related stakeholders. In the context of higher education it is scientifically justified that Learning Factories 4.0 could foster competence development (Abele et al. 2015; Cachay et al. 2012; Gronau et al. 2017).

The contribution to science of this study is the validation of competence development on the much lower educational level of vocational educational training within technical vocational schools (Roll and Ifenthaler 2020b). Nevertheless there is still room for a lot of didactical improvements for integrating Learning Factories 4.0. For example, the Learning Factories 4.0 should be connected to commercial vocational schools, which are learning with ERP software to manage the procurement, marketing and sales (and much more) of manufactured products (Scheid 2017; Wilbers 2017). Wilbers (2017) explained this lack of connection by the insufficient technical connection between these different types of schools. Praxis shows that even if the technical infrastructure is ready, real didactical concepts, which are applicable in the daily business of vocational teaching, are non-existent (Delcker and Ifenthaler 2021). How to provide logical, resource saving teaching to a mix of commercial and technical students should be the next big stage of research on the topic of Learning Factories 4.0 within vocational schools.

Appendix 1. Example questions

Examples of STC
What is the difference in design between a retro-reflective sensor and a through-beam sensor?
The term EVA principle stands for input-processing–output principle (IPO model). Please provide an example for each component
Describe the operation of this pneumatic valve
Explain: What is a smart factory?
Examples of MDC
When commissioning a system, you discover that a distance measurement is not working correctly. Describe your procedureof problem-solving in bullet points
Describe your search strategy on the Internet if you would like to complete this task