- Open Access
A multidimensional structure of domain-specific problem-solving competencies of electronics technicians for automation technology
Empirical Research in Vocational Education and Training volume 8, Article number: 7 (2016)
The ASCOT research initiative funded by the Federal Ministry of Education and Research had the objective to measure vocational competencies in high-quality jobs. To adequately assess these it is important to offer authentic situations, similar to real-life (working) situations. Whether domain-specific problem-solving competencies are a unidimensional or a multidimensional construct has far reaching implications in vocational education e.g., regarding valid assessment of performance in the final exam at the end of vocational training. This article deals with the structure of problem-solving competencies of electronics technicians for automation technology at the end of their apprenticeship.
Realistic problems with the focus on professional activities such as programming (constructive problem-solving) and troubleshooting a programmable logic controller (analytical problem-solving) were employed to measure domain-specific problem-solving competencies. Based on the results structural equation models were calculated and different models are compared for goodness of fit with consideration to content knowledge and fluid intelligence as predictors.
Our results document a two-dimensional structure of domain-specific problem-solving consisting of an analytical and a constructive dimension. This two-dimensional structure remains stable even if content knowledge and fluid intelligence are included as predictors for the domain specific problem-solving competencies (Figs. 1, 5, 6). Only knowledge areas, which are relevant to the problem-solving situation, were predictive for the analytical and constructive problem-solving competencies. Fluid intelligence has a direct effect on both domain-specific problem-solving competencies but the influence is considerably lower when compared to relevant content knowledge.
Based on these findings the final exam at the end of the apprenticeship needs to enclose an appropriate amount of both subdimensions of domain-specific problem-solving competencies. As a practical implication both dimensions have to be supported in addition to the teaching of content knowledge.
The ASCOT research initiative (technology-based assessment of skills and competencies in VET) was launched by the Federal Ministry of Education and Research (BMBF) in Germany in 2011. It pursues the objective of measuring vocational competencies which are essential for working in high-quality jobs in a changing working world (BMBF 2012). The main focus in this article is on electronics technicians in the field of automation technology, who work for companies which develop, manufacture or install automation solutions in the craft trades field (e.g., electrical installation companies, waterworks) or in the trade and industry sector (e.g., automobile and chemical industry, waste disposal industry). Example of their field of work are: looking after automation systems,Footnote 1 analyzing functional interconnections and programming automation systems (European Communities 2002).
For the assessment of professional competencies, it is important to offer authentic and complex test situations that are comparable to real-life (working) situations and approachable to practice, improvement and standard setting (Hartig et al. 2008; Shavelson 2010).
Structures of professional competencies are able to show which subdimensions could theoretically and empirically be separated. These subdimensions indicate the potential to solve specific problem-based tasks and are presumably influenced by curriculum-based learning processes and the confrontation with real-life working tasks. Those seem not only important for the respective markedness but also for their interrelation. The development of professional competencies is not limited to development within individual subdimensions, it also focuses on their relations, whose partial differentiations and processes of integrations during apprenticeship were reported (an overview in Seeber and Nickolaus 2010). This explains the fact that statements concerning structures of competencies, based on cross-sectional studies, like those in this article, are only a snapshot. From a practical view, information about structures of competencies at the end of apprenticeship are important for: (1) references that show which working specifications have been achieved, (2) which subdimensions should be assessed in a valid fashion and (3) which development needs are important. Assessing professional knowledge in an industrial-technical field at the end of an apprenticeship nearly always shows multidimensional models, whereas the structure using this knowledge in different problem situations is unidimensional (an overview in Nickolaus and Seeber 2013). This fact is probably influenced by the assessment of domain-specific problem-solving competencies that focused in previous research only on analytical aspects, like the ability of troubleshooting by prospective car mechatronics technicians or electricians for building technology, because these analytical requirements are very relevant in vocational practice (Becker 2005).
An unresolved issue is whether domain-specific problem-solving competencies are unidimensional or multidimensional in the industrial-technical field. We assume, based on task analysis of electronics technicians for automation technology (Zinke et al. 2014), a multidimensional structure of domain-specific problem-solving competencies at the end of the apprenticeship.
State of research
The following chapter gives an overview of the structure of professional competencies in the industrial-technical field with a strong focus on domain-specific problem-solving competencies. Important results of neighboring sectors are included.
Studies about the structure of professional competence in the industrial-technical field
With the change of input to learning outcome orientation, the valid assessment of vocational skills and competencies has become more important in the field of vocational education and training. Such assessments are important for comparisons, for generating explanatory models and evaluating educational programs. Analyses have confirmed two subdimensions across domains: Content knowledge, and the ability to use this knowledge in different problem situations (an overview in Nickolaus and Seeber 2013). Moreover, other differentiations of content knowledge and manual skills as relevant subdimensions are possible.
Operationalization/structure of content knowledge
Modeling content knowledge in the commercial sector was based on case-situations and learning-fields (Rosendahl and Straka 2011), or based on subject discipline (Seeber 2008). By contrast, in the industrial-technical field, modeling content knowledge on the basis of different technical areas is common. These content areas are influenced by prioritization of the curriculum contents. Strong indications of these curriculum-based learning processes are mentioned by Gschwendtner (2011) and Schmidt et al. (2014). There are inconsistent references to modeling content knowledge based on cognitive psychology criteria (declarative and procedural). In the commercial sector, and in the general didactics of technology, declarative and procedural knowledge is empirically separable (Winther 2010; Walker 2013); however, it is not in the industrial-technical field (Geißel 2008; Gschwendtner 2008). This might be due to the operationalization of content knowledge which, in the professional context, always includes declarative knowledge components. The assessment of declarative and procedural knowledge using paper-pencil-tests probably promotes a common scaling of declarative and procedural knowledge.
Operationalization/structure of domain-specific problem-solving competencies
In the industrial-technical field, we define domain-specific problem-solving competencies as the ability to use content knowledge in different problem situations (Nickolaus 2011a). Content knowledge in the industrial-technical field, is by definition, a necessary condition for solving a domain-specific problem. Trainees and craftsmen in the industrial-technical field are confronted, in vocational practice, with analytical and constructive demand situations (Nickolaus 2011a). So far, only analytical aspects have been acquired in previous research: for example, troubleshooting in technical systems. The assessment of such analytical competencies is realized with real-life working tasks and authentic computer simulations (Nickolaus et al. 2009, 2012). Domain-specific analytical problem-solving competencies are defined as finding malfunctions and making proposals to correct these defects (Wiedemann 1995; Abele et al. 2014; Walker et al. 2015, 2016). An unresolved issue in the industrial-technical field is, whether such analytical and constructive problem-solving competencies are empirically separable.
Findings for the occupations of car mechatronics and electronics technicians for building technology
Further studies of car mechatronics and electronics technicians for building technology showed only a unidimensional model of content knowledge at the end of the first year of training (Gschwendtner 2008; Geißel 2008). This structure becomes multidimensional during the process of training: for example, the results at the end of the apprenticeship of electronics technicians for building technology document a three-dimensional structure of content knowledge, which is separated in different technical areas: basic principles of electrical engineering, traditional electrical installation and modern electrical installation with bus technology (Nickolaus et al. 2011). A similar differentiation (six-dimensional structure) is reported by Gschwendtner (2011); this could also be replicated in the field of car mechatronics (Schmidt et al. 2014).
So far, the importance of content knowledge on domain-specific problem-solving has been reported in all presented analyses between these two subdimensions (r = .61 to .86) of professional competencies (e.g., Gschwendtner 2008; Nickolaus et al. 2011). At the end of the apprenticeship of electronics technicians for building technology, the correlations between the subdimensions of content knowledge [traditional electrical installation (1), basic principles of electrical engineering (2) and modern electrical installation with bus technology (3)] and analytical problem-solving competencies (4) show only high correlations (r1,4 = .38; r2,4 = .67; r3,4 = .61) if the knowledge subdimension is theoretically relevant to the concrete problem-solving situation (Nickolaus et al. 2011). This fact leads to the assumption that electronics technicians for building technology primarily make use of basic principles of electrical engineering and modern electrical installation with bus technology when they solve a problem. This kind of “context specificity”, which is relevant for solving concrete situations, is also mentioned in psychological studies (Funke 2003; Jonassen 2000).
Abele et al. (2012) tested the influence of fluid intelligence (Weiß 2006), and content knowledge on the domain-specific analytical problem-solving competencies of car mechatronics and electronics technicians for building technology. The results document that fluid intelligence has only an indirect effect on domain-specific analytical problem-solving competencies (Abele et al. 2012). On the other hand, Petsch et al. (2015) report an direct effect of fluid intelligence on domain-specific problem-solving competencies in the building sector. They assume a great heterogeneity between cognitive basics, and specific test items support this direct effect.
Findings for the occupation of electronics technicians for automation technology
Until last year there had not been any studies which investigated this job profile and there was a lack of instruments for the assessment of professional competencies. New results indicate that it is possible to measure domain-specific constructive problem-solving competencies of electronics technicians for automation technology using a paper-pencil-test, because the test performances with this sort of test are highly comparable (r ≈ .90) to test performances within real-life programming environments (Link and Geißel 2015; Link 2016). Measuring domain-specific analytical problem-solving competencies is also possible using a simulation of an automated system instead of a real system (r ≥ .90). Here domain-specific analytical problem-solving competencies were operationalized by the ability to find a malfunction in an automated system (Walker et al. 2016). With these assessment instruments, it is possible to investigate the structure of domain-specific problem-solving competencies. Walker et al. (2015) document a two-dimensional structure of domain-specific problem-solving consisting of an analytical and a constructive dimension.Footnote 2 This two-dimensional structure remains stable even if the influence of content knowledge and fluid intelligence was controlled. Additionally, for solving analytical and constructive problems (e.g. troubleshooting or programming a programmable logic controller), only the required dimensions of content knowledge were predictive, which are relevant to the problem-solving situation. Fluid intelligence has only an indirect effect on domain-specific problem-solving, mediated by content knowledge.
Studies about the structure of problem-solving competencies in science education and the influence of fluid intelligence and content knowledge
There is evidence for a multidimensional structure of problem-solving competencies in the PISA study, where multidisciplinary static problem situations (information disclosed to the problem solver is complete) were assessed (Leutner et al. 2012). They define problem-solving competencies as “an individual’s capacity to use cognitive processes to confront and resolve real, cross-disciplinary situations where the solution path is not immediately obvious” (OECD 2004). Multidisciplinary problem situations means, in this context, that these problems are related to mathematics, reading and science, and that, to solve these problems, the content area is not attached to only one single subject area of mathematics, science or reading (OECD 2003). Furthermore, static problem-solving is highly correlated with mathematics (r = .89), science (r = .80) and reading (r = .82) but empirical a standalone construct (Leutner et al. 2012). In addition, there are a direct and an indirect effect of static problem-solving on math (βindirect = .38; βdirect = .12) and science (βindirect = .33; βdirect = .17) at the end of tenth grade (Leutner et al. 2006). Similar results were observed for the fluid intelligence, but the direct and indirect effects on math (βindirect = .19; βdirect = .11) and science competencies (βindirect = .15; βdirect = .09) were marginal (Leutner et al. 2006). Static problem-solving, as assessed in PISA 2003, is a three-dimensional construct, consisting of system analysis and design, trouble-shooting, and decision-making; but these subdimensions are also highly correlated (r = .73–.93) (Leutner et al. 2012).
Studies about domain-specific problem-solving in science education are only partially available. A study by Scherer and Tiemann (2012) in chemistry education is based on the above mentioned PISA framework. They analysed the relationship between different components of problem-solving and strategy knowledge with a virtual environment, where students should solve interactive and static problems (Scherer and Tiemann 2012). At first, a unidimensional structure of chemistry-specific problem-solving was assumed, but this could not be confirmed. Instead, strategy knowledge can be distinguished from problem-solving competencies (Scherer and Tiemann 2012). As covariates for problem-solving competencies, only domain-specific prior knowledge, fluid intelligence, and computer familiarity are significant predictors. The knowledge of problem-solving strategies was not predictive. Overall, the regression-model explained 14.5 % of variance (Scherer and Tiemann 2012).
The main research aim is to analyse which structure of domain-specific problem-solving competencies could be supported. Therefore, a theoretical framework was developed based on the reported state of research and is shown in the following Fig. 1.
There is a direct effect of fluid intelligence on domain-specific analytical and constructive problem-solving competencies (a) and on content knowledge (b). This direct effect of fluid intelligence on content knowledge is only partially in agreement with earlier related research in technical-vocational and science education. On the one hand, some studies in science education document a direct effect of fluid intelligence on domain-specific problem-solving competencies (e.g., Scherer and Tiemann 2012). On the other hand, most studies in industrial-technical fields do not report such an effect (Abele et al. 2012; Nickolaus et al. 2012), with exception of Petsch et al. (2015), who report the influence of fluid intelligence on domain-specific problem-solving competencies in the field of VET. In conclusion, it is unclear if there is a direct effect of fluid intelligence on the domain-specific problem-solving competencies of electronics technicians for automation technology.
Furthermore, content knowledge is separated in different technical areas that are related (c). We suggest three subdimensions for content knowledge on the basis of findings in the field of mechatronics and task analyses of electronics technicians for automation technology. These three subdimensions are: basic principles of electro technology (BP) (e.g., direct and alternating current, safety regulations), electrical engineering (EE) (e.g., drive technology, electrical installation) and automation with programmable logic controllers (AT/PLC).
Additionally, there is a direct effect of these separate context-subdimensions on domain-specific analytical and constructive problem-solving competencies (d), which are related, but standalone (e). The fact that content knowledge is predictive for domain-specific problem-solving competencies was shown in earlier studies in vocational and science education (e.g., Seeber and Nickolaus 2010; Nickolaus et al. 2012; Scherer and Tiemann 2012). In particular, the above-mentioned fact of the task analyses and the findings in the field of interdisciplinary problem-solving suppose a two-dimensional structure of domain-specific problem-solving competencies.
Based on the previous chapter, the following hypotheses are in the focus:
Modeling the structure of domain-specific problem-solving shows a correlated, but separable, two-dimensional structure of domain-specific problem-solving, consisting of an analytical and a constructive dimension (H1.1), even if content knowledge and fluid intelligence are included in the model (H1.2).
Because of the context-specificity of domain-specific problem-solving, only knowledge areas, which are relevant to the problem-solving situation, should be predictive for solving analytical and constructive problems.
If content knowledge is included in the model, fluid intelligence has only an indirect effect on domain-specific problem-solving, mediated by content knowledge.
We tested 211 electronics technicians for automation technology predominantly at the end of their apprenticeship (99.0 % are in the third or fourth year of training). The average age of the participants is M = 20.32 years (SD = 2.11); 90.5 % of the apprentices are male and were doing their apprenticeship in the industrial sector (81.6 %; craft sector = 18.4 %). Most of the trainees (69.2 %) have a basic secondary school certificate (Realschulabschluss). 25.1 % have an upper secondary school leaving certificate (Gymnasium).
To ensure the evidence of validity based on the test content, we integrated experts in the field [teachers, trainers and auditors of the Chamber of Industry and Commerce (IHK)] in the development process of the instruments (AERA 2014; Haladyna and Rodriguez 2013). Based on the theoretical framework of professional competencies presented in Fig. 1, we developed instruments for content knowledge, domain-specific analytical and constructive problem-solving competencies. An extensive overview of all instruments can be seen in: van Waveren and Nickolaus (2015); Link and Geißel (2015); Link (2016); Walker et al. (2015, 2016).
Domain-specific analytical and constructive problem-solving competencies
The approach to measuring domain-specific analytical and constructive problem-solving competencies was a set of realistic problems with the focus on professional activities such as programming (constructive problem-solving) and troubleshooting a programmable logic controller (PLC) (analytical problem-solving). The development of the items was based on the structure of the control program based on Benda (2008). Following this approach for both domain-specific problem-solving competencies, items regarding the operating mode, the step chain and output routine were developed. In total, eight troubleshooting scenarios and eight programming scenarios were generated per problem-solving competence. To measure the analytical problem-solving competence, we used a simulation (Fig. 2, Walker et al. 2016). The development process was made in accordance with Funke and Reuschenbach (2011).
Figure 2 shows, in the lower left corner, a control panel to operate the automation system (e.g., start and stop the system). The current state of the PLC could be seen in top left corner. It is possible to check the current state of all actors and sensors and enlarge selected components via mouse clicks on the right main screen. By clicking on “Simatic” in the main screen, the control program can be opened and observed in real-time processing just like in a real automation system. The reliability of the analytical problem-solving instrument is SEM-ReliabilityFootnote 3 = .75.
In the constructive problem scenario, the apprentices had to program a certain part of the control program of a PLC in a complex automation system where parcels were transported by a belt conveyor, measured by sensors and sorted by a pick-and-place unit with a vacuum gripper (Link 2016). Figure 3 shows the logical functions of an item in two common programming languages: the one on the left is more graphically oriented and is called “function block diagram” (FBD); the other, on the right, is more textual and referred to as “statement list”(STL). In this item, the variables and logic gates for a counter and a signal lamp had to be programmed. The reliability of the constructive problem-solving instrument is SEM-Reliability = .83.
In accordance with the theoretical framework of professional competencies, the items of the content knowledge test cover a three-dimensional content specific structure, consisting of the dimension of automation/PLC (AT/PLC), electrical engineering (EE) and basic principles of electro technology (BP) (van Waveren and Nickolaus 2015). The latent correlations between the subdimensions are high (EE and BP is r = .89, between AT/PLC and BP r = .87 and between r = .83) but standalone. The reliability of the test is about EAP/PV-Reliability = .72 across all dimensions (van Waveren and Nickolaus 2015).
Due to the extensive duration of the tests for professional competencies (see below), we measured fluid intelligence only with part one of the CFT 20-R (Weiß 2006), i.e., the subset of continuing logical progressions, classifications, matrices and topologies were administered. The related loss of reliability from rtt = .95 (part one and two together) to rtt = .92 (only part one) seems acceptable considering the reduction of the test duration (Weiß 2006).
Research design and testing procedure
The duration for testing all components of the professional competencies required 2 days. On the first day, the instruments for fluid intelligence, content knowledge and constructive problem-solving competencies were applied (duration 4.5 h). Analytical problem-solving competencies were tested on day two (duration 4.5 h). To avoid position effects, we used a booklet-design (Frey et al. 2009). Analytical and constructive problem-solving competencies were tested in a cross-over-between and within subject design (Ratkowsky et al. 1993; Jones and Kenward 2003).
All the latent analyses were obtained by using Mplus. For identification of the structural equation models, we fixed the variance of the latent variable to one, and for models with ordinal items the WLSMV estimators were chosen (Muthén and Muthén 1998–2010). The evaluation of the model fit was based on the following indices: Root Mean Square Error of Approximation below .06 (RSMEA ≤ .06) (Hu and Bentler 1999); Comparative Fit Index about .95 (CFI ≥ .95) (Raykov and Marcoulides 2006); the Chi square degree of freedom-ratio below χ2/df ≤ 2 (Schermelleh-Engel et al. 2003) and the weighted root mean residual below 1.0 (WRMR ≤ 1.0) (Yu 2002). Based on Satorra-Bentler-adjusted χ2-values, we used the DIFF Test option in MPlus to test the dimensionality of the domain-specific problem-solving competencies (Hoyle 2012; Wang and Wang 2012).
As shown in Table 1, the fluid intelligence of the tested participants is slightly higher than the average (standardized to mean 100). The results for content knowledge are EAP/PV estimators.
With regard to content knowledge, differences between the three dimensions become obvious. The negative meanFootnote 4 of the AT/PLC knowledge-dimension indicates that the items of this dimension were too difficult for the apprentices at the end of their training. van Waveren and Nickolaus (2015) were also able to be replicate these results in a larger sample of n = 878 electronics technicians for automation technology at the end of their apprenticeship (van Waveren and Nickolaus 2015; Walker et al. 2016). For the other two dimensions (Table 1, EE and BP), the ease of the items match the mean knowledge of the apprentices.
Following the hypothesis H1.1, we expected that a two-dimensional structure of domain-specific problem-solving competencies would better fit the data than a unidimensional model. Therefore, we compared these models (Table 2 on following page, M1 and M2). The likelihood ratio (LR) test supported our assumption for H1.1 and the two-dimensional model M1 (Table 2, comparison of M1 and M2, (Δχ (1) 2 = 26.69, p < .001). Considering that the latent correlation between analytical and constructive problem-solving is high (r = .77) (Fig. 4) in addition to the result of the LR test, it is evident that analytical and constructive problem-solving competencies have substantial common variance but are empirically separable dimensions.
To test the hypothesis H1.2, we included the manifest person ability scores of the three dimensions of the content knowledge as predictors in the two-dimensional and unidimensional model of domain-specific problem-solving (Table 2, M3 and M4).Footnote 5 Analogously to hypothesis H1.1, we compared the fit of the two-dimensional with a unidimensional model of domain-specific problem-solving. And again, the results of the LR test supported the two-dimensional model of domain-specific problem-solving M3 (Table 2, comparison of M3 and M4, Δχ (4) 2 = 48.22, p < .001). After the integration of the three dimensions of the content knowledge, the latent correlation between analytical and constructive problem-solving declined from r = .77 to r = .62, which highlights the need of differentiation between these two dimensions of domain-specific problem-solving (Fig. 5).
Finally, to test hypothesis H1.2, we included fluid intelligence in the models M3 and M4 and compared the fit using the LR test. Despite the integration of content knowledge and fluid intelligence, the two-dimensional model of domain-specific problem-solving fitted the data better after all (Table 2, comparison of M5 and M6, Δχ (5) 2 = 39.82, p < .001). The results of the LR test supported our hypothesis H1.2 and pointed out the stability and the distinctiveness of the analytical and constructive problem-solving competencies.
Our second research question affected the influence of the dimensions of content knowledge on the domain-specific problem-solving competencies. It was assumed, that for solving analytical and constructive problems (e.g. troubleshooting or programming a PLC), only the required dimensions of content knowledge were predictive, which are relevant to the problem-solving situation (e.g. the content knowledge Dimension AT/PLC and not BP) (H2). As Figs. 4 and 5 showed, only the dimension of automation and programmable logic controller (AT/PLC) was predictive for solving analytical and constructive problems. The dimension of AT/PLC explained 35 % of the variance in the constructive, and 46 % of the variance in the analytical problem-solving competency (Fig. 5). These explanation patterns also remained stable after the integration of fluid intelligence into the structural model (Fig. 6). Induced through the addition of the fluid intelligence, the explained variances of the constructive (40 %) and analytical (51 %) problem-solving competencies increased slightly.
Referring to our third research question (H3), fluid intelligence had, not as assumed, a significant direct path to domain-specific analytical and constructive problem-solving competencies. This finding is in line with the results of Fischer et al. (2015). Furthermore, Fig. 6 showed that the influence of fluid intelligence compared to the influence of the dimension AT/PLC on the problem-solving dimensions is noticeably lower.
Discussion and conclusion
The results document a two-dimensional structure of domain-specific problem-solving competencies, which is represented through both an analytical and a constructive subdimension. These subdimensions remain stable even if fluid intelligence and the three-dimensional content knowledge were integrated. In view of the fact that these two domain-specific problem-solving dimensions are empirically separable but correlated, these results have implications for designing the final examination at the end of the apprenticeship. To ensure the evidence for validity based on test-content of the final exam, both subdimensions of domain-specific problem-solving competencies have to be included in an appropriate amount. The high influence of context-specific knowledge dimensions on analytical and constructive problems emphasises the importance of content knowledge for domain-specific problem-solving. Practical implications for teaching structures can also be drawn from the two-dimensional domain-specific problem-solving structure in a manner that both dimensions have to be supported during the teaching process. Furthermore, it is important to ensure that the context-specific knowledge has also been taught. In perspective, the simulation for measuring domain-specific analytical problem-solving could also be integrated in the teaching process. As reported, it is possible to represent industrial programmable logic devices and automation systems via the simulation authentically. Despite of the correlation between the analytical and constructive problem-solving competencies and the similar context (problem-solving in an industrial automation system), the underlying process for solving analytical and constructive problems seems to be disjunctive.
With regard to the transferability of the results concerning the two-dimensional structure of domain-specific problem-solving into other vocational domains, a differentiated consideration is necessary. We assume that the structure of domain-specific problem-solving competence depends on the domain-specific tasks. A replication of these results in the occupation of car mechatronics might be doubtful, because of the smaller range of domain-specific tasks concerning constructive problem-solving competencies. For vocational domains which are more closely related to the occupation of electronics technicians for automation technology, e.g. mechatronic technicians or electronics technicians for industrial engineering, a two-dimensional structure of domain-specific problem-solving seems more likely.
For the occupation of electronics technicians for automation technology at the end of their apprenticeship, fluid intelligence was primarily responsible for the gain of content knowledge. In contrast to our assumption, fluid intelligence was also predictive for solving domain-specific analytical and constructive problems, which is in line with findings of static problem-solving competence in the PISA study.
For further research we see, inter alia, the following desiderata:
A replication-study in which the domain-specific analytical problem-solving competence covers not only troubleshooting a programmable logic controller (software) but also include troubleshooting in hardware (e.g., broken conductors and faulty sensors or actuators). Additionally a replication of the (two-dimensional) structure for problem-solving in vocational domains (e.g., mechatronic technicians) is also indicated.
How far the actual design of the final exam at the end of the apprenticeship in the occupation of electronics technicians for automation technology is suited to provide reliable and valid measures of professional competencies remains unclear. The recent research suggests, that there are substantial variances in the final exams (van Waveren and Nickolaus 2015; Abele 2014; Nickolaus 2011b).
For the development and optimization of approaches which support domain-specific problem-solving competencies, detailed knowledge about the underlying processes during analytical and constructive problem-solving is required. Interesting points of reference for such process analyses and approaches are provided by the researchers Konradt (1995), Schaper (2000), Hochholdinger et al. (2008) and Hochholdinger and Schaper (2013).
Providing a system within automation is for management and supervision of a technical process. Such a system consists of sensors, actuators and processing units: for example, a programmable logic controller (PLC), or software that delivers the program flow.
The basis for this structure were task analyses in the field of vocational practice of the Federal Institute for Vocational Education and Training (BIBB). Zinke et al. (2014) document here two main job assignments: maintenance, and bringing automation systems into service. The ability to program a plc in such systems is very important. Furthermore, the structure is supported by international VET research (feasibility study for a large scale assessment), where “diagnosis of malfunctions (troubleshooting), problem solving and putting systems back into operation” and “programming and handling of bus technologies” are of major importance (Baethge and Arends 2009).
The knowledge scales are logarithmic and their means are zero.
The high correlations between the dimensions occurs because the manifest EAP estimation are shrunken towards each other. This overestimation should not be interpreted (Chen et al. 1998).
Abele S (2014) Modellierung und Entwicklung berufsfachlicher Kompetenz in der gewerblich-technischen Ausbildung. Franz Steiner, Stuttgart
Abele S, Greiff S, Gschwendtner T, Wüstenberg S, Nickolaus R, Nitzschke A, Funke J (2012) Dynamische Problemlösekompetenz. Z Erziehwiss 15(2):363–391. doi:10.1007/s11618-012-0277-9
Abele S, Walker F, Nickolaus R (2014) Zeitökonomische und reliable Diagnostik beruflicher Problemlösekompetenzen bei Auszubildenden zum Kfz-Mechatroniker. Z Pädag Psychol 28(4):167–179. doi:10.1024/1010-0652/a000138
AERA (2014) Standards for educational and psychological testing. American Educational Research Association, Washington
Baethge M, Arends L (eds) (2009) Feasibility study VET-LSA: a comparative analysis of occupational profiles and VET programmes in 8 European countries—international report. Vocational Training Research volume 8. bmbf, Bonn, Berlin
Becker M (2005) Einbindung von Facharbeiterkompetenzen in IKT-dominante Diagnoseabläufe im Kfz-Service. In: Pangalos J, Spöttl G, Kutzen N, Howe F (eds) Informatisierung von Arbeit, Technik und Bildung: Eine berufswissenschaftliche Bestandsaufnahme. Lit, Münster, pp 45–54
Benda D (2008) Das große Handbuch Fehlersuche in elektronischen Schaltungen: Lesen und Auswerten von Schaltungsunterlagen, Fehlersuche mit Methode, Messen und Prüfen mit dem Oszilloskop, Sonderausg. Franzis Elektronik. Franzis, Poing
BMBF (2012) Vocational skills and competencies made visible: the ASCOT research initiative. Federal Ministry of Education and Research (BMBF), Training Policy Division, Bonn, Berlin
Chen S-K, Hou L, Dodd BG (1998) A Comparison of maximum likelihood estimation and expected a posteriori estimation in cat using the partial credit model. Educ Psychol Measur 58(4):569–595
European Communities (2002) Certificate Supplement. https://www.bibb.de/tools/berufesuche/index.php/certificate_supplement/en/elektroniker_automatisierungstechnik_e.pdf. Accessed 5 Dec 2015
Fischer A, Greiff S, Wüstenberg S, Fleischer J, Buchwald F, Funke J (2015) Assessing analytic and interactive aspects of problem solving competency. Learn Individ Differ 39:172–179
Frey A, Hartig J, Rupp AA (2009) An NCME instructional module on booklet designs in large-scale assessments of student achievement: theory and practice. Educ Meas Issues Practice 28(3):39–53. doi:10.1111/j.1745-3992.2009.00154.x
Funke J (2003) Problemlösendes Denken, 1. Aufl. Kohlhammer Standards Psychologie, 1st edn. Verlag W. Kohlhammer, Stuttgart
Funke J, Reuschenbach B (2011) Einsatz technischer Mittel in der psychologischen Diagnostik. In: Hornke LF, Amelang M, Kersting M (eds) Leistungs-, Intelligenz- und Verhaltensdiagnostik, [Vollst. Neuausg.]. Hogrefe, Göttingen [u.a.], pp 595–631
Geißel B (2008) Ein Kompetenzmodell für die elektrotechnische Grundbildung: Kriteriumsorientierte Interpretation von Leistungsdaten. In: Nickolaus R, Schanz H (eds) Didaktik der gewerblich-technischen Berufsbildung: Konzeptionelle Entwürfe und empirische Befunde, vol 9. Schneider Verlag Hohengehren, Baltmannsweiler, pp 121–141
Green SB, Yang Y (2009) Reliability of summed item scores using structural equation modeling: an alternative to coefficient alpha. Psychometrika 74(1):155–167. doi:10.1007/s11336-008-9099-3
Gschwendtner T (2008) Ein Kompetenzmodell für die kraftfahrzeugtechnische Grundbildung. In: Nickolaus R, Schanz H (eds) Didaktik der gewerblich-technischen Berufsbildung: Konzeptionelle Entwürfe und empirische Befunde, vol 9. Schneider Verlag Hohengehren, Baltmannsweiler, pp 103–120
Gschwendtner T (2011) Die Ausbildung zum Kraftfahrzeugmechatroniker im Längsschnitt: Analyse zur Struktur von Fachkompetenz am Ende der Ausbildung und Erklärung von Fachkompetenzentwicklung über die Ausbildungszeit. Zeitschrift für Berufs- und Wirtschaftspädagogik - Beihefte (ZBW-B)(25):55–76
Haladyna TM, Rodriguez MC (2013) Developing and validating test items. Routledge, New York
Hartig J, Klieme E, Leutner D (2008) Assessment of competencies in educational contexts. Hogrefe & Huber, Cambridge
Hochholdinger S, Schaper N (2013) Training troubleshooting skills with an anchored instruction module in an authentic computer based simulation environment. J Tech Educ 1(1):7–22
Hochholdinger S, Rowold J, Schaper N (eds) (2008) Evaluation und Transfersicherung betrieblicher Trainings: Modelle, Methoden und Befunde. Wirtschaftspsychologie. Hogrefe, Göttingen [u.a.]
Hoyle RH (2012) Handbook of structural equation modeling. Guilford Press, New York
Hu L, Bentler PM (1999) Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Model Multidiscip J 6(1):1–55. doi:10.1080/10705519909540118
Jonassen D (2000) Toward a design theory of problem solving. ETR&D 48(4):63–85. doi:10.1007/BF02300500
Jones B, Kenward MG (2003) Design and analysis of cross-over trials. In: Monographs on statistics and applied probability, Bd. 98, 2nd edn. Chapman & Hall/CRC, Boca Raton
Konradt U (1995) Strategies of failure diagnosis in computer-controlled manufacturing systems: empirical analysis and implications for the design of adaptive decision support systems. Int J Hum Comput Stud 43(4):503–521. doi:10.1006/ijhc.1995.1057
Leutner D, Fleischer J, Wirth J (2006) Problemlösekompetenz als Prädiktor für zukünftige Kompetenz in Mathematik und in den Naturwissenschaften. In: Prenzel M (ed) PISA 2003: Untersuchungen zur Kompetenzentwicklung im Verlauf eines Schuljahres. Waxmann, Münster, München [u.a.], pp 119–138
Leutner D, Fleischer J, Wirth J, Greiff S, Funke J (2012) Analytische und dynamische Problemlösekompetenz im Lichte internationaler Schulleistungsvergleichsstudien. Psychologische Rundschau 63(1):34–42. doi:10.1026/0033-3042/a000108
Link N (2016) Problemlösen bei der Programmierung von speicherprogrammierbaren Steuerungen in komplexen automatisierten Systemen. Dissertation, Pädagogische Hochschule
Link N, Geißel B (2015) Konstruktvalidität konstruktiver Problemlösefähigkeit bei Elektronikern für Automatisierungstechnik. Zeitschrift für Berufs- und Wirtschaftspädagogik 111(2):208–221
Muthén LK, Muthén BO (1998–2010) Mplus user’s guide, 6th edn. Muthén & Muthén, Los Angeles
Nickolaus R (2011a) Die Erfassung fachlicher Kompetenzen und ihrer Entwicklungen in der beruflichen Bildung – Forschungsstand und Perspektiven. In: Zlatkin-Troitschanskaia O (ed) Stationen Empirischer Bildungsforschung: Traditionslinien und Perspektiven, 1, neue Ausg. VS Verlag für Sozialwissenschaften, Wiesbaden, pp 331–351
Nickolaus R (2011b) Kompetenzmessung und Prüfungen in der beruflichen Bildung. Zeitschrift für Berufs- und Wirtschaftspädagogik (ZBW) 107(2):161–173
Nickolaus R, Seeber S (2013) Berufliche Kompetenzen: Modellierungen und diagnostische Verfahren. In: Frey A, Lissmann U, Schwarz B (eds) Handbuch Berufspädagogische Diagnostik, 1, neue Ausg. Beltz, Weinheim, pp 155–180
Nickolaus R, Gschwendtner T, Abele S (2009) Die Validität von Simulationsaufgaben am Beispiel der Diagnosekompetenz von Kfz-Mechatronikern: Vorstudie zur Validität von Simulationsaufgaben im Rahmen eines VET-LSA, Stuttgart
Nickolaus R, Geißel B, Abele S, Nitzschke A (2011) Fachkompetenzmodellierung und Fachkompetenzentwicklung bei Elektronikern für Energie- und Gebäudetechnik im Verlauf der Ausbildung - Ausgewählte Ergebnisse einer Längsschnittstudie. In: Nickolaus R, Pätzold G (eds) Lehr-Lernforschung in der gewerblich-technischen Berufsbildung. Steiner, Stuttgart, pp 77–94
Nickolaus R, Abele S, Gschwendtner T, Nitzschke A, Greiff S (2012) Fachspezifische Problemlösefähigkeit als zentrale Kompetenzdimension beruflicher Handlungskompetenz - Modellierung, erreichte Niveaus und relevante Einflussfaktoren in der gewerblich-technischen Berufsausbildung. Zeitschrift für Berufs- und Wirtschaftspädagogik 108(3):243–272
OECD (2003) PISA The PISA 2003 assessment framework: mathematics, reading, science and problem solving knowledge and skills. OECD Pub., [S.l.], Paris
OECD (2004) Problem solving for tomorrow’s world: first measures of cross-curricular competencies from PISA 2003. OECD Publishing, Paris
Petsch C, Norwig K, Nickolaus R (2015) Berufsfachliche Kompetenzen in der bautechnischen Berufsgrundbildung - Strukturen, erreichte Niveaus und relevante Einflussfaktoren. In: Rausch A, Warwas J, Seifried J, Wuttke E (eds) Konzepte und Ergebnisse ausgewählter Forschungsfelder der beruflichen Bildung – Festschrift zum 65. Geburtstag von Detlef Sembill. Schneider Verlag Hohengehren, Baltmannsweiler
Ratkowsky DA, Evans MA, Alldredge JR (1993) Cross-over experiments: design, analysis, and application. In: Statistics, textbooks and monographs, vol 135. Marcel Dekker, New York
Raykov T, Marcoulides GA (2006) A first course in structural equation modeling, 2nd edn. Lawrence Erlbaum Associates Publishers, Mahwah
Rosendahl J, Straka GA (2011) Analysen zur wirtschaftlichen Fachkompetenz angehender Bankkaufleute. Zeitschrift für Berufs - und Wirtschaftspädagogik 107(2):190–217
Schaper N (2000) Gestaltung und evaluation arbeitsbezogener Lernumgebungen. Ruprecht-Karls-Universität Heidelberg, Heidelberg
Scherer R, Tiemann R (2012) Factors of problem-solving competency in a virtual chemistry environment: the role of metacognitive knowledge about strategies. Comput Educ 59(4):1199–1214. doi:10.1016/j.compedu.2012.05.020
Schermelleh-Engel K, Moosbrugger H, Müller H (2003) Evaluating the fit of structural equation models: tests of significance and descriptive goodness-of-fit measures. Methods Psychol Res 8(2):23–74
Schmidt T, Nickolaus R, Weber W (2014) Modellierung und Entwicklung des fachsystematischen und handlungsbezogenen Fachwissens von Kfz-Mechatronikern. Zeitschrift für Berufs- und Wirtschaftspädagogik (ZBW) 110(4):549–574
Seeber S (2008) Ansätze zur Modellierung beruflicher Fachkompetenz in kaufmännischen Ausbildungsberufen. Zeitschrift für Berufs - und Wirtschaftspädagogik 104(1):74–97
Seeber S, Nickolaus R (2010) Kompetenz, Kompetenzmodelle und Kompetenzentwicklung in der beruflichen Bildung. In: Pätzold G, Reinisch H, Nickolaus R (eds) Handbuch Berufs- und Wirtschaftspädagogik, 1st edn. UTB, Stuttgart, pp 95–112
Shavelson RJ (2010) On the measurement of competency. Empir Res Vocat Educ Train 1(2):41–63
van Waveren L, Nickolaus R (2015) Struktur- und Niveaumodell des Fachwissens bei Elektronikern für Automatisierungstechnik am Ende der Ausbildung. J Tech Educ 3(2):62–91
Walker F (2013) Der Einfluss von Handlungsmöglichkeiten auf den Wissenserwerb bei der Durchführung technischer Experimente. Universität Duisburg-Essen
Walker F, Link N, Nickolaus R (2015) Berufsfachliche Kompetenzstrukturen bei Elektronikern für Automatisierungstechnik am Ende der Berufsausbildung. Zeitschrift für Berufs- und Wirtschaftspädagogik (ZBW) 111(2):222–241
Walker F, Link N, van Waveren L, Hedrich M, Geißel B, Nickolaus R (2016) Berufsfachliche Kompetenzen von Elektronikern für Automatisierungstechnik: Kompetenzdimensionen, Messverfahren und erzielte Leistungen (KOKO EA). In: Beck K, Landenberger M, Oser F (eds) Technologiebasierte Kompetenzmessung in der beruflichen Bildung: Ergebnisse aus der BMBF-Förderinitiative ASCOT. Bertelsmann, Bielefeld, pp 139–169
Wang J, Wang X (2012) Structural equation modeling: applications using Mplus. In: Wiley series in probability and statistics. Wiley, Higher Education Press, Chichester, West Sussex
Weiß RH (2006) Grundintelligenztest Skala 2 - Revision - (CFT 20-R). Hogrefe, Göttingen
Wiedemann J (1995) Ermittlung von Qualifizierungsbedarf: Am Beispiel der Störungsdiagnose in der flexiblen Fertigung. Waxmann, Münster, New York
Winther E (2010) Kompetenzmessung in der beruflichen Bildung. W. Bertelsmann Verlag, Bielefeld
Yu C (2002) Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes. University of California, Oakland
Zinke G, Schenk H, Kröll J (2014) Ergebnisse einer Online Befragung zur Berufsfeldanalyse der industriellen Elektroberufe, Bonn
All authors contributed substantially to this work. FW and NL collected the data and wrote the first draft. FW conducted the analyses and processed changes to the manuscript. RN wrote the discussion section and was involved in revising the manuscript critically. All authors read and approved the final manuscript.
This work was supported by the Federal Ministry of Education and Research BMBF [01DB1106].
The authors declare that they have no competing interests.