Open Access

A multidimensional structure of domain-specific problem-solving competencies of electronics technicians for automation technology

Empirical Research in Vocational Education and Training20168:7

DOI: 10.1186/s40461-016-0034-z

Received: 1 March 2016

Accepted: 22 June 2016

Published: 7 July 2016

Abstract

Background

The ASCOT research initiative funded by the Federal Ministry of Education and Research had the objective to measure vocational competencies in high-quality jobs. To adequately assess these it is important to offer authentic situations, similar to real-life (working) situations. Whether domain-specific problem-solving competencies are a unidimensional or a multidimensional construct has far reaching implications in vocational education e.g., regarding valid assessment of performance in the final exam at the end of vocational training. This article deals with the structure of problem-solving competencies of electronics technicians for automation technology at the end of their apprenticeship.

Methods

Realistic problems with the focus on professional activities such as programming (constructive problem-solving) and troubleshooting a programmable logic controller (analytical problem-solving) were employed to measure domain-specific problem-solving competencies. Based on the results structural equation models were calculated and different models are compared for goodness of fit with consideration to content knowledge and fluid intelligence as predictors.

Results

Our results document a two-dimensional structure of domain-specific problem-solving consisting of an analytical and a constructive dimension. This two-dimensional structure remains stable even if content knowledge and fluid intelligence are included as predictors for the domain specific problem-solving competencies (Figs. 1, 5, 6). Only knowledge areas, which are relevant to the problem-solving situation, were predictive for the analytical and constructive problem-solving competencies. Fluid intelligence has a direct effect on both domain-specific problem-solving competencies but the influence is considerably lower when compared to relevant content knowledge.

Conclusion

Based on these findings the final exam at the end of the apprenticeship needs to enclose an appropriate amount of both subdimensions of domain-specific problem-solving competencies. As a practical implication both dimensions have to be supported in addition to the teaching of content knowledge.

Keywords

Analytical problem-solving Constructive problem-solving Multidimensional structure Assessment of competencies

Background

The ASCOT research initiative (technology-based assessment of skills and competencies in VET) was launched by the Federal Ministry of Education and Research (BMBF) in Germany in 2011. It pursues the objective of measuring vocational competencies which are essential for working in high-quality jobs in a changing working world (BMBF 2012). The main focus in this article is on electronics technicians in the field of automation technology, who work for companies which develop, manufacture or install automation solutions in the craft trades field (e.g., electrical installation companies, waterworks) or in the trade and industry sector (e.g., automobile and chemical industry, waste disposal industry). Example of their field of work are: looking after automation systems,1 analyzing functional interconnections and programming automation systems (European Communities 2002).

For the assessment of professional competencies, it is important to offer authentic and complex test situations that are comparable to real-life (working) situations and approachable to practice, improvement and standard setting (Hartig et al. 2008; Shavelson 2010).

Structures of professional competencies are able to show which subdimensions could theoretically and empirically be separated. These subdimensions indicate the potential to solve specific problem-based tasks and are presumably influenced by curriculum-based learning processes and the confrontation with real-life working tasks. Those seem not only important for the respective markedness but also for their interrelation. The development of professional competencies is not limited to development within individual subdimensions, it also focuses on their relations, whose partial differentiations and processes of integrations during apprenticeship were reported (an overview in Seeber and Nickolaus 2010). This explains the fact that statements concerning structures of competencies, based on cross-sectional studies, like those in this article, are only a snapshot. From a practical view, information about structures of competencies at the end of apprenticeship are important for: (1) references that show which working specifications have been achieved, (2) which subdimensions should be assessed in a valid fashion and (3) which development needs are important. Assessing professional knowledge in an industrial-technical field at the end of an apprenticeship nearly always shows multidimensional models, whereas the structure using this knowledge in different problem situations is unidimensional (an overview in Nickolaus and Seeber 2013). This fact is probably influenced by the assessment of domain-specific problem-solving competencies that focused in previous research only on analytical aspects, like the ability of troubleshooting by prospective car mechatronics technicians or electricians for building technology, because these analytical requirements are very relevant in vocational practice (Becker 2005).

An unresolved issue is whether domain-specific problem-solving competencies are unidimensional or multidimensional in the industrial-technical field. We assume, based on task analysis of electronics technicians for automation technology (Zinke et al. 2014), a multidimensional structure of domain-specific problem-solving competencies at the end of the apprenticeship.

State of research

The following chapter gives an overview of the structure of professional competencies in the industrial-technical field with a strong focus on domain-specific problem-solving competencies. Important results of neighboring sectors are included.

Studies about the structure of professional competence in the industrial-technical field

With the change of input to learning outcome orientation, the valid assessment of vocational skills and competencies has become more important in the field of vocational education and training. Such assessments are important for comparisons, for generating explanatory models and evaluating educational programs. Analyses have confirmed two subdimensions across domains: Content knowledge, and the ability to use this knowledge in different problem situations (an overview in Nickolaus and Seeber 2013). Moreover, other differentiations of content knowledge and manual skills as relevant subdimensions are possible.

Operationalization/structure of content knowledge

Modeling content knowledge in the commercial sector was based on case-situations and learning-fields (Rosendahl and Straka 2011), or based on subject discipline (Seeber 2008). By contrast, in the industrial-technical field, modeling content knowledge on the basis of different technical areas is common. These content areas are influenced by prioritization of the curriculum contents. Strong indications of these curriculum-based learning processes are mentioned by Gschwendtner (2011) and Schmidt et al. (2014). There are inconsistent references to modeling content knowledge based on cognitive psychology criteria (declarative and procedural). In the commercial sector, and in the general didactics of technology, declarative and procedural knowledge is empirically separable (Winther 2010; Walker 2013); however, it is not in the industrial-technical field (Geißel 2008; Gschwendtner 2008). This might be due to the operationalization of content knowledge which, in the professional context, always includes declarative knowledge components. The assessment of declarative and procedural knowledge using paper-pencil-tests probably promotes a common scaling of declarative and procedural knowledge.

Operationalization/structure of domain-specific problem-solving competencies

In the industrial-technical field, we define domain-specific problem-solving competencies as the ability to use content knowledge in different problem situations (Nickolaus 2011a). Content knowledge in the industrial-technical field, is by definition, a necessary condition for solving a domain-specific problem. Trainees and craftsmen in the industrial-technical field are confronted, in vocational practice, with analytical and constructive demand situations (Nickolaus 2011a). So far, only analytical aspects have been acquired in previous research: for example, troubleshooting in technical systems. The assessment of such analytical competencies is realized with real-life working tasks and authentic computer simulations (Nickolaus et al. 2009, 2012). Domain-specific analytical problem-solving competencies are defined as finding malfunctions and making proposals to correct these defects (Wiedemann 1995; Abele et al. 2014; Walker et al. 2015, 2016). An unresolved issue in the industrial-technical field is, whether such analytical and constructive problem-solving competencies are empirically separable.

Findings for the occupations of car mechatronics and electronics technicians for building technology

Further studies of car mechatronics and electronics technicians for building technology showed only a unidimensional model of content knowledge at the end of the first year of training (Gschwendtner 2008; Geißel 2008). This structure becomes multidimensional during the process of training: for example, the results at the end of the apprenticeship of electronics technicians for building technology document a three-dimensional structure of content knowledge, which is separated in different technical areas: basic principles of electrical engineering, traditional electrical installation and modern electrical installation with bus technology (Nickolaus et al. 2011). A similar differentiation (six-dimensional structure) is reported by Gschwendtner (2011); this could also be replicated in the field of car mechatronics (Schmidt et al. 2014).

So far, the importance of content knowledge on domain-specific problem-solving has been reported in all presented analyses between these two subdimensions (r = .61 to .86) of professional competencies (e.g., Gschwendtner 2008; Nickolaus et al. 2011). At the end of the apprenticeship of electronics technicians for building technology, the correlations between the subdimensions of content knowledge [traditional electrical installation (1), basic principles of electrical engineering (2) and modern electrical installation with bus technology (3)] and analytical problem-solving competencies (4) show only high correlations (r1,4 = .38; r2,4 = .67; r3,4 = .61) if the knowledge subdimension is theoretically relevant to the concrete problem-solving situation (Nickolaus et al. 2011). This fact leads to the assumption that electronics technicians for building technology primarily make use of basic principles of electrical engineering and modern electrical installation with bus technology when they solve a problem. This kind of “context specificity”, which is relevant for solving concrete situations, is also mentioned in psychological studies (Funke 2003; Jonassen 2000).

Abele et al. (2012) tested the influence of fluid intelligence (Weiß 2006), and content knowledge on the domain-specific analytical problem-solving competencies of car mechatronics and electronics technicians for building technology. The results document that fluid intelligence has only an indirect effect on domain-specific analytical problem-solving competencies (Abele et al. 2012). On the other hand, Petsch et al. (2015) report an direct effect of fluid intelligence on domain-specific problem-solving competencies in the building sector. They assume a great heterogeneity between cognitive basics, and specific test items support this direct effect.

Findings for the occupation of electronics technicians for automation technology

Until last year there had not been any studies which investigated this job profile and there was a lack of instruments for the assessment of professional competencies. New results indicate that it is possible to measure domain-specific constructive problem-solving competencies of electronics technicians for automation technology using a paper-pencil-test, because the test performances with this sort of test are highly comparable (r ≈ .90) to test performances within real-life programming environments (Link and Geißel 2015; Link 2016). Measuring domain-specific analytical problem-solving competencies is also possible using a simulation of an automated system instead of a real system (r ≥ .90). Here domain-specific analytical problem-solving competencies were operationalized by the ability to find a malfunction in an automated system (Walker et al. 2016). With these assessment instruments, it is possible to investigate the structure of domain-specific problem-solving competencies. Walker et al. (2015) document a two-dimensional structure of domain-specific problem-solving consisting of an analytical and a constructive dimension.2 This two-dimensional structure remains stable even if the influence of content knowledge and fluid intelligence was controlled. Additionally, for solving analytical and constructive problems (e.g. troubleshooting or programming a programmable logic controller), only the required dimensions of content knowledge were predictive, which are relevant to the problem-solving situation. Fluid intelligence has only an indirect effect on domain-specific problem-solving, mediated by content knowledge.

Studies about the structure of problem-solving competencies in science education and the influence of fluid intelligence and content knowledge

There is evidence for a multidimensional structure of problem-solving competencies in the PISA study, where multidisciplinary static problem situations (information disclosed to the problem solver is complete) were assessed (Leutner et al. 2012). They define problem-solving competencies as “an individual’s capacity to use cognitive processes to confront and resolve real, cross-disciplinary situations where the solution path is not immediately obvious” (OECD 2004). Multidisciplinary problem situations means, in this context, that these problems are related to mathematics, reading and science, and that, to solve these problems, the content area is not attached to only one single subject area of mathematics, science or reading (OECD 2003). Furthermore, static problem-solving is highly correlated with mathematics (r = .89), science (r = .80) and reading (r = .82) but empirical a standalone construct (Leutner et al. 2012). In addition, there are a direct and an indirect effect of static problem-solving on math (βindirect = .38; βdirect = .12) and science (βindirect = .33; βdirect = .17) at the end of tenth grade (Leutner et al. 2006). Similar results were observed for the fluid intelligence, but the direct and indirect effects on math (βindirect = .19; βdirect = .11) and science competencies (βindirect = .15; βdirect = .09) were marginal (Leutner et al. 2006). Static problem-solving, as assessed in PISA 2003, is a three-dimensional construct, consisting of system analysis and design, trouble-shooting, and decision-making; but these subdimensions are also highly correlated (r = .73–.93) (Leutner et al. 2012).

Studies about domain-specific problem-solving in science education are only partially available. A study by Scherer and Tiemann (2012) in chemistry education is based on the above mentioned PISA framework. They analysed the relationship between different components of problem-solving and strategy knowledge with a virtual environment, where students should solve interactive and static problems (Scherer and Tiemann 2012). At first, a unidimensional structure of chemistry-specific problem-solving was assumed, but this could not be confirmed. Instead, strategy knowledge can be distinguished from problem-solving competencies (Scherer and Tiemann 2012). As covariates for problem-solving competencies, only domain-specific prior knowledge, fluid intelligence, and computer familiarity are significant predictors. The knowledge of problem-solving strategies was not predictive. Overall, the regression-model explained 14.5 % of variance (Scherer and Tiemann 2012).

Research aims

The main research aim is to analyse which structure of domain-specific problem-solving competencies could be supported. Therefore, a theoretical framework was developed based on the reported state of research and is shown in the following Fig. 1.
Fig. 1

Theoretical model of the competence structure of electronics technicians for automation technology at the end of their apprenticeship

There is a direct effect of fluid intelligence on domain-specific analytical and constructive problem-solving competencies (a) and on content knowledge (b). This direct effect of fluid intelligence on content knowledge is only partially in agreement with earlier related research in technical-vocational and science education. On the one hand, some studies in science education document a direct effect of fluid intelligence on domain-specific problem-solving competencies (e.g., Scherer and Tiemann 2012). On the other hand, most studies in industrial-technical fields do not report such an effect (Abele et al. 2012; Nickolaus et al. 2012), with exception of Petsch et al. (2015), who report the influence of fluid intelligence on domain-specific problem-solving competencies in the field of VET. In conclusion, it is unclear if there is a direct effect of fluid intelligence on the domain-specific problem-solving competencies of electronics technicians for automation technology.

Furthermore, content knowledge is separated in different technical areas that are related (c). We suggest three subdimensions for content knowledge on the basis of findings in the field of mechatronics and task analyses of electronics technicians for automation technology. These three subdimensions are: basic principles of electro technology (BP) (e.g., direct and alternating current, safety regulations), electrical engineering (EE) (e.g., drive technology, electrical installation) and automation with programmable logic controllers (AT/PLC).

Additionally, there is a direct effect of these separate context-subdimensions on domain-specific analytical and constructive problem-solving competencies (d), which are related, but standalone (e). The fact that content knowledge is predictive for domain-specific problem-solving competencies was shown in earlier studies in vocational and science education (e.g., Seeber and Nickolaus 2010; Nickolaus et al. 2012; Scherer and Tiemann 2012). In particular, the above-mentioned fact of the task analyses and the findings in the field of interdisciplinary problem-solving suppose a two-dimensional structure of domain-specific problem-solving competencies.

Hypotheses

Based on the previous chapter, the following hypotheses are in the focus:

H1:

Modeling the structure of domain-specific problem-solving shows a correlated, but separable, two-dimensional structure of domain-specific problem-solving, consisting of an analytical and a constructive dimension (H1.1), even if content knowledge and fluid intelligence are included in the model (H1.2).

H2:

Because of the context-specificity of domain-specific problem-solving, only knowledge areas, which are relevant to the problem-solving situation, should be predictive for solving analytical and constructive problems.

H3:

If content knowledge is included in the model, fluid intelligence has only an indirect effect on domain-specific problem-solving, mediated by content knowledge.

Methods

Participants

We tested 211 electronics technicians for automation technology predominantly at the end of their apprenticeship (99.0 % are in the third or fourth year of training). The average age of the participants is M = 20.32 years (SD = 2.11); 90.5 % of the apprentices are male and were doing their apprenticeship in the industrial sector (81.6 %; craft sector = 18.4 %). Most of the trainees (69.2 %) have a basic secondary school certificate (Realschulabschluss). 25.1 % have an upper secondary school leaving certificate (Gymnasium).

Instruments

To ensure the evidence of validity based on the test content, we integrated experts in the field [teachers, trainers and auditors of the Chamber of Industry and Commerce (IHK)] in the development process of the instruments (AERA 2014; Haladyna and Rodriguez 2013). Based on the theoretical framework of professional competencies presented in Fig. 1, we developed instruments for content knowledge, domain-specific analytical and constructive problem-solving competencies. An extensive overview of all instruments can be seen in: van Waveren and Nickolaus (2015); Link and Geißel (2015); Link (2016); Walker et al. (2015, 2016).

Domain-specific analytical and constructive problem-solving competencies

The approach to measuring domain-specific analytical and constructive problem-solving competencies was a set of realistic problems with the focus on professional activities such as programming (constructive problem-solving) and troubleshooting a programmable logic controller (PLC) (analytical problem-solving). The development of the items was based on the structure of the control program based on Benda (2008). Following this approach for both domain-specific problem-solving competencies, items regarding the operating mode, the step chain and output routine were developed. In total, eight troubleshooting scenarios and eight programming scenarios were generated per problem-solving competence. To measure the analytical problem-solving competence, we used a simulation (Fig. 2, Walker et al. 2016). The development process was made in accordance with Funke and Reuschenbach (2011).
Fig. 2

Screenshot of the simulation of an automation system with control panel and PLC

Figure 2 shows, in the lower left corner, a control panel to operate the automation system (e.g., start and stop the system). The current state of the PLC could be seen in top left corner. It is possible to check the current state of all actors and sensors and enlarge selected components via mouse clicks on the right main screen. By clicking on “Simatic” in the main screen, the control program can be opened and observed in real-time processing just like in a real automation system. The reliability of the analytical problem-solving instrument is SEM-Reliability3 = .75.

In the constructive problem scenario, the apprentices had to program a certain part of the control program of a PLC in a complex automation system where parcels were transported by a belt conveyor, measured by sensors and sorted by a pick-and-place unit with a vacuum gripper (Link 2016). Figure 3 shows the logical functions of an item in two common programming languages: the one on the left is more graphically oriented and is called “function block diagram” (FBD); the other, on the right, is more textual and referred to as “statement list”(STL). In this item, the variables and logic gates for a counter and a signal lamp had to be programmed. The reliability of the constructive problem-solving instrument is SEM-Reliability = .83.
Fig. 3

Screenshot of an item in the constructive problem scenario in different programming languages

Content knowledge

In accordance with the theoretical framework of professional competencies, the items of the content knowledge test cover a three-dimensional content specific structure, consisting of the dimension of automation/PLC (AT/PLC), electrical engineering (EE) and basic principles of electro technology (BP) (van Waveren and Nickolaus 2015). The latent correlations between the subdimensions are high (EE and BP is r = .89, between AT/PLC and BP r = .87 and between r = .83) but standalone. The reliability of the test is about EAP/PV-Reliability = .72 across all dimensions (van Waveren and Nickolaus 2015).

Fluid intelligence

Due to the extensive duration of the tests for professional competencies (see below), we measured fluid intelligence only with part one of the CFT 20-R (Weiß 2006), i.e., the subset of continuing logical progressions, classifications, matrices and topologies were administered. The related loss of reliability from rtt = .95 (part one and two together) to rtt = .92 (only part one) seems acceptable considering the reduction of the test duration (Weiß 2006).

Research design and testing procedure

The duration for testing all components of the professional competencies required 2 days. On the first day, the instruments for fluid intelligence, content knowledge and constructive problem-solving competencies were applied (duration 4.5 h). Analytical problem-solving competencies were tested on day two (duration 4.5 h). To avoid position effects, we used a booklet-design (Frey et al. 2009). Analytical and constructive problem-solving competencies were tested in a cross-over-between and within subject design (Ratkowsky et al. 1993; Jones and Kenward 2003).

Statistical procedures

All the latent analyses were obtained by using Mplus. For identification of the structural equation models, we fixed the variance of the latent variable to one, and for models with ordinal items the WLSMV estimators were chosen (Muthén and Muthén 1998–2010). The evaluation of the model fit was based on the following indices: Root Mean Square Error of Approximation below .06 (RSMEA ≤ .06) (Hu and Bentler 1999); Comparative Fit Index about .95 (CFI ≥ .95) (Raykov and Marcoulides 2006); the Chi square degree of freedom-ratio below χ2/df ≤ 2 (Schermelleh-Engel et al. 2003) and the weighted root mean residual below 1.0 (WRMR ≤ 1.0) (Yu 2002). Based on Satorra-Bentler-adjusted χ2-values, we used the DIFF Test option in MPlus to test the dimensionality of the domain-specific problem-solving competencies (Hoyle 2012; Wang and Wang 2012).

Results

As shown in Table 1, the fluid intelligence of the tested participants is slightly higher than the average (standardized to mean 100). The results for content knowledge are EAP/PV estimators.
Table 1

Fluid intelligence and content knowledge of the apprentices (with M = Mean; SD = Standard deviation)

Fluid intelligence

Content knowledge

AT/PLC

EE

BP

M

SD

M

SD

M

SD

M

SD

110.146

14.64

−0.537

0.69

0.110

0.66

0.181

0.88

AT/PLC automation/programmable logic controller, EE electrical engineering, BP basic principles of electro technology

With regard to content knowledge, differences between the three dimensions become obvious. The negative mean4 of the AT/PLC knowledge-dimension indicates that the items of this dimension were too difficult for the apprentices at the end of their training. van Waveren and Nickolaus (2015) were also able to be replicate these results in a larger sample of n = 878 electronics technicians for automation technology at the end of their apprenticeship (van Waveren and Nickolaus 2015; Walker et al. 2016). For the other two dimensions (Table 1, EE and BP), the ease of the items match the mean knowledge of the apprentices.

Following the hypothesis H1.1, we expected that a two-dimensional structure of domain-specific problem-solving competencies would better fit the data than a unidimensional model. Therefore, we compared these models (Table 2 on following page, M1 and M2). The likelihood ratio (LR) test supported our assumption for H1.1 and the two-dimensional model M1 (Table 2, comparison of M1 and M2, (Δχ (1) 2  = 26.69, p < .001). Considering that the latent correlation between analytical and constructive problem-solving is high (r = .77) (Fig. 4) in addition to the result of the LR test, it is evident that analytical and constructive problem-solving competencies have substantial common variance but are empirically separable dimensions.
Table 2

Different structural models of professional competencies of electronics technicians for automation technology at the end of their apprenticeship

Models

χ2/df

Δχ (df) 2

Comparison

RMSEA (90 % CI)

CFI

WRMR

M1: two-dimensional model of problem-solving competencies

1.73

Δχ (1) 2  = 26.69 (p < .001)

M1 and M2

.051 (.040–.062)

.95

0.999

M2: unidimensional model of problem-solving competencies

1.55

.059 (.049–.069)

.93

1.076

M3: integration of content knowledge in M1

1.62

Δχ (4) 2  = 48.22 (p < .001)

M3 and M4

.054 (.044–.063)

.93

1.019

M4: integration of content knowledge M2

1.79

.061 (.052–.070)

.90

1.098

M5: integration of fluid intelligence in M3

1.48

Δχ (5) 2  = 39.82 (p < .001)

M5 and M6

.048 (.039–.056)

.92

0.997

M6: integration of fluid intelligence in M4

1.59

.053 (.045–.061)

.91

1.063

Δχ2 = Difference value of the Satorra-Bentler adjusted χ2-values

Fig. 4

M1 two-dimensional model of domain-specific problem-solving. M1: n = 211, χ2/df = 1.73, CFI = .95, RMSEA = .051 [.040-.062], WRMR = .999. All factor loadings and correlations are significant (p < .001)

To test the hypothesis H1.2, we included the manifest person ability scores of the three dimensions of the content knowledge as predictors in the two-dimensional and unidimensional model of domain-specific problem-solving (Table 2, M3 and M4).5 Analogously to hypothesis H1.1, we compared the fit of the two-dimensional with a unidimensional model of domain-specific problem-solving. And again, the results of the LR test supported the two-dimensional model of domain-specific problem-solving M3 (Table 2, comparison of M3 and M4, Δχ (4) 2  = 48.22, p < .001). After the integration of the three dimensions of the content knowledge, the latent correlation between analytical and constructive problem-solving declined from r = .77 to r = .62, which highlights the need of differentiation between these two dimensions of domain-specific problem-solving (Fig. 5).
Fig. 5

M3 two-dimensional model of domain-specific problem-solving after the integration of the manifest person ability scores of the content knowledge. M3: n = 211, χ2/df = 1.62, CFI = .93, RMSEA = .054 [.044–.063], WRMR = 1.019. All factor loadings and correlations are significant (p < .001). Dashed lines are not significant factor loadings

Finally, to test hypothesis H1.2, we included fluid intelligence in the models M3 and M4 and compared the fit using the LR test. Despite the integration of content knowledge and fluid intelligence, the two-dimensional model of domain-specific problem-solving fitted the data better after all (Table 2, comparison of M5 and M6, Δχ (5) 2  = 39.82, p < .001). The results of the LR test supported our hypothesis H1.2 and pointed out the stability and the distinctiveness of the analytical and constructive problem-solving competencies.

Our second research question affected the influence of the dimensions of content knowledge on the domain-specific problem-solving competencies. It was assumed, that for solving analytical and constructive problems (e.g. troubleshooting or programming a PLC), only the required dimensions of content knowledge were predictive, which are relevant to the problem-solving situation (e.g. the content knowledge Dimension AT/PLC and not BP) (H2). As Figs. 4 and 5 showed, only the dimension of automation and programmable logic controller (AT/PLC) was predictive for solving analytical and constructive problems. The dimension of AT/PLC explained 35 % of the variance in the constructive, and 46 % of the variance in the analytical problem-solving competency (Fig. 5). These explanation patterns also remained stable after the integration of fluid intelligence into the structural model (Fig. 6). Induced through the addition of the fluid intelligence, the explained variances of the constructive (40 %) and analytical (51 %) problem-solving competencies increased slightly.
Fig. 6

Structural model of professional competencies of electronics technicians for automation technology at the end of their apprenticeship (Table 2, model M5). M5: n = 211, χ2/df = 1.48, CFI = .92, RMSEA = .048 [.039–.056], WRMR = .997. All factor loadings und correlations are significant (p < .001). Dashed lines are not significant factor loadings

Referring to our third research question (H3), fluid intelligence had, not as assumed, a significant direct path to domain-specific analytical and constructive problem-solving competencies. This finding is in line with the results of Fischer et al. (2015). Furthermore, Fig. 6 showed that the influence of fluid intelligence compared to the influence of the dimension AT/PLC on the problem-solving dimensions is noticeably lower.

Discussion and conclusion

The results document a two-dimensional structure of domain-specific problem-solving competencies, which is represented through both an analytical and a constructive subdimension. These subdimensions remain stable even if fluid intelligence and the three-dimensional content knowledge were integrated. In view of the fact that these two domain-specific problem-solving dimensions are empirically separable but correlated, these results have implications for designing the final examination at the end of the apprenticeship. To ensure the evidence for validity based on test-content of the final exam, both subdimensions of domain-specific problem-solving competencies have to be included in an appropriate amount. The high influence of context-specific knowledge dimensions on analytical and constructive problems emphasises the importance of content knowledge for domain-specific problem-solving. Practical implications for teaching structures can also be drawn from the two-dimensional domain-specific problem-solving structure in a manner that both dimensions have to be supported during the teaching process. Furthermore, it is important to ensure that the context-specific knowledge has also been taught. In perspective, the simulation for measuring domain-specific analytical problem-solving could also be integrated in the teaching process. As reported, it is possible to represent industrial programmable logic devices and automation systems via the simulation authentically. Despite of the correlation between the analytical and constructive problem-solving competencies and the similar context (problem-solving in an industrial automation system), the underlying process for solving analytical and constructive problems seems to be disjunctive.

With regard to the transferability of the results concerning the two-dimensional structure of domain-specific problem-solving into other vocational domains, a differentiated consideration is necessary. We assume that the structure of domain-specific problem-solving competence depends on the domain-specific tasks. A replication of these results in the occupation of car mechatronics might be doubtful, because of the smaller range of domain-specific tasks concerning constructive problem-solving competencies. For vocational domains which are more closely related to the occupation of electronics technicians for automation technology, e.g. mechatronic technicians or electronics technicians for industrial engineering, a two-dimensional structure of domain-specific problem-solving seems more likely.

For the occupation of electronics technicians for automation technology at the end of their apprenticeship, fluid intelligence was primarily responsible for the gain of content knowledge. In contrast to our assumption, fluid intelligence was also predictive for solving domain-specific analytical and constructive problems, which is in line with findings of static problem-solving competence in the PISA study.

For further research we see, inter alia, the following desiderata:
  1. a.

    A replication-study in which the domain-specific analytical problem-solving competence covers not only troubleshooting a programmable logic controller (software) but also include troubleshooting in hardware (e.g., broken conductors and faulty sensors or actuators). Additionally a replication of the (two-dimensional) structure for problem-solving in vocational domains (e.g., mechatronic technicians) is also indicated.

     
  2. b.

    How far the actual design of the final exam at the end of the apprenticeship in the occupation of electronics technicians for automation technology is suited to provide reliable and valid measures of professional competencies remains unclear. The recent research suggests, that there are substantial variances in the final exams (van Waveren and Nickolaus 2015; Abele 2014; Nickolaus 2011b).

     
  3. c.

    For the development and optimization of approaches which support domain-specific problem-solving competencies, detailed knowledge about the underlying processes during analytical and constructive problem-solving is required. Interesting points of reference for such process analyses and approaches are provided by the researchers Konradt (1995), Schaper (2000), Hochholdinger et al. (2008) and Hochholdinger and Schaper (2013).

     
Footnotes
1

Providing a system within automation is for management and supervision of a technical process. Such a system consists of sensors, actuators and processing units: for example, a programmable logic controller (PLC), or software that delivers the program flow.

 
2

The basis for this structure were task analyses in the field of vocational practice of the Federal Institute for Vocational Education and Training (BIBB). Zinke et al. (2014) document here two main job assignments: maintenance, and bringing automation systems into service. The ability to program a plc in such systems is very important. Furthermore, the structure is supported by international VET research (feasibility study for a large scale assessment), where “diagnosis of malfunctions (troubleshooting), problem solving and putting systems back into operation” and “programming and handling of bus technologies” are of major importance (Baethge and Arends 2009).

 
3

This type of reliability is indicated due to the categorical indicators of the instrument (Green and Yang 2009, 164). The reliability was calculated using the methods of Green and Jang (2009, formula 21).

 
4

The knowledge scales are logarithmic and their means are zero.

 
5

The high correlations between the dimensions occurs because the manifest EAP estimation are shrunken towards each other. This overestimation should not be interpreted (Chen et al. 1998).

 

Declarations

Authors’ contributions

All authors contributed substantially to this work. FW and NL collected the data and wrote the first draft. FW conducted the analyses and processed changes to the manuscript. RN wrote the discussion section and was involved in revising the manuscript critically. All authors read and approved the final manuscript.

Acknowledgements

This work was supported by the Federal Ministry of Education and Research BMBF [01DB1106].

Competing interests

The authors declare that they have no competing interests.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors’ Affiliations

(1)
Technical University of Kaiserslautern
(2)
Vocational School Teacher
(3)
Department of Vocational, Economic and Technical Education, University of Stuttgart

References

  1. Abele S (2014) Modellierung und Entwicklung berufsfachlicher Kompetenz in der gewerblich-technischen Ausbildung. Franz Steiner, StuttgartGoogle Scholar
  2. Abele S, Greiff S, Gschwendtner T, Wüstenberg S, Nickolaus R, Nitzschke A, Funke J (2012) Dynamische Problemlösekompetenz. Z Erziehwiss 15(2):363–391. doi:10.1007/s11618-012-0277-9 View ArticleGoogle Scholar
  3. Abele S, Walker F, Nickolaus R (2014) Zeitökonomische und reliable Diagnostik beruflicher Problemlösekompetenzen bei Auszubildenden zum Kfz-Mechatroniker. Z Pädag Psychol 28(4):167–179. doi:10.1024/1010-0652/a000138 Google Scholar
  4. AERA (2014) Standards for educational and psychological testing. American Educational Research Association, WashingtonGoogle Scholar
  5. Baethge M, Arends L (eds) (2009) Feasibility study VET-LSA: a comparative analysis of occupational profiles and VET programmes in 8 European countries—international report. Vocational Training Research volume 8. bmbf, Bonn, Berlin
  6. Becker M (2005) Einbindung von Facharbeiterkompetenzen in IKT-dominante Diagnoseabläufe im Kfz-Service. In: Pangalos J, Spöttl G, Kutzen N, Howe F (eds) Informatisierung von Arbeit, Technik und Bildung: Eine berufswissenschaftliche Bestandsaufnahme. Lit, Münster, pp 45–54Google Scholar
  7. Benda D (2008) Das große Handbuch Fehlersuche in elektronischen Schaltungen: Lesen und Auswerten von Schaltungsunterlagen, Fehlersuche mit Methode, Messen und Prüfen mit dem Oszilloskop, Sonderausg. Franzis Elektronik. Franzis, Poing
  8. BMBF (2012) Vocational skills and competencies made visible: the ASCOT research initiative. Federal Ministry of Education and Research (BMBF), Training Policy Division, Bonn, Berlin
  9. Chen S-K, Hou L, Dodd BG (1998) A Comparison of maximum likelihood estimation and expected a posteriori estimation in cat using the partial credit model. Educ Psychol Measur 58(4):569–595View ArticleGoogle Scholar
  10. European Communities (2002) Certificate Supplement. https://www.bibb.de/tools/berufesuche/index.php/certificate_supplement/en/elektroniker_automatisierungstechnik_e.pdf. Accessed 5 Dec 2015
  11. Fischer A, Greiff S, Wüstenberg S, Fleischer J, Buchwald F, Funke J (2015) Assessing analytic and interactive aspects of problem solving competency. Learn Individ Differ 39:172–179View ArticleGoogle Scholar
  12. Frey A, Hartig J, Rupp AA (2009) An NCME instructional module on booklet designs in large-scale assessments of student achievement: theory and practice. Educ Meas Issues Practice 28(3):39–53. doi:10.1111/j.1745-3992.2009.00154.x View ArticleGoogle Scholar
  13. Funke J (2003) Problemlösendes Denken, 1. Aufl. Kohlhammer Standards Psychologie, 1st edn. Verlag W. Kohlhammer, StuttgartGoogle Scholar
  14. Funke J, Reuschenbach B (2011) Einsatz technischer Mittel in der psychologischen Diagnostik. In: Hornke LF, Amelang M, Kersting M (eds) Leistungs-, Intelligenz- und Verhaltensdiagnostik, [Vollst. Neuausg.]. Hogrefe, Göttingen [u.a.], pp 595–631Google Scholar
  15. Geißel B (2008) Ein Kompetenzmodell für die elektrotechnische Grundbildung: Kriteriumsorientierte Interpretation von Leistungsdaten. In: Nickolaus R, Schanz H (eds) Didaktik der gewerblich-technischen Berufsbildung: Konzeptionelle Entwürfe und empirische Befunde, vol 9. Schneider Verlag Hohengehren, Baltmannsweiler, pp 121–141Google Scholar
  16. Green SB, Yang Y (2009) Reliability of summed item scores using structural equation modeling: an alternative to coefficient alpha. Psychometrika 74(1):155–167. doi:10.1007/s11336-008-9099-3 View ArticleGoogle Scholar
  17. Gschwendtner T (2008) Ein Kompetenzmodell für die kraftfahrzeugtechnische Grundbildung. In: Nickolaus R, Schanz H (eds) Didaktik der gewerblich-technischen Berufsbildung: Konzeptionelle Entwürfe und empirische Befunde, vol 9. Schneider Verlag Hohengehren, Baltmannsweiler, pp 103–120Google Scholar
  18. Gschwendtner T (2011) Die Ausbildung zum Kraftfahrzeugmechatroniker im Längsschnitt: Analyse zur Struktur von Fachkompetenz am Ende der Ausbildung und Erklärung von Fachkompetenzentwicklung über die Ausbildungszeit. Zeitschrift für Berufs- und Wirtschaftspädagogik - Beihefte (ZBW-B)(25):55–76
  19. Haladyna TM, Rodriguez MC (2013) Developing and validating test items. Routledge, New YorkGoogle Scholar
  20. Hartig J, Klieme E, Leutner D (2008) Assessment of competencies in educational contexts. Hogrefe & Huber, CambridgeGoogle Scholar
  21. Hochholdinger S, Schaper N (2013) Training troubleshooting skills with an anchored instruction module in an authentic computer based simulation environment. J Tech Educ 1(1):7–22Google Scholar
  22. Hochholdinger S, Rowold J, Schaper N (eds) (2008) Evaluation und Transfersicherung betrieblicher Trainings: Modelle, Methoden und Befunde. Wirtschaftspsychologie. Hogrefe, Göttingen [u.a.]
  23. Hoyle RH (2012) Handbook of structural equation modeling. Guilford Press, New YorkGoogle Scholar
  24. Hu L, Bentler PM (1999) Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Model Multidiscip J 6(1):1–55. doi:10.1080/10705519909540118 View ArticleGoogle Scholar
  25. Jonassen D (2000) Toward a design theory of problem solving. ETR&D 48(4):63–85. doi:10.1007/BF02300500 View ArticleGoogle Scholar
  26. Jones B, Kenward MG (2003) Design and analysis of cross-over trials. In: Monographs on statistics and applied probability, Bd. 98, 2nd edn. Chapman & Hall/CRC, Boca Raton
  27. Konradt U (1995) Strategies of failure diagnosis in computer-controlled manufacturing systems: empirical analysis and implications for the design of adaptive decision support systems. Int J Hum Comput Stud 43(4):503–521. doi:10.1006/ijhc.1995.1057 View ArticleGoogle Scholar
  28. Leutner D, Fleischer J, Wirth J (2006) Problemlösekompetenz als Prädiktor für zukünftige Kompetenz in Mathematik und in den Naturwissenschaften. In: Prenzel M (ed) PISA 2003: Untersuchungen zur Kompetenzentwicklung im Verlauf eines Schuljahres. Waxmann, Münster, München [u.a.], pp 119–138
  29. Leutner D, Fleischer J, Wirth J, Greiff S, Funke J (2012) Analytische und dynamische Problemlösekompetenz im Lichte internationaler Schulleistungsvergleichsstudien. Psychologische Rundschau 63(1):34–42. doi:10.1026/0033-3042/a000108 View ArticleGoogle Scholar
  30. Link N (2016) Problemlösen bei der Programmierung von speicherprogrammierbaren Steuerungen in komplexen automatisierten Systemen. Dissertation, Pädagogische Hochschule
  31. Link N, Geißel B (2015) Konstruktvalidität konstruktiver Problemlösefähigkeit bei Elektronikern für Automatisierungstechnik. Zeitschrift für Berufs- und Wirtschaftspädagogik 111(2):208–221Google Scholar
  32. Muthén LK, Muthén BO (1998–2010) Mplus user’s guide, 6th edn. Muthén & Muthén, Los Angeles
  33. Nickolaus R (2011a) Die Erfassung fachlicher Kompetenzen und ihrer Entwicklungen in der beruflichen Bildung – Forschungsstand und Perspektiven. In: Zlatkin-Troitschanskaia O (ed) Stationen Empirischer Bildungsforschung: Traditionslinien und Perspektiven, 1, neue Ausg. VS Verlag für Sozialwissenschaften, Wiesbaden, pp 331–351View ArticleGoogle Scholar
  34. Nickolaus R (2011b) Kompetenzmessung und Prüfungen in der beruflichen Bildung. Zeitschrift für Berufs- und Wirtschaftspädagogik (ZBW) 107(2):161–173Google Scholar
  35. Nickolaus R, Seeber S (2013) Berufliche Kompetenzen: Modellierungen und diagnostische Verfahren. In: Frey A, Lissmann U, Schwarz B (eds) Handbuch Berufspädagogische Diagnostik, 1, neue Ausg. Beltz, Weinheim, pp 155–180Google Scholar
  36. Nickolaus R, Gschwendtner T, Abele S (2009) Die Validität von Simulationsaufgaben am Beispiel der Diagnosekompetenz von Kfz-Mechatronikern: Vorstudie zur Validität von Simulationsaufgaben im Rahmen eines VET-LSA, Stuttgart
  37. Nickolaus R, Geißel B, Abele S, Nitzschke A (2011) Fachkompetenzmodellierung und Fachkompetenzentwicklung bei Elektronikern für Energie- und Gebäudetechnik im Verlauf der Ausbildung - Ausgewählte Ergebnisse einer Längsschnittstudie. In: Nickolaus R, Pätzold G (eds) Lehr-Lernforschung in der gewerblich-technischen Berufsbildung. Steiner, Stuttgart, pp 77–94Google Scholar
  38. Nickolaus R, Abele S, Gschwendtner T, Nitzschke A, Greiff S (2012) Fachspezifische Problemlösefähigkeit als zentrale Kompetenzdimension beruflicher Handlungskompetenz - Modellierung, erreichte Niveaus und relevante Einflussfaktoren in der gewerblich-technischen Berufsausbildung. Zeitschrift für Berufs- und Wirtschaftspädagogik 108(3):243–272Google Scholar
  39. OECD (2003) PISA The PISA 2003 assessment framework: mathematics, reading, science and problem solving knowledge and skills. OECD Pub., [S.l.], ParisGoogle Scholar
  40. OECD (2004) Problem solving for tomorrow’s world: first measures of cross-curricular competencies from PISA 2003. OECD Publishing, ParisGoogle Scholar
  41. Petsch C, Norwig K, Nickolaus R (2015) Berufsfachliche Kompetenzen in der bautechnischen Berufsgrundbildung - Strukturen, erreichte Niveaus und relevante Einflussfaktoren. In: Rausch A, Warwas J, Seifried J, Wuttke E (eds) Konzepte und Ergebnisse ausgewählter Forschungsfelder der beruflichen Bildung – Festschrift zum 65. Geburtstag von Detlef Sembill. Schneider Verlag Hohengehren, Baltmannsweiler
  42. Ratkowsky DA, Evans MA, Alldredge JR (1993) Cross-over experiments: design, analysis, and application. In: Statistics, textbooks and monographs, vol 135. Marcel Dekker, New York
  43. Raykov T, Marcoulides GA (2006) A first course in structural equation modeling, 2nd edn. Lawrence Erlbaum Associates Publishers, MahwahGoogle Scholar
  44. Rosendahl J, Straka GA (2011) Analysen zur wirtschaftlichen Fachkompetenz angehender Bankkaufleute. Zeitschrift für Berufs - und Wirtschaftspädagogik 107(2):190–217Google Scholar
  45. Schaper N (2000) Gestaltung und evaluation arbeitsbezogener Lernumgebungen. Ruprecht-Karls-Universität Heidelberg, HeidelbergGoogle Scholar
  46. Scherer R, Tiemann R (2012) Factors of problem-solving competency in a virtual chemistry environment: the role of metacognitive knowledge about strategies. Comput Educ 59(4):1199–1214. doi:10.1016/j.compedu.2012.05.020 View ArticleGoogle Scholar
  47. Schermelleh-Engel K, Moosbrugger H, Müller H (2003) Evaluating the fit of structural equation models: tests of significance and descriptive goodness-of-fit measures. Methods Psychol Res 8(2):23–74Google Scholar
  48. Schmidt T, Nickolaus R, Weber W (2014) Modellierung und Entwicklung des fachsystematischen und handlungsbezogenen Fachwissens von Kfz-Mechatronikern. Zeitschrift für Berufs- und Wirtschaftspädagogik (ZBW) 110(4):549–574Google Scholar
  49. Seeber S (2008) Ansätze zur Modellierung beruflicher Fachkompetenz in kaufmännischen Ausbildungsberufen. Zeitschrift für Berufs - und Wirtschaftspädagogik 104(1):74–97Google Scholar
  50. Seeber S, Nickolaus R (2010) Kompetenz, Kompetenzmodelle und Kompetenzentwicklung in der beruflichen Bildung. In: Pätzold G, Reinisch H, Nickolaus R (eds) Handbuch Berufs- und Wirtschaftspädagogik, 1st edn. UTB, Stuttgart, pp 95–112Google Scholar
  51. Shavelson RJ (2010) On the measurement of competency. Empir Res Vocat Educ Train 1(2):41–63Google Scholar
  52. van Waveren L, Nickolaus R (2015) Struktur- und Niveaumodell des Fachwissens bei Elektronikern für Automatisierungstechnik am Ende der Ausbildung. J Tech Educ 3(2):62–91Google Scholar
  53. Walker F (2013) Der Einfluss von Handlungsmöglichkeiten auf den Wissenserwerb bei der Durchführung technischer Experimente. Universität Duisburg-Essen
  54. Walker F, Link N, Nickolaus R (2015) Berufsfachliche Kompetenzstrukturen bei Elektronikern für Automatisierungstechnik am Ende der Berufsausbildung. Zeitschrift für Berufs- und Wirtschaftspädagogik (ZBW) 111(2):222–241Google Scholar
  55. Walker F, Link N, van Waveren L, Hedrich M, Geißel B, Nickolaus R (2016) Berufsfachliche Kompetenzen von Elektronikern für Automatisierungstechnik: Kompetenzdimensionen, Messverfahren und erzielte Leistungen (KOKO EA). In: Beck K, Landenberger M, Oser F (eds) Technologiebasierte Kompetenzmessung in der beruflichen Bildung: Ergebnisse aus der BMBF-Förderinitiative ASCOT. Bertelsmann, Bielefeld, pp 139–169Google Scholar
  56. Wang J, Wang X (2012) Structural equation modeling: applications using Mplus. In: Wiley series in probability and statistics. Wiley, Higher Education Press, Chichester, West Sussex
  57. Weiß RH (2006) Grundintelligenztest Skala 2 - Revision - (CFT 20-R). Hogrefe, GöttingenGoogle Scholar
  58. Wiedemann J (1995) Ermittlung von Qualifizierungsbedarf: Am Beispiel der Störungsdiagnose in der flexiblen Fertigung. Waxmann, Münster, New YorkGoogle Scholar
  59. Winther E (2010) Kompetenzmessung in der beruflichen Bildung. W. Bertelsmann Verlag, BielefeldGoogle Scholar
  60. Yu C (2002) Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes. University of California, OaklandGoogle Scholar
  61. Zinke G, Schenk H, Kröll J (2014) Ergebnisse einer Online Befragung zur Berufsfeldanalyse der industriellen Elektroberufe, Bonn

Copyright

© The Author(s) 2016