Evaluating the pedagogical content knowledge of pre- and in-service teachers of business and economics to ensure quality of classroom practice in vocational education and training

Abstract

Background

Research on teachers’ professional knowledge and skills is becoming increasingly important. However, valid assessments of teachers’ knowledge and skills are scarce. In this study, we report on our development and validation (following the international Standards for Educational and Psychological Testing) of a situation-based tool to assess pre-service and in-service teachers’ pedagogical content knowledge (PCK) of business and economics in vocational education and training.

Methods

We present a conceptual model and empirical evidence indicating the validity of test content and internal structure of the new developed assessment. We conducted conceptual analyses, interviews and standardized ratings with experts, and piloted the tool with 338 pre-service and in-service teachers.

Results

Our results indicate that our assessment tool is valid and reliable to measure case-related PCK (a central component of situation-oriented knowledge) of pre-service and in-service teachers.

Conclusions

A video- and performance-based assessment tool for assessing the strategic component of PCK is developed and currently being validated. Together, these tools can serve to assess the professional knowledge and skills of business and economics teachers.

Relevance and background

Empirical research on teachers’ professional knowledge and skills is becoming increasingly important for practice in teacher education and especially in vocational education and training (De Wever et al. 2016). Several debates on educational policies have ensued, and discourse on this topic has featured political catchphrases such as ‘outcome orientation’ (Cochran-Smith et al. 2012; European Commission 2007). The central theme of the discussion on outcome orientation is that both the educational system and teaching can become more successful by focusing less on curricula and teaching objectives, and more on empirically assessed outcomes, such as student learning and resulting knowledge and skills. This also applies to the domain of teacher education.

Empirical findings on the acquisition and promotion of professional knowledge and skills of pre-service and in-service teachers provide indications about the quality and effectiveness of teacher education programs (Blömeke et al. 2011; Fritsch et al. 2015; Kleickmann et al. 2012), which will affect instructional quality and, ultimately, student achievement (Baumert et al. 2010; Hill et al. 2005). Preliminary empirical research in this area, especially on teaching science and mathematics (e.g., Rohaan et al. 2009) has shown that assessments of professional knowledge and skills during teacher education and professional practice require appropriate field-specific assessment instruments, as they determine the quality of analyses and results and, therefore, their consideration in educational policies and practices (Mislevy and Haertel 2006; Zlatkin-Troitschanskaia et al. 2015). However, objective, reliable, and valid assessments of teachers’ knowledge and skills across institutions are still scarce in most countries today (Darling-Hammond et al. 2013).

It is undisputed that teachers’ professional knowledge is key to their professional performance (Desimone 2009; De Wever et al. 2016). Assessment of teachers’ professional knowledge and skills often is based on Shulman’s (1986) description of teacher knowledge, which comprises general pedagogical knowledge, subject matter knowledge, curricular knowledge, and pedagogical content knowledge (PCK). Various studies have investigated teachers’ general pedagogical knowledge (Gold and Holodynski 2015; König et al. 2011); however, domain-specific facets of teachers’ professional knowledge have been assessed in only a few subjects, mainly in mathematics (see review in Depaepe et al. 2013) and science (de Jong and van Driel 2004; Schmelzing et al. 2013; Veal et al. 1999). The lack of assessment of teacher knowledge, particularly PCK, in subjects with more complex curricular layouts may be due to challenges of agreeing on common teaching methods and content structures (Carlson 1990).

This is particularly an issue in the field of business and economics. On the one hand, business and economics is one of the most popular fields of study in vocational education and training (Organisation for Economic Co-operation and Development 2014; German Federal Statistical Office 2014). Economic and financial literacy is becoming increasingly important at all stages of education as well as of professional and social life, not least because of the need to better understand and handle the consequences of momentous economic events such as the recent financial crisis (Blinder 2015; Council for Economic Education [CEE] 2011). Accordingly, business and economics has also been introduced as a school subject in several countries, including the United States and Germany. On the other hand, in secondary education, business and economics is still rarely taught systematically or may be organized very flexibly, for example, according to fields of professional practice in commercial vocational education in Germany. Hence, from the point of view of teacher education, teachers of business and economics need to be prepared for a rather diverse set of content areas, teaching methods, and contexts, and be able to specialize depending on their group of learners and type of school. This diversity poses great challenges to standardized assessment of teachers’ domain-specific knowledge. In this paper, we use the example of PCK in business and economics to show how these challenges can be addressed, step by step, in the development of tools for assessing teachers’ PCK.

First, we present the current state of research on the valid assessment of teachers’ PCK of business and economics. Next, we present our newly developed assessment instrument and discuss its strengths and limitations, including evidence of its reliability and validity for assessing case-related PCK. Finally, we describe a follow-up study currently under way, which validates a complementary performance-based tool to assess the strategic component of PCK of business and economics using video vignettes, and discuss uses of both tools and implications for valid assessment of the professional knowledge of business and economics teachers.

Current state of research on valid assessment of pre- and in-service teachers’ PCK of business and economics

Recent research indicates that success in studying business and economics depends significantly on students’ prior knowledge of business and economics acquired at secondary or vocational schools (Brückner et al. 2015; Gill and Gratton-Lavoie 2011; Walstad 2013). While this prior knowledge can be appropriately assessed, teachers’ PCK of business and economics, which shapes the quality of instruction and likely influences student learning, has not yet been assessed directly, largely due to a lack of appropriate instruments for valid assessment (Zlatkin-Troitschanskaia et al. 2015).

In addition to curricular and subject matter knowledge, teachers’ PCK is a crucial part of their domain-specific professional knowledge (Baxter and Lederman 1999; Berry et al. 2008). PCK is recognized widely as an important indicator of students’ learning outcomes (Grossman et al. 1989; Hill et al. 2005). According to Shulman (1986), PCK includes ‘the ways of representing and formulating the subject that make it comprehensible to others’, and ‘an understanding of what makes the learning of specific topics easy or difficult’ (p. 9). Therefore, appropriate tools to analyze teachers’ PCK should have a strong orientation towards practical situations of everyday classroom teaching of the particular subjects.

Although there are some internationally validated instruments to assess pre-service and in-service teachers’ knowledge of business and economics (Walstad et al. 2007; Förster et al. 2015), instruments to assess teachers’ PCK of business and economics are not yet available internationally, or focused on accounting (Fritsch et al. 2015). The Mexican Higher Education Exit Assessments Test (EGEL) on accounting education by the National Center for the Evaluation of Higher Education is used nationally to assess learning outcomes of pre-service accounting teachers (Vidal Uribe 2013). In some states in the United States, pre-service teachers must pass the Praxis II test on business education to obtain the qualification to teach business administration (Educational Testing Service [ETS] 2011). However, this test assesses predominantly subject matter knowledge and some general pedagogical aspects rather than subject-specific requirements for classroom teaching of business and economics. In this paper, we present a newly developed and validated, situation-based instrument to assess pre-service and in-service teachers’ PCK of business and economics.

Aim of the study

The aim of this study is to enhance teacher training in business and economics and to advance early-stage research on teachers’ PCK. We address conceptual, methodological, and practical challenges of reliable and valid assessment of PCK, including how to systematically design a test, how to confirm criteria of psychometric quality, and how to undertake comprehensive validation (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education [AERA, APA, and NCME] 2014). Regarding validation criteria, we present evidence of the two criteria of test content and internal structure that are fundamental for all tests. Our validation methods include conceptual analysis, interviews with and ratings by experts, as well as a quantitative assessment of pre-service and in-service business and economics teachers.

Validating the assessment of PCK of business and economics

Validation criteria

Test validity is ‘the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests’ (AERA et al. 2014, 11). Very early in the test development process, developers must not only define the construct to be assessed, but also create a validity framework that aligns theoretical and empirical evidence of the test scores and the conclusions to be drawn from them (Pellegrino et al. 2001; Kane 2006).

In our validation analyses, we examined the extent to which the theoretical construct explained the test scores and the underlying item response processes. Following the international Standards for Educational and Psychological Testing (AERA et al. 2014), we built a comprehensive validity argument based on theoretical and empirical evidence of (1) test content, (2) response processes, (3) internal structure, and (4) relationships to other variables. While our validation process addressed all four criteria and related hypotheses, in this paper, we focus on the criteria of test content and internal structure (e.g., Carlton 2012; Knupp and Harris 2012; Wilson 2005), which provide a vital indication of the test’s usability in business and economics teacher education and professional teaching practice. Analyses of response processes and relationships to other variables have been reported in Kuhn (2014) and Kuhn et al. (2013, 2014), and are summarized at the end of the paper.

Conceptual framework for PCK of business and economics

Theoretical model of the structure of PCK

All validity analyses should begin with a clear construct definition. Our construct was based on Shulman’s definition of PCK (Shulman 1986) and a general consideration of modeling and measurement of knowledge and practical skills (Shavelson 2013). Our multi-perspective analyses involved a thorough examination of similar studies in other fields to maintain compatibility with other PCK assessment instruments (e.g., Blömeke et al. 2011; Rohaan et al. 2009; Schmelzing et al. 2013). General standards for teaching and teacher education, guidelines and curricula for business and economics teacher education and professional teaching practice, and international business and economics content standards were examined (e.g., National Council for Accreditation of Teacher Education 2008, Standard 1b; Council for Economic Education [CEE] 2010).

The theoretical construct of knowledge generally consists of two facets: content structure and cognitive structure (Bromme 1995; Shulman 1986). Hence, our comprehensive theoretical model of the construct of PCK of business and economics specified (1) the content structure with regard to subject-specific areas of teaching knowledge and (2) the cognitive structure in terms of types of knowledge and corresponding cognitive processes (see Fig. 1).

Fig. 1

Theoretical model of the structure of PCK (Kuhn 2014, p. 105)

First, consistent with similar studies in other domains (e.g., van Driel et al. 1998), our model defines PCK as consisting of two content components. The first content component is knowledge of teaching objectives, content, and methods, although others have also referred to a narrower category of knowledge of instructional representations. The interdependence of teaching objectives, content, and methods represents the traditional European discourse on didactics (e.g., Bromme 1995; Hopmann and Riquarts 1995). A teaching objective for business and economics, for example, might be to teach students to weigh decisions by considering economic and ecological factors; content might refer to the area of sales; and an appropriate method might be a role-play. These may refer to a mid-term perspective (i.e., with respect to curriculum) and also to a short-term perspective (i.e., with respect to lesson planning). The second content component is knowledge of students’ learning process, also referred to as knowledge of learning difficulties, which includes awareness of common errors students make and their causes. Both content components are embedded in a broader context, which, for the vocational sector specifically, is characterized by a high heterogeneity (i.e., with respect to students’ prior knowledge), among other things (Kuhn 2014).

Second, our comprehensive theoretical model characterizes the cognitive structure of PCK of business and economics in accordance with Shulman’s (1986) differentiation of three types of teacher knowledge: propositional, case, and strategic knowledge. Following Anderson and Krathwohl (2001; Shulman 1987), we specified types of knowledge with regard to underlying cognitive processes, organized hierarchically according to complexity. Propositions present complex ideas in an abstract, simplified way. Thus, propositional PCK presents general principles of effective subject-specific teaching. However, propositional PCK is detached from specific contexts, providing little indication as to how it can be applied in practical teaching situations (Shulman 1986). In our model, we associated teachers’ propositional knowledge with the cognitive processes of remembering and understanding. Thus, to demonstrate this most basic level of knowledge, teachers need to remember and understand various general propositions about teaching and learning, such as knowledge of subject-related teaching principles and their significance for teaching, knowledge of certain key areas of business and economics teaching, or knowledge of student difficulties and their potential causes (Shulman 1987).

To be able to apply general propositions in specific contexts, a second type of knowledge is needed. For Shulman (1986), case knowledge is embedded in context-specific instances of practice. In our model, we linked teachers’ case knowledge to the cognitive processes of analyzing, applying, and creating. We assume that case knowledge enables teachers to analyze and apply general propositions about teaching, learning, and subject matter in this field to specific business and economics teaching contexts and to create their own solutions for their lessons (for example, the didactic principle of conflict-orientation can be represented in a vocational class in retail business by a typical argument acted out by students playing the salesperson and the customer).

Teachers must be able to effectively handle many complex classroom situations that might not be comparable to previous cases and may require choosing among different, even contradictory, teaching principles (Fenstermacher 1994). This requires not only general propositional knowledge and context-specific case knowledge, but also strategic knowledge. Strategic knowledge enables teachers to evaluate and weigh different propositions and cases and to act flexibly in varying contexts, also described as wisdom of practice (Shulman 1986, 13). In our model, we associated teachers’ strategic knowledge with the cognitive process of evaluating different ideas in complex classroom situations (for example, the teacher recognizes that the principle of conflict-orientation has a different relevance for students of a commercial high school discussing “economics vs. ecology” than it has in the aforementioned case, and thus requires a different application).

Test-specific definition of PCK of business and economics

Due to practical limitations of testing time, we could not operationalize the entire model of PCK of business and economics in one test; therefore, we narrowed the construct for this test; while retaining the content distinction between teaching objectives, content, and methods and students’ learning process, we focused on the two aspects of lesson planning and reacting to students’ statements. With regard to cognitive structure, we focused on case knowledge because case knowledge and the associated processes of analyzing, applying, and creating, which are considered important for subject-specific classroom teaching. Furthermore, item contexts were written to represent practical situations in classroom teaching of business and economics.

Test development was guided by the following assumptions about teaching business and economics in Germany; however, we also argue for why the test might be appropriate for use in other countries. In general secondary education in Germany, business and economics concepts are taught mostly in other subjects (e.g., geography, political and social studies). Most business and economics teachers in Germany work in commercial vocational and commercial upper secondary schools, where business and economics content is taught systematically. The curricular structure of business and economics is very diverse, organized in subjects at commercial high school and in fields geared towards various professions at vocational schools. For example, prospective industrial managers take courses on identifying value streams and indicators; prospective wholesalers take courses on controlling buying processes. Therefore, most business and economics teachers in Germany need PCK that enables them to teach diverse, practically relevant, action-oriented content for a large number of commercial professions.

We focused on the content areas of sales, buying processes, and principles of economics because they cover central topics from both business administration and economics, and they are relevant to most professions that rely on business and economic knowledge (Brötz et al. 2011). In addition, these content areas are targeted in national and international assessments (e.g., Winther and Achtenhagen 2009; Walstad et al. 2013) and cover internationally recognized core competencies students should acquire in business and economics classes, such as understanding and solving various problems and social conflicts, making decisions, and seeing things critically and from multiple perspectives (e.g., Council for Economic Education [CEE] 2010; Davies 2012).

Research hypotheses

Test content

One of the five major criteria indicating the validity of an assessment, as defined by the Standards of Educational and Psychological Testing, refers to the connection between the theoretical construct and the test content (AERA et al. 2014). Analyses of content validity aim to determine the extent to which both single items and the compiled test are representative of and relevant to the construct being assessed (Haynes et al. 1995). Items must be derived systematically from a previously defined construct to ensure validity (Kane 2006).

The representativeness of the test items and overall test was deduced primarily from conceptual considerations. Our structural model served to describe the test-specific definition of PCK of business and economics and delineate the item domain, that is, the range of potential items to-be-included (McDonald 1999).

Items were judged relevant if they represented PCK necessary for teaching business and economics. Document analyses had provided a sound basis of relevant content from business and economics curricula and requirements for the teaching profession. To ensure practical relevance of items, we interviewed experts in the areas of both university- and school-based business and economics teacher education during item construction and revision.

Regarding representativeness and relevance of test items, Hypotheses I and II were specified as follows:

• 1. Test items cover the content and cognitive structures of the construct PCK of business and economics as specified in the test definition.

• 2. Test items cover aspects of PCK relevant to teaching business and economics.

Internal structure

Internal structure refers to the dimensionality of the test, which should correspond to the theoretically modeled structure of the construct (AERA et al. 2014). We used confirmatory factor analysis and models from item response theory (IRT; Embretson and Reise 2000) to test the relationship between the theoretically modeled structure of PCK of business and economics and responses to the test items. Since our operationalized test definition of PCK of business and economics focused on case-related PCK only and because research on the dimensionality of PCK in other domains is inconclusive (e.g., Hill et al. 2004; Kleickmann et al. 2012), we assumed a 1D latent construct. Although our definition included knowledge of both teaching and learning processes, analyses of teaching practices have shown that these types of knowledge are mutually dependent. Therefore, our conceptual differentiation between content associated with lesson planning and reactions to student statements might not be reflected in the empirical data. Similarly, we assumed that the cognitive process of creating is very similar to the cognitive processes of analyzing and applying and might not be clearly distinguishable in the empirical data. We tested the internal structure of the data set with a 1D measurement model and with two 2D models for comparison (one 2D model for content and cognitive structure, respectively).

We used differential item analysis to examine whether the test assessed the same latent variable across groups of test takers (AERA et al. 2014). Differential item functioning (DIF) indicates, for similar overall ability, a systematic overestimation or underestimation of particular item parameters for certain subgroups (Camilli 1993). DIF should be considered an item bias compromising estimation of the person ability only if the difference in the item parameters can be explained by a meaningful, construct-irrelevant difference between subgroups. If differences in subsets of items can be explained by construct-relevant factors, differential item analysis indicates relative strengths and weaknesses of subgroups with respect to the assessed construct (Camilli 1993). We expected groups with greater teaching experience to have an edge in completing PCK items. Furthermore, we assumed that female and male participants would perform equally well on the test items.

Accordingly, we specified the following hypotheses:

• 3. PCK of business and economics, as defined in our test definition, is a 1D construct.

• 4. There is no gender bias of the item format.

• 5. Subgroups with greater practical experience have an edge in completing PCK items.

Results

Item construction and test content

Interviews with eight experts, including university- and school-based educators of pre-service business and economics teachers as well as experienced business and economics teachers, and the consultation of practical materials (e.g., teaching materials and exam tasks from university classes, teaching materials from seminars, lesson plans for demonstration lessons) provided descriptions of everyday situations in business and economics teaching that covered all four areas of the item construction matrix (lesson planning and reacting to student statements, crossed with applying/analyzing and creating). Some examples that address particular domain-specific student difficulties in the three key content areas are: difficulties in differentiating between guarantee and warranty, difficulties in understanding the role of the state in national economy (see also Leiser, 1983, p. 308f), difficulties in applying abc analysis, for example to differentiate between value share and quantity share (see also Weber, 1994, p. 154ff). These descriptions were used to develop situation-based item contexts referring to business and economics teaching with a strong orientation towards practical requirements (Carlson 1990).

To represent the theoretically modeled cognitive processes, we selected suitable response formats (Schmeiser and Welch 2006). The processes of analyzing and applying were judged to be elicited adequately with forced-choice questions, while an open-response format was deemed necessary for the process of creating.

All items consisted of four parts: a meta-context, identical for all items in a content area and providing general curricular background information; a sub-context; a question; and an answer format. Figure 2 shows an example item from the content area of buying processes. In this item, the meta-context provides specific curricular details, and test takers are instructed to assume the role of a teacher at a vocational school teaching a class of prospective wholesalers and foreign trade operators.

Fig. 2

Sample PCK item (Kuhn 2014, p. 316)

The sub-context describes the students’ previous knowledge and the targeted competency. In this item, the task proposed in the sub-context does not match the lesson goal. As this item is part of the lesson planning set, test takers are instructed to revise the task, taking into account the lesson goal and target group. To respond correctly to the item, test takers need to remember and understand relevant subject-specific teaching aspects (e.g., it is a class of prospective wholesalers) and subject-matter aspects (e.g., the structure of decision-making processes). Hence, to respond correctly, test takers need subject-specific teaching knowledge and only little subject-matter knowledge because additional information has been provided in the item description (e.g., specifying price as decision criterion rather than simply a quantitative criterion). Test takers need to analyze the relevant aspects given in this item and apply them to the specific learning situation, for example by analyzing what ‘taking uncertain factors into consideration’ means for prospective wholesalers and by creating their own solutions. The open-ended item prompts test takers to suggest two ways of altering the task. Possible alterations include adding another offer to the task in order to create a real decision situation; adding qualitative aspects to the task such as adherence to delivery schedules in accordance with the qualitative comparison of offers; or adjusting the task to the students’ background and life experience, for example, by phrasing the task specifically for trainees of wholesaling and foreign trade.

The initial item pool consisted of 34 items covering the four cells of the matrix. The relevance and other aspects of the initial items were evaluated qualitatively by the aforementioned experts. Think-aloud interviews (Ericsson and Simon 1993; Leighton et al. 2011) were conducted with five pre-service business and economics teachers to detect formal deficiencies in the items, such as inaccurate language or construct-irrelevant demands (Kuhn and Brückner 2013). The revision process, which was conducted in close cooperation with the eight experts, resulted in a paper–pencil test with 17 items, representing the test definition as shown in Fig. 3 with regard to content and cognition (hypothesis I). A standardized rating by 13 experts, six of whom had not been involved in the test construction, additionally confirmed the relevance of the item content for business and economics teaching (hypothesis II). Confirmation means that the experts indicated on a 4-point scale either ‘I strongly agree’ or ‘I agree.’ For all items, relevance of content was confirmed by at least 80 % of the experts; for 11 of the 17 items, at least 90 % of the experts confirmed relevance of content. The experts also confirmed that the underlying business and economics content areas and core competencies are relevant to teaching business and economics.

Fig. 3

Item construction according to 2-by-2-matrix based on the test definition (Kuhn 2014, p. 142)

Administration of the test and internal structure

Survey and sample

In 2011, we administered the test to the target group (Table 1) and collected demographic data on the participants (e.g., gender, completion of content-specific courses). The test took 45 min to complete.

Table 1 Descriptive data of the sample for the empirical study

The test was scored with a maximum of 26 points. In accordance with the theoretically described cognitive requirements, the nine open-ended items requiring creating one’s own solutions had double weighting and, thus, were awarded 0, 1, or 2 points; the eight forced-choice items requiring only applying and analyzing were awarded 0 or 1 point. To ensure objectivity in the scoring of the open-ended items, a coding manual was developed in cooperation with the above-mentioned experts. Two trained coders independently scored 62 % of the open-ended responses. Intercoder reliability, as indicated by Krippendorff’s alpha for ordinal data, showed a good average value of 0.88 and ranged between 0.70 and 0.96 for all items.

Analysis of factor structure

First, we tested the dimensionality of the construct by analyzing the factor structure assuming a less restrictive, congeneric model. To this end, we conducted a confirmatory factor analysis using the software Mplus version 6 (Muthén and Muthén 1998–2010). The model fit of the one-factor model was compared against the fit of the two-factor models. The results were interpreted with regard to the reliability measure Cronbach’s alpha.

The fit measures for the one-factor model indicated a good fit and, thus, confirmed a one-factor structure of PCK of business and economics as specified in the test definition (WLSMV estimator, N = 338: RMSEA = 0.03; CFI = 0.94; TLI = 0.93; WRMR = 0.90). Three of the 17 items had positive, but non-significant factor loadings in the unstandardized solution (p > 0.05). In the standardized solution, only one item had a factor loading with p > 0.05. The analysis indicates a factorially homogeneous group of items, which can be considered representative of a single dimension (hypothesis III).

We tested the one-factor model against both two-factor models. In Alternative Model 1, we divided the items between the two factors of lesson planning and reacting to student statements. In Alternative Model 2, the items were attributed either to applying and analyzing or to creating. The fit of these two models¹ was almost the same as the fit of the one-factor model. However, Alternative Model 1 had a latent correlation of 1.06, which means the model estimate could not be interpreted at all, and Alternative Model 2 had a latent correlation of 0.96, which does not support the assumption of a two-factor structure. To further interpret Alternative Model 1, we conducted an analysis based on a multi-dimensional IRT model in ConQuest (Wu et al. 2007). We calculated a latent correlation of 0.93 between the two factors, lending further support to the hypothesis of a 1D construct.

Testing for a one-factor structure under the more restrictive condition of an essentially tau-equivalent model, we found, as expected, that this restrictive model was not supported.² The empirical results showed that factor loadings were not identical while homogeneity of items was confirmed; hence, the calculated value of Cronbach’s alpha of 0.68 was interpreted as being close to the lower bound of the acceptable reliability range (Krohne and Hock 2007; McDonald 1999). For the purpose of this project and in view of the lack of suitable alternative tests, this reliability was deemed acceptable (Hair et al. 1998).

Rasch scaling

Despite the differences in the factor loadings, which seemed to suggest a two-parameter (2PL) model, the items met the common criterion of fit to the Rasch model for ordinal data (hypothesis III) as calculated using ACER ConQuest 2.0: partial credit model (Masters 1982), N = 338, mean of latent ability distribution = 0. As shown in the Wright map (Wilson 2005; Fig. 4), the thresholds spread reasonably well over the person ability distribution. As expected, the thresholds of the 2-category were in the higher ability range, while thresholds of the 1-category were in the lower ability range. The precision of the person ability estimates was represented by the expected a posteriori (EAP) estimation of 0.71, which showed, as expected, a good match with Cronbach’s alpha. However, the mean threshold of 0.27 meant that there were not enough easy items for precise estimations in the lower ability range.

Fig. 4

Wright map (N = 338; X indicates 2 cases; ‘.1’ indicates the 1-score category, ‘.2’ indicates the 2-score category of an open-ended ordinal item)

The item fit measure weighted mean square (WMNSQ) showed acceptable values (between 0.91 and 1.14); only one item showed a significant deviation from the model with a t value of 2.4, which was beyond the acceptable interval of [−1.96; 1.96]. The point-biserial correlation values of the score categories increased in order of increasing category scores, with negative correlation values for category 0 (Wu et al. 2007).

Differential item analyses provided evidence as to whether the test assessed the same latent variable across groups of test takers. Three forced-choice items showed moderate effects (−0.56; 0.47; 0.56), indicating differences in the item functioning between female participants (N = 150) and male participants (N = 188) (Wilson 2005). However, since differences in item functioning were not observed systematically across all three items (one item was systematically easier for males, while two items were systematically easier for females), the analysis did not indicate a gender bias of the item format. As intended, there was no significant differences in the mean test performances of the female and male participant groups (hypothesis IV).³

In the differential analysis of those with teaching experience (pre-service teachers in their practical phase of teacher education after university, as well as experienced in-service business and economics teachers; N = 162) and those without teaching experience (pre-service teachers in the university phase of teacher education; N = 176), two items showed moderate effects (0.46; 0.52), and 10 items showed large effects (0.65–1.24), indicating that the items functioned differently for the two subgroups. All these items were systematically easier for in-service teachers and pre-service teachers who had already entered the practical phase than for pre-service teachers who were still in the university phase. Therefore, these effects indicated that individuals with greater practical experience scored better. This construct-relevant explanation supported hypothesis V, emphasizing the importance of practical experience (Camilli 1993; Roussos and Stout 1996).

The results indicated that there was not only a factorially homogenous group of items, but also a Rasch homogenous group of items representing the same latent variable. Thus, we considered the manifest sum score of the test a sufficient statistic of PCK of business and economics as specified in our test definition (Masters 1982).

Response processes and relationships to other variables

Comprehensive validation of our test on PCK of business and economics involved assessment of the above-mentioned measures and other validity criteria (AERA et al. 2014). Thus, to present a sound argument for the validity of our test, we highlight key findings from these investigations. The entire validation process of the international Standards for Educational and Psychological Testing is illustrated in Fig. 5.

Fig. 5

Validation process of the Standards for Educational and Psychological Testing (see Kuhn 2014, p. 129)

Cognitive interviews using the think-aloud method with pre-service business and economics teachers provided preliminary empirical evidence that the theoretically predicted thought processes of analyzing, applying, and creating are, indeed, elicited during item response processes, as indicated by verbalized responses (evidence based on response processes; Kuhn and Brückner 2013). Analyses of relationships to other variables provided evidence of discriminant and criterion validity (evidence based on relationships to other variables). For example, comparative group analyses confirmed that the level of PCK among different groups of in-service teachers in the field of business and economics increased as additional classes on subject-specific teaching methodology were introduced and as the number of practical teaching phases increased within their study model. Additionally, as expected, the level of PCK increased with teaching expertise over different stages of teacher education. In-service teachers at the stage of professional, fully autonomous teaching at schools scored higher than pre-service teachers at university (Kuhn 2014).

Discussion

Although the results reported above provide evidence of reliability and validity of our test on PCK of business and economics, we situate this work within the larger landscape of debates about whether knowledge for teaching can be developed and assessed outside of the complex classroom context. Our instrument is consistent with other efforts to develop paper-and-pencil assessments of PCK (e.g., Park et al. 2008). In addition, other researchers have attempted to document teachers’ PCK in written form using structured templates (Loughran et al. 2001; van der Valk and Broekman 1999). However, concerns have been raised about whether these methods really elicit PCK or something else (Alonzo 2007, Hambleton 2006) and whether they capture the complexity of classroom practices (Baxter and Lederman 1999). While it is possible to provide contextual information in written form and, thus, assess part of teachers’ case knowledge (Rohaan et al. 2009), the fact remains that teachers are engaged in tasks that differ from actual teaching. Consistent with earlier claims (van Driel et al. 1998), recent research (Alonzo et al. 2012) points to the importance of a more spontaneous, flexible, and adaptive form of PCK that allows teachers to listen and respond to students’ ideas.

With greater clarity about the construct of PCK of business and economics established in this study, our next step is to explore the use of video to assess pre-service and in-service teachers’ strategic knowledge in our three-year follow-up study (Heinze et al. 2016). Kagan (1990) criticized earlier efforts to elicit teachers’ cognitions as relying upon materials that are ‘so artificially neat that the data they generate are irrelevant to the problems encountered in classrooms’ (p. 456). Videos preserve some of the authenticity and complexity of classrooms situations (Alonzo et al. 2012) and can capture transient moments of classroom instruction. Since we cannot access teachers’ reasoning and use of knowledge while they are engaged in teaching, this is as close as we can come to exploring PCK in action (Janík and Miková 2006) and, thus, to honoring even more the context-specific nature of this knowledge (Grossman et al. 1989). Work to elicit teachers’ PCK using video has demonstrated that this approach may capture different knowledge from that revealed by paper-and-pencil measures of PCK (Kersting et al. 2010). Previous studies have aimed to determine PCK through teachers’ reflections on videos of their own instruction (Alonzo and Kim in press), of others’ instruction (Kersting et al. 2010), and of interviews with students (Alonzo and Kim in press). We see particular promise in eliciting teachers’ strategic knowledge through direct responses to video-recorded displays of student thinking (Lindmeier 2011).

The paper–pencil test we presented in this paper combined with our newly developed video-/performance-based instrument will serve as a valuable tool to foster professional teaching and teacher education in the subject area of business and economics. In the future, a valid and comprehensive measure of teachers’ PCK of business and economics could provide a better understanding of the role PCK plays for student learning and of how teachers acquire PCK of business and economics during teacher education and professional practice. To this end, formative diagnostic assessment could be employed during the different phases of teacher education and practical training. Although the test instrument was created for use in Germany, it is based on internationally agreed upon core concepts of business and economics and adheres to content standards and, therefore, could be adapted for use to support business and economics education in other countries.