Table 6 Criteria for the tasks’ difficulty, including their operationalization and examples

(I) Kind of cognitive process according to the Cognitive System of the ‘New Taxonomy of Educational Objectives’ by Marzano and Kendall ( 2007; 2008 )

Four response categories:

(1) Retrieval

(2) Comprehension

(3) Analysis

(4) Application

(1) A student who is able to reproduce the concept of ‘reliability’ has attained the level of retrieval

(2) A student who is able to decide which data collection method is suitable depending on the presented investigation context has achieved the level of comprehension

(3) A student located on the level of analysis is able to assign presented extracts from a study’s problem definition to its typical elements

(4) A student who is able to set up a research design which is based on a presented research objective or question of a concrete study attained the level of knowledge utilization

(II) Complexity concerning the number of content-related/curricular elements (=solution-relevant variables)

(Adams and Wu 2002 )

Three response categories:

(1) Only one isolated content-related element has to be processed

(2) Two content-related elements have to be processed

(3) At least three content-related elements have to be processed

(1) A task that only demands to describe what the term ‘nominal scale level’ means contains a low complexity

(2) A task which requires a decision if a correlation or a regression analysis is fitting better in order to answer a presented research question shows a moderate complexity

(3) A task prompting to set up a context-related experimental design which has to include all relevant components—such as pretest, treatment, posttest, experimental group, and control group – and which requires considering the concept of randomization involves a high complexity

(III) Degree of familiarity

(~curricular weighting of the task contents; degree of routine with regard to the respective task context)

(e.g., Blum et al. 2003 )

Three response categories:

(1) High degree of familiarity = many learning opportunities

(2) Moderate degree of familiarity = moderate number of learning opportunities

(3) Low degree of familiarity = few learning opportunities

Details concerning the operationalization:

Nine content areas, which depict all contents instructed during the course on empirical research methods were defined

(1) A large proportion of the course’s instruction was spent on performing several in-depth and hence intensive exercises to handle situations belonging to the curricular area of evaluating the adequateness of methods for collecting data

(2) A moderate proportion of the course’s instruction was spent on in-depth exercises with regard to interpreting statistical outputs for explorative factor analyses

(3) The curricular area of regression analysis was treated very superficially during the course (no hands-on applications and repetitions of the respective contents/methods were provided)

→Regarding each item, an index consisting of three criteria was calculated in order to assess the amount of learning opportunities:

(a) How many lecture slides are dealing with the relevant content area? (referring to the first element of the course—the lecture)

(b) How extensive was the respective content area treated within the instruction? (referring to the first and the second element of the course—the lecture and the tutorial moderated by a lecturer)

(i) = low instructional extent

(ii) = high instructional extent

(c) Did the test persons have the opportunity to participate proactively in a case-related hands-on-application concerning the respective content area? (referring to the third element of the course—the project work in small groups supported by advanced students)

(i) = no hands-on-application was performed

(ii) = hands-on-application was performed