Determinants of course completions in vocational education and training: evidence from Australia

Abstract

Background

Completion rates in Australian vocational education and training (VET) are notoriously low. While there are conventional reasons such as issues with course, health, institutional factors, financial and family problems and dissatisfaction with the training experience, more VET specific explanations have included that students may discontinue their studies when they have obtained the specific skills they were seeking or they have gained employment. This present study seeks to examine whether the original intention of students at the time of enrollment along with satisfaction and the benefit that could be obtained from completion have any bearing on completion patterns.

Methods

We model the probability of intending to complete from the 2011 Student Intentions Survey and the perform an out of sample prediction of students’ intention on respondents to the 2011 Student Outcomes Survey. Subsequently a logistic regression model predicting actual completion is developed utilising student intentions, various components of satisfaction, completion pay-offs and some demographic and educational variables.

Results and Conclusions

Main findings of this study include that while students’ initial intentions to complete increase actual completion probabilities, the overwhelming determinants of completion are high annual hours of enrollment and the enrollment in higher VET qualifications. Furthermore, the benefits to completion and satisfaction with the training play only minor roles in shaping completion patterns. This study contributes to the existing body of knowledge about completions in VET by including the original completion intention of students in the examination of completion patterns.

Background

Vocational education and training (VET) in Australia delivers post-compulsory education and training, excluding degree and higher level programmes delivered by further education institutions, and provides people with occupational or work-related knowledge and skills. In addition, VET also delivers programmes that enable students to engage in subsequent educational programmes. In recent years there has been an increased policy focus on qualification completions in both VET and higher education sectors in Australia. Governments are keen to know the extent to which students complete the qualifications or studies they commence because they want to ensure adequate accountability for the expenditure of any public training funds that may have been used. In the university sector it has long been recognised that the completion of an undergraduate degree imparts significant social and economic benefits on students and society as a whole (Borland 2002; James 2001; Marks 2007). As a consequence of the perceived and real benefits of obtaining a university degree, completion rates have been recorded at a relatively high level of 80 %, along with an improving trend (ACER 2011). Completions in vocational education and training (VET)¹ on the other hand have been known to be substantially lower than in the university sector. A variety of reasons have been cited to account for the low VET completions in Australia, including problems with employment and course, health and chance events, institutional factors, and financial and family problems (Long et al. 1995). Other authors concluded from the analysis of the English higher education system that wrong choice of field of study, financial difficulty and dissatisfaction with the training experience (Yorke et al. 1997), as well as inability to cope with the training demands, social dissatisfaction and dissatisfaction with the institutional environment contribute to significant non-completion (Yorke 2000).

One substantial dissimilarity between the completion of a qualification (or degree) in the university and VET sector context appears to be the perception of non-completion. In the university sector non-completion is often (but not always) associated with failure, while in the VET sector it is accepted that non-completion frequently means that students have achieved the specific training goal which prompted them to enroll in a particular program (McInnis et al. 2000; Karmel and Nguyen 2006; Mills et al. 2012). For instance, a plumber may enroll in a certain subject to acquire the skills involving a new technique without the intention of gaining credit toward a qualification. Cohen and Brawer (1996) commented that to vocational students who are seeking a job, completing their training becomes irrelevant as soon as a job becomes available. There is thus less negative stigma attached to categories such as ‘non-completer’ or ‘dropout’. This, in addition to the above-mentioned impediments, may account for the considerable differences in completion rates and additionally necessitates caution when attempting to directly compare completion rates between both sectors, as these numbers need to be interpreted differently.

An additional issue in assessing the difference in completion rates between the university and VET sector is that there has been some controversy about the way in which completion rates are determined in the vocational system. At the current point in time there is no unique student identifier and the concept of commencement of a qualification is not very well defined in the VET system (Mark and Karmel 2010). Completion rates can consequently not be calculated in a straightforward manner. In their paper, Mark and Karmel therefore used a modeling technique in an attempt to derive completion rates in Australian vocational education. This approach enabled them to estimate completion rates for students commencing in 2005 and yielded an overall completion rate of 27.1 %, with significant variation between several categories of students (for example, full-time, part-time, age groups, course level). The result of this analysis has been met with some skepticism. Ross (2011) cited various industry figures who pointed to substantially higher completion rates in selected fields. For instance, an independent analysis by Service Skills Australia (SSA) yielded completion rates substantially higher than the figures published by Mark and Karmel. Reasons cited as responsible for the discrepancies were temporary withdrawals such as for pregnancy, illness, relocation, etc. which were not accounted for in the Mark and Karmel study. The authors of the original study dismissed these objections and suggested that the SSA study relied on cherry-picked training providers and on too short a period of analysis (Ross 2011). Despite these somewhat conflicting research results with respect to actual completion figures, there is a general perception that Australian VET completion rates are low (Mark and Karmel 2010; Azemikhah 2009; Snell and Hart 2007).

In order to investigate the extent to which students do commence their VET studies with the aim of not completing them, the Australian National Centre for Vocational Education Research (NCVER) designed and conducted the Student Intentions Survey (SIS) in 2011 (NCVER 2011). This survey was intended to canvas students’ completion intentions along with the collection of social and educational data. The main result of this survey was that a surprisingly high percentage of students (93 %) set out to complete their qualification. While this figure may have been somewhat inflated (it appears reasonable to speculate that some students may not have been willing to divulge their true intentions as, for instance, government funding may have depended on their declared aim to complete a prescribed course of study), it is of interest to explore the unexpected large disparity between intended and actual completions.

The purpose of this study is thus to explore some questions that surround issues about completion in Australian vocational education. Specifically, the relationship between student satisfaction, the benefits to completion, student age, qualification level and enrollment hours, an well as the intention to complete and actual completion is investigated. Historically, analyses of completion patterns in the Australian VET sector have focused on uncovering determinants of completion in VET without taking the original intention of students into account. In this study we will address this problem by integrating intention information gathered from the 2011 Student Intention Survey into the 2011 Student Outcomes Survey to enable us to make inferences on the relationship between the intention to complete, student age, qualification level of enrollment, student satisfaction, the benefits to completion, and actual completion.

Data and data preparation

This study makes use of two surveys conducted in 2011: The Student Outcomes Survey (SOS) and the Student Intentions Survey (SIS). The SOS is an annual survey that focuses on student outcomes with VET and also collects data on a range of personal and training characteristics, as well as student satisfaction and related measures. The SIS was a one-off survey in 2011 aimed at measuring students’ intentions in respect to their plans to complete the course they were enrolled in. The SOS, designed to enable estimates of key indicators at the institution level in 2011, achieved about 110,000 responses out of about 300,000 sampled students. The SIS featured a much smaller received response of 11,000 out of a sample of 23,000 students, as this survey was designed to provide statistically meaningful estimates at the national level only (NCVER 2011). While both surveys gathered a similar level of administrative, demographic and educational student background data, the SIS limited the information that was collected to issues surrounding the willingness to complete, knowledge about the training at enrollment and attitudes toward the training and vocational education.

The study presented here defines completion based on the classification of graduates and subject completers in the SOS, where graduates are students who have completed a VET qualification and subject completers have successfully completed part of a course (for example, at least one subject) without completing a qualification and who have left the vocational education and training system by the time the survey was undertaken (NCVER 2011a). The benefit of utilising this categorisation was that a large amount of data was readily available for analysis and that the categorisation into these two groups was largely uncontroversial and accepted in the VET community. The primary drawback of this approach was that it omitted those students who enrolled but did not complete a single subject. However, with subject completion rates of around 90 %, it was assumed that this group of students would have accounted for only a very small percentage of non-completing students. Furthermore, the total absence of any data for this type of student at the national level would have made their inclusion in any analysis problematic.

This analysis is aimed specifically at how the benefit to completion, the intention to complete, student satisfaction, several demographic and educational variables, and actual completion are related. The benefit to completion for individual students was determined using SOS Data and employing the methodology described in Karmel and Fieger (2012). The derivation of the benefit of completion described therein is based on two identical models modeling the benefit variable of interest (for instance ‘employment after training’) as a function of a number of predictors, such as sex, age group, location, field of education, qualification level, study status, socio economic status, employment status before training, and prior education. The first model is run only over graduates and then predicts the outcome for graduates and subject completers. The second, identical model is run over subject completers and then predicts the outcomes over subject completers and graduates. As a result, each observation has two predicted scores, one for the completion (graduate) and one for the non-completion scenario (subject completer). The ratio of these two scores defines then the individual completion benefit. Using the example of the ‘employment after training’ outcome variable, where a given individual might have a probability of 0.8 of being employed after training as a completer, and a probability of 0.7 as a non-completer, this would imply a 0.8/0.7 = 1.14 or 14 % benefit to completion for this particular individual. These individual benefits to completion can then be aggregated to provide and overall measure of the completion benefit. We created four new variables, containing the tertiles of the benefit of completion in respect to employment after training, salary, occupational status, and further study. As can be seen in Table 1, there is significant variability in completion benefits between the various outcomes. While there is a strong completion benefit in terms of further study (e.g. a completer is 125 % more likely to be in further study than a non-completer) and also employment after training and improved employment conditions (25 %), the benefit to completion in terms of salary and occupational status are rather diminutive.

Table 1 Benefit to completion for several outcomes

As the samples for SIS and SOS were drawn from two different populations and thus contained different individual students, a method was devised which allowed the transferral of the information about the intention to complete that was gained from the SIS to the SOS which contained the information about actual completion. This was achieved in the following way: First, using the SIS, a model was developed that estimated the intention to complete (Table 2). This model had fairly substantial explanatory power as indicated by a pseudo r-squared of 0.39. Notable findings from this model of intention to complete include a higher propensity of females to complete compared to males, those with a prior year 12 education over all other prior education levels, and those studying for employment and further study reasons over personal reasons. Interestingly, those in employment were less likely to intent to complete than those not employed and those in remote locations were less likely to intent to complete than those in urban and regional settings.

Table 2 Logistic regression parameters for intention to complete

Student satisfaction has been identified as a predictor of completion outcomes in previous studies (Snell and Hart 2008; McInnis et al. 2000; Yorke et al. 1997; Scott et al. 1996). We hypothesised that this inclination may be also exist in our present study. Mean satisfaction composite scores of student satisfaction with teaching, assessment, and general learning from the student outcomes survey were used and three new variables containing the tertiles of these satisfaction scores were generated.

In addition to these measures, a number of new variables were derived from administrative data: an enrollment hours variable was created consisting of 5 categories: 1–30 h, 31–60 h, 61–120 h, 121–400 h, and 400+ h; a three category qualification level variable indicating certificate I/II, certificate III/IV, and Diploma or higher, and a variable containing the age of individual respondents.

Results

Before we conducted our main analysis we undertook an examination of our predictor variables in respect to potential interactions between our predictor variables when modeling the probability of completion. This analysis revealed four substantial interactions between qualification level and age, qualification level and intention to complete, qualification level and course enrollment hours, and age and enrollment hours. These four interaction terms were added to the logistic completion model along with the other predictors. The resulting coefficients from this analysis can be found in Table 3.

Table 3 Logistic regression parameters for actual completion

Table 3 contains coefficients, their standard errors and significance as well as the margin for the base category of categorical variables and the predictive margins for the comparison categories. The overall statistics of our completion model indicate significant power of the independent variables to predict the completion outcome. Likelihood ratio Chi squared value of 36,923 (p < 0.001) and a pseudo R-squared of 0.6 suggest a reasonable explanatory capacity of the model.

The first notable result of this model is that satisfaction with teaching and assessment has only negligible impact on completions in VET. While satisfaction with teaching makes no significant difference across all three tertiles to completion outcomes at all, the two upper tertiles of satisfaction with assessment imply a mildly increased propensity to complete by about one percentage point. Increased satisfaction with the general learning experience on the other hand confers an increased completion probability of about 2 percentage points for the middle tertile and 3 percentage point for the upper tertile compared to the lowest tertile.

All four payoff variables reveal significant differences between their base tertile and higher tertiles in respect to completion, although for the benefit to employment, salary, and occupational status, although their impact is limited to about one percentage point difference in terms of completion outcomes. Only the variable containing the benefit to completion in terms of further study has a more pronounced effect on completion as indicated by a predictive margin of one percentage point increase in completion probability between tertile 2 and 1 and a 3 percentage point increase between tertile 3 and 1.

The variables for number of course hours enrolled, age, qualification level and probability of intention to complete were all entered also as part of interaction effects. The coefficients for these variables thus have to be seen in the context of these interactions. We have graphed the predictive margins for these interaction effects for better visual representation. In all graphs we added a reference line at 0.71, which represents the overall predictive margin of completion.

The first interaction analysed in respect to completion is between the qualification level and student age. The predictive margins have been calculated and are graphed in Fig. 1.

Fig. 1

Age × qualification level interaction

From Fig. 1 it is quite clear that increasing age generally indicates a lower probability to complete. At the age of 15 Students enrolled in Certificate I/II courses have the lowest likelihood of completion, whereas students enrolled in Certificate III/IV as well as Diplomas and higher have a significantly higher propensity of about 10 percentage points to complete. Interestingly, with increasing age the probability to complete decreases more rapidly for Certificate III/IV students, so that at the end of the age scale their completion probability is similar to Certificate I/II students. The negative impact of increasing age on completion has been described before (Clarke et al. 1994), and reasons advanced for this phenomenon have included the higher demands of family circumstances, as well as financial considerations and employment requirements (Ozga and Sukhnandan 1997). However, the discovery that this effect is more pronounced for students enrolled Certificates III/IV represents a new finding. It should also be pointed out that the size of this effect of age is much more pronounced in the case of Certificate III/IV students than for any of the variables discussed before. This is still true even if only the age range with a significant student population (e.g. 15–45) is considered.

A further significant interaction detected is between the qualification level and the intention to complete (Fig. 2). While there appear to be dramatic differences in completion probabilities based on qualification level in the lower ranges of the probability to of intention to complete, it has to be kept in mind that only a minuscule proportion of the student population is actually present at intention probabilities below 0.7. The more pertinent conclusions that can be drawn from this graph are thus associated with the higher probabilities of the intention to complete. Here, it can be seen that the probability to complete substantially increases in a linear fashion for Certificate III/IV and Diploma students as the intention to complete increases, while completion probability increases only moderately for Certificate I/II students. At the end of the intention scale we then find Certificate III/IV and diploma students significantly above the average predictive margin of 0.71 to complete, whereas Certificate I/II students can be found significantly below. The overall margins for Certificate I/II, Certificate III/IV and Diploma were 0.67, 0.72, and 0.77, respectively. Interestingly our estimates reveal slightly higher completion probabilities for Diploma students across the range of completion intentions when compared to Certificate I/II students. This is in contrast to Mark and Karmel (2010) who showed higher probabilities for Certificate III/IV students. The discrepancy however is minor and probably resulting from different assumptions and modeling techniques employed in Mark and Karmel and the present study. Both studies are in agreement in the overall lowest completion rate of Certificate I/II students.

Fig. 2

Intention to complete × qualification level interaction

Figure 3 graphs the interaction between enrollment hours and qualification level. This graph also reveals the main determinant of completions. It is clear that not only are completion probabilities in the lower quintiles of enrollment hours well below the average overall margin (0.71) but also that there are large differences between the three qualification levels. Students at Certificate I/II levels have completion probabilities of less than 10 percent, whereas Certificate III/IV and Diploma students have probabilities in excess of 20 and 35 %, respectively. Enrollment hour quintiles 4 and 5 (e.g. enrollments larger than 121 h/year) show invariably completion probabilities in excess of 90 percent. It should be kept in mind that in Fig. 3 changes across the x-axis can be viewed more as reflecting the population as a whole, as each of the enrollment hours quintiles contains roughly comparable numbers of students. It thus emerges that the main predictor of completion in Australian vocational education is the number of hours an individual student is enrolled in. The overall margins for this variable are 0.22 (1–30 h), 0.28 (31–60 h), 0.48 (61–120 h), 0.96 (121–400 h), and 0.99 (>400 h). Prior research has revealed a relationship between enrollment hours and completions. McGivney (1996) reported differences in completion between part-time and full time student in the British higher education sector in the range between 6 and 17 percent. In Australia, Bender (2003) analysed non-completions of apprentices and trainees and found completion rate gaps between 11 and 23 percent between part-time and full-time students.

Fig. 3

Enrolment hours × qualification level interaction

Finally, our completion model indicated that the impact of student age on completion probability is dependent on enrollment hours. We graphed the predictive margins of this relationship (Fig. 4) and it became apparent that while age does not impact on completion probability at enrollments >120 h, for student with smaller enrollment hours increasing age is associated with a decreasing probability of completion.

Fig. 4

Enrolment hours × qualification level interaction

Discussion

Course completions in vocational education in Australia are notoriously low. A number of generic explanations have been advanced for this, including personal and institutional factors. Increasing satisfaction with the training is generally seen as leading to higher completion rates (Power et al. 1987; Dunn 1995). In the context of the overall results of this present study we could only find a very minor effect of satisfaction on completions. The effect was greatest for general satisfaction and the comparison of the first and third tertile of general satisfaction revealed a 4.3 percentage point increase in the probability to complete.

We also tested the hypothesis whether the benefit to completion in respect to four outcomes (employment, salary, occupational status, and further study) was associated with completion outcomes. The results show that the impact of the benefit to completion is very limited, and in the area of a one percent impact on completion probability. Only the benefit to completion in terms of further study is slightly more pronounced with about a three percentage point increase in completion probability between first and third tertile of the benefit to completion. These results showing a relative lack of impact of the benefit to completion on the probability of completions were somewhat unexpected. Two limitations of these predictor variables should thus be considered. First, it would appear that it is somehow implied that individual students are aware of the benefit of completing their course of study. This may not necessarily be the case and it can thus not be assumed that the benefit to completion plays a role in their decision making on whether or not to complete. Secondly, the way the variables indicating the benefit to completion are constructed means that they represent the overall benefit to completion, rather than the net benefit. In other words, individual economic costs to obtain the benefit are not taken into account. This represents a shortcoming of all the benefit variables and is difficult to overcome with presently available data. In the future it might be possible to address this via a survey that collects data on individual’s time, effort, and financial input into their training.

The affirmation of the initial intention to complete the chosen a course is clearly associated with an increased likelihood of completion. While this is a result could be expected, prior to the completion of the Student Intentions Survey this could not be shown. Figure 2 displays a large range of probabilities of intention to complete and associated probabilities of completion. It should be kept in mind that the overwhelming majority of students had a very high probability of intention to complete and that thus the most reliable representation of probabilities can be found at the higher end of the intention to complete scale.

Our study also showed a fairly strong impact of age and qualification level on completions. We demonstrated increasing age as an impediment to completion, and this was particularly so for students enrolled in Certificates III/IV. Generally, students enrolled in Certificates I/II had significantly worse completion outcomes across the main enrollment ages than those enrolled in other qualifications. Both results in respect to the association of age and qualification level with completion outcomes are well supported by previous research.

The main driver of completion outcomes in Australian vocational eduction identified in this research are the number of enrollment hours. While the prior literature does cite a moderate discrepancy in completion rates between part-time and full-time students, the differences we have found in this study are more substantial. While students enrolled in excess of 120 h/year have completion rates in excess of 90 percent, all qualification levels below this number of enrollment hours have steadily decreasing completion rates as their hours of enrollments decrease. Those in the lowest quintile of enrollments hours have completion probabilities of lower than 10 % (Certificate I/II), below 30 % (Certificate II/IV), and below 40 % (Diploma).

Conclusion

This study analysed completions and predictors thereof. These predictors included student satisfaction, the benefit to completion, age, qualification level, enrollment hours, and the student’s intention to complete at the time of enrollment As there were no data available that tracked a cohort of students from enrollment to completion with respect to their intentions, we had to develop a model that predicted the probability of intending to complete for those students where we knew the completion outcome from the Student Outcome Survey. It needs to be kept in mind that our analysis was based on such modeled data rather than actual data.

Our main finding was that completion issues in the VET sector are predominantly a problem within the student population enrolled at 120 h or less. The analysis also revealed a somewhat weaker relationship between completion and the intention to complete, qualification level, student age, and several interactions between these variables.

A surprising result was the relatively small impact of student satisfaction. While the stronger magnitude of ‘general’ satisfaction over course-related satisfaction found in earlier research in respect to completion outcomes could be confirmed, it was still striking to observe the relatively low contribution of overall satisfaction to the completion model. It may be possible that, due to the unique properties of the vocational education system discussed earlier in this paper (for instance, that a substantial proportion of non-completers fail to graduate because they have gained the skills they set out to acquire, or non-completion due to arising employment opportunities during the training), satisfaction is indeed less related to completion in the VET system when compared with the higher education sector. Furthermore, it is possible that the small contribution of satisfaction to the probability of completion may partly reflect deficiencies in the way satisfaction was measured in the vocational sector.