Technical and vocational education and training (TVET) for young people in low- and middle-income countries: a systematic review and meta-analysis

Abstract

The world is facing a worsening youth employment crisis. In response, technical and vocational education and training (TVET) is back on the development agenda after years of neglect. This systematic review examined the evidence from studies evaluating the impacts of TVET interventions for young people in low- and middle-income countries (LMICs). The 26 included studies evaluated 20 different interventions, predominantly from Latin America. Meta-analyses of the effectiveness of TVET on five outcome measure categories were conducted. The overall mean effects on overall paid employment, formal employment, and monthly earnings were small, positive, and significant; however, significant heterogeneity was observed. Moderator analysis was performed in an attempt to explain between-study differences in effects. The overall paucity of research in this area, together with specific gaps and methodological limitations, affirm the need for strengthening the evidence base. Implications for policy, practice and research are discussed.

Review

Introduction

The global youth unemployment rate rose sharply between 2008 and 2009, from 11.8 to 12.7 per cent, reversing the pre-crisis trend of declining youth unemployment rates since 2002 (International Labour Organization 2011). By 2011, 74.8 million young people were unemployed. In some regions, young people are nearly three times as likely as adults to be unemployed; they are also more likely than older workers to be underemployed or work in the informal labour market, in poor quality jobs that require low levels of skills and offer limited socio-economic security, training opportunities, and working conditions (International Labour Organization 2012). Significant regional variation in youth unemployment exists; and, in many countries, young women are much more likely to be un/underemployed than young men (United Nations 2012). Over 40 per cent of all young people live on less than two US dollars a day, with youth in developing countries disproportionately among the working poor (International Labour Organization 2012). This enormous unlocked potential represents a substantial loss of opportunity for both individuals and society.

One billion young people, the majority from LMICs, are predicted to reach employment age within the next decade (International Labour Organization 2012), compounding what are already severely limited opportunities for integrating youth into the labour market. With increasing policy importance now placed on higher-order skills and their central role in the global knowledge-based economy, comes a renewed focus on the potential of TVET to equip youths with the abilities to seize available work opportunities, and, in turn, impact upon global poverty and social stability (African Union 2007; King and Palmer 2010; Péano et al. 2008; United Nations Educational, Scientific and Cultural Organization Scientific and Cultural Organization 2010, 2012). Neglected by the World Bank and other donor agencies during much of the 1980s and 1990s, there has been growing investment in TVET since 2000. Deciding which programmes to implement requires an understanding not only of which models are effective, but for whom they are effective. Collecting evidence from studies that have analysed these issues is crucial for purposes of policy-making.

Technical and vocational education and training (TVET)

The term TVET, as used in this systematic review, follows the definition used by UNESCO as “those aspects of the educational process involving, in addition to general education, the study of technologies and related sciences and the acquisition of practical skills, attitudes, understanding and knowledge relating to occupation in various sectors of economic life”. It incorporates: technical education, vocational education, vocational training, on-the-job training, and apprenticeship training (or any combination thereof).

Prior reviews of TVET

Although a number of prior reviews have summarised the existing research in this area (see, for example, Angel-Urdinola et al. 2010; Betcherman et al. 2004; Betcherman et al. 2007; Card et al. 2010; Fares and Puerto 2009; Greenberg et al. 2003; Katz 2008; Kluve 2006; Kluve and Schmidt 2002; Puerto 2007; Stavreska 2006; Van Adams 2007), there are a number of limitations to these reviews. Some contain evidence from high-income countries only; others, whilst broader in geographical scope, do not focus specifically on developing countries, TVET, or young people. Very few existing reviews are based on a comprehensive, systematic search for literature, and almost all take either a traditional narrative approach to synthesis or use vote-counting methods where conclusions regarding effective interventions are made based on the number of studies found to demonstrate significant positive results. None of the previous reviews that specifically focus on TVET for youth in LMICs have computed effect sizes for the included studies or examined variability in effects by study, participant or intervention characteristics. These are problems that this review aims to remedy, thereby adding value to the existing body of research on this topic.

Purpose of the present study

This review sought to improve upon prior work by systematically examining the evidence base to provide a clearer picture of the types of TVET interventions being used to improve employment prospects for youth in LMICs, to identify the overall effectiveness of TVET, to examine whether study, participant and intervention characteristics help to explain variability in intervention effects, and to identify areas in which more research needs to be conducted. An extended version of this article is available as a technical report published in the Campbell Collaboration library (Tripney et al. 2013).

Methods

Systematic review methodology was utilised for all aspects of the search, selection and coding of studies. The review was conducted in accordance with Campbell Collaboration procedures and guidelines on systematic review methods, available at http://campbellcollaboration.org/. Full details about the methods for this review can be found in the technical report, and review protocol (Tripney et al. 2012).

Eligibility criteria

Eligibility was restricted to studies that: were conducted in LMICs; were reported between 2000 and 2011; evaluated the impact of a TVET intervention; investigated outcomes for youth aged 15–24 years; included at least one quantifiable measure of employment or employability (broadly defined); and used an experimental or rigorous quasi-experimental design (such as the use of propensity scores to match on pre-tests and/or relevant demographic characteristics or regression analysis to control for differences between groups). Single group pre-post test studies were not eligible. No language or publication status restrictions were applied.

The conceptual definition of TVET used in this review cuts across education level, type of learning arrangement, mode of delivery, setting, and type of provider/regulator. It includes provision of (i) initial training for young people from the age of 15/16 years after compulsory school, but prior to entering work; (ii) continuing education and training for adults in the labour market leading to personal, flexible and/or vocational competencies; and (iii) training for unemployed persons currently available for and seeking work (including retraining for those made redundant). Single- and multi-service TVET interventions were eligible for inclusion in the review, as were interventions delivered for any length of time or frequency. LMICs were defined according to World Bank classification of economies (in effect 1 July 2011 until 30 June 2012).

Literature search and strategy

The review recognised the importance of using a comprehensive and diverse search strategy to locate all qualifying published and unpublished studies (Papaioannou et al. 2009; Schucan Bird and Tripney 2011). First, 10 major bibliographic databases were electronically searched (including ASSIA, Econlit, ERIC, IBSS, Medline, PsycINFO, and SSCI). Searches within each database combined controlled vocabulary and natural language terms, with appropriate wildcards, for three concepts: TVET, employment, and study design. Second, specialist databases and library catalogues were searched, including grey literature, regional and topic-specific sources. Third, the websites of government agencies, research centres, foundations, professional associations, and other relevant organisations were searched. Fourth, the reference lists of previous reviews and included studies were checked. Fifth, citation searches of included studies were conducted. Sixth, information about additional relevant studies was requested from authors of included studies and other key experts in the field.

Data extraction and critical appraisal

Data were independently extracted from all included studies by pairs of reviewers using a coding tool specifically designed for this review. Studies were quality assessed using a tool developed specifically for assessing risk of bias in experimental and quasi-experimental designs based on statistical methods (Hombrados and Waddington 2012).

Analysis

Where data allowed, effect sizes were computed for each study and random-effects inverse-variance weighted meta-analytic methods were used to synthesise the evidence. We followed Petrosino et al. (2012) in the use of standardised mean differences (SMDs) for synthesising both continuous and dichotomous outcomes. The different formulas used in calculating effect sizes are detailed in the full technical report. The included studies were analysed by outcome and Hedges’ g effect sizes were reported. To visibly examine between-study variability in the effect size estimates, forest plots were used to display the estimated effect sizes from each study along with their 95 per cent confidence intervals (CIs). Heterogeneity was explored using both the Q test and the I² index (Deeks et al. 2001; Higgins and Green 2011). Moderator analyses were performed to examine possible reasons for variability in effects. Due to the small number of available studies, an analogue to the ANOVA analysis (univariate) approach was used (Lipsey and Wilson 2001). The categorical variables which identified the sub-groups used in this exercise were specified in advance.

Results

Size of literature reviewed

Of the 8072 articles identified using the electronic search, 145 were deemed potentially relevant on the basis of title and abstract and the full papers were retrieved for a full text review. The full papers of a further 76 items identified through the non-database search were also manually examined. This process resulted in a total of 30 different studies, described in 55 separate publications, being identified as meeting the eligibility criteria. Of these, 26 studies were included in the review.^a The study search and selection process is detailed in Figure 1.

Figure 1

Study search and selection process flowchart.

Descriptive analysis

Publication dates of the 26 included studies ranged between 2001 and 2011, and over three-quarters were published as technical reports. Three studies were randomised experiments and 23 studies used a quasi-experimental design (QED). The QED evaluations used different econometric techniques to address selection bias and net out the impacts of other factors, most commonly propensity score matching (Rosenbaum and Rubin 1983). The majority of study samples included both male and female participants, and a single study focused exclusively on young women. The risk of bias assessments resulted in five studies graded medium quality (Attanasio et al. 2011; Card et al. 2011; Chong and Galdo 2006; Hicks et al. 2011; Espinoza 2010), 10 graded low/medium (Acero et al. 2011; Aedo and Nuñez 2004; Aedo and Pizarro 2004; Analítica Consultores 2006; Chong et al. 2008; Díaz and Jaramillo 2006; Elías et al. 2004; Ibarraran and Rosas-Shady 2006; Jaramillo et al. 2007; Ñopo et al. 2007) and 11 graded low (Alzuá and Brassiolo 2006; Alzúa et al. 2007; Benus et al. 2001; Bidani et al. 2002; Chun and Watanabe 2011; Delajara et al. 2006; Dmitrijeva 2009; López-Acevedo 2003; Medina and Nuñez 2005; Mensch et al. 2004; van Gameren 2010).^d

Twenty different TVET programme interventions were evaluated. Settings include 10 upper-middle income countries (Argentina; Bosnia and Herzegovina; Brazil; Chile; China; Colombia; Dominican Republic; Latvia; Mexico; Panama and Peru); two lower-middle income countries (India and Bhutan); and one low-income country (Kenya).^e Twelve programmes were specifically targeted at youth. The majority were targeted at disadvantaged populations, based on criteria such as household income, education level, and employment experience. A few were occupation- rather than age-focused (targeting, for example, demobilised soldiers or unemployed former employees of state-owned enterprises). A small number of interventions were directed exclusively at either urban or rural residents.

The 20 TVET interventions consisted of one or more of the different models or approaches to vocational skills development (see Table 1). The different types of TVET are not evenly represented, however; for instance, no apprenticeship training programmes were evaluated and a very small number of studies focused on technical education or vocational education. Many of the interventions in the included studies were complex, multi-component programmes. Alongside TVET-related training, some programmes incorporated labour mediation services (such as job readiness skills), some included training specifically for self-employment and some included other types of training (such as basic skills). However, the largest category of multi-component programme involved components consisting of different types of TVET. There were nine two-phase TVET interventions, most of which combined theoretical and practical training, usually in the format of classroom-based vocational training followed by a period of on-the-job training to provide beneficiaries with work experience. These interventions typically focused on short-term semi-skilled training in specific occupations demanded in the private sector.

Table 1 Main intervention characteristics

Primary study authors often did not provide all the necessary data for computing effect sizes. It was possible to calculate at least one effect size for 10 of the studies included in the review (Acero et al. 2011; Aedo and Nuñez 2004; Aedo and Pizarro 2004; Attanasio et al. 2011; Card et al. 2011; Elías et al. 2004; Espinoza 2010; Hicks et al. 2011; Ibarraran and Rosas-Shady 2006; Medina and Nuñez 2005). The findings from these studies were statistically combined using meta-analytic techniques.

Overall mean effects of interventions

The overall mean effect sizes for five outcomes are presented in Table 2. For three outcomes, an overall positive and statistically significant effect on TVET on youth was found: overall paid employment (g = 0.134; 95% CI [0.024, 0.243]); formal employment (g = 0.199; 95% CI [0.055, 0.344]); and earnings (g = 0.127; 95% CI [0.043, 0.21]). For working hours worked, the overall effect was also positive, but non-significant (g = 0.043; 95% CI [-0.017, 0.104]). In contrast, the overall effect on self-employment earnings was negative, and non-significant (g = -0.025; 95% CI [-0.11, 0.061]).^b

Table 2 Overall mean effect size and heterogeneity statistics

Analysis of homogeneity

For three outcomes (overall paid employment, formal employment, and earnings) the grand mean provides some evidence that the TVET interventions were, on average, effective; although, effects are generally small and difficult to detect. Furthermore, in each case the result of the statistical test for homogeneity was statistically significant (Q = 23.8, p < 0.05; Q = 11.1, p < 0.05; Q = 25.5, p < 0.05), indicating that studies disagreed on the magnitude of effect. Our next step was to explore the reasons for this variability between studies.

Analysis of moderator effects

Moderator analyses were performed to examine independent variables that may be contributing to the heterogeneity. The results are presented in Table 3.

Table 3 Moderator analysis results for categorical factors

Relationship of study characteristics to effect size

Two factors potentially associated with treatment effect were examined: study quality and length of follow up (i.e., timing of outcome measurement). As there were no high quality studies included in the review, studies rated medium quality were compared with those rated low or low-medium.^c Short-term treatment effects (measured at approximately six months) were compared with medium-term effects (measured at approximately 12 months). The low quality studies were found to have consistently larger mean effects than the medium quality studies; and, for two outcomes, differences in mean effects observed between studies according to study quality were statistically significant: paid employment (Q_b = 6.49, p = 0.0108); formal employment (Q_b = 10.6, p = 0.0012). Although studies that measured short-term treatment effects consistently had slightly larger mean effects than studies in the medium-term follow up group, no statistically significant relationship between length of follow up and treatment effect was observed for any of the outcomes for which we had data.

Relationship of participant characteristics (gender) to effect size

For one of the five outcome categories (weekly hours), statistically significant differences in mean treatment effects were observed between studies according to gender (Qb = 10.1; p = 0.00151). Treatment effects for female youth were positive (g = 0.16; 95% CI [0.04, 0.28]), while those for male youth were negative (g = -0.09; 95% CI [-0.2, 0.01]). However, male youth participation in TVET demonstrated effects that were not statistically different from zero, lessening our confidence in this result.

Relationship of intervention characteristics (type) to effect size

Slight differences in mean effects between two-phase models of TVET and other models were observed, with two-phases usually, but not always, producing the larger effects. However, these differences were not statistically significant.

Conclusions

This review improves upon prior work by statistically synthesising TVET intervention research, strengthening the evidence base on which current policies and practices can draw. However, the nature of the available evidence, and limitations in the review itself, are such that drawing strong inferences from the results of the analyses is not recommended and considerable caution should be used when interpreting and applying the findings of the review.

Despite renewed investment in TVET and increased pressure for evidence-based decision-making, there remains an overall scarcity of research in this area and specific knowledge gaps. The interventions included in this systematic review are not representative of all TVET programmes in existence. Studies were undertaken in a relatively small number of counties, predominantly in Latin America and the Caribbean. Some types of TVET are poorly represented in the review; for example, no studies examined the effectiveness of apprenticeship training. The quality of the existing evidence base is also a concern. Only three RCTs were identified, and methodological and reporting shortcomings of the included studies were such that none provided high quality evidence to help answer the review questions.

Overall, interventions included in the meta-analyses were found to demonstrate a significant, though small, positive effect on a limited range of labour market outcomes: overall paid employment (g = 0.134; 95% CI [0.024, 0.243]); formal employment (g = 0.199; 95% CI [0.055, 0.344]); and monthly earnings (g = 0.127; 95% CI [0.043, 0.21]). However, the observed heterogeneity was also significant, indicating that different studies point to somewhat different conclusions. Attempts to explain the heterogeneity in overall mean effects suggest that methods matter. The low quality studies have consistently larger mean effects than the medium quality studies. For two outcomes, evidence of a statistically significant relationship between study quality and effect size magnitude suggest that the overall mean effect may be inflated and that our conclusions about treatment effect should be based only on those studies rated medium quality: paid employment (g = 0.06; 95% CI [-0.01, 0.12]); formal employment (g = 0.12; 95% CI [0.05, 0.19]). Treatment effects for both self-employment earnings and weekly hours worked were non-significant. On the whole, the other independent variables that were tested were unable to explain variability in effects. Length of follow up did not demonstrate a significant relationship with treatment effect. Two-phase TVET interventions (classroom-based vocational training followed by on-the-job training) produced statistically similar effects to other TVET models. Gender may have some influence, as an increase in the number of weekly hours worked by youth following participating in TVET appears to be driven by the effect on young women. It is important to note, however, that due to an insufficient number of studies reporting relevant data, only some of the moderator analyses that had been planned a priori could be performed. Furthermore, due to the very small number of studies for most of the variables tested, and thus low statistical power, we may not have been able to detect moderator effects that may indeed be present. In addition, there may be other moderating variables that either were not tested in this study or measured in the primary reports, such as implementation fidelity, whether the intervention was theoretically informed, and so on, which could account for the differences in effects between studies.

While a key strength of this study is its application of systematic review principles to improve upon prior work. There are several important limitations to this review that should be acknowledged. First, not all eligible studies are included in the meta-analysis. Second, the methods for calculating comparable effect sizes from studies using complex econometrics methods, as used in this review, are under-developed and require further research (for a complete discussion, see Becker and Wu 2007; Duvendack et al. 2012; Lipsey and Wilson 2001). Third, no high quality studies were identified and some of the methodological concerns associated with those that were included may mean that the studies have yielded biased estimates of treatment effect. All conclusions from the current review are therefore sensitive to the possibility that the results from the meta-analyses may be over- or under-estimating the effects of TVET on the outcome variables of interest.

Limitations notwithstanding, this systematic review provides some evidence to support the claim that participation in TVET improves the labour market situation of youth in LMICs, on average, when compared to youth who do not participate, with the strength of the evidence strongest for formal employment and monthly earnings. Thus, it is reasonable to conclude that it is worthwhile to continue investment in TVET provision. Although, statistically, the effects of TVET on youth may be small, even a slight increase in the employment rate, for example, could potentially translate into fairly large numbers of young people entering the labour market, especially where programmes are delivered nationally. Through its use of statistical meta-analysis, this review has improved upon previous work; nevertheless, there are two main related areas of likely disappointment for policymakers and programme implementers. First, not only were the observed effects generally small and difficult to detect, the review does not provide conclusive evidence of the effectiveness of TVET on youth employment outcomes. This suggests an urgent need to improve the rigour of the evidence base, and will require policymakers, practitioners and other relevant stakeholders engaging critically with the barriers to effective research production and dissemination in this area. The commissioning of RCTs and robust QEDs is crucial for generating evidence capable of supporting causal claims. To improve potential generalisability, it is also important that appropriate support, financial and otherwise, is given to rigorous outcome research on all types of TVET that have been implemented internationally for young people, and in a broader range of geographical settings than covered by the current evidence base. A cost-benefit analysis was outside the scope of the current review; however, attention should be given to the collection of data regarding the costs of TVET interventions, and examinations of cost-effectiveness should be a priority in future impact evaluations and systematic reviews. Second, the available evidence does not allow recommendations to be made for, or against, investments in a particular model of TVET. Although claims have been made in the literature and media for the success or effectiveness of a particular format of TVET widely implemented in Latin America and the Caribbean, the evidence from this systematic review suggests that such claims are premature. In the absence of evidence in support of any one model or approach to vocational skills development, it seems reasonable in the meantime for policymakers and other stakeholders to select the least costly, most culturally acceptable, and/or most easily implemented programmes, while at the same time becoming good consumers of evidence and working towards the strengthening of the evidence base on TVET. This review has demonstrated that not only is there is a clear need for additional primary research in this area, but that more of the same will not suffice. The methodological shortcomings of the current evidence base, and specific knowledge gaps, suggest a number of future research priorities. These are summarised in Table 4.

Table 4 Summary of recommendations to improve study quality and fill research gaps

End notes

^a Four eligible Spanish-language studies could not be included in the review. Of these, one study was judged as meeting the review selection criteria on the basis of detailed descriptions in previous literature reviews, but efforts to obtain a copy of the publication were unsuccessful. The three remaining eligible non-English language studies were not included in the review due to exhaustion of financial resources. All four studies evaluated ProJoven, a Peruvian programme that was evaluated by a number of studies that were included in the review.

^b A preliminary assessment of the ‘direction of effects’ of the 16 studies not included in the meta-analyses suggests that these studies generally support the meta-analytic findings.

^c For this exercise, all of these studies were grouped together into a single category labelled low.

^d One study (Espinoza 2010) was graded medium quality overall; however, individual assessments for the different outcome categories varied. For employment outcomes, this study was graded as proving medium quality evidence; for monthly earnings, the grading was low/medium.

^e World Bank country classifications have been revised twice since the conduct of this review, and Chile and Latvia are currently ranked as high income economies.

Authors’ information

JT is a lecturer in Social Policy, based at the EPPI-Centre (Evidence for Policy and Practice Co-ordinating Centre), Social science Research Unit, Institute of Education, University of London. JH is a research analyst at J-PAL (Abdul Latif Jameel Poverty Action Lab), based at its regional office in Latin America at the Pontificia Universidad Católica de Chile.