Considerations for career intervention services in global youth workforce development: consensus across policy, research, and practice

Abstract

As yet, despite ongoing gaps in educational and career achievement, earning potential, and other psychosocial risks that perpetuate marginalization among certain populations globally, elements of effective career education and interventions have yet to be agreed upon. Thus, we sought consensus among experts across the domains of policy, research, and practice. We detail the process and results of a recent Delphi study conducted with global youth career and workforce development experts across policy, research, and practice domains who were surveyed about their opinions regarding concepts such as ecological perspectives, needs assessment, trainer preparation, participant curriculum, curriculum delivery, and program evaluation—the six emergent areas of a recent comprehensive synthesis of the literature across these domains (Nassar and Al-Qimlass, in Career builders: key components for effective global youth career and workforce development. RTI Press, Research Triangle Park, 2017a). Our Delphi study yielded 199 consensus statements, subsequently organized into 28 themes, or considerations, for effective global workforce development initiatives. In conclusion, we provide implications, including implementation strategies for key stakeholder groups.

Introduction

Youth constitute one-quarter of the global working population (International Labour Organization [ILO] 2013). On the other hand, youth unemployment and poverty rates are increasing at alarming rates in both developing and developed countries (ILO 2016). Moreover, almost 43% of global youth eligible for the labor market is either unemployed or working, but still living in poverty (ILO 2015). In spite of the fact that decent and sustainable work promotes both physical health and psychological well-being for an individual (Blustein et al. 2017), a deficit of decent work opportunities affects about one in three youth aged 15 to 24 (United Nations 2015).

Literature across policy, research, and practice domains highlights gaps in educational and career achievement, leading to potential psychosocial risks for global youth; hence poverty, lack of social justice, and the consequent marginalization in the society. Conversely, decent and sustainable work facilitates hope, dignity, resilience (Herr 2001) and other positive physiological and psychological well-being among individuals (Blustein et al. 2017) and society (Schiersmann et al. 2012).

Definitions

It is important at this point to discuss the terms found often within the career literature (which can be focused on both youth and adult populations): technical and vocational education and training (TVET), career development, career counseling, career guidance, and career management. TVET is focused training regarding the attainment of knowledge and skills for a specific vocation (UNESCO-UNEVOC International Centre 2018). Career development is a lifelong developmental process that encompasses: interests, abilities, values, personality, background and circumstances (Brown 2002). Career counseling is a term used frequently in the United States and is goal oriented counseling that assists people in understanding themselves and the world of work; in order to make current and future career, educational, and life decisions (Royal 2017). Similarly, career guidance is used more frequently on the global stage and allows a bit more flexibility in goals and delivery in an effort to meet the needs of the community. Similarly, career management is utilized in European and other global contexts to refer to a lifelong process through which individuals guide their respective exploration and acquisition of skills, aptitudes, abilities, and attitudes that can be supported by purposeful interventions (Neary et al. 2016). Building on these aforementioned terms and research within the global youth workforce development literature, this paper will use the term workforce development (WFD) to include all of these aforementioned subtypes and categories.

Gaps and needs

It is also important to note that elements of effective career development interventions have yet to be identified and agreed upon across the various domains within youth WFD implementation system (i.e. research, practice, and policy). For example, many major meta-analyses of career intervention effectiveness have been conducted over several decades (e.g., Brown et al. 2000; Liu et al. 2014; Oliver and Spokane 1988; Whiston et al. 1998, 2017), with one of very few definitive conclusions being the importance of counseling and other social support (Brown 2016). Moreover, one earlier meta-analysis indicated computerized interventions as being among the most cost-effective (Whiston et al. 1998). Further, career interventions worldwide can range from those interwoven throughout the educational system (as early as primary education levels), internship/apprenticeship programs, and standalone programs for youth outside of the workplace (Karacan Özdemir and Nassar 2018). Career management interventions can vary across countries and regions (Neary et al. 2016). Theoretical frameworks, too, can vary widely. Social cognitive theory is one frequently found in the literature (Lent et al. 2002; Liu et al. 2014), along with Bloom’s Taxonomy (1956) which focuses on knowledge, skills, and attitudes; but range from a focus on individuals to one on the larger community (e.g., Nassar-McMillan and Vuorien 2012; Bloom 1956; Bronfenbrenner 1994; Dawis 2002; Gottfredson 2002).

Prior research underpinnings

In response to this lack of agreement or consistency across those who develop and implement, and perhaps most importantly, fund, such interventions, a literature meta-synthesis was recently conducted spanning a variety of documents and disciplines relative to youth WFD globally (Nassar and Al-Qimlass 2017a, b). The authors employed a qualitative content analysis for their systematic content analysis and subsequent interpretation of the data (Holsti 1969; Rosengren 1981). The first step yielded four topics or themes, which included best practices, promising considerations, foundational frameworks, and shortcomings or gaps. The second step of classifying the data sources by relevant topic or theme yielded six key components that need to be addressed for ensuring more holistic and encompassing curriculum for global youth WFD across policy, research, and practice domains: (1) ecological perspectives (i.e. holistic, systemic, and comprehensive perspectives and intervention approaches), (2) needs assessments (i.e. identifying relevant systems, deficits, and resources), (3) trainer (i.e. existing and prospective career education and WFD providers) curricula, (4) participant curricula (i.e. education and skill-focused training materials for all consumer groups), (5) delivery (i.e. where and how interventions will occur), and (6) program evaluation (i.e. relevant data points and sources).

Framework for the study

Building upon the recent 2017 synthesis, our Delphi study utilized these six key components as a framework through which to reach a consensus on critical considerations for effective career and WFD curriculums and programs for youth. Our study was guided by a broad and multi-faceted ecological framework. The Nassar and Al-Qimlass study (2017a) results suggested a variety of ecological, or layered, systems. For example, at the core is consumer systems; embedded within community systems, embedded within implementation systems. Each of these systems is comprised of a variety of domains. For example, the consumer system subsumes groups such as individual youth, their parents, etc. Community systems include employers, trainers, facilities and other infrastructural components. Finally, the implementation system includes policy, research and practice domains.

Prior research has illuminated a wide age range included in definitions of youth across various literature bases (e.g., EQUIP3/Youth Trust 2005; EQUIP3 2008; ILO, n.d.b; National Youth Employment Coalition, n.d.; S4YE, 2015; YouthPower, n.d.). We elected to define youth in the present Delphi study as individuals aged 10–35. Potential experts recruited for participation in the study were those involved in the domains of research, policy, and practice associated with the implementation system of youth WFD programs. The inclusion of these three domains support the ecological framework of the study, due to their varied experiences, expertise, and knowledge base. It was deemed essential for this study to include perspectives from these three domains in order to adequately obtain a holistic understanding of youth WFD programs. Examples of these potential research participants include members of national and international career development associations, members of policy-making institutions around the globe, small scale and large scale practitioners (including programs funded via local, national, or international means), and members of agencies dedicated to developing global youth WFD interventions.

Further, the Delphi method—as a best available forecasting tool—was used to obtain deeper knowledge and opinions of the domain expert-research participants (i.e. research, practice, and policy professionals). This particular method is defined as the process through which consensus is reached regarding a specific topic (Doughty 2009; Hsu and Sandford 2007; Wester and Borders 2014). A group of experts, meeting criteria set forth by the researchers, are asked to provide their input on a given topic. Their initial responses are analyzed and condensed into statements for further review. These statements are then passed back to the experts for their review and continued input regarding their level of agreement with the statements. This cycle of statement review is continued until either consensus among all of the experts is reached, or it is determined that consensus will not be reached (Doughty 2009; Linstone and Turoff 1975; Wester and Borders 2014). This method was chosen for its ability to collectively assess the input of experts representing the implementation system’s three domains of research, policy, and practice. Furthermore, building from the ecological basis of this study, the Delphi method suggests that a more diverse panel will provide greater benefit to the study as multiple points of view will be accounted for (Murphy et al. 1998; Scheele 1975).

Summary

Thus, the purpose of this study was to achieve consensus among experts on what, specifically, constitutes an effective youth WFD intervention. Our study was guided by an ecological theoretical and philosophical approach based on the Nassar and Al-Qimlass (2017a) literature synthesis and our own years of expertise in the field. This ecological underpinning was prevalent at multiple layers and levels. The methodology was based both in spirit and letter on Delphi methodology best practices. The final aim of this paper is to highlight implications of our findings on effective and holistic youth WFD programs. Results of this paper also aims to highlight the implications the findings have on the creation of an effective holistic career development program.

Methods

As stated previously, the Delphi method of inquiry seeks to obtain consensus among content “experts” regarding a specific topic (Doughty 2009; Hsu and Sandford 2007; Wester and Borders 2014). Hallmarks associated with the method include a group of experts, the use of statistical analysis, multiple rounds of data collection, structured or “controlled feedback”, and anonymity of the research participant (Dalkey 1969, 1972; Doughty 2009; Hsu and Sandford 2007; Wester and Borders 2014). Each of these hallmarks is further explained in the following sections.

Research participants

Research participants of a Delphi study are experts within their respective fields; the delineation of what accounts as an “expert” however is determined by the researcher/s and should be in line with the purpose of the study (Doughty 2009; Wilhelm 2001). Due to the centrality of the expert panel to the results, the selection of experts is tantamount to the strength and validity of the study (Clayton 1997). Delphi research participants should represent expertise in their respective domains relative to the content focus of the study (Doughty 2009; Wilhelm 2001) and should represent a diverse group to obtain multiple perspectives (Murphy et al. 1998; Scheele 1975). For the current study, experts in policy, research, and practice across the field of global youth WFD were identified using the following criteria set forth by the researchers: (1) work in the field of career guidance; (2) have worked in career and WFD within the past 5–10 years; and (3) worked with youth, ages 10–35. Additionally, at least one secondary criteria was met based on self-identified domain of expertise:

• Policy: Written policy at local, national, regional and/or international levels.

• Research: Published relevant peer reviewed journal articles or research reports.

• Practice: Have applied experience in developing or implementing youth career and/or WFD programs.

Recruitment

Research participant recruitment included both purposeful and snowball sampling strategies. A list of individuals known by the researchers (e.g., through professional networks, etc.) was first developed in an effort to adequately survey a comparable number of experts within each of the three domains (i.e. practice, policy, and research). From this list, a recruitment email was sent out, as well as a request to forward the information to any other individuals that may meet the selection criteria.

A wide net for research participants was cast due to the fairly common issue of attrition that can be found in a number of research methodologies, but particularly in the Delphi method which can be anywhere from 18 to 70% (Smith 2016). A contributing factor to this high variable rate of attrition of Delphi participants may be their experience of “participant fatigue” due to the amount of time and energy needed to complete the multiple rounds (Powell 2003; Vázquez-Ramos et al. 2007). Additionally, if the initial questionnaire is not well written or engaging, initial research participants may lose interest (Stone Fish and Busby 1996).

Based on the literature, the total number of adequate research participants for a Delphi study ranges from approximately 10 to 15, up to 30 (Clayton 1997; Linstone and Turoff 1975; Doughty 2009; Ziglio 1996). Therefore, the goal of recruitment was to obtain approximately ten experts representing each field of expertise; i.e. research, practice, and policy.

Procedure

After receiving appropriate institutional research approval, we carried out three rounds of data collection and analysis, consistent with Delphi methodology (Fletcher-Johnston et al. 2011; Hsu and Sandford 2007; Wester and Borders 2014). The first round of data collection and analysis utilized a questionnaire developed by the researchers (outlined below), with each subsequent round building and refining upon the initial responses. More detailed accounts of the content and statistical analyses completed at each round are given in the “Data collection and analysis” section.

Instrumentation

Delphi methodology utilizes “structured communication” in the process of obtaining consensus (Doughty 2009; Herlihy and Dufrene 2011; Linstone and Turoff 1975; Smith 2016). This type of structured communication begins with an initial questionnaire that is rooted within an empirical basis (i.e. literature and research findings), but ultimately developed by the researcher/s (Smith 2016). For the current study, the initial questionnaire, presented to first round research participants, was developed based on a 2017 literature synthesis focusing on effective career and workforce development factors for global youth (Nassar and Al-Qimlass 2017a, b). The questionnaire consisted of approximately 55 open ended questions organized by the six key component areas of youth WFD programs previously described (i.e. a holistic, systemic and comprehensive framework; needs assessment; trainer curriculum; participant curriculum; delivery; and evaluation). The nature of the approximately 55 open ended questions came directly from the findings of the 2017 literature synthesis; including best practices, promising methods, theoretical bases, shortcoming and gaps. Additionally, while the first key component of youth WFD programs (or in this case, the first section of the questionnaire) is to have a holistic, systemic and comprehensive framework, we wanted to make sure that each following component (or questionnaire sections) were based in the ecological foundation of the study, as well as guided by the literature. Therefore, each section of the questionnaire included questions regarding the various systems impacted, community specific concerns, general ethical and cultural issues to take into account, and an option for research participants to include items or concerns not already covered. (Note: the systems included in the questionnaire cover Implementation, Consumer, and Community. For the purposes of this research, the Implementation system encompasses those who are directly involved in the application and operation of youth WFD programs; including policy makers, researchers, and practitioners. The Consumer system encompasses those who are both directly and indirectly receiving the youth WFD program’s product such as youth and their parents. Lastly, the Community system encompasses aspects of the society that are unique to the community itself). Sample items from the questionnaire can be found in Table 1; for full access to the questionnaire please contact the authors.

Table 1 Sample questionnaire items

Data collection and analysis

Delphi research is one that uses aspects of both qualitative and quantitative methodologies (Iqbal and Pipon-Young 2009). As previously described, a group of experts, meeting criteria set forth by the researchers, are asked to provide their input on a given topic. Their initial responses are qualitatively analyzed and condensed into statements for further review. These statements are then passed back to the experts for their review and continued input regarding their level of agreement with the statements; now using both qualitative and quantitative analyses. This cycle of statement review is continued until either consensus among all of the experts is reached, or it is determined that consensus will not be reached (Doughty 2009; Linstone and Turoff 1975; Wester and Borders 2014). To complete this cycle, this research study sent out surveys electronically for each round of data collection (a direct link to the questionnaire was included within the recruitment email and subsequent emails), using the online survey tool Qualtrics.

Round one

The first round of data analysis implores the use of content analysis to identify themes from the open-ended questions of the initial questionnaire (Iqbal and Pipon-Young 2009; Powell 2003; Wester and Borders 2014). Upon completion of this round of analysis, a list of statements were generated based on the common themes identified. Statements were then organized by question topic, summarized, and condensed to eliminate redundancy (total statements included in Round two questionnaire; N = 281).

Round two

Experts were asked to rank their level of agreement with the newly created statements using a seven-point Likert scale (one = full agreement; seven = full disagreement). This round also included space for comments from the panel of experts. Analysis involved both content and statistical analyses. Comments were analyzed following the same method as in round one, and reinserted into the third round as new statements to be ranked using the Likert scale. Additionally, descriptive statistics were calculated (using the statistical software, STATA) to determine the level of consensus that was reached for each of the Likert-rated statements; specifically, median scores and interquartile ranges (IQR) which are most common in Delphi methodology (Doughty 2009). Median scores “are used because it is a measure of central tendency, denoting a middle point on a frequency distribution,… does not allow an outlying score or opinion to skew the final score… and median is appropriate to use with small groups” (Wester and Borders 2014). Interquartile ranges “identif[y] the level of consensus within a distribution of scores, representing the middle half of responses” meaning that “a smaller IQR indicates a higher degree of consensus” (Wester and Borders 2014). The criteria for consensus was determined at the onset of the study, and set at a median of 6.0 or higher (due to the use of a seven-point Likert scale) and an IQR of 1.0 or less. Upon completion of round two analysis, those statements meeting full consensus (i.e. both the minimum median score and minimum IQR range) were removed from the questionnaire and recorded. Those statements meeting partial consensus (i.e. either the minimum median score or the minimum IQR range) remained in the questionnaire (N = 77). And finally, those statements that did not meet consensus (i.e. neither minimum median score nor minimum IQR range) were removed from the questionnaire and discarded.

Round three

Experts were asked only to rank their level of agreement with the statements provided. Statements provided included new statements from the content analysis completed in round two, and those statements meeting partial consensus from round two. Analysis of the data collected during this round involved only descriptive statistical analyses to determine level of consensus, and followed the same steps as outlined in Round two. During this final round of data analysis, those statements meeting full consensus (i.e. both the minimum median score and minimum IQR range) were removed from the questionnaire and recorded; all others were discarded.

Results and discussion

Results of the data collection and analysis are provided, along with a review of the expert panel demographics.

Research participants

The initial recruitment email was distributed to approximately 80–100 individuals but the full number recruited is unknown due to the nature of snowball sampling and our procedure allowing for initial contacts to share the information with others of their selection. The second round was sent only to those 20 individuals who completed round one, resulting in 12 completed response, and round three resulted in ten.

Demographics

Of the 20 research participants who completed the first round of questions, 17 indicated expertise in policy, 13 in research, and 19 in practice. Of these 20 research participants, 3 reported a single area of expertise; 5 reported 2 areas of expertise; and 12 reported expertise in all three areas. Additionally, nine were female and 11 were male, ranging in age from 29 to 76. Of the full sample, 10 had Ph.D.s and 7 had master’s degrees. In terms of ethnicity, six self-identified as Anglo-Saxon; two as German; two as Finnish, with the remainder each singly representing various other ethnicities. Regarding geographic areas of expertise, the following continents were represented with the corresponding number of mentions: North America = 3; South America = 1; Africa = 3; Asia = 13; Europe = 22; Australia = 1.

Findings

The items meeting full consensus (N = 199) were reviewed and organized into themes (i.e. N = 28; these represent overarching considerations), within the six key components of youth WFD used to construct the initial questionnaire. Those considerations, along with their corresponding items and median and IQR scores are reported in Table 2.

Table 2 Key components, considerations, and items (with Delphi scores)

Discussion of findings

Table 2 includes all those statements that met consensus among our panel of experts. The statistical data of median scores and IQRs for each statement can lead to a deeper understanding of how the experts converged on a given item, however, as is consistent with Delphi methodology, meeting the minimum criteria for consensus indicates inclusion only and does not speak to priority or importance.

The majority of items that met consensus and became part of the final results were not surprising to the researchers as they were consistent with the majority of recommended best and promising practices; as well as anecdotal data collected throughout the researchers’ work. However, during the data collection process and analysis, it was surprising that some of the items (of which the researchers either felt strongly about or assumed the expert panel would feel strongly about), did not meet the minimum criteria to be included items, such as the use of field practice, the difference between integrated and stand alone program locations, and certain types of outputs and outcomes for youth WFD program evaluation. A possible area of further study could come from an analysis of those items that did not meet inclusion criteria. Additionally, while the overall amount of content or items within each section (or key component) varies, it was not surprising to the researchers to see these differences, because while the initial content (i.e. questionnaire) was driven by the literature, it was refined and developed via the expert panel’s comments and rankings; indicating areas or sections needing more attention and detail.

An interesting point of inquiry stemming from the results is the ultimate level of agreement or disagreement that experts from the different domains had. Unfortunately, this level of inquiry is unable to occur with the current data set because responses cannot be divided along such lines as the research participants were able to self-identify as experts in more than one domain (research, practice, and/or policy). The researchers opted for recruiting a heterogenous expert participant group and this corresponding method of identification to engage their multifaceted expertise instead of using a forced-choice response in which all viewpoints may not have been accounted for.

Results within the key components. Each item that met consensus among our panel of experts indicates a direct measure or component that should be addressed when developing and implementing a youth WFD program. These items can be used by researchers, policy makers, and practitioners as a real-world real-time method to evaluate current and future programs. Further discussion of the study’s findings will review the considerations and items as they relate to the six key components of youth WFD, and will highlight relevance and connection to previous literature and findings.

First, the creation and implementation of a youth WFD initiative should be holistic, systemic, and comprehensive. This key component addressed: being developmental and progressive (e.g., age specific and life-long), ethical and cultural considerations (e.g., being customizable), and being rooted within community (e.g., building awareness and interest in the community). Youth WFD programs should encompass an ecological perspective, that is, recognize the influences of interrelations between subsystems—namely microsystems, mesosystems, exosystems, and macrosystems—within in a larger ecosystem of human behaviors (Bronfenbrenner 1977; Young 1983). For these reasons, it is important to consider factors across all of the interrelated subsystems; starting from the unique differences among individuals to the greater commonalities within the community of focus. Furthermore, it is important to utilize innovative career theories that account for the changing socio-economic sphere, include multiple institutions and organizations, build awareness and interest in the community by incorporating initiatives within the program, and are culturally appropriate to the community. In summary, these considerations create holistic frameworks and provide explanations about the what, why, and how of career development of individuals within a contemporary and culturally appropriate context (Poole 2002).

Secondly, the findings indicated the necessity of needs assessment. Supportively, the National Career Development Association [NCDA] (1997) stated having knowledge of needs assessment and evaluation techniques and practices is one of necessary competencies to develop, plan, implement, and manage comprehensive career development. Under this component, considerations indicated process, logistics, and accessibility issues of implementation. In addition, identifying and involving stakeholder groups is important (Career Education Association of Victoria 2010)—these can include the youth themselves, individuals (e.g., parents, counselors, and researchers), programmatic partners (e.g., educational, employment, and social programs), and professionals (e.g., local and regional agencies, private sector, trade unions, and professional associations). Proactively integrating youth as a key stakeholder group at this stage can promote feelings of personal responsibility and a sense of ownership connected to the program (Olguin and Keim 2009; vanBruinswaardt et al. 2015). Moreover, collaboration between these stakeholders is critical for effective youth WFD programs, and includes communication and joint decision making.

The third key component involves trainers and trainer curricula within global youth WFD programs. Findings stated that trainer curricula should be developed from multiple sources (e.g., demographics of the participant/end-user population, previous curriculum evaluation, aspects of Positive Youth Development etc.), and include psychosocial interventions (e.g., crisis and trauma, substance abuse, disability, etc.) as well as operational and business components (e.g., innovative procedures, social entrepreneurial skills, marketing strategies etc.). Existing community infrastructures (for example, classroom teachers) could provide a base for trainer selection. In addition, country-specific situations (e.g., recruitment of external education and career curriculum experts) should be engaged (Nassar and Al-Qimlass 2017a). For instance, USAID (2005) underlined the importance of developing alliances between trainers as all might have different capabilities. Indeed, the success of youth programs aimed to increase life skills for workplace development and success is based on selection of skilled and talented trainers (International Youth Foundation [IYF], 2014). Moreover, providing in-service training programs and quality standards for teaching staff serves to build and strengthen vocational training systems (Goldin 2015).

Participant curricula emerged as the fourth key component. First, this curriculum should build upon multiple sources and input, including: labor market demands, valid assessment tools, needs of participants, the characteristics of trainers, real-world issues, and feedback from parents and other stakeholder groups. Watts (2001) stated that career interventions should be curriculum based, and should incorporate personal, social, and health components. At the same time, curricula need to incorporate career education, guidance, and counseling concepts, including technical and vocational education and training, soft skills, and transferable skills depending on the expectations of various stakeholder groups (Hughes et al. 2005). The inclusion and incorporation of soft skills into youth WFD programs is becoming more common due to the recognition that these skills are equally as important as occupational skills to function effectively in the workplace and to be successful in labor market (Duncan and Dunifon 1998). Furthermore, as indicated by the study’s findings, it is important to establish a bridge between all of components described, as well as being as reflective as possible of the changing spheres of the world of work (ILO 2011). Last, these programs must also take into account any vulnerabilities within the participant group(s), such as educational, literacy, and developmental levels, and address them accordingly in the curriculum development process (Nassar and Al-Qimlass 2017a, b).

Curriculum delivery, as the fifth key component, applies to all participant groups in terms of the “where” and “how”. The “where” suggests either integrating career curriculum and interventions into existing, broader curriculum structures or implementing the programs in the career centers as a stand-alone model. For example, in most OECD countries (i.e. those 35 countries around the world that are a member of the Organization for Economic Co-operation and Development) career interventions are delivered through the school curriculum (OECD 2003). While in the United States, the Workforce Investment Act of 1998 created a One-Stop Delivery System, located in nearly every community in the country, which provides a network of career training centers (WIA; P.L. 105-220). The “how” encompasses individual versus group approaches, technology-enhanced activities, and peer-to-peer interventions. In fact, previous work suggests that incorporating more than one mode of intervention is more effective than using a single intervention (Mackay et al. 2015). In keeping with the findings, curriculum delivery should pay attention to technology for higher productivity and innovation by considering some crucial issues such as accessibility, local adaptability in terms of culture and language, and quality of content (Goldin 2015). Subsequently, including technology-related skills should be an integral and critical component of youth WFD programs for youth (Making Cents International 2014).

The sixth and final component is evaluation, an imperative part of program development (Nassar and Al-Qimlass 2017a). This study yielded crucial considerations under this component such as the necessity of determining the type of evaluation (i.e. stand-alone measurement vs integrated into a larger evaluation process). In addition, intervention research, as well as outcome (versus solely output) data should be gathered, which could further aid in the reporting required by funding agencies (Nassar-McMillan and Conley 2011). Additionally, when evaluating program outcomes (and outputs), OECD (2003) points out that they may be intended and unintended, behavioral and attitudinal, short- and long-term. Although evaluation can be a challenging step of program development and implementation, it is critical at all stages of development and implementation. Additionally, our research suggests that, although policy makers, researchers, and career practitioners believe that evaluation of career programs is important, there is a need for the development of evaluation assessments and a modification to the current evaluation priorities (Lalande et al. 2006). Furthermore, the current pool of effective (i.e. evidence-based) career development interventions is lacking, which is a result of inadequate outcome-focused intervention research (Baudouin and Hiebert 2007); an issue that intentional and rigorous program evaluation could address. In addition, there is a need for longitudinal research to examine the deep and longer-term effects of career interventions on individuals’ career pathways (Hughes et al. 2016).

Limitations

One of the most notable limitations of the current study and methodology includes the significant, although common, attrition rate from those research participants recruited to those that completed all three rounds of data collection. As the purpose of the methodology is to generate consensus, “[a]ttrition may lead to false consensus or simply to poor summation of the data” (Vázquez-Ramos et al. 2007). The best way to confirm or refute the impact of the current study’s attrition rate would be to replicate the study in some way to confirm consensus.

Another limitation is the exclusion of the “gray labor market” within the initial literature review and synthesis. Workforce development programs typically focus on preparing consumer groups for employment within a legitimized labor market. This may exclude employability within informal markets, sometimes referred to as the gray labor market (Ungerson 1997, 2003).

Conclusions: implications for research, policy, and practice

Results of this study can be used as a means for both program development and program evaluation within the context of ensuring more effective career and workforce development programs for youth. Furthermore, implications should be considered within the overall relationship structure among the implementation systems, in effect, the research, policy, and practice domains. For example, policy makers can solicit input from research and practice for evidence-based and applicable policies. Research can tackle current policies and practices to examine current effectiveness or provide future recommendations. Practice can implement effective career and WFD programs for youth by utilizing research findings and current policies. Coming full circle, policy makers can assess current policies; practitioners can evaluate their career development programs; and researchers can investigate effectiveness of current policies and practices. As stated previously, the manner with which these results can be applied by the various implementation systems fall within the two categories of program development and program evaluation. Below are more detailed examples of how the different implementation systems (i.e. research, policy, and practice domains) can achieve these aims.

How can research benefit from these findings?

Both policy and practice domains need the outputs of research. Current evidence should be provided to inform coherent and effective career and WFD systems for youth (Holzer 2012). Research can also investigate and present the systemic solutions through comparative studies (Sultana and Watts 2006). Moreover, research can evaluate the structure of career development programs from an ecological perspective to foster holistic, systemic, and comprehensive frameworks.

Based on the Delphi results, research can investigate how culturally sensitive holistic, systemic and comprehensive frameworks can be developed as well as serve to structure corresponding evaluations. Secondly, effective career counseling interventions depends on careful assessment of participants’ ecosystems (Flores and Heppner 2002). The needs assessment findings can guide research relative to various stakeholder groups, improving need assessment tools, examining the impact of current needs assessment initiatives on program efficacy, evaluating the current policies on needs assessment, and the like. Regarding trainer curriculum, research can evaluate the effectiveness of the trainer training programs by using these Considerations. In addition, the research can investigate the needs and examine views of trainers with regard to training curricula as well as to guide train-the-trainer initiatives. Moreover, youth WFD programs should be improved consistently by using the existing data (Making Cents International 2014). For example, initiatives can develop soft and transferable skills trainings, investigate the aspects of technical and vocational education and training, and even evaluate the current participant programs according to the needs of the participant groups representing special needs and vulnerabilities. With regard to program delivery, research can use these considerations to explore the current challenges in program delivery and to provide such information for policy makers. Lastly, rigorous research efforts can offer program evaluation toolkits, evidence-based approaches, and even research instruments for evaluation.

How can policy benefit from these findings?

Policies about career guidance, counseling, and education play important roles in a particular nation’s goals to build human capital and develop labor mechanisms (Herr 1996). Consequently, policy makers ideally should receive reciprocal input about gains and costs associated with their current policies and career development practices. Thus, they are concerned with summative evaluations of expected outcomes, along with economic impacts and costs to help inform the development of subsequent policies (Hughes et al. 2005). Therefore, policy needs to receive input from both research and practice domains.

Education policy analysis (OECD 2003) stated that policymakers need basic data regarding the type of services, those who deliver the services, the process of delivery, as well as feedback from participants. The six key components and their corresponding considerations/items developed from this study address each of these issues:

The first key component states that youth WFD program should be holistic, systemic and comprehensive. A holistic perspective requires continuousness, being developmental, taking into account the characteristics of participants and labor markets in an ever-changing world (International Labour Organization 2011). Therefore, policymakers can use each of the results from the current Delphi study to evaluate and adjust (if need be) current policies.

The second key component, needs assessment, focuses on the current needs to be addressed, as identified needs have a crucial role influence on policy making (Kavale 2012). Therefore, policy makers can evaluate the current need assessment policies across different WFD programs, ranging from curricula integrated into existing infrastructures to stand-alone initiatives.

Third, the characteristics of trainer curriculum yielded from this study will inform policy makers about the quality of the current services. As noted by International Labour Organization (2011), a great deal of effort should focus on the quantity and quality of the training, which in turn depends a great deal upon qualified and adequately prepared trainers. Therefore, policy makers periodically call for assessments of trainer preparedness and these assessments can influence decisions such as whether trainer training is compulsory and how training opportunities are provided.

Career development efforts can be effectively conducted throughout the curriculum (Lent and Brown 2006). Moreover, best practices across Europe and beyond include career guidance policies and services for vulnerable groups (Sultana 2004). Correspondingly, the fourth component of participant curriculum could be applied across various groups such as at-risk youth, school age students, disadvantaged groups, young employers, and so forth.

Access to career development services depend on effective policies (Hughes et al. 2005). Program delivery needs to be assessed across these target populations of need to ensure equitable and realistic access.

Finally, to ensure that fiscal decisions and investments are made wisely, policy makers need to ensure that their funded initiatives are indeed evidence based and that they yield evidence of effectiveness (Baudouin and Hiebert 2007). Effective WFD policy can dictate training and evaluation standards based on the sixth component of this study. The current Delphi study results can enable policy makers to enact better preparation of youth for the present and future workforce (Goldin 2015).

How can practice benefit from these findings?

Career practitioners might have concerns regarding formative issues such as how to procure effective outcomes and ensure continuity of the programs, both for individual and organizational stakeholder groups (Hughes et al. 2005). The results of a European survey conducted over 10 years ago (Sultana 2004) indicated paradigm shifts toward career services that have standards of practice and codes of conduct. The findings of our Delphi study could serve to inform such standards of practice. For example, practitioners can utilize the Delphi consensus items as checks and balances to ensure holistic, systemic, and comprehensive frameworks. Next, like for policy makers, needs assessment is a central issue for career practitioners (Kavale 2012). Practitioners can apply the results of the Delphi study as indicators of effective needs assessment procedures and processes, especially considering all stakeholder groups as well as determining end-user consumer needs. Moreover, trainer training being a key Delphi study result, can be utilized by practitioners to evaluate their current strengths and needs and also then, in turn, to advocate for additional resources and support for the professional development of trainers. Although this component requires supportive policies, such as compulsory training and standards, practitioners can employ self-assessment results to advocate for themselves with their employer or institution. Moreover, practitioners need to be able to evaluate their own programs and delivery. The Delphi results can serve as a guide to program development, during the implementation as reference points for process or quality checks, and after program delivery as evaluation items. Delivery items, as the fifth Delphi component, will likewise provide a tool for practitioners to check all aspects of delivery before the implementation process. Lastly, evaluation considerations from the Delphi study will help practitioners to better understand all aspects of their youth WFD programs and interventions. These considerations also provide opportunities for collaborations between local researchers and career practitioners.