EVALUATION STRATEGY FOR 
JOB SERVICES AUSTRALIA 
 
2009 TO 2012 

 

 

 

 
Evaluation and Program Performance Branch 
Research, Analysis and Evaluation Group 
August 2009 

For further information about this publication contact: 
 
Employment Services Evaluation Section 
Research, Analysis and Evaluation Group 
Department of Education, Employment and Workplace Relations 
GPO Box 9880, CANBERRA ACT 2601 
 
General:  1300 363 079 
Switchboard:  13 33 97 
Text telephone for the hearing impaired (TTY):  FreeCALL TM 1800 554 609 
 
http://www.workplace.gov.au/

 

 

 

 

 

© Commonwealth of Australia 2009 

978-0-642-33251-6 – Evaluation Strategy for Job Services Australia 2009 – 2012 [RTF]
978-0-642-33250-9 – Evaluation Strategy for Job Services Australia 2009 – 2012 [PDF]

This work is copyright.  Apart from any use as permitted under the Copyright Act 1968, no part may be reproduced by any process without prior written permission from the Commonwealth available from AusInfo.  Requests and inquiries concerning reproduction and rights should be addressed to the Manager, Legislative Services, AusInfo, GPO Box 1920, Canberra ACT 2601. 
 
Department of Education, Employment and Workplace Relations  
http://www.workplace.gov.au/

Evaluation Strategy for Job Services Australia 

 
Executive summary	1
Introduction	3
A new employment service	3
Evaluation requirement	8
Scope of the Evaluation	8
Evaluation criteria	9
Performance monitoring	11
‘Best’ practice	11
Transition to the new service	11
Specific studies	12
1. Participation in Job Services Australia	12
2. Job Seeker Assistance	12
3. Employer servicing	13
4. Building labour force capacity	13
5. Addressing advantage	14
6. Impact on administrative burden	15
7. Indigenous servicing	15
8. Social inclusion under Job Services Australia	16
9. Effectiveness of Job Services Australia	16
10. Impact of the economic downturn on the employment service	18
Evaluation management	18
Implementation	18
Reporting results	19
Other reviews	19
Resourcing	19
Performance measures	19
Take-up and usage	20
Service type and quality	20
Outcomes	20
Efficiency measures	21
Data sources	21
Employment Assistance Survey	21
Stepping Stones longitudinal survey of income support and employment assistance	22
Post-program Monitoring Survey	22
Service Quality Monitoring Program	22
Administrative data	23

Executive summary 
 
Job Services Australia replaced the Job Network employment service from 1 July 2009. The new service involves four service streams (including a work experience phase in each stream) and seeks to improve the links between labour market assistance and apprenticeships, vocational education and training and state and territory government employment and training programs.  Compared with Job Network, Job Services Australia is designed to place a greater emphasis on addressing skill shortages, social inclusion and targeting more disadvantaged job seekers. 
 
The evaluation will assess how Job Services Australia performs in terms of assisting individuals, particularly those most disadvantaged in the labour market, to obtain skills and secure sustainable employment.  The evaluation will provide information on an ongoing basis to meet the needs of policy makers and program managers.  The timing of the dissemination of findings is intended, in part, to inform future budget decisions and processes for awarding future business.  Information will also be gathered which could potentially be used to identify best practice and subsequently be disseminated to the employment services industry. 
 
The approach adopted in the evaluation includes a number of areas of investigation which reflect the key evaluation issues, the timing of the availability of data and resources.  These areas are: 
	• job seeker participation in Job Services Australia, including initial contact and maintenance of engagement; 
	• the types of assistance delivered to job seekers and the extent to which this assistance is individually tailored to the needs of job seekers and addresses their barriers to employment; 
	• the approach by Job Services Australia providers to employer servicing and the handling of local skill shortages; 
	• raising the skill levels of job seekers to build labour force capacity and the types of training that job seekers are participation in; 
	• participation and outcomes by different types of job seekers (including demographic and disadvantaged groups) to address labour market disadvantage; 
	• reducing administrative burden and its effect on service quality; 
	• servicing Indigenous job seekers including the contribution of the new service to “closing the gap”; 
	• social inclusion; 
	• effectiveness (including impact) and cost-effectiveness from the perspective of the service over all and its main components; and
	• the impact and capacity to respond to the economic downturn. 

The approach to evaluation is flexible, recognising that over time the importance of issues change.  New areas of investigation may be required as a result, while existing ones may need significant redesign or be dropped altogether. 
 
DEEWR has responsibility for managing and conducting the evaluation.  While findings will be disseminated to relevant areas within the department as they become available, it is anticipated that the specific studies will result in a series of evaluation papers.  Subject to Ministerial agreement these papers could be made publicly available on the department’s internet site.  A large scale report consolidating all the evaluation findings is not envisaged. 
 
The main data sources for the evaluation include a longitudinal survey of job seekers, the ongoing Post-program Monitoring Survey and data collected as a by-product of the department’s system for administering the employment services market.  Information gathered through the department’s program for monitoring the quality of services and a number of qualitative studies designed specifically for the evaluation will also be used.   

Introduction 
A new employment service 
Job Services Australia commenced on 1 July 2009 under the Employment Services Deed 2009–2012 (ESD4) and involves an estimated expenditure of $3.9 billion over three years.  The new service replaces Job Network’s continuum of assistance with four service streams (including work experience) and seeks to improve the links between labour market assistance and apprenticeships, vocational education and training, and state and territory government employment and training programs.   
 
A Government review of Australia’s employment services in the first half of 2008 informed the new model of labour market assistance.  The review sought submissions, released a discussion paper1 and held public consultations on the future framework for employment services and how best to implement them.  An assessment of the performance of the existing model (the Active Participation Model) also contributed to the redesign.2
 
Through labour market assistance the Government seeks to ensure that individuals have access to income support and socially inclusive labour market programs which further economic development.3  Job Services Australia contributes to achieving this outcome.  Its objectives are to help individuals to: 
	• obtain the skills they need; and 
	• secure sustainable employment. 

	

1 Department of Education, Employment and Workplace Relations 2008, The Future of Employment Services in Australia: A discussion paper, DEEWR, Canberra. 
2 Department of Education, Employment and Workplace Relations 2007, The APM evaluation: July 2003 to June 2006, DEEWR, Canberra. 
3 Budget:  Portfolio Budget Statements 2008–09, Budget Related Paper No. 1.5, Education, Employment and Workplace Relations Portfolio, Australian Government, Canberra. 

Relative to previous services, the Job Services Australia is also designed to:  
	• increase the focus on the most disadvantaged job seekers;4 
	• achieve greater social inclusion; 
	• boost employment participation; 
	• boost the productive capacity of the workforce; 
	• address skills shortage areas; and 
	• better meet the needs of employers. 

When announced in the 2008 Budget the new employment service responded to concern that the current services were “no longer suited to a labour market characterised by lower unemployment, widespread skill shortages and a growing proportion of job seekers highly disadvantaged and long-term unemployed”.5  Since the budget, the labour market and economy have slowed in response to a downturn in the global economy.  The unemployment rate, for example, has risen from 4.0% in March 2008 to 5.7% in March 2009, and is likely to continue upwards.  This can be expected to increase the inflow of job seekers requiring assistance from the employment service and to make job outcomes harder to secure.   

To reduce the impact on the Australian labour market of the global financial crisis the Government has announced a range of measures to boost job creation and strengthen the economy.  The package includes labour market assistance measures collectively known as the Jobs and Training Compact.  While separate monitoring and evaluation arrangements are being developed for the Plan, this work will be integrated wherever possible with the monitoring and evaluation of Job Services Australia. 
 
The main elements of Job Services Australia are: 
	• Four service streams for work ready job seekers (Stream One) and disadvantaged job seekers (Streams Two to Four) (Figure 1).  The stream a job seeker is allocated to is determined by the job seeker’s assessed level of disadvantage, as measured by the Job Seeker Classification Instrument (JSCI), and, where required, Job Capacity Assessment (JCA).  If the job seeker’s level of disadvantage increases to the extent that they become eligible for a higher stream they can be moved to that stream.   
	• The level of service in each stream will reflect the relative job readiness of job seekers.  Accordingly, service and outcome fees will increase across streams.   

	

4 The most disadvantaged job seekers include those unemployed more than five years, homeless job seekers, people with mental health conditions, Indigenous Australians, job seekers in jobless families and 15 to 24 year-olds who are not working and not in education.  These job seekers are at risk of social exclusion. 
5 Department of Education, Employment and Workplace Relations 2008, The Future of Employment Services in Australia: Fact sheet, DEEWR, Canberra, p. 1. 

	• Job seekers can be fully or partially eligible for stream services.  Job seekers fully eligible for stream services include recipients of Newstart and Youth Allowance (other), including parents and people with disability who have part-time participation requirements,6 or another form of qualifying income support, 15 to 20 year-olds not in receipt of income support and not employed more than 15 hours a week or in full-time education, and CDEP participants.  Other job seekers may register with Centrelink or a Job Services Australia provider as partially eligible for Stream One.  These job seekers are referred to as Stream One (limited) job seekers and are usually those not working or studying full time and not receiving activity-tested income support.  These job seekers are not assessed by the JSCI.  The level of service for these job seekers is limited to the provision of a résumé, access to Australia’s national vacancy database (Australian JobSearch) and advice on the local labour market.  A more comprehensive description of eligibility is included in Appendix B of the 2008 Request for Tender.7 

	

6 Part-time participation requirements involve looking for or undertaking at least 15 hours of work per week and registration with a Job Services Australia provider or undertaking approved study (see http://www.centrelink.gov.au/internet/internet.nsf/filestores/lw054_1007/$file/lw054_1007en.pdf) 
7 Department of Education, Employment and Workplace Relations 2008, Request for Tender for Employment Services 2009–12, DEEWR, Canberra. 

Figure 1:  Job Services Australia, 2009 to 2012 

Notes:  	
1.7 multiplier applies to Employment Pathway fund and service fees in designated remote areas. 	
 	For job seekers who continue in Work Experience for more than 12 months the fee paid will continue at $133 and $67 for each alternate three months (ie, $133 for 13 to 15 months, $67 for 16 to 18 months, $133 for 19 to 21 months, and so on).  	

	• To support closer links between the employment services and the provision training to address skill shortages extra places will be available in the Government’s Productivity Places Program for job seekers eligible to participate in Job Services Australia.  This program is being delivered by States and Territories under a COAG National Partnership Agreement. 
	• A work experience phase, which operates as part of each stream.  Within this phase job seekers aged between 18 and 49 will generally be required to participate in a work experience activity over a 26 week period for every 12 months in the phase.  Job seekers who have completed 12 months of services in Streams One to Four will commence the work experience phase following a stream services review, unless the review suggests a job seeker move to a higher stream or, in the case of Stream Four, further services within this stream.  Job seekers who complete 18 months in Stream Four will automatically move to the work experience phase.  Work experience activities may include participation in programs such as Work for the Dole, Green Corps and Droughtforce, part-time study or paid employment or voluntary work. 
	• An Employment Pathway Fund (EPF) which can be used by providers to help job seekers obtain employment.  Providers will receive an EPF credit for each job seeker.  Credits increase as the job seekers’ levels of disadvantage increase.  The use of credits is not limited to any one job seeker.  Funds can be used flexibly to assist any job seeker or group of job seekers.  Unused funds cannot be retained by providers. 
	• Other elements of Job Services Australia include continuation of the New Enterprise Incentive Scheme, Harvest Labour Services and the National Harvest Labour Information Service: 
	o New Enterprise Incentive Scheme is for job seekers interested in starting and running a small business.  The scheme provides small business training, business advice and mentoring, as well as ongoing income support for up to 52 weeks.  A NEIS panel will decide who is to participate based on the viability of proposed businesses and draft business plans. 
	o Harvest Labour Services for primary producers in areas where the demand for workers cannot be met locally.  Any job seekers legally entitled to work in Australia are eligible for this service. 
	o A National Harvest Labour Information Service to develop and disseminate information about harvest related work opportunities throughout Australia. 
	• An Innovation Fund is to be established to support projects designed to find solutions to address the barriers to employment of the most disadvantaged job seekers.  Funds will be available to members appointed by the department to an Innovation Fund Panel. 
	• Employer Brokers will also be funded to help providers to develop links with local employers that are facing skills and labour shortages.  Brokers will be appointed to a panel and invited to submit proposals for funding activities. 

Participation in Job Services Australia will be supported by a revised compliance framework.  In particular, safeguards will be introduced to ensure job seekers who do not participate in the employment service through no fault of their own are not penalised.  Centrelink will continue to have responsibility for making compliance-related decisions under Social Security legislation and reconnecting job seekers to the employment service.  Centrelink will also be responsible for undertaking Comprehensive Compliance Assessments with job seekers who persistently fail to meet their participation requirements or who are considered to have the potential to do so.  The role of providers will still primarily be to notify Centrelink when a job seeker fails to meet their requirements. 
Evaluation requirement 
Evaluation is part of program management and provides a basis for fine-tuning policies and programs.  The Government requires the development of an evaluation strategy and that Job Services Australia be progressively monitored from implementation.  Specifically, the Government requires that the evaluation consider whether the new service: 
 
	• is provided at the earliest opportunity, consistent with a job seeker’s level of disadvantage while minimising deadweight costs;8 
	• delivers individually-tailored assistance to meet each person’s needs; 
	• delivers assistance and programs that minimise attachment effects;9 
	• contributes to “closing the gap” in employment outcomes and opportunities for Indigenous Australians; and
	• reduces the administrative burden for providers. 

Scope of the Evaluation 
 
The purpose of the evaluation is to assess how Job Services Australia is performing in terms of assisting individuals, particularly those most disadvantaged in the labour market, to obtain skills and secure sustainable employment.  The evaluation will provide information that policy makers and program managers can use when considering changes or improvements.  As a result, the timing of the dissemination of findings will be ongoing so to inform future budget decisions or processes for awarding future business, or otherwise serve to meet the needs of policy makers and program managers.   
 
The evaluation endeavours to answer a set of key questions which are based on the objectives of Job Services Australia and its main elements.  These objectives and related issues, as well as the Government’s evaluation requirements, guided the strategy’s development.  The strategy adopts a flexible approach, recognising that as the evaluation progresses new issues will emerge while some of the issues identified as significant at the outset will become less important over time.  This approach also recognises the current uncertainties on the timing of data. 

	

8 Deadweight refers to the extent to which those assisted would have achieved outcomes in the absence of the assistance. 
9 This is the reduction in job search activity which can result from participation in a labour market program. 

Key differences in approach between the old and new services will be reflected in the evaluation.  For example, the evaluation will place considerable importance on the increased focus on the most disadvantaged and the extra emphasis on training to deliver sustainable jobs when considering measures such as outcome levels, net impact and cost effectiveness.  The impact of the economic downturn will also need to be factored into the evaluation, particularly to the extent that comparisons are drawn between the performance of the new service and that of previous labour market assistance models.  All else being equal, a downturn in the economy would reduce program outcome levels (simply because there are less jobs available in the economy to be filled by job seekers who have been assisted by the service) but will not necessarily lead to reduced impact. 
Evaluation criteria 
Broadly speaking the evaluation will assess performance against three main criteria: participation, effectiveness, and efficiency and cost effectiveness.  The evaluation will also consider how the policy design and implementation of Job Services Australia contributes to performance against these criteria. 
Participation 
Participation in Job Services Australia for job seekers who continue to require assistance emphasises employment-focussed activities, the acquisition of skills relevant to the needs of local employers and job search. 
 
Job seeker participation will be examined in terms of: 
	• Engagement—under Job Services Australia job seekers will participate in a range of activities designed to maintain their engagement with both the labour force and, where appropriate, the employment service.  The evaluation will measure both the degree and form of engagement; 
	• Access and equity—equity of access to assistance, especially by the most disadvantaged job seekers (including those at risk of social exclusion), and the outcomes achieved by these job seekers relative to others; 
	• Quality of service—including responsiveness in assisting job seekers and employers and their satisfaction with services provided.  This has regard to the fact that more disadvantaged job seekers generally require more intensive assistance. 

Effectiveness  
Effectiveness is about results — the extent to which Job Services Australia makes a difference to participants which is consistent with the objectives of the service.  This includes: 
	• Increasing the employment outcomes of those assisted—the primary aim of assistance is to move job seekers into sustainable employment.  Sustainable refers to a minimum period of time (of undefined length) that a job is held.  This issue is discussed further under outcome measures. 
	• Increasing skill levels—the program objective of helping individuals to obtain the skills they need will be reflected both in the match between the skills of job seekers and the skills local employers are attempting to attract and in the contribution of skills acquisition to reducing skill shortages. 
	• Reducing welfare dependency—reductions in welfare dependency can occur by increasing the income of job seekers through the attainment of paid work and increasing exits from income support. 
	• Increasing social outcomes—participants in Stream Four can be assisted to secure a social outcomes, which reflects the progress a job seeker has made towards work readiness.  Currently a comprehensive approach to developing social outcome measures is progressing.  This process is due for completion in time for the 2012 employment services tender and may not be available in time for the evaluation. 
	• Increasing social inclusion— unlike earlier employment service models the new service is explicitly designed to achieve greater social inclusion.  In the context of the new employment service, achieving greater social inclusion is particularly focused at improving the participation of the most disadvantaged job seekers, both in the labour force and in their community, by addressing the multiple barriers often experienced by these job seekers.  Measuring these outcomes calls for an innovative approach which is likely to encompass changes to work readiness among people who remain unemployed as well as greater engagement with local communities.  Social inclusion encompasses social outcomes.  Currently a comprehensive approach to developing social outcome measures is progressing.  This process is due for completion in time for the 2012 employment services tender and may not be available in time for the evaluation.10 

Efficiency and cost effectiveness 
Efficiency can be considered both in terms of the time taken to achieve a result and the cost involved.  Cost-effectiveness combines efficiency and effectiveness criteria.  The evaluation will examine: 
 
	• Reducing the time that job seekers are in assistance. 
	• Reducing the costs of delivering assistance. 
	• Reducing the costs of achieving results.   

	

10 Work on a social inclusion performance measurement framework is underway as part of the broader National Social Inclusion Framework being developed by the Department of The Prime Minister and Cabinet.  The draft performance framework is modelled on the Overcoming Indigenous Disadvantage Strategic Framework (ie, including headline indicators underpinned by strategic change indicators to measure the impacts of government policies and programs and to track longer term progress against key outcomes, as well as highlighting areas for future action).  For more information, see http://www.socialinclusion.gov.au/Pages/default.aspx

Performance monitoring 
Monitoring the operation and outcomes of Job Services Australia is a key element of performance management.  An expert reference group was established to advise the Government on future arrangements.  This advice encompassed key performance indicators, provider performance ratings, social outcomes and service quality and was adopted for the Request for Tender and reflected in the contracts with employment service provides.   
 
Efficiency, effectiveness and quality will be the key performance indicators for the new employment services deed.  Indicators of efficiency (the time taken to achieve results) and effectiveness (the achievement of outcomes) are listed in the Request for Tender.  These indicators will be measured using a regression methodology to produce a rating for each service stream and at the level of each employment service area and Jobs Services Australia site.  The indicator of quality will be measured separately.  It is likely to include measures of job seeker and employer satisfaction.  Further information on the way the performance of employment service providers will be assessed is available in the Request for Tender. 
 
Reporting arrangements for ongoing monitoring of inputs, outputs and outcomes are also being developed.  These reports will provide a significant source of information for the evaluation to assist with conclusions on participation, effectiveness and efficiency and cost-effectiveness. 
‘Best’ practice 
The evaluation will gather information which could potentially be used to identify ‘best’ practice and be disseminated to the industry.  This could cover, for example, the differences in service delivery methods between high and low achieving providers.  As part of arrangements for managing the performance of Job Services Australia the department will foster the dissemination of ‘best’ practice.  If required, service providers are obliged to share ‘best’ practice with the department which could then be used to help underperforming providers. 
Transition to the new service 
Transition arrangements between ESC3 and ESD4 were put in place to facilitate the introduction of Job Services Australia.  Measuring the performance of these arrangements is not in the scope of the evaluation.  Nevertheless, the smoothness of the operation of these arrangements and any subsequent effect they may have on the performance of the new service is an issue which the evaluation will address. 

Specific studies 
 
A number of studies have been designed for the evaluation.  These areas of investigation cover the evaluation issues which are, as noted above, based on the objectives of the new service.  The design of specific studies has regard to the importance of particular issues and the timing and availability of robust and representative data.  Data on participation, for example, will be available to evaluators well before data on outcomes. 
 
The main areas of investigation proposed for evaluation, and the research questions to be addressed in these areas, are described below.  Figure 2 (page 14) gives indicative timings for consideration of these issues.  Because the relative importance of evaluation issues changes over time, the need for some of the studies identified below may diminish over the course of ESD4.  As a result some studies may not be required or may require substantial redesign.  The need for other studies may also eventuate.  
1.  Participation in Job Services Australia 
Job seeker participating in different services will be examined, initially in late 2009 and again a year later, as will the mechanisms (such as the compliance framework) for ensuring entry into the service (for those who need assistance) and ongoing engagement.  Specific issues include: 
	• How quickly are job seekers (disaggregated by demographic characteristics and type of disadvantage) accessing the new service? 
	• Are job seeker participation rates higher in Job Services Australia compared with Job Network during ESC3? 
	• Are job seekers maintaining their engagement with their employment service providers? 
	• Is the administration of compliance measures being adhered to by providers? 
	• To what extent are providers using different engagement and re-engagement strategies to connect job seekers to Job Services Australia and what is the comparative effectiveness of these strategies? 

2.  Job Seeker Assistance 
Entry into different service streams is largely determined by a job seeker’s JSCI score (supplemented where appropriate by a JCA).  For these instruments to be effective they must be capable of both accurately profiling job seekers and of being applied accurately.  Types of assistance and the quality of services will also be addressed within a range of specific issues, including: 
	• Are job seekers being referred to the appropriate streams based on their JSCI (and JCA supplement) assessment? 
	• What types/combination of interventions (including EPF-funded assistance) are job seekers participating in? 
	• Do providers recognise and act upon factors relevant to the individual (eg, caring responsibilities, disability, age and cultural diversity) when developing Employment Pathway Plans and delivering assistance? 
	• Is the JSCI being updated as required?  
	• Are job seekers satisfied with the quality of services from their providers? 
	• Are job seekers satisfied with their input into designing their EPP? 
	• How does target group take-up of specialist providers compare with that of other providers? 

3.  Employer servicing 
The performance of the employment service largely rests with the ability of providers to place job seekers with local employers.  Crucial to this is a sound working relationship between Job Services Australia and employers.  The issues addressed within this area of investigation reflect the factors which contribute to the effectiveness of this relationship, such as: 
	• Are local skills shortages being adequately identified and addressed? 
	• Are the current and future staffing needs of local employers being canvassed effectively? 
	• Are providers working with other local employment agencies (including commercial recruitment organisations) to address the needs of employers and what is the contribution of Employer Brokers to this outcome? 
	• Do employers regard the job seekers referred to them as suitable? 
	• Are providers facilitating the required development activities for job seekers that meet the needs of local employers? 
	• What are the attitudes and responses of employers to the incentives offered by providers (such as wage subsidies) to hire job seekers? 
	• Are providers accessing the training required to address employer’s skill shortages? 
	• Do bonus outcome payments act as sufficient incentives for providers to target local skill shortages? 

4.  Building labour force capacity 
Raising the skill levels of job seekers has the potential to reduce labour market inefficiency and is one means available to government of addressing skill shortages.  Building labour force capacity also increases the employment prospects of those assisted.  This part of the evaluation will examine the following questions: 
	• Is there an appropriate assessment of job seekers’ existing skills? 
	• Is appropriate training for job seekers available locally? 
	• What types of training are job seekers participating in and is the training relevant to the local labour market? 
	• What is the quality of training and skill development delivered to job seekers? 
	• To what extent do employment service providers use the Productivity Places Program to improve job seeker skills? 
	• Are employment outcomes related to the area of training? 
	• To what extent does the training contribute to improving job seeker skills? 
	• What contribution do the individual assistance streams make to help individuals obtain skills? 
	• To what extent are job seekers being referred to education and training services? 
5.  Addressing disadvantage 
Job Services Australia is designed to increase the focus on the most disadvantaged job seekers (including those at risk of social exclusion), in part by the introduction of Stream Four services.  In considering the greater emphasis on these job seekers and especially the introduction of Stream Four, the evaluation will examine: 
	• What is the relative share of assistance11 and expenditure by disadvantage types and levels compared to ESC3 services? 
	• What is the net impact of the assistance for different job seeker groups? 
	• How do the outcomes (and impact) of specialists compare with those of mainstream providers? 
	• How effective have Stream Four services been in helping those who are most disadvantaged to overcome their barriers and build their capacity for work readiness, compared with the former Personal Support Programme (PSP) and Job Placement, Employment and Training (JPET) services (which focussed on pre-employment assistance to overcome non-vocational barriers and then making linkages to employment services such as Job Network).   
	• To what extent have providers made use of concurrent vocational and non-vocational servicing, has this resulted in innovative approaches, and has it been effective in building work readiness and the achievement of outcomes (particularly in comparison with JPET)? 
• To what extent have Stream Four services been delivered in-house, through sub-contracting, via external providers or the local community?  If in-house, how have providers adapted their staffing profile to meet the needs of the most disadvantage job seekers? 
• Has access to and use of the EPF contributed to service delivery and the achievement of outcomes for the most disadvantaged? 
• What types of EPF-funded assistance achieve the best outcomes and impact? 
• Are providers developing particular strategies with local employers for the employment of the most disadvantaged and are these effective? 

11 Relative to the eligible population of each group of job seekers. 
6.  Impact on administrative burden 
Many of the submissions to the review of employment services commented that administrative burden of arrangements under ESC3 adversely affected the capacity of the employment service to deliver assistance.  The new model combines seven former contracts into one and introduces a Charter of Contract Management to “drive efficiencies, cut waste and introduce flexibility [to] allow providers to focus more on jobseekers rather than on administration”.12  In this context the evaluation considers: 
	• To what extent is the administrative burden for providers reduced under the new service compared with ESC3? 
	• What has been the effect on service quality from any reduction in the administrative burden? 
	• Is there scope to include other programs under a single contract model? 

7.  Indigenous servicing 
A significant issue for the evaluation is whether the contribution of Job Services Australia to halving the employment gap between Indigenous and non-Indigenous Australians can be isolated from the contributions of other employment initiatives for Indigenous job seekers.  Bearing this in mind, other issues relevant to assessing the services delivered to Indigenous job seekers include: 
	• How many employment service agencies have hired Indigenous staff members? 
	• Do Indigenous Australians have access to training and skills acquisition opportunities? 
	• Are Job Search Australia agencies developing service level agreements (SLAs) with CDEP and IEP providers to improve servicing for Indigenous job seekers in remote locations? 
	• What is the quality of SLAs and are they delivering outcomes for Indigenous job seekers? 
	• To what extent do the service fee and EPF multipliers and outcome fee bonuses contribute to improved outcomes for remote Indigenous job seekers? 

	

12 Department of Education, Employment and Workplace Relations 2008, The Future of Employment Services in Australia: Fact sheet, DEEWR, Canberra, p. 3. 

8.  Social inclusion under Job Services Australia 
The greater emphasis on social inclusion of Job Services Australia compared with previous employment service models recognises the role of labour market assistance in raising and maintaining participation (including labour force participation) among unemployed job seekers.  Indicators of social inclusion are still being developed however, it is anticipated that the following questions are relevant to this are of investigation: 
	• Does service delivery lead to equitable access to services for disadvantaged groups and invest more intensively in those at risk of or experiencing, social exclusion.  
	• Are interventions tailored to support those experiencing deep and complex social exclusion and are different service providers linking together to address barriers to participation more broadly?  
	• Have social supports and networks that provide information on (and access to) jobs, education, training and opportunities expanded as a result of participating in assistance?  
	• Is the assistance contributing to job seekers being more work ready, particularly in areas and regions that limit life opportunities? 
	• Are activities such as Work for the Dole, volunteering and work experience engaging participants with their local community?  
	• What proportion of job seekers is undertaking paid or unpaid work in social enterprise, as their work experience placement? 

9.  Effectiveness of Job Services Australia  
The evaluation will consider the effectiveness (including net employment impact of the main forms of assistance) of different forms of assistance, such as stream services and work experience.  There is also the potential to measure the effectiveness of different types of work experience placements, including Green Corps and Work for the Dole.  A net impact methodology is proposed as the means of obtaining estimates of effectiveness provided viable comparison groups can be established.13  A comparison group allows a counterfactual to be estimated (ie, the outcomes that would have eventuated for participants had they not participated).  The difference between participant and comparison group outcomes is a measure of program net impact. 

13 The net impact approach involves comparing outcomes of job seekers who have been assisted with those of a group who have not.  The difference in outcomes, all else being equal, gives a measure of net impact.  In the absence of randomly assigning job seekers to a “treatment” or a “control” group, the approach used in Australia to measure net impact relies on selecting a comparison group then using statistical methods to control for the observable differences known to influence the likelihood of an employment outcome between the comparison and participant groups. 

The evaluation will also consider the overall effectiveness of Job Services Australia.  In this case a net impact approach is not feasible because under Australia’s system of universal entitlement a counterfactual against which to compare the performance of the service does not exist.  An alternative approach, which has been used in the past, involves modelling the performance of previous labour market assistance arrangements, using this model to predict the performance of current arrangements then comparing actual performance against predicted.14  This approach assumes performance is a function of local labour market conditions, the demographic characteristics of participants and provider performance.  It gives a relative measure of performance rather than an absolute measure available from a net impact analysis.  The validity of the approach relies on accurately controlling for changing labour market conditions and for differences over time in the composition of job seeker populations.  Measures of effectiveness can be combined with data on costs to estimate the cost effectiveness of services.   

These analyses are not expected to be complete until early 2012.  This timing acknowledges that reliable measures of effectiveness cannot be estimated until a representative sample of job seekers have participated in the service and have had sufficient time to obtain an outcome.  Early measures of effectiveness may be biased, reflecting that a new service takes time to “bed down” and that the job seekers who secure the first wave of outcomes are those easiest to assist. 
 
In estimating effectiveness and cost effectiveness the evaluation will examine: 
	• What proportion of job seekers obtain employment outcomes? 
	• What proportion of Stream One recipients find employment by time in assistance and what proportion remain unemployed after 12 months? 
	• How long do the outcomes last? 
	• Does job quality (ie, the conditions of a job, including remuneration, permanency, hours of work and entitlements) improve over time? 
	• Does assistance contribute to the outcome, and its sustainability? 
	• What is the net impact of the assistance for different job seeker groups? 
	• How do the outcomes (and net impact) of specialists compare with those of mainstream providers? 
	• What types/combination of interventions are most effective? 
	• What types of assistance result in sustainable outcomes? 
	• What is the deadweight of Job Services Australia and its services?  
	• What are the attachment effects of assistance and how do they compare with the attachment effects of services under ESC3? 
	• Efficiency and cost-effectiveness of Job Services Australia. 

14 See Section 9.3 in Department of Education, Employment and Workplace Relations 2007, The APM evaluation: July 2003 to June 2006, DEEWR, Canberra. 

10.  Impact of the economic downturn on the employment service 
Australia’s economic downturn in response to the global financial crisis provides a backdrop to making judgements about performance of the employment service in most of the evaluation’s areas of investigation.  As already noted, for example, job outcomes are likely to fall as a result of the downturn but this will not necessarily mean the effectiveness of the service falls.  This study will examine the impact of the economic downturn on client characteristics and the performance of Job Services Australia, as well as the way the employment service responds to the downturn in terms of the types of services offered.  Isolating downturn effects from other economic and labour market changes will be challenging. 

Figure 2:  Indicative project timelines 

 
Evaluation management 
Implementation 
DEEWR has responsibility for managing and conducting the evaluation.  Development of the strategy has been overseen by the Evaluation and Monitoring Working Group of the Employment Services Steering Committee (ESSC).  Feedback from external agencies (including central agencies and other stakeholder agencies) was also sought during development and has been incorporated into the final strategy. 

Reporting results 
Findings from the evaluation will be disseminated as they become available.  This will include, for example, presentations to policy and program areas and the department’s executive.  Early results, such as those covering the transition period, would be available in the second half of 2009 while the estimated timelines for the evaluation projects is shown in Figure 2.  It is anticipated that the specific studies will result in a series of evaluation papers which, subject to Ministerial agreement, could be made publicly available on the department’s internet site.  To do so represents best practice in official evaluation and demonstrates the Government’s commitment to openness and full accountability in respect of publicly-funded programs.  Release of reports also strengthens the credibility of claims by the department, in particular, and Government, in general, to both serious research and evaluation and evidenced-based policy development.  A large scale report, consolidating all the evaluation findings, is not envisaged. 
Other reviews 
Where appropriate the evaluation will draw on other reviews of labour market interventions and related services.  This may involve using findings and sharing sources of data.  Relevant reviews include the labour market elements of the Government’s response to the global financial crisis.  Other potential reviews which could inform the evaluation include assessments of the transition to the new service, as well as any best practice studies.  As noted earlier, data collected for the evaluation will be an important contributor to the latter. 
 
Resourcing 
 
Of the $3.9 billion allocated for Job Services Australia over three years, $8.3 million (the actual allocation was $10.5 million, which included $1.2 million in 2008–09) has been set aside for monitoring and evaluation, with the bulk of these funds earmarked for a longitudinal survey (Stepping Stones, see below).  These resources supplement the department’s budget for research and evaluation.   
 
The availability of resources in part determines the breadth of much of the analysis which can be conducted in an evaluation.  Even where sufficient financial resources are available evaluation activity may ultimately be constrained by a lack of evaluators who have the necessary analytical and reporting skills.  Research and analysis is currently an area where skills are chronically in short supply. 
 
Performance measures 
 
The evaluation criteria provide a basis for identifying performance measures which in turn dictate the data to collect.  Measuring performance is reliant on the availability of robust and valid data (as noted above in the context of the specific evaluation studies).  Factors which contribute to this include allowing for a settling-in period (it is unrealistic to expect optimal performance from day one), the duration of assistance, the time taken to achieve outcomes and the extent of any refinement to the service post implementation.  The earlier comment on the time taken for data to become available to measure effectiveness applies more generally.  It will be some time before comprehensive data are available which are representative of performance of Job Services Australia as this information relies on a representative cohort of job seekers participating in assistance and being given the opportunity of achieving outcomes.  Where feasible performance data will be disaggregated by, for example, job seeker characteristics (including the groups defined as most disadvantaged or at risk of social exclusion).  The extent of disaggregation possible depends on the numbers participating and the method of data collection. 
 
Consistent with the main criteria for evaluating Job Services Australia the following performance measures will be used. 
Take-up and usage 
Data on referrals, commencements and length of time in assistance provide measures of take-up and usage.  These measures will be available on the number and proportion of participants in the different forms of assistance.  It will be possible to track ‘multiple’ program participation and to make comparisons of take-up and usage with services delivered previously and between voluntary and compulsory participants. 
Service type and quality 
Data on the types of services delivered to job seekers by Job Services Australia and job seeker and employer perceptions of the quality of services will be collected.  These data will be comparable with data for earlier employment service contracts.  Measures of service quality include a job seeker’s perception of their treatment by their employment service provider, confidence in their provider’s ability to find them work and overall satisfaction with services.  Similar information will be available for employers but with less frequency. 
Outcomes 
Outcomes data include: 
	• Employment outcome rates following program participation (ie, the numbers employed as a proportion of the numbers participating).  Education and training outcome rates are also included sometimes as a post-assistance outcome measure since employment service providers can be paid for this type of outcome.  Employment outcomes, combined with education and training outcomes are referred to as positive outcomes.  The data typically refer to outcomes measured around three months after assistance but may also include job seekers whose providers achieve paid outcomes (either 13 or 26 weeks after assistance).  Moreover, longer time frames combined with continuity of outcome will be measured to obtain data on sustainability.  For the net impact analysis the outcomes for job seekers who do not participate in a program will be measured.   
	• Exits from employment assistance.  This refers to job seekers who cease to be registered as requiring assistance. 
	• Off-benefit outcomes.  This refers to the proportion of income support recipients who leave income support.  These “exit rates” rates are measured, for example, each month after commencement in assistance. 
	• Income support reliance, expressed as the proportion of a job seeker’s total income (ie, earned income plus income support payments) which is derived from income support payments, where possible, measured monthly during the 12 months before and after assistance. 
	• Social outcomes.  As noted earlier, participants in Stream Four can be assisted to achieve social outcomes which are designed to reflect an improvement in their work readiness.  The precise nature of these outcomes and how they are to be measured is still being developed. 

Such measures can be used, for example, to estimate the net impact of assistance by comparing the outcomes which would have occurred in the absence of assistance with those of job seekers who have participated in the employment services. 
Efficiency measures 
Measures of efficiency include length of time in assistance, the cost of assistance and the cost of an outcome.  The cost of an employment outcome, for example, can be calculated by dividing the total cost of assistance by the number of job seekers who obtained jobs after participation.  Changes overtime in these measures will need to be interpreted within the context of any changes to the economy and participant characteristics.  Cost per net employment impact of elements of the new service, calculated by dividing the total cost of assistance by the number of job seekers who obtained a job as a result of participation, will also be used. 
 
In general terms, it will be possible to report most performance measures by various demographic and labour force characteristics, including age, gender, education level, duration of unemployment, income support receipt, Indigenous status, disability, culturally and linguistically diverse status, sole parenthood and location.  Other disaggregations may also be used as required. 
 
Data sources 
 
The evaluation will be supported by a range of data sources and collections which will provide the information required to address the research questions.  The main sources are briefly described below. 
Employment Assistance Survey 
The Employment Assistance Survey is a longitudinal survey of job seekers which commenced in January 2008 to collect information on the type, quality and intensity of services delivered by the employment service, as well as data on attitudes to employment and employment assistance and sustainability of outcomes.  The survey sample of about 3,000 was stratified for parents, people with disability, the mature aged, the very long-term unemployed and, for comparative purposes, mainstream job seekers.  The sample was also stratified by type of employment assistance.  Participants in the survey are being interviewed at three to four monthly intervals.  The survey is a key source of baseline data for the evaluation of Job Services Australia. 

Stepping Stones longitudinal survey of income support and employment assistance  
Stepping Stones is a longitudinal survey of job seekers designed in part to collect data not available from other sources for the evaluation of the new service.  The survey will collect information on the experiences and outcomes of job seekers, including their pathways after they leave assistance and go off-benefit.  It will be conducted over four years beginning in the first half of 2009 in order to gather baseline data before the implementation of Job Services Australia.  The survey will involve eight waves of data collection six months apart with a new cohort of around 11,000 introduced every year to allow detailed analysis of job seeker groups of interest as well as the wider working age income support population. 
Post-program Monitoring Survey 
Since the late 1980s the department has used the Post-program Monitoring Survey to measure the labour force and education status of labour market program participants around three months after assistance.  The survey also collects a range of information on whether job seekers are undertaking a period of study or training, the type of educational institution they are attending, tenure of employment, average weekly income, data with satisfaction with services as well as any qualitative comments that job seekers may include in their survey response.  The survey is a valuable source of information on program outcomes and will continue during ESD4.  A mail-out with telephone follow-up is used to collect data.  In the context of job seeker participation in program elements, the approach to administering the survey under Job Services Australia is still being developed. 
Service Quality Monitoring Program 
The Service Quality Monitoring Program collects information on the quality and responsiveness of services delivered by the employment service and by Centrelink.  The program includes regular surveys of job seekers, employment service providers and employers.  Both computer assisted telephone interviewing and face-to face interviewing are usually used.  Key elements of the program during the next four years include: 
	• Job Seeker Omnibus Survey—this survey collects information on job seekers’ experiences with the employment service, including their perceptions of the quality of the service delivered.  An annual survey is planned between 2009 and 2013, conducted in March each year. 
	• Employer Survey—interviews will be held with employers to examine their relationships with Job Services Australia.  Topics include recruitment methods, awareness, understanding and usage of the employment service and perceptions of the quality of services.  Typically surveys are conducted at two-yearly intervals, with the next one planned for August 2010.  This will be preceded by qualitative research with employers in April 2010 to explore some of the issues for inclusion in the main survey. 
	• Survey of Employment Service Providers—this survey has been conducted annually since 1999 and will continue with this frequency during ESC4.  It seeks the views of providers on the quality of services from Centrelink and DEEWR and is designed to monitor the department’s performance in relation to employment service deeds and Centrelink’s performance against service objectives.  Survey information will continue to be collected at site level. 

Administrative data 
As part of its purchasing, regulating and managerial role DEEWR collects information on the activity and performance of employment service providers, including Job Services Australia.  This includes information on fees and EPF expenditure, client numbers and characteristics (eg, age, gender, unemployment duration, educational attainment, and equity group) as well as data on referrals, commencements and outcomes.