Financial resources for adult learning and education tend to be limited across EU Member States. This makes it all the more important to ensure such resources are used effectively and efficiently. But what tells us if money is well invested? Evaluation is increasingly important in public decision-making and can contribute to answering this question, but for an evaluation to be effective, it has to be carefully designed. Important decisions have to be made when planning an evaluation in the field of (mostly formal) adult education. The key aspects in designing an evaluation are below:
- Evaluations can be formative (improvement-oriented) or summative (judgement-oriented). To decide on the main goal of the evaluation is important because some techniques (see below) may need years to deliver results, others (normally more qualitative) can provide insights earlier on.
- It must be decided what exactly shall be evaluated. The following table (Gutknecht-Gmeiner, 2009) gives an overview on subjects and related topics:
|Micro-level: A single course, a training measure||Content, didactics, teachers, framework conditions, usefulness, learned skills|
|Meso-level: An educational institution||Organisation, development of teachers’ skills, curriculum development, infrastructure|
|Macro-level: A policy intervention, a programme or the overall supply of adult education in a region/country||Changes in supply of adult education, satisfaction of participants, increases in labour market success of participants|
3. Concerning the learners or programme participants it must be decided which kind of questions shall be answered: Kirkpatrick has the following four levels that are applicable to all subjects mentioned before:
- Reaction: the (immediate) satisfaction with the intervention (with the course, the institution, the programme),
- Learning: the increase in skills and competencies,
- Behaviour: the change in behaviour among participants (use of acquired skills in the workplace
- Results arising from the change in behaviour, e.g. improved job satisfaction, increased labour market integration etc.,
4. As for methods, it is widely agreed that qualitative and quantitative methods should (in most cases) be used jointly. Especially in large-scale evaluations of policy interventions (macro-level), rigorous quantitative evaluation techniques have their place, particularly if an evaluation focuses on behaviour and results. These dimensions are also often of major interest to policy-makers.
By means of an example I will demonstrate the main challenges in a summative evaluation at the macro-level:
More generally, the example shows that one needs time (and money) to perform rigorous summative evaluations, especially on the macro-level. Simply asking participants once is often not enough. However, this should not be an obstacle when large amounts of scarce public funds are invested in adult education.
Tim Grebe is an evaluation expert and works as a Senior Researcher at InterVal GmbH, Berlin. He has participated in numerous programme evaluations in the field of education and labour market policy.