Measuring the impact of support systems:  a change of perspective and... indicators?

How can we measure the impact of support systems? What indicators can capture the complex range of the effects?

In a context of great debate and controversy over the usefulness (useful for whom? Useful in what way?) of the money spent (the famous proper allocation of public money), support systems are not immune to such tensions over the measurement of their effects.

A colleague, the director of a Mission Locale youth centre, who has long been working on the support of young people in her area and the development of services that contribute to a fairer and more equitable society, confided to me the other day that she was perplexed by the amount of time and energy she and her team spent filling in the many management charts relating to the structure’s activity. And she went on to point out the following: “It’s as if the justification of the activity has taken precedence over the relevance of the projects; that the immediate, visible effects are more important than the deferred effects; that performance is more important than prevention.” No doubt she had not had the time to read in full the remarkable work of Alain Supiot, Professor Emeritus of the Collège de France on “Governance by Numbers”. In an interview on occupational health, he recently said: “Companies have locked themselves into a representation of their activity by figures, with tables and ratios.” And we would add that public structures or those involved in public tenders or services to the public are no exception. This colleague is simply trying to question the meaning of the activity carried out in her organisation, serving the audience it supports. She is also careful to ensure that indicators do not take precedence over the ability to accommodate all audiences. She knows from experience that a map is not representative of a territory, and that in the long run, we risk caring more about the indicators, that are “scrutinised and analysed”, than about the audience or the professionals involved. Roland Gori's text, La Fabrique des Imposteurs, comes back to mind in its analytical accuracy. But more broadly, this professional is looking at what is guiding public action at this time of great upheaval. And she concludes with the lucidity of professionals who have seen all the facets of our professions: “I think that we will be assessed on our ability to encourage our audience to turn to the high-pressure sectors. Full employment is the new mantra. This means that the indicators are changing too.” Does she mean, without explicitly saying so, that support is being pushed towards a new form of social influence?

These reflections from the field (being in the field does not mean that one is not capable of distancing oneself, of theorising and of looking at things critically!) present methodological, ethical and even political questions which encourage a closer look at the foundations in terms of the postulates of public action in the wider field of training and employment and the ways in which its impact is assessed.

This brings up two questions that we will modestly try to clarify and then illustrate: what are the intentions behind developing a support system (in this case, by the public authorities)? Can we then infer processes and evaluation indicators that truly account for the many impacts generated?

A conception of public action in the background: individual responsibility and employment performance

A few days after this discussion, the CEREQ published a very enlightening Working Paper on “The genealogy of the injunction to individual action in employment and training”. The introduction sets out the key issues: “The aim of this working paper is to show how institutions and organisations producing representations such as the OECD and some think tanks and expert circles disseminate the call for responsibility of working people in their relationships to work, employment and training.”

This rhetoric of individual responsibility and of the individual as “entrepreneur of oneself and actor of one's employability” comes up everywhere in the texts as if there were no longer any debate on the subject. We will not attempt to summarise this very dense and enlightening text. We will, however, reaffirm a simple principle. The schemes created by the institutions (state, regions, etc.) are based on a conception of the individual and their relationship to work and therefore define what can help them to become “autonomous”, “self-reliant”, and “having free will”. They deduce how to assess the effectiveness of the funded schemes (services) in relation to these intentions. The methodological framework is therefore quite clear: initially, an individual whose employability and motivation needs to be developed in the light of market expectations (intention); and at the end, access to a job, the nature of which (type of contract, duration, etc.) can be specified, which is the performance indicator. Clearly, the current recruitment tensions call into question this simplified vision. To express reservations about this seemingly common-sense linkage requires looking at both sides of the debate simultaneously: i.e. both the underlying conceptions and the indicators.

Looking at impacts in terms of prevention, facilitation and empowerment

The difficulty is that this apparently simple assessment process renders invisible an essential part of the contributions of support, firstly by focusing only on results in terms of changes in situations, and secondly by not taking into account preventive and developmental aspects (ensuring that the conditions for access to a mobilising working life are facilitated for each person). Broadening the approach could free us from invasive semantics (the famous “removing barriers”) to focus on the mobilisation of the person's resources and their involvement in a facilitating environment.

These questions need to be considered in conjunction with the many studies carried out over the last ten years on the notion of impact. France Stratégie published a report on the subject: “Impact(s), responsibility and global performance”, which places the notion of impact in its historical context (companies in the Social and Solidarity-based Economy have worked on this heavily to demonstrate their contributions to society) and in its current situation in companies. It reads: "Looking at the impact of the company in terms of the consequences of its activity makes it possible to think more deeply about the responsibility of the company, by defining the way in which it contributes to sustainable development and how it takes into consideration the dialogue with its stakeholders, based on concrete data objectified with the help of impact measurement methodologies.”

Sustainable development? The public as a stakeholder? What about support systems? When we try to measure the impact of the support work, we soon realise that there is a danger of focusing only on the effects on the person. However, a measure of the impact of a system, even if it is aimed at the individual, cannot isolate the person from the many interactions that make up their life. Nor can we ignore the things that are valuable to them. If we want to reconcile collective impact and individual mobilisation, we can ask the question as Jean Guichard puts it. “How can we help people create working lives that contribute to sustainable, equitable and human-scale development?” He continues: “The concept of “capability” developed by Amartya Sen seems like an ideal candidate to replace the notion of skill(s).” What exactly does the capability approach refer to? Capability can be understood as “The opportunity to achieve the combinations of human functioning valued by the person: what they can actually do or be.” Without going into the details of the model, we can simply agree that it is about paying attention to the conditions for each person to have access to effective freedom with regard to what is of value to them. And this cannot be reduced to simple quantitative data and a ratio (regardless of the interest of having these elements) because in the field of support, this introduces an essential subjective dimension.

Creating multi-purpose observation models

In concrete terms, we proposed and experimented with multi-criteria assessment processes aimed at prevention, and not reduced to the simple collection of quantitative data. We created grids of indicators which aim to take into account the individual in a wider environment (them and the world); which are not content with simply looking at changes in their situation (in employment or not); which are interested in the conditions for exercising their effective freedom (going towards what makes sense for them); and which allow them to contribute to the observation process through ongoing reflection. This approach implies moving away from a logic of strict performance evaluation based on a few indicators determined for all once and for all (exit rate, access to employment) and to apprehend both preventive and facilitating aspects, i.e. to be interested in reopening possibilities (to use the philosopher François Julien's concept) and to develop the conditions facilitating the mobilisation of the individual's empowerment so that they feel able to move forward towards what is important for them.   

The table below provides a simplified version for ease of reading:

These five criteria have the advantage of not being solely centred on the person but take into account their possibilities of interaction with the environment. They make it possible to simultaneously address the question of the point of view (what I think of myself and what I think I am capable of doing) and the question of action (taking part in a fab-lab, for example, learning to weld, etc.); they make it possible to focus on favourable environments and pedagogical methods for facilitating these developments; they can be discussed with the people being supported themselves during the process; they also serve as a means of shared management of action and allow for a totally individualised assessment, but with common criteria.

In short, this criterion-based grid is at least as much a pedagogical engineering structure (what situations and contexts allow these criteria to be developed?) as it is a model for assessment.

Once again, with reference to the working paper cited in the introduction, it is important to remain vigilant, and avoid placing all the responsibility on the individual (their employability, their motivation, their courage) but to facilitate their interactions with the environment, the enriched use of the external resources proposed, but also the possibility of finding in them resources that had never had the opportunity to be mobilised.

We are constantly adapting this model for observation and steering in the various projects we work on with the teams involved in support. We will give a concrete illustration of this in a forthcoming blog article on “Accompaniment: taking the side roads” where we will present the original initiatives developed by Insercall, a programme for integration through economic activity with two areas of activity: teleconsulting and communication (print, web, video).

Observing support systems by making visible effects that are not immediately perceptible is also a matter of social justice. These criterion-referenced approaches do not replace the inevitable numerical indicators, but they do show the multiple and little visible impacts that a truly experiential and personalised support system can generate.

Because experience also teaches us that if you don't express what you do and allow, there will always be someone to tell you what you don't do and show you what you lack.

https://www.cereq.fr/sites/default/files/2023-03/WP18.pdf

https://www.strategie.gouv.fr/publications/impacts-responsabilite-performance-globale

André Chauvet