Investeste in oameni! FONDUL SOCIAL EUROPEAN

POS DRU 2007-2013 - Axa prioritară 1: „Educaţia şi formarea profesională în sprijinul creşterii economice şi dezvoltării societăţii bazate pe cunoaştere” Domeniul major de intervenţie: 1.5 „Programe doctorale şi post-doctorale în sprijinul cercetării” Numărul de identificare al contractului: POSDRU/159/1.5/S/134650 Titlul proiectului: “Burse doctorale si postdoctorale pentru tineri cercetatori in domeniile Stiinte Politice, Stiinte Administrative, Stiintele Comunicarii si Sociologie”

National School of Political Science and Public Administration

Doctoral thesis

“Applying contingent evaluation to lifelong learning:

policies of the European Union”

- Summary -

PhD supervisor:

Prof. Univ. Dr. Cezar Bîrzea

Candidate:

Clain Alexandru Nicolae

2016

2

Taking into account the increased use of public policy and programme evaluation as a means

for ensuring the responsible use of European funds by the European Union’s member states, as well

as the increasing involvement of the European Union in the field of educational policies which have

a direct impact on the member states and their citizens, the current research brings into focus the idea

of developing an alternative evaluation approach, which can be used for evaluating the European

Union’s policies in the educational field in particular, as well as in other policy domains, in general.

Thus, the general objective of this research consisted in identifying the opportunities and

benefits which the contingent evaluation approach can bring in the process of evaluating European

policies and programmes in the field of lifelong learning. The specific objectives which were taken

into consideration during the research are the following:

- Identifying the unintentional effects and dysfunctions of evaluation in the context of new

public management;

- Analysing the historical evolution of the lifelong learning concept in the European

- Analysing the European Union’s framework of strategies, policies and programmes in the

field of lifelong learning;

- Designing a model for contingent evaluation which can be used for evaluating lifelong

learning policies;

- Designing an instrument which can be used for measuring lifelong learning;

The research hypotheses which stood at the foundation of the current research and which were

tested throughout the paper are the following:

- When the complexity of policies and programmes which are going to be evaluated

increases, the ability of traditional evaluation models to offer explanations regarding causal

relationships decreases;

- When the flexibility of the model for evaluating public policies and programmes increases,

the evaluator will be able to use a wider range of methods and instruments for measuring

the results of the policy / programme;

The research was structured around the following research questions:

- Which are the unintended effects and dysfunctions of evaluating public policies and

programmes in the context of new public management?

- Which type / model of evaluation has the greatest potential of decreasing the appearance

of these dysfunctions when evaluating complex policies and programmes in the

educational field?

3

- What is the degree of complexity of the European Union’s policy framework in the field

of lifelong learning?

- How can we measure the short term and long term results of complex policies in the

educational field at the level of the European Union?

The expected results of this current research include obtaining a deeper understanding of the

most important characteristics of the European Union’s policies and programmes in the field of

lifelong learning, identifying the evaluation requirements which appear when evaluating these types

of policies and programmes, as well as the development of an instrument which can be used in order

to measure the different dimensions of lifelong learning.

The added value of this research consists on the one hand in the elaboration of a contingent

evaluation model, which can be used in order to evaluate both complex policies in the educational

field in general, and in the field of lifelong learning in particular, as well as in other policy domains

at the level of the European Union. The second contribution, and at the same time an element of

novelty, is represented by the elaboration of the Index for Lifelong Learning, which can be used as an

instrument for measuring the short and long term results of the European Union’s policies and

programmes in the field of lifelong learning.

The theoretical foundation of the research resides in the major changes which the new public

management paradigm brought on beginning with the 1980’s at the level of public administrations in

the European space. Some of these changes include the orientation towards performance based

management, which implies the constant measurement and evaluation of the results and activities of

public institutions, especially through the use of performance indicators. This shift in paradigm

contributed to the appearance of what Sanderson describes as the “evaluative state” 1, in which

performance is analysed through a variety of instruments, while Dahler-Larsen talks about the

“evaluation wave” which has swept through the majority of developed countries.2 Van der Meer also

points out that the desire to ensure the public accountability of decentralized agencies has led to the

necessity of developing mechanisms for control, monitoring and evaluation, in order to determine the

performance and results of different agencies.3 Still, as it will be pointed out later on, focusing

exclusively on the use of performance indicators as a means for determining the merit and worth of a

1 Ian Sanderson, „Performance management, evaluation and learning in ‘Modern’ local government”, Public

Administration, Volume 79, Issue 2, 2001, p. 298 2 Peter Dahler-Larsen, „Evaluation and public management”, în Ewan Ferlie, Laurence E. Lynn, și Christopher Pollitt,

The Oxford Handbook of Public Management, Oxford University Press, 2007, p. 616 3 Frans-Bauke van der Meer, “New Public Management and Evaluation”, pp. 167 - 168, în Christopher Pollitt, Sandra van

Thiel şi Vincent Homburg (Ed.), NPM in Europe: adaptation and alternatives, Basingstoke: Palgrave Macmillan, 2007

4

public programme or policy can lead to the appearance of dysfunctions and unintended effects of

evaluation, which leads to the necessity of identifying a new evaluation approach, which can

overcome these obstacles. More specifically, the approach which will be studies during this research

is represented by the contingent view in evaluation, which was first developed by Huey-Tsyh Chen.4

With regards to methodology, the primary method which was used throughout the research is

represented by document analysis, both of primary sources, such as different normative acts, policy

documents and strategies at the European level, as well as of secondary sources such as works from

specialized literature or reports from different international organizations. Regarding the thematic area

of the documentary sources which were used, it includes elements such as new public management,

theory and practice of evaluation in the context of new public management, approaches and models

for evaluating public policies and programmes, the evolution of the concept of lifelong learning,

lifelong learning in the European Union, and the development of instruments for measuring the results

of policies in the field of lifelong learning.

Secondly, the research which was conducted in chapter IV of the thesis entailed the use of a

wide range of methods for analysing statistical data. The main sources which were used for collecting

statistical data regarding lifelong learning in the European Union are Eurostat and the 2007 and 2013

Eurobarometers. The process of developing and validating the Index for Lifelong Learning required

the use of statistical methods and procedures such as correlation analysis, frequency analysis, linear

regression, normalization of data, weighing and aggregating statistical data.

The thesis is made up of four main chapters, the first of which discussed aspects connected to

new public management and the way in which the evaluation of public policies and programmes takes

place within this paradigm. Thus, the chapter presents the fundamental concepts which define the new

public management paradigm, as well as aspects regarding the theory and practice of evaluation.

Another issue which is discussed is represented by the unintended effects and dysfunctions of the

process of evaluating public policies and programmes. The first chapter is thus meant to answer the

first two research questions: which are the unintended effects and dysfunctions of evaluating

public policies and programmes in the context of new public management and which type /

model of evaluation has the greatest potential of decreasing the appearance of these dysfunctions

when evaluating complex policies and programmes in the educational field?

4 Huey-Tsyh Chen, ”The contingence view and The Program Theory Perspective”, în Practical program evaluation.

Assessing and improving planning, implementation and effectiveness, Sage Publications, 2005

5

The analysis which was conducted in chapter I points out the fact that the use of traditional

evaluation models and approaches when placed in the paradigm of new public management can lead

to the appearance of dysfunctions and unintended effects. In the context of an excessive focus on

aspects such as increasing efficiency and efficacy, parsimony in using public resources and increasing

public accountability, evaluations which are conducted mainly focus on determining the results of the

public policy or programme, without looking at other aspects which might prove to be significant.

This in turn leads to specific problems, such as the weak analysis of causal relationships between the

interventions which are being evaluated and the effects which have been obtained after the

implementation period, the lack of a real and direct connection between performance indicators which

are defined in order to evaluate a certain intervention and the objectives which are established in the

respective intervention, as well as the low adaptability of evaluation models to changes which can

appear during the implementation of a public intervention, more specifically of contextual conditions

such as institutional landscapes, social structures or cultural values. Thus, chapter I also confirms the

first research hypothesis, namely when the complexity of policies and programmes which are

going to be evaluated increases, the ability of traditional evaluation models to offer explanations

regarding causal relationships decreases.

The last part of the chapter discusses and alternative evaluation model, namely contingent

evaluation. In the case of policies and programmes in the field of lifelong learning, using this

contingent approach could lead to the generation of more relevant and trustworthy evaluation results.

On the one hand, even from the initial stages, such an approach would allow the evaluator to identify

the specificities of lifelong learning in all its aspects, the characteristics of the policy or programme

which is going to be evaluated, as well as the socio-economic context in which the respective

intervention was implemented. On the other hand, in increased flexibility of this approach would allow

for a rapid adaptation to a wide range of evaluation contexts and situations, evaluation objectives and

objectives of evaluation beneficiaries, since the methods and instruments which are used can be

adapted for each specific situation.

The second chapter of the thesis focuses on discussing the concept of lifelong learning in the

European Union. With regard to methodology, methods and techniques such as the analysis of

documents from specialized literature were used in order to conduct this analysis. The chapter includes

a discussion on the different views of established authors in the field such as Aspin, Hager, Faure,

Kallen, Bagnall, Field, Jarvis, Sharples, McGivney. Furthermore, official documents belonging to

international institutions and organizations which were involved in the process of developing the

6

concept of lifelong learning were analysed. These included documents from organizations such as the

Organization for Economic Cooperation and Development (OECD), the United Nations Educational,

Scientific and Cultural Organization (UNESCO), the European Commission and the Group of Eight

(G8).

After analysing works from specialty literature, as well as different official documents

belonging to OECD, UNESCO, G8 and the European Commission, we can see that the process of

shaping the concept of lifelong learning consisted in two main stages. The first stage, which took place

between the beginning of the 1970’s and the end of the 1980’s is characterized by the existence of a

wide range of concurrent concepts such as permanent education, recurrent education or lifelong

education, which were in fact highly similar. These concepts placed great emphasis on the importance

of formal and non-formal education, since it was considered that only these types can be seen as

education in the true meaning of the word, while the importance of informal learning was not

recognized at that moment. On the other hand, in the case of the second stage, which began in the

early 1990’s, we can notice that both established authors in the field, as well as the main international

organizations focus on a common concept – lifelong learning, a concept which brings into attention

the centrality of the learner. Thus, it is believed that educational systems and educational offers should

be capable to respond to the learning needs of members of society at different stages in their lives.

Furthermore, this new concept distances itself from its precursors by recognizing the importance of

informal learning, which represents a significant proportion of the learning which occurs throughout

an individual’s life. Thus, this chapter brings a partial answer to the third research question - what is

the degree of complexity of the European Union’s policy framework in the field of lifelong

learning?

The main objective of the third chapter was to identify and briefly analyse the main strategies,

policies and instruments of the European Union in the field of lifelong learning, while the specific

objective was to understand the way in which the European Union encourages participation in lifelong

learning for different target groups, such as pre-school children, school children, students from

vocational education, students in higher learning and adults. With regard to methodology, the main

method which was used is represented by content analysis, which was applied to official documents

from the European Commission, the European Parliament and the Council of the European Union, as

well as to different programmes, policies and instruments for lifelong learning.

Chapter III thus enriches the answer to the third research question by analysing the European

Union’s framework of strategies, policies and instruments in the field of lifelong learning. The

7

research which was conducted in this chapter highlights the existence of a complex policy framework,

which shows that the European Union has significant stakes in stimulating the participation of citizens

from EU member states in lifelong learning activities. Thus, initiatives and strategic documents of the

European Union in this field such as the Strategic Framework for European Cooperation in the field

of education and vocational training (ET2020), the European Plan for Adult Learning, the framework

for Key Competences for Lifelong Learning, the European Qualifications Framework for lifelong

learning and the Lifelong Learning Programme 2007 – 2013 create a complex framework of

interconnected policies, in which interventions are designed based on the principle of

complementarity, which leads to obtaining a greater potential for transforming lifelong learning into

reality at the level of the European Union.

Out of all these initiatives special attention was given to the Lifelong Learning Programme

2007 – 2013, which represents one of the most important initiatives of the European Union in the field

of lifelong learning. The foundations of this programme were set through the Decision no.

1720/2006/CE of the European Parliament and the Council from the 15th of November 2006 for

establishing an action programme in the field of continuous learning. The general objective of the

programme is to transform the European Union into a knowledge-based society with the help of

lifelong learning, a society with a sustainable economic development, more and better jobs, higher

social cohesion, while also ensuring the protection of the environment. The programme also aims to

encourage exchanges, cooperation and mobility between the education and training systems from EU

member states5. The programme has a systemic approach, by including beneficiaries from all levels

which are relevant to lifelong learning, starting from active learners, which represent the main focus

of the programme, and continuing up to all the actors which make up the organizational framework

of lifelong learning.

However, as it will be shown throughout the case study, the complexity of the policy

framework raises a number of challenges and difficulties throughout the evaluation process, so that it

becomes necessary to use an evaluation model which is different from classical approaches.

The fourth chapter of the thesis consists in applying a contingent evaluation model to lifelong

learning, as well as in developing and validating the Index for Lifelong Learning, which is one of the

main contributions of this research. The case study which was conducted in chapter IV was meant to

5 Council of the European Union, „Decision no. 1720/2006/CE of the European Parliament and of the Council of 15

November 2006 establishing an action programme in the field of lifelong learning”, Official Journal of the European

Union, 24.11.2006, Title I, Chapter I, Article 1 (2)

8

give answers to the last research question, namely how can we measure the short term and long term

results of complex policies in the educational field at the level of the European Union? Thus,

Thus, a series of evaluation models were discussed in the first part of the case study, such as

outcome evaluation, programme theory evaluation, the CIPP evaluation model, and empowerment

evaluation in order to identify those specific elements which can be used in order to give shape to a

contingent evaluation model. Later on a model for contingent evaluation was developed, which could

then be used in order to evaluate the Lifelong Learning Programme 2007 – 2013. This programme

was chosen due to the fact that it is the most representative, and at the same time the most complex

programme of the European Union in the field of lifelong learning.

The first component of the contingent evaluation model was represented by context evaluation.

With regard to the main research methods and techniques which were used at this stage we can note

the analysis of strategic and policy documents of the European Union, as well as of secondary sources

from specialty literature, which can bring valuable information based on the perspectives of specialists

in the field of lifelong learning. What is more, relevant statistical data was analysed, such as

participation rates at different levels of education, participation in extra-curricular activities during

the time period which was analysed.

The context evaluation process firstly highlighted the fact that the Lifelong Learning

Programme 2007 – 2013 was created in a context of relative economic and social instability, being

meant to provide solutions for the negative effects of the social and economic crisis by stimulating the

participation of EU citizens in lifelong learning. Among the most notable negative effects of the

economic crisis on the educational systems from EU member states we can note the decrease of budget

allocations for education, as well as a decrease in the number of teaching staff which can lead, both

in the short and long term, to a decrease in the participation of EU citizens in lifelong learning.

Furthermore, the context evaluation led to the identification of a series of factors which can have a

negative impact on the programme’s implementation process and implicitly on its short and long term

results, such as reduced administrative capacity, language barriers, economic and structural barriers

(lack of financial resources for participating in educational activities, long distances between the

residence of those who want to learn and educational centres, the lack of facilities for disabled

individuals in educational institutions), mental and cultural barriers (precarious educational culture

within the family, low educational aspirations).

The second component of the contingent evaluation model consists in inputs evaluation. With

regard to methodology, the analysis of strategic documents from the European Union was the main

9

method which was employed, also with a focus on secondary sources from specialty literature.

Following the logic of the contingent approach, in the case of evaluating lifelong learning at the level

of the European Union, inputs evaluation was used as an instrument for identifying and quantifying

the policies and programmes which are currently being implemented by the European Union in the

field of lifelong learning, the main objective being to understand the way in which all these

interventions interact with one another.

The research which was conducted during the inputs evaluation stage led to the identification

of two main issues. On the on hand, the analysis of the European Union’s strategic documents pointed

out the fact that the Lifelong Learning Programme 2007 – 2013 is coherent with the EU’s main

policies and strategies from the 2007 – 2013 period, being meant to bring a significant contribution to

reaching the EU’s different strategic objectives in the field of lifelong learning, and in the field of

increasing economic competitiveness. On the other hand, the inputs evaluation highlighted the

existence of overlaps between the objectives and actions which are proposed in the Lifelong Learning

Programme and those included in other initiatives of the European Union which were implemented in

the same time frame, which can lead to the appearance of a lack of coherence and coordination

between the different initiatives, thus diminishing the potential impact of the main actions and

measures. What is more, the existence of similar objectives in different programmes hampers the

evaluation process at the end of the implementation period, making it difficult to evaluate and attribute

results to a certain programme. Thus, it is recommended to increase the target area of a future

programme in the field of lifelong learning, so that it will also include those initiatives which overlap

with already existing objectives in order to ensure higher coherence, as well as a better attribution of

short and long term results.

The last component of the contingent evaluation model is represented by results evaluation.

Due to the fact that the object of evaluation is highly complex, being influence by multiple initiatives

and programmes of the European Union in the field of lifelong learning, the evaluation of results

cannot focus exclusively on the results of a single programme, such as the Lifelong Learning

Programme 2007 – 2013. Thus, the results evaluation process actually entailed a meta-evaluation,

which aimed to measure the combined impact of existing initiatives, thus offering a clearer and more

coherent image on the results which have been obtained.

Thus, the main objective of the results evaluation was to offer an overall image on the impact

which the initiatives implemented by the European Union during the 2007 – 2013 period have had on

the participation of EU citizens in lifelong learning, because only analysing the results which were

10

obtained after the implementation of the Lifelong Learning Programme 2007 – 2013, without taking

into account the existence and influence of other interventions would not have produced relevant

results. The first aspect which was taken into consideration refers to the participation rate of adults

aged 25 to 64 in lifelong learning, which is one of the European Union’s reference indicators in the

field. After analysing the available statistical data we can see that the evolution of this indicator in the

2007 – 2013 period is a positive one, which can probably be attributed to a significant extent to the

use of European funds allocated through the European Union’s programmes and policies in the field

of lifelong learning. Another aspect which needs to be taken into account is that this positive evolution,

which is significant in some cases, has occurred in a context of economic and social crisis, which can

be interpreted as a result of the attempts of EU members states to surpass the negative effects of the

economic crisis by offering new opportunities for the continuous education and training of adults, as

a means for decreasing unemployment rates during the times of crisis.

The table below presents the evolution of the participation rate of adults aged 25 to 64 in

lifelong learning during the 2007 – 2013 period.

Member state 2007 2013

Evolution

(percentage)

Austria 12.8 14.0 +9.4%

Belgium 7.2 6.7 -6.9%

Bulgaria 1.3 1.7 +30.8%

Czech Republic 5.7 9.7 +70.2%

Cyprus 8.4 6.9 -17.9%

Croatia 2.7 2.9 +7.4%

Denmark 29.0 31.4 +8.3%

Estonia 7.0 12.6 +80.0%

Finland 23.4 24.9 +6.4%

France 6.1 17.7 +190.2%

Germany 7.8 7.8 0.0%

Greece 2.2 3.0 +36.4%

Ireland 7.6 7.3 -3.9%

Italy 6.2 6.2 0.0%

Latvia 6.9 6.5 -5.8%

Lithuania 5.2 5.7 +9.6%

Luxembourg 7.0 14.4 +105.7%

Malta 5.9 7.6 +28.8%

Netherlands 16.6 17.4 +4.8%

Poland 5.1 4.3 -15.7%

Portugal 3.9 9.3 +138.5%

United Kingdom 20.0 16.1 -19.5%

11

Romania 1.3 1.8 +38.5%

Slovakia 3.9 2.9 -25.6%

Slovenia 14.8 12.4 -16.2%

Spain 10.6 11.1 +4.7%

Sweden 18.6 28.1 +51.1%

Hungary 3.6 3.0 -16.7%

UE-28 8.96 10.48 +16.96%

Source: Eurostat6

The results evaluation stage also entailed the development and validation of the Index for

Lifelong Learning. From a conceptual point of view, the index for lifelong learning is based on the

maximalist approach in understanding lifelong learning, which is discussed by Hager. More to the

point, the maximalist approach interprets lifelong learning through a wider conceptual framework –

the knowledge society, which means that lifelong learning takes place in formal and non-formal

setting, as well as in informal settings, which takes place both at the individual level, as well as at the

level of groups or communities.7 Consequently, lifelong learning will be analysed from the

perspective of its triadic nature, so the index will take into consideration three main dimensions:

participation in formal education, participation in non-formal education and participation in informal

education.

A series of indicators, as well as the variables which are going to be measured were defined

for each of these dimensions in the following manner:

Participation in formal learning

Participation in early / preschool formal education – the variable which is measured is

represented by the percentage of children aged between 4 years and the age for

compulsory education who are enrolled in formal education institutions (ISCED 0-1);

Participation in secondary formal education – the variable which is measured is

represented by the percentage of individuals aged between 20 and 24 who have

graduated superior secondary education and post-secondary non-tertiary education

(ISCED 3-4);

6 Eurostat, Participation rate in education and training (last 4 weeks) by sex and age, Table trng_lfs_01, Age from 25 to

64 years, disponibil online la adresa: http://ec.europa.eu/eurostat/web/products-datasets/-/trng_lfs_01 7 Paul J. Hager, „Concepts and definitions of lifelong learning”, pp. 19 – 20, în Manuel London, The Oxford Handbook of

Lifelong learning, Oxford University Press, 2011

12

Participation in tertiary formal education – the variable which is measured is

represented by the percentage of individuals aged between 30 and 34 who have

graduated from tertiary education (ISCED 5-8);

Participation in non-formal learning

Participation in continuous non-formal education and training – the variable is

expressed by the percentage of individuals aged between 25 and 64 who have

participated in the last 12 months in continuous professional training courses;

Participation in online training courses – the variable is represented by the percentaje

of individuals aged 25 to 64 to have used the internet in the last three months before

data collection in order to participate in an online training course;

Participation in informal learning

Learning through cultural activities – visiting monuments and historical sites – the

variable which is measured is represented by the percentage of individuals aged 15 to

64 who have visited monuments or historical sites (palaces, castles, churches, gardens

etc) 3 to 5 times in the last 12 months;

Learning through cultural activities – visiting museums or galleries – the variable

which is measured is represented by the percentage of individuals aged 15 to 64 who

have visited museums or galleries 3 to 5 times in the last 12 months;

Learning through cultural activities – reading activities – the variable which is

measured is represented by the percentage of individuals are 15 to 64 who have read

books 3 to 5 times in the last 12 months;

Before being able to aggregate all the indicators in order to obtain the final value of the index

for lifelong learning, an important step is represented by the normalization of data for each of the

indicators, a process which is necessary due to the fact that most of the times indicators within a data

set have different units of measurement.8 The Min-Max method was used in order to normalize the

data, which consists in substracting the minimum value which was registers in the entire range of

values and dividing to the absolute amplitude of the indicator, which represents the difference between

the maximum and the minim values of the indicator. One of the potential risks of this approach is that

extreme values can distort the normalized indicator. Still, in the case of indicators which have reduce

8 Organization for Economic Co-operation and Development, Handbook on Constructing Composite Indicators, 2008, p.

27

13

intervals of variance, Min-Max normalization can lead to an increased amplitude of the indicator, thus

increasing its effect on the composite indicator.9

Consequently, for each indicator 𝑥𝑞𝑐𝑡 for a generic country c and a moment in time t is

transformed into 𝐼𝑞𝑐𝑡 =

𝑥𝑞𝑐𝑡 − 𝑚𝑖𝑛𝑐(𝑥𝑞

𝑡 )

𝑚𝑎𝑥𝑐(𝑥𝑞𝑡 )− 𝑚𝑖𝑛𝑐(𝑥𝑞

𝑡 )∗ 100, where 𝑚𝑖𝑛𝑐(𝑥𝑞

𝑡 ) and 𝑚𝑎𝑥𝑐(𝑥𝑞𝑡 ) represent the

minimum, respectively maximum values of the indicator 𝑥𝑞𝑐𝑡 in all countries c at the moment t, so that

the normalized indicators will have values between 0 (minimum) and 100 (maximum).

The linear aggregation method was used in order to aggregate the indicators from the main

dimensions, which entails summing up the normalized and weighed indicators. Thus, the values for

the three composite indicators belonging to each of the dimensions of lifelong learning was calculated

based on the formula 𝐶𝐼𝑐 = ∑ 𝑤𝑞𝐼𝑞𝑐𝑄𝑞=1 , where ∑ 𝑤𝑞 = 1𝑞 and 0 ≤ 𝑤𝑞≤ 1 for all indicators from q =

1 to Q and for all countries from c = 1 to M. The OECD guide for composite indicators points out the

fact that this method of aggregation imposes a series of constraints regarding the nature of the different

indicators. Thus, obtaining a quality composite indicator depends on the quality of the individual

indicators, as well as on their unit of measurement.10

With regard to the method for weighing the indicators, the method of equal weighing was

chosen. The choice for equal weighing is based on the assumption that all the indicators which have

been defined have equal importance in describing the current state of lifelong learning at the level of

EU member states. In other words, all the indicators will have an equal weight in each of the composite

indicators.

Lastly, the value for the Index for Lifelong Learning was obtained through the linear

aggregation of the three main composite indicators, respectively participation in formal learning,

participation in non-formal learning and participation in informal learning. The calculations were

made based on the formula 𝐼𝐼𝑃𝑉𝑐 = ∑ 𝑤𝑞𝐶𝐼𝑞𝑐𝑄𝑞=1 , where ∑ 𝑤𝑞 = 1𝑞 and 0 ≤ 𝑤𝑞≤ 1 for all composite

indicators from q = 1 to Q and for all countries from c = 1 to M.

With regard to the main limitations of the index, they refer to the lack of quantitative or

qualitative data at the level of the European Union’s member states regarding the variables which

have been taken into consideration. This problem is especially visible in the case of indicators which

are based on qualitative data, due to the fact that at the present time no relevant statistical data could

be identified in the database available on Eurostat. Thus, the full use and testing of the index is actually

9 ibidem, pp. 27-28 10 ibidem, p. 103

14

conditioned by the application of a new questionnaire at the level of EU member states, which

surpassed the possibilities of research in the present thesis. However, in order to compensate for this

issue, the missing values were estimate by regressing the indicators which have missing values with

the other indicators, or by using a temporal series which provides more statistical data.

The table below shows the values which were obtained for the Index for Lifelong Learning for

the years 2007 and 2013, where ICIF represents the composite indicator for formal learning, ICINF

represents the composite indicator for non-formal learning and ICIIF represents the composite

indicator for informal learning.

Year /

Member state

2007 2013

ICIF ICINF ICIIF IIPV ICIF ICINF ICIIF IIPV

Austria 56.92 27.91 46.97 43.49 61.67 24.56 63.33 49.35

Belgium 74.78 27.84 49.82 50.31 65.32 23.29 56.21 47.79

Bulgaria 55 0.86 7.63 20.95 53.57 0 28.99 27.25

Czech Republic 58.8 15.62 45.94 39.72 50.94 14.64 41.52 35.34

Cyprus 61.1 21.61 9.83 30.54 51.09 13.91 26.76 30.28

Denmark 66.43 82.32 74.15 73.56 71.46 74.88 79.04 74.38

Estonia 65.34 36.17 50.99 50.32 65.16 30.9 58.18 50.9

Finland 65.87 91.69 81.4 78.86 57.36 83.08 51.45 63.32

France 70.1 18.87 43.35 43.66 69.47 45.92 41.99 51.94

Germany 60.18 34.82 57.59 50.35 63.09 23.67 54.4 46.58

Greece 34.78 16.11 17.47 22.56 40.77 12.04 13.42 21.86

Ireland 48.09 33.58 42.05 40.83 82.51 19.68 46.85 49.18

Italy 58 22.35 28.47 35.91 55.92 21.58 33.8 36.73

Latvia 53.28 41.31 39.6 44.28 68.72 12.77 51.61 43.92

Lithuania 57.82 28.58 25.23 36.84 71.72 37.08 30.97 46.13

Luxembourg 63.8 38.07 71.82 57.32 77.28 56.89 59.21 63.82

Malta 45.87 35.07 26.92 35.59 49.49 23.86 43.85 38.68

Netherlands 68.58 47.56 79.59 64.59 70.55 45.03 77.33 63.66

Poland 43.12 4.99 33.39 26.89 59.55 4.09 21.39 28.06

Portugal 31.1 13.14 21.87 21.82 43.53 12.55 4.76 20.08

United Kingdom 58.23 75.93 47.75 60.03 68.22 60.69 58.35 61.79

Romania 41.55 5.39 28.47 24.88 37.63 6.12 28.99 24

Slovakia 44.85 11.13 45.42 33.46 36.09 12.61 30.75 26.22

Slovenia 67.7 37.58 39.99 47.94 73.24 20 37.23 43.05

Spain 57.74 62.27 61.34 59.84 51.26 55.21 37.28 47.44

Sweden 77.08 48.06 71.95 65.04 79.01 73.4 86.8 78.94

Hungary 60.52 17.61 31.19 36.08 64.24 12.61 16.89 30.94

UE-27 57.283 33.2 43.71 44.28 60.7 30.41 43.754 44.50

15

The comparative analysis of the values of the Index for Lifelong Learning for the years 2007

and 2013 shows, at least regarding the EU-27 average, that the participation of EU citizens in lifelong

learning was relatively constant, since the average value of the index for the year 2013 is only 0.22

percentage points higher than the value from 2007. However, if we look at the individual evolution of

the member states, we will see that this apparent stability is caused in reality by the fluctuations in the

participation rate in lifelong learning for each of the member states. On the one hand, in the case of

14 out of the 27 member states of the European Union which were analysed there are significant

decreases of the index for lifelong learning, among the most notable cases being Slovakia, with a

decrease of 21.6%, Spain, with a decrease of 20.7% and Finland, with a decrease of 19.7%. At the

opposite spectrum, 13 of the 27 EU member states registered significant increases, the most notable

cases being that of Bulgaria, with an increase of 30.1%, Lithuania, with an increase of 25.2% and

Sweden, with an increase of 21.4%, thus counter-balancing the decreasing tendency.

Looking strictly at the dimension of participation in formal learning at the EU-27 level, which

essentially represents the field of action of the European Union’s policies and programmes in the field

of lifelong learning, we can see an increase of 3.42 percentage points, which indicates a positive

evolution of the participation of EU citizens in formal learning, in spite of the negative effects of the

economic crisis. This evolution can in turn indicate a cumulative positive impact of all the EU

initiatives in the field which have been implemented in the 2007 – 2013 period, and implicitly of the

Lifelong Learning Programme 2007 – 2013. On the other hand, this increase is not very high, which

can possibly be explained through issues such as the negative effects of the economic crisis, which

has had a significant impact of budget allocation for the educational sector in the majority of EU

member states and implicitly on the capacity for co-financing EU interventions in this field, as well

as economic, social and cultural factors which have been previously discussed, which affect the

willingness of individuals to participate in formal education activities.

The figure below comparatively presents the values for the Index for Lifelong Learning for

each member state of the European Union during the period 2007 – 2013.

16

The results of the evaluation process which was conducted in this chapter confirm the second

research hypothesis, namely when the flexibility of the model for evaluating public policies and

programmes increases, the evaluator will be able to use a wider range of methods and

instruments for measuring the results of the policy / programme. The contingent evaluation model

which was developed allowed the use of a wide range of indicators for measuring the results of the

European Union’s policies and programmes in the field of lifelong learning. As it was previously

pointed out, one of the major limitations of the current research is represented by the impossibility of

0 10 20 30 40 50 60 70 80 90

Suedia

Danemarca

Luxemburg

Olanda

Finlanda

Regatul Unit al Marii Britanii

Franta

Estonia

Austria

Irlanda

Belgia

Spania

Germania

Lituania

UE-27

Letonia

Slovenia

Malta

Italia

Cehia

Ungaria

Cipru

Polonia

Bulgaria

Slovacia

Romania

Grecia

Portugalia

Lifelong Learning Index 2007 / 2013

IIF2007 IIF2013

17

using qualitative research methods, a limitation which derives both from the high complexity of the

Lifelong Learning Programme 2007 – 2013, as well as from the fact that the reach of the programme

is too high to be covered in this research. Thus, the results which have been obtained after applying

the contingent evaluation model to lifelong learning at the level of the European Union can be

improved or confirmed once more by using qualitative research methods, such as applying interviews

at the level of EU member states, conducting case studies or applying questionnaires.

Putting things into perspective, the contingent evaluation model which was developed in this

research has the potential of being used for evaluating the different policies and programmes of the

European Union. The increased flexibility of the model makes it easy to adapt it to different evaluation

needs, which requires a prior identification and definition of the purpose and objectives of the

evaluation process, as well as a clear identification of the difficulties which might arise due to the

specificities of the policy / programme which is being evaluated. With regard to the structure of the

evaluation process, the dimensions of context evaluation, inputs evaluation and results evaluation have

proved to be sufficient in order to provide an overall image on the merit and worth of an intervention.

The model also allows the use of a variety of research methods and techniques, so that, depending of

financial and time allocations, a mix of qualitative and quantitative methods can be used in order to

better attribute results and identify causal links more precisely.

With regard to the Index for lifelong learning, the statistical tests which have been applied in

the case study demonstrate the fact that it is an instrument which can be characterised as having a high

degree of confidence, since the results which have been obtained are valid. Obviously, the index can

be enriched by including additional indicators within the three main dimensions, a process which is

mainly conditioned by the existence of statistical data for each member state of the European Union.

Another aspect which needs to be taken into account is that the logic and structure of the index can be

replicated in order to measure the results of different policies and programmes of the European Union

from other policy domains.

18

Selected bibliography

Works by author

1. Aspin, David; Bagnall, Richard; Chapman, Judith; Evans, Karen. „Introduction and

overview”. In David Aspin, Richard Bagnall, Judith Chapman și Karen Evans (Eds.), Second

International Hanbook of Lifelong Learning, Part one, Springer, 2012

2. Chen, Huey-Tsyh. Practical Program Evaluation: Theory-Driven Evaluation and the

Integrated Evaluation Perspective, Second Edition, Sage Publications, 2014

3. Chen, Huey-Tsyh. Theory-driven evaluations, Sage Publications, 1990

4. Dahler-Larsen, Peter. „Evaluation and public management”, in Ewan Ferlie, Laurence E.

Lynn, și Christopher Pollitt, The Oxford Handbook of Public Management, Oxford University

Press, 2007

5. Donaldson, Stewart I.; Scriven, Michael (Eds.). Evaluating social programs and problems:

visions for the new millennium, New Jersey: Lawrence Erlbaum Associates, Publishers, 2003

6. Dunleavy, Patrick; Margetts, Helen; Bastow, Simon; Tinkler, Jane. „New public management

is dead – Long live Digital-Era Governance”, Journal of Public Administration Research and

Theory, Vol. 16, Issue 3, 2006

7. Field, John; Leicester, Mal. „Introduction – Lifelong learning or permanent schooling?”. In

John Field, Mal Leicester (Eds.), Lifelong learning: Education across the lifespan, Routledge,

2000

8. Hager, Paul J. „Concepts and definitions of lifelong learning”. In Manuel London, The Oxford

Handbook of Lifelong learning, Oxford University Press, 2011

9. Homburg, Vincent; Pollitt, Christopher; van Thiel, Sandra. “Introduction”. In Christopher

Pollitt, Sandra van Thiel și Vincent Homburg (Ed.), NPM in Europe: adaptation and

alternatives, Basingstoke: Palgrave Macmillan, 2007

10. Hood, Christopher. „A public management for all seasons?”, Public Administration, Volume

69, Issue 1, 1991

11. Jarvis, Peter. Globalisation, lifelong learning and the learning society. Sociological

perspectives., Lifelong learning and the learning society, Volume 2, Routledge, 2007

12. Kushner, Savile; Norris, Nigel. “The new public management and evaluation”. In Saville

Kushner, Nigel Norris, Dilemmas of engagement: evaluation and the new public management,

Elsevier, 2007

19

13. Lane, Jan Erik. New public management, Routledge, 2000

14. Lincoln, Yvonna; Guba, Egon. Fourth generation evaluation, Sage Publications, 1989

15. Pollitt, Christopher; Bouckaert, Geert. Public Management Reform. A comparative analysis

– New Public Management, Governance, and the Neo-Weberian state, Third Edition, Oxford

University Press, 2011

16. Scriven, Michael. Key evaluation checklist (KEC), The Evaluation Center, Western Michigan

University, 2013

17. Stufflebeam, Daniel. “Evaluation models”, New Directons for Evaluation, No. 89, Jossey-

Bass, 2001

18. van der Meer, Frans-Bauke. “New Public Management and Evaluation”. In Christopher Pollitt,

Sandra van Thiel şi Vincent Homburg (Ed.), NPM in Europe: adaptation and alternatives,

Basingstoke: Palgrave Macmillan, 2007

Official and programmatic documents

1. Council of the European Union, “Council conclusions of 12 May 2009 on a strategic

framework for European cooperation in education and training (ET 2020)”, Official Journal

of the European Union, C 119/2

2. Council of the European Union, “Council Resolution on a renewed European agenda for adult

learning”, Official Journal of the European Union, C 372/01

3. Council of the European Union, “Recommendation of the European Parliament and of the

Council of 18 December 2006 on key competences for lifelong learning”, Official Journal of

the European Union, 30.12.2006

4. Council of the European Union, “Recommendation of the European Parliament and of the

Council of 23 April 2008 on the establishment of the European Qualifications Framework for

lifelong learning”, Official Journal of the European Union, 06.05.2008

5. Council of the European Union, „Decision no. 1720/2006/CE of the European Parliament and

of the Council of 15 November 2006 establishing an action programme in the field of lifelong

learning”, Official Journal of the European Union, 24.11.2006

6. European Commission – Directoare-General for Education and Culture, Interim Evaluation of

the Lifelong Learning Programme (2007 - 2013) – Final Report, 2011

7. European Commission, Communication from the Commission. Europe 2020. A strategy for

smart, sustainable and inclusive growth, Brussels, 2010

20

Guides and reports

1. Bertelsmann Stiftung, ELLI European Lifelong Learning Indicators. Making Lifelong

Learning Tangible!, 2010

2. Organization for Economic Co-operation and Development, Handbook on Constructing

Composite Indicators, 2008

Statistical data

1. European Commission, Directorate-General for Education and Culture, Special

Eurobarometer 278 – European Cultural Values, September 2007

2. European Commission, Directorate-General for Education and Culture, Special

Eurobarometer 399 – Cultural access and participation, November 2013

3. Eurostat, Individuals who have used Internet, in the last 3 months, for doing an online course

(of any subject), Table isoc_bde15cua, available online at:

http://ec.europa.eu/eurostat/web/information-society/data/database

4. Eurostat, Participation in early childhood education, Table tps00179, Age between 4-years-

old and the starting age of compulsory primary education, available online at:

http://ec.europa.eu/eurostat/tgm/table.do?tab=table&init=1&plugin=1&language=en&pcode

=tps00179

5. Eurostat, Participation rate in education and training (last 4 weeks) by sex and age, Table

trng_lfs_01, Age from 25 to 64 years, available online at:

http://ec.europa.eu/eurostat/web/products-datasets/-/trng_lfs_01

6. Eurostat, Participation rate in education and training (last 4 weeks) by type, sex and age – Non-

formal education, Table trng_lfs_09, available online at:

http://ec.europa.eu/eurostat/web/education-and-training/data/database

7. Eurostat, Population by educational attainment level, sex and age (%) – main indicators, Table

edat_lfse_03, available online at: http://ec.europa.eu/eurostat/web/education-and-

training/data/database

8. Eurostat, Tertiary educational attainment by sex, age group 30 – 34, Table t2020_41, available

online at:

http://ec.europa.eu/eurostat/tgm/download.do?tab=table&plugin=1&language=en&pcode=t2

020_41