Evaluation of Education for Democracy and Tolerance

by Gabriele Ketterle

The project "Education for Democracy and Tolerance” at the Center for Applied Policy Research (Munich) has been accompanied by a formative evaluation in the adaptation, development and testing of its workshop programs ("Betzavta” from Israel, "A world of Difference” from the US and "Respect (+) Tolerance” which was developed by the project). While formative evaluation helped to steadily improve the quality of the programs, a summative evaluation is now about to look at the effects of the programs over a period of three years. Its goal is to identify which changes in attitude and behavior have taken place with the participants. Since a comprehensive concept of summative evaluation of civic education does not yet exist in Germany, the first task was to establish such a theoretical framework.

First of all it is necessary to consider the conceptual basis of the educational project since this is also the basis for framing the evaluation. There are three dimensions of impact: 1. The project aims at providing knowledge as well as practical competencies concerning tolerance and democracy. 2. A social space for personal introspection and reflection is being created and experiential learning allows for a change of perspectives. 3. Additionally, since the programs are process-oriented, there can be unintended side effects which were not planned initially. Especially the last two dimensions show that "evaluation can not be comprehended as an unproblematic recording of data”. Therefore an approach with qualitative methods is chosen.

The underlying basis of the project’s work is social constructivism. This means turning away from traditional moralistic vocabulary that sets up dichotomies like good / bad and better / worse. In terms of evaluation it means turning away from the idea of reaching and measuring certain levels of morality . However this does not mean giving up criteria of quality and success. The development of such criteria was conceptualized in a way as to pay respect to the particular pedagogical approach. The following criteria for the evaluation were developed:

• All stakeholders involved in the evaluation process are being integrated into the design of the evaluation
• The addressees of the evaluation are being regarded as acting and autonomous subjects
• The process of evaluation is open and transparent
• The findings will be discussed and validated with the participants of the evaluation
• A pluralism of values and perspectives will be accepted and the openness to different concepts of impact and effect will be assured.

From these categories it becomes clear that evaluation is seen as a self-reflexive dimension which does not come from the outside, but stems from within the project itself.

The evaluation is divided into three areas according to the three dimensions of impact of the educational programs: 1. Evaluation of knowledge and competencies via standardized methods. 2. Evaluation of ability to reflect and switch perspectives, which "is the fundamental essence of the evaluation”. Here qualitative case studies will be conducted. 3. Evaluation of unintended side effects. Here methods are being used that do not work with preconceived lists of criteria.

The concept introduced here has been discussed with external institutes which proposed evaluation designs on its basis. For each of the three programs a different approach was chosen: The evaluation of the program "Betzavta” focuses on the trainers who are not only interviewed but also included into the evaluation in form of a steering group. Especially their understanding of the didactical basis and experience in working with the program will be taken into consideration and be compared with its goals. For "A world of Difference” the evaluation will look at two school class which will be accompanied over a period of time. Not only the students but also the "products” of the trainings as well as the teachers and parents will be included into the evaluation. A steering group with all stakeholders will make the process transparent. The program "Respect (+) Tolerance” will be evaluated by looking at the totality of social interaction in one week-long workshop. Individual narrative and group interviews will cover as many aspects of the social life of participants as possible.

I find this particular translation of the pedagogical basis of the project into a concept of evaluation to be very interesting and relevant. Evaluation of political education can not only rely on the usual standards, but has to furthermore integrate democratic aspects into its very design and process. One difficulty is the strict following through of a constructivist conception. The communication of the results can in this way not be reduced to a "formula of effect” which some stakeholders, as e.g. funders, might expect. With this very open concept there is also the risk of finding out not only unintended, but downright uneasy and counter-productive answers undermining the project´s work. Nevertheless, in the age of a pluralistic and open society this approach is necessary and authentic.

Index: Podium

Intergroup Relations Programs: How Effective Are They?
by Walter G. Stephan & Cookie White Stephan

4th Annual Network Conference
by Michael Seberich

Presentation of the "Power of Language" - Handbook at the Council of Europe
by Viola Georgi

Empowerment and Human Security in a time of transition - impressions from a visit in South Africa
by Katrin Uhl

The Aquino Foundation

Evaluation of Education for Democracy and Tolerance
by Gabriele Ketterle