The new Millennium has seen an increased concern over our lack of understanding of what works in international development, and what lessons we can draw from seemingly successful and unsuccessful development initiatives. However, many of the approaches used by international development agencies to assess the impacts and effects of their development interventions can potentially result in the findings having a positive bias so that the impacts and benefits of the interventions are over-estimated, while the proportion of the target population who do not benefit may be under-estimated and the potential negative consequences of the interventions may frequently be ignored. These biases are due to a combination of factors including: budget and time constraints, limited access to data – particularly baseline data; how evaluations are commissioned and managed; and political and organizational constraints and pressures. These biases have important consequences when evaluation findings are used to inform future management and policy decisions. The above factors can also produce negative biases whereby the findings can, internationally or inadvertently, under-estimate program outcomes. While these biases are more obvious for under-resourced evaluations with unrealistically short deadlines, there are also a number of potential positive biases that can also affect well resourced “strong” quantitative evaluation designs. A number of recommendations are proposed to reduce positive bias and strengthen the validity of evaluation findings by strengthening how evaluations are managed and through strategies to strengthen the evaluation methodology, even when operating under budget and time constraints.
Read the full paper by registering as a participant in the conference and going to the closed area of the Downloads Section of the website. Then click on Michael Bamberger’s name. This will take you to Michael Bamberger’s front page. You can then link to her Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of Michael Bamberger’s work by clicking on the buttons on his front page.
In addition to access to this paper, registration as a participant provides you with access to discussion groups, and to downloads of other papers, case studies and teaching materials from the conference. Register by clicking on the Participants button in the main menu.
“The important tasks of investigating what works and what does not in the fight against poverty cannot be monopolized by one method.”
Evaluating for development results opens questions:These are some of the questions I want to speak to here in discussing methodological issue we struggle with in evaluating for development results.
Let me declare my own position: development results are not about the project or program being implemented but about the change that is taking place on the ground. It is therefore essential to consider results from the perspective of what change is happening on the ground, not what change is happening in the program or the project.
Read the full paper by going to the closed area of the Downloads Section of the website. Then click on and clicking on Fred Carden’s name. This will take you to Fred Carden’s front page. You can then link to Fred Carden’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons his front page.
In addition to access to this paper, registration as a participant provides you with access to discussion groups, and to downloads of other papers, case studies and teaching materials from the conference. Register by clicking on the Participants button in the main menu.
The traditional top-down approach to program outcome evaluation stresses the need for strong evidence to establish an intervention’s efficacy or effectiveness. This approach’s principles and methods are rooted in Campbellian typology and have been applied intensively in outcome evaluation. Yet lessons learned from such applications suggest that in addressing stakeholders’ interests and needs, this approach has limitations. To be stakeholder-responsive, evaluation must go beyond the top-down approach’s focus and strategies. This paper proposes the integrative validity model as an alternative perspective and a bottom-up approach to outcome evaluation. The new perspective enables evaluations to meet both scientific and practical requirements.
Read the full paper by registering as a participant in the conference and going to the closed area of the Downloads Section of the website. Then click on Ross Conner’s name. This will take you to Ross Conner’s front page. You can then link to his Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of Ross Conner’s work by clicking on the buttons on his front page.
In addition to access to this paper, registration as a participant also provides you with access to discussion groups, and to downloads of other papers, case studies and teaching materials from the conference. Register by clicking on the Participants button in the main menu.
Collaborative approaches to evaluation represent a considerable point of departure from traditionalist genres, such as evaluation framed by experimental and quasiexperimental designs or those associated with measuring and explaining variation in program performance. There are numerous variants, all of which involve persons trained in evaluation working in partnership or in collaboration with members of the program community be they program developers, managers, implementers, funders, intended beneficiaries, or other relevant stakeholder groups. Over the years, several forms of collaborative evaluation have emerged with such descriptive designations as ‘Practical Participatory Evaluation’ (Cousins & Earl, 1992; 1995), ‘Transformative Participatory Evaluation’ (Tandon & Fernandes, 1982; 1984), ‘Participatory Rural Appraisal’ (R. Chambers, 1994, 2003), ‘Deliberative Democratic Evaluation’ (House & Howe, 2000), ‘Utilization Focused Evaluation’ (Patton, 1997, 2008), and Empowerment Evaluation’ (Fetterman, 2001; Fetterman & Wandersman, 2005), with each type distinguishable by the level and nature of stakeholder involvement and justification /rationale for the approach.
Situated among these myriad approaches the term participatory evaluation has taken on multiple meanings and understandings over time, a fact that may serve to obscure the heterogeneity found in practice (Gregory, 2000) and may further obfuscate the discernable differences that might be evident at methodological and practical levels. As Gregory (2000) notes, “there are different degrees of participation and …consistency in approach cannot be assumed across evaluations” (p. 180). As we have noted elsewhere (Cousins, 2007; Cousins & Whitmore, 1998), conceptual clarity is essential to deepening our understanding of collaborative inquiry. Also essential, in our view, is the systematic analysis of a growing body of empirical research in the area. That empirical research is essential to the advancement of evaluation as a field (Christie, 2003; Cousins, 2004; Mark, 2008; Smith, 1993), is a point that is especially relevant at this juncture as the research knowledge base has matured to a point that warrants a systematic rendering of the participatory evaluation landscape, identifying not only what is known, but also what remains pressing to know.
The intent of this paper is to provide a comprehensive and critical review and synthesis of empirical research on participatory and related forms of collaborative evaluation. To accomplish this task we settled on a set of guiding questions of considerable scope:
Read the full paper by registering as a participant in the conference and going to the closed area of the Downloads Section of the website. Then click on Jill Chouinard’s name. This will take you to Jill Chouinard’s front page. You can then link to her Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of Jill Chouinard’s work by clicking on the buttons on his front page.
You can also access the paper via Brad Cousins’ front page.
In addition to access to this paper, registration as a participant also provides you with access to discussion groups, and to downloads of other papers, case studies and teaching materials from the conference. Register by clicking on the Participants button in the main menu.
After several years of discussion and development that began in 1997, the International Organisation for Cooperation in Evaluation (IOCE) was created in 2003 by representatives from 24 national and regional evaluation organizations and networks. Following the foundational work of a transitional board, IOCE’s first official Board of Trustees and Executive Committee were designated at a meeting in Toronto, Canada, in late 2005. On 1 January 2006, IOCE began its first official year and its first 2-year term of planned activities. This paper is a description of IOCE, an analysis of its first official two years, and an assessment of its strengths, weaknesses, challenges and opportunities. The sections below focus on IOCE’s general mission and vision, its central focus on advocacy, its main strategy of partnerships, its governance and recent activities, as well as on its funding and membership. The paper concludes with a discussion of the successes and challenges and of the needs and opportunities for the organization, along with some recommendations as IOCE moves forward in its development.2
Read the full paper by registering as a participant in the conference and going to the closed area of the Downloads Section of the website. Then click on Ross Conner’s name. This will take you to Ross Conner’s front page. You can then link to his Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of Ross Conner’s work by clicking on the buttons on his front page.
You can also access the paper via Paul Garbe’s front page.
In addition to access to this paper, registration as a participant provides you with access to discussion groups, and to downloads of other papers, case studies and teaching materials from the conference. Register by clicking on the Participants button in the main menu.
Collaborative approaches to evaluation represent a considerable point of departure from traditionalist genres, such as evaluation framed by experimental and quasiexperimental designs or those associated with measuring and explaining variation in program performance. There are numerous variants, all of which involve persons trained in evaluation working in partnership or in collaboration with members of the program community be they program developers, managers, implementers, funders, intended beneficiaries, or other relevant stakeholder groups. Over the years, several forms of collaborative evaluation have emerged with such descriptive designations as ‘Practical Participatory Evaluation’ (Cousins & Earl, 1992; 1995), ‘Transformative Participatory Evaluation’ (Tandon & Fernandes, 1982; 1984), ‘Participatory Rural Appraisal’ (R. Chambers, 1994, 2003), ‘Deliberative Democratic Evaluation’ (House & Howe, 2000), ‘Utilization Focused Evaluation’ (Patton, 1997, 2008), and Empowerment Evaluation’ (Fetterman, 2001; Fetterman & Wandersman, 2005), with each type distinguishable by the level and nature of stakeholder involvement and justification /rationale for the approach.
Situated among these myriad approaches the term participatory evaluation has taken on multiple meanings and understandings over time, a fact that may serve to obscure the heterogeneity found in practice (Gregory, 2000) and may further obfuscate the discernable differences that might be evident at methodological and practical levels. As Gregory (2000) notes, “there are different degrees of participation and …consistency in approach cannot be assumed across evaluations” (p. 180). As we have noted elsewhere (Cousins, 2007; Cousins & Whitmore, 1998), conceptual clarity is essential to deepening our understanding of collaborative inquiry. Also essential, in our view, is the systematic analysis of a growing body of empirical research in the area. That empirical research is essential to the advancement of evaluation as a field (Christie, 2003; Cousins, 2004; Mark, 2008; Smith, 1993), is a point that is especially relevant at this juncture as the research knowledge base has matured to a point that warrants a systematic rendering of the participatory evaluation landscape, identifying not only what is known, but also what remains pressing to know.
The intent of this paper is to provide a comprehensive and critical review and synthesis of empirical research on participatory and related forms of collaborative evaluation. To accomplish this task we settled on a set of guiding questions of considerable scope:
This electronic document is intended to stimulate thought and discussion about emerging ideas for improving the quality of modern evaluation practice. It will focus on lessons learned from recent debates about what constitutes credible and actionable evidence in evaluation practice. This document primary consists of excerpts from a recent book on What Counts as Credible Evidence in Applied Research and Evaluation Practice? (Donaldson, Christie, & Mark, 2008) and supporting web-based resources hot linked throughout so that readers can explore the topics introduced in much more depth.
Recent Debates about EvidenceIn 2001, Donaldson and Scriven (2003) invited a diverse group of applied researchers and evaluators to the Claremont Colleges in southern California (USA) to provide their visions for a desired future for evaluation practice. The heat generated at this symposium suggested that whatever truce or peace had been achieved in the quantitative-qualitative paradigm wars raging in the 1970s and early1980s, remained an uneasy peace (Mark, 2003). For example, Yvonna Lincoln and Donna Mertons envisioned a desirable future based on constructivist philosophy, and Mertons seemed to suggest the traditional quantitative social science paradigm, specifically randomized experiments were an immoral methodology (Mark, 2003). Thomas Cook responded with a description of applied research and evaluation in his world, which primarily involved randomized and quasi-experimental designs as normative and highly valued by scientists, funders, stakeholders, and policy makers alike. Two illustrative observations by Mark (2003) highlighting differences expressed in the discussion were (1) “I have heard some quantitatively oriented evaluators disparage participatory and empowerment approaches as technically wanting and as less than evaluation,” and (2) “It can, however, seem more ironic when evaluators who espouse inclusion, empowerment, and participation would like to exclude, disempower, and see no participation by evaluators who hold different views.” While the symposium concluded with some productive discussions about embracing diversity and integration as ways to move forward, it was clear there were lingering differences and concerns about what constitutes quality applied research, evaluation, and credible evidence. A full account of these discussions, supporting resources, and the volume that resulted can be found at: http://sites.google.com/site/evaluatingsocialprograms/.
Donaldson and Christie (2005) noted that the uneasy peace seemed to revert back to overt conflict in late 2003. The trigger event occurred when the U.S. Department of Education’s Institute of Education Sciences declared a rather wholesale commitment to privileging experimental and some types of quasi-experimental designs over other methods in applied research and evaluation funding competitions. At the 2003 Annual Meeting of the American Evaluation Association (AEA), prominent evaluators discussed this event as a move back to the “Dark Ages” of evaluation (Donaldson & Christie, 2005). The leadership of the American Evaluation Association developed a policy statement opposing these efforts to privilege randomized control trials in education evaluation funding competitions:
AEA Statement:
American Evaluation Association Response
To U. S. Department of Education
Notice of proposed priority, Federal Register RIN 1890-ZA00, November 4, 2003
"Scientifically Based Evaluation Methods"
The American Evaluation Association applauds the effort to promote high quality in the U.S. Secretary of Education's proposed priority for evaluating educational programs using scientifically based methods. We, too, have worked to encourage competent practice through our Guiding Principles for Evaluators (1994), Standards for Program Evaluation (1994), professional training, and annual conferences. However, we believe the proposed priority manifests fundamental misunderstandings about (1) the types of studies capable of determining causality, (2) the methods capable of achieving scientific rigor, and (3) the types of studies that support policy and program decisions. We would like to help avoid the political, ethical, and financial disaster that could well attend implementation of the proposed priority.
Donaldson and Christie (2005) documented an important response to the AEA Statement from an influential group of senior members of the American Evaluation Association. This group opposed the AEA Statement, and did not feel they were appropriately consulted as active, long-term members of the association. Their response became known as “The Not AEA Statement.”
The Not AEA Statement:
Posted on Evaltalk on: 12-3-2003
AEA members:
The statement below has been sent to the Department of Education in response to its proposal that "scientifically based evaluation methods" for assessing the effectiveness of educational interventions be defined as randomized experiments when they are feasible and as quasi-experimental or single-subject designs when they are not.
This statement is intended to support the Department's definition and associated preference for the use of such designs for outcome evaluation when they are applicable. It is also intended to provide a counterpoint to the statement submitted by the AEA leadership as the Association's position on this matter. The generalized opposition to use of experimental and quasi-experimental methods evinced in the AEA statement is unjustified, speciously argued, and represents neither the methodological norms in the evaluation field nor the views of the large segment of the AEA membership with significant experience conducting experimental and quasi-experimental evaluations of program effects.
We encourage all AEA members to communicate their views on this matter to the Department of Education and invite you to endorse the statement below in that communication if it is more representative of your views than the official AEA statement. [Comments can be sent to the Dept of Ed through Dec. 4 at comments@ed.gov with "Evaluation" in the subject line of the message].
************************************
This statement is in response to the Secretary's request for comment on the proposed priority on Scientifically Based Evaluation Methods. We offer the following observations in support of this priority.
Read the full paper by registering as a participant in the conference and going to the closed area of the Downloads Section of the website. Then click on Stewart Donaldson’s name. This will take you to Stewart Donaldson’s front page. You can then link to his Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of Stewart Donaldson’s work by clicking on the buttons on his front page.
In addition to access to this paper, registration as a participant provides you with access to discussion groups, and to downloads of other papers, case studies and teaching materials from the conference. Register by clicking on the Participants button in the main menu.
Empowerment evaluation is the use of evaluation concepts, techniques,
and findings to foster improvement and self-determination. It is aimed
at increasing the probability of achieving program success by (1)
providing people with tools for assessing the planning, implementation,
and self-evaluation of their programs, and 2) mainstreaming evaluation
as part of their planning and management (Fetterman and Wandersman,
2005).
It is a global phenomenon. It has been used throughout the world, from
Australia to the United States and Brazil to Spain, as well as Ethiopia,
Nepal, New Zealand, and South Africa. Empowerment evaluation’s impact on
the Stanford University’s School of Medicine curriculum was found to be
statistically significant (Fetterman, 2009, 2010). Empowerment
evaluation has also been used in Native American reservations and
tobacco prevention programs. (See
http://www.davidfetterman.com/empowermentevaluation.htm for videos
of empowerment evaluation projects.
2The author is grateful to members of IOCE’s 2006-2007 and 2008-2009 Executive Committees for input, comments and suggestions on earlier versions of this paper. The author’s summary comments in the ‘Discussion’ section, however, are his own.
Read the full paper by going to the closed area of the
Downloads Section of the website. Then click on David Fetterman’s name.
This will take you to David Fetterman’s front page. You can then link to
David Fetterman’s Paper, Powerpoint, Biography, List of Publications and
also hyperlink to other areas of his work by clicking on the buttons his
front page.
You can also click on the hyperlink David Fetterman has provided in the
Introduction above to see how the programme evaluation methodologies he
advocates have been implemented in practice.
In addition to access to this paper, registration as a participant
provides you with access to discussion groups, and to downloads of other
papers, case studies and teaching materials from the conference.
Register by clicking on the Participants button in the main menu.
Good day to all. I send greetings from Urbana-Champaign, Illinois, USA – a small urban community surrounded by a majestic prairie with deep, rich soil on which Illinois farmers grow vast quantities of corn and soybeans. These crops are made into products ranging from food for animals to fuel for automobiles. Illinois farmers also grow and produce the blueberries and peaches, the fresh greens and broccoli, the bell peppers and onions, and the very delicious goat cheese that I purchase at our Saturday morning farmers market, all summer long.
... It is indeed a privilege to live here and to benefit so much from the generosity of the land.
... Yet, I know that not all community residents share the realities of my experience of privilege.
I also have the privilege from living near and working at the highly
regarded University of Illinois, located in Urbana-Champaign. This flagship
university of the state of Illinois is home to wonderful scholars and
educators of all kinds. Here are some snapshots of current activities
at my university (www.illinois.edu, retrieved July 2, 2010).
The FORmative Evaluation Consultation And Systems Technique (FORECAST) is an approach to formative evaluation that offers specific models and tools for program improvement. In this article, we will 1) describe and present updates/refinements to FORECAST, and 2) discuss contributions of the FORECAST approach to formative evaluation practice.
According to Michael Scriven (1991), the proper function of evaluation is making a judgment of a program’s value or effectiveness – i.e., summative evaluation. Under this premise Scriven regarded formative evaluation as a strategy for providing a preview, or “early-warning,” of summative evaluation results. While we accept Scriven’s (1967) notion that formative evaluation is temporally prior to summative evaluation, we consider formative evaluation to be fundamentally a strategy for program implementation and improvement. Formative evaluation produces information during the developmental stages of a program that is used to improve programming to increase the likelihood of positive outcomes (Goodman & Wandersman, 1994).
FORECAST was first used in a five-year project funded by the Center for Substance Abuse Prevention (CSAP) (Goodman & Wandersman, 1994) as an approach to formative evaluation that provides models and tools for program development and improvement (Goodman & Wandersman, 1994). Within this project, community partnerships in South Carolina were established and subsequently tasked with assessment of community needs and implementation of comprehensive alcohol, tobacco, and other drug (ATOD) prevention programming. CSAP partnerships used FORECAST to develop program models and integrated the use of tools for planning, implementing, and evaluating ATOD interventions.
A model of the problem diagrams the perceived causes and effects of the health or social problem of interest, and a model of the solution depicts how the program or intervention will address the problem. The CSAP partnerships developed a model of the problem that outlined risk and protective factors for ATOD abuse (i.e., individual differences, family influences, and community influences) (see Appendix A). Partners subsequently used the model of the problem as a basis for developing a model of the solution depicting a comprehensive strategy for needs assessment, planning, implementation, and evaluation of ATOD prevention programs (see Appendix B).
Markers are short-term steps which afford incremental success in implementing the model of the solution. Measures assess whether markers are accomplished by a certain timeline, and meaning provides quality standards or benchmarks for each of the markers. As with models, the development of markers, measures, and meaning involves mutual decision-making between the evaluator and other project stakeholders (Goodman & Wandersman, 1994). Project markers adopted by CSAP partnerships included the selection of local leaders, mechanisms for committee meetings, and strategies for collecting and monitoring needs assessment data see (Appendix C).. Examples of measures were meeting minutes, telephone and appointment logs, and surveys to assess meeting effectiveness. Meaning included agreed-upon benchmarks for sufficient meeting quality (e.g., minimum attendance level, inclusion of relevant agenda items).
Read the full paper by registering as a participant in the conference and going to the closed area of the Downloads Section of the website. Then click on Jason Katz’s name. This will take you to Jason Katz’s front page. You can then link to his Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of Jason Katz’s work by clicking on the buttons on his front page.
You can also access the paper via Abe Wandersman’s front page.
In addition to access to this paper, registration as a participant also provides you with access to discussion groups, and to downloads of other papers, case studies and teaching materials from the conference. Register by clicking on the Participants button in the main menu.
I am writing just under the wire, at the tail end of this interactive conference, under the category of “Program Evaluation Methodologies Applicable in Developed and Developing World Contexts.” I write from the context in which I live and work, the Midwestern United States. Although I travel and consult internationally, I have little direct experience with international evaluation projects. The organizations with which I typically work, however, share the features of many development settings. On the resource side, they are frequently small, subject to the whims of funders’ evolving interests, and, if not strapped for cash, thinking hard about where their continuation funding will come from. Money to support program evaluation is often limited or non-existent.
On the accountability side, funders often require these organizations to complete compulsory program evaluations (sometimes as an unfunded mandate) that in some cases require quantitative outcome or impact data pre-maturely, occasionally demanding causal proof after only a year or two amidst a swirl of changes in an unstable social environment. The evaluator in me recognizes the impossibility of valid measurement in such a context, but the demand is there, nevertheless. Staff members routinely discuss cultural differences among themselves, funders, and participants, but all parties profess a desire and commitment to improve the conditions of those whom they serve, however they serve them. In this paper, I will use three examples in discussing my current practice: an education program in a large school district; a university-community evaluation collaboration in development, and a faith community’s social action dilemma.
My two-part thesis is straightforward: (1) Meaningful community data can and must play a meaningful role in focusing programs and later in focusing an accompanying evaluation (a value position), and (2) inexpensive participatory methods for generating information can enable every evaluator to collect and analyze such data (a pragmatic position). In other words, it is important to gather data from a variety of sources both as programs are being established and later as they are being evaluated, and everyone—even staff and communities without extensive evaluation resources—can afford to use these data-collection methods. My goal, difficult (perhaps impossible) to attain, is to provide accountability to the people, moving beyond the funder to the ultimate beneficiaries of our programs. I reference here Robert Kennedy’s mid-1960s desire, as I understand it, to provide individual community members final say about the success of social programs in their neighborhoods. The subsequent development of the field of program evaluation in the USA surely deviated from this vision, but for those of us who vividly remember the ‘60s, it remains a guiding hope. Hence the importance of community-based needs assessment in general and participatory needs assessment in particular.
Read the full paper by registering as a participant in the conference and going to the closed area of the Downloads Section of the website. Then click on Jean King’s name. This will take you to Jean King’s front page. You can then link to her Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of Jean King’s work by clicking on the buttons on her front page.
In addition to access to this paper, registration as a participant provides you with access to discussion groups, and to downloads of other papers, case studies and teaching materials from the conference. Register by clicking on the Participants button in the main menu.
Program evaluators in Africa face many of the same challenges
that evaluators do world-wide. However, uniquenesses associated with
their history and contexts need to be given consideration in
planning, implementing and using evaluations. Issues related to
social justice and human rights are particularly salient in the
African context given its history of colonization, poverty, and
violence. Evidence of a desire for change and the resilience of
oppressed communities also need to be given consideration. The
transformative paradigm is offered as a philosophical framework that
is specifically focused on providing a guide to clarifying
assumptions that rest on the desire to immerse oneself in cultural
issues, be cognizant of differential access to power, act on the
implications of challenges in the form of oppression and
discrimination, and involve communities in meaningful ways in the
process.
Complexities in communities in terms of dimensions of diversity
contribute to the need to carefully examine assumptions for
evaluation work that has the potential to bring about social
transformation. For example, in Africa, people have long been
targets of discrimination based on economic status, religion, tribal
affiliations, and the color of their skin. These bases of
discrimination resulted in significant economic disparities that are
apparent in the large proportion of the population who live in
poverty in townships, shanty towns, and illegal squatter camps, many
times in housing that is not connected to water or electric
services. These economic disparities are also associated with
exposure to higher rates of violence and poorer access to quality
education, employment opportunities, and health care. When other
dimensions of diversity are added to this list, such as language,
disability, deafness, and gender, the picture becomes even more
complex (Mertens & Musoyka, 2007; Musyoka, 2007).
Simply listing the characteristics of people that are used as a
basis for discrimination is not sufficient for evaluators to conduct
an evaluation that has the potential for social transformation. The
evaluator has the responsibility to be cognizant of the relationship
between these characteristics and inequities in terms of power. As
Symonette (2004) reminds us:
The complexities of communities, differentials in terms of access to power and resources, a legacy of discrimination and oppression, and a desire for an increase in social justice and human rights constitute the historical and modern context in Africa that necessitates the clarification of the ethical principles that guide evaluators in their work. To this end, this presentation describes the Transformative Paradigm (Mertens, 2009; 2010) as a framework that prioritizes concerns about social justice and human rights, while providing guidance in terms of the methods of systematic inquiry used by evaluators.
Read the full paper by going to the closed area of the Downloads Section of the website. Then click on Donna Mertens’ name. This will take you to Donna Mertens’ front page. You can then link to Donna Mertens’ Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons her front page.
I am writing from my office on the Mississippi River just a short distance from where the main interstate highway bridge that runs through Minneapolis collapsed August 1, 2007 killing 13 people. The subsequent investigation discovered that inspections of that bridge done four years earlier revealed the buckling of the steel gusset plates that join the supporting pillars and beams of the bridge, the failure of which led to the bridge’s collapse. But those findings were never used or acted on.
I mention this because it is a cautionary tale about what remains I believe a primary challenge to development effectiveness evaluation -- and that is the failure to carefully design evaluations with use in mind and to actually use evaluation findings once they are produced. A great deal of evaluation continues to be compliance activity – mere paperwork procedures -- done to meet accountability mandates rather than to seriously support learning and decision-making. A common example of poor evaluation timing is the widespread practice of requiring end-of-project evaluations that are produced too late to actually inform real future planning or to inform decision-making about a major project since, in fact, any such decisions would be taken well in advance of the end of a project. While on the surface, it might appear that evaluating projects at the end would be sensible, that timing often means evaluation results are not produced in a timely fashion that connects to real planning and budgeting processes. So, the first lesson learned I would offer is the importance of planning for and actually using evaluations.
Read the full paper by registering as a participant in the conference and going to the closed area of the Downloads Section of the website. Then click on Michael Patton’s name. This will take you to Michael Patton’s front page. You can then link to his Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of Michael Patton’s work by clicking on the buttons on his front page.
In addition to access to this paper, registration as a participant provides you with access to discussion groups, and to downloads of other papers, case studies and teaching materials from the conference. Register by clicking on the Participants button in the main menu.
In this paper I consider evaluation in environmental, resource and conservation settings and posit that evaluation in these settings occurs at the intersection of human and natural systems. This characteristic not only distinguishes evaluation in these settings but also introduces the concept of a two system evaluand. The paper describes the difference these settings make to evaluation infrastructure including preparation of evaluation consumers and practitioners and the difference that a two system evaluand makes to evaluation methods. Finally I briefly sketch a weak intellectual infrastructure for evaluation in these settings.
Read the full paper by registering as a participant in the conference and going to the closed area of the Downloads Section of the website. Then click on Andy Rowe’s name. This will take you to Andy Rowe’s front page. You can then link to his Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of Andy Rowe’s work by clicking on the buttons on his front page.
In addition to access to this paper, registration as a participant provides you with access to discussion groups, and to downloads of other papers, case studies and teaching materials from the conference. Register by clicking on the Participants button in the main menu.
In 1974 Barry MacDonald, one of the first evaluators in the United Kingdom when evaluation became a distinct, contemporary field of activity, wrote a paper outlining a concept of democratic evaluation. The paper, ‘Evaluation and the control of education’, received a great deal of attention (both in practice and critique) and has been published in several places (see, for example, MacDonald, 1974, 1976, 1987). In this paper MacDonald posits a political classification of evaluation in terms of ideal types, bureaucratic, autocratic and democratic. It is only the democratic model which concerns us here, as this is the focus of this paper, though see in the text for a comment on its emergence in relation to the other two types.
Let me declare my interest. I was a Senior Research Associate and Evaluator in the Centre for Applied Research in Education at the University of East Anglia when the model was first introduced and had the opportunity to try it out in the field. I took a literal interpretation at first but modified this in practice as I came to realize that just as each case is different, so is the application of the model in different unique socio-political contexts. While maintaining adherence to the key concepts and principles which underpin the model, the exact way in which it needed to be interpreted in practice differed. My experience in using this model which has underpinned my evaluation
What is important about these key concepts is how they are interpreted and used, and they have often been misinterpreted, which I will come to later in the critique. But essentially they form a procedural basis for the sharing of evaluation knowledge equitably and fairly. Confidentiality helps to secure the trust and conditions necessary to gather honest, valid data. Negotiation is the means through which data that is not harmful to individuals and which has been agreed by them is released for public knowledge. Accessibility refers to the need to communicate to audiences beyond the case in ways they understand.
An early set of democratic procedures I adopted in a school case study ran like this:
In subsequent evaluations I modified these procedures to take account of different socio-political contexts, the difference between private and professional knowledge and the time it takes to negotiate, especially in contexts where there are multiple stakeholders and where individuals wish to avoid clearance of data and/or restrict publication. These modifications accepted that there might be instances where it was not possible for the evaluation to change a report on the basis of suggested editions, where to do so would compromise other persons’ contributions or the integrity of the findings. It included a procedure that did not allow individuals to restrict information under the confidentiality principle when it was already public knowledge. On the time issue, a distinction was made between ‘on the record’ and ‘off the record’ data and strict deadlines introduced for the return of comments to avoid intentional delay. (See Appendix 1).
In some respects the ethical procedures are similar to those customarily adopted in social science research such as informed consent, confidentiality and pre-publication access. However in democratic procedures much more weight is given to:
The role of evaluation in a democracy has been a common theme in evaluation discourse since evaluation first became a field of study evaluating social and educational programmes, and several approaches to working with democratic values in evaluation have been advanced over the years. This paper explores and critiques a particular model of democratic evaluation that was first introduced into the evaluation literature by Barry MacDonald, thirty five years ago when control of education was in contention. It outlines the premises on which this model is based and the context in which it arose. It gives examples of how it worked in the field and cites critiques that have been made of it in theory and practice. The paper concludes with a comment on the role of such a model and various other interpretations of democratic evaluation in the 21st Century.
Read the full paper by registering as a participant in the conference and going to the closed area of the Downloads Section of the website. Then click on Helen Simons’ name. This will take you to Helen Simons’ front page. You can then link to his Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of Helen Simons’ work by clicking on the buttons on her front page.
In addition to access to this paper, registration as a participant provides you with access to discussion groups, and to downloads of other papers, case studies and teaching materials from the conference. Register by clicking on the Participants button in the main menu.
In the last few months, I have reflected on what I think is important about educational program evaluation, and how responsive evaluation fits into the practice of professional evaluators. I am especially concerned about the tendency to see informal and formal evaluation as worlds apart.
Education is infused with evaluation. Social service is infused with evaluation. All professional work is infused with evaluation. It is not a matter of choice. One cannot do one’s work without sensing the quality of things (Eiseley, 1971).
Improvement in the quality of teaching and learning--and quietly thinking--requires refinement, refinement in the recognition of merit and shortcoming (Scriven, 1967). Some of an evaluating person’s refinement comes naturally with age, and with challenge and commitment, but unfortunately, just as naturally, come distortion and diminution of skill. With good meta-evaluation, i.e., with good management of evaluation, the eye becomes sharper and the will becomes stronger. We can train ourselves to be better evaluators. But ever and ever, some judgments go awry.
Formal evaluation is the conscious disciplining of judgment. We train ourselves better to see value. Each conscious step taken to manage, to refine, to validate judgment extends its formalization. Formalization is disciplined thinking, often a way of making thinking impersonal and artificial. We become more exact in identifying the things we evaluate, the evaluands. Soon we have styles, protocols, models, and mechanisms to refine our formalizations, and we argue about which are the better. Personally I look for ways to be responsive to the situation, empathic with stakeholders. I do not worry much about “going native.”
Formal and informal evaluation are part of the same act. People who become professional evaluators, or experts in any way, or arbiters or leaders, formalize their ways of recognizing quality. They rely still on the informal, as well as the formal. The boundary between formal and informal is indistinct, the mix is a gradation of refinement. In the ubiquity of evaluation, only some refinement will be apparent. It should be more apparent in the work of those of us who see evaluation as a skill to be sharpened and sensitized to complication, resistant to simplification.
Read the full paper by registering as a participant in the conference and going to the closed area of the Downloads Section of the website. Then click on Robert Stake’s name. This will take you to Robert Stake’s front page. You can then link to his Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of Robert Stake’s work by clicking on the buttons on his front page.
In addition to access to this paper, registration as a participant provides you with access to discussion groups, and to downloads of other papers, case studies and teaching materials from the conference. Register by clicking on the Participants button in the main menu.
For the past few years I have been engaged in conducting metaevaluations of evaluations of various reform programs in both developed and developing countries. Based on those experiences, and without going into detail, it seems to me that evaluation clients would achieve more satisfactory evaluation services if they and their evaluators proceeded about as follows:
(this paper is co-authored with Jason Katz, Robert M. Goodman, Sarah Griffin and Diane Wilson).
Abe Wandersman has also contributedThe FORmative Evaluation Consultation And Systems Technique (FORECAST) is an approach to formative evaluation that offers specific models and tools for program improvement. In this article, we will 1) describe and present updates/refinements to FORECAST, and 2) discuss contributions of the FORECAST approach to formative evaluation practice.
According to Michael Scriven (1991), the proper function of evaluation is making a judgment of a program’s value or effectiveness – i.e., summative evaluation. Under this premise Scriven regarded formative evaluation as a strategy for providing a preview, or “early-warning,” of summative evaluation results. While we accept Scriven’s (1967) notion that formative evaluation is temporally prior to summative evaluation, we consider formative evaluation to be fundamentally a strategy for program implementation and improvement. Formative evaluation produces information during the developmental stages of a program that is used to improve programming to increase the likelihood of positive outcomes (Goodman & Wandersman, 1994).
FORECAST was first used in a five-year project funded by the Center for Substance Abuse Prevention (CSAP) (Goodman & Wandersman, 1994) as an approach to formative evaluation that provides models and tools for program development and improvement (Goodman & Wandersman, 1994). Within this project, community partnerships in South Carolina were established and subsequently tasked with assessment of community needs and implementation of comprehensive alcohol, tobacco, and other drug (ATOD) prevention programming. CSAP partnerships used FORECAST to develop program models and integrated the use of tools for planning, implementing, and evaluating ATOD interventions.
A model of the problem diagrams the perceived causes and effects of the health or social problem of interest, and a model of the solution depicts how the program or intervention will address the problem. The CSAP partnerships developed a model of the problem that outlined risk and protective factors for ATOD abuse (i.e., individual differences, family influences, and community influences) (see Appendix A). Partners subsequently used the model of the problem as a basis for developing a model of the solution depicting a comprehensive strategy for needs assessment, planning, implementation, and evaluation of ATOD prevention programs (see Appendix B).
Markers are short-term steps which afford incremental success in implementing the model of the solution. Measures assess whether markers are accomplished by a certain timeline, and meaning provides quality standards or benchmarks for each of the markers. As with models, the development of markers, measures, and meaning involves mutual decision-making between the evaluator and other project stakeholders (Goodman & Wandersman, 1994). Project markers adopted by CSAP partnerships included the selection of local leaders, mechanisms for committee meetings, and strategies for collecting and monitoring needs assessment data see (Appendix C).. Examples of measures were meeting minutes, telephone and appointment logs, and surveys to assess meeting effectiveness. Meaning included agreed-upon benchmarks for sufficient meeting quality (e.g., minimum attendance level, inclusion of relevant agenda items).
Read the full paper by registering as a participant in the conference and going to the closed area of the Downloads Section of the website. Then click on Abe Wandersman’s name. This will take you to Abe Wandersman’s front page. You can then link to his Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of Abe Wandersman’s work by clicking on the buttons on his front page.
You can also access the paper via Jason Katz’s front page.
In addition to access to this paper, registration as a participant provides you with access to discussion groups, and to downloads of other papers, case studies and teaching materials from the conference. Register by clicking on the Participants button in the main menu.
INTRODUCTION
The idea of theory-based evaluation has sparked considerable interest in recent years (Chen and Rossi, 1987, Lipsey and Pollard, 1989, Scheirer, 1987, Patton, 1989, Chen 1990, Bickman, 1987, 1990, Mark, 1990, Smith, 1990, Weiss, 1995). The root idea is that the beliefs and assumptions underpinning a program intervention are expressed in a sequence of cause and effects, i.e. a “program theory”. For example, a program that trains youth for jobs expects that trainees will learn skills; with skills, they gain in self-confidence; having skills and self-confidence, they aggressively seek jobs; with skills, self-confidence and aggressive search, they are hired. The evaluation seeks data to see how each step in the sequence is in fact borne out. This approach to evaluation offers a way in which evaluation can tell not only how much change has occurred, but also, if the sequence of steps appears as expected, how the change occurred. If the posited sequence breaks down along the way, the evaluation can tell at what point the breakdown occurred.
Evaluation based on a program’s theory of change (TOC) helps evaluators to:
build and test a theory about more general causal mechanisms responsible for the outcome behavior….. generalize the results…. to other settings, populations and treatments….. [and] better predict whether they [the results] will be produced in other contexts (Judd and Kenny, 1981 : 603).
The aim of this paper is to give a practical description of what a “theory” in theory-based evaluation looks like, and a set of recommendations on where to get the information for constructing one. The paper discusses the advantages and limitations of the approach and concludes with suggestions about when it is likely to be warranted.
Read the full paper by registering as a participant in the conference and going to the closed area of the Downloads Section of the website. Then click on Carol Weiss’ name. This will take you to Carol Weiss’ front page. You can then link to her Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of Carol Weiss’ work by clicking on the buttons on her front page.
In addition to access to this paper, registration as a participant provides you with access to discussion groups, and to downloads of other papers, case studies and teaching materials from the conference. Register by clicking on the Participants button in the main menu.
Fetterman’s approach to evaluation (1996; 2001; 2005) is relatively new to the literature. Adding to other approaches, Fetterman argues that empowerment evaluators facilitate evaluation, draw evaluees into the process and train them to ‘drive’ it in part and sense a measure of empowerment in the process. Unusually, values are upfront, its principles declaring the value-stance of an empowerment evaluator. They provide a framework formally shared with evaluees and used, amongst others, to secure buy-in to the process and achieve results. The purpose of this paper is to discuss values underpinning empowerment evaluation and illustrate these in a micro-evaluation of Tapologo, a self-initiated faith-community based HIV/AIDS prevention and treatment programme in rural North West Province. Reflection on the evaluation leads to the conclusion that values driving the approach may make it more useable in this country than initially believed.
Value
The term ‘value’ is used in two ways in this paper. It refers to gauging, estimating or attributing worth to a ‘thing’, a program for example, to finding how well a program is [or is not] doing. Value in this sense defines evaluation as field: “evaluation is the determination of a thing’s value” (Worthen and Sanders, 1987: 22; Scriven in Davidson, 2006: 1). In this widely accepted definition, value refers to the formal determination of a thing’s worth, to establishing quality, determining effectiveness or worth. It entails establishing standards for judging quality, collecting relevant data and applying standards to determine quality. Definitions such as this generally assume that value is attributed by an outsider, applying developed instruments, who also make recommendations for evaluees to improve their program.
Attributing value in this sense differs from evaluees attributing value to their own program, as argued by Fetterman (1996; 2005): “empowerment evaluation is the use of evaluation concepts, techniques and findings to foster improvement and self-determination” (1996: 10). They argue that, with training, value is self-attributed by those who know a program best, namely those inside it, to increase the probability of program success. Here value is declared by evaluees. Facilitated by an evaluator, they declare both what is working well and what is not working well, as it is in their best interest to do so to develop quality for their own good and the public good, and to uncover what may be hidden or missed by an outsider in traditional approaches to evaluation. Fetterman (2005: 12) observes that “People are demanding much more of evaluation and are becoming intolerant of the limited role of the outside expert who has no knowledge or vested interest in their community” (2005: 12). In their view, attributing value is an enabling and emancipatory concept. It helps individuals and communities to have control over decisions that affect them, and to take charge of their environment. It has psychological power, helps them help themselves and can be a liberating and emancipatory experience.
Value
Value, in the sense of values, secondly, emphasises substance or meaning of the term ‘value’ and desirability. House and Howe (1999: xvi, 5 ) consider values one end of the fact-values continuum, and facts and values being intertwined and blended. Facts, such as the law of gravity, mean that objects fall in a vacuum at the same speed. At the other end of the continuum, a value such as inclusion suggests desirability, as in a code prioritizing one meaning over others, which formalizes thinking and guides action. Facts are established in scientific research through testing hypotheses to find new relationships between phenomena yielding results we have confidence in. In contrast, inclusion derives meaning from its codification, as in a code of ethics or a policy, for example to include mildly challenged learners in mainstream schools [White Paper 6], where deliberation clarifies, argumentation grows and consensual agreement construes what is entailed in the term. House and Howe argue that facts and values are intertwined or blended through deliberation between a wide range of people dialoguing meaning. Value in this sense is not concerned with attributing worth. Nor are values arbitrary. Value here refers rather to “the property or aggregate properties of a thing by which it is rendered useful or desirable” (wsmith@wordsmith.org, 14/7/2009). Properties or meanings attributed to the term ‘values’ and why a value is useful or desirable, entails the second and primary use of this term in this paper. Values made explicit in empowerment evaluation principles, clarify the value orientation of an evaluator and underlie its procedure and facets, and are considered next.
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Ray Basson’s paper next to his name. This will take you to Ray Basson’s front page. You can then link to Ray Basson’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
Systematic policy, programme and project evaluations are relatively new tools in the arsenal of the public manager in developing countries, although it has been an established tool for the improvement of policy outcomes and impacts for a number of decades now by governments in more developed societies. The increasing significance of the evidence based paradigm for public policy making and implementation in developing countries as a result of the increasing capacity to process huge data sets in the information era, and the continual development of new evaluation approaches informing the evaluation of public policies and programmes, necessitates a rethink of what exactly the purpose of an evaluation is, and how to fine-tune its use to maximise its impact on public policy processes and outcomes.
What we want to achieve with an evaluation will determine what we have to
do and how we have to do it. It is crucial for the public manager to understand how a
systematic evaluation can achieve different strategic goals, and what a decision about what the main strategic purpose of the evaluation is, implies for the evaluation design and methodology.
This paper starts by summarising the influences of the policy and social sciences on the evaluation profession. It then provides a brief overview of alternative evaluation approaches, before suggesting guidelines how to select the most appropriate
evaluation design to achieve stated strategic goals. The paper argues that the main
variables that influence the most appropriate evaluation design are the following:
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Fanie Cloete’s paper next to his name. This will take you to Fanie Cloete’s front page. You can then link to Fanie Cloete’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
Gender equality and women’s empowerment remains among the world’s unsolved development problems. Yet this is not for want of trying. For over three decades assertive programming in human rights, social justice and in particular women's rights have generated and expanded the literature as well as instruments, created a number of institutions global and local and, above all popularized the notion and language of (universal and attainable) human rights. The investments have been massive and in many instances the gains have been, critical and significant. But the outcome has cannot be said to be equivalent to the investments.
The UN Economic Commission for Africa evaluation report of the ten-year implementation period for the Beijing Platform for Action, one of the most ambitious declarations in favor of women's rights, reports that African women and girls remain victims of grave violations of their human rights-economic, social, cultural, political as well as all the others. Thus despite being on the international development agenda as a programmatic commitment for over 35 years and with a good number of multilateral, bilateral and private development institutions in addition to many NGOs pursuing the cause, gender equality and women's advancement or empowerment has still not been fully achieved in most parts of the world as in Africa. This is despite the multiplicity of signed, ratified and even domesticated legal instruments designed to protect (and assure) these rights. Much of the action in favour of gender equality and women’s empowerment has been supported by donor funding with many countries especially in Africa appearing to be unwilling followers in this path.
Donor and partner country behaviour was expected to change radically with the introduction of the 2005 Paris Declaration on Aid effectiveness. The Paris Declaration has been hailed as a having consolidated global commitments for implementing changes in the planning, coordination, monitoring and evaluation of aid. This new aid agenda is seen as attempting to perform two related functions of increasing the volume and quality of aid, and improving the effectiveness of aid. The PD has thus ignited a rising activism in defence of Aid Effectiveness. Since AE cannot be divorced from overall development effectiveness and development effectiveness cannot be guaranteed without real gender equality and the full attainment of women’s human rights, it is imperative that no effort is spared to integrate GE in the implementation of the PD.
In this paper, we take a look at the Paris Declaration (PD) in relation to Women’s rights and gender equality. We reiterate the now familiar verdict that the PD and the Accra Agenda for Action (AAA) do not fare too well fare in relation to their gendered credentials as reflected in the indicators and we recount the Africa Gender Evaluators Network engagement with this powerful aid instrument and movement. The paper is divided into the following four sections;- background, the invitation, the project, outreach-upscaling-outscaling & conclusion.
Background
The Paris Declaration1
The Paris Declaration is a statement, adopted at a High Level Ministerial Forum (HLF) on March 2 2005. The HLF, organized by the Development Assistance Committee (DAC) of the OECD, was attended by Aid Ministers from 22 OECD donor countries, representatives of international organizations, recipient developing countries and 14 civil society organizations. Over 100 partner governments, bilateral and multilateral donor agencies, regional development banks, and international agencies—endorsed the Paris Declaration on Aid Effectiveness, committing thereby to actions that would promote effective use of aid funds.
The declaration consists of three sections namely; the statement of resolve, partnership commitments, and indicators of progress. Much of the discussions and contentions to date have revolved around the 5 principles that underlie the partnership commitments: - ownership, alignment, harmonisation, managing for results, and mutual accountability. The statement of resolve highlights the raison d’etre of the declaration, as the scaling up of aid, and identifies the management and implementation processes of the new approach, including adaptability to differing country situations, the targets and their timelines. It also suggests a monitoring and evaluation schema.
The 50 partnership commitments in respect of the 5 principles of ownership, alignment, harmonisation, managing for results, and mutual accountability, are partner specific. The fact that the numbers of commitments vary for each principle and for each partner is itself pregnant with interpretations.
The Indicators of Progress which constitute the final section of the PD consist of a table of the 12 indicators and their corresponding targets to be achieved by the year 2010. The difference in the number of indicators for each of the principles with Ownership, Managing for results and Mutual accountability having one indicator each while Harmonisation has 2 indicators, and Alignment has nine indicators is indicative if their relative importance. It has thus been suggested, on account of the number of indicators for each of the five principles that he PD is principally an instrument for pursuing structural reforms of aid delivery mechanisms with a distant hope for improving aid efficiency. In the short term, it has limited positive impact on development effectiveness or aid effectiveness in general and gender equality in particular.
Two monitoring and evaluation exercises of the implementation of the PD in partner countries have been completed since the PD was endorsed; in 2006 and 2008. The third round of Monitoring the Paris Declaration will begin in October 2010 and will be completed by March 2011. At the global level an independent evaluation exercise of the outputs and outcomes of the PD is currently being undertaken supported by the OECD.
Despite the limitations, the PD is currently the lighthouse document for Aid Effectiveness and had been described by the OECD (2007) as;
‘an ambitious attempt to increase the impact of aid on development by promoting more mature partnerships between donors and partner countries. It also seeks to enhance partner countries’ ability to manage all development resources more effectively; and enable their citizens, and parliaments, to hold governments accountable on its use. As well as committing all parties to the Declaration to a clearly specified set of actions and behavioural changes, it also calls for periodic monitoring at the country level, so that the governments of developing countries and their external partners are increasingly accountable to each other for the progress being made’(OECD, 2007,Chapter 3).
While many of the key reviews of the PD (DCD/DAC, 6-2006, 7; DCD/DAC, 2007; Fleming S. etc al, 2007; DCD/DAC, 2-2007) suggest that the narrow focus of the PD others see the opportunities offered by and in the gaps as valuable entry points for the work of engendering it and its outcomes. Among the strongest reasons for engaging and interrogating the PD from a gender equality perspective is the complete absence of gendered pictures and references in the results of the first monitoring exercise conducted in 2006, and in the PD indicators themselves (DCD/DAC, 2-2007).
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Florence Etta’s paper next to her name. This will take you to Florence Etta’s front page. You can then link to Florence Etta’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
Focuses of this Paper
This paper focuses on the model used to validate a test which can be used for screening purposes to identify children with learning difficulties. The instrument uses error profiles to identify children requiring special treatment programmes, based on the preponderance of particular types of phonic errors.
Our evaluation model has been a mixed one linking traditional psychometric evaluations with case studies of the instrument in use. The initial stages in the validation process involved content and construct validations conducted longitudinally with the instrument, as well as two large-scale evaluations which compared the types of phonic errors made by learners in different grades at primary school. These produced evidence of distinctive patterns of error made by children in mainstream and remedial classrooms, indicating that the profiling system could be used for predictive and discriminatory purposes.
Given findings indicating the internal and external validity of the instrument, a number of evaluations have also been conducted which focus on the validity of the instrument in use. These studies have focused on the pragmatics of using the test for screening purposes as well as for planning and monitoring instruction in classrooms and clinics. This evidence has been produced through seventeen case studies of the instrument in use, which have then been integrated through meta-evaluation based on aggregative case survey analysis.
The results indicate that the instrument is reliable, and both content and construct valid when measured against other commonly available scholastic tests of reading, writing and spelling. In addition the test yields detailed diagnostic information on the types of phonic errors made by children, and has good potential as a group test which can be used as a screening instrument to identify children with dyslexia, tapping error patterns which are predictors of learning difficulties. The type of information yielded by the instrument is also useful for planning remedial instruction which targets the specific types of errors made by individual children in the classroom, for clinical use in planning individual instructional programmes, and for monitoring and evaluation purposes.
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Dina Grasko’s paper next to her name. This will take you to Dina Grasko’s front page. You can then link to Dina Grasko’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
Evaluation is an emerging field in social science in the past two decades; as a result, this discipline does not have the rich psychometric history that psychology enjoys. A lack of strong psychometric history in the field necessitates the development of reliable and valid instruments. However, developing and testing the psychometric properties of an instrument can be time consuming, and evaluators may choose simply to develop instruments that make intuitive sense without a careful examination of their psychometric properties.
Exploratory factor analysis (EFA) is especially useful in such a scenario. EFA is a statistical procedure that is used to identify the interrelationships among a large set of observed items. Through data reduction, EFA groups a certain number of observed items into a common factor (Nunnally & Bernstein, 1994). A common factor is also referred to as a latent variable or a construct, as it is not measurable directly (DeVellis, 1991).
Since most programs, or at least part of a program, aim to change a psychological construct of their participants, a valid survey with multiple items is warranted. For example, evaluators cannot assess parent-child relationship directly because it is a latent variable, but they can examine it through indictors (i.e., items on a survey) of the relationships.
Exploratory factor analysis can provide validity evidence through the examination of internal structure of newly developed instruments. In this article, we first present an in-process case study that utilized EFA. In the methods and results sections, we argue the appropriate use of exploratory factor analysis in the initial testing phrase of a newly developed instrument, namely the many decisions, such as rotation and extraction methods, that evaluators must make while conducting an EFA. Finally, we discuss the implications and the use of EFA in the present as well as other evaluation studies.
Despite its popularity and usefulness in educational and psychological research, exploratory factor analysis (EFA) may not be as frequently used in evaluation studies. We argue that EFA is especially important in such studies when study-specific instruments are employed. EFA can begin to provide evidence of measurement validity through the examination of internal structure of the measurements. Such evidence can increase the utility of the instrument in evaluation. However, too often EFA is conducted using software default settings (e.g., principal component analysis with varimax rotation) which may not be most appropriate for the
We use an in-process case study as an example to provide guidelines for appropriate uses of EFA.
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Laura Hill’s paper next to her name. This will take you to Laura Hill’s front page. You can then link to Laura Hill’s Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
Governments and other development actors are constantly taking decisions on policies, programs, and projects. In doing so they may weigh opportunities and costs of acting or not acting, starting, continuing, revising, or ending any of them. Some of this happens based on sound evidence or information, some on weak, absent, or faulty evidence, some on opinion, and much on a range of other factors not related to evidence at all.
Much has been written about building evaluation capacity and the need for improving the quality of evaluation to address gaps in evidence based policy and programming. There are calls for more training; funds are being established to support evaluations; and a range of organizations are trying to address capacity gaps. In parallel, there is an increasing call to shift evaluation use from serving donor needs to serving the needs of the countries wherein programmes are undertaken (Segone, 2006, Segone 2008, Carden 2010). Such a transition will require improvements in quality, supportive institutions and structures, and significantly strengthened in‐country evaluation capacity.
It would be naïve and simplistic to assume that strengthening the supply of, and the demand for, evaluation would make decision making transparent, technocratic, rational and linear. As Boyle, Lemaire and Rist (1999) note evaluation findings may be ‘drowned out’ by other aspects in the political context, and, ‘often for good reason’ (pg. 5). The idea of evaluation field building developed in this paper, encompasses both the need for strengthening the quality and practice of evaluation, and, broadening the space and platforms for using evaluation knowledge in decision making. It calls for a more deeply contextualized understanding of what quality, rigor, and use should entail. In doing so it embraces, rather than ignores, the complexity of decision making and implementation systems.
Making evaluation matter in the cycle of policy, programming, research, and evaluation that must constitute sound, evidence‐based, aspirational, development is not simple. ‘Aspirational’ is used here intentionally, to reflect a view of evaluation that goes beyond measuring performance or management oriented functions; evaluation, here, embodies a recognition that development is not adequately meeting the needs of most citizens. This recognition brings with it a need for shifts, critiques, and democratization processes to fundamentally change the systems and institutions of decision making, and the ways that both connect with citizens.
Evaluation can be used to reinforce existing and dominant development systems, discourses, and approaches. It can also be used to challenge them. Evaluation, here, is conceptualized as being part of dynamic, critical, and change oriented processes. Taking the case of South Asia, this paper explores and develops a framework for evaluation field building. The paper suggests elements a robust evaluation field should include and maps these against the current scenario in South Asia. The paper then proposes strategies for field building to support and strengthen this evolution (indeed revolution) in evaluation practice and use.
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Katherine Hay’s paper next to her name. This will take you to Katherine Hay’s front page. You can then link to Katherine Hay’s Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
Monitoring educational quality in South Africa is a relatively recent development (Howie 2008) and the field is still in its infancy methodologically locally on both a systemic level and a school level. The utilisation of complex modelling techniques in monitoring educational quality has been primarily restricted to more developed environments internationally for some time, with a few exceptions locally in the past decade (Howie, 2002; Mohandas 1999, Scherman, 2007). National and international assessments have been used to monitor educational quality on system-level in a variety of ways (Howie & Plomp, 2006) and multilevel analysis has been used effectively internationally for exploring and understanding the systemic contextual factors associated with student performance. School effectiveness as a field has a long tradition in developed countries and has been associated with monitoring educational quality on school level (Scheerens, 2001) and multilevel analysis has been used most effectively in school effectiveness research in developed contexts. Although some literature exists for a developing world context (see for example Howie, 2002; Fuller & Clark, 1994; Passos, 2010; Scheerens, 2001; Scherman, 2007, van Staden, 2010; Wyatt, 1996) a dearth of studies still exist, especially in the context of Africa and South Africa in particular. These studies and other literature suggests that contextual factors should be included when exploring pupil achievement (a primary indicator of educational quality) as these factors do influence achievement and provides the context within which performance should be interpreted. While school effectiveness models are not as volatile as critics would have it, they are still not as firmly established as some enthusiasts proclaim (Wyatt, 1996). Studies of school effectiveness in developing countries should make use of the advanced statistical analysis available, i.e. multilevel analysis, an element that has been missing in school effectiveness research generated in developing countries. Likewise, the baseline measures of outcomes should be encouraged if school effects are to be inferred (Riddell, 1997).
In South Africa, the inclusion of contextual factors are particularly important for South Africa, given the apartheid era and disparities in schooling in terms of analysing and understanding pupils’ performance. The poor performance of South African pupils in international assessments underscores the importance of contextual data (Howie et al, 2009). South Africa has not performed well in international comparative assessments like the TIMSS studies in 2003, 1999, and 1995 (Howie 1997, 2001, HSRC, 2006), PIRLS 2006 (Howie et al, 2009) as well as the SACMEQ study (Moloi & Strauss, 2005) where pupils performed well below the international averages and below those of many countries. Likewise the pupils performed well below expectation in the Systemic Evaluation in Grades 3 and 6. The disappointing results could be due to pupils being ill prepared (for a number of reasons) in terms of the content areas and therefore being unable to achieve the expected assessment standards (National Department of Education, 2005).
Globalisation is an important consideration for educational quality. The issues of educational quality can be linked to globalisation. Education and globalisation are two very powerful forces at play in the international landscape (Mortimore, 2001). Where education has the power to change lives, globalisation has the power to enable education to reach all people so that they may have the opportunities to better themselves as well as become viable to compete on the global circuit. Improving the quality of education systems worldwide would enable nations to secure economic competitiveness in a globalised economy (Kellaghan & Greaney, 2004), in a process where nations are drawn together to form a single entity (Mortimore, 2001). In this view the world is becoming increasingly mobile and it
“is important to stress quality in the context of our mobile world: the diversification of societies, largely as a result of migration, urbanization and cultural change, combined with increased sensitivity to the numerous different aspects of individual and group identity, places new burdens upon education systems. These threaten to undermine education quality if they are not dealt with adequately, but at the same time they encourage a dynamic of research, experimentation and exchange of experience that can mightily advance the agenda of quality education (Pigozzi (2006, p. 3)”.
Educational quality can be thought of from a variety of perspectives such as the following:
For purposes of this paper, educational quality is used in the meaning of Van der Werf et al, 1999. The aim of this paper is to discuss the application and implications of using complex statistical techniques such as a multilevel analysis for the purpose of monitoring educational quality in a developing context such as South Africa. Two examples of modelling are provided on two levels, namely monitoring on school-level, using a small-scale study and secondly monitoring on system-level based on large-scale national data from an international comparative study.
The paper is structured in the following way: firstly a short description is given of two studies that are examples of monitoring educational quality in South Africa and utilised complex statistical models to analyse the data. This is followed by a description of the conceptual framework used in monitoring educational quality at school and system-level as well as the frameworks used in the two examples. Thereafter the research methods for the two studies are briefly summarised. The results of the two studies are given separately before the conclusions and reflections conclude the paper.
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Sarah Howie’s paper next to her name. This will take you to Sarah Howie’s front page. You can then link to Sarah Howie’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
MDGs, EFA and the South African context
According to the Bill of Rights included in the Constitution of the Republic of South Africa, 1996 (Act 108 of 1996), everyone has the right to a basic education, including adult education; and further education, which the state, through reasonable measures, must make progressively available and accessible. Anticipating the formal adoption of the Constitution, new the Ministry of Education, established after the historic elections in May 1994, had already created a single national educational system, to be managed by the nine provincial legislatures. Universities and Technikons fell under the direct responsibility of the national government. The South African Act of 1996 (Act 84 of 1996) provided for two categories of schools, public and independent, for learners between five and 16 years of age or those reaching the ninth grade, whichever occurred first. In 1995, the South Africa Qualifications Authority (SAQA) was established (Act 58 of 1995) with a range of responsibilities including overseeing the implementation of the National Qualifications Framework (NQF). Curriculum 2005, with its emphasis on outcomes-based education, was introduced at Grade One level in 1998 (Shalem, Allais, and Steinberg, 2004).
The goals of Education for All (EFA)
A decade before the declaration of the MDGs, a global commitment to basic education was made in 1990 at the World Conference on Education for All in Jomtien, Thailand. In 2000, the World Education Forum meeting in Dakar, Senegal re-affirmed these commitments and made a new statement of the goals to promote Education for All (EFA) (World Education Forum, 2000).
Figure 2: The six EFA goals agreed at the World Economic Forum in 2000
Whereas only two of the eight MDG goals related directly to education, all six of the EFA goals do so. They also refer to early childhood education; to adult literacy, albeit only in relation to young people and adults up to age of twenty-four; and to life skills.
This paper reviews the MDG reporting in South Africa paying attention to current measures that are used and proposed alternative and additional measures as appropriate. Much of the analysis is an attempt to shift the emphasis away from what Zafar (2007) refers to as the ‘coarse grain indicators’ used in the MDG approach. Globally education reporting by UIS and as used in the GMR report tends to take a more comprehensive and holistic approach to measuring progress compared to that used in the MDG reporting approach. The conclusion to the report makes recommendations about key issues which should be considered and discussed in order to move beyond the ‘coarse grain’ approach.
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Anil Kanjee’s paper next to her name. This will take you to Anil Kanjee’s front page. You can then link to Anil Kanjee’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
Culturally-sensitive participatory program evaluation (referred to from this point as CSPPE) is a somewhat elusive concept. While some treat is as a socially-oriented research approach, others perceive it as a full fledged research theory or meta-theory (Caldwell et al., 2005; Donaldson & Scriven, 2003; Letiecq & Bailey, 2004; Minkler, Glover-Blackwell, Thompson & Tamir, 2003). Some even present it as reduced to certain methodologies, which are not necessarily dependent on a certain theoretical or philosophical framework (Patton, 2002). These different viewpoints are the result of various preferences and attitudes towards research and evaluation, which this workshop is not designed to delineate. For our learning purpose, CSPPE will be addressed as a group of methodologies that share the following basic work assumptions:
As such, cultural-sensitivity in participatory program evaluation is presumed to enable:
Theoretically speaking, CSPPE can be said to represent what happens when four major frameworks conjoin: Culturally-sensitive research, Culturally-sensitive evaluation, Participatory program evaluation, and Generalized evaluation practices and guidelines. Each of these can be widely discussed. However, for the purpose of gaining a basic understanding of CSPPE, the following elements involved in each of these concepts provide a useful foundation.
Culturally-sensitive research
Generally, cultural-sensitivity in research can be defined as conducting studies with a raised consciousness of the impact cultures have on a) The people and/or phenomena that is being studied, b) The research process and c) the researcher/evaluator (Henderson, Sampselle & Oakley, 1992). Reasons for utilizing cultural-sensitivity in research are usually divided into two groups. First, are reasons pertaining to research quality and rigor. Certain, if not most, research paradigms contend that paying attention to cultural issues influencing research is of great importance when planning and executing "good" research. In this vein, choosing to ignore such issues is seen as detrimentally affecting studies' ability to reach accurate and reliable "truths" (Hopson, 2003). Positivists, or researchers that see themselves as primarily identified with quantitative research methods, may view cultural-sensitivity as means for choosing the right research variables, control variables and research measures. A culturally-sensitive approach is also closely related to the ability to generalize research findings across countries and populations. Researchers that are more affiliated with constructivist paradigms or qualitative research methods may view cultural-sensitivity as an expression of their perception of human phenomena as being dynamic and influenced by several social and individual interactions (Patton, 2002). A second set of reasons for using a culturally-sensitive approach to research has to do with ideological motivations and attitudes, which mostly pertain to using research as means of promoting empowerment and equality for excluded populations (Grinnell & Unrau, 2008).
Researchers practicing cultural-sensitivity in various stages of their studies are commonly expected to (Grinnell & Unrau, 2008; Patton, 2002):
Culturally-sensitive evaluation
"Research" and "evaluation", although sharing several common grounds, are not
synonymous terms or concepts (For a detailed distinction between these two, see Mertens, 2010).
CSPPE is inspired and influenced by both their approaches towards cultural-sensitivity. For this reason,
it is worth dedicating part of our discussion not only to this approach as it manifests itself in research,
but in program evaluation as well. Culturally-sensitive evaluation differs from other types of culturally-sensitive research
mainly in the social role it attributes to its application of cultural-sensitivity.
This is well-expressed in Hopson's (2003) guidelines for culturally-sensitive program evaluators. It is of value to make the effort to capture the spirited social motivation that these pointers reflect:
Having said this, "good" culturally-sensitive program evaluations are therefore expected to include the following (Hopson, 2003):
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Lia Levin’s paper next to her name. This will take you to Lia Levin’s front page. You can then link to Lia Levin’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
Developing country public sector capacity has proven difficult. Despite the financial resources provided and international commitments made, reviews have acknowledged results are below expectations (World Bank, 1998; World Bank, 2005; Commission for Africa, 2005). Jones and Kettl (2003) argue the inability to identify problems affects the design and implementation of reforms is the key issue. While donors have historically thrown technical solutions at political and social problems, Reeler (2007) advocates for good theories of change to assist those involved in development to build their thinking and understanding of complex social processes. Similarly, in a recent review of the quality of the UK Department for International Development’s (DfID) evaluation, Perrin (2009: 28) argues that:
there is a need for the development and use of more appropriate theory-of-change models that articulate and begin to encompass the complex and diffuse nature of the linkages between what DFID does and wider impact.
This paper proposes that surfacing theories of change, that is the collection of assumptions, principles and propositions to explain the relationship between the program’s actions and expected outcomes, are a possible avenue to aid better understanding of development contexts, change processes and the use of capacity development strategies. It is based on the premise that the act of articulating helps assumptions and biases to be brought into the open for discussion, clarification and testing in different contexts to increase learning. As a consequence, the capacity development strategies used may be more effective in achieving desired outcomes.
The paper presents a case study which sought to surface theories of change underpinning strategic planning, monitoring and evaluation and client service charters being introduced to the Government of Tanzania. Partial, missing and confused theories of change are presented along the implications for the public sector capacity development.
The paper presents a case study which sought to surface theories of change underpinning strategic planning, monitoring and evaluation and client service charters being introduced to the Government of Tanzania. Partial, missing and confused theories of change are presented along the implications for the public sector capacity development.
Several documents relating to the Public Sector Reform Program in Tanzania were analysed to identify components (context, mechanism and outcomes) of theories of change as they pertained to strategic planning, monitoring and evaluation and client service charters. The document analysis sought to answer the following questions:
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Donna Loveridge’s paper next to her name. This will take you to Donna Loveridge’s front page. You can then link to Donna Loveridge’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
The South African Radio Learning Programme worked in South Africa over the period between 1992 and the end of 2009. It was supported over this period by international donor funding to the Open Learning Systems Education Trust (OLSET), a South African NGO.
The aim of the programme was to offer direct support to the South African National and Provincial Departments of Education in improving the quality of teaching in primary schools. This was done by targeting the development of English language competencies in the junior primary phase, through the medium of interactive radio.
The type of interactive radio programme developed in South Africa differed from interactive radio programmes developed elsewhere (USAID, 1990; Leigh, Naidoo & Ramafoko, 1995; Potter & Naidoo, 2006), in providing a curriculum providing support to teachers at school and classroom-levels, and a curriculum of classroom-based activities for learners. Teachers were provided with materials which included a teacher’s manual and printed classroom posters and alphabet friezes to support a series of audio lessons, which are broadcast daily by the South African Broadcasting Corporation (the SABC).
OLSET also provided direct support to schools in the form of in-service teacher training. This involved ongoing personal contact between regional coordinators and the teachers and principals in participating schools, provision of in-service training workshops and assistance in the development of teacher support groups.
Focuses of this Paper
This paper focuses on the development of a set of procedures for undertaking classroom observation in the programme, and linking these to the broader evaluative procedures supporting OLSET’s work. These methods were developed as part of an internship programme involving Yale University, the University of the Witwatersrand, Johannesburg and the Open Learning Systems Education Trust.
Our aim was to use classroom-based observation as a basis for formative evaluation directed at improving teaching. The methods would focus on those teaching transactions observed and recorded in the classroom by the regional coordinators in the programme. These would then be used as a basis for discussion and reflection on teaching practices.
The procedures would involve use of classroom observation for formative evaluation purposes, to improve classroom teaching. The data would also be used for summative evaluation purposes, to provide evidence about whether the programme was achieving results in relation to its aims and envisaged outcomes.
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Gordon Naidoo’s paper next to his name. This will take you to Gordon Naidoo’s front page. You can then link to Gordon Naidoo’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
While feminist evaluation and gender approaches are used in Sub-Saharan Africa they are often confused. This lack of distinction may be a result of the limited examples of feminist evaluation in academic journals, books, and published papers, compared with the many examples of gender approaches to evaluation. Denise Seigart suggests that feminist evaluation studies may be limited either because many practitioners are hesitant to label their approach as feminist or because this approach is relatively new to the evaluation field (Seigart, 2005). In addition, discussions that focus on evaluation methods often do not include feminist evaluation, and when gender approaches are taught they rarely focus on feminist evaluation. In 2008 Suzanne Hodgkin (2008) wrote that although an increasing number of journal articles and other published evaluation studies examine the use of mixed-method research, considerably fewer demonstrate the use of mixed methods in feminist evaluation or research.
In contrast, published examples of gender approaches in evaluation are numerous. With the term “gender” as a search word, the World Bank’s evaluation Web site alone identifies 33 papers, journal articles, and books on gender evaluation and gender approaches published after 2001 (World Bank, 2008). The Web site of the United Nations Development Fund for Women (commonly known as UNIFEM) posts 111 research and/or evaluation books and articles published after 2000 that focus on women or women-focused interventions. Of the 111 publications, 53 carry the term “gender” in their title; none are labeled as feminist (United Nations Development Fund for Women, n.d.).
Purpose of this Paper
This paper has two purposes. First, the paper seeks to provide clarity between feminist evaluation and gender approaches. Second, the paper intends to demonstrate how this clarity led to a useful evaluation being conducted in Southern Africa. A comparative framework will be used to clarify the divergence between feminist and gender approaches and identify where they overlap. A sub-Saharan African based case study will demonstrate how these approaches can lead to a useful evaluation.
Situating the Author
As a feminist evaluator, I acknowledge that there are multiple realities and even numerous definitions of feminism, and I use the term “feminist” with the following understanding: “A common belief that guides feminism is that gender bias exists systematically and is manifest in the major institutions in society … Feminism examines the intersection of gender, race, class, and sexuality in the context of power” (Mertens, 2005, p. 154).
Living in South Africa I have had numerous opportunities to apply both feminist evaluation and gender approaches in Africa. In the past few years, most clients rejected the proposed use of feminist evaluation yet they readily accepted a gender approach. One common response was that feminist evaluation was a ‘western concept’. These same clients readily accepted other evaluation approaches such as Utilization Focused Evaluation or Empowerment Evaluation, which ironically were developed in the United States. Yet other clients bristled at the term ‘feminist’ yet when I was permitted time to explain an approach based on feminist evaluation tenets, they agreed that the approach was useful.
Initial research confirmed that the perceptions associated with feminist and gender approaches to evaluation contrast significantly (Longwe, 1995; Ogundipe-Leslie, 1994). Longwe (1995) provides an example of how these two approaches are viewed within the international development context. She contends that most development agencies work with patriarchal host governments in the developing world context. It stands to reason that development agencies, dependent on this relationship, would not upset funding sources with “outrageous” feminist thought or theory and therefore would use gender approaches, which are less offensive (Longwe, 1995).
These findings encouraged further research on the differences between feminist evaluation and gender approaches, as well as the usefulness of their application. As a first step the terms “gender” and “sex” are clarified.
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Donna Podems’ paper next to her name. This will take you to Donna Podems’ front page. You can then link to Donna Podems’ Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
This paper presents is an initial mapping of where human rights’ thinking has been integrated in programme and policy evaluations. It shows that human rights have been well established as a values framework for evaluation. There is a multitude of tools and criteria, some consistent trends emerging in how human rights have been integrated into evaluation and areas where further investigation is required. This finding stands in contrast to a number of books and articles discussing the application of human rights in a cross-disciplinary way./
`
Within the first few paragraphs of these books there is often a sentence that resembles the following:
Though human rights and social sciences work is mutually reinforcing, they have
largely been working in parallel (Sen 1999: 62; Freeman 2002: 178; Donnelly 2003: 2;Jonsson 2005: 47; Marks 2005: 23; Robinson 2005: 31; Gasper 2007: 1).
Ironically, each of these authors then does the service of reinforcing the links between human rights and a number of disciplines. Such an analysis needs to be refreshed.
While some evaluation practitioners have forged ahead in using human rights as a framework for describing merit,
worth or value, there is no map of these efforts for the discipline of evaluation to draw on, learn from and advance or discard.
As this has been a fresh attempt to scan and map some emerging areas of interest the claims are humble, the scope limited, while the frameworks used are tested rather than being claimed as good practice. This paper, first, discusses in gaps in the literature; second, a guiding framework is developed for the analysis of evaluations informed by human rights approaches; third, a summary of specific evaluation is presented; finally, some initial conclusions, questions and areas for deeper review are raised.
Discussion of Gaps in Literature
The paucity of reviews of human rights within the discipline of evaluation makes
you think there is only an island of practice; rather there is a continent to traverse.
This paper responds to gaps in the literature and mainstream practice of evaluation
and human rights where there is only emerging recognition of their potential
importance to each other. As House (2005: 1072) has stated concisely, “values lie
at the heart of evaluation”. Meanwhile, the concept of human rights holds the
position of having an “international normative universality” (Donnelly 2003: 1
with original italics) – it is the guiding values system to regulate the conduct
affairs between people globally. Put another way, values are core to evaluation,
while human rights are the ideal global values framework, an ‘ought’ that would
need to be evaluated. From this interaction at the level of values a complex set of
consequences unfold into practice – specialised language, concepts, and taxonomy
emerge. Being specialised means that in many instances human rights is new to
evaluators, while evaluation new for human rights practitioners. Curiously a
middle ground has been found in development practice, where the interplay
between the two is especially important.
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Stephen Porter’s paper next to his name. This will take you to Stephen Porter’s front page. You can then link to Stephen Porter’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
Marks, grades and ratings are commonly used by class teachers as indicators of the progress made by children at school. The evaluations reported in this paper have focused on a different type of indicator, namely on the phonic error patterns made by mainstream and remedial primary school children.
This paper focuses on the model used to validate a test which can be used for screening purposes to identify children with learning difficulties. The instrument uses error profiles to identify children requiring special treatment programmes, based on the preponderance of particular types of phonic errors.
Our evaluation model has been a mixed one linking traditional psychometric evaluations with case studies of the instrument in use. The initial stages in the validation process involved content and construct validations conducted longitudinally with the instrument, as well as two large-scale evaluations which compared the types of phonic errors made by learners in different grades at primary school. These produced evidence of distinctive patterns of error made by children in mainstream and remedial classrooms, indicating that the profiling system could be used for predictive and discriminatory purposes.
Given findings indicating the internal and external validity of the instrument, a number of evaluations have also been conducted which focus on the validity of the instrument in use. These studies have focused on the pragmatics of using the test for screening purposes as well as for planning and monitoring instruction in classrooms and clinics. This evidence has been produced through seventeen case studies of the instrument in use, which have then been integrated through meta-evaluation based on aggregative case survey analysis.
The results indicate that the instrument is reliable, and both content and construct valid when measured against other commonly available scholastic tests of reading, writing and spelling. In addition the test yields detailed diagnostic information on the types of phonic errors made by children, and has good potential as a group test which can be used as a screening instrument to identify children with dyslexia, tapping error patterns which are predictors of learning difficulties. The type of information yielded by the instrument is also useful for planning remedial instruction which targets the specific types of errors made by individual children in the classroom, for clinical use in planning individual instructional programmes, and for monitoring and evaluation purposes.
Based on longitudinal evaluation of the instrument conceptually and in use, we suggest the value of an instrument which can pinpoint the types of phonic errors made by children in the classroom. The profile of errors can be used for purposes of planning individualized instruction. What we have also found is that phonic error profiles produce two types of evidence for a teacher or school administrator which class marks, grades and ratings cannot provide.
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Charles Potter’s paper next to his name. This will take you to Charles Potter’s front page. You can then link to Charles Potter’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
Systematic policy, programme and project evaluations are relatively new tools in the arsenal of the public manager in developing countries, although it has been an established tool for the improvement of policy outcomes and impacts for a number of decades now by governments in more developed societies. The increasing significance of the evidence based paradigm for public policy making and implementation in developing countries as a result of the increasing capacity to process huge data sets in the information era, and the continual development of new evaluation approaches informing the evaluation of public policies and programmes, necessitates a rethink of what exactly the purpose of an evaluation is, and how to fine-tune its use to maximise its impact on public policy processes and outcomes.
What we want to achieve with an evaluation will determine what we have to
do and how we have to do it. It is crucial for the public manager to understand how a
systematic evaluation can achieve different strategic goals, and what a decision about what the main strategic purpose of the evaluation is, implies for the evaluation design and methodology.
This paper starts by summarising the influences of the policy and social sciences on the evaluation profession. It then provides a brief overview of alternative evaluation approaches, before suggesting guidelines how to select the most appropriate
evaluation design to achieve stated strategic goals. The paper argues that the main
variables that influence the most appropriate evaluation design are the following:
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Babette Rabie’s paper next to her name. This will take you to Babette Rabie’s front page. You can then link to Babette Rabie’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
The longitudinal evaluation reported in this paper focuses on the error patterns made by mainstream and remedial primary school children. It evaluates the reliability, validity and usefulness of an instrument we have developed for analyzing and profiling the types of written spelling errors made by children in the classroom.
The design used in the evaluation has been based on use of time and methodological triangulation to build firm inferences. This has enabled us to establish that the instrument we have developed yields profiles of errors which can be used for purposes of both diagnostic and screening purposes. It can also be used as part of a wider programme procedure for planning instruction in classrooms, as well as individualized instruction for children.
THEORETICAL BASIS OF THE INSTRUMENT
It is apparent from the literature that learning disabilities and dyslexia are terms which are differently understood and variously defined. They can be defined either in terms of their congenital and physiological basis (Kolb & Whishaw, 1996; Rains, 2002), or functionally as related to underlying cognitive processes (Naglieri, 2000; 2008; Naglieri & Crockett, 2005; Vellutino, Scanlon, Small & Fanuele, 2006).
Despite these definitional differences, there appears to be agreement in the literature indicating that early diagnosis of children with dyslexia and learning disabilities is of paramount importance (Shaywitz, 1998; 2003), and that priority needs to be placed on screening all children to identify those likely to require learning support. Procedures are also needed to assist teachers in the classroom to find practical solutions to the problems of the need for early identification of children with learning problems, as well as screening procedures which can be used in group teaching situations in the classroom.
The reason for this is that it is most often the teacher who first notes that a child has a problem with school work (Wadlington & Wadlington,2005), and it is the child with early difficulties with language acquisition, phonological and phonemic awareness who is likely to experience later learning difficulties relating to the acquisition of reading, writing and spelling (Adams, Foorman, Lundberg, & Beeler, 1998; Shaywitz, 2003).
The development of our instrument (which uses an error classification system based on phonic errors) took its departure point both from the Harvard studies (Chall, 1967), as well as other research (e.g. Adams, 1990; Liberman, Shankweiler, & Liberman, 1989) which indicated the value of use of structured phonics in the classroom. It embodies a response to Coltheart, Masterson, Byng, Prior and Riddoch’s (1983) observation that there is a close similarity between the reading and spelling performance of both acquired and developmental dyslexics, supporting the view that errors made in both reading and spelling may be useful not only for diagnosing or screening learning disabilities, but also for planning instruction.
The instrument we have evaluated identifies the phonic errors made by children as a precursor to teaching them. Our evaluations have been conducted ipsatively as well as comparatively, in order to establish
We focus this paper on the applied as opposed to the theoretical implications of our evaluations of the use of the Phonic Inventories for diagnostic and teaching purposes. Extensive analysis of the literature on reading, writing and spelling are presented elsewhere in other areas of our work (Els, 2005; Grasko, 2005; Pereira, 2008; Potter. 2009a; Potter, Grasko & Pereira, 2006; Ravenscroft, 2008) . The instrument we evaluate is based on the assumption that both reading and spelling reflect the ability to map letters and letter combinations to sounds (Moats, 2006). For this reason the types of spelling errors made by children can provide information which is useful for diagnosis, screening and planning instruction. This type of information is not only potentially useful to teachers and clinicians (Bhamjee, 2008; Callander, 2007; MacReadie, 2001; Mazansky, 2007; Sfetsios, 2002; Wilson, 2001), but also to school administrators (Potter, Capelo, Fridjhon, Human, Ravenscroft, & Rees, 2009).
This paper describes a longitudinal evaluation of an instrument for identification of children with learning difficulties, based on profiles of errors made in writing words. Our evaluation design has been based on a series of formative, comparative, and summative evaluations of the instrument, involving aggregative case survey analysis of seventeen small-scale evaluative studies conducted between 2001 and 2005, two longitudinal evaluative case studies conducted over a period of eight years, as well as two large-scale evaluations comparing the types of spelling errors made by learners in different grades at primary school.
Our results suggest that the types of spelling error made by children with learning difficulties are persistent as they progress through school, providing indicators of learning difficulties. The error patterns yielded by our instrument also provide useful information when used for planning individual instructional programmes as well as instruction in classrooms. Based on evidence of distinctive patterns of error made by children in both mainstream and remedial schools, we are currently conducting developmental evaluations involving action research with the procedure as used by teachers and clinicians in classrooms and clinics. We are also developing computerised scoring and profile analysis to enable the instrument to be used on a large-scale.
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Gregg Ravenscroft’s paper next to her name. This will take you to Gregg Ravenscroft’s front page. You can then link to Gregg Ravenscroft’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
Monitoring educational quality in South Africa is a relatively recent development (Howie 2008) and the field is still in its infancy methodologically locally on both a systemic level and a school level. The utilisation of complex modelling techniques in monitoring educational quality has been primarily restricted to more developed environments internationally for some time, with a few exceptions locally in the past decade (Howie, 2002; Mohandas 1999, Scherman, 2007). National and international assessments have been used to monitor educational quality on system-level in a variety of ways (Howie & Plomp, 2006) and multilevel analysis has been used effectively internationally for exploring and understanding the systemic contextual factors associated with student performance. School effectiveness as a field has a long tradition in developed countries and has been associated with monitoring educational quality on school level (Scheerens, 2001) and multilevel analysis has been used most effectively in school effectiveness research in developed contexts. Although some literature exists for a developing world context (see for example Howie, 2002; Fuller & Clark, 1994; Passos, 2010; Scheerens, 2001; Scherman, 2007, van Staden, 2010; Wyatt, 1996) a dearth of studies still exist, especially in the context of Africa and South Africa in particular. These studies and other literature suggests that contextual factors should be included when exploring pupil achievement (a primary indicator of educational quality) as these factors do influence achievement and provides the context within which performance should be interpreted. While school effectiveness models are not as volatile as critics would have it, they are still not as firmly established as some enthusiasts proclaim (Wyatt, 1996). Studies of school effectiveness in developing countries should make use of the advanced statistical analysis available, i.e. multilevel analysis, an element that has been missing in school effectiveness research generated in developing countries. Likewise, the baseline measures of outcomes should be encouraged if school effects are to be inferred (Riddell, 1997).
In South Africa, the inclusion of contextual factors are particularly important for South Africa, given the apartheid era and disparities in schooling in terms of analysing and understanding pupils’ performance. The poor performance of South African pupils in international assessments underscores the importance of contextual data (Howie et al, 2009). South Africa has not performed well in international comparative assessments like the TIMSS studies in 2003, 1999, and 1995 (Howie 1997, 2001, HSRC, 2006), PIRLS 2006 (Howie et al, 2009) as well as the SACMEQ study (Moloi & Strauss, 2005) where pupils performed well below the international averages and below those of many countries. Likewise the pupils performed well below expectation in the Systemic Evaluation in Grades 3 and 6. The disappointing results could be due to pupils being ill prepared (for a number of reasons) in terms of the content areas and therefore being unable to achieve the expected assessment standards (National Department of Education, 2005).
Globalisation is an important consideration for educational quality. The issues of educational quality can be linked to globalisation. Education and globalisation are two very powerful forces at play in the international landscape (Mortimore, 2001). Where education has the power to change lives, globalisation has the power to enable education to reach all people so that they may have the opportunities to better themselves as well as become viable to compete on the global circuit. Improving the quality of education systems worldwide would enable nations to secure economic competitiveness in a globalised economy (Kellaghan & Greaney, 2004), in a process where nations are drawn together to form a single entity (Mortimore, 2001). In this view the world is becoming increasingly mobile and it
“is important to stress quality in the context of our mobile world: the diversification of societies, largely as a result of migration, urbanization and cultural change, combined with increased sensitivity to the numerous different aspects of individual and group identity, places new burdens upon education systems. These threaten to undermine education quality if they are not dealt with adequately, but at the same time they encourage a dynamic of research, experimentation and exchange of experience that can mightily advance the agenda of quality education (Pigozzi (2006, p. 3)”.
Educational quality can be thought of from a variety of perspectives such as the following:
For purposes of this paper, educational quality is used in the meaning of Van der Werf et al, 1999. The aim of this paper is to discuss the application and implications of using complex statistical techniques such as a multilevel analysis for the purpose of monitoring educational quality in a developing context such as South Africa. Two examples of modelling are provided on two levels, namely monitoring on school-level, using a small-scale study and secondly monitoring on system-level based on large-scale national data from an international comparative study.
The paper is structured in the following way: firstly a short description is given of two studies that are examples of monitoring educational quality in South Africa and utilised complex statistical models to analyse the data. This is followed by a description of the conceptual framework used in monitoring educational quality at school and system-level as well as the frameworks used in the two examples. Thereafter the research methods for the two studies are briefly summarised. The results of the two studies are given separately before the conclusions and reflections conclude the paper.
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Vanessa Scherman’s paper next to her name. This will take you to Vanessa Scherman’s front page. You can then link to Vanessa Scherman’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
Private sector agencies operating in emerging markets have increasingly come to appreciate the importance of stakeholder engagement and stakeholder participation for the success of their business ventures. They increasingly recognise the reputation risks that come from poor stakeholder relations and lack of participation and place a growing emphasis on corporate social responsibility and transparency.
Participation can be defined as the process through which people with an interest (stakeholders) influence and share control over initiatives and the decisions and resources that affect them (ADB, 2001).
The concern with successful stakeholder participation has obviated the importance of evaluating the interactions between stakeholders in business ventures.
In this paper, a dialogical-activity theoretical framework approach to evaluating stakeholder engagement in business development projects is proposed (Engeström, 1995). It is argued that such framework enables insight into the stakeholder process as well as providing an impetus for positive change. The paper first presents the dialogical- activity theoretical approach. This is followed by an application of this approach to a social impact assessment (SIA) of a mining project in Africa in which the author was involved.
A dialogical-activity framework
Activity theorists embrace activity as the explanatory nexus of human behaviour. They regard activity as ‘a purposeful process in which a subject (i.e. corporate agency) is connected to an object ( business output/profit) through culturally constituted mediational means (i.e labour/management ) (R.Engestrom, 1999, p.35). In other words, human behaviour is seen as socially and culturally mediated activity towards a purpose, obtaining meaning within a social context.
Leont’ev’ (1978) makes an important contribution to the conceptualisation of the relationship between the individual and the social, by making a distinction between Activity, Action and Operation. Activity is collective in nature and represents the overarching object to which an Activity system (i.e. business venture) is directed. Operations are the routinised and iterative sets of responses on the part of the individual human subject that are often performed below the level of conscious awareness. Operations are the constitutive elements of (and subsumed beneath) Actions (i.e labour), which are, characterised by goal-directedness and intentionality. The utility of this three-part typology is the manner in which it allows for a heuristically useful distinction between individual goal-directed actions (i.e. actions of business agencies or stakeholders) and overall collective object-orientated activity (i.e. a business venture).
According to Y. Engestrom, (1993, p.68) activity systems are not stable and harmonious, but evolve through the resolution of their inner contradictions and tensions or contradictions which arise from the injection of new elements into the activity system. Y. Engestrom argues that these contradictions are the driving forces in what he calls ‘learning by expanding’ (Engestrom, 1987). This is a process by which the participants in an activity system construct a new object based on a resolution of inner and outer contradictions.
R. Engestrom (1995, 1999) expands the Activity Theory analytic schema through the inclusion of semiotic and communicative elements inspired by the ideas of Mikhail Bakhtin. She takes Bakthin’s notion of utterance as a unit of analysis. In accordance with Bakthin (1981) she argues that utterances are culturally shared and distributed cognitive artefacts. They exist on the borderline between oneself and the other. In order to operationalise the utterance as a unit of analysis for activities, R. Engestrom (1995) invokes Bakthin’s notions of Voice, Social language and Speech genre and bridges them to the Activity theoretical concepts put forward by Leontev and Y. Engestrom. This is presented in Table 1.
Table 1: Conceptual schema of the proposed similarities between levels of activity,
components of the Activity System and Speech.
| Bakhtin | Leontev | Engestrom> |
| Social Language | Activity | Object/Outcome |
| Voice | Action | Subject -object |
| Speech Genre | Operation | Rule/tool (mediational means) |
In this paper, we will discuss the role of evaluators with change. How does evaluation relate to change management? Evaluators have a significant change management role in the organizations and systems they evaluate. Change Management Theory research includes an Images of Managing Change Theory with six different Images of Managing Change created on a matrix using two dominant factors. Factor 1 is Management of Change: management as controlling activities or management as shaping capabilities and Factor 2 is Outcomes of Change: intended, partially intended, or unintended outcomes.
How do these Images relate to evaluators? Taking these two factors into account, this paper is proposing that evaluators are managers of change that follow one of the six different images, the Image of Interpreter. The Image of Interpreter is a combination of the Factor 1, Management of Change: management of shaping capabilities and the Factor 2, Outcomes of Change: partially intended outcomes. The Interpreter Image of managing change places the change manager in the position of creating meaning for organizational and/or system members and helping them to make sense of organizational systems.
How are the organizations and systems we evaluate being impacted by our work? If we agree on this Image of Managing Change and that we have a key role in Change Management, what can we do differently to enhance our impact on organizations and systems? How will this knowledge impact our methods and practices?
In this paper, we will discuss the role of evaluators with change. How does evaluation relate to change management? Evaluators have a significant change management role in the organizations and systems they evaluate. Change Management Theory research includes an Images of Managing Change Theory with six different Images of Managing Change created on a matrix using two dominant factors. Factor 1 is Management of Change: management as controlling activities or management as shaping capabilities and Factor 2 is Outcomes of Change: intended, partially intended, or unintended outcomes.
How do these Images relate to evaluators? Taking these two factors into account, this paper is proposing that evaluators are managers of change that follow one of the six different images, the Image of Interpreter. The Image of Interpreter is a combination of the Factor 1, Management of Change: management of shaping capabilities and the Factor 2, Outcomes of Change: partially intended outcomes. The Interpreter Image of managing change places the change manager in the position of creating meaning for organizational and/or system members and helping them to make sense of organizational systems.
How are the organizations and systems we evaluate being impacted by our work? If we agree on this Image of Managing Change and that we have a key role in Change Management, what can we do differently to enhance our impact on organizations and systems? How will this knowledge impact our methods and practices?
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Joan Vermillion’s paper next to her name. This will take you to Joan Vermillion’s front page. You can then link to Joan Vermillion’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
Innovative learning and evaluation practices: examples from Asia
“The most important thing for the CBNRM course is to have the mechanism to improve it according to the suggestions and advice from different stakeholders. Maybe the form this monitoring takes is not the most important aspect. What counts is how to make every teacher really respect the suggestions and advice from different stakeholders, and to make other stakeholders accept the teacher’s willingness to participate. In the existing system of our course evaluations, the evaluation of the course is just a formality. Nobody cares about the students’ needs, interests, or opinions. And the actual way the evaluation is done is too stiff. But the CBNRM course gives us an example of how to solve this. Yes, we can do it! Everyone feels that they are the master of the course. Teachers are not afraid to face different needs of different students; and students are not the ducks waiting to be fed. So everyone feels very happy in the course.”
MSc student, China Agricultural University/College of Humanities and Development, January 2007 (quoted in Vernooy et al. 2010)
Well-designed, regular monitoring and evaluation (M&E) can contribute to good practices in Community-Based Natural Resource Management (CBNRM) capacity development (Horton et al. 2003; Vernooy et al. 2003). Most research and development organizations routinely monitor and evaluate their capacity development efforts, but mainly in terms of immediate outputs, e.g., changes in knowledge associated with training activities or distribution and readership of publications and knowledge products. This kind of evaluation does not adequately track changes beyond the level of a specific activity.
A coherent and meaningful evaluative learning framework helps enhance the existing pool of knowledge by offering valuable lessons on and insights into effective strategies and factors that lead to success in capacity development (Engel et al. 2007; Baser and Morgan 2008; Taylor and Ortiz 2008). It can also provide a critical view of issues surrounding scaling up, sustainability, and institutionalization.
Such frameworks consist of a clear definition of context, content, capacity, the capacitated, and capacity development. Collaborative learning provides a platform for those seeking to evaluate capacity development, by enabling them not only to conduct evaluation, but also to develop their capacity to evaluate. It also allows participants to draw on their individual and collective experiences to build a practice-informed theory on evaluating capacity development. The effectiveness of evaluation can be greatly enhanced if it is built into and becomes integral to the capacity development process and is fully embraced by all those involved. Defining clear utilization of evaluation results is another important factor contributing to greater effectiveness (Patton 2003).
An adaptive mode of learning is likewise critical to successful evaluation, as continuous conceptual and methodological refinement occurs with increased understanding of the contexts and purposes of evaluation (Vernooy et al. 2010). There is, however, no single way to integrate M&E into learning processes. It is worth experimenting with a variety of methods and tools, adapting one’s practice along the way while keeping a critical eye on the time and energy spent on M&E. In the three case studies from Asia highlighted in this paper (more details in the following section), the organizations involved have tried or are trying such an adaptive approach. They have done so by joining forces with other partners in the region interested in strengthening evaluation expertise for CBNRM capacity development (see Box 1).
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Ronnie Vernooy’s paper next to his name. This will take you to Ronnie Vernooy’s front page. You can then link to Ronnie Vernooy’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
Randomised control trials (RCTs) have become popular in the development community as part of the response to the results agenda. These studies are well placed to address the question of which programs work or not. And, properly designed, they can be embedded in a broader evaluation design which also addresses questions of why an intervention works in a specific context or not, and at what cost. Usually the appropriate design will not be a simple randomization drawn from an eligible list. So there are several decisions to be made in designing a RCT, and these decisions should be recorded in a study protocol which is placed in the public domain before the study begins. Many of the criticisms of RCTs can be responded to. But some require more reflection and research and have implications for the conduct of RCTs.
Driven by the focus on results, impact evaluation has become a prominent part of the development agenda in the last decade. Much of the discussion of results has focused on outcome monitoring, such as the attention devoted to tracking the Millennium Development Goals. Whilst useful, outcome monitoring cannot tell us the impact of an intervention, and so cannot be used to make an assessment of the contribution an agency has made to development.
But there has been growing use, notably amongst economists and political scientists, of a range of approaches which do directly tackle this question of what difference an intervention has made. Prominent amongst these approaches are experimental designs, or randomized control trials (RCTs).
The purpose of this paper is to provide a short, non-technical introduction to RCTs. More technical treatments are available from Bloom (2006) and Duflo et al. (2006). The paper deals briefly with what is meant by impact evaluation, before moving onto the problem of selection bias and how it can be dealt with through experimental and quasi-experimental designs. Practical concerns in designing and implementing a RCT are then discussed before moving on to some of the criticisms which are commonly made of this approach.
What is impact evaluation?
Within the development community many think of ‘impact’ as meaning long-run effects. This usage is the DAC definition and is embodied in many versions of the log-frame. However, as I have discussed elsewhere (White, 2010) it is not at all what I mean by impact evaluation. Impact evaluation in my usage refers to looking at what difference a program made: did it improve lives, save lives even? Impact evaluation is a ‘with versus without’ analysis: what happened with the program (a factual record) compared to what would have happened in the absence of the program (which requires a counterfactual, either implicit or explicit).
Another name for impact evaluation is attribution analysis. We want to attribute some part of observed changes to the policy, program or project being evaluated. Again, many in the donor community mean something different by attribution. They mean attribution to their agency. I am not concerned here with that issue. I am interested in attribution to a specific intervention, regardless of who funds it. Impact evaluation is about development effectiveness not aid effectiveness. Having said that, impact evaluation of programs supported by donor funds either directly (project aid) or indirectly (program aid) should clearly play an important role in addressing the issue of that agency’s contribution to development.
So, where is the counterfactual to come from? The answer depends on the nature of the intervention. For ‘large n’ interventions, in which the intervention is delivered to many units (households, schools, clinics, firms, villages, districts or whatever) then statistical analysis is the most appropriate means of constructing a counterfactual. Specifically, the counterfactual is constructed by identifying a comparison group, which is similar in all respects to those receiving the intervention, except that it does not receive the intervention. Then the differences in the indicators of interest (usually outcome-level indicators) are compared in the project and control groups after the intervention, called an ex-post single difference design. It is preferable to have data on the indicator from before the intervention also, that is a baseline survey, so a double difference impact estimate can be calculated. The double difference is the change over time in the difference between the groups of the indicator, or, equivalently, the difference in the change.
So, the next question is, how to identify a suitable comparison group?
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Howard White’s paper next to his name. This will take you to Howard White’s front page. You can then link to Howard White’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
Evaluation is an emerging field in social science in the past two decades; as a result, this discipline does not have the rich psychometric history that psychology enjoys. A lack of strong psychometric history in the field necessitates the development of reliable and valid instruments. However, developing and testing the psychometric properties of an instrument can be time consuming, and evaluators may choose simply to develop instruments that make intuitive sense without a careful examination of their psychometric properties.
Exploratory factor analysis (EFA) is especially useful in such a scenario. EFA is a statistical procedure that is used to identify the interrelationships among a large set of observed items. Through data reduction, EFA groups a certain number of observed items into a common factor (Nunnally & Bernstein, 1994). A common factor is also referred to as a latent variable or a construct, as it is not measurable directly (DeVellis, 1991).
Since most programs, or at least part of a program, aim to change a psychological construct of their participants, a valid survey with multiple items is warranted. For example, evaluators cannot assess parent-child relationship directly because it is a latent variable, but they can examine it through indictors (i.e., items on a survey) of the relationships.
Exploratory factor analysis can provide validity evidence through the examination of internal structure of newly developed instruments. In this article, we first present an in-process case study that utilized EFA. In the methods and results sections, we argue the appropriate use of exploratory factor analysis in the initial testing phrase of a newly developed instrument, namely the many decisions, such as rotation and extraction methods, that evaluators must make while conducting an EFA. Finally, we discuss the implications and the use of EFA in the present as well as other evaluation studies.
Despite its popularity and usefulness in educational and psychological research, exploratory factor analysis (EFA) may not be as frequently used in evaluation studies. We argue that EFA is especially important in such studies when study-specific instruments are employed. EFA can begin to provide evidence of measurement validity through the examination of internal structure of the measurements. Such evidence can increase the utility of the instrument in evaluation. However, too often EFA is conducted using software default settings (e.g., principal component analysis with varimax rotation) which may not be most appropriate for the
We use an in-process case study as an example to provide guidelines for appropriate uses of EFA.
Read the full paper by going to the closed area of the Downloads Section of the website. Click on Proceedings. Then click on the title of Kai Chi Yam’s paper next to his name. This will take you to Kai Chi Yam’s front page. You can then link to Kai Chi Yam’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
The main pressure for Monitoring and Evaluation (M&E) in South Africa comes from the accountability strand of M&E in government, but slowly the need for reliable data for data-driven decision making is taking on more importance. This based upon the prominence of treasury regulations, for example, the Treasury regulations (2005: 60) in accordance with the Public Finance Management Act (RSA: 1999) demands that annual reports ‘include information about the institution’s efficiency, economy and effectiveness in delivering programmes and achieving its objectives and outcomes against the measures and indicators set out in any strategic plan for the year under consideration.’
Despite this strong regulatory envrionment many government departments and civil society organisations are not yet ready for evaluation as they are still struggling to set up monitoring systems. The Presidency noted this ‘evaluation lag’, but hopes to institutionalise some existing evaluation activities (Engela & Tania, 2010).
Nevertheless, there is a growing demand for a variety of related services on the institutional level:
Read the full paper by clicking Downloads on the main menu of the website. Scroll down and click on Conference Proceedings. Scroll down through the papers and then click on the title of Jennifer Bisgard’s paper next to her name. This will take you to Jennifer Bisgard’s front page. You can then link to Jennifer Bisgard’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
Psychologists are frequently confronted by individual differences when they are required to deal with individuals who present with a unique combination of difficulties, abilities and deficits. In particular, when investigating the treatment of developmental disorders such as dyslexia, it may be necessary and preferable to consider individual cases rather than group studies. Evaluating the efficacy of an intervention in single case studies raises a different set of methodological issues to those appropriate in group studies.
A number of programme evaluation theorists have focused on case study methodology as being appropriate to the evaluation of programmes as well as the study of individuals. Drawing on this literature, strategies to address some of the challenges involved in conducting studies of children with developmental disorders are discussed. These are illustrated in their application in the treatment of a child with developmental dyslexia
Read the full paper by clicking Downloads on the main menu of the website. Scroll down and click on Conference Proceedings. Scroll down through the papers and then click on the title of Yvonne Broom’s paper next to her name. This will take you to Yvonne Broom’s front page. You can then link to Yvonne Broom’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
The central characteristic of collaborative partnerships is the breadth of their engagement in the community to develop and sustain social change (Roussos & Fawcett, 2000, 370). Collaborative partnerships have been defined as “an alliance among people and organizations from multiple sectors, such as schools and businesses, working together to achieve a common purpose” (Roussos & Fawcett, 2000, 369).
The collaborative partnership is a form of inter-organizational alliance extensively used by many public health, justice and community arts organizations. Projects stand as important exceptions to normal organizational behavior. Projects cross normal organizational boundaries, combining people and resources from parts of an organization that do not normally work together for special innovative purposes. Projects among collaborative partnerships are anomalous for typical projects. The collaborative partnership project lacks the unifying force of one organizational structure, and the goals of these projects are complex and diffuse.
Collaborative partnerships focus on community change, but they use a variety of strategies: “aspects of social planning, community organizing, community development, policy advocacy” (Roussos & Fawcett, 2000, 369). Their central characteristic is not necessarily participatory structures; they can involve top down or bottom up management styles.
Collaborative partnerships represent a genre of community intervention that is familiar and important. In the United States infra-structures have developed to help collaborative partnerships develop especially among public health, justice and arts organizations. CADCA (Community Anti-Drug Coalitions of America) trains community anti-drug coalitions. Some national mutual aid organizations have layers of affiliation, with a national organization supporting states chapters which in turn support local, often county, affiliates. Examples include the National Alliance on Mental Illness and The ARC (ne Association for Retarded Citizens). Self-help organizations partner routinely with professional service systems. Many U.S. federal programs require local partnerships among several organizations. Some partnerships emerge by necessity as in special docket courts such as drug courts, domestic violence courts, and mental health courts. The concept of public-private partnerships is based on collaborative partnerships.
This paper presents a general model for collaborative partnership projects and tests the model with activities in 2006 at Twanano Papermaking which were part of the project called New Partners/New Knowledge (NP/NK) from 2005 - 2006. Twanano Papermaking was founded through a collaborative partnership that included Phumani Paper, the Department of Arts, Culture, Science and Technology (DACST) and Technikon Witwatersrand (TWR). Phumani Paper founded 20 other organizations like Twanano in townships around South Africa. Twanano remains part of this collaborative partnership, even though some partners have left and the relationships among them have changed.
Phumani Paper’s history illustrates the importance among collaborative partnerships of foundational moments and the unrelenting pressure on the collaborations to change, transform and even expire. The collaborative partnerships of Phumani Paper have continued to focus on their core goals, but the partnerships themselves have undergone constant change.
Twanano was affiliated with several organizations in a collaborative partnership, and NP/NK was a special project that sought an ambitious set of goals. NP/NK sought no less than to help Twanano’s parent organization, Phumani Paper, to “re-capitalize” its affiliated sites through re-development of human, social and cultural capital.
The general model for collaborative partnership projects suggests that considerable participation is required. For example, participation at Twanano provided a way to attract and renew successive generations of creative people; and it helped adapt to the needs of local communities. The model also suggests that a collaborative project consists of an inner and outer collaborative process which help to structure and direct the complexities of these projects. The model shows that skillful mediation is required in both processes to direct and frame participation.
NP/NK provides illustrations of the paradoxical combination of participatory organization and mediated, directed leadership among collaborative partnerships. This paper suggests that planners, managers and evaluators of projects among collaborative partnerships can be helped by this model to address this fluidity and these paradoxes.
Read the full paper by clicking Downloads on the main menu of the website. Scroll down and click on Conference Proceedings. Scroll down through the papers and then click on the title of Mark Creekmore’s paper next to his name. This will take you to Mark Creekmore’s front page. You can then link to Mark Creekmore’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
Psychologists are frequently confronted by individual differences when they are required to deal with individuals who present with a unique combination of difficulties, abilities and deficits. In particular, when investigating the treatment of developmental disorders such as dyslexia, it may be necessary and preferable to consider individual cases rather than group studies. Evaluating the efficacy of an intervention in single case studies raises a different set of methodological issues to those appropriate in group studies.
A number of programme evaluation theorists have focused on case study methodology as being appropriate to the evaluation of programmes as well as the study of individuals. Drawing on this literature, strategies to address some of the challenges involved in conducting studies of children with developmental disorders are discussed. These are illustrated in their application in the treatment of a child with developmental dyslexia.
Read the full paper by clicking Downloads on the main menu of the website. Scroll down and click on Conference Proceedings. Scroll down through the papers and then click on the title of Tracy Hadfield’s paper next to her name. This will take you to Tracy Hadfield’s front page. You can then link to Tracy Hadfield’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
This paper focuses on the potential contribution which empowerment theory can make as a basis for evaluating development programmes in educational settings. It is suggested that community psychology's ecological perspective and contextualist epistemology, and more specifically empowerment theory, provide a useful way of conceptualising school development. This, in turn, provides a basis for understanding school development programmes and evaluating their work.
The paper begins with a discussion of some of the theories relevant to community psychology, concerning ecological perspectives and contextualist understandings. Empowerment is then examined as a theoretical framework to guide the evaluation of community (such as school) development programmes.
Empowerment is defined as a process occurring at a number of levels. To illustrate its application in evaluation in this paper, empowerment theory is applied in an evaluative case study of a school development programme. The evaluation design is based on use of an empowerment framework as a reference point for establishing whether the work of a school development programme has been valuable, and produced tangible evidence of empowerment in schools and teachers. This is done through examination of both quantitative and qualitative data drawn from this particular programme.
Based on the analysis, conclusions are drawn the end of the paper, concerning the possibility of using community psychological theory, and empowerment theory in particular, as a framework for evaluating development programmes working in educational settings. Discussion is also undertaken concerning the possibility of using community psychological theory as a basis for analyzing education more generally.
EMPOWERMENT AS A CONCEPT
Rappaport (1987) makes the case for empowerment as a useful concept for examining development in the various contexts and settings in which community psychologists work in practice. He feels that, “whatever our area of study; children, adults, the elderly, organisations, neighbourhoods or social policies what hold these diverse efforts together is a concern for empowerment” (p. 129).
Empowerment is conceptualised by Rappaport and other community psychologists such as Zimmerman (1995; 2000) as one of the terms which can be used to understand the way in which social and community development occur. Empowerment is thus a contextual and ecological term, which needs to be understood with reference to how particular individuals live and work in particular community settings.
DEFINING EMPOWERMENT
Empowerment as a term has been frequently used in community psychological as well as programme evaluation literature. Despite its frequent usage, there has been a lack of clarity on what is actually when writers refer to the term empowerment (Perkins, 1995; Perkins & Zimmerman, 1995). Rappaport (1984), for example, suggested that empowerment is easy to define in its absence – alienation, powerless, helpless – but difficult to define positively because “it takes on a different form in different people and contexts” (p. 2).
In this paper, empowerment is used to suggest a belief in the power of people to be both the masters of their own fate and involved in the life of their several communities. It is a process by which people, organisations and communities gain mastery over issues of concern to them whether those be events, outcomes or resources (Rappaport, 1987). As Rappaport comments: “Empowerment is viewed as a process: the mechanism by which people, organizations and communities gain mastery over their lives” (1984, p. 2).
This paper also follows Rappaport (1984) in suggesting that empowerment is a process, which occurs at multiple levels of analysis. Swift & Levin (1987) and Eylon & Au (1999) have suggested that empowerment is an enhancing and energising process. It is context-specific, and expands the feeling of trust and control in oneself as well as in one’s colleagues and one’s organisation, and which consequently leads to certain individual and organisational outcomes. This type of definition suggests that empowerment can occur not only at the individual level but also at the level of the group (e.g. the community and organisation).
Read the full paper by clicking Downloads on the main menu of the website. Scroll down and click on Conference Proceedings. Scroll down through the papers and then click on the title of Alex Hassett’s paper next to his name. This will take you to Alex Hassett’s front page. You can then link to Alex Hassett’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
The Open Learning Systems Education Trust (OLSET) has been working in South Africa since 1992. Its aim has been to develop a model for teaching English as a second language in South African primary schools, through the medium of interactive radio. It has provided in-service training and support to large numbers of teachers involved in implementing the programme, and in using its audio and print materials in the classroom.
This paper outlines shifts in the programme’s implementation theory which have taken place over the seventeen year period from 1992 to end 2009, from a view of curriculum as skills-based to a process view of curriculum. It focuses on the role evaluation has played in the development of the programme and its vision over this period, and describes the participatory evaluation approach which has been developed to support the programme’s implementation at large scale.
Evaluation has been used formatively and developmentally to assist the project staff in gaining acceptance of the programme by teachers and learners, as well as its endorsement by principals and educational officials, as it has grown to scale to providing support to an estimated 32 500 teachers and 1 300 000 learners over the period 2004 –2009.
Read the full paper by clicking Downloads on the main menu of the website. Scroll down and click on Conference Proceedings. Scroll down through the papers and then click on the title of Sabrina Liccardo’s paper next to her name. This will take you to Sabrina Liccardo’s front page. You can then link to Sabrina Liccardo’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
Developing country public sector capacity has proven difficult. Despite the financial resources provided and international commitments made, reviews have acknowledged results are below expectations (World Bank, 1998; World Bank, 2005; Commission for Africa, 2005). Jones and Kettl (2003) argue the inability to identify problems affects the design and implementation of reforms is the key issue. While donors have historically thrown technical solutions at political and social problems, Reeler (2007) advocates for good theories of change to assist those involved in development to build their thinking and understanding of complex social processes. Similarly, in a recent review of the quality of the UK Department for International Development’s (DfID) evaluation, Perrin (2009: 28) argues that:
there is a need for the development and use of more appropriate theory-of-change models that articulate and begin to encompass the complex and diffuse nature of the linkages between what DFID does and wider impact.
This paper proposes that surfacing theories of change, that is the collection of assumptions, principles and propositions to explain the relationship between the program’s actions and expected outcomes, are a possible avenue to aid better understanding of development contexts, change processes and the use of capacity development strategies. It is based on the premise that the act of articulating helps assumptions and biases to be brought into the open for discussion, clarification and testing in different contexts to increase learning. As a consequence, the capacity development strategies used may be more effective in achieving desired outcomes.
The paper presents a case study which sought to surface theories of change underpinning strategic planning, monitoring and evaluation and client service charters being introduced to the Government of Tanzania. Partial, missing and confused theories of change are presented along the implications for the public sector capacity development.
Several documents relating to the Public Sector Reform Program in Tanzania were analysed to identify components (context, mechanism and outcomes) of theories of change as they pertained to strategic planning, monitoring and evaluation and client service charters. The document analysis sought to answer the following questions:
Read the full paper by clicking Downloads on the main menu of the website. Scroll down and click on Conference Proceedings. Scroll down through the papers and then click on the title of Donna Loveridge’s paper next to her name. This will take you to Donna Loveridge’s front page. You can then link to Donna Loveridge’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
There are many opportunities, indeed, demands from international donor agencies for scientists to bridge the gap between science and policy-making. This paper discusses how the Health Economics and HIV/AIDS Research Division (HEARD) at the University of KwaZulu-Natal in South Africa has used advocacy, subsidiary research and marketing principles to achieve a practical result for a project that was based on a quantitative study on child welfare in the context of HIV/AIDS. The purpose of this evaluation is to document the processes followed in this component of the programme, to identify results achieved, and to highlight lessons learned from an initiative that combined research and advocacy to promote utilisation of research evidence on an applied level.
Background: The Amajuba Child Health and Wellbeing Research Project (ACHWRP) conducted in South Africa had two aims. The scientific aim was to document and compare material and psycho-social welfare of orphans and non-orphans in a district via three annual surveys of approximately 700 children and their caregivers. The practical aim was to facilitate the development of a district child welfare management plan.
Method: Achieving the practical and applied aim involved subsidiary research and advocacy, based on a social marketing exercise. This aimed to remove the constraints that inhibited local child welfare agencies, particularly local government departments, from improving child welfare services. Outcome mapping was applied in identifying strategic partners as the basis for implementation and evaluation of this component of the programme.
Results: The results of the evaluation indicated that the social marketing exercise succeeded in obtaining the inclusion of a child referral mechanism into local government policy and also in practice in three district wards.
Conclusion: The evaluation suggests that successful intervention to get research into policy and practice requires a continuum of interventions. It also challenges common assumptions amongst researchers, donors, government and non government agencies about the principles and practice of advocacy linked to primary research.
Read the full paper by clicking Downloads on the main menu of the website. Scroll down and click on Conference Proceedings. Scroll down through the papers and then click on the title of Luke Potter’s paper next to his name. This will take you to Luke Potter’s front page. You can then link to Luke Potter’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
There are many opportunities, indeed, demands from international donor agencies for scientists to bridge the gap between science and policy-making. This paper discusses how the Health Economics and HIV/AIDS Research Division (HEARD) at the University of KwaZulu-Natal in South Africa has used advocacy, subsidiary research and marketing principles to achieve a practical result for a project that was based on a quantitative study on child welfare in the context of HIV/AIDS. The purpose of this evaluation is to document the processes followed in this component of the programme, to identify results achieved, and to highlight lessons learned from an initiative that combined research and advocacy to promote utilisation of research evidence on an applied level.
Background: The Amajuba Child Health and Wellbeing Research Project (ACHWRP) conducted in South Africa had two aims. The scientific aim was to document and compare material and psycho-social welfare of orphans and non-orphans in a district via three annual surveys of approximately 700 children and their caregivers. The practical aim was to facilitate the development of a district child welfare management plan.
Method: Achieving the practical and applied aim involved subsidiary research and advocacy, based on a social marketing exercise. This aimed to remove the constraints that inhibited local child welfare agencies, particularly local government departments, from improving child welfare services. Outcome mapping was applied in identifying strategic partners as the basis for implementation and evaluation of this component of the programme.
Results: The results of the evaluation indicated that the social marketing exercise succeeded in obtaining the inclusion of a child referral mechanism into local government policy and also in practice in three district wards.
Conclusion: The evaluation suggests that successful intervention to get research into policy and practice requires a continuum of interventions. It also challenges common assumptions amongst researchers, donors, government and non government agencies about the principles and practice of advocacy linked to primary research.
Read the full paper by clicking Downloads on the main menu of the website. Scroll down and click on Conference Proceedings. Scroll down through the papers and then click on the title of paper next to his name. This will take you to Tim Quinlan’s front page. You can then link to Tim Quinlan’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
This paper reports the results of an evaluation which focused on the educational background and experience of a small sample of English-additional language students entering engineering education in South Africa. The aim of the evaluation was to contribute to a growing understanding of the causes of high failure rates and transitional difficulties which many South African students encounter when they enter higher education in the country.
Using a thematic content analysis of interview transcripts and questionnaires, the learning behaviours of the students were identified and classified. The themes from the analysis indicated that the students’ educational background had developed in the majority of them ‘surface-oriented learning practices’ which were problematic in a tertiary learning environment. These practices were deeply engrained and pervasive in their influence affecting not only the way they approached learning but how they interacted with teachers and textual material, how past examination papers were used in their study strategies, and how they perceived knowledge and learning.
Using the themes from the interview and questionnaire data as well as classroom observations, we also investigated the first year experience of the students and identified some of the difficulties the students commonly encountered and the learning behaviours that emerged during their first year at university as a result of these difficulties. Analysis of the data indicated that many of the difficulties which the students experience when they enter university can be explained and understood as the result of the ‘cultural distance’ between the educational culture in which the students were raised and the educational culture existing at the university they enter. Specific features of this cultural distance are the learning practices of the students and their lack of proficiency with English as a medium of instruction.
The results of the evaluation indicated that transitional distance was a particular problem for possibly the majority of students entering a first year engineering programme in South Africa. The transitional difficulties which students faced on entering university could be understood in terms of a journey of adjustment that was necessitated by the distance between the culture the students come from and the culture they were entering.
Following Stake (1983; 2010), such understanding can provide a credible basis for designing learner responsive modifications to a first year programme. Suggestions are made about what educators can do in the way they design and deliver first year curricula in order to address these culture-derived difficulties and hopefully minimize their magnitude and influence.
Read the full paper by clicking on Downloads on the main menu of the website. Scroll down and click on Conference Proceedings. Scroll down through the papers and then click on the title of Laurie Woollacott’s paper next to his name. This will take you to Laurie Woollacott’s front page. You can then link to Laurie Woollacott’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
The Society for Family Health (SFH) Zambia had two peer education intervention programmes—the sexual-health peer education and water-purification peer education. Targeting schools, clinics, markets, and taxi stations, the aim of the latter intervention was to motivate individuals to purify drinking water, especially using CLORIN. CLORIN is a home water disinfectant that consists of concentrated chlorine solution, a chemical that kills most disease-causing bacteria in water.
The SFH peer education programme designed the water-purification intervention around the Knowledge Deficit Model and the Health Belief Model. Referring to water-purification, the Knowledge Deficit Model assumes that an individual does not know that untreated water is contaminated and such water can cause diarrhoea. Therefore, the model emphasis that educators should communicate that treating water with CLORIN lessens contamination levels and therefore diarrhoeal diseases. The Health Belief Model requires educators to communicate that diarrhoea can be a serious illness, everyone is at risk of getting diarrhoea, and there are benefits to treating drinking water with CLORIN.
The peer education presentations were participatory and took the form of question-and-answer discussions. This way, Peer Educators expected their audience to increase their knowledge and improve their attitude. This paper answers the question “was the peer education intervention effective in increasing knowledge on and improving attitude towards water purification?”
Read the full paper by clicking on Downloads on the main menu of the website. Scroll down and click on Conference Proceedings. Scroll down through the papers and then click on the title of Kambidima Wotela’s paper next to his name. This will take you to Kambidima Wotela’s front page. You can then link to Kambidima Wotela’s Paper, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
INTRODUCTION
The integrated curriculum evaluated in this study thematically integrated three academic disciplines for students in Grades 8-10 in a non-racial private school in Johannesburg. The curriculum allowed students to explore common issues in the early seventies, to learn about themselves, build ‘bridges’ across difference, and make learning relevant to life. Ethnographic methods were used both to understand the innovation in its own terms--as a design following the principle of themes--and to point to strengths and weaknesses to develop it where needed. The curriculum was taught in parallel with several academic curricula not integrated into the innovation in these Grades, and was succeeded by discipline curricula in Grades 11-12 leading to a matriculation certificate. The curriculum was developed in-house by teachers from multiple disciplines and was the defining innovation of the school at that time.
Evaluation Design
An ethnographic approach to evaluation seeks to understand phenomena as a social system from the perspective of those ‘within’ it. It endeavors to study a phenomenon as it occurs ‘naturally’ undisturbed by the process. It is sensitive to issues ‘foreshadowed’ within social arrangements, probes issues in some depth, and describes and interprets phenomena. The evaluation started with familiarization of the evaluator with the site as a ‘cultural scene’ (Spradley and McCurdy 1972), that is of the integrated curriculum as one amongst others in the school as a social system and more widely in schools in the region. It was followed by an extended time in which key informants were selected by deliberate decision for their knowledge of the curriculum and their ability and willingness to share their knowledge of the curriculum with the evaluator [12 students, 8 teachers]. Data gathering followed, multiple in-depth interviews of informants (Spradley 1979; Fetterman 1989, 2009) being arranged in the same time frame as naturalistic observations of lessons both to capture the curriculum from the perspective if teachers and students, and to check data through triangulation, that is against data from different instruments, sources and by re-presenting data to informants. Data was also gathered from unscheduled events where information relating to the evaluation was collected and stored in an ethnographic record. Fieldwork extended over 14 months, entailing approximately 300 observations and 70 interviews.
Data were analysed by inductive coding to establish broad emerging themes in the first instance, and thereafter the more detailed data which added substance to themes. Themes assist with describing the programme from the perspective of informants, and suggest parameters guiding the manner in which it may be interpreted. Adjudications follow and aim to persuade to views about the curriculum grounded in data.
Key words: ethnography in integrated studies, Chett Bowers, James Spradley. curriculum evaluation, Joseph Schwab, Laurence Stenhouse, Decker Walker, ethnographic evaluation, David Fetterman.
Read the full case study by going to the closed area of the Downloads Section of the website. Click on Case Studies/Teaching Materials. Then click on the title of Ray Basson’s case study next to his name. This will take you to Ray Basson’s front page. You can then link to Ray Basson’s Case Study, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
Use the key words provided to assist in searching on this site and on the web for the work of evaluation theorists and other evaluators using similar methodologies. Use these in reading about evaluation as a field, in developing your own evaluation designs, or for developing materials for teaching purposes.
INTRODUCTION
This case study is a technical report which focuses on the methodological applications used in applying Alkin’s CSE model of evaluation (1969; 1985) in formative and summative evaluation of an interventionary programme. It focuses in particular on the use of time and methodological triangulation for purposes of drawing firm inference from non-experimental data, and highlights certain of the strengths and limitations of combined use of quantitative and qualitative analyses in longitudinal evaluation.
The case study first describes the use of a longitudinal multimethod evaluation design in undertaking an evaluation into the high failure rates in the first year Engineering Graphics course at the University of the Witwatersrand, Johannesburg. It describes the methods used to establish the nature of the problem and the need for an intervention through a series of predictive analyses, the use of action research in establishing a programme designed to overcome the problem, the use of formative evaluation to establish whether the programme was working as intended, and the use of longitudinal evaluation strategies to establish programme effects, as well as continuing needs for the intervention.
Key words: spatial perception, technical education, curriculum evaluation, Joseph Schwab, Decker Walker, Laurence Stenhouse, CSE evaluation model, Marvin Alkin, multiple methods, Donna Mertens, critical multiplism, Thomas Cook, Will Shadish.
Read the full case study by going to the closed area of the Downloads Section of the website. Click on Case Studies/Teaching Materials. Then click on the title of Peter Fridjhon’s case study next to his name. This will take you to Peter Fridjhon’s front page. You can then link to Peter Fridjhon’s Case Study, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
Use the key words provided to assist in searching on this site and on the web for the work of evaluation theorists and other evaluators using similar methodologies. Use these in reading about evaluation as a field, in developing your own evaluation designs, or for developing materials for teaching purposes.
INTRODUCTION
This evaluation attempted to establish whether high imagery instructional methods could be successfully used to teach autistic children. A multimethod evaluation design was used to establish whether there was a change in selected verbal and non verbal abilities of three autistic children after a 5 month period of high imagery instruction, as compared against a baseline of response to previous instruction. Analysis of test results, in depth interviews, developmental diaries and school assessments was conducted.
At baseline all three children had phonological weaknesses. Two of the children in the sample showed little to no response to the high imagery instruction. The third child in the sample showed an increase in phonological skills and in reading, writing and spelling abilities as well as an increase in both vocabulary and the non-verbal abilities involved in drawing. It was noted at base-line as well as throughout the study that visual memory was an area of strength for this child.
The success of high imagery teaching strategies when used with a child with well developed visual memory abilities would suggest that high imagery instruction could be a useful teaching strategy for children with autism who learn through visual modes. The lack of success of the other two children in the sample would suggest that not all children with autism learn through visual strategies.
Key words: curriculum evaluation, Joseph Schwab, Laurence Stenhouse, Decker Walker, case study evaluation, Helen Simons, Robert Stake, Rob Walker, autism, mental imagery, non-verbal abilities.
Read the full case study by going to the closed area of the Downloads Section of the website. Click on Case Studies/Teaching Materials. Then click on the title of Tracy Hadfield’s case study next to her name. This will take you to Tracy Hadfield’s front page. You can then link to Tracy Hadfield’s Case Study, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
Use the key words provided to assist in searching on this site and on the web for the work of evaluation theorists and other evaluators using similar methodologies. Use these in reading about evaluation as a field, in developing your own evaluation designs, or for developing materials for teaching purposes.
INTRODUCTION
The South African Riding for the Disabled Association (SARDA) offers recreational and therapeutic riding for mentally and physically disabled individuals. With private patients, the individual’s level of function at the time of entry into the programme is considered and the individual’s guardians or parents and SARDA’s instructors decide on realistic developmental goals appropriate for the individual. Schools catering to disabled individuals attend group rides in which all participating students ride together in an arena with overarching goals guiding development.
I conducted a needs assessment based on Parlett and Hamilton’s (1972) illuminative evaluation approach for SARDA in 2009. I decided to take this flexible and responsive approach to the needs assessment to allow for a variety of issues to emerge from the programme rather than constraining the needs identified from the outset by applying a more rigid measurement based model. I consulted programme documents, observed the programme and conducted interviews with key personnel and identified the four needs described below.
Identified Needs
SARDA struggles to attract volunteer instructors; at the time of the needs assessment, SARDA had three instructors out of the ideal eight. Since the inception of the Highveld branch, participants have joined the programme on an open-ended basis: there has been no expected time frame and thus no time limit imposed. Given that diminishing gains are typical after the first twelve sessions, extended participation is not therapeutically justified. The Highveld team recently introduced the twelve week programme (which has long been instituted international) for new clients from the long waiting list in order to include more clients and help more individuals. Such a change warrants evaluation.
The needs assessment drew attention to four key aspects of the SARDA programme that could benefit from evaluation. These areas were SARDA’s marketing (which is constrained by limited resources), SARDA’s instructor training system (which may be an obstacle to volunteers becoming instructors), the recent introduction of the twelve week programme system, and the satisfaction of SARDA’s stakeholders (a key indicator of the programme’s success). I selected these aspects of the programme for evaluation on the basis of the instructors’ expressed needs and my observations.
Evaluation Design
The design includes elements of Patton’s Developmental Evaluation (1994) as a form of Utilisation-Focused Evaluation (1984), and involves aspects of Stake’s Responsive Evaluation (1973), Parlett and Hamilton’s Illuminative Evaluation (1972), and the Practical Participatory approach to evaluation described by Cousins and Earl (1992). A responsive three step process is recommended involving the creation of an evaluation team which will oversee the identification of issues and the evaluation process as a whole, the review and selection of a method of data collection and the enactment of this method, and the interpretation and utilisation of the findings. The evaluator will provide technical expertise and ensure that data collection methods selected will address the questions posed; as such, an advisory role is implicated over a leadership role.
While I have identified funding as SARDA’s main problem, evaluation does not appear to be a feasible solution to the fundraising deficit. As such, needs that are amenable to resolution via evaluation (and which may have implications for the fundraising issue) will be focused on in the evaluation design. Each need relates to a distinct aspect of the programme, and each is tackled in this evaluation design via the introduction of a stakeholder feedback and ongoing evaluation system.
I view stakeholder satisfaction as the most pressing issue which may be moderated by perceptions about and issues around the other aspects in need of evaluation; this issue will be addressed via the formation of an evaluation task force which will address issues that would detract from stakeholder satisfaction. SARDA is successful if disabled individuals are given the opportunity to ride and are, in their parents’, caregivers’ and instructors’ opinions, reaching realistic therapeutic goals. Ongoing stakeholder satisfaction assessment and evaluation is thus useful and should be implemented as an integral part of the programme.
Key words: riding as therapy for physical disabilities, developmental evaluation, Michael Patton, illuminative evaluation, Malcolm Parlett, David Hamilton, participatory evaluation, Bradley Cousins, stakeholder approaches in evaluation, Robert Stake, Carol Weiss.
Read the full case study by going to the closed area of the Downloads Section of the website. Click on Case Studies/Teaching Materials. Then click on the title of Laura Harris’ case study next to her name. This will take you to Laura Harris’ Tracy Hadfield’s front page. You can then link to Laura Harris’ Case Study, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
Use the key words provided to assist in searching on this site and on the web for the work of evaluation theorists and other evaluators using similar methodologies. Use these in reading about evaluation as a field, in developing your own evaluation designs, or for developing materials for teaching purposes.
INTRODUCTION
This evaluation examined whether women’s groups played an empowering role in the lives of 10 Muslim women in South Africa. A qualitative methodology was employed to gain information from two women’s groups in Johannesburg. An open-ended questionnaire, focus groups, and individual interviews were used to elicit information. The data were initially analysed using a computer-based content analysis programme (Atlas-ti), to identify a priori and emergent themes and categories in the data. These themes and categories were then used as the basis for a qualitative interpretive analysis, focused in particular on issues of empowerment with respect to psychological, community, economic, intellectual and political empowerment, and more specifically on gender equality. The results indicated that while these groups did have many positive outcomes in adding value to the lives of the women in the sample, the issues dealt with in women’s groups cannot be isolated from the broader social context in which Muslim women live. While the processes of empowerment appear to have begun, gender equality still seems far off.
Keywords- Community empowerment, gender equality, Muslim women, women’s groups, human rights education evaluation, Felisa Tibbitts, democratic evaluation, Barry MacDonald, Helen Simons, evaluation for democracy, Jennifer Green, Donna Mertens.
Read the full case study by going to the closed area of the Downloads Section of the website. Click on Case Studies/Teaching Materials. Then click on the title of Zarina Hassem’s case study next to her name. This will take you to Zarina Hassem’s front page. You can then link to Zarina Hassem’s Case Study, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
Use the key words provided to assist in searching on this site and on the web for the work of evaluation theorists and other evaluators using similar methodologies. Use these in reading about evaluation as a field, in developing your own evaluation designs, or for developing materials for teaching purposes.
INTRODUCTION
This case study focuses on the potential contribution which empowerment theory can make as a basis for evaluating development programmes in educational settings. It is suggested that community psychology's ecological perspective and contextualist epistemology, and more specifically empowerment theory, provide a useful way of conceptualising school development. This, in turn, provides a basis for understanding school development programmes and evaluating their work.
1. The Programme
The programme evaluated was a school development programme working with 24 primary schools in a township outside of Pretoria. Ten of the schools had been working with the project for between 3-4 years, 6 for 2 years and 8 for 1 year.
The programme was offered by a non-governmental organisation. The programme’s aim was to ‘facilitate the development of the primary schools in that area so that they were functioning organisations providing quality education’ (Outreach, 1996, p.1). The programme used a whole school development approach to organisational change (Outreach, 1999a). In line with the work of Davidoff and her colleagues (Davidoff, 1995, Davidoff & Robinson, 1992; Davidoff & Lazarus, 1997) the focus of this approach was on the development of the school as an organisation (Outreach, 1998).
1.1. A Focus on Organisational Change
The central focus of the organisational change programme was on the drawing up and implementation of a school development plan. This focus was in line with several other similar programmes being implemented at the time in Southern Africa (Halliday & Coombe, 1994, Potterton, 1998; Schofield, 1995) and was also in line with regional education department policy. In addition, the programme focused on the issue of empowerment, particularly in terms of developing empowered outcomes through leadership development (Outreach, 2000) and school development planning (Outreach, 1999a).
1.2. The Process of School Development Planning
The process followed in school development planning involved the implementation of a multi-dimensional, whole school strategy, which aimed to bring key stakeholders together within the school. The programme’s stakeholders then identified problem areas, agreed where improvements could be made and then decided on how to make change happen with the resources the schools had available (Hargreaves & Hopkins, 1991; 1995; Hopkins, Ainscow & West, 1994).
With respect to empowerment, the focus of school development planning was based on the assumption that in order for a school to be empowered as an organisation it needed to take charge/control of its own development process. Through the school development planning process, the schools would be empowered to determine their own developmental path. Empowerment would be achieved where schools actively implemented the school development plan and took steps to achieving the goals set out in the plan (Outreach, 1998).
2.1. Evaluation Design
The evaluation was theory-based (Jobin; 2008; Weiss, 1972; 1998) in using social science theory as a basis for conducting evaluative work. The design involved an attempt to establish whether empowerment had taken place in this particular educational setting at the individual, organisational and community levels distinguished by Zimmerman (2000), and Peterson and Zimmerman (2004).
These theories were used as a conceptual framework for evaluating the work of the programme. This was done by examining the programme’s theory of implementation on individual, community and organisational levels, and by identifying whether empowerment processes had led to changes among participants in the programme in terms of empowerment outcomes.
Keywords: whole school development, empowerment of teachers, theory-based evaluation, Carol Weiss, Denis Jobin, Craig Russon, use of multiple methods in evaluation, John Creswell, Donna Mertens, Abbas Tashakkori, Charles Teddlie.
Read the full case study by going to the closed area of the Downloads Section of the website. Click on Case Studies/Teaching Materials. Then click on the title of Alex Hassett’s case study next to his name. This will take you to Alex Hassett’s front page. You can then link to Alex Hassett’s Case Study, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
Use the key words provided to assist in searching on this site and on the web for the work of evaluation theorists and other evaluators using similar methodologies. Use these in reading about evaluation as a field, in developing your own evaluation designs, or for developing materials for teaching purposes.
INTRODUCTION
The HIV Voluntary Testing and Counselling (VTC) programme evaluated was run in the Chris Hani Baragwanath Antenatal Clinic (the ANC) and controlled by the Perinatal HIV Research Unit (the P.H.R.U.). It incorporated three projects: the VTC of pregnant women; the provision of affordable formula-feeding; and clinical trials of Nevirapine in preventing mother-to-child transmission (MTCT) of HIV. The intended aims of the programme included provision of social services; successful implementation of clinical trials; and a ground-level contribution to country-wide interventions against HIV/AIDS (PHRU, 2000b).
The primary functions of the counsellors working in the programme were: to ensure informed consent was obtained from ANC patients prior to undergoing HIV testing; to disclose test results; to conduct full pre- and post-test counselling; and, in the case of positive status, to dispense Nevirapine to reduce potential MTCT of the virus. In addition, counsellors were expected to attend group and individual supervision to receive on-going training and ensure skills development. Although the counselors received minimal monthly remuneration, they were volunteers and not formal programme employees (PHRU, 2000a).
Evaluation Brief
It is a long-standing principle that evaluations should be based on real, specific questions facing those requesting the evaluation and that both the methods selected and results reported should reflect those needs (Holtgrave, Reiser & Di Franceisco, 1997). The VTC programme evaluation was requested by the counsellors themselves, as a means of formalising and illustrating to themselves, the P.H.R.U and the larger community their role in the programme and the difficulties they faced. The evaluation conducted thus aimed to examine the day-to-day running of the HIV-VTC programme via a close exploration of the counsellors’ opinions, knowledge, beliefs and attitudes regarding the programme’s aims, daily operation and efficacy.
Evaluation Design
‘Illuminative Evaluation’ was selected as an appropriate theoretical framework. Developed by Parlett and Hamilton (1972). This evaluation methodology is based on the traditional axioms of the naturalistic paradigm and is intended to describe and interpret (as opposed to measure and predict), with a particular emphasis on context, i.e. to move beyond the ‘instructional system’, the elements of the programme and their organisation, to include an examination of the ‘learning milieu’, the social and material environment in which the programme takes place (Guba & Lincoln, 1983; Parlett & Hamilton, 1972).
Parlett (1974) proposed five stages of illuminative evaluation: setting up of the evaluation (developing rapport); open-ended exploration (familiarisation through observation; emergent design); more focused inquiries (incorporation of additional data); interpretation; and the report (audience-tailored presentation) (Parlett, 1981; Parlett & Dearden, 1977; Parlett & Hamilton, 1972; Potter, 1999; Worthen & Sanders, 1987). The design is extremely flexible, becoming progressively focused as particular thematic lines emerge but still open to the inclusion of new themes and ‘negative’ evidence (Parlett, 1981).
In addition, Parlett & Hamilton (1972) emphasised that in order to evaluate a programme effectively, one had to capture its ‘recognisable reality’ by adopting a research style and methodology suitable to the programme; and to this end espoused a multi-method approach incorporating both quantitative and qualitative measures, allowing the strengthening of validity via cross-checking and triangulation (Parlett, 1981).
The design and data collection process for the evaluation paralleled Parlett’s (1974) structure fairly closely. The evaluation design was thus idiographic, inductive and qualitative in nature, aimed primarily at understanding and describing the programme. It was emergent and progressively-focused, allowing for the incorporation of issues arising during the research, as well as the pre-ordinate issues deemed important. Three qualitative methods were used: field observation, analysis of background documentation and in-depth interviews with the counsellors.
Keywords: illuminative evaluation, Malcolm Parlett, David Hamilton, use of observation, interviews and other qualitative methods in evaluation, Egon Guba, Yvonna Lincoln, Michael Quinn Patton.
INTRODUCTION
There are major differences in preparedness for higher education in South Africa as a result of continuing socio-economic and educational inequalities. These present obstacles to university access and throughput, not only at our own university, but more broadly across the country. The University of the Witwatersrand, Johannesburg has implemented the Bale Scholarship Programme which aims to identify, select, enroll and support black female students in Science, Engineering or Architecture degrees so that they graduate within minimum time.
This case study focuses on the design of an evaluation undertaken to establish whether the methods and procedures used for selecting the Bale students have been well conceptualised, and whether the support they have received while at university has been effective. The evaluation design is multimethod, and is based on analysing both qualitative and quantitative evidence collected from 20 Bale students and 10 black female scholarship Science, Engineering and Architecture students living in university residence, against a broader data base collected from contrast groups of other students in these Faculties. The aim will be to establish a blueprint for future programmes and contribute to the understanding of student access, adjustment, retention, throughput and success within the South African context.
Keywords – Transformation, Academic and Societal Integration, Evaluability Assessment, Joseph Wholey, Logical Framework (Logframe) Evaluation, Rolf Sartorius, Student Access, Adjustment, Retention, Throughput and Success.
Read the full case study by going to the closed area of the Downloads Section of the website. Click on Case Studies/Teaching Materials. Then click on the title of Sabrina Liccardo’s case study next to her name. This will take you to Sabrina Liccardo’s front page. You can then link to Sabrina Liccardo’s Case Study, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
Use the key words provided to assist in searching on this site and on the web for the work of evaluation theorists and other evaluators using similar methodologies. Use these in reading about evaluation as a field, in developing your own evaluation designs, or for developing materials for teaching purposes.
INTRODUCTION
A request to evaluate the introduction of information and communication technology (ICT) in Western Cape schools to improve learner performance in Grade 12 mathematics provided an opportunity to conduct a comprehensive evaluation in a local setting. The project started with a systematic review of the literature on the role of ICTs in learner performance. The evaluation proper included a number of overlapping phases: an assessment of the programme’s evaluability, pre-programme activities and preparation, and implementation; as well as an expert analysis of the intervention itself, and its early outcomes.
A particular feature of this evaluation was that it contained an outcome element in a programme that clearly was still under development, and the considerations and precautions taken to protect the programme against premature conclusions are described. It is argued, as it was to the programme stakeholders, that despite the inclusion of an outcome evaluation aspect in the overall design, the results needed to be regarded as formative rather than summative. The paper concludes with reflections on the evaluation itself.
Keywords – Time-on-task; technology; mathematics; education, systematic evaluation, Ralph Tyler, Peter Rossi, Howard Freeman, Mark Lipsey, evaluability assessment, Joseph Wholey, comprehensive evaluation, Peter Rossi.
Read the full case study by going to the closed area of the Downloads Section of the website. Click on Case Studies/Teaching Materials. Then click on the title of Johann Louw’s case study next to his name. This will take you to Johann Louw’s front page. You can then link to Johann Louw’s Case Study, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
Use the key words provided to assist in searching on this site and on the web for the work of evaluation theorists and other evaluators using similar methodologies. Use these in reading about evaluation as a field, in developing your own evaluation designs, or for developing materials for teaching purposes.
INTRODUCTION
This case study describes the evaluation of the development and implementation of the Open Learning Systems Education Trust’s (OLSET’s) “English in Action” programme from 1992 to the end of 2009. It focuses on the programme’s development from a uni-modal distance education model focused on enhancing learner involvement and learner gains to a multi-modal model focused on enhancing teacher and learner involvement, teacher and learner gains, school, classroom and teacher support and in-service teacher training. It highlights the expansion of schools, teachers and learners involved in the programme over a seventeen year period.
Based on evidence from longitudinal formative evaluation of the programme, reasons for the wide-spread support of the programme by teachers, principals and educational officials in South Africa are identified, and constraints and limitations highlighted. Summative comment is then provided in relation to the resurgence of interest in the potential of using radio learning as a basis for teacher development, which is evident more broadly internationally.
Keywords: distance education, open learning, teacher development, in-service training of teachers, information and communication technologies, participatory evaluation, Bradley Cousins, responsive evaluation, Robert Stake, stakeholder evaluation, Carol Weiss.
Read the full case study by going to the closed area of the Downloads Section of the website. Click on Case Studies/Teaching Materials. Then click on the title of Gordon Naidoo’s case study next to his name. This will take you to Gordon Naidoo’s front page. You can then link to Gordon Naidoo’s Case Study, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
Use the key words provided to assist in searching on this site and on the web for the work of evaluation theorists and other evaluators using similar methodologies. Use these in reading about evaluation as a field, in developing your own evaluation designs, or for developing materials for teaching purposes.
INTRODUCTION
Empowerment evaluation is an evaluation approach which entered the evaluation arena in 1993 (Miller & Campbell, 2006). It was initially introduced at the American Evaluation Association conference by David Fetterman. Since its inception, it has been utilized in a variety of contexts including individualistic, organizational, institutional and community-based settings (Fetterman, 1998).
Empowerment evaluation draws its origins from empowerment theory, community psychology, and action anthropology. It emerged in rejoinder to Positivist models which have been criticised as being too value-free in their aim to maintain objectivity (Potter, 1999). Its primary aim is “to help people help themselves” (Fetterman, 1996, p.5). Thus, it endeavours to improve existing policies and programmes, in addition to providing capacitation skills for community growth.
Empowerment evaluation can essentially be classified as a model which falls within the realm of critical-emancipatory approaches to programme evaluation (Potter, 1999) since it aims to challenge the status quo via its recognition of social problems. This set of teaching materials is designed to introduce individuals to empowerment evaluation and, it is intended to be a resource for facilitating an introductory lecture on the topic. The materials consist of a PowerPoint presentation and a set of content notes to assist the facilitator.
Keywords: empowerment evaluation, David Fetterman, Abraham Wandersman.
Read the full set of teaching materials by going to the closed area of the Downloads Section of the website. Click on Case Studies/Teaching Materials. Then click on the title of Stephen Porter’s contribution next to his name. This will take you to Charles Potter’s front page. You can then link to Stephen Porter’s Teaching Materials, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page. A poster is also provided under the poster section of the conference which can be used to support a presentation in the classroom using these materials.
Use the key words provided to assist in searching on this site and on the web for the work of evaluation theorists and other evaluators using similar methodologies. Use these in reading about evaluation as a field, in developing your own evaluation designs, or for developing materials for teaching purposes.
INTRODUCTION
This case study is a technical report which focuses on the methodological applications used in applying Alkin’s CSE model of evaluation (1969; 1985) in formative and summative evaluation of an interventionary programme. It focuses in particular on the use of time and methodological triangulation for purposes of drawing firm inference from non-experimental data, and highlights certain of the strengths and limitations of combined use of quantitative and qualitative analyses in longitudinal evaluation.
The case study first describes the use of a longitudinal multimethod evaluation design in undertaking an evaluation into the high failure rates in the first year Engineering Graphics course at the University of the Witwatersrand, Johannesburg. It describes the methods used to establish the nature of the problem and the need for an intervention through a series of predictive analyses, the use of action research in establishing a programme designed to overcome the problem, the use of formative evaluation to establish whether the programme was working as intended, and the use of longitudinal evaluation strategies to establish programme effects, as well as continuing needs for the intervention.
Key words: spatial perception, technical education, curriculum evaluation, Joseph Schwab, Decker Walker, Laurence Stenhouse, CSE evaluation model, Marvin Alkin, multiple methods, Donna Mertens, critical multiplism, Thomas Cook, Will Shadish.
Read the full case study by going to the closed area of the Downloads Section of the website. Click on Case Studies/Teaching Materials. Then click on the title of Charles Potter’s case study next to his name. This will take you to Charles Potter’s front page. You can then link to Charles Potter’s Case Study, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
Use the key words provided to assist in searching on this site and on the web for the work of evaluation theorists and other evaluators using similar methodologies. Use these in reading about evaluation as a field, in developing your own evaluation designs, or for developing materials for teaching purposes.
INTRODUCTION
This case study describes an evaluation of a five-year multi-sectoral international development program aiming at integrating gender concerns in programmatic activities in various countries. It focuses on the process of carrying out the “Interim Assessment of the Short-Term Technical Assistance & Training (STTA&T)” which purpose was to assess the impact of assistance provided by the Women in Development Office (WID) of the United States Agency for International Development (USAID) to missions and other operating units to accomplish the goals of integrating gender issues in development assistance activities. Integrating gender issues into the programs sponsored by USAID field missions is a key part of the WID Office's mandate.
The assessment considered three issues in addressing the impact of the STTA&T in several countries: (1) Appropriateness of STTA&T to USAID/Missions’ needs; (2) Usefulness of STTA&T and (3) Effect of STTA&T on USAID staff, operating units and implementing partners. Besides reviewing existing government and project documents, the assessment team conducted a series of telephone and face-to-face interviews with USAID staff and development practitioners who benefited from STTA&T. Here we discuss the challenges of gathering information to assess the impact of programs that were implemented in several countries a few years prior to the assessment.
We argue that the impact of distinct short-term technical assistance activities —such as gender trainings, gender evaluations, and sectoral assessments—on the countries were they were implemented is difficult to measure as a result of development assistance. Nonetheless, we were able to assess stakeholders’ level of satisfaction, their ability to use what they learned from the activities, and the extent to which involvement in these activities have influenced ways of thinking about gender within USAID missions and implementing partners. The opinions expressed here are those of the author and do not necessarily reflect the views of USAID.
Keywords: technical assistance, gender integration, international development programs, impact evaluation, Peter Rossi, David Freeman, Mark Lipsey, logical framework evaluation, Rolf Sartorius, USAID, AUSAID.
Read the full case study by going to the closed area of the Downloads Section of the website. Click on Case Studies/Teaching Materials. Then click on the title of Flavia Ramos-Matoussi’s case study next to her name. This will take you to Flavia Ramos-Matoussi’s front page. You can then link to Flavia Ramos-Matoussi’s Case Study, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
Use the key words provided to assist in searching on this site and on the web for the work of evaluation theorists and other evaluators using similar methodologies. Use these in reading about evaluation as a field, in developing your own evaluation designs, or for developing materials for teaching purposes.
INTRODUCTION
The First Year Psychology for the Health Sciences Tutorial Programme is designed to help first year Psychology for the Health Science students understand and discuss the course material in smaller classes which promote interaction and expression. The primary aims of the Tutorial Programme are to develop reading, writing and learning skills within students, and to provide students with individual attention. The objectives of the programme are to help students understand basic psychological principles, to promote critical abilities, to help students understand human behaviour (including their own) and to assist the students in being able to identify where and how psychological theories and principles may be applicable in their own medical careers.
The programme corresponds with the lecture programme of the Psychology course. The programme is both designed and implemented by the Psychology Department in the Faculty of Humanities. Its activities consist of one compulsory 45 minute tutorial every Monday and these tutorials are led by the tutors, who consist of Honours and Masters Psychology students. Activities which are promoted within the tutorials include exercises designed to improve knowledge and understanding, as well as essay and examination revision and referencing guidelines. The Tutorial Programme begins in the month of February of every year and ends in the last week before the final end-of-year examinations. The theory on which the Tutorial Programme was developed and is implemented is the theory of Problem-Based Learning. The fundamental principles of this theory, which are pertinent to the programme’s implementation as well as to its evaluation, are that learning in context increases knowledge, that social interaction leads to development and growth, and that meta-cognitive reasoning and self-directed learning aid in long-term knowledge retention.
Evaluation Design
A LogFrame approach (Sartorius, 1991) will first be used to map the programme’s theory of implementation which summarises the primary facets of the programme. The evaluation will then be conducted in two phases, as follows:
Phase 1: Needs Assessment
The first phase involves a needs assessment (Alkin, 1979; Rossi and Freeman, 1985), the results of which indicates that there are six primary areas within the programme on which an evaluation is both desirable and necessary. These areas are as follows:
Based on the above, the evaluation will take the form of both a formative and a summative evaluation conducted by both an internal and an external evaluator.
Phase 2: Formative Evaluation (internal evaluator only)
Design:
The evaluation design consisted of a flexible multi-paradigmatic approach comprising both qualitative and quantitative methodology. The overall design for the evaluation is based on a synthesis of the Logical Framework Approach as well as a Practical Participatory Approach. A combination of these two evaluation frameworks was chosen so as to allow for a comprehensive and dynamic review of the programme.
The Logical Framework or LogFrame approach was originally developed in the 1970’s to augment the accountability of evaluation results (Sartorius, 1991). It became evident that it was of fundamental importance that the implementers of a programme, and the programme itself, be accountable to the funders and recipients of the programme (Sartorius, 1991). The LogFrame approach thus introduced the idea of a programme matrix which divides a programme and its implementation into separate areas/aspects and consequently allows for an evaluation to be focused on investigating these separate aspects (Gasper, 2000). This approach was chosen to aid in the current evaluation design because of its practicality, its systematic foundation and its inherent basis for adjudication (in that the aims, objectives and outcomes of a programme are the criteria for programmatic judgments) (Gasper, 2000).
The second theory underpinning this evaluation design is the theory of Practical Participatory evaluation. This approach was developed to further the progression of the field of programme evaluation towards the utilization of evaluation results (Cousins & Whitmore, 1998). A primary assumption of this approach is that stakeholders should be involved and ‘participate’ in the evaluation process, by helping with evaluation support, the gathering and analysis of evaluation data, and through guiding the evaluator with their grassroots knowledge and contextually specific understanding of the processes involved in the programme and its implementation (Brisolara, 1998). This approach thus seems particularly useful for the current evaluation design as there have been significant investments as well as firmly expressed hopes for future implementations of the programme by key stakeholders of the programme (namely the programme co-ordinator and the Psychology course co-ordinator). When this is viewed in combination with the programme’s limited size, we can see that participation of stakeholders within this evaluation would encourage ongoing learning throughout the process of the evaluation as well as increase the likelihood that the evaluation results will be correctly utilized for instrumental purposes (to make decisions regarding the programme as were decided upon by key stakeholders prior to the evaluation).
Therefore, while it is evident that both approaches are separately applicable and advantageous to this evaluation design, their utility is substantially increased when applied in a combined evaluation design. Whilst the LogFrame provides an overarching evaluation design, practical participatory evaluation delineates the ways in which stakeholders can participate in investigating the various facets of the matrix and also provides the ways in which these facets can be measured. The inclusion of the participatory approach also helps in transferring evaluation knowledge so that this evaluation of the programme may lead to possible future annual evaluations. In summation, this double-theoried approach to the evaluation design increases the comprehensiveness and expected quality of the evaluation.
Keywords: practical participatory evaluation, J Bradley Cousins, stakeholder evaluation, Robert Stake, Carol Weiss, logical framework evaluation, Rolf Sartorius, USAID, AUSAID.
Read the full case study by going to the closed area of the Downloads Section of the website. Click on Case Studies/Teaching Materials. Then click on the title of Tanya Samouilhan’s case study next to her name. This will take you to Tanya Samouilhan’s front page. You can then link to Tanya Samouilhan’s Case Study, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
Use the key words provided to assist in searching on this site and on the web for the work of evaluation theorists and other evaluators using similar methodologies. Use these in reading about evaluation as a field, in developing your own evaluation designs, or for developing materials for teaching purposes.
INTRODUCTION
The aim of these materials is to introduce students of evaluation to the types of articles, papers and practical manuals which exist in the literature on programme evaluation. Commentary is provided on the origins and theoretical underpinnings of participatory evaluation, as well as its methodologies and applicability. Commentary is also provided on the relevance of these articles to the field of programme evaluation in a South African context.
We provide seven core references, and also refer to others in the accompanying powerpoint. We focus in particular on the relevance of two types of participatory evaluation, practical and transformative, with the aim that the reader can see their relevance and applicability both within South Africa as well as internationally. The articles have been chosen with the aim of illustrating the expectations of an evaluator working on participatory assumptions, and also with the aim of providing basic information to readers interested in using participatory approaches in evaluation.
In addition to a written handout, we provide a powerpoint which summarises the main principles of practical and transformative participatory evaluation. We suggest that participatory programme evaluation has its origins in responsive evaluation and in stakeholder approaches to programme evaluation. We also indicate how participatory assumptions have influenced and shaped other approaches in the field of programme evaluation.
We suggest that practical participatory evaluation is constructivist in character and is usually conducted from an interpretive standpoint. Transformative participatory evaluation, in contrast, is usually based on critical-emancipatory assumptions (Potter, 1998; 2006). The evaluator works to promote a social agenda, working actively to promote social transformation based on democratic values. This form of participatory evaluation can thus be characterized as based within a transformative paradigm (Mertens, 2007).
We believe that participatory evaluation has a contribution to make in South Africa, especially in situations in which programme evaluations can be used to help alleviate some of the social problems and unequal power relations inherent in social problems and their respective programmes. In this way we believe participatory evaluation can contribute both to the development of the field of evaluation within this country as well as more broadly internationally.
Keywords: practical participatory evaluation, J Bradley Cousins, transformative participatory evaluation, transformative paradigm, Donna Mertens.
Read the full set of teaching materials by going to the closed area of the Downloads Section of the website. Click on Case Studies/Teaching Materials. Then click on the title of Tanya Samouilhan’s contribution next to her name. This will take you to Tanya Samouilhan’s front page. You can then link to Tanya Samouilhan’s Teaching Materials, Powerpoint, Biography, List of Publications and hyperlink to other areas of her work by clicking on the buttons on her front page. A poster is also provided under the poster section of the conference which can be used to support a presentation in the classroom using these materials.
Use the key words provided to assist in searching on this site and on the web for the work of evaluation theorists and other evaluators using similar methodologies. Use these in reading about evaluation as a field, in developing your own evaluation designs, or for developing materials for teaching purposes.
INTRODUCTION
Empowerment evaluation is an evaluation approach which entered the evaluation arena in 1993 (Miller & Campbell, 2006). It was initially introduced at the American Evaluation Association conference by David Fetterman. Since its inception, it has been utilized in a variety of contexts including individualistic, organizational, institutional and community-based settings (Fetterman, 1998).
Empowerment evaluation draws its origins from empowerment theory, community psychology, and action anthropology. It emerged in rejoinder to Positivist models which have been criticised as being too value-free in their aim to maintain objectivity (Potter, 1999). Its primary aim is “to help people help themselves” (Fetterman, 1996, p.5). Thus, it endeavours to improve existing policies and programmes, in addition to providing capacitation skills for community growth.
Empowerment evaluation can essentially be classified as a model which falls within the realm of critical-emancipatory approaches to programme evaluation (Potter, 1999) since it aims to challenge the status quo via its recognition of social problems. This set of teaching materials is designed to introduce individuals to empowerment evaluation and, it is intended to be a resource for facilitating an introductory lecture on the topic. The materials consist of a PowerPoint presentation and a set of content notes to assist the facilitator.
Keywords: empowerment evaluation, David Fetterman, Abraham Wandersman.
Read the full set of teaching materials by going to the closed area of the Downloads Section of the website. Click on Case Studies/Teaching Materials. Then click on the title of Bronwynn Sherriff’s contribution next to her name. This will take you to Bronwynn Sherriff’s front page. You can then link to Bronwynn Sherriff’s Teaching Materials, Powerpoint, Biography, List of Publications and hyperlink to other areas of her work by clicking on the buttons on her front page. A poster is also provided under the poster section of the conference which can be used to support a presentation in the classroom using these materials.
Use the key words provided to assist in searching on this site and on the web for the work of evaluation theorists and other evaluators using similar methodologies. Use these in reading about evaluation as a field, in developing your own evaluation designs, or for developing materials for teaching purposes.
INTRODUCTION
This case study is an evaluation of a programme of support offered to teenage girls who have lived and worked on the streets or begged for a living, and who now live in a residential shelter for street girls in Johannesburg.
The evaluation is being conducted in three stages. The first stage examines the social and policy context within which the residential facility conducts its work, in order to provide the context of the study. A needs analysis based on examination of the developmental and psychosocial needs of two samples of girls living at the shelter is also being undertaken. Thereafter, an evaluation of the programme of support at the residential shelter will be carried out.
The data will then be integrated across the three stages of the evaluation design, with the intention of developing a conceptual framework and a programme implementation theory. This will include a set of criteria for assessing to what extent the existing programmes of support at the facility are logical theoretically, relevant contextually and achieving their intended objectives.
Evaluation Design
The evaluation design is theory-based (Jobin, 2008; Weiss, 1972; 1978). The evaluation as a whole is contextualised within the Human Needs paradigm, and Human Needs theory as set out in Max-Neef’s (1992) Human Scale Development matrix. The matrix is used to provide a set of primary codes or core categories which can be used as indicators against which to examine the girls’ needs relative to their quality of life (Testa, 1996).
The evaluation design is also theory-seeking (Rosenthal and Rosnow, 1991). Theory will be evolved through the analysis of the empirical data. Max-Neef’s theories are used to provide an initial set of codes within which to examine the empirical data gathered in the three stages of the evaluation. The intention is to develop a conceptual framework and a programme implementation theory, grounding the theory through the analysis (Strauss and Corbin, 1990a; 1990b; 1997). This will then be used to inform programme development and evaluation both in this particular residential facility as well as in other similar programmes in the future.
The evaluation is guided by the following three questions:
A Multistage Multimethod Investigation based on Data and Time Triangulation
The evaluation is being implemented using a multistage hybrid needs assessment-based evaluation design. This shares many of its assumptions with systematic evaluation approaches based on needs assessment (e.g. Alkin, 1969; 1985; Rossi and Freeman, 1982). It also draws on contextually-based approaches to conducting evaluations (e.g. Stufflebeam, 1968; 1972).
The first stage in the evaluation design follows Stufflebeam in undertaking analysis of context as the initial stage of a multi-stage evaluation. This is then followed by a needs assessment. On the basis of the contextual analysis and needs assessment in the first two stages, the third stage of the evaluation is then conducted.
Each of the three stages essentially has its own design (Rosenthal and Rosnow, 1991). Overall, the evaluation is conceptualized as a multimethod investigation (Teddlie and Tashakkori, 2009), with analysis of evidence in each stage of the design being based on different forms of triangulation (Denzin, 1970; 1978; Denzin and Lincoln, 1998).
In implementing the evaluation, the first two focusing questions are being used to ground the investigation theoretically, and to identify the needs of a particular sample of street girls. This information will then be used to develop a set of indicators relative to conducting the evaluation, which follows from the third question.
The design as a whole is based on the suggestions made by Johnson and Christensen (2007) that time is an important variable in multimethod evaluation. Data are thus being collected from multiple sources at different stages in the investigation. The data from each of these stages will then be integrated at the end of the process to yield the evaluation conclusions, and to develop a conceptual framework for programme design and implementation (Weiss, 1998).
Keywords: street children, human needs paradigm, context evaluation, Daniel Stufflebeam, theory-based evaluation, Carol Weiss, Dennis Jobin, Craig Russon, logical framework evaluation, Rolf Sartorius, USAID, AUSAID, use of multiple methods, Donna Mertens.
Read the full case study by going to the closed area of the Downloads Section of the website. Click on Case Studies/Teaching Materials. Then click on the title of Athena Tudoric-Ghemo’s case study next to her name. This will take you to Athena Tudoric-Ghemo’s front page. You can then link to Athena Tudoric-Ghemo’s Case Study, Powerpoint, Biography, List of Publications and also hyperlink to other areas of her work by clicking on the buttons on her front page.
Use the key words provided to assist in searching on this site and on the web for the work of evaluation theorists and other evaluators using similar methodologies. Use these in reading about evaluation as a field, in developing your own evaluation designs, or for developing materials for teaching purposes.
INTRODUCTION
This case study is a technical report which focuses on the methodological applications used in applying Alkin’s CSE model of evaluation (1969; 1985) in formative and summative evaluation of an interventionary programme. It focuses in particular on the use of time and methodological triangulation for purposes of drawing firm inference from non-experimental data, and highlights certain of the strengths and limitations of combined use of quantitative and qualitative analyses in longitudinal evaluation.
The case study first describes the use of a longitudinal multimethod evaluation design in undertaking an evaluation into the high failure rates in the first year Engineering Graphics course at the University of the Witwatersrand, Johannesburg. It describes the methods used to establish the nature of the problem and the need for an intervention through a series of predictive analyses, the use of action research in establishing a programme designed to overcome the problem, the use of formative evaluation to establish whether the programme was working as intended, and the use of longitudinal evaluation strategies to establish programme effects, as well as continuing needs for the intervention.
Key words: spatial perception, technical education, curriculum evaluation, Joseph Schwab, Decker Walker, Laurence Stenhouse, CSE evaluation model, Marvin Alkin, multiple methods, Donna Mertens, critical multiplism, Thomas Cook, Will Shadish.
Read the full case study by going to the closed area of the Downloads Section of the website. Click on Case Studies/Teaching Materials. Then click on the title of Errol van der Merwe’s case study next to his name. This will take you to Errol van der Merwe’s front page. You can then link to Errol van der Merwe’s Case Study, Powerpoint, Biography, List of Publications and also hyperlink to other areas of his work by clicking on the buttons on his front page.
Use the key words provided to assist in searching on this site and on the web for the work of evaluation theorists and other evaluators using similar methodologies. Use these in reading about evaluation as a field, in developing your own evaluation designs, or for developing materials for teaching purposes.