  |
The
following are four types of commonly conducted
evaluations.
Needs Assessment: A needs assessment
answers the question, What do we need to do? It accurately
describes the problem or need through the collection of
information from a variety of sources—local and/or
national surveys of students, or interviews with school
staff, community providers, and parents. It can identify
the intended program participants and determine whether
the program should target all or a specific group of students.
It can also help determine realistic and achievable goals
and objectives.
Outcome Evaluation: An outcome evaluation
answers the question, What changed because of the program?
The questions in the evaluation should flow directly from
the program goals and objectives. The types of outcomes
that are targeted will also determine the evaluation methods
to be used, such as surveys, direct observations of behavior,
or focus group discussions.
Process Evaluation: A process evaluation
answers the question, What makes the program work and
why? This includes the answers to these questions: What
does the program look like? How does it operate? What
are its pieces and parts? This evaluation also looks at
whether the program is being implemented according to
the original plan, whether the intensity of the program
has changed over time, and how staff members implement
the program differently—all of which have implications
for program effectiveness and outcomes.
Cost-Benefit Analysis: A cost-benefit
analysis answers the question, Is the program cost-effective?
It might include an assessment of how much the program
costs to implement per individual or per school, or how
much the program saves in relation to other costs (e.g.,
the costs of keeping a child in a regular versus an alternative
school setting). Costs also include such things as materials
and staff time for training, and collection of data from
participants or records.
Programs may not have the need to conduct each type of
evaluation, but they should be aware of the role that
each type can play in providing data on best practice
strategies for violence prevention. |
| |
|
Suggested
Resources
Flannery, D. Improving School Violence
Prevention Programs through Meaningful Evaluation. ERIC
Digest No. 132: ERIC Clearinghouse on Urban Education; 1998.
Flannery, D. and Seaman,
M. “Challenges to Evaluating School and Community-Based
Violence Prevention Programs: A Framework for Action.”
In Evaluating School Violence Programs, ed. E. Flaxman.
(ERIC Clearinghouse on Urban Education; 2001.) |
|
|
Schools and communities around the country are implementing violence
and substance abuse prevention programs with increasing frequency.
This is due both to demands from policy makers for school safety
plans and to the public perception that school violence is increasing.
The No Child Left Behind Act requires enhanced accountability for
results, proven methods, and rigorous scientific research.
According to Daniel J. Flannery, Ph.D., director of the Institute
for the Study and Prevention of Violence at Kent State University,
few resources exist for the non-research-oriented professional who
needs to conduct a basic program evaluation. Not all school staffs
have expertise in program evaluation and not all researchers have
experience in conducting research in schools. Conducting an effective,
efficient, and scientifically valid evaluation of a school violence
prevention program can be difficult work.
There are several basic steps for conducting any evaluation:
Be clear about program goals and objectives. This
first step drives the rest of the evaluation design. What do you
need to change? What is a problem at your school? What do you want
to find out or show when the program is done? Decide how to determine
whether the program achieves what it sets out to do. When will it
be apparent that the program has demonstrated something positive
or worthwhile? Is the focus on behavior change, cost-effectiveness,
the program’s applicability to other groups, or all of the
above?
Determine the scope of the evaluation. This will
depend on how comprehensive the evaluation is, as well as on the
availability of resources. Typically, 10 to 15 percent of a program’s
budget should be allocated for evaluation.
Decide on the research design. There are three
basic components that will make the evaluation results more valid:
collection of data before the program is implemented (to compare
before- and after-effects); assessment of a control group not participating
in the program; and—the most difficult to achieve— random
assignment of participants to either the program or the control
group.
Implement the evaluation. Be sure to revisit the
original research question to make sure that the evaluation has
not strayed from the original program goals and objectives. If the
goal has changed, the evaluation may need to also change.
Analyze the data. Describe what was done, and for
whom. Review the program goals, outcome questions, and program monitoring
issues. Document what the duration, intensity, and frequency of
the program or activity has been.
Disseminate the findings. Consider the audience
who will receive the results (e.g., program funders, project staff,
parents) to determine the most effective means of describing and
disseminating the data.
It is important to recognize the advantages of conducting an evaluation,
even when implementing one of the identified “best practices”
programs previously evaluated by some other group. The potential
benefits that program evaluation provides, according to Dr. Flannery,
are:
Understanding of the population served. Every school and
community is different—participants will vary by ethnicity,
age, family structure, neighborhood, socio-economic status, and
individual experiences. The impact of a program will vary accordingly.
Understanding of the services provided. What is
developed on paper is seldom a complete reflection of what happens
in reality. Even the best-designed program will work differently
in one school than it did in another.
Improvement of the services. Evaluation data that
is provided back to programs on a regular basis can be used to modify
or improve services.
Data for collaborators. Prevention initiatives
are increasingly collaborative and complex. Effectiveness requires
participation by all partners because of the differing impact each
discipline has on projected outcomes.
Data to support funding of the program. With increasing
competition for limited resources, programs must be able to provide
objective data demonstrating effectiveness.
Enhanced scientific knowledge. Despite the growth
in the number of school-based violence prevention programs, well-designed
evaluation studies of program effectiveness are still lacking.
Conducting a program evaluation can be hard work, but a well-designed
and implemented evaluation will increase the capacity for all schools
to make better use of limited resources, and broaden the base of
best practices for violence prevention across the country.
Daniel J. Flannery, Ph.D., is the director of the Institute for
the Study and Prevention of Violence at Kent State University. This
article was adapted from his presentation, “Techniques and
Challenges to Evaluating School- and Community-Based Violence Prevention
Programs,” 2002 National Technical Assistance Meeting, Washington,
DC.
|
|
