Evaluation

DH450 : The Class : Evaluation : Evaluation : Evaluation

Evaluation

Program evaluation can be thought of as a type of applied research. The primary purpose is to discover how well programs work and to help in making decisions. Questions that are asked are:

Did the program accomplish what it was designed to do?
Did the program work better than other similar programs?
Did the program reduce health costs?
How could the program be improved?
Should the program be continued?
Does the program merit continued funding?
Should the program be expanded?

Evaluation is an essential part of program planning. The evaluation decisions and tools are designed at the time that the program is planned.

The results of the program are measured against the goals and objectives. The results of the evaluation provide accountability to funding sources, administration, and key stakeholders. The evaluation answers whether the program was successful in reducing or eliminating the identified need or problem. It answers the question, "How well did the program work compared to other programs?"

Link first to this site for an excellent overview of program evaluation.
What Should You Include in a Program Evaluation Plan?

Effectiveness Evaluation:

Does the program do what it was designed to do?

Performance
Outcome
Impact
Eligibility

Performance:

How many patients did you see?
How many procedures did you do?

Outcome:

How successful were changes in client or environment?
What changes occurred from pre to post test ?
What you are changing? (oral hygiene, perio, dental caries.)

Impact:

What in the process matters most?
Are you measuring it?
Mark this and measure it.

Eligibility:

How similar is the target population and client population?
Is the program serving the intended client population?
Are services being utilized enough and appropriately?

Efficiency Evaluation.

Cost Analysis
Cost Benefit
Contract Compliance
Quality Assurance

Cost Analysis

The cost per service (i.e., cost per sealant)
The cost per reduction in need (i.e., $ spent on sealant saves ## dollars in treatment costs.)
The cost per client

Cost Benefit

Untreated cost

The cost for treating a child with early childhood caries is approximately $3,000 per case.
Cost of providing vs. the cost if don't put on program.

Planners look at tangible as well us non-tangible benefits: i.e., pain and suffering, self esteem, overall development.

Contract Compliance

Are qualified personnel being hired for the program?

Quality Assurance

Subjective and Objective
Flow analysis (is client or patient flow matching flow chart?)

Perceptual Evaluation.

General Population
Target Population
Clients
Staff
Agencies
Politicians

When you don't have data to do anything else, you survey these groups:

General population

People only know what effects them directly

Target population

People program is designed to serve

Ask why not being served?

Clients

People who are actually served

What is their perception of the program?

What is their evaluation of the staff?

Staff

Will always blame people who supervise them.

Agencies

Ask for their perception of the program

i.e., for a school-based sealant program, you might assess the perceptions of school staff.

Politicians

They are likely the ones with the purse strings

The following images help demonstrate how different people can have varying perceptions of the same thing.

Figure of a woman

Different backgrounds (colors)

Management Information System

A system which provides the information necessary to manage a program and make decisions.
This can be on paper or computer.
Aim for simplicity (KISS rule)
This should be user friendly

For the homebound program, easy to use forms were designed using Excel software. They help track patient flow, the budget, student requirements, and production of students and professionals. Quarterly reports to the funding agencies can be written using these forms.

Sometimes evaluators use standard instruments to measure changes or differences that programs make.

For example, health promotion programs are often planned with the intent to change knowledge, attitude and behavior (KAB). Hence, evaluation instruments are designed to measure changes in KAB. CAPS instruments) are designed and tested for use for HIV prevention planners and designers. Using standardized evaluation instruments allows for the ability to compare programs.

There are several other standard evaluation tools. Some of these can be found on the Internet.
See Community Health Internet sites.

Standard tools may not be available, especially if a program is innovative. In that case, forms, questionnaires or surveys will need to be developed.

I designed instruments to evaluate a community early childhood caries promotion project. The first form is double sided, designed to assess risk for decay on the first page, and on the second page to track changes in white spots, decay, treatment status, oral hygiene levels, incentives given and numbers and dates of visits. To determine the best time of day, and site for interventions, a form will track numbers of visits at various sites.

Once you have finished the lesson you should:

Go on to Why Evaluate?
or
Go back to Program Evaluation

E-mail Tricia Moore at Tricia.Moore@nau.edu or call (520) 523-4012