This article was written by Christy Sullenberger, MS, Director of Member Services, and Rebecca J. Davis, Ph.D., Executive Director of NCHN, for the “Networking News” monthly newsletter. The Network Technical Assistance Project is funded by the Federal Office of Rural Health Policy, Health Resources and Services Administration, U.S. Department of Health and Human Services through a contract to Rural Health Innovations, LLC, a subsidiary of the National Rural Health Resource Center.

Evaluating the network organization or a specific network program is an essential element of performance and process improvement, as well as overall assessment of effectiveness. Evaluations are used to improve programs, build organizational capacity, demonstrate value, and provide a basis for decision-making. While there is no set method for network evaluation, an evaluation typically addresses inputs, processes, outputs, and outcomes. Each of these elements provides insight into a different aspect of the network organization. Taken together, evaluation is a useful tool for the network leader, Board, stakeholders, and present and future funders.

A well-executed evaluation will also provide a statement of value, which is notoriously difficult for many health networks. Historically, health networks can illustrate positive value and promising programs, but quality quantitative data is lacking. In order to provide a good illustration of a network program’s impact, and ultimately the value of the network an evaluation plan should be developed and executed. The process should begin by determining clearly stated and measurable goals for the program, define metrics (how will you measure the impact of project), collect data, analyze the data against program goals and objectives, and then illustrate the connection of the program’s outcomes to the network’s value.

An evaluation should show a direct link between the program activities and outcomes and should address a number of questions, including:

• What do I need to know to make program decisions and adaptations?
• What is working well and what is not?
• How well does the program deliver value to members and stakeholders?

Potential funders, including private foundations and governmental agencies, are seeking justification for the investments they make in rural health networks. A good evaluation plan, along with previous outcomes, assists them in achieving this goal. A good example of the importance of evaluation to HRSA Rural Health Network Development Programs, is stated in a recent RFP Guidance,

“Evaluation is a very important component of the RHND Program. The collection of performance measures from past RHND cohorts and numerous rural health network case studies demonstrated positive outcomes. But, due to the lack of evidence and challenges using traditional quantitative methodologies to measure network outcomes, it is difficult to ascertain the significance and uniqueness of rural health networks that support positive health outcomes in rural communities. Project-level evaluations of RHND grantees will assist in determining and validating the reasons why rural health networks are an important strategy in the improvement of rural healthcare. A comprehensive evaluation approach should contain contextual, implementation and outcome evaluative components. And the process and result of evaluation should not only assist in the understanding of the benefits of rural health networks but be utilized in a manner that enhances and improves the functions and activities of the network.” (p.5, Rural Health Network Development Program Funding Opportunity Announcement, FY 2014, Health Resources and Services Administration, Office of Rural Health Policy (ORHP))

Rural Health Network Evaluation: When and How?

As part of your rural health network development grant proposal, you have already outlined program activities and expected outcomes. Prior to developing the proposal, you likely completed a needs assessment and know exactly what your vision is for the implemented project. You will now need to execute your evaluation plan. The evaluation approach you choose guides you in the collection and organization of data, so it is important to develop an evaluation plan early in the process of implementation. In addition, as you begin collecting data, you can conduct an ongoing evaluation, which determines if implementation is going as expected. This will lead to the final evaluation of the project and will determine if the stated objectives were met.

An evaluation may be goals-based, process-based (formative), outcomes-based (summative), or a combination of these. A goals-based evaluation determines whether you are meeting your overall objectives. A process-based evaluation addresses how your program works and highlights strengths and weaknesses. An outcomes-based evaluation addresses the benefits of your program to network members and/or the community. A final evaluation will often include pieces of all of these approaches.

Preparation

Before developing your evaluation plan, it is essential to sit down and re-summarize your proposed program’s activities and objectives. One common guideline for developing goals and objectives is the SMART acronym. All objectives should be Specific, Measurable, Attainable, Realistic, and Time bound. Ensure that you have a strong list of realistic goals, achievable objectives, and appropriate activities that link directly to the desired results. Consider the purpose of your evaluation, outline the specific questions that the evaluation will answer, and decide how and how often data will be collected. You may want to consider using a logic model to assist in planning (see more at The Logic Model). Once this template is developed, you can use it to identify specific metrics, determine the kind of data that you want to collect, and choose an evaluation approach that is appropriate to the project.

Evaluation Methods

There are a variety of evaluation methods and models. Each method has strengthens and weaknesses. The important question to answer, is which method will best provide actual data that can be used to determine the effectiveness of the proposed project. An evaluation plan for the proposed project may need to incorporate different approaches.  Some basic methods of program evaluation include the following:

• Questionnaires and surveys: can be analyzed and presented numerically/quantitatively
• Interviews: provide primarily qualitative outcomes and can be conducted in person or on the phone and should be targeted and clear
• Documentation review: can be inexpensive, but may not provide a complete picture
• Focus groups: can provide a range of feedback, but may be slightly difficult to present analytically
• Case studies: can provide an in-depth look at a program and many variables
• Others (see Overview of Methods to Collect Information for a list of  primary methods and the pros and cons of each)

The guiding principle in the selection of an evaluation method/s is to collect and present the most useful information about a program. Throughout the process, keep in mind that the method you choose will determine how the results are collected. And, in order to avoid introducing bias, you will need to develop a process that ensures that data is collected in the same way each and every time.

Common Components of an Evaluation Report  

When you have identified your method/s and collected data, you will then need to communicate your findings. There are many ways to structure the evaluation, and organization and content will depend on your process and methodology, but the following is a common structure of an evaluation:

1. Title page
2. Table of contents
3. Executive summary
4. Purpose of the evaluation
5. Organization and program background
6. Overall evaluation goals
7. Methods used
8. Interpretations and conclusions
9. Recommendations
10. Appendices

For more detail on the above, see Contents of an Evaluation Plan. As you embark on your program evaluation and tie it into your network assessment, keep in mind that the most important element in the evaluation process is that you start early in the process of implementation and that you are consistent in your collection methods.

Additional Reading

Basic Guide to Outcomes-Based Evaluation for Nonprofit Organizations with Very Limited Resources (Free Management Library)

Basic Guide to Program Evaluation (Including Outcomes Evaluation) (Free Management Library)

Critical Components of Evaluation by Alana Knudson, Ph.D., National Rural Health Resource Center Evaluation Workshop (August 6, 2014)

Designing Evaluations (GAO/PEMD-10.1.4, United States General Accounting Office - Program Evaluation and Methodology Division, 1991)

Evaluation for Nonprofits (Nonprofit Answer Guide)

Tools and Resources for Assessing Social Impact (TRASI) (Foundation Center)

Tools and Strategies for Managing Health Networks: Network Evaluation (NCHN)