Continuous Improvement and Evidence Building

Nonprofit organizations (NPOs) face high demand from communities for services and increasing expectations from funders and policymakers for evidence of impact. For many years, traditional evaluations, such as randomized control trials (RCT), have been used to demonstrate the impact of programs. This evidence building method, if done prematurely, can often leave NPOs with findings of minimal or, in some cases, negative impacts on program participants. This can result in a misrepresentation of program impacts, and sometimes in disinvestment from funders.

Project Evident recommends an approach to evidence building that includes continuous program improvement with small scale tests of change to innovate and improve on the design or delivery of an intervention. Making many intermediate improvements prepares an organization for a successful large-scale impact evaluation, when needed. By approaching evidence building as an ongoing process instead of the all-too-common "one study at a time" approach, practitioners avoid delayed findings and constrained adaptations.

Common Struggles

We find that many organizations that struggle with the “one study at a time” approach often have unstructured methods for data collection and analysis, lack of timely data, and face a disconnect between improvement efforts and program decision making. Too often, the program improvements being tested are hypothesized and immediately scaled across an entire program model without adequate use of data to test that the improvement truly had an impact.

An improvement is always a change, but a change is not always an improvement.

At the other end of the data-use spectrum, some organizations ask evaluation teams to lead program improvement efforts. This can lead to initiatives and tests that are slowed due to lack of organizational buy-in and participation, and often change ideas that are not well-informed by practice. Project Evident believes practitioners too often are in the caboose of evidence building. It is important to empower practitioners---whose daily work will improve through using timely data, and empower them to be in the driver’s seat, determining program improvement activities that would most benefit their work.

At Project Evident, we see ideal evidence building as continuous, with data being  used for program improvements and to demonstrate program impact. This is important for several reasons:

1.   Evidence building at NPOs should not occur infrequently or in a vacuum; it must be a part of the culture of an organization. Building evidence constantly, internally and externally, through structured program improvement methodologies and traditional evaluation methodologies helps to create a data and learning culture where data is being used continuously to support decision making and to demonstrate impact.

2.   Evidence building should include a structured approach to test improvements that is low risk, small scale, and uses measurement to act on ideas with promising evidence of improvement

3.   Evidence building internally and externally helps NPOs spread and scale what works and accelerate the impacts of their programs to achieve desired outcomes for their beneficiaries. Many evidence-building activities can be done internally---for example, program improvement initiatives and early, small-scale impact studies. Internal evidence should be used by organizations to develop and improve their programs, and to orient organizations on their evidence-building journey.

Project Evident supports NPOs in identifying and tailoring evidence building approaches perfectly suited for their unique organizational context, to construct or strengthen their evidence building continuums. For program improvement, Project Evident relies on practitioner-led Improvement Science methodologies as a structured approach for internal evidence building. Improvement Science or Improvement methodologies such as Continuous Quality Improvement(CQI) have been used in several sectors to systematically identify, describe, and analyze the strengths and weaknesses in an organization or system; and then test, implement, learn from, and revise test of changes. More simply, one can describe Improvement work as an ongoing cycle of collecting data and using it to make decisions to gradually improve program processes.

Continuous Quality Improvement and evidence building should provide structure for the organization. Some key questions that can guide this process are: (1) What are you trying to accomplish? -- What are your organization’s SMART goals connected to your Theory of Change and/or strategic plans (2) How will you know it is an improvement?-- Relevant and timely measurement using qualitative and quantitative data to demonstrate improvement (3) What changes can we make that will result in improvement? -- What ideas do you have that you can test? Potential improvements are then tested using improvement cycles to provide a structured approach to using data for learning in a low risk way. There are several methods for conducting improvement cycles but each method helps the organization to do the following:

• Establish a plan for implementing an improvement strategy
  This should include: (a) hypothesis for what the change should accomplish; (b) plan for implementation -- when, where and how will the change be implemented; (c) what are the indicators of success and how will they be measured.
• Introduce the change and collect data
  This involves implementing the change and collecting data according to the plan. The data collection should include documentation of implementation challenges and any unintended consequences.
• Analyze the data and summarize learnings
  This involves analysis of data that is relevant to the test, which could be quantitative  or qualitative (e.g. feedback from participants or staff). When possible, consider using comparison groups for tests of change.
• Use the results to inform decisions about further testing and improvements
  A helpful improvement practice is to frame program decision-making as adopt-- the test was a success, conduct again in a different context or group, adapt-- the test needs some small tweaks, test again or abandon-- it is clear that the test is not going to be an improvement, abandon and try a more promising test of change

The final step in our evidence building continuum is traditional impact evaluations. As previously mentioned, these methods have been used for many years by NPOs to demonstrate program impact.  Impact evaluations can be used to rigorously evaluate the desired outcomes of a program using a variety of methods, including  pre-post tests, Quasi Experimental Designs (QED), and Randomized Control Trials (RCT).

What is important to note here is the internal program improvement evidence building that has occurred along the continuum gives NPOs confidence in what they can expect from the results of an impact evaluation. Continuous evidence building reduces the all-around risks of conducting an expensive and time-consuming impact evaluation, both by ensuring that the evaluated program has been improved substantially prior to more formal evaluation, and by ongoing data collection and review, so that more information is known beforehand.

User Case:

Project Evident has supported continuous evidence building with many of the direct partner work we have done. Several of these organizations are what we would call constructively dissatisfied and seek to use data and evidence building more consistently and effectively in their work, but often need a little extra support and guidance building the muscle for continuous evidence building.

For example, we worked with an organization with a strong organizational culture of using data for learning, that sought to use more program improvement strategies. This organization wanted to be able to better use data to inform program work and continuously test improvements and innovations in an evidence based way.

Project Evident initially worked with the organization to create a Strategic Evidence Plan (SEP) that included recommendations around program improvement. One year later, the organization reached out for support because they were having challenges successfully implementing the program improvement recommendations in a structured way and sustained way.

In order to help this organization Project Evident worked with them to identify their key challenges which included:

• The organization was trying to test big long term changes and measure success using long-term outcomes.
• Only managers & up received prior training in improvement
• Roles and responsibilities were unclear and improvement efforts were not being followed through
• Data being used to study the improvements were often custom reporting and burdensome, making it hard to use for real-time decision making
• Test of change were going on for too long and learning was not happening in a timely and relevant manner

It was clear from the assessment that the organization could use support building more structure to their continuous evidence building efforts. To address these challenges Project Evident delivered a 1.5 day training with a large session spanning the organization, a management session on nurturing a Culture of Learning, and Measurement & Learning team session on supporting improvement efforts.  We encouraged the organization to collect, analyze and monitor data often through regular meetings.

Project Evident would consider an organization like this as a best case example. They already had a strong data and learning culture with a team already using data consistently. If they struggle with having a structured approach to continuous improvement and evidence building, it is fair to assume other organizations are also struggling with the same challenges. Project Evident is dedicated to supporting NPOs who seek to be data informed and delivering high impact programs to their communities.