Evidence Definitions The Office of State Budget and Management created the following tiered levels of evidence. Each of the definitions below outline the criteria needed to qualify for each level of evidence. The defined tiers of evidence are intended to be used in implementing the NC Results First Initiative and to support budget proposals for selected programs and service areas. These definitions will be most applicable to agencies that cover select policy areas. As an example, Results First currently has the following policy areas: crime & delinquency, child and family well-being, education, employment and job training, mental health, public health, sexual behavior & teen pregnancy, and substance use. Establishing a standard framework and terminology for tiers of evidence is an important step toward building a strong system of evidence-based policymaking. Government leaders and other stakeholders can use these definitions to identify what works and to ensure their limited funds are invested in programs that will achieve strong outcomes. The tiers of evidence are ordered based on the direction of impact, with positive impact at the top and negative impact at the bottom. Tiered Levels of Evidence Proven Effective A service or practice that is proven effective offers a high level of research on effectiveness for at least one outcome of interest. This is determined through multiple qualifying evaluations outside of North Carolina or one or more qualifying North Carolina-based evaluations. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs. Promising A promising service or practice has some research demonstrating effectiveness for at least one outcome of interest. This may be a single qualifying evaluation that is not contradicted by other such studies but does not meet the full criteria for the proven effective designation. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs. Theory-based A theory-based service or practice has no research on effectiveness or research designs that do not meet the standards for “promising” or “proven effective.” These services and practices may have a well-constructed logic model or theory of change that has not been tested. This ranking is neutral. Services may move to another category after research reveals their causal impact on measured outcomes. Mixed Effects A mixed effects service or practice offers a high level of research on the effectiveness of multiple outcomes. However, the outcomes have contradictory effects, and there is not additional analysis to quantify the overall favorable or unfavorable impact of this service. This is determined through multiple qualifying studies outside of North Carolina or one or more qualifying North Carolina-based evaluations. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs. No Effect A service or practice with no effects has no impact on the measured outcome. It does not include the service’s potential effect on other outcomes. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs. Proven Harmful A service or practice that is proven harmful offers a high level of research that shows participation adversely affects outcomes of interest. This is determined through multiple qualifying evaluations outside of North Carolina or one or more qualifying North Carolina-based evaluations. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs. Resources: Tiered Level of Evidence Handout
Evidence Definitions The Office of State Budget and Management created the following tiered levels of evidence. Each of the definitions below outline the criteria needed to qualify for each level of evidence. The defined tiers of evidence are intended to be used in implementing the NC Results First Initiative and to support budget proposals for selected programs and service areas. These definitions will be most applicable to agencies that cover select policy areas. As an example, Results First currently has the following policy areas: crime & delinquency, child and family well-being, education, employment and job training, mental health, public health, sexual behavior & teen pregnancy, and substance use. Establishing a standard framework and terminology for tiers of evidence is an important step toward building a strong system of evidence-based policymaking. Government leaders and other stakeholders can use these definitions to identify what works and to ensure their limited funds are invested in programs that will achieve strong outcomes. The tiers of evidence are ordered based on the direction of impact, with positive impact at the top and negative impact at the bottom. Tiered Levels of Evidence Proven Effective A service or practice that is proven effective offers a high level of research on effectiveness for at least one outcome of interest. This is determined through multiple qualifying evaluations outside of North Carolina or one or more qualifying North Carolina-based evaluations. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs. Promising A promising service or practice has some research demonstrating effectiveness for at least one outcome of interest. This may be a single qualifying evaluation that is not contradicted by other such studies but does not meet the full criteria for the proven effective designation. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs. Theory-based A theory-based service or practice has no research on effectiveness or research designs that do not meet the standards for “promising” or “proven effective.” These services and practices may have a well-constructed logic model or theory of change that has not been tested. This ranking is neutral. Services may move to another category after research reveals their causal impact on measured outcomes. Mixed Effects A mixed effects service or practice offers a high level of research on the effectiveness of multiple outcomes. However, the outcomes have contradictory effects, and there is not additional analysis to quantify the overall favorable or unfavorable impact of this service. This is determined through multiple qualifying studies outside of North Carolina or one or more qualifying North Carolina-based evaluations. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs. No Effect A service or practice with no effects has no impact on the measured outcome. It does not include the service’s potential effect on other outcomes. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs. Proven Harmful A service or practice that is proven harmful offers a high level of research that shows participation adversely affects outcomes of interest. This is determined through multiple qualifying evaluations outside of North Carolina or one or more qualifying North Carolina-based evaluations. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs. Resources: Tiered Level of Evidence Handout
