Skip to main content
NC OSBM logo NC OSBM

Topical Navigation

  • Home
  • Budget
    Budget
    • Budget 101
    • Budget Manual
    • Budget Instructions
    • Budget Memos
    • Governor's Budget Recommendations
    • Certified Budget
    • NC IBIS
    • Training Resources
  • Facts & Figures
    Facts & Figures
    • Population & Demographics
    • Economy
    • Geography
    • State Data Center
    • LINC
    • State Demographer Data
  • Operational Excellence
    Operational Excellence
    • Strategic Planning
    • North Carolina Results First
    • Evaluation Fund Grants
    • Regulatory Analysis & Impact Analyses
    • NC Office of Strategic Partnerships
    • Performance Management Academy
    • Governor's Advisory Committee on Performance Management
    • Case Studies
  • Stewardship Services
    Stewardship Services
    • Directed Grants
    • Internal Audit
    • Report Fraud, Waste or Abuse
    • Grants Management
    • OPEB
    • Reports to the NC General Assembly
    • Fee Reports
    • Cost Collections Report
    • Archived Reports
  • About OSBM
    About OSBM
    • Leadership Team
    • Contact Us
    • Upcoming Events
    • Employment Opportunities
    • News
  • NC.GOV
  • AGENCIES
  • JOBS
  • SERVICES
NC OSBM »   Operational Excellence »   North Carolina Results First »   Evidence Definitions

Evidence Definitions

Tiered Levels of Evidence

The Office of State Budget and Management created the following tiered levels of evidence. Each of the definitions below outline the criteria needed to qualify for each level of evidence. The defined tiers of evidence are intended to be used in implementing the NC Results First Initiative and to support budget proposals for selected programs and service areas.

These definitions will be most applicable to agencies that cover select policy areas. As an example, Results First currently has the following policy areas: crime & delinquency, child and family well-being, education, employment and job training, mental health, public health, sexual behavior & teen pregnancy, and substance use.

Establishing a standard framework and terminology for tiers of evidence is an important step toward building a strong system of evidence-based policymaking. Government leaders and other stakeholders can use these definitions to identify what works and to ensure their limited funds are invested in programs that will achieve strong outcomes. The tiers of evidence are ordered based on the direction of impact, with positive impact at the top and negative impact at the bottom.

Tiered Levels of Evidence

Proven Effective

A service or practice that is proven effective offers a high level of research on effectiveness for at least one outcome of interest. This is determined through multiple qualifying evaluations outside of North Carolina or one or more qualifying North Carolina-based evaluations. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs.

Promising

A promising service or practice has some research demonstrating effectiveness for at least one outcome of interest. This may be a single qualifying evaluation that is not contradicted by other such studies but does not meet the full criteria for the proven effective designation. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs.

Theory-based

A theory-based service or practice has no research on effectiveness or research designs that do not meet the standards for “promising” or “proven effective.” These services and practices may have a well-constructed logic model or theory of change that has not been tested. This ranking is neutral. Services may move to another category after research reveals their causal impact on measured outcomes.

Mixed Effects

A mixed effects service or practice offers a high level of research on the effectiveness of multiple outcomes. However, the outcomes have contradictory effects, and there is not additional analysis to quantify the overall favorable or unfavorable impact of this service. This is determined through multiple qualifying studies outside of North Carolina or one or more qualifying North Carolina-based evaluations. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs.

No Effect

A service or practice with no effects has no impact on the measured outcome. It does not include the service’s potential effect on other outcomes. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs.

Proven Harmful

A service or practice that is proven harmful offers a high level of research that shows participation adversely affects outcomes of interest. This is determined through multiple qualifying evaluations outside of North Carolina or one or more qualifying North Carolina-based evaluations. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs.

Resources:

  • Tiered Level of Evidence Handout

North Carolina Results First

  • Adult Mental Health
  • Child and Family Health
  • Evidence Definitions
  • Juvenile Justice
  • Reports

Share this page:

  • Facebook
  • Twitter
  • Email

How can we make this page better for you?

Back to top

Contact Information

North Carolina Office of State
Budget and Management

Mailing Address: MSC 20320, Raleigh, NC 27699-0320
Physical Address:
116 W. Jones Street, Raleigh, NC 27603

984-236-0600

Website Feedback Form

Report Fraud Hotline

Fraudulent activities should always be reported to your local law enforcement office. The following is additional information on how specific types of fraud complaints or cases of suspected fraud can be submitted to state agencies.

Visit the Hotline!

Follow Us on Twitter

Tweets by NCDemographer

  • Employee Directory
  • Accessibility
  • Terms of Use
  • Privacy Policy
  • Open Budget
NC OSBM
https://www.osbm.nc.gov/operational-excellence/north-carolina-results-first/evidence-definitions