What is evidence and how should we use it?

At Research4Schools (R4S), we believe that education practices should be based on evidence. But what is evidence? The Every Student Succeeds Act (ESSA) considers evidence to be the findings from formal research studies. The law sets standards for what constitutes high-quality evidence, with randomized experiments being the gold standard. However, there are a number of other forms of evidence that schools and districts can rely upon that meet the criteria set forth by ESSA.

How should we use evidence? We conceptualize evidence-use as falling on a continuum that ranges from no or merely anecdotal engagement with research on one end, to deep inclusion of scientific research on the other (see Depth of Research Use in our conceptual framework).  Evidence-use is often defined by the education community as either a) the extent to which schools implement programs based on scientific research (e.g. are implementing a “proven” curriculum), or b) the practice of incorporating evidence, broadly construed, into decision-making processes (Hood, 2003).

While there is value in both perspectives, R4S focuses on the latter, with particular interest in instrumental uses of research—situations in which practitioners can cite specific ways that research evidence was used in decision-making, regardless of where their engagement with research falls on the “depth of use” continuum. Therefore, our definition of what constitutes evidence-use is broad and extends beyond the definitions used in federal guidelines such the No Child Left Behind Act of 2001 (NCLB) and ESSA.

Differences between ESSA and NCLB

States submitting and amending plans to the U.S. Department of Education under ESSA should be aware of important differences between the new law and NCLB regarding how research is defined. Previously, NCLB placed a heavy emphasis on scientifically-based research, indicating only “a preference for random-assignment experiments” in impact evaluations of programs or policies (NCLB, 2001).

When ESSA replaced NCLB in December 2015, it granted states more flexibility in designing and implementing educational evaluations but asserted a clearer desire for the use of programs already proven to be effective. The legislation replaces “scientifically-based” with “evidence-based” programs, defined as any “activity, strategy, or intervention that shows a statistically significant effect on improving student outcomes or other relevant outcomes” (ESSA, 2015). Unlike NCLB, ESSA uses a tiered system to order the rigor of the research, with Tier 1 being most rigorous:

• Tier 1, Strong Evidence: Randomized Experiments
• Tier 2, Moderate Evidence: Quasi-Experiments
• Tier 3, Promising Evidence: Correlational Studies
• Strong Theories: Require ongoing self-evaluation

With this guidance, the burden of determining the quality of evidence now largely falls on schools and districts.

Resource: Compare NCLB and ESSA Definitions of Research Evidence

recommendations for states

ESSA affords a unique opportunity to rethink how states have approached education reform and accountability in the past. It is an opportunity to transform programs, to plan for sustainable and equitable change, and to capitalize on all of the assets that schools, districts, community organizations, and institutions of higher education have to offer. Together with the University of Delaware Partnership for Public Education (PPE), R4S offers the following recommendations to states for incorporating evidence-based research into their ESSA plans.

Tiers of evidence

A central feature of ESSA is its emphasis on tiers of evidence underlying local education agency (LEA) improvement plans. In response to the need for criteria to evaluate the underlying evidence, we recommend first that the state department of education assemble an advisory board that includes experts in evaluating and/or implementing evidence-based interventions to build local and state capacity for, and support meaningful implementation of, ESSA provisions related to evaluating and improving LEA plans.

• Implement a process for continuous evidence-use. We recommend that LEA’s develop a process to engage in effective, continuous evidence-use, beginning with identifying where needs are greatest and where evidence might support improvement. The U.S. Department of Education recommends that LEA’s use the following protocol:

  

• Strive for a body of evidence. While ESSA requires “at least one study” on an intervention to provide strong evidence, moderate evidence, or promising evidence, state education agencies (SEAs), LEAs, and other stakeholders should consider a comprehensive body of relevant evidence to ensure a robust use of research in the planning process.
• Review LEA evidence-use. The state DOE, either on its own or with support from an advisory committee, should review the quality of evidence of LEAs’ plans using available tools and guides, such as the What Works Clearinghouse (WWC). This can be done by evaluating the quality of the evidence along with assessing fit to school/district.
• Evaluate strong theories through partnerships. If evidence underlying an LEA plan does not fall in Tiers 1–3, we recommend that the LEA be required to partner with an evaluation or research group that will measure the effectiveness of the plan, contributing not only to local improvement but to the knowledge base that can then be utilized by other schools under improvement.
• Create a state-approved list of evidence-based interventions. The state DOE should create an ongoing list of state-approved, evidence-based interventions that serves as a resource, but not a mandate, for comprehensive or targeted support and improvement. A preliminary list can pulled from resources such as WWC, but should be updated annually, with schools and other agencies encouraged to add to the list.
• Train staff in kinds of evidence. We strongly recommend professional development for LEA and state DOE staff on the tiers of evidence that provides examples of each and how to interpret research relative to rigor and needs.
• Hire staff dedicated to do the work. Additional capacity building should include hiring internal research staff within state DOE to lead work in evidence use and to engage in partnership-building with the state’s institutes of higher education (IHEs) and other organizations to better connect research expertise to the work and goals of the state DOE.

Don’t stop there.

Throughout each section of a state’s plan, formative and summative evaluation—coupled with strategies for piloting and scaling up—are essential ingredients in comprehensive improvement and should be required components of all local and state initiatives. This will require additional capacity but is necessary for meaningful improvement. States should look to their institutions of higher education to play a leading, constructive role in conducting such studies.

ESSA Resources

Reference: Hood, P.D. (2003). Scientific Research and Evidence-Based Practice. Retrieved from https://pdfs.semanticscholar.org/f52d/d63172a7c98449a8b824c85a93061fd192fe.pdf