If something Actually Worked in Education… 4 things to Look For

“Education” is a huge conversation, but almost all of the conversation is about the education problem. There is not a serious or mature conversation about solutions.

Meanwhile lots of resources and, as important, attention, are going into an ever-changing, wide variety of programs which, it is hoped, will be part of some solution.  How do we know when we’re looking at any given program, whether it can have game-changing impact on K-12 schools, or has no chance? I call the former, programs that “Actually Work”.

Here I identify and describe four characteristics of any program that Actually Works: Scope, Results, Robustness, and Scalability. Failure at any one of these is failure to Work.

1 – Scope

A program that Actually Works must cover or affect an entire course curriculum. For example, let’s think of mathematics. A program that Actually Works can be as small as one grade level, or one course. But the scope can’t be less than that — it can’t be just a specific concept — because you need to be able to Tell if something Actually Works, and that takes an end-of-course assessment. And what Actually Works needs to help educators and students with what they are held accountable to: summative assessments. So, in mathematics, a cool interactive visualization of parabola parameters on a graphing calculator doesn’t meet the Scope requirement.

2 – Results

To Actually Work, a program needs Positive Results. As stated above in Scope, the results need to be measured using a summative assessment. The Positive Results need to be essentially this: for educators and teachers using the program, all the students become Proficient in the given subject area. I’m willing to accept Proficient as defined by states as meeting state standards on a state standardized test, and “all” as 90%+, but a more general way to define Proficient is: functionally competent to create work product, or solve problems. So, do these Positive Results need to pass a clinical, random-assignment gold-standard evaluation to scientifically attempt to identify the Cause? No. For a program that Actually Works, the results will speak for themselves, robustly at scale in any real world setting, and there need be no argument about what caused the results. Empirically, on implementing the program, you get the Positive Results. In a wonderfully luxurious later phase of this education conversation, when multiple programs Actually Work, we can fret about what we don’t yet know. We can fret over whether extra hours made the difference, or what component came from human or material or process components of the program, or motivation, or factors external to a school. We can fret over whether one program that Actually Works has been properly compared to another program that Actually Works. And we can fret over exactly how do we evaluate the answers to those questions. But the breakthrough step is to have “one” exemplar program that Actually Works.

3 – Robustness

If a program Actually Works, it will work for any educator and any student.  A program that only works for some educators — for the most motivated, or the most experienced, or the most skilled — does not Actually Work. A program that Actually Works needs to work for a first-year teacher with low confidence, right? I mean if it doesn’t work in that situation, then we have a Big Problem, right? And a program that Actually Works has to work with every student subgroup. In the area of math, for example, that means that it works for students who are multiple grades below grade-level. It works for students who have been testing low. It works for students who don’t get much help at home on math. It works for students who have low confidence in their math abilities. It works for students who are suffering from low motivation to do well. And it works for students who are low in language arts proficiency, and students who are learning English.

And the educator or student, for whom the program Actually Works, can be in any school, any school district, in any city and in any calendar year. In other words, Positive Results should be replicable across geography and time.

4 – Scalability

If a program that Actually Works can’t Scale, then it is of no use. By Scale I mean reach millions of students. And there are three aspects of scaling: Implementability, Economics, and Speed.

First Scalability Aspect: Implementability. The program needs to be able to be implemented across the dominant education model. In other words, it must fit within the existing structure of facilities, people, and time. The program should *not* then be requiring some additional changes, such as re-tooling school buildings; re-tooling school hardware; re-tooling school personnel; requiring home technology, or changing the time or location of education. Another consideration for scalability is that the program must not only be do-able by teachers, but also be embraced by teachers. Anything less than a full embrace by teachers will result in failure, failure to achieve Positive Results, either from outright rejection of the program, or by spotty or sub-par implementation of the program.

Second Scalability Aspect: Economics. The program needs to fit within current school economics. This means both for initialization of the program as well as for sustaining the program. For start-up, there is not only the program outright cost but also the costs for site facilitization (see above) and especially in teacher time for training. Outright cost needs to be in the neighborhood of current instructional materials costs per student. Facilitization costs need to be near zero. And teacher training time costs need to also be near zero, as PD days and substitute days become artifacts of the past. Sustaining costs (e.g. annual renewals) need to be low enough to survive tough priority battles during even the toughest budget years (see above: teachers must embrace and highly value any program, for it to be Scalable).

Third Scalability Aspect: Speed. If there is a program that Actually Works with all of the above, to be worthwhile it must also have the characteristic of quick scale-up. By quick I mean that it must have the capability, if the market demands it, of adding millions of students and their teachers per year to the program. In a country with 49 million students and 3.5 million teachers in K-12, anything less than this means we have no solution at all.

About these ads
Tagged

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 174 other followers

%d bloggers like this: