Category Archives: Policy

Why Not: 3 Ingredients Enable Universal, Annual Digital Program Evaluations

This post originally appeared in an EdSurge guide, Measuring Efficacy in Ed Tech. Similar content, from a perspective about sharing accountability that teachers alone have shouldered, is in this prior post.

Curriculum-wide programs purchased by districts need to show that they work. Even products aimed mainly at efficiency or access should at minimum show that they can maintain status quo results. Rigorous evaluations have been complex, expensive and time-consuming at the student-level. However, given a digital math or reading program that has reached a scale of 30 or more sites statewide, there is a straightforward yet rigorous evaluation method using public, grade-average proficiencies, which can be applied post-adoption. The method enables not only districts, but also publishers to hold their programs accountable for results, in any year and for any state.

Three ingredients come together to enable this cost-effective evaluation method: annual school grade-average proficiencies in math and reading for each grade posted by each state, a program adopted across all classrooms in each using grade at each school, and digital records of grade average program usage. In my experience, school cohorts of 30 or more sites using a program across a state can be statistically evaluated. Once methods and state posted data are in place, the marginal cost and time per state-level evaluation can be as little as a few man-weeks.

A recently published WestEd study of MIND Research Institute’s ST Math, a supplemental digital math curriculum using visualization (disclosure: I am Chief Strategist for MIND Research) validates and exemplifies this method of evaluating grade-average changes longitudinally, aggregating program usage across 58 districts and 212 schools. In alignment with this methodological validation, in 2014 MIND began evaluating all new implementations of its elementary grade ST Math program in any state with 20 or more implementing grades (from grades 3, 4, and 5).

Clearly, evaluations of every program, every year have not been the prior market norm: it wasn’t possible before annual assessment and school proficiency posting requirements, and wasn’t possible before digital program usage measurements. Moreover, the education market has greatly discounted the possibility that curriculum makes all that much difference to outcomes, to the extent of not even trying to uniformly record what programs are being used by what schools. (Choosing Blindly: Instructional Materials, Teacher Effectiveness, and the Common Core by Matthew Chingos and Russ Whitehurst crisply and logically highlights this “scandalous lack of information” on usage and evaluation of instructional materials, as well as pointing out the high value of improving knowledge in this area.)

But publishers themselves are now in a position, in many cases, to aggregate their own digital program usage records from schools across districts, and generate timely, rigorous, standardized evaluations of their own products, using any state’s posted grade-level assessment data. It may be too early or too risky for many publishers. Currently, even just one rigorous, student-level study can serve as sufficient proof for a product. It’s an unnecessary risk for publishers to seek more universal, annual product accountability. It would be as surprising as if, were the anonymized data available, a fitness company started evaluating and publishing its overall average annual fitness impact on club member cohorts, by usage. By observation of the health club market, this level of accountability is neither a market requirement, nor even dreamed of. No reason for those providers to take on extra accountability.

But while we may accept that member-paid health clubs are not accountable for average health improvements, we need not accept that digital content’s contribution to learning outcomes in public schools goes unaccounted for. And universal content evaluation, enabled for digital programs, can launch a continuous improvement cycle, both for content publishers and for supporting teachers.

Once rigorous program evaluations start becoming commonplace, there will be many findings which lack statistical significance, and even some outright failures. Good to know. We will find that some local district implementation choices, as evidenced by digital usage patterns, turn out to be make-or-break for any given program’s success. Where and when robust teacher and student success is found, and as confidence is built, programs and implementation expertise can also start to be baked into sustained district pedagogical strategies and professional development.

Tagged , , , , , , ,

Common Core meme provocation and response – 9 ways to Go Faster!

The “Angry Mother Destroys Common Core” internet meme got my goat when I browsed the comments section. I grant that the actual meme problem was overbearing (it’s the “write a letter to Jack”  about subtraction error using the number line). But in the comments I saw a lot of complaints about how the number line approach was not the fastest way to the answer, and thus a confusing waste. Not those commenters’ faults in a way; they are just sticking with the one-right-way paradigm about math they were taught back in the day.

So I yielded to temptation and uploaded a response comment on the meme’s thread, sharing here for (I hope) your amusement:

Apparently to many folks, the Fastest way to get to The Answer is the Point of math. Here are 9 ways to go faster:

1) Memorize lots of formulas – as many as you can.
2) Once it seems to you like one of your formulas can be applied, run it.
3) Even for addition, use a calculator (We all have one in our phones, sweet!)
4) Download a powerful calculator that has lots of formulas pre-coded for you to just punch in a few #’s. Less memorizing!
5) Have your mom do the problem for you (that’s faster, right?)
6) Have your mom just give you the answer (now we’re getting fast!)
7) Have your smartphone solve it (it knows more than Mom!) http://www.nydailynews.com/…/new-photomath-app…
8) Don’t answer the question – heck what difference does it make if you answer someone else’s canned math question anyway?
9) Don’t even bother taking a math class and save yourself years of grief – other people know how to do the math that you can’t do with a calculator or app anyhow, right? And aren’t there lots of people to tell you they never broke out a binomial or even an ‘x’ in their worklife, ever, so what a waste?

Oh, and now that speed has been served, please enjoy your trip into the real world after school, where people get good pay if they can solve real problems via pattern recognition, efficient problem abstractions, and a myriad of hierarchical solution methods, using logical thinking. Within a few months of hiring on, managers know who can solve problems and who is weak. Calculators don’t cut it.

Oh yeah one more thing: in the real world there is no 1 right answer in the back of the book.

My kids will see your kids out in the world competing for jobs. Good luck to them all!

Tagged , ,

Is “Doogie Howser” acceleration through formal school our ideal?

An iconic TV series in the early 90’s features a teenager, Doogie Howser, who “earned a perfect score on the SAT at the age of six, completed high school in nine weeks at the age of nine, graduated from Princeton University in 1983 at age 10.” Then, he slowed down and took four looong years to get through medical school – I guess he had to slow down because scripting a tween as the M.D. lead of a hospital sitcom didn’t hit the target advertiser demographic.

“Completed high school in nine weeks.” Let that sink in deep for, like, a millisecond. Did that just slip by? Did it strain credulity? Did it sound great? Are you excited that self-paced digital learning can make this accelerated ‘learning’ happen for more and more Doogies in the near future? If only we can break out of that 19th century assembly-line seat-time mentality..

Hey, earlier is just better, right? I clearly remember attending a LAAMP-sponsored community meeting at an auditorium in Occidental College in 2000, where attending teachers were excitedly being informed from the stage that new state standards meant their students would be “tested on high level content they currently only get to in college.” I was stunned by this earlier-is-better strategy, thinking that we hadn’t really nailed most student’s learning of the good old high school stuff quite yet. By 2006 some unintended but damaging consequences of this value judgement that earlier is better were described by Tom Loveless of the Brookings Institute’s Center for Educational Policy.

[In 2013 California dropped its earliest-in-the-nation requirement that all eighth graders take algebra. Here is the LAAMP final report on disappointing results from its 6 year $53M project in Los Angeles.]

Here is my rant: Stop this faster is better one dimensional talk. No more, “…so with self-paced online schooling Suzie could get her competency based GED at age 12!” Just like Doogie, our hero and model for success.

Having started UCLA at age 15 myself, I feel quite the laggard compared to this ideal. But then I also keenly experienced the social downside risk to acceleration – no sports, no girlfriends, no prom. Too young + too academically successful = stick to your geek math club friends. Seriously folks, scrambling the social institution that is high school for 14-18 year olds needs to be a consideration when you hear “…well when they finish the 3rd grade software, just advance ’em right to 4th and then if they can to 5th.” And one does hear it. What’s that you say? What about the non-digital aspects of learning? “Nah, it doesn’t matter where the teacher or the rest of their peers are! Speed is a goal! Get with it!”

Yes it’s exciting that some types of digital content enable quicker uptake. I’m saying quicker uptake as a primary goal is dangerous.

Here’s an analogy about learning in another domain, a more physical domain I think we intuitively understand better. Suppose you were taking guitar lessons. But rather than once a week in a guitar shop with a  teacher, and near daily practice, you were watching lessons on YouTube. And say there was a 20 hour playlist of 40 half-hour video lessons. Wow! That is easily watchable in a week! What’s that you say? Sure, I passed the online ‘competency’ quizzes and final. It was easy, man! Bam! Done! I just learned guitar! High school in 9 weeks!

Really? What was the purpose of “learning” guitar? The learner’s purpose, your purpose? Were you supposed to be able to, like, really play it? Have your fingers fret clean notes with automaticity? Feel motivated to play; enjoy playing? Experience playing in a band and making music? ‘Own’ playing it as a lifelong skill? Learn to read sheet music so that you could play new songs and more easily learn some other instrument too? Learn more about music in general? Understand key guitar features and why they work how they do? Launch you on a quest to conquer more and more challenging guitar playing, way beyond your training?

Or was the purpose more just a ‘badge’ to display. “Got in in one week!” “Got through 3rd grade math online in one month!” Hey, I got a good score, a good grade, what more do you want?

What gets lost in the conversation about acceleration is GOING DEEP in learning. There is another dimension to accelerate besides calendar time: accelerate diving deep. Use digital content to dive deeper. Don’t promote 12 year old college students as an ideal. Please, the social costs are too high. If you are testing for competency, test DEEP. No badge until you successfully perform a duo gig and earn tips at the local coffee shop.

Take extra time you find to dive deeper into 3rd grade math. As MIND Research Institute co-founder Dr. Matthew Peterson said, if a 3rd grader gets quickly through 3rd grade digital content then, “let’s get them a Ph.D. in 3rd grade math! And then a Nobel Prize in 4th grade math!” The vertical dimension, depth of learning, is bottomless in every content area.  For example, fractions concepts can open up a deeper exploration of rationals and irrationals.

Are you satisfied with how ‘deep’ the current learning demanded to beat the system is? How do you feel about the current pacing through content? Is what you want from digital tools to rip through inch-deep learning much faster?

I’m here to say, personally as well as pedagogically, the future can’t be about getting American history dates and people and quadratic formulae crammed into some 9 week high school frenzy. It’s not about 14 year olds in Ph.D. programs. Let’s use digital tools in order to get more powerful learning. Aim to go deep, not fast.

Tagged , , , , , ,

Transforming the Education Market: Look to Non-Profits

This piece was originally published on the Huffington Post on May 3.

The convergence in the last 20 years of advancements in computer, cognitive & neuro-sciences has made game-changing educational programs a possibility. In the area of mathematics education, inventors can now see a path to give teachers powerful yet easy to use, radically different digital tools to get all students to be proficient in math and even algebra. The breakthroughs go beyond math for math’s sake, beyond proficiency on tests, to the fundamental purpose of math education for all students: every person possessing the powerful thinking, analytic and problem-solving skills that mathematical literacy promises. To put a number it, a 2011 Organisation for Economic Co-operation and Development study estimated that even modestly increased student math skills would add over $40 trillion to the U.S. economy over current students’ lifetimes.

But the K-12 education market is a poster child for unhealthy markets. Products are developed to politicized market specifications, which are far below their potential. Digital solutions from the past 30 years have not worked, so expectations are low. Curriculum is treated as a commodity, and not even evaluated. Not only do most educators not seriously expect teacher tools or content to make a game-changing difference, but K-12 purchasers are looking for approaches familiar to what they experienced in school. The market even lacks understanding of the indicators of program quality and effectiveness necessary to meet the market spec of standardized test success, so simple or secondary features hold sway instead.

In this market, I believe the leaders achieving radically higher educational goals are mission-driven, not-for-profit organizations. Non-profits, almost by definition, need support beyond current market forces. Visionary and savvy business social investments, though relatively small change to $1B businesses, can be and should be vital support for continuous invention and prove-out.

Many business leaders are looking to “help education” through their corporate social responsibility strategies. Unfortunately, based on my experience inside a non-profit education researcher and publisher, what most businesspeople will be looking for is just marginal improvement. There is an unconscious acceptance of the familiar, of “inch deep” learning rather than breakthrough deeper learning. There is a focus on increasing speed and reducing cost. There is a lack of appreciation for the vital role of the teacher.

So, for those visionary businesses and foundations interested in accelerating a quantum leap forward for all teachers and students in K-12 education outcomes (their future workforce/customers), I’d like to suggest that the following considerations are crucial:

1)  Look for non-profits pursuing radically different approaches. Digitizing existing content and approaches is just more of the same, even if ported onto a glossy-screened touch-tablet. People are actually inventing new learning tools, content and processes — like inventing powered flight. Real transformation is going to use teaching and learning models that seem and look and feel radically different from how you learned. Search for that.

2) Look for truly scalable approaches. These three questions will help you evaluate a program’s scalability:

i) Are they applicable to all teachers and students, from gifted to struggling to English learners?

ii) Are they aimed at the heart of the problem right now: at all schools and teachers and training processes as-is? Think Los Angeles, New York, Chicago, Houston and D.C. public schools?

Don’t just focus on and invest in the fringes of the school market, which will take years if ever to reach the majority of students in the community, and the nation. Each year, 4 million more children are passing through an untransformed pipeline.

iii) Can they scale up fast and without limit, driven ultimately by non-philanthropic funds? That is, do the economics of the solution enable eventual demand and resources from the main market, i.e. government public schools, to adopt, scale and sustain?

While some non-profits aimed at breakthrough transformation will be in the early stages of research and invention, you can also look for others with solid evidence that their program delivers results, and that there will be market demand and that it is economically scalable.

There are examples of businesses following all of these principles in their social investment in education. Corporate foundations, CEOs and chairs of Cisco, Broadcom, Emulex, Microsemi, PwC, Bank of America, Chevron and others  have come together to support a breakthrough math program using instructional software to tap students’ visual reasoning. You would not recognize math taught this way, it is that different.

The lowest performing elementary schools in Orange County were provided grants to launch this math program. Over 80% of those schools are now participating. A 45,000-student district serving predominantly economically disadvantaged English learners, Santa Ana Unified, went district-wide and closed its “achievement gap” with the California state average. And the targeted schools at a county-wide level are greatly outpacing similar schools in Academic Performance Index growth. The proven results have attracted district funding at over 1,000 additional sites. Over 500,000 students are being served, and scale-up is economical.

For any company interested in helping education, keep an eye out for the pioneering non-profits that fit this profile. Your social investment will then be poised to go beyond “help,” to transform and scale to millions.

Tagged , , , , , ,

Instructional Software: Just a Cleanup Activity after Ineffective Teaching??

How instructional software is positioned in the minds of many educators and others is outdated, and misleading.

The conventional model of instructional software (I am thinking “core” subject like mathematics) began, naturally, as a digital extension of conventional teaching. So: practice problems, read onscreen, with multiple choice answers, instant grading and perhaps with some gamification of scores. But if you didn’t “need” the “extra teaching,” you didn’t “need” the instructional software. So it was optional: for some students, some of the time.

There were two logical models for deciding for whom and when to use instructional software: use it for remedial students, or periodic diagnostic tests (eventually also online) for everyone, to determine which skills the student needed more practice in; then assign instructional software just for those specific skills. The metaphor is “filling the holes in the Swiss cheese.”

Implicit in those models was that some students did not need any instructional software: those who learned sufficiently from the standard, no-software-involved, teaching. The instructional software served a role of “cleaning-up” whatever gaps were left unfilled or incomplete after the normal teaching. By observation then,  the regular teaching on its own was ineffective in achieving the learning goal for some students some of the time. (The reason it was ineffective could include many things outside of the teacher’s control, of course.)

Despite the recent emergence of “blended learning” as a desirable future model of combining digital content with teacher & chalkboard learning, at present the preponderance of students still use zero instructional software in their studies. And frequently, even in 2013’s “state-of-the-art” blended learning examples, the role of the digital content is still essentially more practice, like a digitization of homework reps, albeit with intelligent sourcing of problems and with instant scoring.

Similarly, in many 2013 RFP’s the instructional software is specified for RTI tier 2 interventions for struggling students only. This means that not only do the RTI tier 1 “OK” students not need any digital component in the normal course of their learning, it’s not even seen as a way to prevent “OK” students from slipping into tier 2.

All of the above makes sense if you see the role of instructional software as just enabling “more.” More of what teachers ideally, technically “could,” but in the real world can’t, deliver because of the constraints of scarce time, and thus the impossibility of differentiating teaching to productively engage each learner and suit the pace of each learner. So the instructional software provides more time for those students and situations who just didn’t get enough time from conventional teaching.

But consider: more time for students has been tried, and tried, and doesn’t get game-changing results. By game-changing I mean ensuring that every student understands and gains content mastery and confidence in a subject – like math. If more of the same did work for challenging situations, then the mere, but very expensive, application of additional teacher time (double-block, repeated courses, pull-outs) would be shown to “fix” the problem. Which in math, certainly, it doesn’t — not at a scale and cost which can be universal and sustained (i.e. beyond a one-on-one tutorial). So instructional software’s role to give “more of the same” is not a fix.

This pigeon-holing of instructional software as for “clean up” is too limiting. If that’s your model, you wouldn’t even think of buying — or making — instructional software that has fundamental and vital value for every student. Fundamental and vital is how we view… textbooks. Lawsuits are filed and won to ensure that every student has a textbook. When the day comes that a lawsuit is filed, fought and won to ensure that every student has effective instructional software we will know that the pigeon-holing is over.

Here’s an analogy of this positioning problem to the world of exercise and health. It’s as if instructional software is seen as physical therapy, rather than as physical conditioning. It’s as if it’s just for those who are in some way injured, or chronically weak, rather than for everyone who wants to get in shape. You get diagnosed for your injury, perhaps a shoulder tweak, you do your therapy reps with rubber bands, and one happy day you’re healthy enough to quit doing the P.T., forever.

The future, additional role of instructional software is as a vital component of the learning environment, for every student and teacher. It’s like joining and then diligently using a gym’s facilities and moreover its trainers, motivation and social aspects. Properly designed and trained and supported, it’s a gym program that gets everyone more fit. No one gets to “test out”. No one gets to work “just on their weaknesses.”

And it’s not implicit that “ineffective teaching” is the raison d’etre for instructional software. This is turned completely inside-out: instructional software, in the hands of a teacher, makes teaching and learning more powerful and effective generally, throughout the school year: differentiating to reach every student (including the strongest), engaging and motivating each student at an appropriate level and pace, and providing multiple opportunities for the teacher to assess, diagnose, and consolidate student learning.

Tagged , , , , , , , ,

Common Core Pessimists: Consider the Incremental Advance on Information Quality

In response to Rick Hess’ blog “The Common Core Kool-Aid” lamenting the latest silver-bullet ed-reform groupthink. I agree with Rick’s consistent themes that “it’s about the quality” and incremental improvement.

So, from the perspective of my experience in a non-profit math content publisher and ed researcher, a simpler take on CC: improves quality/quantity of widely relevant, widely comparable, statistically valid, actionable information for feedback into improving learning systems nationwide. State-level standards/assessments quality have empirically shown wide variation such that the low quality end has generated suspect information (e.g sky high state proficiency rates vs. NAEP rates). Apparently it’s politically more likely for the standards/assessment system to be gamed at the state, vs. federal, level.

So, not a panacea, nor a catalyst or driver for any specific “reform” agenda, just an incremental but i.m.o. highly significant and valuable improvement on the current state-of-affairs in information quality, transparency, and widespread relevance. Would be wonderful if over time CC turned out to provide a higher quality “floor” under teaching-to-the-test, too.

Tagged

Assessments are the Ultimate Game-Changer

“Real change in teaching and learning requires real change in assessment” – Justin Reich.

This arcsparks post is 100% inspired by EdWeek’s Justin Reich’s excellent post “Everything Hinges on Assessment“, which powerfully backs up, and shows examples in support of, his quote above. My post below was also left as a comment on Justin’s blog.

In other words one might say, to profoundly change the game we need to change the scoring. Then, how people play the game to win has to change too.

And the only scoring on K-12 frequent enough, direct, clear, important to all the players, and timely enough to matter to anyone at all is tests, whether in-course or end-of-course.

I”ve been with a non-profit working for 11 years to provide a digital tool for a changed game that very few if any are seriously playing yet.

Specifically math instructional software for blended learning to ensure that students deeply understand math concepts. In the absence of hard-edged assessment, that tests the highest-order conceptual student outcome results we’re spec’ing into our instructional design, we rely on the soft evaluations made by educators on a personal level – i.e. on specific visionary educators, whether at district, site, or class level, seriously insisting on more than is tested for.

Assuming that our program and the educator are actually successful together, this is still an unstable situation, susceptible to specific educators leaving the district/school, etc. to be replaced by another with de facto lower goals for student learning. To put it another way, sustained learning above a floor-level set by the assessments is an unstable exercise in defying gravity.

At my shop, our experience is that a good program will show results on any level of quality assessment of the content, and we are perfectly happy to see results show up (and they do) on fill-in-the-bubble assessments. Nonetheless our strategy is to design for max conceptual understanding, regardless of the assessment.

Quality in instructional materials and programs is extremely expensive – much more expensive than the market thinks it is. Economics alone would dictate designing as cheaply as possible, to the lowest quality required to meet the assessment spec: designed for the test. Fortunately for my shop, we’re a non-profit so our bottom line is learning, not earning and we avoid this temptation.

Bottom line: couldn’t agree more with the points in this post; dramatically raising the assessment’s requirements on student learning will be a profound, irresistable, stable game-changer for educators, students, parents, and all publishers.

Tagged , , ,

What if a balloon corp. decided funding for the Wright brothers’ R&D proposals?

An insightful Rick Hess blog on the potential for federal government funding of true innovation in education: ARPA-ED: A Qualified Thumbs-Up – Rick Hess Straight Up – Education Week.

As usual Rick reports out the risks not just the promise. Working at a digital content innovator turned down twice by i3 reviewers (U.S. Education Department’s “Investing In Innovation” competitive grants program), it is clearly evident to me that the incumbent education establishment is not open to “breakthrough capabilities” or even “fresh thinking and ideas”. It is as if those reviewing a potential breakthrough aviation project, like stealth technology, already have strong beliefs on what will or won’t work, if not their own competitive stealth projects underway. Selection of ARPA-ED projects to be funded: applicant eligibility, who evaluates project proposals, on what basis, and under what oversight, will be a make-or-break. Just allowing the path-of-least-resistance and least-political-risk of incumbent status-quo thinking will *not* generate education game-changers.

Tagged

Does that Program really work? 6 questions to ask about Research

At MIND Research Institute, we have focused intently on studying the real-world, at-scale effectiveness of ST Math, our visual math software. In the process, we’ve gained a perspective unique among publishers on what constitutes valid effectiveness research.

That’s why we offer these guidelines on how to assess research about K-12 curricular programs.

1. What was the size of the study? Some of the results need to be at substantial scale, meaning at least five school-wide (not classroom) implementations. Super-motivated teachers can and will get outperforming results using ANY program. With large-scale studies (e.g. MIND has several from 25-50 schools) there are a substantial number of teachers involved, and it’s unlikely they are all champions.

2. Are the results repeatable over years? Only having one year of results is not compelling. It’s very important to see whether improvement is maintained or even grows in Year Two, and to see that two different sets of students achieved similar results.

3. What were the results compared to? Just having outcomes improve with program use means nothing; the improvement must be greater than a similar situation without the program. Also, the added improvement from the program must withstand a test of statistical significance.

4. Are the results repeatable across states? Look for large-scale results across multiple states. This means several different assessments show positive outcomes—an indication that the program is affecting something deeper than test prep for a specific type of assessment.

5. What outcomes were measured, and for which students? For a program to “stick,” you should look for gains in standardized assessments or other aspects of an accepted accountability system. Also, the research should disaggregate improvements by initial proficiency levels, so you’d know whether there was improvement among both low and high-performers.

6. Is the program implementation replicable and sustainable? The study should identify conditions of implementation that led to results, and you should evaluate whether these conditions—training, time, acceptance by teachers, and cost—are replicable and sustainable for you.

Tagged

If something Actually Worked in Education… 4 things to Look For

“Education” is a huge conversation, but almost all of the conversation is about the education problem. There is not a serious or mature conversation about solutions.

Meanwhile lots of resources and, as important, attention, are going into an ever-changing, wide variety of programs which, it is hoped, will be part of some solution.  How do we know when we’re looking at any given program, whether it can have game-changing impact on K-12 schools, or has no chance? I call the former, programs that “Actually Work”.

Here I identify and describe four characteristics of any program that Actually Works: Scope, Results, Robustness, and Scalability. Failure at any one of these is failure to Work.

1 – Scope

A program that Actually Works must cover or affect an entire course curriculum. For example, let’s think of mathematics. A program that Actually Works can be as small as one grade level, or one course. But the scope can’t be less than that — it can’t be just a specific concept — because you need to be able to Tell if something Actually Works, and that takes an end-of-course assessment. And what Actually Works needs to help educators and students with what they are held accountable to: summative assessments. So, in mathematics, a cool interactive visualization of parabola parameters on a graphing calculator doesn’t meet the Scope requirement.

2 – Results

To Actually Work, a program needs Positive Results. As stated above in Scope, the results need to be measured using a summative assessment. The Positive Results need to be essentially this: for educators and teachers using the program, all the students become Proficient in the given subject area. I’m willing to accept Proficient as defined by states as meeting state standards on a state standardized test, and “all” as 90%+, but a more general way to define Proficient is: functionally competent to create work product, or solve problems. So, do these Positive Results need to pass a clinical, random-assignment gold-standard evaluation to scientifically attempt to identify the Cause? No. For a program that Actually Works, the results will speak for themselves, robustly at scale in any real world setting, and there need be no argument about what caused the results. Empirically, on implementing the program, you get the Positive Results. In a wonderfully luxurious later phase of this education conversation, when multiple programs Actually Work, we can fret about what we don’t yet know. We can fret over whether extra hours made the difference, or what component came from human or material or process components of the program, or motivation, or factors external to a school. We can fret over whether one program that Actually Works has been properly compared to another program that Actually Works. And we can fret over exactly how do we evaluate the answers to those questions. But the breakthrough step is to have “one” exemplar program that Actually Works.

3 – Robustness

If a program Actually Works, it will work for any educator and any student.  A program that only works for some educators — for the most motivated, or the most experienced, or the most skilled — does not Actually Work. A program that Actually Works needs to work for a first-year teacher with low confidence, right? I mean if it doesn’t work in that situation, then we have a Big Problem, right? And a program that Actually Works has to work with every student subgroup. In the area of math, for example, that means that it works for students who are multiple grades below grade-level. It works for students who have been testing low. It works for students who don’t get much help at home on math. It works for students who have low confidence in their math abilities. It works for students who are suffering from low motivation to do well. And it works for students who are low in language arts proficiency, and students who are learning English.

And the educator or student, for whom the program Actually Works, can be in any school, any school district, in any city and in any calendar year. In other words, Positive Results should be replicable across geography and time.

4 – Scalability

If a program that Actually Works can’t Scale, then it is of no use. By Scale I mean reach millions of students. And there are three aspects of scaling: Implementability, Economics, and Speed.

First Scalability Aspect: Implementability. The program needs to be able to be implemented across the dominant education model. In other words, it must fit within the existing structure of facilities, people, and time. The program should *not* then be requiring some additional changes, such as re-tooling school buildings; re-tooling school hardware; re-tooling school personnel; requiring home technology, or changing the time or location of education. Another consideration for scalability is that the program must not only be do-able by teachers, but also be embraced by teachers. Anything less than a full embrace by teachers will result in failure, failure to achieve Positive Results, either from outright rejection of the program, or by spotty or sub-par implementation of the program.

Second Scalability Aspect: Economics. The program needs to fit within current school economics. This means both for initialization of the program as well as for sustaining the program. For start-up, there is not only the program outright cost but also the costs for site facilitization (see above) and especially in teacher time for training. Outright cost needs to be in the neighborhood of current instructional materials costs per student. Facilitization costs need to be near zero. And teacher training time costs need to also be near zero, as PD days and substitute days become artifacts of the past. Sustaining costs (e.g. annual renewals) need to be low enough to survive tough priority battles during even the toughest budget years (see above: teachers must embrace and highly value any program, for it to be Scalable).

Third Scalability Aspect: Speed. If there is a program that Actually Works with all of the above, to be worthwhile it must also have the characteristic of quick scale-up. By quick I mean that it must have the capability, if the market demands it, of adding millions of students and their teachers per year to the program. In a country with 49 million students and 3.5 million teachers in K-12, anything less than this means we have no solution at all.

Tagged
%d bloggers like this: