The quality of education studies varies about as much as the quality of public schools -- maybe even more. While some studies stand out for using rigorous research methodology, others clearly seek to serve a political goal at the expense of serious research. Policymakers, journalists, parents and all of us would benefit from tools aimed at helping sift the gold-standard studies from the river of research.

To help with this, the National Alliance for Public Charter Schools has identified 284 studies that examine charter school achievement. Of these, 203 use serious research methods, and are included in our soon-to-be-released research synthesis. For more information on this research synthesis, please contact pressroom@publiccharters.org.

Of all the research, only seven of the highest-quality studies are peer reviewed and only four are examined by Think Tank Research Quality. To rely too heavily on those 11 studies that both adopt serious research methods and have undergone a review process -- a process that naturally requires reviewers to select certain individual studies-- automatically skews the importance of these few studies.

Even if the education sector adopts a proper peer-review process to augment the work of groups like Think Tank Research Quality, there is still a huge challenge for education research. That is that a single, solid review of a discreet study cannot stand alone. No single study should determine how the subject of research is judged, despite wide media traction or attention from policymakers. Instead, a body of research needs to be used collectively to inform everyone.

Our educational research base should be a beacon of scientific understanding, a bright light illuminating better practices, better programs, and better policies. Unfortunately, the beacon is dim, and our country is, for the most part, still in the dark. Crews have been dispatched to the scene but repairs are proceeding slowly. Some fixes seem to obscure the light of reason even more. If the beacon is indeed growing brighter, something must be obscuring it from view. Either that, or we’re all going blind.

Mr. Hess, Mr. Kress, and Mr. Finn each offer thoughtful commentary. As Mr. Hess notes specifically, whatever level of quality we enjoy today in educational research is due at least in part to the work of IES. IES research standards are clear and, frankly, I think they’re the only standards we have. I just wish more people knew about them. I also wish more educational researchers took them to heart.

Though educational research has improved, how we use it has not. One could argue that education simply hasn’t had enough time yet to mature into a responsible research-based culture. But I have grown weary of excusing education’s weaknesses time and again. It seems that we perpetrate upon our system of schooling the same injustice it perpetrates upon our children: “the soft bigotry of low expectations”, as President Bush so aptly described it.

Let’s raise our standards and start measuring with the proper yardstick. It seems the debate here is about the quality of the research itself—the peer review process, data systems, oversight, ethics, etc.—but shouldn’t research in an applied science be evaluated by the degree to which it is successfully applied in the field?

Educational research should reflect its applied nature through the successful translation of data, theory, and analysis into replicable models of real-world success. But this rarely occurs. Most studies seem to serve merely as tenure trophies, funding mechanisms, marketing brochures, or agitprop advance materials for entrenched special interests. Few, if any, do much to help teachers and kids.

As I work in schools across the country, I see good things happening here and there, but few that are based on research. Most good things in schools today are based on common sense, creativity, and good old-fashioned hard work—three things we don’t study very much, probably because they’re hard to count. Sometimes I think that what counts doesn’t matter nearly as much as we think it does—and that what matters doesn’t seem to count.

I also see many bad things happening in schools, things that are justified by studies that are contradictory, inconclusive, ethically compromised, or just sham science. Sometimes, even good research is bastardized and bowdlerized to justify programs and practices that actually have nothing to do with it—and everything to do with someone’s agenda.

For example, my local school district is struggling right now to understand its own “research-based” instructional framework. What’s the problem? They didn’t read the research correctly and now no one is sure what the framework represents, what specific practices it recommends, or to what extent it is being implemented. This serves perfectly the anti-reform interests of many of our administrators and teachers because it means we can’t easily create change.

We would be infinitely better off without a research base. At least then we’d have to talk directly about practice and observe directly in classrooms rather than being able to avoid these uncomfortable activities by arguing over the meanings of words in studies that most of our teachers and administrators don’t understand.


BUT WAS IT ALWAYS THIS WAY?

When I began working in education 15 years ago, I followed educational research with consistency and devotion. Nowadays, I give each study a reasonable read and file it away in my “research” folder. I must have hundreds of studies in there, but I can’t remember the last one that inspired me to craft a useful real-world implementation of something for a classroom, school, or district where I work.

When it comes to educational research, we seem to forget that we work in an applied science, not a theoretical science, or a lab science. There’s little use for “pure research” in education; nor do we need any more preciously parsed “bean countings” that reframe educational statistics and recast outcomes to justify hidden agendas. If a study doesn’t have valid data, or clear and unambiguous implications for policy or practice, it’s not worth the e-paper it’s distributed on.

As for policy-focused studies, in particular, these have been virtually useless in recent years—CREDO v. HOXBY being a perfect example. All we get are conflicting viewpoints conveniently aligned with the political orientations of the studies’ authors. You say “potayto”; I say “potawto”; let’s call the whole thing off. The most responsible research-based conclusion we can draw about charter schools is that the research base on charter schools is inconclusive. Yet our President and our Secretary push charters as though the matter was settled years ago. I guess they say “potayto”.

Personally, I’m very interested in charter schools. But I recognize that my interest is personal, biased, and not at all based on research. I support charters under limited circumstances because I see a limited number of them that are very successful, and I’d like to see more—of the successful ones only, please. When the subject of charters comes up, I don’t make knee-jerk comments about educational privatization, nor do I run around the country handing out Hoxby studies like Gideon Bibles.

For me, charters represent a personal interest in a new structure of school governance that might better support some of the more innovative practices I’ve developed. I want more lab schools, more places with fewer restrictions where we can do more interesting research. I am willing to admit that charters suit my whim more than they seem to suit the needs of our children at this time—and this is why I refrain from recommending their unchecked expansion.

President Obama and Secretary Duncan, however, are unwilling to be so transparent. Like many of us, they are merely exercising their personal interests in charters, too. But from what they say, and the policies they support, one would think they had access to a research base full of crystal clear confirmation that the rest of us have never seen. What is the value of policy-focused studies if we don’t use them to focus policy? Even if we had the best research imaginable, I wonder if we would use it.


SO HOW DO WE USE RESEARCH IN SCHOOLS?

While I deeply respect the opinions of everyone who has contributed this week, and I readily admit that each of you has read more research than I, my hunch is that I’ve had more direct experience when it comes to the lowly matter of seeing how research is actually used in schools. Even good research isn’t used well. For the most part, it isn’t used at all. And often, it is used with the intent of delaying progress not inspiring it.

My experience working in schools—seeing how teachers, administrators, and school board members bandy about claims of conclusive analyses where none exist simply to justify their petty prejudices, parochialisms, and idiosyncrasies—convinces me that while research may be improving, the way we are using it is not. So there are really two issues here: the quality of the science available and our ability to use it well. While the former may be getting better, the latter is almost certainly getting worse.

The central problem of educational research today derives from an unfortunate collision of four factors:

(1) Commercial entities conducting studies of their own products and services that read less like lab reports and more like marketing briefs in order to qualify their offerings for federal funding under NCLB and Title I.

(2) Administrators struggling with change who seek easy ways to exert control over their districts and who are looking to published programs—research-justified or not—that represent safe decisions for which they cannot easily be blamed.

(3) The use and interpretation of the term “research-based” in NCLB, language that has allowed, and even encouraged, school leaders to seek out any research, valid or not, in order to avoid using newer and more innovative approaches that might accelerate the pace of change.

(4) A lack of ability among educational leaders to effectively interpret and apply research in their schools and districts, and a lack of willingness in these same people to seek out and apply the best research rather than just the most convenient research.

Together, these factors have contributed to a proliferation of ineffective programs and the safe haven of plausible deniability for the people who have implemented them. If research in an applied science doesn’t lead to positive results in its application, what can be said of its value? Even worse, if the people who have to use the research can’t understand it or won’t seek it out, would higher quality studies make a difference?

In the future, we may have to ask these uncomfortable questions with even more frequency—and actually be willing to answer them. Race to the Top just gave a group of states money for the construction of large-scale longitudinal data systems. It also funded a multi-hundred-million dollar program for two assessment consortia to create new tests. This means that more educational data will soon be available than ever before in our nation’s history.

More data means more studies. More studies mean more analysis by people who will use these studies to recommend new programs—which can then generate additional data which can be studied all over again. This metrical merry-go-round could run on its own energy forever like the fabled perpetual motion machine. The actual value of the research, in terms of practical application in our schools, would be irrelevant. An entire “research economy” could exist, not as a means to an end, but as an end in itself, in perfect symbiosis with the system it studies.

Given the relatively poor quality of educational research now, and its even poorer application by educators in the field, this scenario represents an insidious and self-serving feedback loop where the real winners are data gatherers, researchers, and policy analysts, not teachers, children, and families. The next decade will be the best in history for educational research. But it may not be best for education.

The danger here is an obvious feedback loop of misinformation. Bad data—of the kind we have seen most states gather and report as they have rigged their testing systems to show false growth—leads inexorably to bad research. Bad research leads to bad policy, and bad policy leads to bad programs.

An increase in the volume of educational research may make things worse, not better. A plethora of studies on any given topic gives risk-averse school leaders an easy out when pondering local change. Defenders of the status quo need only select the study that advocates the least change or whose conclusions can be construed as supporting what is already being done regardless of the results being achieved.

Our obsession with data could lead us down a blind alley, and if states continue to lie about their achievement information, we might not even know it. Gargantuan RttT-funded longitudinal data systems could wind up as repositories of junk data ready to be cooked up into junk science.

The mere existence of a new magnitude of educational data will likely unleash a torrent of studies the likes of which we’ve yet to experience. Given the current quality of education research, and the difficulties we have applying it in the field, I foresee more research-inspired confusion, not less, and more data-based donnybrooks than logic-inspired agreements.

Higher standards and better processes for vetting research will be helpful but only if people use them. Peer review is well and good. But it’s hard to peer review the government. We know that states are disinclined to render a fair accounting of educational statistics. Perhaps we need more research on this! Far from the stuff of Tea Party-esque conspiracy theory; this is mainstream educational history. It’s well documented; we all know about it.. And yet, as we stand on the edge of a new era of government-funded data acquisition and research, no one is talking about how corrupt our data might be.

We can say that educational research is getting better. But can we prove it? Is this a subject we’ve researched? I’m not being facetious. We can choose to study anything we want. Why not study the application and value of educational studies?

If we measure educational research by the real-world yardstick of successful application in the field, we have a very poor record to contend with. Of all the studies that have been conducted since “A Nation at Risk” was published in 1983, what percentage have conclusively contributed to practices, programs, or policies that have led directly to statistically significant improvements in student achievement?

We have a very limited history of effective educational research in this country. We also have colossal multi-billion-dollar "research-based" failures like Reading First on our scorecard. It’s not as if we can point to a time in our history, or to a group of people, or to a set of laws, or to a method of funding, or to a political party that has raised the quality of educational research to that of other disciplines or secured better application of that research in the field. The one thing we know for sure is that we don’t know for sure how to create and apply successful educational research.

For an education reform effort of national scope and generational consequence that is supposed to derive entirely from research, much rides on the existence of strong science and its responsible application. At present, however, such science is weak, and its application is weaker. So far, there seems little connection between the research we have available to us, the policies we have chosen, and conclusive evidence of improvement in our schools.

I believe that educational research can be better. But I also believe that it would have gotten better long ago if quality had been important to us. I think the current insufficiency of educational research is simply the result of greed. For the first time in our history, there’s a lot of money for educational research and a lot more money riding on research outcomes.

The obvious implications of financial gain often taint the research process. Political posturing and a risk-averse education culture prepare the soil for the sewing of dubious studies sprouting questionable conclusions like weeds. In 2001, we wrote a law that said school had to be research-based. People read the law and commissioned studies to support their pre-existing points of view. People conducted these studies because because research money pays their bills.

In a terrible irony, we seem to have put the cart of reform before the research that was supposed to have pulled it. Even today we resist the creation of a research-driven culture of educational change. If we embraced such a culture, we’d be doing very different studies, and very different people would be doing them.

I don’t know why all this has happened. My hunch is that it’s just human nature and the nature of chaotic systems at times of great change. Poor educational research is not the problem. The problem is that no one has proposed ways of improving it or ensuring that improved research is used more effectively.

If educational research is getting better, it is improving more by accident than by design. Clearly, research-based reform cannot get better without better research and the appropriate application thereof. So improving the quality and use of educational study must become a national priority. Continuing to educate future generations on accidental science is unconscionable.


WHAT “RESEARCH-BASED” REALLY MEANS IN SCHOOLS TODAY

A few years ago, a Language Arts curriculum director called to ask my opinion about a reading program she was thinking of buying for her district. She had picked one of the most popular programs in the country. I told her that my experience with it in ten or so districts had not been good. The program was fine, I said. But it wasn’t great, and it certainly wasn’t the best that she could do.

A big problem I had noticed with this particular program was that it came with so much “stuff” that reading was about the last thing teachers ended up teaching. Instead, most tended to simply “follow the book” and otherwise ignore the process of teaching their kids to read, or improving their skills as reading teachers, in favor of merely implementing the program.

There were several other programs that were better, I told her. I also pointed out that the best thing to do was simply to hire teachers who knew how to teach reading or, if that wasn’t possible, to train teachers to teach reading well.

Programmed instruction is less effective than well-trained human beings, regardless of the subject. It’s also much cheaper when districts do a proper cost-benefit analysis and own up to the inevitable costs of purchasing new programs every few years and starting all over again with new approaches.

Training isn’t cheap, and it requires vigilant adminstration. But well-trained teachers can’t have their learning erased by every new curriculum adoption. Teachers who learn to rely on programs have to start over from scratch every time programs change. After several changes, their teaching becomes a pastiche of semi-random practices or an unthinking exercise of turning pages in a textbook.

With reading, in particular, programmed instruction comes up short when teachers have to address the individual needs of diverse learners. When everyone is learning the same thing, at the same time, the same way, on the same day, effective differentiation is impossible. As I told the curriculum director, teachers who know how to teach reading can make better use of programs even if you do eventually adopt one, so building capacity for quality instruction is a better approach in general than relying on programs as packaged substitutes for human ability.

The curriculum director told me that she liked the program she had chosen because there was so much research on it. She had never used it herself and was not interested in running a small pilot program for reasons I could not understand. Her attraction was based solely on the illusion that research existed to support the program’s use.

She directed me to the publisher’s website to make her point. The publisher must have had hundreds of positive studies available (and not a single negative or even inconclusive one). But these glowing reviews were at odds with my direct experience, so I hopped over to the What Works Clearinghouse to check out the research there.

Curiously, only 30 studies had been accepted at the Clearinghouse on this particular program. None of these was considered valid or statistically significant. IES’s conclusion was that there was no proof of the program’s effectiveness. The obvious research-based decision for my curriculum director would be to make a different decision. But that’s not what happened.

When I mentioned IES’s conclusions, and sent my client to the Clearinghouse, she was astounded. She had no idea that the Clearinghouse existed, she had never heard of IES, and she couldn’t believe that a publisher would reference research that wasn’t considered valid by our government.

She also didn’t understand that most of the research she’d seen on the publisher’s website was conducted by the publisher. Nor did she think there was anything wrong with publishers conducting research on their own products. Most curious of all, however, was the fact that she had no interest in using the Clearinghouse to evaluate other programs.

She purchased the popular reading program shortly after talking to me and with no further investigation of other options. Why? Because, by law, she could, and because it was the easiest thing for her to do. The program was, at least according to its publisher, “research-based”.

A set of studies, valid or not, was all she needed to make a near-million-dollar purchase, a purchase that turned out to have negative consequences for her district in the end as reading scores held flat in the coming years but reading teachers lost their interest in actually learning how to teach reading, thus making the next adoption even harder.

Each time this district picks a new reading program, they will start from scratch with regard to improving instruction. Over many years, and many funding cycles, it is highly unlikely reading will improve, even though teachers will surely have had to work harder and harder to keep up with new programs that work no better than old ones. All of this behavior will be justified through research, just as it is in thousands of schools and districts across the country.

Perhaps the best educational research we could develop would be research on how we use educational research. This might provide useful insights. Most important of all, it might help us craft policies to improve the quality of the studies we conduct, and improve the way their results are used in our schools.


AND YOUR POINT IS?

Yes, educational research is getting better. But, practically speaking, it’s still awful, at least when measured by results in the field. In a research-driven reform effort such as ours, we have to remember that the scientific stakes are very high. A few bad studies, or even bad implementations of good studies, can catalyze a cascade failure. Just think of Reading First. Education is a highly conformist culture. The most likely thing for a school to do is exactly what another school is already doing. This is why fads ripple through our system like tsunamis. Failure is viral. Bad decision-making is the rule, not the exception.

Too many people use sham science to support extra-educational agendas. Even though IES has a great resource to sort these issues out, few educators know about it, and even those who do prefer to ignore it—just like my curriculum director did. After all, if you’re an administrator with a lot of money to spend, and responsibility for the most important subject in your district’s curriculum, why take a risk trying to build capacity for positive change when you know you can never be criticized for purchasing a popular program even if it fails?

The problem with educational research is that we don’t do very much of it. Most of the “research” we do is not research at all; it’s marketing or propaganda. What passes for research in education probably wouldn’t pass in any other discipline. Nor would so much shoddy and obviously tainted research be used to make so many high-dollar decisions that affect so many people’s lives.

Our goal in the coming years must be to address this situation thoroughly by raising the quality of educational research to that of other disciplines, and by training educators in the field in its proper application. Research-driven reform will fail if we don’t reform research.


LIVE BY THE SWORD, DIE BY THE SWORD

Back in 2001, almost no one mentioned the potential negative implications of the phrase “research-based” as it was used throughout NCLB. For the first time in our history, schools were going to implement "scientific” practices and programs—as a matter of law. We even set up the What Works Clearinghouse as the official arbiter of what would constitute the official research base. This was a good idea. But I don’t think the authors of NCLB could have predicted from their vantage point in our nation’s capitol how the phrase “research-based” would be used—and abused—in our nation’s schools.

Reform-oriented educators like me working at the building and district levels, naively thought that the “research-based” requirement would bolster our cause. After all, we were the ones pushing research-based practices already. But we were wrong.

Where before we had research, and opponents of change had only excuses and political maneuvering, protectors of the status quo found better defensive weapons in, of all places, research. Where I might have used a dozen studies, professional literature, and empirical evidence right inside a district’s own classrooms to support a particular approach, district administrators now had only to find a single study that supported a different approach or, more often, the same approach they’d been using for years.

In the game of study versus study, the party in power always takes the push. Because the most innovative practices are always supported by the smallest group of people (see “early adopter” and “Rogers Innovation Adoption Curve”, 1962), the least innovative practices are the most likely to be implemented. And when these least innovative approaches can be supported by research, there is no effective way to oppose them, even they are based on an inferior or incorrectly applied research base.

In a valiant and responsible attempt to bring research to education, the creators of NCLB ironically commoditized educational research. To most educators, all studies are the same. At this point in time, education culture has embraced the notion that research exists, but not the idea that significant qualitative differences exist between studies. Scientific inquiry in education has effectively been reduced to an exercise in confirmation bias.

Even worse, most educators working in the field are now so tired of being manipulated by research-based justifications of useless programs and practices that most of us don’t even try to justify what we do with research. Why bother? Anyone with more power who doesn’t like us or the change we represent can simply find a study that supports their agenda.

Even if the opposing study is horribly flawed, or simply an inappropriate match to the context in question, teachers will lose to principals, principals will lose to superintendents, and superintendents will lose to school boards. In effect, we’re right back where we were before NCLB, only now we can’t actually make persuasive evidence-based arguments for programs and practices. We can only hurl studies at each other.

Having to defend everything one does with a study means that nothing new can ever be used even if it is shown to be effective through empirical observation. About ten times a year, I get an e-mail from a teacher asking for the research base on my work. Typically, this teacher and perhaps her department or grade level, have enjoyed great success over several years using something I’ve created. They need the research base to justify their continued use of the practices because they have been told that everything they do must be research-based.

I have no research base for my work. I created it, along with about twenty other educators, over a 10-year period by simply refining ideas in thousands of classrooms across the country—using an empirical approach. Education is an applied science; I created my work through application in the field. I know that it works because thousands of teachers have told me that it does and because I’ve seen it work in my own practice and that of many teachers my company has trained.

But because I can do no research (I have no relevant credentials or sufficient organizational affiliations in this regard), and because my work is apparently unconnected to past research in education, most teachers lose the right to use it after several years of success. Of course, even if I was able to do my own research, not even I would except it as valid. Someone who isn’t me would have to study my work, and because I give my work away for free, there is no economic motivation for anyone—like a potential publisher, for example—to pay for a study. Nor would anyone ever receive grant funding to study it because it is not based on applications of past educational research.

Here’s what I find strange, however. Teachers who have experienced years of sky-high test scores using my materials are the research base. If, right in your own school or district, a small group of people using the same techniques has been getting higher test scores, year after year, than any other group of people, you have successfully performed a valid field study in my opinion. A small set of practices can easily be connected to a convincing set of results; causal relationships can even be identified in some cases. But if you do not have a research study completed by an academic, a government agency, or a “legitimate” institution, you will be told that you cannot continue to use the practices with which you have been so successful.

This is the effective result of the “research-based” requirement of NCLB: it stifles innovation. Anything truly new and different, can’t be used—even if we can see in our own district data that it works.

I know this has happened in at least 10 districts where I’ve worked. I’ve also received another 20+ e-mails regarding similar occurrences from people who have used my free teaching materials on their own in districts where I haven’t worked.

I can hardly assume that I’m the only creator of original teaching practice who has experienced this phenomenon. Therefore, I conclude that the research-based requirement of NCLB, our nation’s unhealthy obsession with educational quantification, and the fact that any study can now be used to support any agenda, has stifled the best source of innovation we have: good teachers doing good things and getting good results.

Research is now an accepted proxy for sound judgment, and the term “research-based” is now a euphemism for sound practice. The folks who wrote NCLB are not to blame for this. There’s nothing wrong at all with encouraging the use of good research. But when we don’t have good research, we don’t know how to use good research, and we are required by law to use it, we end up supporting programs and practices our laws were designed to screen out. Instead of raising the bar for education decision-making, we encourage charlatanism. If a commercial publisher or an ideologically-minded organization needs to push an agenda, all they have to do is publish a study favorable to their point of view. Because research has become commoditized, one study is as authoritative as the next.

The fact that we have a wonderful source of information about educational research at IES is irrelevant in light of the law that created it. Very few of my clients in the last decade have ever visited the What Works Clearinghouse; many have never even heard of it, or so they say when I bring it up on their computer screens. Even after they know about it, they still don’t use it. It’s too complicated. And they don’t really understand research anyway.

Like the curriculum director I was attempting to advise, most school leaders have neither the time, the knowledge, nor the inclination to thoughtfully interpret research. And, frankly, they don’t have to because thoughtful interpretation is not required. Any practice or program with a study of any kind can be considered “research-based” for the purposes of school decision-making and most state and federal funding. Many school leaders will gladly appropriate the conclusions of any study as long as it supports a decision they have to make or an agenda they need to hide behind.

Prior to NCLB, people often made decisions for reasons unrelated to research. This wasn’t good, and those decisions weren’t necessarily sound, but at least they had to be explained in their own terms; the decision-maker had to offer a substantive rationale. This is no longer the case. As research has become commoditized, so has educational judgment. It is now possible to lead an entire district while knowing little at all about one’s discipline. By simply copying words from studies, a Chief Instructional Officer can use the law of the land to justify any practices for which he or she can cite research of any kind.

Once it has been determined that an idea is “research-based”, no further discussion of the matter is required. Any time someone seeks to challenge a decision, leaders need only invoke the “research base” again. Logic, experience, even direct evidence from our own schools is not considered valid. There is no recourse because there is no discourse. There is only research.

If I want to challenge a policy in my local school district, all my superintendent has to say to hold me off is that “research links X with Y.” He doesn’t even have to show me the studies. But if I want to challenge him, I have to find studies disproving his claim and supporting my own, even if I already have definitive data from our own district to make my case. Even if I have both research studies and empirical evidence from within our own district, he doesn’t have to accept these things. Neither does our board. As long as one powerful person can find one study to support his or her agenda, it eventually becomes district policy—or at least a reason to postpone indefinitely any change to a new policy. This happens not by force of reason but by force of research—any research at all.

The theory of forcing educators into making decisions based on research was correct—in theory. The “research-based” requirements in NCLB were meant to minimize the subjective nature of politicized educational decision-making and to lay a foundation for systematic improvement. But this intention was predicated on the availability of good research and the ability of educators to use it thoughtfully, ethically, and in an intellectually honest fashion. This has not happened, nor is it likely to happen in our lifetime. Education culture is simply not a culture of research. And educational research seems unlikely ever to capture the truth of education culture.

Research is a mighty sword. But it cuts away the good just as easily as it cuts away the bad. In many cases—and at times I think most cases—educational research is used as a weapon to empower incompetence and to embolden demagoguery. Skeptical empiricism, rational decision-making, and simple logic seemingly have no place in discussions of reform. Direct experience, individual initiative, and personal judgment have been deprecated to anonymous, commoditized research.


BUT THIS “RESEARCH-BASED” IDEA MAKES THINGS EASIER, DOESN’T IT?

If your teachers can’t teach reading, if you don’t know how to train them (or just don’t want to), and if you don’t want your boss to kick your butt when reading scores slide, buy the “IBM” of reading programs. After all, nobody ever got fired for buying IBM, right?

The publisher of the program will gladly supply you with dozens of studies so you can appear appropriately responsible, and so you can spend federal Title I dollars to pay for it. If things don’t go well, you know you’ve got plenty of statistical CYA to rebut any criticisms.

Who needs to know that the studies aren’t good? Or that the program isn’t worth nearly what you paid for it? Or that for half or a third of your budget you could have trained a corps of teacher-leaders in your school or district to teach reading effectively so you wouldn’t need to buy yet another expensive program of questionable effectiveness a few years down the road?

The fact that very few educational studies meet our government’s standards for quality and validity is something most of us have been willing to overlook. Maybe that was necessary in the beginning; I don’t know. But it’s certainly not necessary now. In fact, it’s downright dangerous. Research abounds in education these days. And the “market” for research is insatiable owing to our fetish for data-driven decision-making.

I use the word “fetish” with intention. It describes exactly what I see happening in the schools where I work. A fetish is an “excessive or irrational devotion” to something. This is exactly what we have right now when it comes to research and data in our schools. Anything that can be counted can be used to justify a course of action—even if the beans don’t add up, the analysis is sloppy, or the metric itself is meaningless. As long as we have numbers—any numbers—we can use it to drive a decision.

But numbers are not necessarily research, and good educational research doesn’t necessarily have to be based on numbers. The course of true science never did run smooth; in an applied social science, the path from data to conclusion is rarely straight and true. What about all those things that can’t be easily reduced to numbers or blessed by someone with a Ph.D.? Empirical observation is generally held to be a powerful thing outside of education. What we can see and experience directly, even if it can’t be readily quantified, should still be considered valid for the purposes of discourse and decision-making.

Many are the times when I have heard district leaders tell me that they are making one decision when they have seen with their own eyes that a different decision represents a more responsible choice. Several curriculum directors I know have said in open meetings that they would prefer that everyone used the same inferior program of instruction even if it meant taking away more effective teaching methods from their best teachers. And all they need is a study supplied by the program’s publisher to justify this decision.

Am I against data-driven decision-making? Not at all. I certainly use it to make decisions in my business and in my personal life. But when I check my website stats with Google Analytics, I know I’m getting honest numbers, and I know that I’m measuring things with numbers that numbers were meant to measure. When I check my investment portfolio against the returns of popular indices, I know how to tell whether I’m investing wisely or if I need to consider different vehicles to reach my goals.

Even though past performance is no guarantee of future returns, I am happy to use numbers in ways that are appropriate and responsible. But if a broker hands me a study and tells me to move my money into REITs during a housing bubble, should I take his advice just because he has a study? At the same time, how I can make a decision to buy credit default swaps if I don’t even know they exist? If I don’t keep up with the latest thinking—because I don’t understand even simpler ideas—how will I be able to use something like a “Black Swan” strategy to maximize my return, and manage risk, in a chaotic market?

Of course, this is what brokers are for, right? Well, maybe not. Brokers are just people, too. They read their studies, they study their stats. And often they merely follow the herd. Too much groupthink. Too much reliance on second-, third-, or nth-hand information.

When we cede our judgment to things we don’t understand, we tend to make poor decisions. Worse yet, as a group, we tend to follow other people who make poor decisions in the hope, however ridiculous, that they must know something we don’t. This is what I have seen in education now that we’re all using “research-based” practices and data-driven decision-making.

People don’t choose practice now, research does. People don’t make decisions either, data does. Chanellor Rhee didn’t fire those teachers. Data from the IMPACT evaluation did. We don’t have the largest achievement gap in the nation here in Chapel Hill; our new metrics show little variation at all between majority and minority students. (In some cases, there is no variation at all. Why? Because the metrics were defined to hide the variation that exists.) Lies, damned lies, and education statistics! We’ve got an opening next year for a new superintendent. Oh, if D’Israeli would rise from the grave and take the position.

If people are forced to justify their decision-making with research, and if as a result, research becomes the primary justification for decision-making, the market for research will grow dramatically. If the market grows dramatically, the quality of the work will vary widely. If educational decision-makers don’t understand research, variance in quality will be irrelevant, and research will become commoditized. If people who make decisions decide that all research is created equal, then no decision can be said to be any worse or any better than any other provided any research can be found to support it.

As a result of the requirement to base decisions on research, and the resulting commoditization thereof, people who don’t make good decisions to begin with don’t gain the ability to make better decisions in the future. What they do gain, however, is plausible deniability for their decision-making. As such, in an unprecedented era of educational accountability, we may be less accountable now than we used to be. As Flip Wilson might say if Dr. Geraldine Jones were a school superintendent: “The data made me do it!”

The strength of data-driven decision-making is also its Achilles’ Heel. The decisions we make are only as good as the data we have to make them with, and our ability to interpret that data in rational ways that lead to positive results. If we’ve got bad numbers, or if we don’t know how to use good numbers well, our decisions won’t pan out. But if data truly drives our decisions, we are not driving them ourselves. Therefore, we are not personally accountable. If we allow our accountability to reside in statistics, formulae, and the conclusions of others, we admit that it does not reside in ourselves.

Though well-intentioned, our desire for research-based programs and practices, and our unshakeable belief in the inherent rightness of data-driven decision-making, have sent reform off the rails just a bit. Without good research, and proper application, we’ll never reach our destination. Unfortunately, in the short run, educational research and data-driven decision-making have decreased the likelihood that insightful dialog will occur in our schools, reduced our potential for innovation, and stifled even the most reasonable dissent.

School officials have never been keen on explaining their reasons for their decisions. Now, thanks to data-driven decision-making and the notion of “research-based” educational programs and practices, they need not explain much of anything at all. All they have to say is, “We’ve got data on that!” and no further questioning will be tolerated.


YEAH, BUT WHAT’S THE REAL PROBLEM WITH EDUCATIONAL RESEARCH?

Since I’m not a member of the academy, I can’t speak to issues academic. I read a lot of educational research but very little of it squares with my experience in education. I’ve worked in hundreds of schools and thousands of classrooms so my vantage point is mundane, prosaic, quotidian. When I read research, I often think of myself as looking up at some grand theoretical model from the humble linoleum floors of the classroom.

Still, I’m curious and I want all the information I can get. So I read many studies and policy papers, and I scour raw data when I have the time and patience to do so. Often, however, when I get down to the fine print, or the source data, most of what I discover is that the phenomenon in question fairs poorly and that results are at best inconclusive or statistically insignificant.

Today, for example, I received in my inbox a craftily worded work from the Center on Education Policy. It seems they want very badly to make the case that state test scores are improving relative to the NAEP. But their own language about how the data was parsed, and the results interpreted, seems lawyerly, perhaps even Clintonesque in its attempt to redefine terms that already seem to have definitions. My favorite moment is the part where they try to defend the notion that “proficient” on a state test should not be matched to “proficient” on the NAEP but to the lower level of “basic”. Their reasoning for this reads as follows:

“…the term ‘proficient’ means fundamentally different things on state tests and on NAEP (sic). The NAEP definition of proficient is aspirational, signaling where the National Assessment Governing Board (NAGB) believes students should be and embodying the knowledge and skills that NAGB believes should be included in a well-designed curriculum for that subject area. To reach the NAEP proficient level, students must demonstrate ‘solid academic performance’ and ‘competency over challenging subject matter, including subject-matter knowledge, application of such knowledge to real-world situations, and analytical skills appropriate to the subject matter.’ To reach the NAEP ‘basic’ level, students must demonstrate ‘partial mastery of prerequisite knowledge and skills that are fundamental for proficient work at each grade’ (National Assessment Governing Board, n.d.).”

So is CEP saying here that “solid academic performance” and “competency…” are aspirational? And that we really should only look to states to make sure that kids have “partial mastery of pre-requisite knowledge…”?

Wiktionary defines “aspirational” as being ambitious. CEP seems to think that an ambitious education is one where kids exhibit “solid academic performance.” I don’t know about you, but that seems sort of B-plus-ish to me. Lotsa room here before we get into anything ambitious or aspirational. Worse yet, CEP seems to think that kids should be judged proficient if they demonstrate “partial mastery of prerequisite knowledge…”. This sounds pretty “basic” to me. And obviously it sounds basic to the folks who made the NAEP because this is how they define “basic” on their test.

The NAEP isn’t the “gold standard” of testing for nothing. It’s the “gold standard” because most of us believe it is the best test our nation has yet devised. That judgment may change some day. But for now, if the NAGB says a kid has “basic” abilities in reading or math, it seems disingenuous to equate that with proficiency and to elevate state test results on that basis.

But this is all just semantic jousting, isn’t it? And isn’t this exactly the problem?

Here we have a major study of a major issue in education and the publishers of this study have to use carefully crafted re-definitions and odd equivocations of vocabulary in order to show numbers that indicate a favorable result.

Knowing, too, that what we’re looking at here is state testing information, aren’t we at least going to wonder if the source data itself is valid? Yet I can find nothing in the study that assures me that the “New York Problem” was screened for. If states are pumping their data and CEP is pumping state results, this study’s conclusions could be highly inflated.

I’m not saying that the study’s conclusions are inflated, I’m merely saying that they could be—and that we have no way of knowing if they are. For that matter, CEP may not even know because states aren’t in the habit of divulging how they arrive at their results, least of all to the people who study them.


CUI BONO?

Whose interests are served by this kind of work? Why not show the data as it is? (Perhaps this depends on what the meaning of “is” is.) Why not use the terms as they are actually used? Why the lawyerly language? Couldn’t the data be interpreted just as accurately with traditional definitions of traditional terms? Is the source data valid? Questions abound. In fact, don’t most educational studies leave us with more questions than answers these days? (Don’t answer that!)

Now, I’m not fully equipped to evaluate research methodology. But I think I know a smoke screen when I see one. If, to do this study properly, language has to be tortured to justify a favorable conclusion, why conduct the study in the first place? Probably because someone paid for it.

Aye, there's the rub; for in that sleep of death what dreams may come, when we have shuffled off this mortal coil, must give us pause.

Pause, indeed. For I have heard it said of late that we are witnessing “reform by ransom”, and while I don’t subscribe to this theory myself, I think the precedent set by Race to the Top, and the idea previously floated that Title I dollars might be dolled out through a similarly coercive approach, makes many people nervous. And well it should. People who criticize people who criticize competition often don’t realize that the people they are criticizing don’t mind competition at all—as long the scoring is sensible, transparent, and fair. How would we rate RttT on these three criteria?

Do I think that CEP is a disreputable organization, or that they’ve got a bad study on their hands? Not at all. As educational studies go, it’s one of the best I’ve seen. But what I’ve seen is that virtually all educational studies suffer from the same problem: an overly legalistic definition of what counts, a definition so contorted that unraveling it to even the tiniest degree unravels the entire study.

The recent and rather masterful unraveling of the Hoxby study is a prime example. With simple sleuthing and spreadsheeting, a savvy analyst shredded Dr. Hoxby’s statistical methodology and exposed her conclusions as biased and incorrect. And yet, pro-charter folks still hail this study as the definitive work in their domain. If they seem to have no response to the de-Hoxbyizing of their movement, it’s probably because they know they don’t need it. They already have a study showing that charters work. In a world where every study is as good as every other, no study can be invalidated once it exists.

Let’s not blame the charter folks for pushing their side of the argument; let’s blame ourselves for not taking a serious look at a study that was seriously flawed—the day it came out. Again, would something like this have gone unnoticed for so long in any other discipline? This is more than just a peer review problem, isn’t it? Deep down in places we don’t talk about at parties…. Well, no need to repeat that one. We want studies on that wall because we don’t want to walk that wall ourselves. We’re not sure what to do. We know that what is happening isn’t good. We know that lives are on the line. And we’re unwilling to take responsibility for our results.

CEP’s study may not be a bad study. But I have no way of knowing that given the way the data is interpreted. And I’m not a savvy spreadsheet jockey with time to investigate the minutiae of their methods. So how am I to interpret CEP’s conclusions? If I am an ed policy person, am I to interpret that current policies are working or that current policies are failing? Or am I merely to conclude that we need to make definitional adjustments to state tests and the NAEP so that “proficient” means “proficient”? But if this is the case, should the states come up or should the feds come down? Should we go off the gold standard? Should we let the currency of educational decision-making float? Sure would make things easier. But I don’t think it would make things better.

CEP rightly refers to the NAEP as the “gold standard” yet clearly all that glisters clearly isn’t gold in their estimation. State testing systems with decidedly tarnished reputations can be looked upon with great favor in this study so long as we can agree on an unusually modest definition of the word “aspirational” and that the term “basic” really means “proficient”. I’ll admit that it’s a silly argument. But sadly, it’s the only argument we have in this situation.

If my analysis here has left you bored and frustrated, join the club. I read half a dozen of these studies every week and I’m completely fed up with statistical spin and lawyerly language. We all know the value of simple metrics simply measured. We run our businesses on them, we run our lives on them, some of us probably even run our families on them. We all tell our children that honesty is the best policy. We don’t say, “Honesty is the best policy—just remember to put the truth down in the fine print using complicated language no one can really understand.”

Studies like CEP’s are everywhere these days. I don’t even have to look for them. They materialize in my inbox like junk mail ads for Viagra. So I’m left to wonder, who benefits from all this research? I certainly don’t. I know my local education leaders don’t. I suppose my state department of ed might like to put out a press release saying that someone said they were doing a slightly better job than the horrible job done by the people who preceded them—as long as by “better” we mean “basic” redefined as “proficient”, and “proficient” redefined as “aspirational”.

This isn’t science, is it? It’s really Language Arts.


SO WHY DOES THIS HAPPEN?

Perhaps the worst aspect of educational research for me is that I no longer enjoy social events. As soon as people find out that I work in education, they ask me about test scores and studies, and I have to explain what I see going on. As the situation is complex, it does not yield to the characteristic brevity of cocktail party banter. Even when someone buttonholes me for 15 or 20 minutes, they end up slack-jawed and incredulous, asking over and over again, “What are you saying? How can it be like this?”

Well, I'll tell ya how. There are two really exciting things about education these days: there’s a lot of money in it and there are some very easy ways to influence the system. We’re very close now to a national mono-curriculum. And only a few years away from having two tests that will be so similar, they might as well be one in the same. We have only one large and truly solvent educational publisher left. And most philanthropy is dolled out by just three organizations.

With the government staunchly behind such a small set of policies, education is ripe for a hostile takeover. But this battle won’t be won by accumulating shares, dodging poison pills, and waging proxy wars. The strategies of choice will be philanthropy, legislation, and research.


ARE YOU SURE THIS IS REALLY HAPPENING?

I can’t speak for anything but my own personal experience. My company has worked in a few hundred schools, taught in a few thousand classrooms, attempted to help maybe 20,000 teachers. In all, a tiny faction of the totality that is American education.

I don’t do research myself. I don’t know stats. I don’t have an MPP. I don’t even know how to set up a Pivot Table in Excel. But time and again I have seen research used to stop good things from happening. I have seen it used to smother change, derail innovation, and stifle dissent. I have seen it used to send large amounts of money to one person or one company, or large degrees of influence to one group or one ideology—while little of true value ever goes to teachers or children.

What I have never seen, however, is research used to improve anyone’s education. I have never seen anyone take a study into a school district office and say, “Here’s a really good thing. Let’s do it!” I’m not saying this hasn’t happened. I’m just saying I’ve never seen it.

I’ve been in more closed door district policy meetings than I like to admit. They always make me nervous because I know that at some point, one of my clients is going to ask for my opinion, and that I am going to give it to them. No, that multi-million dollar program is not as good as the publisher says it is. No, that particular curriculum is not as rigorous as your state says it is. No, your test data is not as valid as you think it is. No, that study is not as solid as it seems.

And then it’s time for me to go home.

My work in education virtually always comes to a screeching halt whenever I point out anything untoward about a study or a stat. So maybe this is all just sour grapes, right? Maybe educational research is great and I’m the greedy bastard in the game. If that’s your opinion, I’m sure you can find a study to support it.

On the Fordham Institute website this morning, there is a post that arises out of this National Journal discussion. The post is very aggressive and is about NEPC, me, and – in particular – my colleague, Professor Alex Molnar. I won’t presume to respond for Alex. But beyond the personal attacks, the post is disappointing to me in that it avoids the substance of the think tank reviews and the issues they bring to the surface. In fact, there’s absolute nothing there about the specific merits of any think tank review itself, beyond an unsupported, broad intimation that the reviews are of low quality and are biased.

There are many conversations worth having. It’s worth having a conversation about the value of peer review. It’s worth having a conversation about the use, non-use, and misuse of research in policy making. It’s worth have a conversation about the nature of objectivity. It’s worth having a conversation about the quality of think tank reports. And it’s worth having a conversation about the quality of the reviews that we produce. While none of these importing issues are taken up in the Fordham post I read this morning, we do hope to have those conversations in the future.

I have enjoyed my communications with Fordham’s Mike Petrilli in the past and hope to continue those positive interactions with him and with others at the organization. We invite Fordham to work with us to organize a public forum to take up these important issues. By sticking to the substance of our differences – which is a core principle of my own work and of our work at NEPC – we can better inform our respective (and mutual) audiences.

Dear debate participants on this blog thread: You are all wrong and -- even worse -- not coming clean about the underlying issues of self-interest. Yes, ed schools can be dogmatic and social science research can be shoddy and obscure (though it's getting better, as Rick noted). Yes, education think tanks have turned into shiny-shoed PR operations (or, better yet, journalists!). But what's really going on here is ed schoolers and think tankers trying to discredit each other so as to soothe egoes, win research dollars, and prevail politically (or at least ideologically). And the real issue is that even quality research is regularly ignored by lawmakers, the press, and the public.

Are things really moving in a positive direction, as Mr. Kress suggests? What are the criteria or measures that would support that conclusion? I clicked through on the link he provided, and the practice guides at IES do indeed generally look helpful. But in terms of the overall direction of our schools, are we better off today than we were 10 years ago? Is policymaking more or less evidence-based than it was then? I wish I saw progress, but instead I see the same thing Diane does -- Compared to research and evidence, ideology matters most to policymakers and the media.

There was a crisis ten years ago, but progress has been made. Look, for example, at the solid and practical research on timely and important topics available in IES practice guides published since the mid-2000s.http://ies.ed.gov/ncee/wwc/publications/practiceguides/

Will we sustain, and indeed increase, this recent commitment to solid research? I don't know.

We've moved to more "diamonds" and less "dirt," but, without a commitment to the standards and criteria inherent in scientific research, we'll easily return back to the world in which "research" is just another person's opinion.

The IES publishes its criteria for separating junk from science. It'll be interesting - and important - to see if these two groups use the same criteria.

I received the following response from guest contributor Bruce Baker, an associate professor in the graduate school of education at Rutgers University. His blog can be found at http://schoolfinance101.wordpress.com/, and his e-mail is bruce.baker@gse.rutgers.edu .

"I take issue with Rick Hess’ perspective that the need for Think Tanks to bypass the peer review process when producing education policy research is driven by the bias that persists throughout peer review in education research journals because peer reviewed outlets are invariably controlled by education professors. Full disclosure here - I’m not only a professor in an Ed School, but I’m also a reviewer for the Think Tank Review Project. I also sit on the editorial boards of the two major journals in my specific niche in education policy research.

I would point out that peer reviewed research journals publishing research related to education policy are not, by any stretch of the imagination, uniformly controlled by “ed schools” or “ed faculty.” There exist a wide array of public policy, economics, political science and other interdisciplinary journals that currently publish research related to education policy. These range from the Journal of Policy Analysis and Management, to the American Economic Review, to the recently created Journal of Education Finance and Policy. I would also argue that the editorial boards of journals such as Educational Evaluation and Policy Analysis have become increasingly diversified across social science fields including economics, political science and sociology. Education policy research and education policy research journals have become a melting pot of social science research. I agree that quality has improved as a result. But, it is a mischaracterization to argue that the majority of peer reviewed outlets are currently controlled by a single political ideology based in “ed schools.” For example, the journal Education Finance and Policy from MIT Press has an editorial board of over 40 members, far fewer than half of whom are based in “ed schools” and many of those based in ed schools are labor economists or public finance economists who have transitioned to ed schools. I would argue that the “ed school” monopoly to which Rick Hess refers is a myth.

That said, does political bias play a role in the selection of which research gets published and which does not? Probably so. I’ve certainly encountered it more than once, and from both sides of the aisle. Further, is it possible that the Think Tank Review project and Think Tank Review authors are at times, offering little more than a political counter-punch? That’s certainly possible as well. But I would point out that the Think Tank Review panel is also diversified beyond Ed Schools. I would also point out that the critical reader should look for the substance of the critiques offered, not just the political counter punch if there is any.

One advantage of the Think Tank Review process is that it engages top experts in critiquing at times, very complex policy analysis. Many of these individuals know, by their experience with specific policy issues or statistical issues, where to look for problems. The average reader can’t always do that. The Think Tank Review project provides the average reader or policy maker the opportunity to look into the peer review process. The project takes non-peer reviewed reports, and subjects them to the critique of the “cranky third reviewer.” In academe, we all know about that cranky, disagreeable reviewer #3.

My own personal experience with Think Tank Reviews has put me in a position to review stuff that doesn’t pass even the most basic smell test, but still requires a deeper look than it might get from the average reader or from the popular media. The most absurd piece I’ve been called upon to review was the Weighted Student Funding Yearbook, from the Reason Foundation. Here’s one of the conclusions from my review: “Perhaps most disturbing is the fact that in a third of the specific districts presented in the report, the evidence of success provided predates the implementation of the reforms, and the Reason press release makes the outright claim that past improvements are somehow a function of yet-to-be-implemented reforms.” Yes, this non-peer reviewed think tank report claimed that districts implementing weighted student funding had positive effects on student outcomes - but in 1/3 of the cases to which they referred, the outcome changes occurred before the policy implementation. This kind of absurd stretch of logic needs to be brought to light. This isn’t about a political counter punch to a libertarian think tank from a “liberal” ed school perspective. This is merely shedding light on an analysis that is wrong, by any account. But it takes time, effort and expertise (and a great deal of patience) to thoroughly evaluate these reports. Not every critique delivers a knock-out punch. Most Think Tank Reviews, and most chapters of the book offer well-reasoned critique of the methods of analysis and related findings, and most are written by individuals highly qualified to offer those critiques. This additional critical lens is a valuable public service."

There is plenty of rigorous research about all these topics, but advocacy think tanks – and too often policymakers – ignore it. Instead, we see shoddy research pushed to the forefront by well-funded marketing and media campaigns. The state of public discourse on education is woeful, with academic researchers conducting high-quality studies but talking mainly to one another, while along a parallel track run meaningful conversations between policymakers and well-connected advocacy think tanks.

The conclusion we reach in the Think Tank Research Quality book is not that there is a crisis of quality in education research. Instead, our conclusion is that the ways in which research affects policymaking are determined more by communications and networking strategies than by the actual quality of the research itself.

Drawing on our Think Tank Review Project, the book presents 21 reviews of recent think tank reports on key issues such as school choice, early-childhood education, education finance, teacher quality, and standards-based accountability. Our expert third party reviewers use the standards of academic peer review to scrutinize think tank reports, writing brief reviews in plain language that are then published on the Project website (http://nepc.colorado.edu/think-tank-review-project). Reviews are intended to support the public debate about education and to help a non-academic, non-expert audience of policymakers, reporters, and citizens assess the merits of the think tank reports reviewed. In our view, the best ideas emerge from an open process of rigorous critique and debate.

The real strength of a good review is in the analysis and the explanations it provides – explanations that help readers understand in what ways a report is sound and in what ways it is not. Our expert reviewers have found persistent violations of the fundamental tenets of social science research. Here are common examples:

• The selective use of research to bolster ideologically pre-determined findings;

• A failure to describe adequately the methods used or to provide sufficient data for other researchers to confirm the results reported;

• Flawed research methods, such as the confusion of correlation with causation, the failure to account for “selection bias,” or arbitrarily switching to nonstandard significance levels;

• Overstated conclusions; and

• Unjustified leaps from evidence to unsupported recommendations.

The Think Tank Review Project does not put forward its reviews as definitive, nor do we think of the Project as a “watchdog.” We publish honest reviews grounded in rigorous social science, and we value a public dialogue about the quality of the evidence offered to support education policy recommendations.

With that background in mind, let me then return to the question asked in the prompt, “Is there an area of education research—charters, teacher evaluation, and teacher pay, among others—that isn’t sufficiently rigorous?” The Think Tank Research Quality book includes three reviews of think tank reports on charter schools, one of which comes out looking pretty sound and two of which have serious flaws. The book includes two reviews of reports concerning teacher quality, and neither of those reports fares well. But our reviewers have pointed out that in each of these areas a great deal of rigorous academic research is available. The goal is to have policy guided more by such high-quality research and less by the shoddy stuff. We hope our Project contributes to that goal.

As others weigh in on the prompt and on our new book, I hope they’ll consider one core question: What additional steps might we as a community of scholars and experts take to help bring the highest-quality evidence to bear on policymaking? Our conclusion is that the current dynamic is governed by its own form of Gresham’s law, with the bad research driving out the good. Or perhaps this is better stated as a problem of incentives: the incentives to do good research are rarely linked to incentives to influence policy, while the incentives to influence policy are rarely linked to incentives to do good research. How might that change?

Yes, we do have a problem. Why do policymakers act despite evidence and research? Why did policymakers believe the press releases about the "Texas miracle" in 2001 without demanding rigorous studies? Why is the Obama administration promoting school closings and charters even though this strategy failed in Chicago? Why have charters become the darling of the media and the Obama administration despite research and evidence showing that they are extremely variable in quality?

Yes, we have a problem: Compared to research and evidence, ideology matters most to policymakers and the media.

Diane Ravitch

Of course, we need better education research. And we need peer-reviewed research. But the problem today is that policymakers are proceeding to impose their preferences despite good research and evidence to the contrary. Some researchers and think tanks release their "findings" to the media without peer review, and the media looks for the sensational headline. Why have charter schools become the darling of the Obama administration and the media even though the Stanford CREDO study and the facts on the ground show that their single most predictable quality is variability and that there are many shoddy and mediocre charter schools? Why do they persist in promoting charters as a panacea even though charter students have never outperformed regular public school students on NAEP? Why does the media ignore the abysmal results of Arne Duncan's Renaissance 2010 in Chicago, which is apparently his model for the nation? Why do the media ignore the mountain of studies about the unreliability of value-added models to assess teacher quality? Why do they never ask about what high-performing nations have done to become high performing?

Yes, we have a problem. Ideology trumps evidence, research, even common sense.

Diane Ravitch

Rick is right, overall. But he neglects to emphasize an integrally related problem, which is that essentially all would-be ARBITERS of research and research quality, including the two outfits that are bringing out this new book, are nobody's idea of impartial or unbiased. They have a long history of doing --and promoting--research that advances their favored policy agendas (in this case, policies favored mostly by labor and the left) and of trashing research, however credible, that tends to validate or advance other policy directions.

There’s long been a dearth of significant, high-quality research in education. But there’s no new “crisis”. Indeed, matters are improving. The ability of think tanks and others to bypass conventional education journals and forums is only problematic if they are providing meaningful and unbiased quality control—and I have long thought that presumption to be unjustified. Rather, far too often, the reviewing process has turned more on the biases of education professors than on the rigor of the research in question.

As I note in my forthcoming book The Same Thing Over and Over, seeking to challenge education dogmas has too often been brutal, exhausting, and career-killing work for would-be academics. Allowing researchers an opportunity to bypass politicized an ideological gatekeepers has been, on balance, a plus for the field of ed research.

Overall, the quality of educational research is higher today than it was five years ago, much less, fifteen years ago. This is due to Russ Whitehurst’s relentless efforts in his tenure at IES, maturing data systems, the ability of new technologies to break the publishing stranglehold of the status quo, and the consequent emergence of a rising generation of sophisticated education researchers less wedded to education school norms and mores.

There are dramatic changes in how research is disseminated. In recent decades, new technologies, new data sets, and the emergence of new entrepreneurial providers (like the NewSchools Venture Fund) and advocacy and research operations (like the Education Trust or the Fordham Institute) have upended the comfortable gatekeeping role of post-World War II ed schools, national associations, and education journals. This has discomfited those in the bastions of the old order, like my good friends Kevin Welner and Alex Molnar, and fueled some pretty excited attacks on voices and findings that were stifled or shut out by the old order. I find it telling that Kevin, Alex, and their colleagues and predecessors never expressed similar concerns when the handful of establishment entities with the resources to publish and disseminate in an earlier era—outfits like AACTE, the Learning First Alliance, NCATE, the NEA, NSBA, or CCSSO—would self-publish influential research or reports more to their liking.

Now, there is a continuing, worsening embrace of what I’ve called “the new stupid” in how data and research are used, but that points to problems with how practitioners and policymakers use the research that they read. As I argued a couple years ago in Educational Leadership, “Today's enthusiastic embrace of data has waltzed us directly from a petulant resistance to performance measures to a reflexive and unsophisticated reliance on a few simple metrics.”

Helping tame such behavior does call for thoughtful efforts to critique and consider all kinds of research—however it’s published. I welcome efforts by the Think Tank Review Project to be one contributing filter in such an ecosystem; but it’s misguided either to regard that effort as an arbiter of good research or to take at face value complaints that breaking the stranglehold of the education professoriate is bad for science or schooling. For those interested in all this, I’d recommend taking a look at the collection of terrific essays by the likes of Jeff Henig, Jimmy Kim, Dan Goldhaber, and Dom Brewer in my 2008 Harvard Education Press book When Research Matters.