Do schools need independent auditors? I think the question pre-supposes that most of the shenanigans that go on around test scores happen at the school level. They don’t. They happen legally and often in plain sight at the state level.

Everybody wants higher test scores – especially the higher-ups in state departments of education and in state legislatures and governor’s offices. But states can make their tests as easy or as hard as they choose. And they can change things as often as they want. No auditor is going to find anything wrong with schools that merely comply with state testing changes. But that’s exactly where most problems occur.

What I would support is an independent reviewer who would report, in plain language, each school year, the state of every state’s testing system. In general, the public at large has no idea how corrupted state test scores have become over the last decade. In our state, for example, we began with tests that were far too easy. Then, three years ago, we made the tests harder. Scores went down as a result. But that makes state leaders look bad. So they added a “retesting” component to the process this year that gives kids who fall just short of passing a second chance to take the test. Needless to say, this change had a dramatically positive effect on scores. So now we have harder tests that are easier to pass. This makes no sense. Furthermore, all the changes that have been made in the last three years make it impossible to compare scores over the 13-year history of high stakes testing in our state. It really isn’t possible anymore to tell if or how much our kids have improved. Yet few people know about this because our state Department of Public Instruction says they now have a clearer picture of student achievement, and our local newspapers simply don’t cover these issues thoroughly.

Well-trained state testing reviewers could also provide complete explanations of how state testing systems work. Recently, I spent 16 weeks, more than 20 e-mails, and one long phone call with the fellow who heads up our state testing system. Nice as he was, he was quite reluctant to explain to me how the testing system worked. His claim was that it was just too complicated for someone like me to understand – even though I’ve studied the issue for a long time, read widely about it, and even have a working understanding of contemporary testing concepts like Item Response Theory. Frankly, I don’t think he’d ever been asked about the testing system in detail and just didn’t want anybody to know. To this day, I’ve never received a complete set of answers to my questions.

So, the kind of auditing we need in education today should not be targeted at the school level. Yes, some schools have fudged their data; some districts, too. But these small-scale improprieties pale in comparison to the outright perversion of testing data by states who change the way their tests are scored, the types of items they include, where they set arbitrary cut-off points for passing and failing, and the process they use to administer their tests.

The taxpayers who pay for these tests have a right to know what they measure and what they mean. They also have a right to know the rationales behind changes that occur from year to year. Finally, every citizen in every state needs to understand how their state tests compare in difficulty to the tests of other states. While none of this seems excessive or unusual to me, I note that none of it exists. The only comparison data we can look at comes by way of benchmarking individual state scores to the NAEP test. But then, this begs the question: Why doesn’t everyone take the NAEP or some single national set of tests?

As the old joke goes, the reason we’ll never have national testing is that Democrats don’t like the word “testing” and Republicans don’t like the word “national.” And yet, with the development of national standards already begun, it seems we are indeed moving in this direction.

A national test providing both criterion-referenced and norm-referenced data may be distasteful to some, but it provides a solution for many. Should this ever occur, auditors/reviewers will only be needed at the national level and, in due time, we may begin to understand how our tests work and if and what our kids are learning.

As something of an obvious postscript, I’d like to point out that we have had a form of national testing for a long time. The SAT and ACT tests represent the highest of high-stakes tests and are taken by 2-3 million kids each year across state lines. More and more, even the inequity of access to test prep courses is being solved by high schools providing this information through normally scheduled classes. Most people seem to have accepted these tests as necessary and useful. I’m not sure why they couldn’t be used as a model for other tests we may wish to give.

Anyone who knows me knows that I’m not a big fan of testing. But I recognize its potential to improve education. What I typically fight against is what goes on routinely at state levels when tests are manipulated with an eye toward changing the data the comes back from schools in subsequent years. In general, harder tests serve us better than easier ones even if scores are low in the first few years. Beyond that, we should work diligently to control how and how often kids can take these tests. When tests can be taken multiple times (and, yes, I’m aware that kids can take the ACT and SAT multiple times), we can’t really be sure where individual knowledge ends and test familiarity begins.

I think it’s also important to note that testing has been with us for a very long time. I remember taking several tests in various subjects and in various grades when I was in school during the 60s and 70s. In general, these were norm-referenced multiple-choice tests in core subjects – not much different than many of the tests states are using now. Of course, back then, we pretty much ignored the data. Unfortunately, today’s state test data deserves to be ignored as well because virtually every state has corrupted its own process by making so many changes that year-to-year reliability has been compromised.

Finally, the era of high-stakes testing has ushered in a very dangerous form of teaching known as “teaching to the test.” It’s easy to see why so many schools and districts have taken this approach. But it’s also easy to see how it affects teaching quality and, ultimately, how it corrupts the teaching profession. Whether we end up with auditors, reviewers, inspectors, or a neutral test proctor in every classroom, we will not correct this problem without a significant change in teaching culture and teacher training. Rather than teaching to the test, teachers should teach to their students. This is the only sensible approach. But the more testing comes into play, school seems to make less and less sense to everyone in it.
 

Collapse

Do schools need independent auditors? I think the question pre-supposes that most of the shenanigans that go on around test scores happen at the school level. They don’t. They happen legally and often in plain sight at the state level.

Everybody wants higher test scores – especially the higher-ups in state departments of education and in state legislatures and governor’s offices. But states can make their tests as easy or as hard as they choose. And they can change things as often as they want. No auditor is going to find anything wrong with schools that merely comply with state testing changes. But that’s exactly where most problems occur.

What I would support is an independent reviewer who would report, in plain language, each school year, the state of every state’s testing system. In general, the public at large has no idea how corrupted state test scores have become over the last decade. In our state, for example, we began with tests that were far too easy. Then, three years ago, we made the tests harder. Scores went down as a result. But that makes state leaders look bad. So they added a “retesting” component to the process this year that gives kids who fall just short of passing a second chance to take the test. Needless to say, this change had a dramatically positive effect on scores. So now we have harder tests that are easier to pass. This makes no sense. Furthermore, all the changes that have been made in the last three years make it impossible to compare scores over the 13-year history of high stakes testing in our state. It really isn’t possible anymore to tell if or how much our kids have improved. Yet few people know about this because our state Department of Public Instruction says they now have a clearer picture of student achievement, and our local newspapers simply don’t cover these issues thoroughly.

Well-trained state testing reviewers could also provide complete explanations of how state testing systems work. Recently, I spent 16 weeks, more than 20 e-mails, and one long phone call with the fellow who heads up our state testing system. Nice as he was, he was quite reluctant to explain to me how the testing system worked. His claim was that it was just too complicated for someone like me to understand – even though I’ve studied the issue for a long time, read widely about it, and even have a working understanding of contemporary testing concepts like Item Response Theory. Frankly, I don’t think he’d ever been asked about the testing system in detail and just didn’t want anybody to know. To this day, I’ve never received a complete set of answers to my questions.

So, the kind of auditing we need in education today should not be targeted at the school level. Yes, some schools have fudged their data; some districts, too. But these small-scale improprieties pale in comparison to the outright perversion of testing data by states who change the way their tests are scored, the types of items they include, where they set arbitrary cut-off points for passing and failing, and the process they use to administer their tests.

The taxpayers who pay for these tests have a right to know what they measure and what they mean. They also have a right to know the rationales behind changes that occur from year to year. Finally, every citizen in every state needs to understand how their state tests compare in difficulty to the tests of other states. While none of this seems excessive or unusual to me, I note that none of it exists. The only comparison data we can look at comes by way of benchmarking individual state scores to the NAEP test. But then, this begs the question: Why doesn’t everyone take the NAEP or some single national set of tests?

As the old joke goes, the reason we’ll never have national testing is that Democrats don’t like the word “testing” and Republicans don’t like the word “national.” And yet, with the development of national standards already begun, it seems we are indeed moving in this direction.

A national test providing both criterion-referenced and norm-referenced data may be distasteful to some, but it provides a solution for many. Should this ever occur, auditors/reviewers will only be needed at the national level and, in due time, we may begin to understand how our tests work and if and what our kids are learning.

As something of an obvious postscript, I’d like to point out that we have had a form of national testing for a long time. The SAT and ACT tests represent the highest of high-stakes tests and are taken by 2-3 million kids each year across state lines. More and more, even the inequity of access to test prep courses is being solved by high schools providing this information through normally scheduled classes. Most people seem to have accepted these tests as necessary and useful. I’m not sure why they couldn’t be used as a model for other tests we may wish to give.

Anyone who knows me knows that I’m not a big fan of testing. But I recognize its potential to improve education. What I typically fight against is what goes on routinely at state levels when tests are manipulated with an eye toward changing the data the comes back from schools in subsequent years. In general, harder tests serve us better than easier ones even if scores are low in the first few years. Beyond that, we should work diligently to control how and how often kids can take these tests. When tests can be taken multiple times (and, yes, I’m aware that kids can take the ACT and SAT multiple times), we can’t really be sure where individual knowledge ends and test familiarity begins.

I think it’s also important to note that testing has been with us for a very long time. I remember taking several tests in various subjects and in various grades when I was in school during the 60s and 70s. In general, these were norm-referenced multiple-choice tests in core subjects – not much different than many of the tests states are using now. Of course, back then, we pretty much ignored the data. Unfortunately, today’s state test data deserves to be ignored as well because virtually every state has corrupted its own process by making so many changes that year-to-year reliability has been compromised.

Finally, the era of high-stakes testing has ushered in a very dangerous form of teaching known as “teaching to the test.” It’s easy to see why so many schools and districts have taken this approach. But it’s also easy to see how it affects teaching quality and, ultimately, how it corrupts the teaching profession. Whether we end up with auditors, reviewers, inspectors, or a neutral test proctor in every classroom, we will not correct this problem without a significant change in teaching culture and teacher training. Rather than teaching to the test, teachers should teach to their students. This is the only sensible approach. But the more testing comes into play, school seems to make less and less sense to everyone in it.
 

Collapse

An independent auditor may be able to provide a more accurate analysis of test scores.  However, only local, state and federal school agencies and policy makers look at the data. Most parents or students do not pay any attention to the massive amounts of student data that is collected by school districts. The data should be used by the consumer. I have participated in many parent meetings as a teacher and administrator. Most parents still want to know what their child’s grade is. Parents do not ask what does my child know, what can they do and are they on target to graduate? And when they graduate, can they get a job or continue their education?

I have been involved in week long accountability processes where a group of state level department staff and local school district educators visit a local school district to determine if the district is accountable by providing a high quality education for their students. This process did not change the system. The system will change when the consumer demands change. Policy makers that are concerned about the amount of money expended on education will not be able to force change in the system by policing accountability systems.

There is plenty of data that indicates we need to change some practices in public education. A united effort by all stakeholders to raise our children to be productive, responsible citizens and not allow excuses for not taking advantage of opportunities is necessary for change. School administrators can lead effective change. Research on effective school leadership is available from Mid-continent Research for Education and Learning (McREL) at http://www.mcrel.org .

Collapse

 Concerns of districts “spinning” student performance data are increasingly widespread as high-stakes tests take center stage in U.S. accountability systems.  Having an independent auditor or inspector to evaluate such data is not a bad idea, but by itself, it’s not likely to accomplish much to improve school systems. An auditing agency that looks only at test scores is a very partial vision of the kind of school evaluations that would actually move us forward in improving instruction.  To secure a broader analysis of what and how schools are doing – and what will help them improve -- we should look at the inspection systems offered by higher-achieving countries such as Australia, Hong Kong, and Great Britain. 

In these nations and others that operate school inspectorates, trained experts, usually highly respected former practitioners, evaluate schools by spending several days visiting classrooms, examining random samples of student work, and interviewing students about their understanding and their experiences, as well as looking at objective data such as test scores, graduation rates, and the like.  In some cases, principals accompany the inspectors into classrooms and are asked for their own evaluations of the lessons, allowing the inspectors to evaluate the instructional and supervisory competence of principals.  Inspectors also play a role in assuring the quality and comparability of school-based assessments (as in England and Australia), as well as schools’ internal assessment and evaluation processes (as in Hong Kong).   

In most countries’ inspection systems, schools are rated on the quality of instruction and other services and supports, as well as students’ performance and progress on a wide range of dimensions including and going beyond academic subject areas, such as extracurriculars, personal and social responsibility, the acquisition of workplace skills, and the extent to which students are encouraged to adopt safe practices and a healthy lifestyle.  Schools receive extensive feedback on what the inspectors both saw and recommend.  Reports are publicly posted.  Schools requiring intervention are then given more expert attention and support, and placed on a more frequent schedule of visits.  (See Grading Education: Getting Accountability Right by Richard Rothstein for details about how such processes can support accountability.)

An American version of the inspectorate system, called a School Quality Review, designed by former members of the British inspectorate with U.S. educators, has been piloted in several states and cities, including New York, Rhode Island, and Chicago, and has recently been piloted among charter schools in Washington, DC with the support of the DC Charter Schools Association.  The National Center for Restructuring Education, Schools, and Teaching has a wealth of data and resources on school quality at its website, www.tc.edu/NCREST.

The process has proved an extremely effective strategy for enabling schools to get an objective look at their practices, creating an evidence base that honors the broader goals of education and complements test information, and providing diagnostics and recommendations that are essential for any serious improvement ultimately to occur.  Practicing educators who are among the members of the teams also learn directly about colleagues’ practices and how to evaluate education in ways that travel back with them to their own schools, creating a learning system across the district or state.   [These

Such inspectorates are proven effective in supporting both accountability and improvement in nations around the world.  While test-based accountability measures have gained greater prominence in the United States, they do not by themselves develop or ensure quality and equity in schools, particularly those that are under-resourced and poorly functioning.  Inspectors with educational expertise can help improve the quality of learning in our public schools, focusing on how to develop real results over the long term, rather than smoke-and-mirrors style quick fixes. Rather than just keep the box scores provided by tests, inspectorates offer real opportunities to identify the improvements and innovations that can have a real, lasting impact on the quality of student learning.

Collapse

To understand school quality, we need clear, simple, meaningful data that everyone can understand. Parents – who, along with students, are the real stakeholders – need to be able to answer the question: "Will this school put my child on track for college?"

Low cut scores and artificially inflated passing percentages don’t tell parents anything they need to know. Auditors can’t change this. All auditors can do is enforce an already flawed system, ensuring "accuracy" without any real meaning. The last thing we need is more bureaucracy in the form of auditing agencies.

Parents need simple answers to simple questions. For high schools: How many kids go on to college? How many get 4-year degrees? How many complete other kinds of post-secondary education? How many complete college-prep coursework? What’s the average SAT or ACT score? How many take AP exams and how well do they do?  

Ideally, the US will develop a common, national assessment benchmarked to rigorous, internationally-competitive national standards. This could be used to gauge quality in all grades, giving parents early, objective feedback about their child’s learning. It would go a long way toward providing transparent information on school quality.

If the best accountability system is active parents and citizens, as many commentators have suggested, then the #1 priority needs to be making sure parents are well informed. Accountability by policymakers for policymakers bypasses those with the greatest stake in the game. Empowered, informed parents — including lower-income parents — could catalyze change in a way no auditor possibly could.

Collapse

Steve Peha, President of his own Education Consulting Company, Teaching That Makes Sense, submitted the following:

Do schools need independent auditors? Past history suggests they do. 
But even a crack team of our country’s best psychometricians won’t 
improve education.

Yes, the data we get is bad. But it’s bad in two ways, only one of 
which an auditor can deal with. While an auditor can certify accuracy, 
he or she can’t certify that the data is put to proper use. And that’s 
where the real problem in the education data stream lies.

Here’s something that has bugged me about data-driven decision-making 
for years: No one I’ve ever worked with can make a decision based on 
the data they have.

I mean, if your reading scores are below 20, I suppose you could 
“decide” to put more  time, money, and effort into reading. But one 
hardly needs data to drive a decision like this.

Now that virtually everyone in the country is aware that most state 
data is tainted in some way, I would think that school leaders would 
stop using it. But just the opposite is true. It’s as if everyone 
believes that someone else’s data is messed up, but not theirs. I 
can’t tell you the number of principals I sit down with every year to 
“go over the data.” And when we’re done, most of them smile awkwardly 
and go about making the same safe decisions they would have made 
without the data.

Of course, it’s not just state test data that has this effect on 
people’s reasoning. Internal school assessments are poured over like 
the Rosetta Stone. Yet when I ask, “What do those numbers tell you 
about what to teach next?” the room goes oddly quiet.

When viewed from the “production” side, educational data looks like a 
boon. But from the “consumption” side, it’s a boondoggle. Whether our 
current system is producing accurate data or not is irrelevant if 
people can’t use it to drive sound decision-making. And believe me, 
most school leaders can’t.

Frankly, I’m surprised about how quiet people like Arne Duncan and 
Barack Obama have been on what could easily be termed a national “data 
crisis” in schools. As NCLB comes up for reauthorization, I suspect  the issues of data integrity and data use will be glossed over once again. Nobody seems to want to take this on. Yet one could easily  argue that it is the top educational story of our era. (See “Campbell’s Law” on Wikipedia for more details on the accuracy issue.)

Data is not our salvation; it is the ghost in the machine, the poltergeist that keeps laughing at us every time we try to catch him. 

Who ya gonna call? Ghostbusters?

Collapse

            As previous contributors to this week's discussion have noted, manipulation of test score data in the NCLB era is rampant. An auditor of these data would solve little, however. Even if the most egregious forms of manipulation – lowering cut scores, reducing the difficulty of test items, making test items more predictable – could be eliminated, a more serious problem would persist: NCLB and accompanying state accountability systems attempt to judge the overall quality of schools by reporting only on the most easily measured of their outcomes, basic skills in math and reading. This accountability system creates incentives for schools to distort their curricula, downplaying other important goals (including more sophisticated math and reading skills) in order to focus on basic skill test scores.

            Such distortion is not unique to education. In other public and private fields, analysts have long observed similar distortions. When police commanders attempt to hold officers accountable for arrest (or ticket) quotas, community relations and crime prevention gets forgotten. When Medicare attempts to rate cardiac surgeons by patient survival rates, the relatively healthier patients get priority for surgery. When the Department of Labor attempts to rate job training agencies by the number of unemployed workers placed in new jobs, finding the lowest-skill temporary jobs becomes the priority, with education and re-training programs forgotten.

            Harold Levy notes (below) that Generally Accepted Accounting Principles requires standardized financial data for public corporations. But no matter how standardized, these data alone tell us little. If even the most sophisticated analysts could agree on their meaning, the stock market (a way to bet on whose interpretations of standardized data are most accurate) would disappear. In the private sector generally, accountability policy almost always places greater reliance on qualitative judgment than on quantitative measures. Management experts recommend "balanced scorecards" and "360 degree evaluations," not management by numbers.[1]

            Here is a tragic example of the consequence of our current obsession with test scores in educational accountability policy. Public health experts report a national crisis of diabetes, early heart disease, and resulting shortened life expectancy in the adult African-American population. And they also note an increase in the obesity and overweight status of children, particularly minority children. Obesity and overweight status are among the predictors of diabetes and heart disease.

            Many minority children, including children in Chicago where math and reading test scores described in the Commercial Club report are an obsession, live in communities without safe places to play outside – school physical education programs can be their only opportunity for healthy exercise.

            Yet national survey data show that in the NCLB era, schools serving minority children are particularly likely to cut back on physical education periods, in order to boost test scores by providing more time for drill in basic math and reading skills. Even if these schools were successful in boosting such skills, they would be doing so at the risk of the health, productivity and longevity of the children they are charged with serving.

            Certainly, developing basic skills are a critical part of what schools should be doing, but it must be balanced with schools' many other responsibilities – teaching science, history and other academic subjects, developing an appreciation of the arts, music, and literature, promoting students' physical health and knowledge of healthy practices, improving students' character, sense of responsibility, ability to cooperate and resolve conflicts peacefully. A school which, even by accurate data, raises math and reading scores by diminishing appropriate attention to these other areas is not a school worthy of public confidence.

            Last month, a committee of the Broader, Bolder Approach to Education (BBA) campaign (www.boldapproach.org) issued a proposal for a new accountability system to replace our single-minded obsession with math and reading scores.

            BBA calls for a re-authorized Elementary and Secondary Education Act to expand the National Assessment of Educational Progress so that it reports state-level results in a broader range of both cognitive and behavioral skills, not only math and reading as NCLB currently requires. A sophisticated science assessment (including hands-on tasks) and a physical fitness assessment would be good places to start. As a matter of national policy, we need better information on how states compare in the broad range of educational outcomes. And state policy makers need to know where their state performance falls short within this broad range.

            * As Cynthia Brown recommends below, BBA calls for a re-authorized Elementary and Secondary Education Act to require states to begin the development of qualitative evaluation systems (inspectorates) for schools. Eventually, a sufficient corps of professional inspectors should be trained to visit schools on a regular basis to interpret test score data and report on the quality of instruction and administration, the balance of the curriculum, the quality of student work portfolios, and the maturity of student behavior. The reports of these inspection teams should be the primary accountability tool, replacing simplistic test score data as the means by which we evaluate schools.

            Other nations, recognizing the dangers of test score-based school accountability are far ahead of the U.S. in the development of such inspectorates. Schools are evaluated by such means in the U.K., in the Netherlands, in France, in New Zealand, and in Australia, to name the most developed examples.

            This "balanced scorecard" approach to school accountability will not be perfect. It will be difficult to ensure the consistency of inspectors' reports. But while making fine distinctions between schools may not be possible, this system will be able to identify schools in need of improvement, and be able to direct support and intervention to those schools. That is the purpose of an accountability system. Our current test-obsessed NCLB undermines that purpose.

[1] For a more detailed discussion of the experience in other fields with accountability by narrow quantitative measures, and of the use of qualitative judgment in private sector accountability, see chapter 5 of my 2008 book, Grading Education: Getting Accountability Right (Teachers College Press and the Economic Policy Institute).

Collapse

The problem is that our current "accountability" system mis-focuses attention on boosting standardized test scores through a punitive approach, thereby encouraging teachers and administrators to game the system. The Chicago Commercial Club – whose "Renaissance 2010" proposals Arne Duncan was implementing – now says the solution is policing. This is like addressing the destructive consequences of high-stakes uses of standardized tests on schools and classrooms by ratcheting up surveillance and punishment to counter cheating, rather than by changing the dysfunctional system.

In Massachusetts, an alliance of groups has proposed an accountability system that would include an inspectorate, modeled on those in England and New Zealand (which Richard Rothstein and the Broader, Bolder Alliance are now promoting); limited standardized testing, but without high stakes attached to those tests; and local assessments, including classroom-based evidence. The tests and local assessments would be anchored in (hopefully slimmer and improved) state standards. The Joint Organizational Statement on NCLB, signed by 151 national organizations, calls for including locally-based evidence in accountability systems.

Framed another way, assessment information used for accountability should be largely based on classroom performance over time. In other words, let’s look at actual student work and see if it’s getting better. Quality checks through re-scoring of samples of student work (moderation), visiting teams of experts (accreditation model), and comparison of standardized test results with classroom evidence, together would constitute a built-in audit function for quality control, not something pasted on top of dysfunctional high-stakes tests. Implemented properly, as has been done in some places in the U.S. and some other nations, the local assessments can avoid the danger that using different local assessments will hide inequality in learning outcomes.

Assessment data should be used for assistance instead of punishment. It’s becoming more common to hear the rhetoric of assistance from policymakers, and there is increased federal funding for improvement. But the improvement effort still occurs within a context of high stakes sanctions based on standardized test scores. The Forum on Educational Accountability, which I chair, has proposals for overhauling the accountability structure, as do others.

By using multiple sources of evidence that includes an inspectorate, and by shifting to building school capacity rather than attacking public schools, we'd be constructing a healthy system that would overcome the problems identified in Chicago and that are common across the nation under the test-and-punish regime.

Collapse

I’m with Rachel Tompkins and Mike Antonucci. Truly independent auditors exist, and they are called parents, and press. Given a chance to make any decisions about their child’s school, the availability of un-touched data, and a glimpse at the options, parents vote with their feet.  The press help feed that information, as well as demand answers, and thanks to the Internet the media is a much bigger, better watchdog than ever.

It’s amazing that anyone well-schooled in education politics would support the “independent auditor” idea, knowing that any shred of independence in this business is compromised by whoever holds the purse strings, those paid by the purse strings – and by those who lobby those who strings are tied, period.
 

Apropos to our conversation this week, the Wall Street Journal reported recently on the ability of states, like Illinois, to create the illusion of achievement by tinkering with tests.

The article points out that No Child Left Behind resulted in Illinois public officials creating new tests that made it easier for students to obtain higher marks. Evidence for this claim came from the report by the Civic Committee of the Commercial Club of Chicago (the same one linked to in our question.)

The story does a good job of teasing out the issue of self-evaluation and Secretary Duncan's involvement. Read the full article here

I've been a life-long student of government, and, yet, I'm totally confused by the idea on the table this week.

A state currently reports, or allows to be reported, "X" as education data.

If this proposal is adopted, and the same state hires someone other than the current source and who is supposedly "independent" to report the data, do we seriously think the state will then, all of a sudden, report "Y" or "Z"?

Data can be fudged.

Almost every large school district reports 90% average daily attendance yet their data also shows that 30% of the students are absent in total a month each year. Similarly, the state reading scores for both Chicago and Illinois show dramatic improvement, yet the NAEP scores are largely flat. This data may be able to be reconciled but right now, the public doesn’t know whom to trust.

Relying on misleading or bogus data is not new. When George W. Bush rose to power on the success of the Texas Miracle in education and persuaded Congress to enact No Child Left Behind it was in part of the back of the Houston data that ultimately was shown to be highly flawed and misleading. Rod Paige was hounded by the misleading statistical results almost his entire tenure. In contrast, Sec. Duncan has been vociferous in denouncing the accuracy of school data –as Diane Ravitch has pointed out in her blog post – and that alone merits commendation.

The most elemental data can be manipulated. Even definitional issues can be tough.

-is a GED recipient a graduate or a dropout?
-is a student who signs in at homeroom but cuts all his classes truly “present?”
-is a student who returns each winter to the Dominican Republic with his family a truant, a student with an "excused absence", or should his name be removed from the register? (and no matter how she's categorized,way how is she being educated?)

School administrators cannot be trusted to present their own results any more than AIG. Data needs to be standardized much like GAAP(Generally Accepted Accounting Principles) require of public corporations; analysis needs to be routinely made available to the public at large in an understandable way.

Parents should not need a degree in statistics to understand how their schools rate. Outside, independent auditors are the key.

Collapse

The best accountability system is active parents and citizens who pay attention to the schools, ask questions and hold elected officials on the school board (or mayors) accountable. This group in Chicago is part of such a system. I understand this is a quaint idea to many of my colleagues—maybe like rabbit ears and analog TV.

While most of the attention in the NCLB era has been paid to a few large cities, all of my attention has been focused on rural America. In some of the 7600 or so districts where 50% or more of the children reside in rural places, there are many examples of this successful accountability with high standards.

 I recall overhearing a reporter’s interview with a school superintendent in Nebraska a few years ago. When asked how many students dropped out, one superintendent said he had only been in the district nine years but they had no dropouts in those years.

 “Well,” queried the reporter, “how many went on to college?”

 “Other than a few who go into the military, all of our graduates go on to some type of secondary education.”

That’s not true in every rural district in Nebraska but graduation rates in rural districts there are always substantially above 90%.

 If we spent as much time encouraging active citizen engagement as we did devising tests and external systems of accountability, we might make some progress.

 As for independent auditors, I think there is merit in considering some system akin to the inspection regimes in some European and Asian countries that do well in all these international comparisons we love to knash our teeth over. The accountability group at the Bold Approach (www.boldapproach.org) has just issued some interesting thoughts on moving beyond the narrow test driven NCLB regime.

Collapse

We already have independent auditors. They're called the press.

If they're not getting the job done, let's find out why before we start creating another bureaucracy.

Misleading reporting of student achievement gains certainly isn’t new.  Remember the Lake Wobegon scandal of the late 1980s when most students in every state scored above average?  We can thank John Jacob Cannell, a West Virginia doctor for exposing this nationally.  Back then the test scores of choice were norm reference tests.  At least we’ve progressed to measuring student achievement against a standard.  But there is a long way to go, which is why the recent movement on national or common standards is important.  Still, such common standards won’t mean much if they are not tied to common assessments with uniform definitions and cut scores for proficiency as well as basic and advanced performance.

Independent state auditors won’t solve much of the problem.  A national one could, but then a common, national assessment might erase the need.  At the state level a kind of inspectorate would be more valuable. This may sound a lot like an independent auditor, but there's no reason why it couldn't be done with regional consortia, and it would have a far greater emphasis on instructional practice than an auditor.  

Is an independent audit and report on school performance necessary? Absolutely. 

Within the charter school sector, this is a core function of  quality charter school authorizers, the agencies that approve and monitor schools and decide which schools are good enough to stay open.  The best authorizers annually report on the performance of their schools, and strong authorizers like Thomas B. Fordham Institute in Dayton, Ohio, and the Indianapolis Mayor's Office have executed on this with a high level of professionalism that allows all constituents to take a long, hard look at the performance of schools within their

portfolio.   More importantly, authorizers also have the power to act decisively to close the schools that come up short in their performance.

The charter school sector separates the function of operating schools from the functions of monitoring and reporting on results and acting upon low performers.  This type of separation of functions would benefit all of public education.

Sweeping statements about what states are doing need parsing just like average scores need disaggregation.  All states did not lower standards, in fact the standards statements were by and large untouched.  Many states reduced cut scores on the summative high stakes tests required by NCLB.  In some cases the cut scores had been set too high in the zeal of the moment or out of a misunderstanding of what students should know and be able to do.   Independent auditors could shed additional light on student achievement if they had something other than the current crop of tests to look at.

Auditors would need a more complete picture of achievement because the tests are such weak indicators of learning that they provide only a glimpse at student learning.  The Broader Bolder group has suggested another way to get better information about student learning that would be less cumbersome than auditors.  And the common Core proposed by NGA and CCSSO offer another path to most transparency.  Auditor

Auditors would provide information to the public and policy makers but not to students who have been the victims of a hide the cheese strategy where learning targets and their progress towards the targets are not apparent to them.  Inshort where schools are not regularly visited by parents and community members auditors might help.  But independent auditors are not sufficient to take the next step in education by making learning targets and student progress transparent to parents, students, teachers and administrators.

Collapse

Updated at 8:50 a.m. on July 20.

For years, states have been lowering standards and districts have been passing along "juiced" test score increases. It happened during the NCLB era. It happened before NCLB came along. It'll probably still happen under whatever law comes next.

The only thing that's going to limit the practice is if lawmakers and bureaucrats are at least occasionally held accountable. And that's why the Civic Committee report presents such an important opportunity. During his stint heading the Chicago public schools, Arne Duncan touted some pretty hard-to-believe test score increases. These scores were passed along without much scrutiny during the process in which he became Obama's education secretary.

So far, at least, it seems like Duncan's getting a free pass. His press people offer evasive or defensive responses when queried about the scathing report. Meanwhile, the Secretary goes around the country chiding others for "lying" about school performance. Was Duncan unaware that Illinois was watering down its tests when he was in Chicago, or was he unwilling to tell the truth? We still don't know --no one's really pressed him on it. But it's not too late. And getting to the truth behind Duncan's involvement in the Illinois test score scandal will do more to prevent future manipulations of test score results than anything else I can imagine.

Collapse

Why Are We Lying To Children?

No Child Left Behind created a perfect storm of lying and fraud, all of it completely legitimate. The federal government told the states that every child must be proficient in reading and mathematics by the year 2014 or face serious sanctions. Schools that failed to meet their annual progress targets for six years in a row might have their entire staff ousted, be closed, turned over to state control, or converted to private or charter management. The mandate to states, districts, schools, and teachers was clear: Produce higher scores or die.

In a slight bow to federalism, the law also told the states to set their own standards, choose their own tests, and establish their own definition of “proficiency.”

What an invitation to creative thinking! Since no state or nation has ever managed to bring all its students to proficiency, the states came up with some neat tricks to ensure that their proficiency rates would climb—sometimes dramatically-- year after year.

Secretary Duncan is aware of this scandal. He told New York Times columnist David Brooks on March 13 that “States are lying to children. They are lying to parents.” On May 29, the Secretary said in a speech at the National Press Club that states have dumbed down their standards. He said, “What is most troubling to me on the standards issue is that far too many states, including the state that I come from, Illinois — I think we are fundamentally lying to children.” Children are told that they are meeting state standards, but the standards are so low that the students “are in fact barely able to graduate from high school.”

Secretary Duncan knew that Illinois was lying. The Civic Committee of Chicago’s report shows that the gains in Chicago to which President Obama referred when he first introduced Secretary Duncan were a mirage (i.e., a lie). Instead of dramatic improvement, Chicago’s students made modest improvements at best, and any gains “dissipated” in high school. The explanation? The state lowered the passing mark (or cut score) on its tests, which caused a big jump in test scores. Anyone who looked at Chicago’s scores on the National Assessment of Educational Progress would have known that the city had flat scores from 2003-2007.

Illinois, as the Secretary noted, is not alone.

New York dropped its cut scores from 2006 to 2009 and, not surprisingly, state scores soared. In 2006, third-grade students had to answer 44 percent of the questions on the state math test to meet state standards, but in 2009, it was necessary to answer only 28 percent correctly. Most people, thinking of their own school days, assume that a score of 44 percent is a failing score, not a proficient one, and 28 percent? Forget about it. To call such a score "proficient" is embarrassing; it's a lie.

New Jersey, on the other hand, has decided to “toughen” its standards. Up until now, third and fourth grade students were deemed “proficient” if they correctly answered only 40 to 45 percent of the questions on the state tests in mathematics and English language arts. But now the state board of education has decided that they will have to get 50 percent correct to be considered “proficient.” But didn't a score of 50 on a test mean you failed? Or used to mean that.

Certainly it would be worthwhile to have an independent auditor in every state, a professional agency whose mission is to tell the truth about student performance and graduation rates. What we have now are political appointees at the state and district level who think that their job requires them to show constant improvement, and if necessary, to fabricate it with statistical magic. State legislatures should demand independent auditors, and if they don’t, the U.S. Department of Education should insist on it.

Secretary Duncan is right. Our schools will not improve unless we have accurate information. Nor will they improve if Congress and the administration insist on imposing unreachable goals that invite--nay, demand--lowered standards and lying.

Collapse