I do believe that it makes sense for the Department to seek legislative relief from the eligibility requirement that districts receiving I3 funds make AYP for the two years prior to their application, but not because I support removing AYP requirements themselves, as do some others weighing in here. In fact, maintaining a high bar on the definition of AYP will be a very important tool for ensuring that districts and state education agencies continue to be intensively focused on the hard work of school and district turnaround that is critical to closing the nation’s achievement gap.

But much as we need to preserve AYP requirements overall in the upcoming debate on ESEA’s renewal, there’s really no reason why the I3 grant program should be strictly linked to AYP. Indeed, one of the central goals of this competitive grant program is not simply to direct investment into what’s already been working for years on end, but also to spur innovations that can bring dramatic academic progress to the districts where it’s needed most.

This distinction is well borne out by considering the situation in Connecticut. Forty-five of the state’s 166 districts did not make AYP last year, including every single one of the state’s urban districts. Yet several of these urban districts, most notably Hartford and New Haven, have embarked on ambitious reform plans. In part because Connecticut has not implemented a growth model for NCLB compliance, a district like Hartford, whose academic gains have tripled the state average over the past two years, and which is on pace to eliminate its achievement gap within less than a decade, would be ruled out for an innovation grant.

This elimination is despite the fact that this district has been innovating on numerous fronts, from an “all choice” approach to school selection for parents to the application of differentiated autonomy for schools based on performance and a weighted student funding formula that has dramatically boosted the resources actually reaching students for classroom instruction. Likewise, without relief from the AYP eligibility requirement, New Haven would also be ruled out for an I3 grant, despite the overwhelming ratification by their teachers’ union earlier this week of a breakthrough new contract allowing for teacher evaluation based on student achievement and charter conversions of failing schools.

Collapse

The purpose of the i3 Fund awards is not accountability but innovation, and restricting eligibility to those who had met AYP for 2 years would tell everyone who hadn’t met that hurdle, “Just give up on improving through innovation and getting support for that. We’re not even going to look at what you propose.”

Maybe I am looking at this in a simplistic way, but I think a basic effect size measure would be more practical than what Steve Peha proposes, largely because you do not want to make model assumptions about the relative difficulty of assessments when the main question is, “Does a program help children compared to not having the program?”

In the wake of flat NAEP test scores (which correlates to flat achievement, safe for some pockets of the country and some groups of kids), it’s a good exercise to ask just what is intended to be accomplished with our latest entry to the alphabet soup of education - i3. There seems to a lot of disagreement about whether or not AYP should be used as one criterion as districts apply. Tom Vander Ark says scrap them for this purpose, as does Diane Ravitch. But as Nelson Smith suggests, charters may be ill-affected in their pursuit because of the variety of AYP challenges they experience. KIPP DC’s Susan Schaeffler recently pointed out during a panel I moderated that her school failed AYP even as KIPP KEY Academy has become the highest performing middle school in the District of Columbia. So, her school is great, but the DC standards and assessment portfolio seems to ignore that. This is a disconnect Education Secretary Arne Duncan seems to appreciate. Indeed, as the New York Times article on the “sluggish” (how about pitiful?!) math scores points out, many states got away with lowering the requirements of their state testing regimes – “an unintended consequence of NCLB” that Duncan calls the “race to the bottom.” Why then does Duncan want to use such unreliable indicators to judge districts on i3?

Wait, I know the answer. It’s because there is nothing more reliable, right? NCLB may not be the best-implemented law. It suffers not only from state and district game playing but from the very political decisions made during its creation to leave the states and districts alone to come up with their own standards without benchmarking to NAEP - which many of us wanted and that, indeed, seemed to be the solution to exactly what ails us now.

Despite its issues, we do know more than ever before about how and what schools are doing for and to kids, and the very glaringly obvious achievement gap is now clearly recognized. Those of us who have long decried the state of public education (and have been stoned for doing so) have finally been vindicated. But progress is slow precisely because the levers that would allow change are locked up not only by low standards, but by a lack of real consequences, union dominion in schools and, yes, little choice for those who are stuck.

Innovation funds need to be spent, right? AYP is not always reliable. It’s particularly not reliable when the states’ standards are so low that schools failing by all other indications make AYP while schools like Susan’s fail. So why not offer an additional, comparative measure of performance and ensure enough good reviewers for i3 that such data actually gets used to evaluate applications? Any school or school district that can demonstrate solid gains over time – value added – and has a reliable data system to show it – gets into the i3 pool for further review. If they don’t – and you know most don’t – they get left behind.

How’s that for ensuring districts start doing more with data?

Collapse

What bothers me in some of the commentary here is the passion for eliminating the AYP requirement, period.

I can well imagine that there may be tremendously innovative and deserving applicants who do not reach AYP as currently defined. So fine; allow the exceptionally grand applications to "win" even in the absence of meeting the AYP requirement, but make everybody report their current AYP status. Just so we know.

AYP is not a perfect formula, but it is far superior to having student achievement get defined in so many ways that it has no defnition and children go invisible. Let's fix it, but let's keep the information we do have front and center and top of mind while we get it right.

I can certainly understand why critics and opponents of AYP (Monty, Diane) would want AYP removed from the i3 eligibility process. Ditto for the pro-innovation crowd (Tom, Nelson). And of course no one wants the money wasted. But there are practical, political, and legislative concerns to consider beyond the ideal and the ideological. Once you lift or modify AYP for districts getting federal money you've created yet another tier. We already have NAEP ratings, and state by state AYP, and waivers and pilots. We've got the common core stuff coming down the road, and Race to the Top, too, all of which could invite further AYP modificaitons. What happens next? What legitimacy do i3 districts have with their neighboring districts if their improvement is measured by something other than AYP? What little comparability exists is diminished, and an unweildy "rolling reathorization" of NCLB is begun.

Monty, I was born and raised on your side of the equation so I'm pretty much always gonna come down on your side in one form or another. Can't disagree in this case with a word you said. I, too, oppose the use of AYP and pretty much all of the "not ready for prime time" tools we have in our current accountabilty arsenal.

I think that Sandy Kress. however, had a good point and -- even more to the point -- he stuck to the confines of the question as posed while I and many others used it as a platform for knocking AYP and other ideas altogether -- whether they had anything to do with i3 or not.

Pertaining to the specific question about the use of AYP in i3, I think Sandy's point was mostly a practical one. What other -- better -- indicators of success do we have that could be substituted at this stage in the game? (I like the ideas you mention in your link but those -- sadly -- haven't yet come to federal fruition.)

You probably understand better than any person here the nascent nature of our accountability policies. But the pace of reform marches on -- often faster than any of us can keep up with it.

While I may not like AYP, I do agree with Sandy that we shouldn't give innovation money to schools that haven't had at least some modest level of success. AYP may not be a very reliable indicator in this regard. But it may be all we can come up with at the moment that is part of our federal accountability system.

I would bet that you, Sandy, and me (and probably most everyone else here) want to see i3 dollars and other targeted fed ed funds focused on the schools that will do the most with them. Helping good schools get even better is a smart strategy early in a reform period because it increases the likelihood of developing larger numbers of successful models that -- ideally -- could be replicated in short order. We take the same approach here at Teaching That Makes Sense when consulting with individual schools by giving the most successful teachers most of our consulting time. What we work for, early in a contract, is the quick creation new leaders who can help to bring others along.

Collapse

Sandy,

Yes, I agree. Hadn't thought about adding socio-economics into the mix but it's the best way to figure out who is gaining because of changes they are making, and who is gaining just because their schools draw from a well-educated and well-heeled community.

When I lived in Washington State they did a nice thing when they presented their state ed data online. One type of search allowed you to view test score and other relevant data in cohort groups of schools bounded by free-and-reduced lunch percentage ranges (for example, "give me all schools between 10% and 20% free-and-reduced lunch with reading scores above 75%"). This made it easy to find so-called "under-performers" and "beating-the-odds" schools.

The most interesting thing about this for me as a consultant was that many of my clients were schools that faired relatively well overall but ended up 12th out of 12 or 19th out of 20 in their socio-economic cohort.

These were very easy schools to "improve" of course because they had enormous amounts of untapped potential that wasn't very hard for me to tap. But, even though these schools could improve quickly, they would not be good candidates for innovation because their school culture was based on the assumption that since their families were fairly well off, their schools would always score high.

There's no way I can see for accounting quantitatively for school culture -- although I know some researchers have taken a shot at this. But it's probably the most important thing I've experienced as an indicator of the potential for innovation.

Collapse

Concurring with Diane, not only should AYP requirements be scrubbed from i3, but that entire component of NCLB should be completely overhauled ASAP. As several respondents explained, excluding needy districts based on AYP when most are now not making AYP will exclude some districts in which improvement is both sorely needed and possible with an infusion of well-used resources.

For those continuing to wonder about the consequences of NCLB, it is interesting to see once again the trend lines for NAEP improvement are flattening, continuing a trend that began in 2003, which is about when NCLB started to have an effect. Comparing 2003-2009 with previous periods (1993- 2003 and 1990-1996) we see overall trend rates in grade 8 math slowing. From 2007 to 2009, Blacks and Hispanics gained only one point, while Whites gained 2. From 2007 to 2009, Grade 4 math scores are flat overall and for most demographic groups. More evidence that punitive accountability attached to standardized test scores hurt, not help, school improvement.

I must disagree with my friend Steve Peha. Building an improvement system on the flawed and inadequate tests we now have, compounded by the fact that 'growth' modeling is not ready for high-stakes accountability, and we'll get another complete mess. Evaluating improvement and having improvement expectations make sense, but we need a richer set of indicators of learning and other relevant factors. For an example, see proposals from the Forum on Educational Accountability (FEA), which I chair). We also must not use tools that could help in determining who needs assistance of what sort for determining high-stakes punitive actions when those tools are not ready for the job, as is now the case with 'value added.'

Richard points to some interesting comments from Sec. Duncan, and reads them in an encouraging light. In his piece on the EPI website, he goes on to draw cautions based on such things as the actual RTTT proposals. RTTT is NCLB on steroids, extending to teachers the direct test-based accountability now inflicted, with great damage, on schools. Comments from

FairTest, NEA, AFT and FEA, as well as recent comments from the National Academy of Sciences, thoroughly dismantle the flawed thinking behind the RTTT proposals and suggest superior approaches to school improvement. I hope Sec. Duncan listens and that his recent comments cited by Richard indicate a shift in direction.

I also agree with Richard that despite my deep desire for a fast and thorough overhaul of NCLB, it is not likely to happen fast – and thorough is very much up in the air. Winning a real positive transformation in federal education policy will depend on public education, organizing and mobilizing as those who want to maintain or worsen NCLB have deep pockets.

Collapse

Steve - obviously the devil would be in the details, but I think you've made a good start. We certainly would want to put a premium on districts and schools that do an especially good job of improving achievement for disadvantaged students, don't you think?

Mr. Kress,

You are 100% "on the money" as they say. As for an AYP alternative, how about this:

1. Take state test data.

2. Index it relative to difficulty of all stat test against equivalent NAEP data so all schools in all states play on the same scale. (This has been done many times, I think, over the years.)

3. Create an "improvement factor" or an "improvement index" to qualify schools that have made decent gains over the last 3-5 years.

Schools that can't match some minimum degree of improvement, can't play. Reward improvement over time (favoring multi-year consistency over single-year "events") and give money to schools that have the best record of "sustained improvement".

This is similar to AYP but without some of the oddities that can occur whereby some well-performing or improving schools miss the mark for a single sub-group or one of the other AYP factors.

I'm not a data jockey, so I really don't know if this would fly. But it seems to me a better place to start. What do you think?

Well - at the risk of being the skunk at the party - I'll gladly and strongly go against the budding consensus here.

I don't particularly want to defend ayp in this comment. Just for fun, I would remind the readers that the ayp provisions of NCLB were universally supported in the Congress by both Democrats and Republicans. Further, I encourage all interested to google a very thorough and interesting article in the New Yorker in the summer of 2001 on the very subject of the ayp provisions. I don't want to take the space here, but you will be fascinated to see who was arguing for flexibility, and who wasn't!

As to whether ayp has worked, I'll lay off that one, too, save for suggesting that readers google the letter signed by most of the major civil rights organizations in 2007 strongly opposing any weakening of accountability in NCLB through the appropriations process. It, too, will surprise you.

What I want to say here is simply this. The feds have limited money to spend on these innovation grants. They have every right to invest the first monies with districts and others who have shown success. After all, while we want innovation to help all, we should entrust these funds to groups and entities who have shown success with innovation in order to promote effective practice. Then others can see successful innovation and presumably the use of effective, innovative practice will spread. Funding those who have not been successful, even though they, too, have needs, doesn't seem to me to be a smart use of limited dollars.

Now, if my fellow bloggers don't like the use of ayp, I enccourage them to devote some time to describing the objective criteria of success the Secretary ought to use in its place to assure that those receiving funds have the best track record of success, thus evidencing that investing in them in lieu of others makes sense.

So far, I haven't seen any such proposal. All I've seen is essentially trust the applicants or let them use whatever data they have and want to share. I'd hate to be on a peer review panel making judgments on that flimsy a basis - comparing applicants with different measures, different data, and different notions of what success means. Please know that I am well aware that making ayp means different things in different places and is, therefore, of limited use here. But bagging ayp with no other objective standard in its place will simply bring us back to the old days of either sending the money to everyone or sending it to favorites or those with the best grant writers.

Until someone does better than what we've seen so far, I'm proud to be a minority of one who answers Eliza's first question with a firm "no" and her second question with a firm "yes."

Collapse

Steve Peha just observed that “So far at least, Mr. Duncan has only indicated his desire to patch [NCLB] up, not to replace it.” This is true, but few realize how radical are the “patching” principles that Arne Duncan has articulated. For example, he has denounced a fixed achievement goal for all students as “utopian,” insisting that we should hold schools accountable for student achievement at all points in the distribution. He has insisted that schools alone are not responsible for learning, but that city parks departments, health services, social services, and others also share responsibility. He has said that accountability must not focus exclusively on math and reading, but must promote “a well-rounded education that draws children into sciences and history, languages and the arts.” Tacking an AYP requirement onto such principles is, I suppose, theoretically possible, but would require impractical complexity far beyond the existing capacity of either the federal or state governments. Developing this capacity will take many years; all that can be done immediately is to support states in experimental efforts to evaluate schools using methods which must necessarily, in part, be qualitative.

It is true that the Secretary has sent mixed signals. He does call for early re-authorization of ESEA, but the re-authorization principles that he has embraced make early action inconceivable. I have explored these implications in somewhat greater detail in an EPI Policy Memorandum, issued today and available here.

Collapse

What’s that you say? AYP doesn’t make much sense? Well that’s the shocker of the new fall season, isn’t it?

Since everyone here has been following ed reform for quite a while, I’m not sure that discussing the merits of using AYP for grant eligibility is worth too many more words. Clearly, according to everyone here, it’s not.

But what interests me more is Mr. Duncan’s habit of using this “policy as leverage” approach to get states and schools to change their ways.

Mr. Duncan may be new to his role as Secretary of Education but he is not an unintelligent or unambitious man. He obviously understands just as well as we all do that using AYP for just about anything is dubious at best.

So where did this idea come from? (Did someone at DOE just lose their job?)

More generally, does the “policy as leverage” approach make much sense at a time when much of what Mr. Duncan is trying to get states to do is either odd (like the AYP thing) or manipulative (a la RttP).

Perhaps the Secretary is using the only leverage he has. But what interests me more is why he needs much leverage at all.

So I ask, however naively, “Why wouldn’t states just want to do the right thing?”

Maybe they’re not so excited about reforming educatio. Maybe they don’t believe in some reforms (like charters, for example). Maybe they think federal reforms represent unfunded mandates (see Utah). Maybe there’s just too much that’s politically dangerous about taking ed reform seriously. Who knows?

And that’s the root of the problem.

Reform is hard, especially when you do it poorly, and politicians – at the state level at least – are getting more and more tired of taking risks on things we push down their throats when some of those things have marginal evidence of their effectiveness.

Maybe some parts of NCLB don’t help schools, teachers, and kids. I would say that’s something everyone agrees on. NCLB is so seriously flawed that many smart people are calling for us to start over again from scratch. Certainly there were many people opposing NCLB at the beginning who voiced the same criticisms then that most of us agree with now. What prescient voices those were; how thoroughly they were ignored or ridiculed.

So it’s no wonder that states are showing a bit of reform fatigue when it comes to processing ideas from the federal government. From where I sit, the federal government has one strike against it already. What’s worse, some of the more idealistic among us thought the Obama-Duncan combination would be a chance to move forward with bright, shiny, “Yes we can!” ideas that would erase past errors in judgment, as opposed to perpetuating them.

Is ed reform starting to feel like Afghanistan to you? Full of tough choices and no changes? NCLB has not been successful. It may even have poisoned the well. Let’s put it all behind us – including things like AYP.

Of course, Mr. Duncan is free to put whatever requirements he wants to put on federal education dollars. That is his right now that he’s in charge. But is it smart? I don’t think so. Nobody likes to be bullied. And how well will i3 and RttP winners do if they feel constrained or unprepared to live up to the Secretary’s demands?

Another thought: Why do we have so many requirements on the small money (i3 and RttP) when we had virtually no requirements at all on the big money (ARRA)? It would seem to me that putting thoughtful restraints on the use of $60 or $90 billion would achieve more than focusing all of our energies (and enduring a little controversy) constraining the use of $5 billion.

In the meantime, we’re holding off on NCLB reauthorization indefinitely. Mr. Vander Ark, in a recent post on his own website, made a good case for putting it off yet another year. So far at least, Mr. Duncan has only indicated his desire to patch it up, not to replace it -- and to reauthorize it sooner rather than later. Perhaps the more time he takes, the more change we will see. (Maybe he’ll take a call from Mr. Vander Ark on this issue.) Then again, he’ll have Congress to convince no matter what he comes up with, so what we get in the end may not be much different than what we have now regardless of where the Secretary stands.

At some point, and the sooner the better, we’re all gonna have to sit down for a big “Come to Jesus” meeting about ed reform. Are we willing to go another decade continuing to assume that the five-fingered knuckle-sandwich of testing, standards, charters, vouchers, and merit pay will do the trick? Or will we continue with what has seemed such a ham-fisted approach?

Collapse

The point of i3 is to disseminate innovation -- it's not an accountability program. AYP as a gateway requirement might make sense if only AYP-certified districts could create useful innovations. But a district might fail AYP because it misses test administration by one point, while creating a sensationally effective special-ed program that deserves replication. Among charter schools (half of which are LEAs and directly eligible for i3 grants), you can see plenty of single-campus schools that perform well overall but miss AYP due to a single factor. That shouldn't disqualify them from offering what does work to other public schools.

Great topic, Eliza. Thanks for bringing it up. Alas, I'm not as enthusiastic about this as those who've already commented. Not that I'm opposed to revamping AYP. This just seems like an overly convenient, ad hoc way to do it. The NCLB waivers and pilots we've had in the past were widely discussed before they went into effect. And, even done carefully, a carveout for i3 districts risks setting a precedent for making a (further) mess of AYP. Pretty soon, you're rewriting NCLB via appropriations language. Not good. For more of my thoughts on this, click here. Thanks again. / Alexander

The Department of Education should eliminate AYP requirements from this program and proceed as rapidly as possible to revise NCLB to remove all its toxic mandates, including the absurd goal of all children being proficient by 2014. Setting impossible goals does not improve American education.

Diane Ravitch

Looks like we all agree that dropping the AYP requirement would be a good idea. It creates the opportunity for schools and districts that need help to partner with an innovative approach or a scaled solution.

It makes great sense for the Department of Education to ask Congress to modify the requirement that school districts applying for Innovation Fund grants make AYP for two years and such an action would not weaken accountability. Districts that have been identified as in need of improvement obviously face very serious challenges in substantially improving student achievement. For districts to meet the challenges they need a clear vision and sense of purpose, a sound theory of change, creativity in designing a comprehensive and aligned set of actions, and a leadership team that is persistent and focused on implementing new approaches. Troubled school districts are not likely to be able to do this without strong and effective partners—partnerships overtly encouraged by the i3 fund. To exclude from funding our lowest-performing districts when they find such strong partners defeats the purpose of the program, i.e. to provide incentives for these partnerships to form where students are most in need.

Allowing school districts not making AYP to be awarded i3 funds does not remove them from the accountability system and it should not. But to date, state intervention with districts needing improvement has been mostly weak to non-existent. The i3 program will provide a rare opportunity for low-performing districts to engage in promising new ventures with some of the nation’s leading innovators.

Collapse

Developing education innovations that work is difficult and chancy work. The pool of cutting edge ideas is smaller than most think (quick - name three) and very few of those have evidence of effectiveness that would qualify them for moderate or big bucks under the proposed standards for i3. The last thing the competition needs is an artificial barrier to entry based on whether a district has met AYP. Further, it is the districts struggling to meet AYP that are most in need of effective innovations. Congress would be well-advised to eliminate this requirement.

Great! Finally there is recognition that the scorekeeping system built for NCLB Title I had problems. Transparency is a good thing, especially in education. An easily understood system that shows students, parents community members as well as state and federal policy makers how schools and the school district as a whole are doing is essential both to help the system continually improve and to permit easily asking and answering questions about effectiveness and efficiency.

Congress constructed an accountability system that would provide test scores for important sub groups of students and for all students school by school and school district by school district as a means of judging effectiveness. The system assumed that the test scores would provide sufficient, fair and accurate insight into student achievement to judge the effectiveness of educators. The system also assumes that a counting system could be created that deal fairly and accurately with tricky issues such as privacy and important intervening variables such at the languages students spoke, student disabilities and student mobility.

Every new activity encounters problems and challenges that were unforeseen or misunderstood, and then people make misjudgments or had inaccurate or inadequate preconceptions about the work of public educators. For example some measures of achievement were clearly not up to the same standard of validity and reliability, such as other measures of achievement but they were treated as equal by rule makers. Test scores for students who were just learning English or students on the lower end of the spectrum of cognitive of learning disabilities were clearly not as valid or reliable as test scores for students for English was their native language or students who did not have cognitive or learning disabilities. Further complicating judging effectiveness were statistical questions of how many test scores were needed to permit faith that the averages used to make effectiveness judgments, 10. 20, 30 or 67 were sufficiently accurate. And what is sufficiently accurate? Because the rule makers made mistakes or came down on the side of rigidity when flexibility was needed, what looked like accurate information to the public and policy makers looked inaccurate and unfair to school people who were doing the work of teaching or organizing the learning environment. At a macro level state were permitted to cut deals on accountability so there was no common pattern. Some states back loaded achievement targets so much that achievement increases were very unlikely to meet the large increases in annual measurable objectives.

Secretary Duncan has recognized that some school districts may not have gotten out of school improvement, but they might be making progress. Whether progress is sufficient is complicated by back loaded achievement targets making even solid progress seem like failure. Recognizing that the NCLB accountability system needs improvements in accuracy for some student groups and some circumstances permits USED to consider the progress a school district is making when selecting school districts for grants to award innovation. Judging by the Secretary's comments accountability 2.0 will be better than accountability1.0. Lessons learned over 8 years ought to be applied to the 3i grants rather than waiting for accountability 2.0 to emerge from reauthorization. Who knows we may be in for an outbreak of common sense.

Collapse