All Rhode Island students are going to graduate ready for success in college, careers, and life. To make sure that happens, I have identified five priorities, one of which is to develop user-friendly data systems that will drive continuous improvement.

Rhode Island has a wealth of data, but we have to ensure that we are using the data effectively. For example, districts make decisions all the time about how to invest in professional development. They do not always do so based on what the data tell them about the needs of their teachers and students. In part, this is a matter of training – we have to do a better job preparing school leaders to use data to drive improvement. In part, it’s a matter of policy-development. For example, at the state level, we should not approve expenditure plans unless they are backed up by supporting data.

Rhode Island has a sound data warehouse; however, we want to improve its usability. We need to be able to provide relevant data to teachers at the classroom level. Teachers need timely, accurate, complete data on student performance so that they can assess their own teaching practices as well as the needs of individual students. If teachers have good data and they are well prepared to use the data, they will improve student achievement.

In addition, we want to integrate data from early-childhood education, afterschool programs, other state agencies, and nonprofit education organizations.

As I noted in the Education Blog last month, Rhode Island is not yet able to automatically measure how many of the students from our PK-12 system graduate from college. Fortunately, we will soon be able to connect our PK-12 data not only to our higher-education data systems but even to databases on the Rhode Island workforce. That will be a huge step toward effective use of data to drive continuous improvement!

Because my mission is to prepare students to succeed in college and careers, I certainly need to know whether they’re doing so. It is essential that we know how our students have succeeded in college and beyond. We also need to know why they succeed or not. What high schools did they attend? What course of study did they pursue? What qualities and characteristics resulted in success or failure? This kind of data, when put to use, will help us prepare all students for success.

Of course, we must all be aware of the need for privacy and confidentiality whenever we collect data at the individual student level. This issue becomes more complex when we collect data from outside the academic world, such as data on financial status, employment status, and earnings. We have to ensure that all data systems, no matter how extensive, include privacy protections.

With those protections in place, there is no doubt that more extensive and comprehensive data about postsecondary success will help us improve our education system across the Pk-16 spectrum.

Collapse

And then there were eight.

This blog is beginning to feel like an Agatha Christie novel. We’re at our lowest point of participation yet. Perhaps this week’s topic was a little dry. Perhaps, no matter what side one is on, we all recognize the inevitability of large-scale data acquisition in education, and therefore don’t feel that much discussion is needed. Either way, I'm confient we'll all come roaring back to our usual numbers in the weeks to come.

First off, Kudos to Mr. Carey for what was probably the most articulate and thorough post of the week. He obviously knows this issue well. And with Mr. Kress’s endorsement, he easily moves to the head of the class.

I did, however, note some comment-worthy comments in Mr. Huggins’ post.

“We will never be able to consistently improve student achievement and maximize our use of time and money in schools if we don’t have reliable data to inform our decisions.”

The phrase “reliable data” is the key piece for me. When NCLB gives states the power to change their tests at any time, we end up with data that can only be used reliably if the changes at one point can be correlated accurately with the changes at another. To my knowledge, no state or third party has ever done this. And, in any case, some types of changes states make are virtually impossible to correlate.

In my current home state of North Carolina, we’re on version three of our tests. At first they were easy, then they got a little harder. Now, with the recent addition of a retesting option, they appear to have gotten slightly easier again. But it’s impossible to tell for sure. Our test data has within it serious discontinuities which make longitudinal interpretations unsound at best and irresponsible at worst.

Even if we enter all test data with total accuracy, and put this data into the most elegantly accessible database, we won’t have ANY reliable test data upon which to draw conclusions. At best, short-term guesses could be made based on certain periods of time when we suspected the testing system was relatively stable; at worst, horrendous errors could be created by policy makers who were unaware of the changes that have occurred over many years. And, by the way, state education folks are not exactly forthcoming about the changes they make. I know; I tried for months to pry this last set of changes out of them to no avail.

Again, from Mr. Huggins’ post:

“Some of the most sought-after NCLB reforms, such as the ability to measure student growth or teacher effectiveness, are impossible without sophisticated data systems to organize and sort information.”

They’re also impossible without uniformity in the types of tests given, the scoring methods used, the cut scores set for passing, and a whole host of other variables that are not easy to account for.

Again, we have what could only be described euphemistically as 50 highly fluid testing systems. In my opinion, these systems are unlikely ever to settle down because there’s just too much political advantage in changing them all the time. And, what’s probably worse, states will never fully disclose the effects of the changes.

Finally, having just read the Commission on NCLB’s recommendations for high quality data systems, I see no recommendations regarding the quality and reliability of the data states gather. Perhaps this will be addressed during reauthorization but I have the feeling that every state will want to preserve it’s right to monkey with it’s own data in any way and at any time it pleases.

It seems to me like we’ve got a cart-before-the-horse issue brewing here. On the one hand, we all see the value in big, flexible data systems that track educational data on a longitudinal basis. However, we have no standards whatsoever that would encourage states to put reliable data into these systems.

If NCLB has proven one thing it is that Campbell’s Law applies to the creation of educational data: “The more any quantitative social indicator is used for social decision making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

With 50 state longitudinal data systems in various stages of being created, we run a grave risk, I think, of making a “Campbell’s Soup” (pun intended) out of our quest for the Holy Grail of school statistics.

What concerns me most is not the possibility of mismanagement or outright corruption. These things exist as potentialities in any venture. But I am surprised that no one seems to be thinking about wayward states, Campbell’s Law, and the forthcoming Campbell’s Soup problem. If we don’t find solid ways of dealing with these things, we could end up making poor policy decisions, based on bad data, for generations to come.

 

Collapse

We will never be able to consistently improve student achievement and maximize our use of time and money in schools if we don’t have reliable data to inform our decisions. Some of the most sought-after NCLB reforms, such as the ability to measure student growth or teacher effectiveness, are impossible without sophisticated data systems to organize and sort information. That is why the Commission recommended that all states develop high-quality longitudinal data systems with increased federal funding support. These systems should be crafted to illuminate student and teacher performance and inform key policy questions, but also to maintain privacy rights guaranteed by law. As other commenters have noted, the experience of states like Florida shows that this can be done.

The Commission also recommended that states provide professional development for all individuals who use and maintain these data systems. This is critically important not only to ensure that data in the system is accurate, used properly, and actually informs decisions by educators and policymakers, but also to ensure that anyone who has access to the system is trained to protect the privacy of student and teacher information. 

Since the release of our recommendations, states have made great progress in developing data systems that inform reform efforts, and federal funding for building data systems has also increased, most recently through stimulus dollars. However, as the Data Quality Campaign has noted, there remains much work to do in refining, linking, and better using these data systems, including making user-friendly data more accessible to parents, educators, and policymakers. The stakes are too high not to do this work well.

Collapse

Like many public policy questions, this is not a question of good versus bad, but a question of two goods, both with downsides.

Collecting longitudinal data on students from kindergarten to higher education to employment represents the Holy Grail for policy researchers. It would provide them with the type of data that would allow for a more complete and accurate understanding of student enrollment, persistence, and completion. State policy makers would have access to the type of data necessary to make well informed policy decisions on educational issues. It seems like a no-brainer.

However, such a databases raises three important concerns that can’t be brushed aside or soft-pedaled. They must be addressed before this effort moves forward.

First, the costs and complexity involved in creating statewide longitudinal databases that link K-12, postsecondary education, and work force employment will be considerable, both for the states and for institutions already laboring under the burden of excessive reporting requirements. The complexity alone can sink such an effort—Real ID, a plan to link driver’s license databases nationwide, was in the works for years and never got off the launch pad.

Second, incentives in the legislation for states to link their databases with other states’ will create a national database. It may not be a federal database, but it will be national. And once it exists, it will be used in ways that we can’t even imagine today, for purposes that are no longer limited to research and public policy. 

Third, many believe that FERPA and other privacy protections currently in place will be sufficient to protect students, but in this day and age, when stories about compromised bank information, electronic passwords and Social Security numbers are commonplace, it’s naive to believe that these databases will be inviolable. Remember, this is 50 separate databases.  Someone will lose a laptop or a thumb drive. And if we achieve the goal of database interoperability that means that it won’t just be one state’s data that are compromised—it will be the entire national database.

The federal government has already authorized such databases in two previous pieces of legislation, the America Competes Act and the American Recovery and Reinvestment Act, and approximately $500 million has been set aside to fund them. It’s reasonable to ask if we need a third statutory authorization to accomplish what Congress has asked for, especially if the third authorization is more complex than the preceding two.

Collapse

To increase higher education attainment to the national goal of 60 percent, educators and policymakers need to know what’s happening to every student at each stage along the educational journey. They must be able to pinpoint trouble spots, isolate gaps in learning and achievement and identify specific populations of students who need extra help. They also need high-quality data that can follow highly mobile student populations from P-12 schools through both community colleges and four-year institutions into the workforce. They need information provided by linked data systems to determine where and why students drop out and to determine how they move (or fail to move) into the workforce.

All states should create longitudinal data systems to link K-12 data with early learning, postsecondary education, and workforce data. State policy should ensure that data can be accessed, analyzed, and communicated to all stakeholders to promote continuous improvement and build the capacity to use longitudinal data for effective decision-making. State data systems should be linked to permit the analysis of student movement across states and to permit better analysis of the costs and benefits, including college completion and student learning outcomes, of higher education. 

Linking education and labor data at both the federal and state levels would offer extraordinary benefits for strengthening not just workforce development programs, but also the regular academic curricula of colleges and universities. Nationally, the Data Quality Campaign (DQC) has done important work in focusing state policymakers’ attention on the importance of building robust and coordinated data systems. As part of that effort, DCQ has identified ten elements that are essential in such systems.

Still, data systems alone are not enough. It is essential that data be used to spur improvements in performance throughout the educational system. The community colleges involved in Lumina’s Achieving the Dream initiative, for example, are using data daily to learn how to make their programs more effective for the students they serve. Incorporating data-driven decision-making into the fabric of higher education is the key to improving performance and better results.

The issue is not whether or not states and the federal government should collect data on higher education. We are already drowning in data on higher education. The problem is that very little of this data relates to the results of higher education for students. Because of inadequate data, we know very little about how students move through the educational system, which schools and interventions are most effective in helping K-12 students to prepare for success in college, and how well prepared graduates are for employment and further education.  

Collapse

I hope Kevin is ok with what I'm about to do. Plus it's truly a rare thing for me to have nothing new to add to a conversation. But I want to align myself entirely with the views of Kevin Carey!

Before getting into education, I was a high tech entrepreneur. Among other things, I worked on a bunch of database projects. The first thing we always asked clients was “What questions do you want to answer with this collection of data?” The reason we asked it was to make the creation of the database, and the client’s data-gathering process, as streamlined as possible.

Peter Ewell and Hans L’Orange take a shot at this in their paper, “The Ideal State Postsecondary Data System: 15 Essential Characteristics and Required Functionality.” While I’m not sure this overview gets the job done, I’m glad that at least a couple of people are thinking about how these vast data repositories might be constructed. But I’m concerned that simultaneously funding large technology projects across 50 states without a more specific sense of why we’re doing it, can only lead to the creation of a high tech Tower of Babel.

The idea, as I understand it having read the article Ms. Krigman linked us to, is that via threat of losing out on generous federal funding, states will create their own post-secondary data systems, and that these 50 state data systems will somehow be merged into a national system at some point in the future.

The first challenge I see is that we have no adopted standards across the states for what to gather and how to store it. Last time I worked in tech, we based our stuff on open source standards like the Dublin Core Metadata Initiative. Without standards of this kind, the hope of a nifty national database is a policymaker’s pipe dream.

The second challenge is privacy (which apparently is not a problem because anyone who thinks it is  has no sway in the discussion). While people like Mr. Nassirian rightly and eloquently bring up privacy issues, I have no doubt that the federal government and complicit states can extort all the data they want from college kids and their families simply by withholding financial aid or even admission. As thoroughly distasteful as this process may be, Mr. Nassirian points out in the article, “Aggressive Plan for State Data Systems” that he has “long since given up hope that the headlong rush toward state data systems can be stopped or even slowed.” Mr. Carey mentions FERPA as a good piece of protection. And Mr. Vander Ark uses Florida as a positive example. So I guess we’re OK on the privacy front. As yet, I don’t think any college kids have weighed in, but if the popularity of Facebook quizzes is any indication, most students don’t really care about privacy either.

The third challenge is the existence of private institutions who will no doubt be loathe to comply. Unless the government can somehow twist their arms, too, this will leave holes in the data that could turn tight conclusions into sloppy conjecture. This leads me to where the entire Data-Driven Decision-Making Movement in education has tended to go wrong in the past: we often make decisions based on incomplete or invald data.

Take, for example, the amount of corrupted test data from high stakes state tests. Virtually every state has rendered its data collection efforts utterly useless by mucking around with changes to cut scores, scoring methods, test forms, minimum sub-group sizes, confidence intervals, and the like. No wonder so many of us tend to prefer the NAEP, SAT, and ACT when we want to draw conclusions about student performance. But, alas, even these sources are subject to certain difficulties.

Finally, what if a state or two (or more) decides to sit this one out? I’ve heard rumblings that Utah might try to pull out of NCLB. Why wouldn’t other scarcely populated states just say no, punching more holes in the data and adding to the head-scratching for data jockeys and policymakers.

So, will a national database of post-secondary data be useful? That depends, of course, on the quality of the data gathered and how it is used. My hunch is that the education policy industry will reap the early rewards. After that, who knows? Further carrot-and-stick strategies via the federal government may be required to get institutions to implement the policies the data helps us support.

The long term issues we face in this endeavor are similar to what comes up during a census. Some people are happy with statistical sampling; others want every person physically accounted for. In our case, MBAs and policymakers want as much data as they can get, as fast as they can get it; educators would prefer to restrict the flow of information to whatever degree possible (futile though it appears to be). The level of squabbling is, of course, directly proportional to the consequences of numbers tipping one way or another.

Let’s consider something else as well: Don’t institutions of higher learning keep this kind of data already? If they do, we might do well to learn something from them about what they gather and how they use it. If they don’t, or if they don’t act effectively to apply the data they gather to the creation of more enlightened policies and practices, we should probably take a moment find out about that, too. Rushing off willy-nilly to gather our rosebuds while we may seems imprudent and unnecessary. As we’ve seen with NCLB, the data juggernaut can be a powerful weapon for scuttling ships that might otherwise be rising with the tide. And the fear of such an occurrence, even when it is remote, doesn’t always catalyze sound decision-making.

Like many people these days, I count myself among the growing list of Americans who are rather unhappy about the state of data in education and how people use it to confuse the general public and advance their careers and ideologies. The point of using data was, I thought, to help us reason effectively, free from the taint of ideology. But this has rarely been the case during the last fifteen years or so. As Mr. Nassirian notes, it is foolish to oppose the drive for data. And no one likes to look foolish. So I say let’s go out there and get it all. But let’s also raise our collective ethical standards for the data we get and what we decide to do with it.

 

Collapse

The terminal outcomes of K-12 aren't only test scores and graduation rates, they should include college completion and employment.  We can't really measure K-12 effectiveness without the ability to know where high school leavers/graduates go.  

Our college dropout rate is even worse than our high school drop out rate.  We won't be able to help students make better post secondary choices or hold higher education accountable for bad completion rates without the data.  

Florida's experience seems to indicate that we can add this important information without comprimising privacy.

It's a very good idea. There's no way to craft smart, fair policy without good information, and there are currently huge holes in our knowledge about K-12 schools, colleges and universities, and the transitions students make between them. Current graduation rate measures, for example, are limited because we can't give colleges credit for students who start at one college and then transfer (as many do) and finish at another. Data systems would also allow for much more robust and multi-dimensional measures of high school success. Everyone seems to agree we should be preparing high schooler graduates to succeed in college and the workplace. So let's see if high school graduates actually do succeed in college and the workforce! Only data systems can provide this information in a reliable, cost-effective way.

Privacy is very important, but I don't believe it should be seen as a significant risk here. First, there are already strong federal privacy laws on the books: the Family Educational Rights and Privacy Act (FERPA). It's a felony to release individually identifiable education records, which is one of the reasons this hardly ever happens. Second, these data systems already exist in a number of states, and to date there have been few if any problems. See, for example, this report (http://www.fldoe.org/fetpip/pdf/0506pdf/hg0506ads.pdf) generated  by the Florida data system, which details, for every high school in the state, the percentage of graduates going on to higher education (broken down by sector), employment rates, earnings, and more. Florida has produced these reports for a number of years without a hitch.

Meanwhile, in addition to state data systems, the vast majority of colleges and universities are already submitting individual student records to the National Student Clearinghouse (http://www.studentclearinghouse.org/), a non-profit organization founded by the student loan industry. The Clearinghouse has been in operation for more than a decade with no instances of student privacy violation. In each case the combination of sensible oversight and strong existing privacy laws has protected individual students while providing a great deal of valuable and necessary information. 

Collapse

Barmak Nassirian, associate executive director of external relations, the American Association of Collegiate Registrars and Admission Officers, submitted the following:

 Ideally, public policy would be developed on the basis of robust data. In practice, data are more often only an afterthought used to rationalize positions that have been arrived at through other means. In reality, most of our political discourse is eerily independent of facts, having more to do with a priori ideology and judgments about the expedient.

Indeed, the trajectory of the “State Longitudinal Data Systems” that are now included in the House and Senate versions of student loan reform legislation stands as a textbook example of selective use of facts to justify a prior conclusion. Several hundred million federal dollars have already poured into state projects that would, if completed, enable state and federal policymakers (and possibly researchers and unknown others who might be granted access to the data) to track individuals from pre-birth through elementary, secondary, and postsecondary educational institutions and into the job market and beyond. As the mantra for the creation and funding of these systems has been taken up by various voices, Congress has not so much as held a single hearing to examine whether the creation of these systems is wise or, for that matter, feasible. The process by which its advocates—institutions cynically advocating unrealistically grand data systems to deflect more modest accountability measures, state officials clamoring for more power over institutions, researchers and foundations seeking more data for the sake of data, and information technology companies hoping to cash in on contracts—have lobbied for the idea illustrates the folly and the futility of funding and building such systems if data-driven public policy is supposed to be their most compelling justification. Lost in the cacophony are any concerns about individual privacy and likely future uses of such an unprecedented repository of very personal information.

 The proposed systems violate some of the most basic principles of fair information practices. First, the very existence of this treasure trove of information about individuals would be kept secret from them, thus violating the basic principle of notice and awareness that private information about them is being compiled. Second, the data system would obtain the data from third-parties who collected it for other stated purpose, thus violating the principle that personal data collected for one purpose should not be used for other purposes without informed consent. Third, there would be no right of access review and correction to individuals’ own records, to at least ensure that the information being involuntary maintained about people is correct. This is a particularly significant issue because of the broad scope of data collection envisioned in the two bills. These data systems would pull record fragments from multiple unrelated data sources and will, based on our experience with other similar large data systems, certainly have very high error rates. Fourth, terms of access and disclosure to the systems are as fuzzy and vague as the basic reason as to why they are being built. There is no clarity as to exactly who will have access precisely because we are rushing to build a tool before we know what we intend to do with it. Add to these foundational problems the absence of a grievance or redress process and what we have a credible example of how Big Brother could come to be.

And if all of the above isn’t enough for perfectly rational citizens to consider moving off the grid, consider what future Congresses would do with a database that contains individual-level health, education, employment, social welfare, juvenile justice, and criminal data about all or a vast majority of citizens. Would such a database remain, like census data, strictly statistical? Or would the temptation to use it for other federal or state purposes prove irresistible? Would Homeland Security be denied the right to search for terrorists in these databases? Could we keep the IRS, the FBI, or the DoD out of these systems? 

 There are better ways of collecting information to inform public policy. Tracking representative sample populations through multiple longitudinal studies—long a staple of social science research—would provide as much, if not more, useful information about the issues that putatively justify the gigantic state tracking systems promoted by boosters of state data systems. And they would do so at far lower costs without violating privacy rights of individuals.

Collapse