Saturday
November 22, 2008
Advertisement
News, Views and Careers for All of Higher Education
Feb. 19, 2007
Huge IPEDS Lives
The U.S. Education Department is quietly moving ahead with plans to significantly expand the information and data it collects from colleges each year through an online survey — including an entirely new section that would require institutions to report on the accountability measures they use and their scores on those tests or tools.
Related stories
- Getting the Government’s Attention (in a Good Way), Oct. 19, 2007
- College Accountability Movement Moves Online, Sept. 17, 2007
- AAU Gets on Board, May 7, 2007
- A Warning on Measuring Learning Outcomes, Jan. 26, 2007
- Virtues and Vices of ‘Value Added’, Aug. 10, 2006
The proposal appears to be another prong in the department’s multi-faceted campaign to carry out the recommendations of Education Secretary Margaret Spellings’ Commission on the Future of Higher Education. By proposing this expansion of what it collects through the Integrated Postsecondary Education Data System (IPEDS), the department could go a long way (without potentially controversial legislation or regulatory changes) toward achieving its goal of establishing a federal system for reporting student learning outcomes and other information on colleges’ performance, as called for in the final report of Spellings Commission. The department’s announcement says that most of the new information it is seeking to collect would be added to the department’s existing Web site for college information, the College Opportunities Online Locator.
Inside Higher Ed reported in December that the department was contemplating such an expansion, which its officials had internally dubbed “Huge IPEDS,” as an alternative to the more controversial federal “unit records” system. (IPEDS is the federal government’s primary database for information about colleges, their staffs and their students, although it doesn’t collect information about individual students, like the unit records system the department also coverts.) In recent weeks, department officials had seemed to back away from the idea, telling a meeting of college association leaders as recently as this month that no such expansion of IPEDS was planned soon.
But in a January 24 announcement in the Federal Register, the department seemed to be laying the groundwork for just that. The announcement invited comments on its annual proposal to revise what it collects from colleges through the postsecondary database system. In a document explaining its request, which must be approved by the federal Office of Management and Budget, the department said that most of the changes were based on recommendations made by an advisory panel of users of the IPEDS database.
The document acknowledged, though, that some of the new information requests were driven by the report of the Spellings Commission, which called for the department to “collect data and provide information in a common format so that interested parties can create a searchable, consumer-friendly database that provides access to institutional performance and aggregate student outcomes in a secure and flexible format.” (The Spellings report added: “The strategy for the collection and use of data should be designed to recognize the complexity of higher education, have the capacity to accommodate diverse consumer preferences through standard and customizable searches, and make it easy to obtain comparative information including cost, price, admissions data, college completion rates and, eventually, learning outcomes.")
Some of the department’s requests, even if they flow from the Spellings Commission report, are unlikely to be particularly controversial. The data collection plan asks institutions, for instance, if they post their transfer of credit policies online, and to provide a link. It also asks them to report how many full- and part-time students are enrolled exclusively in online programs.
The two most significant categories of new information that the department is requesting (which, if approved by OMB, would be voluntary in 2007-8 and required in 2008-9) would be what the department calls “a new accountability part” and an expanded section of information about financial aid, which seems to be designed to help the department come up with a method of reporting on the “net price” that different categories of students might really pay (as opposed to the “sticker price” that gets widely reported) to attend a particular college.
Under the “new accountability part,” colleges would be asked a set of four questions. Some are straightforward; the department asks if institutions have online “fact books” and if they post information on their Web sites about assessment or student learning outcomes, and requests links to those pages, which the department says it would add to the Web-based College Opportunities Online Locator.
But the department also asks whether colleges use specific student learning assessments, such as the National Survey of Student Engagement, Community College Survey of Student Engagement, Collegiate Learning Assessment, and National Measure of Academic Proficiency and Progress, and to specify which other assessment tools they use. Colleges would also be asked to say if the institution makes its results on these measures available online on its own Web site, and to provide the appropriate Web address, which would also be added to the COOL Web site.
The department’s plan would also ask (and by 2008-9 require) colleges to provide, in matrix form, data on all accountability measures they use and “the institution’s score” on those measures. (The document does not make clear whether this information would be shared with the public, but if it would, colleges that now use these surveys and tests for internal purposes only would presumably be forced to reveal them.) The department’s request that a college report a single score for the institution is likely to renew concerns higher education leaders have expressed that the Spellings Commission’s push for accountability is overly simplistic, since most accountability measures that institutions use can’t be summed up in one “score.”
George D. Kuh, a professor of higher education at Indiana University and the director of the National Survey of Student Engagement, said the idea of collecting information about which accountability measures colleges were using and posting links on the department’s Web site to their results made good sense.
But trying to collect information about colleges’ scores on various accountability measures through one or two cells in a spreadsheet is “singularly problematic,” Kuh said. The fewest number of scores an institution could report to even begin to make its NSSE results meaningful, he said, is 10 — scores for both first-year students and seniors on each of the survey’s five main measures. But even that, he said, would fail to tell any interested parent or student how that student might fare at that institution, because it wouldn’t take into account his or her gender or race, whether he or she started at the institution or transferred in, etc.
“That’s where it becomes problematic and potentially very misleading to the public,” Kuh said. “It may not be very well thought through at this point.”
Clifford Adelman, a longtime Education Department researcher who is now a senior associate at the Institute for Higher Education Policy, echoed Kuh’s comments, describing the accountability grid as taking an “incredibly dangerous, reductionistic” approach. “It implies that there is a finite, standardized set of acceptable measures on which institutions can receive ’scores,’ and that these ’scores’ (whatever that means) can be annualized,” Adelman said.
More Information on Financial Aid
The other major category of expanded information that the department proposes collecting relates to financial aid. The proposal calls for colleges to report significantly more information about the kind of financial aid that their students receive (from the current four categories to seven (Pell Grants, Academic Competitiveness Grants, other federal grants, state grants, institutional grants, federal subsidized loans, federal unsubsidized loans, and other loans).
Then colleges would then be asked to fill out a 9-cell grid in which they would break those aid recipients down by their dependency status (dependent, independent with dependents, independent without dependents), their living arrangements (on campus, off campus without family, off campus with family), and, at public institutions, whether they attend in-district, in-state, and out-of-state.
The additional financial aid information that the department is seeking to collect would appear to further its goal of reporting information related to colleges’ net price rather than their sticker price. Collecting information about financial aid recipients based on whether they live on campus or off, etc., would go part of the way toward allowing the department to report how much different types of students really pay for their college educations on average.
But college financial aid administrators warn that while the proposed expansion of IPEDS would provide better information than the department now has at its disposal, the information the department would be able to provide would be too generalized to significantly help individual students. “I’d be concerned about how accurate it would be, if families depend on it too much,” said Pam Fowler, director of financial aid at the University of Michigan.
Fowler also noted that some of the information the department seeks to collect in this part of the new collection would be extremely difficult and time consuming to track down. For instance, her office does not even collect data on students’ living situations, which would only be available from the Free Application for Federal Student Assistance.
A department spokeswoman, Samara Yudof, said that some of the changes the department is seeking in its IPEDS proposal, including some related to the race and ethnicity of students, have been mandated by federal law. “Others are merely under consideration and reflect an interest by [the National Center for Education Statistics] in gathering additional information,” she said. “The department will carefully consider any comments received during the public comment period before making decisions about changes in IPEDS.”
Comments are due by March 26.
Want it on paper? Print this page.
Know someone who’d be interested? Forward this story.
Want to stay informed? Sign up for free daily news e-mail.
Advertisement
Comments
IPEDS is too important a data collection to contaminate with what this Department is trying to do with accountability measures. What you read here is a tragic testimony to ignorance. The first item on the proposed list accountability measures is enough to make you howl: NSSE and CCSSE are NOT “assessments.” They have nothing whatsoever to do with student learning. They are good institutional process feedback surveys. If I were back in a role of a dean, I would use them, but not as accountability measures. Then, the fact that the ED drafters didn’t even cite the ETS Major Field exams or licensure exams in the list of assessments indicates that the Department has no interest in content knowledge as an accountability measure (and, even then, the Major Field tests and licensure exams cover barely 10 percent of all majors in which degrees are awarded). This gap reflects one of the Spellings Commission’s blind sides. In every society and economy on the face of the globe, the primary purpose of institutions of higher education is the distribution of knowledge. You would never know that the Commissioners or the ED folks who wrote this bullet ever had kids who went to college—-or maybe they just never listened to their kids. Because what my kids and theirs talked about learning at the family dinner table includes ferro-liquids, people you never heard of in the French Enlightenment, migration theory in anthropology, the effects of international trade agreements in business cycles, etc., i.e. real stuff. That’s what they are proud of; that’s what colleges exist to pass on. They don’t talk about “critical thinking” or “problem solving” or other generalized cognitive operations that are cited in the Commission’s scatter-shot approach to assessment—and reflected in the friends-of-the-court list of assessments cited here. Are CLA and MAPP bad tests? No. In fact, they are pretty good for what they do, but what they do reflects a fraction of why your kids go to school.
Then we are presented with a matrix of reporting putative outcomes, something that looks like it was designed by a third-grader with dreams of becoming an autocrat: it is an incredibly dangerous, reductionistic—-and stupid—-item. It implies that there is a finite, standardized set of acceptable measures on which institutions can receive “scores,” and that these “scores” (whatever that means) can be annualized. If my college gives comprehensive examinations in every one of its 93 majors as its summative assessments—-which is a legitimate and powerful way of figuring out what my students have learned, and, by analysis of the components of student performance, a guideline for improvement in instruction—-the U.S. Department of Education will not recognize it.
The Spellings Commission undermined itself by putting all its half-literate drafts with missing or erroneous citations on the Web. In the same way, the accountability section of this proposed IPEDS overhaul damages the rest of the proposed changes—-many of which are pretty good and needed.
Clifford Adelman, Senior Associate at Institute for Higher Education Policy, at 9:30 am EST on February 19, 2007
Cliff Adelman writes that “NSSE and CCSSE are NOT ‘assessments.’ They have nothing whatsoever to do with student learning. They are good institutional process feedback surveys.”
Here’s a sample of actual questions from NSSE, which anyone can view at http://nsse.iub.edu/index.cfm :
“In your experience at your institution during the current school year, about how often have you done each of the following?
a. Asked questions in class or contributed to class discussions
d. Worked on a paper or project that required intergrating ideas or information from various sources
During the current school year, how much has your coursework emphasized the following mental activities?
a. Memorizing facts, ideas, or methods from your courses and readings so you can repeat them in pretty much the same form
c. Synthesizing and organizing ideas, information, or experiences into new, more complex interpreations and relationships
During the current school year, about how much reading and writing have you done?
d. Number of written papers or reports between 5 and 19 pages
In a typical week, how many homework problem sets do you complete?
To what extent does your institution emphasize each of the following?
a. Spending signficant amounts of time studying and on academic work
To what extent has your experience at this institution contributed to your knowledge, skills, and personal development in the following areas?
b. Acquiring job or work-related knowledge and skills
f. Analyzing quantitative problems”
Etc., etc. The entire survey is available on-line.
It would be correct to say that these are not direct measures of student learning. They’re measures of educational practice and student engagement. But to say that these questions have “nothing whatsoever to do with student learning” is, obviously, incorrect.
I think everyone would agree that NSSE, like any survey instrument, has limitations. But let’s be accurate when we desribed what it is, and what it isn’t.
Kevin Carey, Research and Policy Manager at Education Sector, at 10:15 am EST on February 19, 2007
FAFSA
All students are not legally required to complete the FAFSA. Only those students seeking Federal Financial Aid. Students who can pay and are not interested in Pell Grants, Stafford Loans, Work Study, etc. are not required to complete this document.
Xerrin, at 10:25 am EST on February 19, 2007
Failures defined
The broader message put forth by both Cliff and Kevin is clear: assessing student learning outcomes is for all intents and purposes impossible. The proposed changes to IPEDS would satisfy some pundits and higher ed researchers in the short run but eventually the findings and interpretations using the new data would face the same scrutiny and riducule that current measures do. More information would be requested and the same absurd cycle would start again.
All of these assessments are “point estimates” in that they measure outcomes at a discreet point in time. When talking to the many self-proclaimed assessment researchers (often with PhDs or years of experience), I take great pleasure in asking them to explain to me simple terms from introductory economics or history courses and then watch them stumble helplessly through the answers. No matter how you dice these exercises the simple fact remains that we have no way of knowing two extremely important facets of student learning, which is why the whole exercise is ridiculously useless: 1) was it the student’s or faculty member’s fault that he did not learn what was taught, and 2) whether what was in fact learned was not flushed away hours, days, weeks or even months later. Without knowing where the blame lies for lack of progress we cannot say whether the interventions necessaary should be on the demand or supply side. Without knowing whether the student spent all night cramming for a test or just simply forgot basic tenets of what was learned over the course of time, we have no idea about the extent to which any policy interventions were actually successful.
Clearly, the good people at Education and the policy pundits amongst us would hope we retain this information over the course of our lives and use it productively. Then again, if a PhD-trained expert on education assessment can’t remember what year the Scopes trial took place or what the elasticity of a demand curve is, how can we possibly accept their assertions that measuring learning outcomes will provide some benefit to anyone else except those who study assessment and publish their findings in peer-reviewed journals?
The Spellings Commission’s (and Education’s in general) push to focus on learning outcomes by penalizing institutions who don’t tow the line coupled with their narrow-minded belief that it can be achieved by more data is naive and terribly bad where policy formulation is concerned. One can only hope that this fad passes into oblivion, like the many others that have emerged over the years, relatively soon.
P. Rof, at 11:50 am EST on February 19, 2007
Responding to Kevin—-he’s wrong!
Kevin Carey is smart guy and a great guy. But on this one, he’s plumb wrong. None of the NSSE questions he cites tell anyone what students have really learned. They tell us what students have done, or how often they do it, or what they think of their academic interactions and institutional environment. All of that is just fine, and it is very helpful for academic administrators to get that kind of feedback. Provided they can link the data to student histories, they can generate some hypotheses for improvement, tentative action plans, etc. These are steps toward accountability, but they are not indicators of student learning.
Clifford Adelman, Senior Associate at Institute for Higher Education Policy, at 3:15 pm EST on February 19, 2007
Ballet is nice...but I like figure skating more.
So I keep hearing that accountability and learning outcomes are far too complex to measure let alone reduce to a small set of variables. Seems quite possible to me considering we’re talking about boiling down 5 or more years of academic pursuits. And aren’t we talking about something quite ellusive? What would we even call it? Intellectual growth or maybe just learning...no, that’s too simple. How about enlightenment! OK, before I get too cynical let me lay get to the point. There _is_ such a thing as accountability and measurement of ellusive outcomes such as learning. Sure it’s hard but not impossible. And its not like there aren’t options. We don’t have to dedicate ourselves to one avenue. We can quantify the type and quality of programmatic attributes that highly correlate with positive academic success. We can create benchmarks by institution and college type/size/focus to determine if it has optimized its deployment of resources and use of funds. We can quantify by student cohort institutional abilities to gradudate students. We can continue to survey student engagement, instructor effectiveness and student employment post-graduation. And, as has been done in K-12, we can begin to assess and _analyze_(read, not survey) instructor success in assisting student academic progression. Standardized testing would be needed but this can and should be done.
Let’s not lose sight that higher education needs renovation. The first steps into this renovation by the federal government may be much less than desirable but the action has just begun and the conversation will likely never end. I suggest we all embrace the change. Students have already been demanding for some time the exact same kind of accountability and substantiation of a learning environment that Spelling and others are now calling for. Remember, we do live in a highly capitalistic culture. The institutions that rise to the market demand will have tremendous success while those who resist it will falter and probably wonder why.
Take care,
Kelly
Kelly Deal, at 4:30 pm EST on February 19, 2007
Kevin’s point is well taken. As of yet, we’ve not established an effective means of evaluating learning. We can offer analysis of all sorts of methods of teaching, but it fails to account for learning.
Certainly higher educaton functions like a business in many ways, but students are not widgets off the line. They’re more like bottle wine or scotch—they ripen well after they finish undergraduate work. I think we are stuck because to really evaluate the learning in college we must evaluate the student’s life after college. How do we evaluate a lifetime and judge whether he or she lived the good life? It’s sysiphean task offering no solid solution.
—As I write the word solution, it implies a problem. I’m not so sure a problem exists in regards to the nature of IPEDS because they don’t really offer anything useful.
In one of my master’s class our professor described this sort of data as a shield to give the public so that the institution may carry on with its work. I only hope that the IPEDS is—as mentiond above—just a fad.
Tucker, at 7:46 pm EST on February 19, 2007
HUGE IPEDS and the FAFSA
Xerrin wrote that not all students are legally required to file the FAFSA. That is certainly true, but my guess is that about 90% of college students do file it because it is required in order to get any federal aid.
My point was that much of the data talked about in HUGE IPEDS is already available by simply aggregating it properly. Adding yet another unnecessary reporting burden on schools is not a very productive notion particularly at a time when costs are rising faster than the general economy.The fact that not everyone files the FAFSA is not an argument for the validity or necessity of HUGE IPEDS. My guess is that a family wealthy enough to have no need to file the FAFSA will also have little need for HUGE IPEDS data anyway.
feudi pandola, at 10:31 am EST on February 20, 2007
Echoing Clifford’s comments, it seems clear that the Dept of Ed folks, with Secretary Spellings front and center, either are not really concerned with learning or they have confused the K-12 testing regime with learning. Accountability seems to be the new buzz word: what is the public getting for its’ money? How many students are graduating in 5 or 6 years? Increasing access to college. What is the preparedness of students for the world environment? These are the points that come out of the Secretary’s public comments. While each of them is important, it is not clear how they relate to the higher part of higher learning. The approach treats colleges as if their missions should all be geared towards workforce preparedness. This approach does not really create a discussion about what is good and not so good about college curricula. Or about what works in teaching and student learning. The Secretary is attempting to federalize the accreditation process, by forcing the accreditors to go along with whatever new rules or interpretations the Department devises on “outcomes.” Mission and the independence of the college are being set aside in the name of accountability. Either the bar will be set in such a way that colleges that cater to students that need more help and time will be hurt, or the bar will be set so low that it will be meaningless, but colleges will stay in good stead by adopting a testing regime and publishing the results on their website. Too bad we are not talking about what the Core should consist of. An opportunity wasted for a political agenda.
Jeff Martineau, Director at American Academy for Liberal Education, at 12:00 pm EST on February 20, 2007
Kelly Deal
Kelly’s point is well taken, but for this observation. The responsibility for developing measures for assessing student achievement should be in the hands of the chartered accrediting associations and their members—not the U.S. Department of Education. The Department has not engaged the higher education community, and is moving to impose its own single-minded standards on a universe of thousands of diverse institutions. The Department wants a uniform system in order to deliver to the public a quantitative score for each institution. To what end does this Procrustean bed serve? Accuulation of statistical data? This is not a function that our Department of Education should be engaged in. It is expensive leading to increases in tuition costs. It is ultimately meaningless when applied to educational institutions as opposed to institutions serving a vocational purpose.
Dick Bishirjian, President at Yorktown University, at 12:30 pm EST on February 20, 2007
INEXPERIENCE SHOWS IN COMMENTS
Once upon a time I was a Federal employee managing a $10 million program. I was enamored (like Kelly and several others) by the elusive goal of accountability. “All educational outcomes are measurable,” I confidently replied to to the critics or skeptics.
We published guidelines, money was spent, reports were filed—and poof!—it is all gone, not even a memory, except for a few old geezers like me who were young pups dduring the Nixon administration.
With the benefit of wisdom and vastly more experience at the institutional level (including collection and interpretation of IPEDS data) I can see that the Federal perspective on accountability was worse than naive or condescending—it was politically dangerous.
The hidden message from inside the Beltway is not merely anti-Democrat, but anti-democratic: viz., If we Feds can find out exactly what you’re doing out there in college-land, we will be able to control you.
Technology be damned, political fads be damned — what happens in college classrooms is vastly more complex and important than can be captured, interpolated or summarized in any survey or questionnaire.
Feed the beast his thin gruel if you must, but don’t waste too much time on Federal accountability. Real learning takes place when students critically absorb the notes you put in the margins of their papers, or discover the joy of reading, or share the reports of their transformational journeys (interior and exterior), or ask intelligent questions during class discussions.
The rest — all of it — is window-dressing.
John R., at 1:41 pm EST on February 20, 2007
Feudi Pandola and Pam Fowler have “hit the nail on the head".
The financial aid data the Department is proposing to collect exist within its own databases. To require colleges to regurgitate Department data back to the Department provides no valuabe services to students,families or the community at-large.
DD, at 9:36 am EST on February 21, 2007
Outcomes assessment is hardly an easy task in something as “multivariate” as higher education. We don’t have a simplifying and unifying Rosetta stone, like profit (if anything profit concerns seem to make higher ed worse). Trying to find out what is learned by millions of students across thousands of schools and hundreds of majors is hard, very hard.
So are we to be forgiven if we fall back on satisfaction or self-assessment surveys and call them outcomes? The DoE seems to be willing to do so. For my part I thought Unit Record reporting was a pretty good idea and was disappointed to see it fall by the wayside. We may not be able to easily measure student learning but we can see how schools are doing at system-wide retention/graduation of the students they choose to accept. That, at least, would have been measurable with Unit Record reporting and could have given students, in my view, something they’d actually find useful when evaluating 2nd, 3rd and 4th tier schools.
Michael Lane, Assoc. Director, at 3:50 pm EST on February 21, 2007
Advertisement
Jobs Related to Huge IPEDS Lives
or search for jobs directly.
Adjunct Instructor, Mass Communication
Lone Star College System
Located just north of Houston, Texas, our five campuses serve 1,400 square miles. Our student enrollment is nearly 50,000 in ... see job
Assistant/Associate Professor/Computer Engineering
Lebanese American University
School of Engineering & Architecture Department: Electrical & Computer Engineering Discipline(s): Computer Engineering ... see job
Adjunct Assistant Professor, Administration of Criminal Justice
Johnson County Community College
A career at Johnson County Community College is more than a job. We believe it’s important to invest in our employees and ... see job
Assistant/Associate Professor of Finance
Kentucky State University
The successful candidate teaching courses in finance at the graduate and undergraduate levels, research and scholarly ... see job
Office Systems Technology Faculty
Broward College
Broward College invites applicants for a full time tenure track teaching position in Office Systems Technology. Candidates ... see job
Assistant Director
Babson College
Assistant Director, Financial Aid — Student Financial Services
Reports to:
Senior Associate ... see job
Clinical Assistant or Associate Professor
East Carolina University
East Carolina University, a constituent institution of the University of North Carolina, is a doctoral institution with an ... see job
Admin & Financial Officer
University of Pennsylvania
The nation’s first university, Penn is a world-renowned leader in education, research, and innovation. Situated on a ... see job
Assistant Professor of Criminal Justice
Fairmont State University/Pierpont Community and Technical College
Fairmont State University and Pierpont Community & Technical College, with a 120-acre main campus in Fairmont, WV, is part of ... see job
CYTOPATHOLOGIST — Clinician-Educator
University of Pennsylvania
The Department of Pathology and Laboratory Medicine at the University of Pennsylvania’s School of Medicine seeks candidates ... see job
Home
| Sign up | Log in
| About us
| Work for Us
| Press Releases
| Rights and permissions
| Contact us
| Advertise
| Partners
| RSS 
| Podcasts
| Links
| Privacy
| My Account
© Copyright 2008 Inside Higher Ed
HUGE IPEDS is a huge waste of time!
From everything I’ve read, the federal government ALREADY has most of the data that this new reporting system will collect at it’s fingertips! For example, HUGE IPEDS proposes to collect financial aid information on grant recipients (Pell, SEOG, SMART, etc) based on a variety of factors that are all already available to the U.S. Department of Education. The FAFSA that all students are legally required to file has all of the information that HUGE IPEDS proposes to collect. All it would take for the Department to aggregate this data is a co-ordinated effort between a few sections within the Departmnent of Education.
To instead, erect HUGE IPEDS, and force the cost of education up for all schools, is dumb, unecessary, and a very expensive waste of time.
feudi pandola, at 8:55 am EST on February 19, 2007