Aug. 18

Let the Assessment PR Wars Begin

Among the many reasons critics have cited for opposing the overuse of nationally comparable measurements of student learning outcomes is that they will become yet another oversimplified way of comparing one college’s performance against another. Just look, they say, at how quick many colleges are to boast about how they fare in the annual U.S. News & World Report rankings, even as many academic leaders bemoan the rankings’ attempt to sum up institutional excellence in a few (flawed, they argue) statistics.

Looking for a job?

See all 209 new postings

Browse all job listings:

Faculty: 3,626
Administrative: 1,892
Executive: 169
FEATURED EMPLOYERS

The University of Nebraska at Omaha issued a news release on Thursday with the provocative headline “UNO First in U.S. for Value-Added Education.” University officials based their assertion on their students’ performance last year on the Collegiate Learning Assessment, a performance-based examination sponsored by the Council for Aid to Education. The annual report that Nebraska-Omaha officials received from the council informed the university that it “contributes more to the learning gains made by students than 100 percent of the 176 four-year undergraduate institutions participating in the 2007–2008 CLA.”

That led to UNO’s boast, in the news release, that it “contributes more to the learning gains made by students than any other institution that participated in a recent national examination, including schools such as Duke University, the University of Texas, the University of Michigan, University of North Carolina and Arizona State University.”

The CLA, which was championed by the Secretary of Education’s Commission on the Future of Higher Education and has been embraced by several major higher education groups as one of a new generation of tools for measuring student learning, is controversial among critics for some of the same reasons that proponents find it attractive. First, it purports to measure certain kinds of learning — general education skills such as critical thinking and problem solving — that many traditional standardized tests do not.

Second, the way it is typically used — administered to small (100-student) groups of freshmen and seniors — aims to help colleges show the extent to which they have helped their students improve over the years, allowing them to show how much “added value” they have delivered over the course of students’ careers at an institution.

Accountability advocates have encouraged the CLA’s use because they hope it will become a nationally accepted way of helping students and families compare the success of different institutions in educating their students. Although the test’s sponsors have argued that the CLA is designed primarily to inform colleges and universities about how they can improve their internal teaching and learning practices, CLA advocates have not aggressively opposed the examination’s use as nationally comparable accountability tool.

But some experts on assessment, and some institutional officials who have used the CLA and similar tools in recent years, have cautioned about the limitations of “value added” measurements, noting among other things that such tests tend to closely track the incoming credentials (scores on SAT scores, etc.) of their students and that colleges may have perverse incentives to have their freshmen score poorly in order to make the performance of their seniors look especially good.

“A value-added score, calculated using the same methodology for all higher education institutions in America, would enable an institution with limited resources that admits students with very poor high school records and very low SAT scores but graduates students who have pretty good GRE scores (as an example of an exit exam) to get a 100 percent score because the improvement or value-added is large,” John V. Lombardi, president of the Louisiana State University system, wrote in a 2006 essay for Inside Higher Ed.

“Colleges with superb facilities and resources that admit students with very high SAT scores and very fine high school preparation and graduate students with very good GRE scores could get a 50 percent score because the improvement measured by the tests would be modest (from terrific coming in to terrific going out). Then, in the national rankings, the first institution could claim to be a much better institution for improvement than the second one.”

Nebraska-Omaha has done just that. The university’s interim assistant vice president for academic affairs, Steve Bullock, said in an interview that the 225 randomly selected freshmen who took the CLA last fall performed less well (finishing in the 9th percentile of all test takers) than they were projected to based on their average composite ACT score of 22. The 98 seniors who took the CLA last spring, by contrast, scored better on the CLA than projected, scoring in the 86th percentile.

As a result, Nebraska-Omaha’s “value added estimate” in its CLA report from the Council for Aid to Education found it to have scored in the 100th percentile among the 176 participating colleges in the difference in performance between its freshmen and seniors.

Bullock said he and his colleagues at UNO know that the CLA results are “not a definitive measurement of how we’re doing, and that we see it as a means to spark further inquiry into what we’re doing.” Officials said they would use the results to explore why the freshmen the university admits appear to be underperforming compared to their academic credentials.

But Nebraska-Omaha officials also decided they could not pass up the opportunity to do a little boasting about scoring so highly on the CLA’s value added scale, Bullock said. “Our administrative leadership felt it was important to get this sort of information out,” he said. “We haven’t done a very good job historically of promoting what we do at the institution, and it’s pretty rare that an institution like ours gets to be No. 1. Of course, the CLA is skewed in favor of schools like ours where students come in and have some room for growth, whereas Harvard or Yale, regardless of what you do, have very good students coming in and they’re going to be smart when they leave.”

Marketing or Misuse?

Nebraska-Omaha’s boasting about its CLA scores did not surprise some college leaders and assessment experts, given the sort of self-promotion that many colleges do about their finish in the U.S. News rankings. Richard Ekman, president of the Council of Independent Colleges, said in an e-mail message that his association’s Collegiate Learning Assessment Consortium has focused “on getting colleges to use the CLA to improve their teaching and learning effectiveness,” and to “trust one another enough to share results and learn from one another about uses of the CLA to diagnose weaknesses and find solutions.”

But “we fully expect a college that happens to score very well to boast about it,” Ekman said. “That’s what UNO is doing, and I don’t fault the university for playing up a distinctive characteristic. UNO may be pushing it a little far — to imply that if you enroll there, you’re guaranteed to learn more than at ABC University — but that’s the admissions game these days.”

Carol Schneider, president of the Association of American Colleges and Universities, said in an e-mail message that her group’s Liberal Education and America’s Promise and Valid Assessment of Learning in Undergraduate Education programs have encouraged colleges to use a broad array of learning measures, including electronic portfolios, to “guide all students, not just a tiny sample, to show that they have achieved the most important outcomes of college.” The CLA, “like any standardized test, can be part of a comprehensive strategy for tracking outcomes, but it is much too narrow and much too generic to become a single arbiter for educational quality,” Schneider said. “I worry that the CLA will become our apple in the garden of assessment — tempting us to settle for simple metrics when we all know very well that learning is multi-faceted and — at the most advanced level — grounded in the students’ actual field of study.”

Although the Council for Aid to Education may have encouraged Nebraska-Omaha’s crowing by informing the university that it finished in the 100th percentile of participating colleges, council’s officials bristled at UNO’s news release.

“We strongly discourage schools from doing things like this,” Roger Benjamin, president of the Council for Aid to Education, said in an e-mail message. “Ranking schools is not our goal,” added Jim Hundley, the council’s executive vice president and chief operating officer. “We’re not trying to identify who’s the best in the nation. What we encourage schools to do is to use their results as a signal and use it with other data, including testing other than the CLA. But how people use the results is not something we have control over.”

Given America’s love for rankings and the thirst for readily comparable (and sometimes oversimplified) measures of quality, however, Nebraska-Omaha’s use of the CLA could be the next battleground in the college rankings competition.

“The danger would come if a federal official (or a USA Today) were to read the release from UNO and get the bright idea to require all colleges to reveal their scores and to create a single ranked list that was presented as if it were the only measure of quality,” said Ekman, of the independent college council. “That’s what’s so troublesome about the U.S. News rankings.

“So, I’d say to UNO: use what you’ve got; but recognize that you may be encouraging Big Brother.”

— Doug Lederman

Want it on paper? Print this page.
Know someone who’d be interested? Forward this story.
Want to stay informed? Sign up for free daily news e-mail.

Comments

Hats Off to Gaming the System?

Is this a suprise to anyone? For quite some time, those who warned about the results of standardized tests being used to rank schools one to another have sometimes been dismissed for being too cynical. But this is the second case I know of (at least widely reported in the press) where an institution has sent out press releases about the its students’ results on a standardized test.

For those concerned with student learning, this kind of “gamesmanship” simply gets in the way; it “politicizes” the student learning process. My institution has faculty, staff, and administration deeply committed to enhancing student learning through the use of rubric grading and other forms of direct assessment, but it appears as though much of these local efforts (of the faculty especially) are being short shrifted simply because the results cannot be compared to other universities and colleges. Yet, we have faculty who have taken local measures to heart because these local measures are more meaningful and useful to them. It is simply unfortunate (yet not unexpected) that “gamesmanship” appears to be developing so quickly.

Sean McKitrick, Assistant Provost for Curriculum, Instruction, & Assessment at Binghamton University (SUNY), at 7:30 am EDT on August 18, 2008

Shame on UNO

Shame on UNO.

Jim, at 7:50 am EDT on August 18, 2008

Ass-ass-ment?? Bah! Humbug!

Never have so many fought for so little that benefits so few! The Bush sold the war is the same way administrations sell assessment. Lies, distortions, made up data, packaging, and humbug!

Diogenes, at 8:15 am EDT on August 18, 2008

Perverse Result

How’s this for ‘perverse’. UNO’s boast works and attracts a stronger applicant pool. Suddenly, their CLA scores show that students are learning less. Unless of course they somehow encourage first year students to tank the exam.

mythbuster, at 9:10 am EDT on August 18, 2008

Not the first

I have in hand a Spring 2008 magazine from Charleston Southern University, the largest item on the cover of which is “CSU: Top 100, Top 10%". The top 10% is results of the CLA. But I can hardly blame them, or UNO, because they are up against a media environment obsessed with rankings. They didn’t make the rules (who did?) The term “league tables” used in Europe for what we call college rankings is revealing: it’s a sports metaphor. Someone has to win and others have to lose. This is just wrong (and Consumer Reports is better than this). I wish our policy makers would take a look at the work of Grady Bogue, who in his 1992 book _The Evidence for Quality_ asked the questions “Is quality in limited supply?” and “Is quality to be expressed in a single performance indicator or evidence, or does it take more than one data point to circumscribe the richness of both individual and organizational performance?”

Grocheio, Asst VP Planning and Institutional Effectiveness at Shorter College, at 9:20 am EDT on August 18, 2008

Pandora’s Box

Well, having already opened Pandora’s box with USNWR, I would submit that the CLA and the accumulation of more such prominent tools in the future is the only way to mitigate misleading marketing. The point at which students and families encounter rankings/assessment feature fatigue is the point at which related marketing efforts will be in vain and institutions can more widely begin to craft and use instruments appropriate only to their internal value-adding processes. When everyone is #1 in something, number ones will matter a great deal less. Perhaps then students and families will favor informed discussion of institutions over conceived formulas and lists. Good luck to CLA, and may the continuation of this conversation result in any number of corollaries and competitors.

Wossamotta U., at 9:25 am EDT on August 18, 2008

Samples and False “Accountability”

These problems will appear anywhere a test like CLA is used to “document” something we call “accountability.”

First (and you can easily obtain the data), UNO has 11,000 undergraduates, 24% of whom are part-time. Only 55% are even counted in its calculation of graduation rates, which tells you that 45% either entered part-time or not in the fall term, hence are “non-traditional” (actually, that is a new tradition, but we’ll let that pass here). The 6-year graduation rate is 41%, with 30% of its students transferring out (the proportion transferring in is another story, though indeterminable from publicly accessible data).

Now, how representative do you think a group of 100 student test-taking volunteers, presumably all graduating seniors, will be of this population, a population very typical of places like UNO? And how much of whatever controlled change in test scores are marked do you think is uniformly attributable only to their experience at UNO (or any other institution)? Your brother-in-law, with two kids, two cars, and two jobs, may be among the volunteers. Your daughter, who attended two other institutions while on her way to a degree, may be among the volunteers. The 100 volunteers may consist entirely of engineering and nursing majors—or journalism and psychology majors. A third of the volunteers could have cooperative education placements or externships in work-based learning settings. Or they could just have jobs that consistently present opportunities for developing the kinds of cognitive operations that the CLA (or other tests) measure.

The point is that the potential mix of backgrounds and learning experiences is much too volatile and complex for any statistical gymnastics to control with samples of volunteers who represent 1 percent of a diverse undergraduate population.

If U.S. higher education really believed that the CLA (or any other test) was a valuable measure of what it expects every student to learn, was, in fact, the ultimate measure of what it does, then it should require all students—and not just unrepresentative samples—to take and pass the exam in question as a condition of graduation. Each school can set its own cut score (there are standard psychometric techniques for doing so), and give students 3 shots to pass.

At that point, the credibility of the enterprise will rise. Until then, what we are doing is just a hollow show. Besides, our students don’t get degrees in “critical thinking” or “making and breaking arguments.” They get degrees in engineering, nursing, anthropology, etc.—and that’s the way we organize our faculties and curriculum as well.

Think about it.

Cliff Adelman, Senior Associate at Institute for Higher Education Policy, at 9:25 am EDT on August 18, 2008

Assessment PR wars begin...

In addition to the much more serious concerns about overall misuse of standardized assessments, especially CLA, the last time I looked, it is impossible to be at the 100th percentile as UNO claims. Nobody can be better than 100% of all schools since UNO’s scores are included. They cannot be better than themselves. I’m glad UNO is proud of their supposed achievement, but somebody better check their statistics!

Stephen J. Langendorfer, Ph.D., Director, general education at Bowling Green State University, at 10:00 am EDT on August 18, 2008

So What’s The Big Deal?

Just last week – and as a preface to The Manley Plan — I wrote ...

“ ... third, is it the student’s ‘absolute knowledge’ at the conclusion of hir undergraduate experience or the so-called knowledge value-added that will be, somehow or other, aggregated to tell us whether we’re getting our money’s worth from our colleges and universities? I’m wagering that we’ll side with the elites in this matter. Poor Shenandoah Valley University argues, ‘Our graduates aren’t wonderfully brilliant upon graduation, but they learned a lot during their undergraduate experience here.’ The elite Dukemouth, however, says, “That’s unfair! Our incoming freshmen are brilliant to begin with, so expecting us to add a great deal of value is unrealistic.’”

http://www.insidehighered.com/views/2008/08/07/fryshman

This hoopla – the sort to which academic administrators feel compelled to resort when they’ve got nothing better to do – tells parents nothing they didn’t already know. If your college-age child is bright, thoughtful, well-informed, creative, energetic, and a high-achiever ... well Yale is the place (just ask Rory Gilmore). But if your youngster is not that bright to begin with (maybe even a little slow) or wasn’t smart enough to use hir high school experience to prepare for the future ... the University of Nebraska at Omaha fits the bill. I thought that was already well-known.

But since everyone seems to be Hell-bent on ranking colleges and universities, I think I’ll recommend that “The National Enquirer” get into the business. Using available information and an Excel spreadsheet, they can employ the “quick and dirty” Manley Index to get a best-of-all-possible-worlds ranking system. For each college or university, let E(ACT) be the average American College Testing score of incoming students, let E(CLA) be the average Collegiate Learning Assessment value-added score of graduates, and let USN&WR be the U.S. News & World Report ranking of the school. The Manley Index is ...

MI = 5*E(ACT) + 2*E(CLA) — 0.0001*(USN&WR).

Obviously bigger is better, so ranking these index numbers will (1) tell every parent in the U.S. where to send hir children and (2) give every college administrator in the land additional free time s/he can devote to hustling endowment funds.

I just love to solve problems!

Frizbane Manley, at 10:15 am EDT on August 18, 2008

Words and Actions

If the CLA doesn’t want to be in the business of ranking colleges then why do they provide the percentiles? Colleges interested in this exam could receive their own scores and use them to benchmark their efforts in improving teaching and learning that the CLA supposedly leads to. If absolutely necessary aggregated benchmarks for other groupings of universities (private, public, open access, highly selective, and so on) could be made available for comparisons to institution’s individual summary score. It doesn’t take the so-called critical thinking skills the CLA is supposed to measure to see the inconsistency between this organization’s words and actions and to see who benefits from it the most.

Assessment Coordinator, at 10:45 am EDT on August 18, 2008

Excellent Start

None of the strategists moving accountability in this direction believe that the CLA (or its next three generations for that matter) provides a comprehensive picture upon which one can make valid inter-institutional comparisons. Neither do they believe that such comparisons are necessarily helpful. What they seek are small points of leverage with which to dismantle the Mandarin controlled, self-serving institutions that refuse to become accountable to their primary stakeholders (students, not the professors).

This is a small move in the right direction. Go for it UNO! Gain as much market share as you can with this small victory. Let the competition begin. The marketplace will sort out the conceptual foundations, metrics, and benchmarks, finally taking it out of the hands of the Mandarins who will never run out of ways to argue against any idea for increased openness and accountability.

No, a percentile rank of 100 is not possible; 99th percentile is the limit.

Senior Professor, at 10:45 am EDT on August 18, 2008

Hurrah for UNO!

Three cheers for UNO! Why cheers for a school that has only done what any sensible institution in a competitive marketplace would do? Because their action has likely inaugurated a great new era in higher education—the era of instruction accountability. Obviously, this era isn’t going to be great for everyone, as the comments on this article by administrators makes clear. Gone are the days when they could sit back and have students do the essential job of instruction assessment for them by filling out simplistic and demonstrably invalid instructor evaluations that have led to out-of-control pandering to the worst students via grade inflation and expectations deflation (see, e.g., Valen Johnson’s “Grade Inflation: A Crisis in Higher Education"). However, the future is obviously sunny for instructors who have actually been trying to educate students and have been kicked in the *** by students and administrators alike for doing so.

Are there problems with the CLA? Of course, but many of the objections to it and similar instruments that commentors to this article have made so far just don’t hold water. Are there problems with its validity or comprehensiveness? Perhaps, but I’m sure that schools can bring enough pressure to bear to either force the CLA’s improvement or band together to create their own. Does the CLA put places like Duke at a disadvantage? Or, do high CLA results one year doom a school to lower results the next, because of the boost to the quality of their incoming class? Perhaps, if only raw, undifferentiated scores are reported. But if those who initially have high CLA scores are smart, they will find a way to report scores that are adjusted for students’ initial preparation levels (using SAT scores and/or highschool rank, etc.).

Similarly, assuming Duke is really as smart as its reputation, it’ll quickly figure out that it can also level the playing field by finding a way to factor out students’ initial preparation levels.

Where will they get the data for this? It shouldn’t be hard to come by; indeed, as indicated, the government is likely to push data reporting soon. Further, the CLA’s makers have every incentive to start reporting an adjusted score to mitigate against the devaluation of its measure, regardless of the attitude that one of its administrators clearly has about all this. And if the CLA’s maker doesn’t figure this out, some other organization will eventually see the opportunity of creating a normed instrument and seize it.

Now that the cat is out of the bag, there is no way to put it back in, regardless of the rather naive comments reported in the article. As everyone in higher ed. should understand but many often fail to, education is a business, and just like any other business, those who successfully compete in it will do so because of they created perceived value. For those school’s who don’t have snob value, which is only of principal importance to our country’s small elite, they will HAVE to compete with each other based on measures of intellectual improvement because that’s what a great many in the *market* will, quite logically, perceive to be the most important criteria for assessing value. And its the market the drives everything in our world. For once, at least from my perspective, that seems to be a good thing.

Again, Kudos to UNO!

ex-prof, at 1:15 pm EDT on August 18, 2008

“It’s the market that drives everything in the world.”

And it’s too bad, too, ex. prof. How about a non-market supply and demand model, based on DE-centralized planning? We would educate for that, of course, which would do interesting things to “assessment,” namely assessing the ability of each and every worker/consumer to participate in economic decisions in proportion to the degree s/he is affected by the overall result. Planning goes through 7 or 8 iterations until a match is made between how much citizens want to produce and how much they want to consume.

The market as such many find corrosive and corrupting. It mis-prices things (like gasoline throughout the 90s) leading to just the problematic nature of education in such a steeply sloping, hierarchic society. I submit that one reason assessment is so vexing, or has become the issue it has, is indeed market-driven, which is sad.

The right-wing notion that the exchange between worker and capitalist is a free exchange, for example, is one of the great howlers in the history of ideas.

A.D., at 2:50 pm EDT on August 18, 2008

We are in danger of replaying the same argument that has unfolded in measuring K12 schools’ contributions to achievement.

What seems to be difficult to grasp — or easy to ignore — in that argument is that schools — and universities — operate on distributions of fairly significant student differences. Changes in measures of central tendency for outcomes that matter to students (as well as society) for different types of students are poor representations of distributional changes. This article touches on this point in contrasting the likely change in central tendency measures for well-resourced schools with top performing students versus less well-resourced schools attempting to attract and educate students who may not be top performing for any number of reasons.

Michael Gerber, Professor & Chair at UCSB, at 3:30 pm EDT on August 18, 2008

PR WARS VS. AFFIRMATIVE ACTION

I find it very interesting that at this site there seems to always be more, and more substantive, reactive responses to how institutions are compared to one another, than how students are compared to one another.

When it comes to what’s called affirmative action,’ most responses at this site rarely question how students are admitted. That is, if they don’t ‘make it in the rankings,’ i.e., SAT/ACT they should simply accept it and their lower academic readiness status (cf. responses to the Colorado vs. Ward Connally article).

But, when it comes to institutional rankings, measures and all their ramifications become the center of many, many responses. The unique differences of institutions become key. The inadequacies of any ranking measures also become central.

It seems to me that for many who respond to articles (I don’t know if the respondants are faculty or students, or both) standardization should be borne by (prospective) students only, but seldom (if ever) by the institutions that accept or reject them. Why is that?

Somehow, we believe that diversity of academic institutions is so valuable that we should use non-standardized methods for accrediting them, if not ranking them (the former being much more important than the latter). But, students (almost always African American, but sometimes females of all ethnicities) should live or die by only rankings and just suck it up.

Why? Because that’s fair? Hmmm!

Dr. Gadsden, Morgan State University, at 9:00 pm EDT on August 18, 2008

thoughts on ass-ass-ment

Forgive me for getting to the party late. For Diogenes and others of similar mind, my thoughts on ass-ass-ment are here: http://howadminworks.wordpress.com.

Anonymous Administrator, at 5:00 am EDT on August 19, 2008

Jobs Related to Let the Assessment PR Wars Begin

or search for jobs directly.

Librarian
American University of Afghanistan

The American University of Afghanistan (AUAF) offers a unique opportunity to teach and work with a dedicated international ... see job

Chair — the College of Liberal Arts
University of Minnesota, Twin Cities

The College of Liberal Arts at the University of Minnesota, Twin Cities, invites applications and nominations for the ... see job

Professor/Dept Head Computer Science and Engineering
Mississippi State University

Professor/Department Head Computer Science and Engineering Mississippi State University see job

Campus Solutions Consulting — PeopleSoft Financials
CedarCrestone

Consulting Opportunities! Take your career to the next level with CedarCrestone. see job

Director of Student Learning Assessment
Ithaca College

Job Description: Ithaca College invites applications for the position of Director of Student Learning ... see job

Director of Standardization, Campus Coordinator
Kaplan Higher Education

Based in the greater Chicago area!! Kaplan Higher Education (see job

Associate Dean
Western Carolina University

Western Carolina University’s Hunter Library invites applications and nominations for the position of Associate Dean of ... see job

Assistant Director, Military Outreach 002653
University of Maryland University College

Are you seeking a challenging and rewarding position? Look no further than UMUC... see job

Assistant Director, Continuing Education
Trinity Washington University

The Assistant Director of Continuing Education is the front line person for all faculty and course management in the ... see job

Associate Academic Officer
University of North Carolina School of the Arts

The University of North Carolina School of the Arts (UNCSA), a constituent campus of the University of North Carolina with ... see job