The University of Calgary and 11 other universities across the country will no longer participate in Maclean’s magazine university rankings.
In a letter to Maclean’s magazine, dated Aug. 14, the 11 universities–including four which placed in the top five of their categories in the 2005 survey–stated their withdrawal from the Maclean’s university rankings issue. The letter cited inaccurate research methods as the primary reason for withdrawal.
“The decision was based on continuing disquiet with the way we are asked to produce data for Maclean’s,” said U of C provost and vice-president academic Dr. Alan Harrison. “We’ve been asked to spend time and money on an exercise that we feel is fundamentally flawed.”
This spring, the U of C also refused to participate in the Maclean’s undergraduate student survey, which hit stands June 19, and asked recent graduates to rate their experiences.
In the second letter–sent to Maclean’s last month–the presidents of the dissenting universities said they would not invest public funds in the survey used for the university rankings, stating the data the magazine uses are “oversimplified and arbitrary.”
The U of C has a long history of poor performances in the Maclean’s rankings. Though last year the U of C’s overall ranking remained unchanged at 14th place, it improved in the areas of student quality and awards, faculty awards and grants, percentage of budget allocated to student scholarships and bursaries and alumni support.
However, Harrison insists the U of C’s rankings in the previous surveys were not the only factor influencing the decision to withdraw.
“We improved, but we withdrew anyway,” said Harrison. “It illustrates the position we took was a principled position. We do have fundamental concerns with what Maclean’s was doing.”
Over the past years several universities have voiced concerns over the way Maclean’s analyses its data.
“We’d been expressing these reservations for some time,” said Harrison. “But the universities didn’t coalesce around a single issue. Quite frankly we weren’t getting our act together.”
Harrison suggested there are several problems with Maclean’s data analysis, including their attempt to combine a variety of different measures together.
“How do you get an average [across institutions] of a scholarship budget or entering grades?” asked Harrison. “It’s like comparing apples and oranges.”
Harrison admitted the U of C also grades students, but uses a fair method of evaluation.
“We do grade you in a fundamentally sound way,” said Harrison. “We grade out of 100, and both 84 per cent and 82 per cent are good grades. We wouldn’t say that 82 per cent came in second. We don’t rank students. It’s not a very realistic way of assessment. Most universities do a very good job on limited resources. Some do better than others, but all do a good job.”
U of C Students’ Union VP academic Shannon O’Connor said the SU is supportive of administration’s decision. While the SU wants students to have as much information as possible, it is inappropriate to rank different universities against each other, she said.
“Ranking the schools, as Mac-lean’s was, didn’t communicate the correct information to prospective students,” said O’Connor. “It would be better if Maclean’s highlighted the pros and cons of each university individually, rather than trying to compare different institutions that can’t be reliably compared.”
Tony Keller, Maclean’s managing editor, special projects, confirmed the 2006 university rankings issue will go ahead, with or without university cooperation.
“They can’t decline to participate because of FOIP [the Freedom of Information and Protection of Privacy Act],” said Keller. “This isn’t a decision for them to make.”
Though the magazine may not be able to force the universities to answer its own questionnaire, third party surveys are available.
This spring, Maclean’s used a FOIP request to access results from the National Survey of Student Engagement, conducted in 2004, and the Canadian Undergraduate Survey Consortium, conducted in 2005, for publication in their June 19 University Student Issue.
Keller defended the Maclean’s ranking system.
“I don’t buy the argument that what we do is unscientific,” said Keller. “Everything we use is objective data. When universities compare themselves to other universities they lay out their measures along almost exactly the same lines as we do. Universities can criticize Maclean’s and some of that may make us better but it won’t shut it down. We’re journalists, we cover higher education and we’ll continue to do our jobs.”
Despite the continuing controversy, there are many schools that continue to participate in the Maclean’s questionnaire, including Acadia University in Nova Scotia.
“It’s our view that the comparisons are inevitable and Maclean’s is one source of information that parents and students use,” said Acadia senior director of communication and public affairs Scott Roberts. “Those folks that say the data in Maclean’s are flawed, that’s a valid point. We have no issue with people who decide not to be part of it.”
Roberts stressed the institutions surveyed have always sought to be measured fairly and accurately, suggesting that concerns over the magazine’s analysis might be eased if the data sets were separated from the rankings.
“Take class sizes or classes taught by tenure track professors–there are a lot of data points presented in the Maclean’s survey that are useful to students and parents,” said Roberts. “The rankings themselves are immaterial. It’s the information that is valuable.”
Roberts does not think third party surveys will take the place of the Maclean’s questionnaire when students try to compare institutions.
“We believe those are extraordinarily valid sources of information for institutions who are trying to improve the way they operate,” said Roberts. “But at the same time, they are not commonly or publicly available.”
In the end, said Roberts, it all boils down to the need for information.
“Somewhere out there someone is going to present data,” he explained. “You can either be part of that, or just know that they’re out there without being able to have a say.”
The universities not participating in the fall 2006 Maclean’s university rankings survey are: Dalhousie University, McMaster University, Simon Fraser University, University of Alberta, University of Calgary, University of British Columbia, University of Lethbridge, University of Manitoba, Universite de Montreal, University of Ottawa, University of Toronto and Queen’s University.