[csaa-forum] removal of journal ranking system

Mark Gibson mark.gibson at monash.edu
Thu Jun 2 09:37:47 CST 2011


Thanks for this Guy. Very informative to have a good succinct summary of the
UK system. You confirm what I thought though: we still don't *really* know
how ERA will drive differential funding here. We are acting on the
expectation that it will do, but without knowing exactly how.

You are right of course about the need for continued lobbying. One thing
that worries me about cultural studies specifically is that we are not
terribly well organised for doing this -- at least not collectively.

On a trivial point, I'm intrigued by the ubiquity of 'stars' in these
research measurement systems: Our journals were ranked A*, A, B ...; in the
UK, units were ranked 5*, 5, 4 ... Why don't they just recalibrate the
rankings to make it A, B, C or 5, 4, 3? Is it just to soften the blow for
those who don't make the top rank? We can feel good about being '5' or 'A',
even if we miss out on the star, whereas being ranked 4 or B on a
recalibrated scale might be humiliating? It adds to the appearance of
silliness, where you think they might want to do everything they can to
avoid that. Why not give those in the top level a smiley stamp?

-- Mark

On 2/06/11 9:32 AM, "Guy Redden" <guy.redden at sydney.edu.au> wrote:

> Hi Mark,
>  
> The funding is indeed an important question. Other research frameworks that
> use rating scales (e.g. RAE in UK and Hong Kong, PBRF in NZ) are used for
> differential funding. I¹d be very surprised if ERA isn¹t. In the UK
> RAE-based funding formulas changed quite often as the government made
> different year-to-year decisions about how to divide up the research pot.
> I¹ve lost touch with RAE 2008 and its funding implications, but up to 2001
> units of assessment were rated on a 7 point scale (1 to 5*). Before 2004
> anything 3a and above received some funding. In 2004 the lowest rating to
> get funding was raised to 4. So from 2004 onward this meant 4-rated units
> got one unit of resource and a 5* almost twice that as below. Actual
> payments to unis were also determined by the research volume and subject
> cost weightings.
>  
> 1 = 0
> 2 = 0
> 3b = 0
> 3a = 0 (³National excellence in two thirds of outputs and international
> excellence in some²)
> 4 = 1
> 5 = 1.5
> 5* = 1.88 
>  
> We should be clear about this: universities where peer review had found
> there to be significant research activity (as per 3a descriptor) in certain
> disciplines have received zero block grant for research under the RAE.
>  
> How the Australian government funds is a matter of speculation (and I¹m
> aware my last email in this thread included speculation and hearsay that may
> not be accurate), but I went to an NTEU meeting recently in which a senior
> union figure reported from a meeting she and others had with Kim Carr. Carr
> was asked how the ratings will be used for funding. His reply as reported by
> her was ŒWhy should the government fund research that is not excellent¹?
> If the UK approach is followed 4 and 5 rated units of evaluation will
> receive the real dollars, 3s a token amount if any. But let¹s hope for a
> less blunt approach in Australia in the name of sustaining a diverse and
> accessible research culture.
>  
> The current Australian HE headline attributes the dropping of the journal
> rankings to us who raised a fuss, so I think we should be on the case about
> possible funding implications too! Otherwise I guess they will at some point
> be presented as a fait accompli.
>  
> Cheers,
> Guy
> 
> 
> 
> On 1/06/11 10:21 PM, "Mark Gibson (Arts)" <mark.gibson at monash.edu> wrote:
> 
>> Yes my mistake David. It's only ERA1 RIP. That was my point too: we
>> need to be asking about ERA2.
>> 
>> On the question of 'financial consequences', this has always seemed to
>> me rather obscure. It has been difficult ever to get a straight answer
>> on precisely how ERA will drive funding. Carr made another statement
>> earlier in the year but the most concrete it got was to say that ERA
>> 'may' drive funding through the Research Training Scheme.
>> 
>> While we have been running around worrying about ERA, we still seem to
>> be living substantially with the old HERDC formula. Is there good
>> intelligence on how (or even whether) this will change?
>> 
>> Mark
>> 
>> On Wednesday, 1 June 2011, David Rowe <D.Rowe at uws.edu.au> wrote:
>>> 
>>> Dear CSAA
>>> 
>>> ERA is not dead. The rigidly constructed journal ranking system has been
>>> discarded, but ERA 2012 will still happen and will affect us all, especially
>>> as it
>>>  will have direct financial consequences. So there¹s still plenty more to be
>>> anxious about, sorry!
>>> 
>>> All Best
>>> 
>>> David
>>> 
>>> David Rowe
>>> Centre for Cultural Research, University of Western Sydney
>>> 
>>> 
>>> 
>>> From: csaa-forum-bounces at lists.cdu.edu.au
>>> [mailto:csaa-forum-bounces at lists.cdu.edu.au]
>>> On Behalf Of Mark Gibson (Arts)
>>> Sent: Wednesday, 1 June 2011 4:00 PM
>>> To: csaa-forum at lists.cdu.edu.au
>>> Subject: Re: [csaa-forum] removal of journal ranking system
>>> 
>>> 
>>> ERA RIP.
>>> 
>>> Like others, though, I¹m uneasy about what the alternative might be. There
>>> was always one thing to be said for ERA: it was an attempt to give
>>> recognition to publication outputs rather than give credit only (or mainly)
>>> according to funding inputs. The upside
>>>  of that was obvious for a field like cultural studies, where some of the
>>> best work has been produced with very little funding support.
>>> 
>>> Graeme provides a useful insight into how some assessment of outputs might
>>> survive without journal rankings (via Jon Stratton¹s last email). If his
>>> description is right, then judgements will still be made about publication
>>> quality. It will just be Œin camera¹
>>>  (and hopefully more nuanced), avoiding the unintended consequences of
>>> public
>>> journal rankings.
>>> 
>>> Let¹s hope this system does work and that those making the judgements on
>>> quality do so wisely. My worry is that it might also prove too hard (or
>>> perhaps just too bloody time-consuming!) and they end up throwing up their
>>> hands and going back to taking grant
>>>  income as a proxy for research quality. It would be so much easier.
>>> 
>>> Is it worse seeing the journal you publish in ranked B or C, or seeing
>>> publication in general rated at only 6% of research performance (as it was
>>> under HERDC)? It would be nice to think that we can do better than both, but
>>> I don¹t think there¹s any certainty.
>>> 
>>> Mark
>>> 
>>> On 1 June 2011 14:50, Jackie Cook <Jackie.Cook at unisa.edu.au> wrote:
>>> 
>>> 
>>> Love Paul¹s qualitative twist! Try this:
>>> 
>>> Q: ³How many high-profile quantitative research exercises on academic
>>> evaluation  have the Unit Evaluators completed and published in
>>>  peer-refereed journals?²
>>> A: ³They¹re  really extraordinarily flexible, a product of sector-leading
>>> evaluative experiencesŠ²
>>> 
>>> Expect to hear something very like that, very soonŠ
>>> 
>>> Jackie Cook
>>> UniSA
>>> 
>>> 
>>> 
>>> From:
>>> csaa-forum-bounces at lists.cdu.edu.au
>>> [mailto:csaa-forum-bounces at lists.cdu.edu.au]
>>> On Behalf Of Paul Magee
>>> Sent: Wednesday, 1 June 2011 2:02 PM
>>> To: csaa-forum at lists.cdu.edu.au
>>> 
>>> 
>>> Subject: Re: [csaa-forum] removal of journal ranking system
>>> 
>>> 
>>> 
>>> 
>>> Colleagues,
>>> 
>>> 
>>> 
>>> 
>>> I¹m still a little confused as to what a Œunit of evaluation¹ is. Is that a
>>> new term for Œacademic¹?
>>> 
>>> 
>>> The way quality is so swiftly defined (Œthe introduction of a journal
>>> quality
>>> profile, showing the most frequently published journals¹) in terms of
>>> quantity also confuses me. Do you think we could do it the other way round,
>>> so that whenever people ask us to
>>>  quantify a phenomenon we respond in terms of its qualities?  Q: How many
>>> plastic spacemen does it come with? A: They¹re really extraordinarily
>>> flexible, a product of sector-leading injection moulding.
>>> 
>>> Best
>>> 
>>> P
>>> 
>>> 
>>> Dr Paul Magee
>>> 
>>> Associate Professor of Poetry
>>> Faculty of Arts and Design
>>> University of Canberra
>>> ACT 2601
>>> 02 6201 2402
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Australian Government Higher Education (CRICOS)
>>> Registered Provider number: #00212K
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On 1/06/11 11:11 AM, "James Arvanitakis" <J.Arvanitakis at uws.edu.au> wrote:
>>> 
>>> 
>>> 
>>> 
>>> Hey everyone
>>> 
>>> I am also pleased with this ­ though always nervous about what is around the
>>> corner
>>> 
>>> The journal ranking system has been devastating to emerging researchers.
>>> 
>>> Friends and colleagues have asked me Œhow to publish in an A* journal¹ ­ and
>>> feel rejected if they fail. I have always responded that good quality
>>> research should be published in the appropriate journal and the rankings are
>>> arbitrary ­ and will change. This
>>>  was confirmed to me by a publisher I met while in Europe who was literally
>>> laughing at the rankings system. He said some of his weakest and moribund
>>> journals had received A rating and had been revived from the decision ­ but
>>> could find no reason why this was
>>>  the case. Meanwhile, excellent journals get a low ranking and people have
>>> chosen to ignore them.
>>> 
>>> 
>>> We then compared an ŒA Journal¹ that averages an average of 3.4 readers to a
>>> newspaper piece based on our research that has 10,000 reads: how should
>>> these
>>> compare?
>>> 
>>> While I have been lucky to be surrounded by excellent mentors and have
>>> supported my decision to ignore rankings, I have seen senior management at
>>> certain universities base their entire decision-making around the ranking
>>> system. Maybe if we are ever faced with
>>>  such a system, we can agree to boycott it.
>>> 
>>> Thanks to those who put up a good fight on this
>>> 
>>> 
>>> James Arvanitakis, PhD
>>> 
>>> Senior Lecturer - School of Humanities and Languages
>>> Head of Program - Dean Scholars
>>> Research member - Centre for Cultural Research
>>> Member of the Ally Program for GLBIT students
>>> Fellow - Centre for Policy Development
>>> 
>>> www.jamesarvanitakis.net
>>> 
<https://email.uws.edu.au/owa/redir.aspx?C=b8a0230cbcb44c06b284e6f5777c2892&>>>
U
>>> RL=http%3a%2f%2fwww.jamesarvanitakis.net>
>>> 
>>> 
>>> Harper Lee: Real courage is when you know you're licked before you begin,
>>> but
>>> you begin anyway and see it through no matter what.
>>> 
>>> University of Western Sydney
>>> Rm UG05, Building U
>>> Kingswood Campus
>>> Ph: +61-47360391
>>> Mob: +61-438-454-127
>>> www.uws.edu.au 
>>> 
<https://email.uws.edu.au/owa/redir.aspx?C=b8a0230cbcb44c06b284e6f5777c2892&>>>
U
>>> RL=http%3a%2f%2fwww.uws.edu.au>
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________
>>> 
>>> csaa-forum
>>> discussion list of the cultural studies association of australasia
>>> 
>>> www.csaa.asn.au
>>> 
>>> change your subscription details at
>>> http://lists.cdu.edu.au/mailman/listinfo/csaa-forum
>>> 
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________
>>> 
>>> csaa-forum
>>> discussion list of the cultural studies association of australasia
>>> 
>>> www.csaa.asn.au
>>> 
>>> change your subscription details at
>>> http://lists.cdu.edu.au/mailman/listinfo/csaa-forum
>>> 
>>> 
>>> 
>>> 
>>> --
>>> Dr Mark Gibson
>>> Communications and Media Studies Program
>>> National Centre for Australian Studies
>>> School of Journalism, Australian and Indigenous Studies
>>> Faculty of Arts, Monash University
>>> Caulfield East, Victoria 3145
>>> Australia
>>> 
>>> Caulfield Campus, B4.17
>>> 
>>> Tel: +61 3 9903 4221
>>> Fax: +61 3 9903 4225
>>> 
>>> 
>>> 
>>> 
> 




More information about the csaa-forum mailing list