[csaa-forum] ERA rankings

Tue Jul 8 12:44:42 CST 2008

  Thanks for this, Greg; and, Andrew,thanks for your incisive comments on the effects of treating the Humanities in the same way as the Sciences.  I have to say that, working in a university of technology, I am constantly confronted by the ill consequences of this mistake.  And, I should add, one of these consequences is the misapprehension that Humanities journals can be ranked in a simple heirarchic way as, it would seem, Science journals can be ranked.  This fallacy appears to be at the root of the ranking heirarchy with which the ERA exercise has confronted us.
   What I think, though, is most iniquitous is something that Greg has already raised.  That is, that, regardless of the extent to which we are able to claim journals succeed in fulfilling the criteria for, say, an A categoristion, a large number of journals indeed 50%, will have to be placed in the C category because of the use of the bell curve.  
   Thank you, Greg, for providing us with the criteria for the three categories used for the European exercise.  When I identified the ERA B category as dodgy I think that I was thinking of the consequence of the invention of the A* category.  The structural consequence of this is to devalue the B and C categories.  This is especially so when the criteria are so clearly constructed to reflect claims about quality.  This is particularly obvious when we compare the criteria for the ERA exercise against the criteria for the European exercise.  I am, here, really only repeating Greg's point, but, I believe, its importance is such that it bears repetition.  This is that, whereas the European criteria make explicit reference to their C category not being a residual category, the ERA exercise, by implication, absolutely does make C residual.  We are simply told that C journals are ones that don't qualify for the other categories.  Now, how can this claim be made when, at the same time, we are told that 50% of all journals must be categorised as C? In other words, the more successful people are in moving their journals from B or C to A* or A, the more journals will have to be moved down from those categories into B and C. 
    The interesting thing about the European ranking from this point of view is that, unlike the ERA ranking system,  it becomes a form of differentiation rather than an exercise in quality per se.  Already, I have received emails from people who are very happy that their journals have been categrorised as A* or A.  I have also received emails from people who are unhappy that their journals have been categorised as B or C.  Clearly, the assumption that the ERA ranking is about quality and that A* or A is what a journal needs to be respected is already becoming entrenched.    
cheers,
Jon

________________________________

From: csaa-forum-bounces at lists.cdu.edu.au on behalf of Greg Hainge
Sent: Tue 8/07/2008 9:55 AM
To: csaa-forum at lists.cdu.edu.au
Subject: RE: [csaa-forum] ERA rankings

For information, I just wanted to make one more point in regards to the rankings of journals in response to Jon's comments that, potentially, "B is considered dodgy and C, forget it" and his observation that these exercises can become self-fulfilling as they drive people away from lower ranked journals. This is all probably spot on and of course for that reason very worrying, especially when you look at the descriptors for the different ranks. These are the descriptors found in the guidelines on the ERA process: 

Tiers for the Australian Ranking of Journals

Overall criterion:  Quality of the papers

A* (top 5%)

Typically an A* journal would be one of the best in its field or subfield in which to publish and would typically cover the entire field/subfield.  Virtually all papers they publish will be of a very high quality.  These are journals where most of the work is important (it will really shape the field) and where researchers boast about getting accepted.  Acceptance rates would typically be low and the editorial board would be dominated by field leaders, including many from top institutions.

A (next 15%)

The majority of papers in a Tier A journal will be of very high quality. Publishing in an A journal would enhance the author's standing, showing they have real engagement with the global research community and that they have something to say about problems of some significance.  Typical signs of an A journal are lowish acceptance rates and an editorial board which includes a reasonable fraction of well known researchers from top institutions.

B (next 30%)

Tier B covers journals with a solid, though not outstanding, reputation.  Generally, in a Tier B journal, one would expect only a few papers of very high quality. They are often important outlets for the work of PhD students and early career researchers.  Typical examples would be regional journals with high acceptance rates, and editorial boards that have few leading researchers from top international institutions.

C (next 50%)

Tier C includes quality, peer reviewed, journals that do not meet the criteria of the higher tiers.

One would think from this that in fact A* A and B would be very good journals, whilst C might be the only dodgy category. As this drives changes in publication amongst researchers independently or at the behest of institutions will this be how the B category is considered? Who knows. 

What I find particularly worrying about these descriptors and their respective percentile bands, however, is that 50% of journals are expected to be in category C, in other words below standard (if you take it that the descriptor for category B really describes an academic journal showing acceptable quality control, with a level of integrity and process and a range of contributing authors). This seems to me to show an extraordinary lack of faith in the academic community's ability to self-regulate. Are 50% of journals out there really below the standards laid out in B? 

Now contrast this to the equivalent paragraph from the European Reference Index for the Humanities guidelines (that drew up procedure for the European Science Foundation's equivalent exercise). Look carefully at the amount of attention given to the possibility of difference from one discipline to another, the emphasis on ensuring that even C journals are considered reputable, the careful differentiation of category A and B and the flexible percentile bands given (10-25% in the A and B category, with no hard line on the split between those two). Little wonder that their lists (yes that's right, individual lists drawn up by different discipline areas!) seem somehow much much better than the one we're currently trying to stick band aids all over. (and if you want to see their lists, be they final or initial, see here: http://www.esf.org/research-areas/humanities/research-infrastructures-including-erih/erih-initial-lists.html <http://www.esf.org/research-areas/humanities/research-infrastructures-including-erih/erih-initial-lists.html> ) 

Standards :

All journals included must fulfil normal international academic standards, i.e. selection of articles is based on an objective review policy. This quality control is normally through peerreview, and it is expected that journals would depart from peer review only where there is another system ensuring quality control. In some scholarly traditions peer-review is an unfamiliar procedure. It is one aim of ERIH to encourage top-journals to adopt a coherent peer-review system. The journals must fulfil basic publishing standards ( i.e. ISSN, timeliness of publication, complete bibliographic information for all cited references, full address information for every author).

ERIH strives only to list "good scientific journals". The only way to avoid category "C" being perceived as a residual category is to apply this principle thoroughly also to category "C". Only journals that fall into the following three categories should be included.

Categories :

1) Journals category A: i.e. high-ranking international publications with a very strong

reputation among researchers of the field in different countries, regularly cited all over

the world.

2) Journals category B: i.e. standard international publications with a good reputation

among researchers of the field in different countries.

3) Journals category C: research journals with an important local / regional significance

in Europe, occasionally cited outside the publishing country though their main target

group is the domestic academic community.

Nota bene:

6 For categories A and B, journals published in the whole world can be considered

6 For category C, only European journals must be considered.

6 It is recommended that in category A, only 10 to 25% of the total list should appear;

this percentage target will differ from one discipline to another.

2

"International journals":

1) A journal is international (Categories A and B) when the following requirements are

fulfilled in addition to those that apply to all journals:

* A genuine, varied and regular international cohort of contributors and readership

* Consistently high-quality scholarly content

* Broad consensus within the field concerning international status and visibility

2) In addition, they will have some, though not necessarily all, of the following characteristics:

* Active international advisory board

* Open to unsolicited contributions

* Highly discriminating and selective in the choice of articles published

* Published on time and to an agreed schedule

The difference between category 'A' and category 'B' journals is likely to be the degree to which they conform to 1) above, and both the number of characteristics under 2) to which they conform as well as the degree of conformity. Generally, 'A' journals should conform to more of these characteristics, and to a greater extent, than 'B' journals. Language : Main international languages in this context are English, French, German, Spanish and Russian. However, journals in other languages can also be "international", when they are being used as "forum language" for specific research communities. Similarly, there may be non-European journals, that are international in character, in languages other than European languages, which can be included in categories A and B. Process

* ESF (SCH) Member Organisations provide initial lists of journals in 15 disciplines,

and contribute through further consultation

* 15 ERIH Expert Panels analyse, harmonise and finalise lists of graded journals in

categories A, B and C.

* ERIH Steering Committee supervises and unifies methodology ("bottom up") and

reports to Standing Committee for the Humanities, and through ESF / SCH to HERA

/ European Commissions

* ESF Standing Committee for the Humanities approves final reference lists and

ensures acceptance

* They develop a financial continuity plan and make recommendations on future

development of ERIH

In the hope that this further informs deliberations

Greg

Dr Greg Hainge, Senior Lecturer in French, French Coordinator,
School of Languages and Comparative Cultural Studies, University of Queensland, Qld 4072, Australia.
tel: (Int. + 61) (07) 3365 2282  fax: 3365 6799
personal web page: geocities.com/ghainge/
                    ******
President of the Australian Society for French Studies 
                    ******
_Culture Theory and Critique_ Editorial Board.
                    ******
_Contemporary French Civilization_ Editorial Board.
                    ******
_Etudes Celiniennes_ Editorial Board.
                    ******
Australia and NZ Representative of the Société d'Études Céliniennes
-----------------------------------------------------------
CRICOS Provider No:00025B
-----------------------------------------------------------
This email message is intended only for the addressee(s)and contains information that may be confidential and/or copyright.  If you are not the intended recipient please notify the sender by reply email and immediately delete this email. Use, disclosure or reproduction of this email by anyone other than the intended recipient(s) is strictly prohibited. No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient. 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://bronzewing.cdu.edu.au/pipermail/csaa-forum/attachments/20080708/4d089ee3/attachment.html 