Peer-review and its discontents

The latest issue of the Post-Autistic Economics Review is now out, available here.   It has an interesting article by philosopher Donald Gillies arguing against the centrally-organized reviews of university research activities which British academics have had to endure these last 20 years, and which now look likely to be adopted in Australia, NZ and elsewhere.  One argument he makes is that one’s peers are usually quite bad at judging the long-run impact and quality of one’s research, especially when the research is innovative, and Gillies gives the example of Frege’s Begriffsschrift, the first axiomatic treatment of propositional and predicate calculus.  When this was published in 1879, it was slammed by Frege’s contemporaries, and it was only recognized for the seminal work it is two decades later.  If Frege had been working in a British University a hundred years later, both he and his department may have faced termination by his university administration, given the hostility that his own peers felt towards his work; lots of departments have been closed, and academics made unemployed, as a result of the peer assessments of the British Research Assessment Exercise (RAE).

A longer version of Gillies’ paper is available on his web-site, here.  

8 thoughts on “Peer-review and its discontents

  1. I’ve also commented on Donald Gillies’ paper on my blog. Donald was my PhD supervisor and has always had what to my way of thinking is a refreshing attitude to philosophy which demands of philosophers that they look at developments in other fields, in his case probability theory, artificial intelligence, and more recently medicine.

    I’d be interested to know from Americans whether they believe the tenure system has similar effects.

  2. The British assessment system is really 20 years old? I was under the impression that it was a fairly new development. I haven’t had a chance to read Gillies’ paper yet, but to be convinced, I’d want a more recent example of how badly the academic establishment misjudge the importance of something, only to realize later it’s significance. A post-World War II example would be particularly convincing.

    I could believe that tenure has the same effect, but I would imagine it varies by field. As the time to tenure in physics gets longer and longer, I could see researchers getting more and more conservative, to the point that once they get tenure they forget how to be mavericks. In other fields where the process is not quite so prolonged, it would be easier for someone to just suck it up for a few years until they get the chance to take risks.

  3. Walt, according to Wikipedia, the first British RAE was in 1986.

    IMHO, to refute the claims of a universalist process such as the RAE, only one important counter-example (of the failure of peer-review to support paradigm-shifting work) would be needed. I don’t see why such a counter-example would be more convincing if it occurred in the last 60 years. Perhaps it may be more convincing if it happened since the widespread adoption of the Web, since that has probably affected day-to-day academic practice more than any other innovation of the last few hundred years. But there are still respected academic journals in the humanities which solicit papers, send them for peer-review, make editorial decisions, and obtain final copy of accepted articles, all as if the Internet did not exist.

    As someone in a UK department eager to retain its high RAE rating, I can attest that a large amount of university money, faculty time and faculty and administrative effort is spent preparing RAE submissions, including mock submissions with paid external mock assessors. Although I am sure it increases our chance of a good grade, I very much doubt that such preparatory activity is positively correlated with our research capabilities or achievements. Like so much of life in Britain under Tony Blair (eg, the National Health Service, the education system, the police service), the emphasis is now on form-filling to the detriment of real work.

    Regarding peer reviewing, you may be amused by this witty game-theoretic treatment of the journal peer-review process by Chalmers and Herzberg. (The paper is available electronically from Jstor, for those with access.)

    @ARTICLE{chambers:herzberg:as68,
    author = “J. M. Chambers and A. M. Herzberg”,
    title = “A Note on the game of refereeing”,
    journal = “Applied Statistics”,
    year = “1968″,
    volume = “17″,
    number = “3″,
    pages = “260–263″}

  4. I should have thought that the situation in philosophy is worse than in physics in terms of detecting, before they reach say 30, talented people who can lead the field in promising new directions. The adoption of a new orientation in philosophy often entails that the majority of work in the field is misguided. E.g., if the Wittgenstein of the late 30s and 40s is right, much of the philosophy of the time was leading nowhere. If Collingwood was right that metaphysics is the historical study of the absolute presupposition of the various sciences (in the broadest sense), then just about everything done since in metaphysics is wrongheaded.

    Of course, one might say that the RAE has nothing to do with it. Had Wittgenstein first come to Russell with his later ideas on language games already in place, he would probably have politely been shown the door. His initial ideas were just right to gain him powerful backers. Now, it is hardly surprising that in ones early years, one engages with existing thinkers. The problem comes for someone whose natural starting points are currently out of favour, quite possibly the kind of person with the potential to do something novel. In this case an assessment exercise such as the RAE coupled to intense competition for jobs is a recipe for disaster.

  5. I didn’t doubt that the RAE was so old; I was just surprised to learn the fact, since I’d only heard of it a few years ago. (One thing the internet has made possible is the ability to hear complaints from academics throughout the English-speaking world. :-) ) I’d like a more recent example of the problem just because whenever someone brings up an example of academics totally missing the import of some work, the examples themselves always seem to be fairly old. (Grassman is a standard example I’ve heard before.) The post-World War II academic environment is so different from what came before (at least in the United States) that I don’t know if the same conditions hold.

    I don’t know much about the job market in philosophy, so it could be much worse than physics. Physics has the peculiar feature that you can go fifteen years between entering grad school and getting tenure, which has to weigh heavily on the minds of potential mavericks.

  6. On post-war peer-reviewing, Roy Weintraub (reference below) has a fascinating account of how the key paper in post-war mathematical economics came to be published. This was a paper in the journal Econometrica by Ken Arrow and Gerard Debreu in 1954 which showed the existence of an equilibrium in an economy (strictly, in a certain class of mathematical abstractions of an economy, which were not anything like real economies).

    This was a case of one reviewer not being independent of the authors, and maybe not reading the paper very carefully because he knew and trusted the authors. The other reviewer opposed its publication, for what may have been sound mathematical reasons, although it is also not clear that this reviewer read or understood the paper. Their reviews were followed by the editor ignoring the opinion of the second reviewer, seemingly because he was not from an Ivy League institution. It is hard to sustain any belief in the objectivity of the peer-review process after reading this account.

    @BOOK{weintraub:book02,
    author = “E. Roy Weintraub”,
    title = “How Economics became a Mathematical Science”,
    publisher = “Duke University Press”,
    year = “2002″,
    series = “Science and Cultural Theory”,
    address = “Durham, NC, USA”}

    @ARTICLE{arrow:debreu:econ54,
    author = “Kenneth J. Arrow and Gerard Debreu”,
    title = “Existence of an equilibrium for a competitive economy”,
    journal = “Econometrica”,
    year = “1954″,
    volume = “22″,
    number = “3″,
    pages = “265–290″}

  7. Walt –

    I am interested in what you say about mavericks in physics. How well are cosmologist Max Tegmark’s ideas on possible worlds accepted in physics? He has suggested (if I understand him correctly) that there may be not only alternative possible worlds having the same constants and laws of physics as ours, but worlds with different constants or laws, and even worlds just comprising different conceptual and mathematical entities.

    These last type of worlds sound very similar to biologist Rupert Sheldrake’s concept of morphic fields. Yet Sheldrake was attacked by the scientific community when he first proposed them: the journal Nature even said that his books should be burnt. Tegmark, by contrast, has been published in Scientific American. Why the difference? Is it just a matter of time, or are physicists more inclined by nature to accept weird ideas than biologists?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>