Below you will find the responses to our last cyberconference which ran from Friday, 28 May to Monday, 14 June 1999. This was a global cyberconference on peer review in the social sciences.

Towards a more inclusive absolute peerage?
Conf: Peer Review in the Social Sciences

From: Steve Fuller
Date: Friday, May 28, 1999 10:40 AM
If everyone working in a given field — ‘absolute peers’ in the literal sense — contributed to selecting the pool of peer reviewers, both the people selected and the judgements reached would be substantially different from what we now see. What do you think?

From: Morris Minor
Date: Thursday, June 10, 1999 07:52 AM
Yes, I agree. And this is why funding agencies need to be more proactive about soliciting peer reviewers.

Topic: Relative Peers: beyond research ghettos (1 of 2), Read 107 times
Conf: Peer Review in the Social Sciences

From: Steve Fuller
Date: Friday, May 28, 1999 10:41 AM
A researcher’s ‘relative peerage’ may be subject to considerable prejudice. For example, anecdotal evidence suggests that feminist researchers rarely peer-review men, regardless of overlap in substantive interests; whereas men may be periodically asked to evaluate feminist research because of such overlap—and are generally asked to review proposals farther afield than women are. In this respect, relative peerage is the final frontier of anti-discriminatory knowledge policy. To continue the example: to involve more women in peer review — only so that they can review other women — is little more than to ghettoize women’s knowledge practices. If a journal or agency is truly committed to integrating traditionally suppressed perspectives, then it must remedy this basic problem in the definition of relative peerage. But how? What do you think?

From: Merle Jacob
Date: Sunday, May 30, 1999 01:12 PM
Much of the problem with peer review in this respect is also related to the way academe itself is structured. Those who have more resources in the system (e.g. tenure) have a higher chance of being in a position to gather more resources. One of the little details, it may be interesting to investigate with respect to the research ghetto argument (as it applies to women) is ‘how are women distributed in the academy?’ Are there more women doing feminist research either within traditional women studies settings or elsewhere than women doing research in other fields? Similar questions could be raised about some areas within cultural studies?

Topic: Peers away from the front lines of research (1 of 3), Read 81 times
Conf: Peer Review in the Social Sciences

From: Steve Fuller
Date: Friday, May 28, 1999 10:43 AM
Finally, attention needs to be paid to the fact that there may be a partial division of labour between people who publish a lot in a field and those who publish much less but read much more widely in the field. The relative peers I have in mind here are people who have withdrawn from, if they ever spent time at, the front lines of research. These people may be primarily teachers or administrators, and may well constitute the ‘silent majority’ of ‘users and beneficiaries’ of research, even within the academy. Because these people are not personally invested in the future of particular lines of research, they may be better able to offer dispassionate peer evaluations. What do you think?

From: Jochen Glaeser
Date: Monday, May 31, 1999 01:08 PM
This division of labour we can observe well within the field. I am not sure how much the ‘Old Boys’ do empirical research of the kind they have to review. In both cases (mine and the people Steve has in mind) the danger is that the peers have lost their ability to evaluate the practical side of research, i.e. whether proposed work is methodologically doable or whether the data a conclusion is based upon are trustworthy.

To give an extreme case as example: In a conference on fraud in science we had a guest from the natural sciences who happened to be ‘substitute reviewer’ of a falsified article. His institute’s chief was the official reviewer but had not enough time or felt incompetent and therefore delegated the work. After reading the article, our ‘substitute reviewer’ stated that none of the authors could ever has applied the methods described in the article:The methods were presented in a way they can’t work, in a way that is not state of the art etc., i.e. in a way no scientist who works with th methods would describe them. In the ‘normal’ review process the ‘peer away from the front lines’ wouldn’t have seen this discrepancy because he hasn’t done empirical work for years …

Thus, I think one has to be very careful in selecting fields/ journals where ‘remote peers’ can be of some help.

From: Anonymous
Date: Monday, June 07, 1999 11:59 PM
If you select peers to review a submission, I would suggest one consider the submission to be reviewed, and the skills needed to review it. If the submission involves complex experimental or analytic methods, surely one seeks a methodological specialist to review such content. Indeed, it seems to me that in an area such as epidemiology, it is often a good idea to get a bio-statistician as one reviewer, rather than depend only on several epidemiologists (who might all tend to be competent but not expert in biostatistics). There is a lot of implicit knowledge involved in research, and reviewers who “who publish much less but read much more widely in the field” are unlikely to have mastered such implicit knowledge. Comparably, if the submission involves broad theoretical or conceptual issues, one would perhaps deliberately seek out the widely read to provide a review of the quality and completeness of the thinking in the submission.

In their editorial in the current edition of Science, Nathanson and Auerbach write “To advance a more cross-disciplinary and innovative program, the Office of AIDS Research has established a priority-setting process that involves advice from extramural scientists and community advocates and identifies areas that require concerted support because of trends in the epidemic, as well as developments in scientific knowledge.” In this case, NIH is (I think very appropriately) involving people so far away from “the front lines of science” that they are characterized as “community advocates”. In organizing the program review, the staff has decided that they need knowledge and understanding beyond that which they expect to find in the research community.

As I noted in another intervention, one of the problems of any review system is allocating resources (money, journal space) among disparate submissions – a good apple versus a good orange. I will indulge in ageism. I often look to older scientists with experience in science administration to provide advice on such allocations, rather than to new Ph.D.s deeply involved in the conduct of their own research.

Topic: Should track record be treated as a plus or minus? (1 of 2), Read 62 times
Conf: Peer Review in the Social Sciences

From: Steve Fuller
Date: Friday, May 28, 1999 10:48 AM
Some people regard track record as very much akin to the principle of natural selection in evolutionary theory, namely, a mark of survival that is intrinsically worthy of respect. Others are more sceptical, believing instead that track record is a liability that should be, in some sense, penalized. For example, if Professor Fart has been doing largely the same research — albeit reliably — over the last two decades, shouldn’t he forced to do something different before receving his next grant? What do you think?

From: Anonymous
Date: Wednesday, June 02, 1999 04:41 PM
In this, as in the question on ‘track record’, Steve seems to think it is easy to figure out from a proposal whether it represents a good project. Would that it were so.

Does anyone know how much of the varience in scores assigned to proposals in an important review process is due to differences in the actual quality of the work proposed, and how much is due to the failure of the proposals to provide the information needed to judge this quality? (Assuming that the varience is not primarily introduced by the review process independent of the quality of the research or proposals.)

Anyway, the qualifications and track record of the researchers proposing a piece of research are useful clues as to whether they will be successful; and it seems likely that the poor reviewers need all the clues that they can get.