Peer Review - Do Away with It!

The peer review system in science, whereby prior to publication manuscripts are reviewed by anonymous peers, has come under recent criticism in a debate at the Royal Society, at which a case for its abolition was argued.

The concept that a piece of research should not be reviewed by other experts in the field is, of course, ridiculous. But I think doing away with peer review in its present form is feasible. I wrote about this in 2012, and reproduce what my ideas would be here, with some tweaks.

There are two major current problems: the cost of publishing and access, and the inequities of the peer review process. Both could be cured at once.

We need ONLY ONE, publicly-run, open-access scientific publishing domain to which ALL ARTICLES are uploaded for free, in whatever format wished for by the authors and without prior peer review. This domain would subsume all existing primary research publishing. After an initial, publically-funded development phase, the small costs of maintaining this domain could be obtained through discreet advertising revenues. The model thus obviates both the need for  charging huge amounts for access to  journals and the need to charge authors for each publication submitted.

Once an article is uploaded it will easily be able to be found by a keyword search, such as exists in PubMed or Web of Science. Reviewing would not be solicited but would be open and online, in the fashion of "comments" to a blog entry.  Any given article might thus receive many or no reviews. PubPeer goes a step in this direction. However, in my opinion no reviews would be anonymous: only registered reviewers who have revealed their identities would be allowed to post comments. 

Before entering comments, the qualifications of the commenter would be verified e.g. PhD in the field of the article. All may comment, even the unqualified, but their qualifications would be public. I would suggest separate comment threads for specialists and non-specialists. Many of the reviews, even from the specialists,  are likely to be incompetent, but the reviews would also be open to review, and ranking, as would the reviews of the reviews etc.. The paper itself would be continuously modifiable by the authors (as in Wikipedia), to add results or respond to criticisms etc.

In the above system there would be no need for a decision to be made a priori as to whether an article is "of sufficient general interest" before publication - this would all be decided by the readers afterwards: a points system could be devised whereby an article gains prominence depending on how many times it is accessed from different computers, cited later on, and on the reviews received. As an article rises in points, so would its visibility in the web domain. Extremely hot articles would be expected to very rapidly gain prominence.

Any objections?

Peer Review, Again

Peer review can be exasperating, but also fun. A while ago I reposted some amusing reviewer comments from the journal "Environmental Microbiology". Well, just after that we submitted a paper for which one reviewer was raving about the work and the other thought it boring - chalk and cheese it was.

Of course I can't repost the above anonymous reviews here. But I thought it would be fun to compose two reviews myself, the first scathing and the other glowing.  Although fictitious, they are composed of actual reviewer comments about our manuscripts submitted over the last year or so. (And, yes, one of the reviewers, who presumably has a Ph.D., does spell the word "know" as "no".)

Reviewer I.

 Essentially, the paper either tells us what we already know, or makes novel assertions which are uncheckable and based on needless ad-hoc-erie. The authors should change the tone of their paper to be constructive rather than "criticizing" the work of other people - by comparing to such works (sic), the authors are doing themselves a disservice, as they should no (sic) that is really no longer acceptable. This submission will not help the reputations of any of the coauthors as it is really not at the level one would expect for three of the main authors. They should all provide a bit more guidance, mentorship and advise (sic)  to the apparently junior author, in the opinion of the review (sic) who is very familiar with the very nice high quality publications of Smith. The impression left on this reviewer is one of being cheated, as if only the first half of a fine review article was provided. Poorly designed and provides only trivial results, if any. The ball-and-spring model is too simplistic and I do not think that any of the conclusions can be reliable.

Reviewer II.

An important and sorely-needed contribution to the field and will have an impact, or should have, not only on practical strain development, but also in basic research into microbial physiology. The approach stimulates new ideas, is interesting and very promising judging from the results exposed in this manuscript.  The paper is clearly written, well executed, and richly supported by an abundance of supplemental materials. The work is thorough, complete and well written. This interesting paper provides new insights and suggests a scheme which has many interesting elements and should make an important contribution to the field. In summary, this manuscript is an important physical contribution to our understanding of conformational transitions in proteins.This is an outstanding piece of work: an exciting and unexpected discovery that may also be of importance to other biocatalysts. It is an important step towards interpreting neutron scattering data and obtaining quantitative agreement between MD results and neutron data. The simulations are carried out competently, the mathematical analysis is thorough, the results appear to be of very high quality and the presentation is clear. The work should be of general interest for all kinds of organisms. It provides a good example of this type of approach and has been carefully performed –

Hmm...accept or reject, I wonder?

Massive Climate Data Scam?

OK, so I respond to the comment on the previous post claiming that climate change data are phony and rigged to achieve a predetermined conclusion. The principal reason that this is unthinkable is that hundreds climate data have passed the peer review system by which anonymous experts opine on the veracity of data and conclusions in any scientific paper. Anonymity means that the authors are not told the identity of the reviewers who are then freer to criticize without fear of professional retribution. Now, the peer review system is highly imperfect – the reviews themselves are frequently wrong or contradict each other, and thus rubbish gets published, including in the very top journals. We’ve had famous examples of that, even from respected professors, such as the water memory/cold fusion affairs, and, in the other direction, future Nobel-prizewinning science has been initially rejected. Furthermore, peer review, and the funding process, does not fully protect against cronyism and gatekeeping: a paper or grant proposal challenging orthodoxy is more likely to be rejected (unless clearly irrefutable) as the editor and reviewers are more likely to support the mainstream. But this is natural: extraordinary claims require extraordinary evidence. Further, these papers can be resubmitted and, if they are of some merit they will likely eventually appear in some other peer reviewed journal, where they will then likely attract unusual subsequent attention. The nature of science is such that these published results can be objectively tested independently, and by this process the truth normally is established. Further, scientists in any field LOVE to prove each other wrong because it is a great holier-than-though schadenfreude feather to have in one’s cap. I have a colleague who spent almost his entire illustrious career doing that. Science is highly egotistical and competitive – we work against more than with each other.

In the case of climate change, there are large uncertainties and unknowns in the science. However, the critical point is that we are not just talking about one set of data or a single paper, but a large collection of observations and calculations: 928 papers between 1993 and 2003, for example, none of which disagree with the consensus. If the data supporting the general warming trend were to be mostly ‘phony’ and ‘rigged’ then there would have had to have been a fraud of absolutely unprecedented proportion. Thousands of brilliant, accomplished and respected scientists, reviewers and editors from all over the world, working for many different disparate, independent organizations, would have had to have conspired to create an enormous, insulated, fantasy world of rigged publications. If ever that happens public mistrust will be such that society may well give up science altogether and go back to the days of pre-enlightenment irrationality: a state of affairs in nobody's interest. Even had there been a political incentive for a massive scam, which there was not, it would have been unworkable.