Charlton BG. Peer usage versus peer review BMJ 2007; 335: 451
It is often asserted that peer review is the essence of scientific evaluation, but this is incorrect. Peer review is not specific to science but is employed by all academic subjects from English literature to theology. Neither is it necessary to science. Until a few decades ago—and during the scientific golden age of the mid-20th century—there was very little peer review in the modern sense. So peer review is neither necessary nor sufficient for scientific progress.
The truly definitive scientific evaluation is in fact "peer usage," which entails testing facts and theories not by opinion but in actual practice. This means that, even when published in the best journals, new science should never be regarded as valid until its predictions have been retrospectively validated by use in further relevant research by competent scientific peers.
Peer usage is essential to science because it evaluates how research stands up when used for intervening in the natural world. This is often termed "replication"; however, it is not usually repetition but instead a process by which ideas and facts are incorporated into future successful research. As long as later research that is built on earlier research continues to grow and thrive, then that earlier science is provisionally regarded as valid.
But peer usage is a retrospective process, and testing science by usage is slow and expensive. It involves persuading other scientists that it is worth their while to expend energy and resources. Evaluation by peer usage has a timescale of years. Published research must be noticed, read, understood, incorporated; new work must be planned and executed, then published, noticed, read, etc. Peer usage is also incomplete, because more scientific theories and findings are published than can ever be checked in practice. Only a small percentage of published science ever actually gets evaluated by peer usage.
As a result, there has been a major shift away from retrospective peer usage towards the predictive process of peer review. Peer review is faster (taking weeks rather than years) and cheaper (because it asks only for opinions). In effect peer review is prospective filtering by a consensus of informed judgment.
Although peer review is not specifically scientific, in principle it can identify ideas and facts that are probably correct, so long as research is an incremental extrapolation of established knowledge, methods are standard and well established, and investigations are performed by researchers of validated competence. In other words, peer review usually works well for applied science or "research and development."
However, peer opinion becomes markedly less valid when research is more ambitious and radical. Many or most major scientific advances were initially rejected by peer review. This implies that there is a continuing need for other methods of evaluating radical and ambitious science.
Traditionally, editorial review is the main alternative to peer review. A scientist editor or editorial team applies a sieve, with varying degrees of selectivity, to research submissions. Strictly, this process should not attempt to predict whether ideas and facts are "true," because truth can be established only in retrospect. Instead, editorial selection works within constraints of subject matter on the basis of factors such as potential importance and interest, clarity and appropriateness of expression, and broad criteria of scientific plausibility. Even probably untrue papers may be judged worth publishing if they contain aspects (ideas, perspectives, data) that are potentially stimulating to the development of future science.
In my personal experience, editorial review remains a viable model for publishing in modern biomedical science. Medical Hypotheses explicitly uses editorial review and aims to publish bold and radical ideas; yet the journal has a 2006 impact factor of 1.299, and papers are downloaded an average of 26 000 times per month. This implies that the journal is being used by other scientists to a significant and worthwhile extent.
The most prestigious scientific journals like to imply that their publications are not just radical but also true. This is simply hype. When published science is (almost certainly) true then it cannot be important; and when science is potentially revolutionary then it cannot be regarded as true (until subjected to evaluation by peer usage).
Peer review is valuable for predicting the probable validity of modestly incremental science; but there remains an important role for journals that use editorial review, on the basis that true scientific validity can be established only after publication, by the slow and rigorous methods of peer usage.
