Charlton BG. Medical Hypotheses 2006 impact factor rises to 1.3 – A vindication of the ‘editorial review’ system for revolutionary science. Medical Hypotheses
2007; 69: 967-969
Editorial
Summary
The Thomson Scientific Impact Factor (IF) for Medical Hypotheses has risen to 1.299 for 2006. This means that the IF has more than doubled since 2004, when it stood at 0.607. Using Elsevier’s Scopus database; in 2004 there were 437 citations to Medical Hypotheses papers published in the previous two years – by 2006 this had trebled to 1216 citations. Monthly internet usage of Medical Hypotheses run at an average of about 26 000 papers downloaded per month. An IF of 1.3 means that Medical Hypotheses has now entered the mainstream level of ‘respectable’ medical journals, in terms of its usage by other scientists. This is particularly pleasing given the aim of the journal is to publish radical and speculative ideas. A healthy IF is important to Medical Hypotheses because the journal deploys a system of editorial review, rather than peer review, for evaluation and selection of papers. Editorial review involves selection of a journal’s content primarily by an editor who has broad experience and competence in the field, assisted by a relatively small editorial advisory board. The great advantage of editorial review is that it is able, by policy, to favour the publication of revolutionary science. But since editorial review relies on hard-to-quantify and non-transparent individual judgments, it is important for its outcomes to be open to objective evaluations. Scientometric measures of usage such as citations, impact factors and downloads constitute objective evidence concerning a journal’s usefulness. Since Medical Hypotheses is performing adequately by such criteria, this provides a powerful answer to those who fetishize peer review and regard any other system of evaluation as suspect. Journal review procedures are merely a means to the end, and the end is a journal that serves a useful function in the dynamic process of science. Medical Hypotheses can now claim to perform such a role.
***
I am pleased to report that the Thomson Scientific Impact Factor (IF) for Medical Hypotheses has risen to 1.299 for 2006. This means that the IF has more than doubled since 2004, when it stood at 0.607 (www.scientific.thomson.com).
The IF is (approximately) a measure of the average number of times a paper in a journal is likely to be cited. Although there are important differences in citations according to research fields, and although IF has limitations if used to evaluate the potential importance of specific articles or specific scientists over the short term; nonetheless, I regard the general level of IF as a broadly valid measure of a journal’s importance among scientific peers.
Another measure of a journal’s profile in the scientific literature is the total number of citations per year, and here too Medical Hypotheses is thriving. Using Elsevier’s Scopus database (www.scopus.com), in 2004 there were 437 citations to Medical Hypotheses papers published in the previous two years – by 2006 this had trebled to 1216 citations. The journal’s influence is clearly expanding.
Furthermore, the monthly internet usage of Medical Hypotheses 2005–6 runs at an average of about 26 000 papers downloaded per month, which again indicates a very healthy level of interest from the broad scientific community (www.intl.elsevierhealth.com//journals/MeHy).
Aside from professional scientific considerations, Medical Hypotheses has an unusually high media impact (as can be seen by looking at internet news sources or performing web searches). Of particular interest to Medical Hypotheses readers is the imminent publication of a book about the journal, written by Roger Dobson and published by Cyan Books (London, UK). The book is titled: Death can be cured: and 99 other Medical Hypotheses. I have written a foreword, and can recommend the book as an edifying and amusing journey through some of the more stimulating ideas that have been published in the journal over recent years.
An IF of 1.3 means that Medical Hypotheses has now entered the mainstream level of ‘respectable’ medical journals, in terms of its usage by other scientists, and the probability is that this figure will rise further over the next three years. This is particularly pleasing given the aim of the journal is to publish radical and exploratory ideas [1] and [2], which inevitably means a greater risk that papers will be ignored by other researchers as being too speculative.
Another reason that a healthy IF is important to Medical Hypotheses is that the journal usually deploys a system of editorial review, rather than peer review, for evaluation and selection of papers.
Peer review in journals is used in a wide range of academic subjects including the arts and social sciences, and for most of its history science did not use modern methods of peer review; so that contrary to common assertions, peer review is neither distinctive nor essential to the natural sciences. What characterizes science is in fact evaluation by ‘peer usage’ – extrapolating facts and ideas to predict future observations, and making such observations in order to test published facts and ideas [3] and [4]. In other words, scientific evaluation comes after publication, and not before.
Currently, peer review in journals is a system whereby a prospective paper is vetted by gathering the opinions of two or three specialists in the field, any of which can (in practice) usually veto publication. The result of rigorous peer review (although most peer review, being unpaid and done as a favour, is probably far from rigorous) tends to be that published work is more reliable but less ambitious [1] and [5]: peer reviewed publications tend to be incremental advances on previous knowledge, performed by accredited professionals, and attained by standard techniques [4].
If Kuhn’s distinction between ‘revolutionary’ and ‘normal’ science is used [6] then peer reviewed research is therefore usually valuable for normal science, and when rigorous may then enable research to be regarded as probably valid at the time of publication. But the intrinsic tendency is for peer reviewed research to be limited in its scope and ambition so peer review intrinsically discriminates against revolutionary science [5].
Since Medical Hypotheses aspires to be a journal of revolutionary science ideas [1] and [2], it is clear that this aim would tend to be thwarted by peer review. Research which aspires to be ‘revolutionary’ science is relatively unlikely to get through the innate conservatism of peer review [5], since bold ideas are indeed less likely to be correct than cautious ideas. Looking back at the golden age of revolutionary biological science during the mid-twentieth century, it seems obvious that modern peer review would likely have killed the necessary ferment of radical speculations; some which turned-out to be accurate, but most of which were mistaken [7].
Until a few decades ago, the evaluation of scientific papers for journals was mainly by a process which I term ‘editorial review’. Editorial review involves selection of a journal’s content primarily by an editor who has broad experience and competence in the field, assisted (to a greater or lesser extent) by a relatively small editorial advisory board. The great advantage of editorial review – from the perspective of Medical Hypotheses – is that it is able, by policy, to favour the publication of revolutionary science.
Naturally, editorial review relies on hard-to-quantify and non-transparent individual judgments, which is why it is particularly important for its outcomes to be open to objective evaluations. Scientometric measures of usage such as citations, impact factors and downloads – albeit incomplete and imperfect – are all objective evaluations which tend to quantify a journal’s usefulness. So long as Medical Hypotheses is performing adequately by such criteria, this provides a powerful answer to those who fetishize peer review and regard any other system of evaluation as suspect.
In the end, journal review procedures are merely a means to the end, and the end is a journal that serves a useful function in the dynamic process of science [3]. Medical Hypotheses can now claim to perform such a role.
Acknowledgement
Thanks are due to Tanya Wheatley, Senior Publishing Editor in Health Sciences at Elsevier, for some of this data; and for her invaluable editorial support and strategic advice. I also like to thank the authors of Medical Hypotheses publications during the time of my editorship so far: upon their work, everything else depends.
References
[1] D.F. Horrobin, Ideas in biomedical science: reasons for the foundation of Medical Hypotheses, Med Hypotheses 1 (1975), pp. 1–2.
[2] B.G. Charlton, Inaugural editorial, Med Hypotheses 62 (2004), pp. 1–2.
[3] D.L. Hull, Science as a process, Chicago University Press, Chicago (1987).
[4] B.G. Charlton, Conflicts of interest in medical science: peer usage, peer review and ’CoI consultancy’ (Editorial), Med Hypotheses 63 (2004), pp. 181–186.
[5] D.F. Horrobin, The philosophical basis of peer review and the suppression of innovation, JAMA 263 (1990), pp. 1438–1441.
[6] T.S. Kuhn, The structure of scientific revolutions, Chicago University Press, Chicago (1970).
[7] H.F. Judson, The eighth day of creation: makers of the revolution in biology, Jonathan Cape, London (1979).
Sunday, 23 September 2007
Saturday, 1 September 2007
Peer usage versus peer review
Charlton BG. Peer usage versus peer review BMJ 2007; 335: 451
It is often asserted that peer review is the essence of scientific evaluation, but this is incorrect. Peer review is not specific to science but is employed by all academic subjects from English literature to theology. Neither is it necessary to science. Until a few decades ago—and during the scientific golden age of the mid-20th century—there was very little peer review in the modern sense. So peer review is neither necessary nor sufficient for scientific progress.
The truly definitive scientific evaluation is in fact "peer usage," which entails testing facts and theories not by opinion but in actual practice. This means that, even when published in the best journals, new science should never be regarded as valid until its predictions have been retrospectively validated by use in further relevant research by competent scientific peers.
Peer usage is essential to science because it evaluates how research stands up when used for intervening in the natural world. This is often termed "replication"; however, it is not usually repetition but instead a process by which ideas and facts are incorporated into future successful research. As long as later research that is built on earlier research continues to grow and thrive, then that earlier science is provisionally regarded as valid.
But peer usage is a retrospective process, and testing science by usage is slow and expensive. It involves persuading other scientists that it is worth their while to expend energy and resources. Evaluation by peer usage has a timescale of years. Published research must be noticed, read, understood, incorporated; new work must be planned and executed, then published, noticed, read, etc. Peer usage is also incomplete, because more scientific theories and findings are published than can ever be checked in practice. Only a small percentage of published science ever actually gets evaluated by peer usage.
As a result, there has been a major shift away from retrospective peer usage towards the predictive process of peer review. Peer review is faster (taking weeks rather than years) and cheaper (because it asks only for opinions). In effect peer review is prospective filtering by a consensus of informed judgment.
Although peer review is not specifically scientific, in principle it can identify ideas and facts that are probably correct, so long as research is an incremental extrapolation of established knowledge, methods are standard and well established, and investigations are performed by researchers of validated competence. In other words, peer review usually works well for applied science or "research and development."
However, peer opinion becomes markedly less valid when research is more ambitious and radical. Many or most major scientific advances were initially rejected by peer review. This implies that there is a continuing need for other methods of evaluating radical and ambitious science.
Traditionally, editorial review is the main alternative to peer review. A scientist editor or editorial team applies a sieve, with varying degrees of selectivity, to research submissions. Strictly, this process should not attempt to predict whether ideas and facts are "true," because truth can be established only in retrospect. Instead, editorial selection works within constraints of subject matter on the basis of factors such as potential importance and interest, clarity and appropriateness of expression, and broad criteria of scientific plausibility. Even probably untrue papers may be judged worth publishing if they contain aspects (ideas, perspectives, data) that are potentially stimulating to the development of future science.
In my personal experience, editorial review remains a viable model for publishing in modern biomedical science. Medical Hypotheses explicitly uses editorial review and aims to publish bold and radical ideas; yet the journal has a 2006 impact factor of 1.299, and papers are downloaded an average of 26 000 times per month. This implies that the journal is being used by other scientists to a significant and worthwhile extent.
The most prestigious scientific journals like to imply that their publications are not just radical but also true. This is simply hype. When published science is (almost certainly) true then it cannot be important; and when science is potentially revolutionary then it cannot be regarded as true (until subjected to evaluation by peer usage).
Peer review is valuable for predicting the probable validity of modestly incremental science; but there remains an important role for journals that use editorial review, on the basis that true scientific validity can be established only after publication, by the slow and rigorous methods of peer usage.
It is often asserted that peer review is the essence of scientific evaluation, but this is incorrect. Peer review is not specific to science but is employed by all academic subjects from English literature to theology. Neither is it necessary to science. Until a few decades ago—and during the scientific golden age of the mid-20th century—there was very little peer review in the modern sense. So peer review is neither necessary nor sufficient for scientific progress.
The truly definitive scientific evaluation is in fact "peer usage," which entails testing facts and theories not by opinion but in actual practice. This means that, even when published in the best journals, new science should never be regarded as valid until its predictions have been retrospectively validated by use in further relevant research by competent scientific peers.
Peer usage is essential to science because it evaluates how research stands up when used for intervening in the natural world. This is often termed "replication"; however, it is not usually repetition but instead a process by which ideas and facts are incorporated into future successful research. As long as later research that is built on earlier research continues to grow and thrive, then that earlier science is provisionally regarded as valid.
But peer usage is a retrospective process, and testing science by usage is slow and expensive. It involves persuading other scientists that it is worth their while to expend energy and resources. Evaluation by peer usage has a timescale of years. Published research must be noticed, read, understood, incorporated; new work must be planned and executed, then published, noticed, read, etc. Peer usage is also incomplete, because more scientific theories and findings are published than can ever be checked in practice. Only a small percentage of published science ever actually gets evaluated by peer usage.
As a result, there has been a major shift away from retrospective peer usage towards the predictive process of peer review. Peer review is faster (taking weeks rather than years) and cheaper (because it asks only for opinions). In effect peer review is prospective filtering by a consensus of informed judgment.
Although peer review is not specifically scientific, in principle it can identify ideas and facts that are probably correct, so long as research is an incremental extrapolation of established knowledge, methods are standard and well established, and investigations are performed by researchers of validated competence. In other words, peer review usually works well for applied science or "research and development."
However, peer opinion becomes markedly less valid when research is more ambitious and radical. Many or most major scientific advances were initially rejected by peer review. This implies that there is a continuing need for other methods of evaluating radical and ambitious science.
Traditionally, editorial review is the main alternative to peer review. A scientist editor or editorial team applies a sieve, with varying degrees of selectivity, to research submissions. Strictly, this process should not attempt to predict whether ideas and facts are "true," because truth can be established only in retrospect. Instead, editorial selection works within constraints of subject matter on the basis of factors such as potential importance and interest, clarity and appropriateness of expression, and broad criteria of scientific plausibility. Even probably untrue papers may be judged worth publishing if they contain aspects (ideas, perspectives, data) that are potentially stimulating to the development of future science.
In my personal experience, editorial review remains a viable model for publishing in modern biomedical science. Medical Hypotheses explicitly uses editorial review and aims to publish bold and radical ideas; yet the journal has a 2006 impact factor of 1.299, and papers are downloaded an average of 26 000 times per month. This implies that the journal is being used by other scientists to a significant and worthwhile extent.
The most prestigious scientific journals like to imply that their publications are not just radical but also true. This is simply hype. When published science is (almost certainly) true then it cannot be important; and when science is potentially revolutionary then it cannot be regarded as true (until subjected to evaluation by peer usage).
Peer review is valuable for predicting the probable validity of modestly incremental science; but there remains an important role for journals that use editorial review, on the basis that true scientific validity can be established only after publication, by the slow and rigorous methods of peer usage.
Subscribe to:
Posts (Atom)