Peer Review or Poor Review? - You Decide

[davesmith_au]

April 24, 2011 ~ Dave Smith

In 2008 we published a thunderblog which highlighted some of the misgivings of peer-reviewed science. More recently a paper has been published in a peer reviewed journal which examines and explains many of the pitfalls of peer review. The paper titled "Classical peer review: an empty gun"[1] by Richard Smith is a study which is focused on medical review but which applies to all scientific disciplines. [More...]

[mharratsc]

Reviewers could be identified whilst authors and their affiliations retain anonymity until publication. This would have the two-fold effect of allowing the research to stand on it's own merits and add accountability to the reviewers. All comments regarding a paper could be published to allow third parties to evaluate any criticisms offered.

Hit the nail square on the head with that one!

[psychegram]

Reviewers could be identified whilst authors and their affiliations retain anonymity until publication. This would have the two-fold effect of allowing the research to stand on it's own merits and add accountability to the reviewers. All comments regarding a paper could be published to allow third parties to evaluate any criticisms offered.

Hit the nail square on the head with that one!

Yes, that stuck out to me, too, as an excellent suggestion.

There are enormous problems with the peer review system ... and it would seem to me, as a scientist, that with the introduction of peer review the development of the sciences has almost ground to a halt. Oh, sure, we get narrow technical advancements, improvements in models, etc., but paradigms seem to be locked in as never before. Debate has been squelched.

Another point from the blog post that deserves mentioning:

So where does this leave us? The public at large have, quite understandably, grown suspicious of many of the sciences. One result of this is the explosion of a plethora of web sites questioning much of what we are led to believe. But with much information comes much disinformation and it is becoming more difficult to draw the line between sound criticism of a popular paradigm and dissent for the sake of dissent.

And that puts its finger squarely on the issue. Science is rapidly losing its support. Youth are increasingly uninterested in a career in science, or even an education in its basics. The promises of science are increasingly turning out to be hollow ... witness the catastrophes of nuclear power, genetic engineering, pharmaceutical medicine, etc. Skepticism regarding 'global warming' is growing and when the ice age hits - and I believe we are now seeing the beginning of one - billions of humans will be asking, "How did the billions of dollars spent on climatology not foresee this?" And then of course, as denizens of this forum are only too aware, there is the issue of cosmology....

The danger is that the scientific paradigm will completely break down, opening the door to a new Dark Age of obscurantist mysticism ... and this will be driven, in part, by reaction against a materialist/objectivist paradigm which denies the reality of all of those subjective sensations - emotions, love, etc - that humans instinctively know to be real. You can already see this happening.

The challenge, then, is to navigate a new path, one giving birth to a new science, a science that is not in bondage to corporate or state interests, that does not spend its time thinking of new ways to control, to kill, or to extract profit, that is not based upon reductionism, on the dismembering of phenomena into their constituent parts ... but rather a science that is in service to humanity, and to the world: a holistic practice that seeks not just an understanding of the world, but a relationship with it ... and not just with those aspects that are 'understood' but with those that are not, a relationship with the unknown as such. The emphasis of such a science would necessarily shift from ingenuity to wisdom. It is likely that even terms such as 'understanding' and 'science' would have to be exchanged for a new vocabulary ('instanding' might be a better term in the former case, and as to 'science', it's root meaning of 'to split' is rather at odds with a more holistic view.)

Cosmology is a key - perhaps THE key - aspect to this. Change the cosmology, change the world. The EU project is, I feel, an important part of this. But it is not the only part, and indeed one of the things I most appreciate about this forum is the members here seem to appreciate this. Anyone is free to come and present their ideas, and while the ideas themselves certainly garner criticism, the atmosphere is far more open, far more one of 'no one person or community has all the answers', than what you find elsewhere. It is not assumed that, unless one has an advanced degree and a knowledge of higher mathematics, one is not qualified to speculate cosmologically. Intuitive understandings are just as much valued.

At any rate, to bring this all back to the subject at hand, the peer review system obviously cannot survive such a transition, not in its current form. It is very much a relic of the old paradigm ... so old that it does not even take heed of the basic Kuhnian principle that scientific paradigms themselves have more to do with scientists and their communities than with the phenomena they study. At the same time, it is highly unlikely that ApJ, A&A, MNRAS or any of the other prestigious journals will change their practices any time soon ... and so there is no way forward but to establish new journals, which put the suggestions of R. Smith into practice.

So I'd like to propose something to the EU community: why not expand the project to include a scientific journal?

Or perhaps I should phrase that as a question: are there already journals which put these principles into practice?

[jjohnson]

It sounds too much like acute observation and good reasoning combined with common sense to be adopted by the current peer review process. To change itself, the peer review process would have to first... change itself.

And, as noted, this criticism seems to be growing. The scientific process for getting the best explanations of how things work and what the effects of this or that "cause" or "perturbation" will be is increasingly mistrusted by the general public. Lay that at the feet of both the lay public, who are typically not well educated in what "science" actually is, as well as to those scientists who, due to one cause or another, tend to mislead the public or to obfuscate facts by employing confusing jargon. And let's not forget popular press writers and video shows that tend to sensationalize and, at the same time, "dumb-down" science for their target audience. Not all do this, and it is probably well-intentioned, to make science more exciting and interesting, but it seems to have the opposite effect. Science may be interesting and absorbing, for some of us, but it should not be presented as entertainment, per se. For me, that's kind of offensive. Subject and audience both get dissed in that transaction.

This apparent distrust is not limited to just the disciplines we might "traditionally" think of as EU-related, as the author is from the medical field. This disconnect is troubling because science and the pursuit and synthesis and applications of science are vital to us all. Healthy, articulate and critically thought-out skepticism is one thing; distrust is not that at all.

As ambassadors of a wide and steadily growing set of ideas, we have to remember to try to be clear and straightforward in discussing and writing about our ideas, to be open to honest and constructive criticism, and to avoid "bad-mouthing" those who have come to other conclusions. Science, in a way, is nothing but theories in evolution; everyone competes; one wishes the best ideas always float to the top and the rest are winnowed out. One may choose to build upon and improve "established facts", and one may choose to establish better facts, and ideas that spring from them. Both occur all the time. Help keep that happening.

I see psychgram and I are posting nearly simultaneously here. Well said, psychgram. The IEEE publish a journal on plasma physics, which is one of the cornerstones of EU ideas. It may or may not touch on cosmology very often (I don't subscribe so do not get it) - and I doubt that it involves itself in applying its papers to the wider world and other disciplines nearly as much as this web site and its publications do. Several universities have plasma physics research programs, including dusty plasma research as well, and they publish papers, many in the peer reviewed press. Some are rather good, as you might expect. You may ascertain that EU ideas generally are not yet accepted there, however. Nonetheless, there are nuggets in some of those papers, too, and the better read we are, the better prepared and the more resourceful and imaginative we can be.

A printed journal is avery expensive proposition, and I doubt that many would be up to paying the typical page charges that such exercises usually require. A web journal, like, say, arXiv, might be a possibility. Who serves as webmeister and how submittals are handled would be a subject in itself. Interesting suggestion. Readers? Sponsors? Volunteers? Anyone interested?

Jim

[psychegram]

Oh, I wasn't suggesting a printed journal! Definitely a web publication.

Submissions would be a two-step process. In the first, review, process, a paper could be submitted, anonymously, by anyone, for free. There would then pass a period of time during which reviewers could have a crack at it and (not anonymously) make any concerns/criticisms they have public. The original author would retain access to the publication and have the ability to revise, respond to criticisms, etc ... all out in the open, with the key thing being that no one knows who the author is. After a certain period of time has passed, the author can submit to the (volunteer/donation supported) editorial staff, who would make the final decision over whether or not to give the paper the Journal's official stamp of approval.

[mharratsc]

All processes you both mentioned could easily be done with a distributed .pdf doc.

[psychegram]

All processes you both mentioned could easily be done with a distributed .pdf doc.

Indeed! The monetary barriers to entry are essentially nil. It is a matter of time

[David Russell23]

During the last 10 years I've published 8 papers through the peer review process in The Astrophysical Journal, Astrophysics&Space Science, and the Journal of Astrophysics & Astronomy. I've also had a number of papers rejected through that process. And one time I served as a referee for a paper for ApJ Letters.

I've had some very good experiences in which the referee was thoughtful and helpful. I've also had experiences in which the referee seemed not to have read the paper because things he/she said I hadn't discussed were right there in the paper. And in some instances the referee just didn't know what he/she was talking about.

Based upon those experiences I would have to say the greatest improvement that could be made to the peer review process is to allow the author's interpretation of data to stand. Papers should be sent back for corrections if there are mistakes in handling the data. But a paper should not be rejected based upon a referee having a different opinion on the interpretation of the data. If an author makes mistakes in their interpretations then of course they should be asked to correct those. For example if the data is claimed to rule out model "X" and prove model "Y" when in fact the data is consistent with both models it is fair to ask the author to correct that claim.

I also like the idea of having a paper published with brief referee comments following the paper and then a brief response by the author to the referees comments.

Unfortunately - the article linked to the OP points out in the conclusion that journal editors and scientists almost universally like the peer review process as is ... so nothing is going to change any time soon.

[tayga]

I also like the idea of having a paper published with brief referee comments following the paper and then a brief response by the author to the referees comments.

My experiences are very similar to yours, David, and I think that this is a good suggestion. It improves on the current system where rebuttals and replies are published separately. Of course, any exchanges in this comment section should also be completed with author anonymity.

[David Russell23]

I also like the idea of having a paper published with brief referee comments following the paper and then a brief response by the author to the referees comments.

My experiences are very similar to yours, David, and I think that this is a good suggestion. It improves on the current system where rebuttals and replies are published separately. Of course, any exchanges in this comment section should also be completed with author anonymity.

I don't know what a better system would look like. As an author you can always modify a rejected paper to try and account for the referee's reaction and then resubmit to another journal. I think there should be some sort of feedback before a paper is published. The problem is that there is abuse by referee's in some cases. It gets reported all the time - and the results of the peer review process can be arbitrary. One time I had a paper recommended for publication with revisions suggested. After I made the revisions and re-submitted the original referee was unable to review it again so it went to another person who rejected the paper.

There is merit to the idea of having papers reviewed without the reviewer knowing the authors names. Unfortunately - a true expert in the field will still have a pretty good idea who the author is and depending upon the instrumentation used and observations made the idea of anonymity for the authors might be a fantasy anyway.

Just brainstorming here - another option is to have only the data analysis reviewed by the referee. So the referee would not even see the authors interpretations of the data. But I'm sure flaws could be found with that idea as well.

And another level of problem with research publication mentioned in the paper cited in the OP of this thread is that successful publication of a paper does not mean that people pay any attention to it. In fact, the more a papers results are at odd's with what the mainstream point of view "knows", the more likely it is to be ignored.

[mharratsc]

Mr. Russell said:

fact, the more a papers results are at odd's with what the mainstream point of view "knows", the more likely it is to be ignored.

Now you've hit the nail on the head, sir.

This is exactly why most of the papers regarding plasma cosmology or the Electric Universe (or any of its components) are published in the IEEE journal rather than 'mainstream' channels- you cannot break through the prejudice of the paradigm at all.

The problem stands- everyone knows it's broke, but there seems to be very little agreement on how to fix it. :\