Edit: This looks remarkably like an attempt by Science to smear their open-access competitors, given that it didn't even bother to test whether non-open-access journals have the same flaws (and in fact we know they do, including Science!), yet it's being presented as evidence that open access journals specifically are flawed. I'd hope this article would never survive peer review.
> This looks remarkably like an attempt by Science to smear their open-access competitors, ....
That may be. I wish this had been done differently, and I wish "open access" were not the term used. It may be true that the author is trying to paint with an overly broad brush, throwing some FUD at the movement to make scientific papers easier to get at; I'm not sure.
But it is clear to me that this article is describing a real problem. The problem is the recent proliferation of new journals with very low standards.
> ... and in fact we know they do, including Science! ....
I don't agree. Problems with the peer review system have been demonstrated. It is inconsistent, sometimes lazily done, and subject to corruption in some fields.
But if someone were to claim that you could take 300 journals of the caliber of Science, send them all a worthless paper, and get acceptances from half, then I would call that ridiculous. The occasional acceptance of a poor paper means that the review system needs work. The acceptance of poor papers as a matter of course indicates a lack of standards. There is a difference.
I think the point of the comment is that Science's study is purely self-serving, since they did not also send their sting paper to the more traditional journals to test their peer review rejection rate of this bogus paper.
A point needs to be brought out here. This article is not about traditional academic journals. It concerns journals of a kind that have been popping up like mushrooms in recent years. The article refers to them as "open access". I'm not fond of the term, since there are certainly selective, reliable journals that still allow submissions from anyone and publish articles on the web without a paywall. Regardless, plenty of more-or-less fake journals have been founded recently. As a researcher, I get solicitation e-mails from them every day.
It is true that it can be difficult for someone not in the relevant field to tell the difference between different kinds of journals. I know the difference, since I've heard of the major journals in my field, I know many of the editors, etc. But a science journalist might have more trouble with it.
For fields like medicine, this can matter a lot. It's the same problem we see with SEO and the like: how, in the modern world, do we form a robust measure of reputation and reliability?
Well put. I think Science does have some conflict of interest here, but I also think they are one of the good guys in the publishing business, and that this is a worthwhile subject for an expose.
Like you, I get (about 1/week) blind offers to submit articles to this or that journal, and even (at a lesser frequency) offers to be an editor (unpaid!). It stinks to high heaven [1]
Getting back to Science, their individual subscription rates are reasonable (~$150/year = 52 issues), their public health articles are available immediately after publication, and their other articles are available after 1 year. They are run by a non-profit institution (AAAS), so they are less motivated to jack up their fees.
I was not able to find institutional fees for Science.
[1] Here are the most recent 5: "International Transaction of Electrical and Computer Engineers System" (sounds like a joke), "Entropy", "The Open Software Engineering Journal", "Journal of Robotics", "The Open Electrical & Electronic Engineering Journal".
It's a problem. How do we get people who aren't in a particular field to distinguish between something like PLoS ONE, which is legitimately selective and reliable, but just trying something innovative and new, or the legitimately high quality CS conference proceedings that publish only on arXiv; from journals that accept anyone and publish online, with no barrier to their entry into publishing, or the many people who simply throw their papers onto arXiv and leave things at that?
The internet has many opportunities for academic publishing, but many dangers as well. Academic publishing has relied on the barrier to entry in publishing to filter out incredibly bad journals, or make it clear that they are scams that simply charge high publishing fees; now anyone can create a journal, with low publishing fees, that may be quite legitimate or may be a scam.
Peer review is receiving a bad review, no pun intended. Please remember that good journals exist, and insiders know good from bad. A bad acceptance from a heretofore-unknown journal with a seeming credible title is not indicative that all peer review is flawed or done by impatient baboons.
And I say this having had a terrible time getting a decent review in recent conferences. But my journal review process has been largely positive, helping to improve my paper significantly, even if they miss some things. Certainly, when I receive a paper to review, I and my colleagues give it our best shot, but we largely work only for the established upper-echelon in our field.
while often true, science and scientific journals do not serve solely the scientific community, they serve humanity, which includes the non-professional public. while you may be an honest person and would never knowingly abuse wrong information to harm others, gain, or advance a broken agenda, plenty of people do. the "vaccines lead to autism" people, for example, do that.
the history of science shows that careful, accurate and correctly performed experiments and publications are what give humanity progress. their absence, history has again shown, causes humanity to retreat, sometimes with disastrous consequences.
this is not an idle, "i sure wish more 'scientists' could publish" debate, at some level the very question of human progress is at stake.
Major Comments:
The author make a compelling case that a sizable number of Open Access publishers have potentially flawed review policies that allow substandard work to enter the literature. This is a promising first finding, however the authors then use this data to build a narrative of the nature of Open Access journals without having performed the corresponding experiment with an appropriate control arm.
Without a comparison to conventional publishing, these findings cannot be extended as far as the author does in the conclusions. Such an assertion, without support, is not justified or supported.
This isn't that surprising. It's common knowledge in the academic community that:
* There are bogus open access journals which exist solely to make money from author fees.
* There are bogus conferences that follow a similar business strategy.
* There are lower-quality journals and conferences that do not attract many top-quality submissions, so tend to have a lower barrier for acceptance: you might get work that's rigorous and unexciting, work that's exciting but preliminary or rough around the edges, or work that's neither rigorous nor interesting.
* Peer reviews vary widely in quality, from being thorough
to overly critical and nit-picking to being cursory. This is extremely frustrating to many people, but is hardly an undiscussed issue. One particular problem is that more junior reviewers are often reluctant to criticize a paper unless they're very confident in their understanding, either due to it being slightly outside of their core expertise, or due to convoluted presentation: it's easier to assume it's correct than to go out on a limb criticizing something.
I would say that being published in a peer reviewed article is just the start of a gauntlet that any serious scientific claim has to run, regardless of the quality of peer review. Plenty of important work has started out as being extremely controversial: even once it got published people didn't necessarily think it was right. The consensus does continue to evolve.
I've never met a scientist who will believe something just because it's in a peer reviewed article.
There is a serious ethical problem with submitting dummy papers in that it abuses the trust of the editors and reviewers. You are wasting the time of volunteer reviewers who have nothing to gain but plenty to lose if they screw up. Reviewers assume papers are submitted in good faith, and are not traps.
Editors have practice spotting cranks but trap papers don't have that smell about them since they are written by smart but naughty people trying to trick reviewers.
We read the journals. Many readers of an article are qualified to review it. Many also personally know the editor or associate editors and will raise a stink if the journal starts to suck. We _are_ these journals. We write the articles for free, we review them for free, we proudly share our results. We hate it when a shitty paper gets in.
Edit: There _are_ shitty journals that no one reads. Everyone in the business knows which ones these are. There is a problem explaining this to lay people, since the shitty journal web site doesn't have a sticker on it saying "zero credibility - please ignore these articles". This is not really a problem for researchers.
That is not relevant to the issue of whether or not the peer review process works in deciding what articles to publish. Researchers might be able to identify bad journals and bad articles (though in my experience, quite a few researchers only spend time reading the results), but the point of the review process is to ensure that bad articles are not published in the first place. If the reviewers are allowing bad articles to slip through the cracks, then the process is not working and we need to seriously reconsider the entire journal system (though I would say that the Internet already requires us to reconsider that system).
My real point was that submitting blatantly bad articles serves a purpose, which is to test whether the current process of reviewing articles is effective at weeding out bad articles. We should not blindly assume that journals, even top-tier journals, are only publishing good articles.
If the author is to be believed, the flaws of the paper were numerous, obvious, and fatal. Meaning, easy to spot in five minutes by any reader with a bit of domain knowledge. Those are not subtle traps.
Now, while I understand how journals can assume good faith, they should never assume competence. They are supposed to make sure the paper is good, dammit! If they let that one slip through, they are bound to let genuine fatal mistakes slip through.
No matter what, papers should go through actual peer review before publication. This study, if conducted the way I have read it, is solid evidence that many journals do not do that. They broke trust in the first place.
Anyone whos ever tried to publish. I used to love science before I actually became an academic and realised how dark and corrupt it could be behind the scenes, I suppose the same stands for every field though.
Edit: This looks remarkably like an attempt by Science to smear their open-access competitors, given that it didn't even bother to test whether non-open-access journals have the same flaws (and in fact we know they do, including Science!), yet it's being presented as evidence that open access journals specifically are flawed. I'd hope this article would never survive peer review.