My Website: petejudo.com
Follow me:
Behavioral Science Instagram: @petejudoofficial
Instagram: @petejudo
Twitter: @petejudo
LinkedIn: Peter Judodihardjo
Good tools I actually use:
Shortform: https://www.Shortform.com/pete
Ground News: https://ground.news/Pete
source
ANYTHING done by humans is subject to all the flaws of human beings: hubris, greed, envy, vengeance, laziness, etc. Whether it’s organized religion (graft and coverups) or governing bodies (Olympic committee corruption anyone?) people are HUMANS and we must always accept our own frailties regardless of the “higher calling”
It is not a broken system. How else is it supposed to be done? There is only so much that can be done for free.
Peer review comes in two parts, the initial screening prior to publication, which only a few people are involved it, and finally the opinion of the greater scientific community ONCE it has been published.
Just because something is published does not mean it is cannon or written in stone. There are lots of papers that are controversial or dismissed as poorly formulated, poorly controled or just plain wrong.
The issue is that lay people seem to think that something being published makes it fact when actually all it is is a record of work that is done which the reader can take or leave according to their personal assessment of the work. If that work is correct, misguided, misinterpreted or wrong is up to the reader to decide, and if they are sufficiently familiar with the field they usually get those assessments right 😂
ok, you presented an issue. What is your solution?
Seems that sometimes Peer review = volunteering so the publisher can make more money.
Deleted my first comment b/c you addressed part of it lol. Great video, looking forward (if possible) to hear some proposed solutions to strengthen the peer review process.
I think we'll live to see the day when peer review is crowd sourced, with a whole economy of tokens backed by the blockchain to incentivise good and honest reviews. You want to submit a paper? That's 100 Research Coins (RC). You get 5 RC for doing a review, and 1 RC for reviewing somebody else's review. If your reviews are well rated by the community, maybe you can get a few more RC.
The biggest problem with peer review in social science right now is how it validates certain entirely problematic discipline.
As an hypothetical example, let's say if you try to publish a study of physical evidence of ghosts, and the paper would never pass peer review in the discipline of physics. That wouldn't be a problem if enough people believe in the BS, and come up with a entirely new different sub-discipline called "Ghostly Physics" or something. Now only experts in Ghostly Physics who believe in the same nonsense can peer review your paper because they are the only experts that can understand. See the problem?
This sounds like such a crazy example, but this is exactly how so many of sub-disciplines and XXX-ism in social sciences get to this level of lack of rigorousness and scrutiny. All you need is quite a small amount (in the grand scheme of things) of academics that believe in it.
Although I agree that all aforementioned problems are true, I believe their scale is over exaggerated. For example, it is true that sometimes you can reliably guess who is the author of double blind paper, it does not happen often. It also worth mentioning that some of the problems are different from domain to domain. For my field of computer science it is virtually impossible to publish in adequate single blind venue.
And then the question is if peer review is so broken, what do you replace it with? Especially if you do not want to make things more expensive. On the other hand there are many known ways to improve peer review.
Peer review is like democracy, it is the most terrible system, except for all the alternatives.
Glad you are covering this topic. It is important to clearly recognize the flaws in the current system. As for bias, it can rarely be completely removed, but we can put systems in place to mitigate the worse of its effects.
Computer graphics has this issue of double blind-ish.
There's not many of us, we kinda know a large chunk of our peers and what they work on.
Especially within a niche.
Some of us kinda also cheat.
They are well known and tweet the crap out of their results when they get them.
Turns out they tweet images that then appear in their articles…
I had that issue of massive delay and waste of time on my work.
A publication I wrote was rejected a first time by someone who clearly hadn't fully read it (citing our limitations described in the article as things we failed to disclose).
months later it was rejected on the premise of "NO DIRECT APPLICATION IN ENGINEERING" (which I wasn't aware was our goal when doing exploratory work).
Also on that second round, reviewers criticized us for adding something that round one criticized us for not having.
And then finally on the third try it got through.
It's taken a year and a half, weeks upon weeks of rewriting to fit templates, countering nit picks of reviewers, writing rebuttals, waiting for answers, etc…
The peer review system more often than not functions at best as an absolute barrier – or at least like a time barrier – to actual peer reading, evaluation and replication. At its worst, the peer-review system has become a space for censorship by rivals and even theft of clues, findings, and opportunities (including careers and awards).
A little beyond the issue of peer review, there are taboo researches that make publication or even research funding unfeasible, as well as connections and almost mandatory statements for the publication of articles or research funding. Not least important, there are "mandatory" conflicts of interest in very expensive surveys, as well as impossibility of independence when certain papers involve hundreds or at least all the most renowned researchers in a certain area or subject.
And we haven't even touched on the problem of bias in favor of positive results or, more than positive, very striking results, a problem that is at the base of the fraud scandal of Francesca Gino, who was only caught because her position led her to be very careless. How many more humble or risk-averse researchers have simply been more careful in their deceptions?
Why isn't data part of the peer review system? The data could be the major factor in the paper to begin with, it would seem to me, data should be one of the most important parts of the review, if in fact data applies to the paper to begin with!
4:09 I have done some peer-review. Not a huge amount but some. I have also published papers that have been peer reviewed. All the peer review I have done or had done on my papers were double blinded. I did not know who was reviewing my paper and I did not know who's research I was reviewing. How is this not the standard across all peer-review systems?
The peer review process has a lot of problems because it is inherently subjective. If we want to continue doing peer review, perhaps we should double blind the process and classify peer reviewers based on readability vs. logic combined with mandatory data submission. Perhaps we should also combine this with adequate compensation for peer review without incentivizing sloppy peer review (getting as many papers done as possible).
I like the open access movement, but there's a major flaw associated with it. Primarily the cost to the authors writing the paper can be incredibly burdensome and/or the target of scams. One journal I tried submitting to charged my PI 2150 Swiss francs or so as a processing fee to make it open access. If the money is not going to the peer review process, I am confused on where the money is going besides staffing costs and server maintenance.
one thing I would also add is that if your paper is controversial or in a controversial subject the odds of it making it through the peer review process drop as well. Scientists are humans too, they can fall prey to the same kinds of bias that any professional can.
I had a peer reviewer once correct Friedrich Schiller's (or at least his translator's) grammar.
The current scientific system was just not made to be used as a career. Science was built from people who dedicated their life to finding the truth and could probably have done a more prestigious job had they wanted it. That culture kind of continued on for a while after science became a career, but I do believe it has been dying out the past 50 or so years and we haven't quite caught up with our culture yet.
Why do you talk about peer "reviewer" in the singular, as if there is only one reviewer? In my field, there are three reviewers minimum. That entirely dismantles your premise about subjectivity. Bad, poorly constructed argument.
Wait wait wait, then you talk about bias as if DOUBLE BLIND REVIEW is not a thing? You're absolutely dead wrong and spreading misinformation at this point.
Peer review is a very new part of science, and it was historically only used for biomedical research. now it's used to sell more journals.
I think it is worth pointing out that these three categories of problems reinforce one another.
12 minutes of worthwhile analysis but only a graphic and a comment on potential future videos on alternatives? Peer review is at least some form of review. I hope those follow-up better solution videos come soon. Otherwise this is ammo for the anti-science crowd.
It's so wild to me that providing the data is optional during review of a paper that uses data to make a conclusion. A review without checking the data itself couldn't possibly be reliable, no matter how knowledgeable and fair the reviewer is.
One time I used a website to research academic papers in the U.S, and every file had unreadable, broken English and meaningless graphs written by Chinese students. I don't know who to blame. I couldn't find the resources I needed.
critique could be incentivised! especially if it is deconstructive and in an accommodating way of the structures of the presentor/author .. moreover constructive.
I once had a business man tell me taxes audits are good for the department. Trains employees and attunes the system….
critique can be practical doing something in the world. Our public discourse is so muddled we can't seem to ground our information into social knowledge circuits, not enmasse. It seems obvious academia in general is orientated in a liberal human subject perspective and doesn't acknowledge that!?! A big fish but not the territory nor even the only fish.
Grounding requires finding a common ground. That's tough to do to be fair, but i feel like that's a constant feedback loop requirement for social organization. We have many many many ways to measure and validate narratives for humans to participate politically, socially, etc. consentual as it happens moral codevelopment. What stories/Circuits did the research become of, where's it too. Function and telos!
As we make practical and connect dots. Peer review and critique could be shown to make systems useful in a direct way. it gets more complicated, so map it!
Francesca Gino's work was obviously suspect. Amazing she is still on full pay.
The peer review process need to be… Reviewed
One other problem: some reviewers tend to suggest to cite their papers to bribe acceptance.
My colleague papers shared me the review summary for one of his papers where the reviewer asked him to cite a bunch of papers with a common author in the first review and a few more in the revision – many of which is not even related to his work. Talk about unethical practice.
Abolish this crap.
Yes, I'd love to hear more about alternatives. I'd also like to know if there are any discussions on how to fix peer review. Maybe peer reviewers should get compensated somehow? Some kind of incentive would be helpful. Finding some way to alleviate the stress of ceaselessly competing for funds would be nice too.
Nice video, I agree. But a peer review taking years? Are you sure? Even going through 2 or 3 different peer reviews would still take less than a year, at least in psychology.
As an additional argument you could use the Sokal Squared experiment as proof that in several academic fields peer review works against objective evidence. In these cases, the more the article adheres to field orthodoxy and pushes this further, the more chance of publication and positive peer review.
I once wrote a review on non NRTI treatment regimens and alternative regimens for HIV. The peer reviewers were so biased against it, they pretty much said, "aside from these trials showing no change in viral load, where is the proof". I just moved on and didn't want to deal with it as I wasn't in academia and didn't care about a publication; I just wanted the concise information out there. Fast forward 3 years later and someone else notices that there is a missing review on the same topic and writes similar thing. I think it was the Lancet that picked it up.
Glad it was published and really only bitter to the peers that prevented this info that could have helped out patients years earlier.
You do great work!
Our current system chokeholds our ability to evolve. The incentive structure is all crooked from bottom-up. As a person who has experienced some of what academia has to offer, for most places innovation and advancement of knowledge almost happens despite of academia than because of it. You can't have a profit motive and do science it's simple as that. You can't have corporations dictate research. These are things a toddler can understand. You can't have departments close down because it isn't beneficial for the market. Scientific progress isn't a linear progress nor can it be derived from coercion. Of course huge developments do happen and that is because of the greatness of human ability however "huge" is only relative to what we find possible under this regime.
Peer review is unpaid and unmanaged! Editors have no power!
It's funny because these are problems that all people have. Here it's scientists, other times it's people who make burgers. We're never gonna escape bias, because we won't put ourselves second.
"peer-review" means "my friends took a look at it"
If anything blind review makes things worse. If someone is keeping a legit paper from passing review, their name should be known
Thanks for this video; I am in the Economics field; just a few comments: in most Econ journals, you would not be able to review a paper from someone from the same institution. In principle one should reveal conflicts of interests. Also, I don't think that peer review is a "full tax" on the reviewer. You usually get to evaluate papers you should probably be reading anyway. I usually enjoy reading other people's work and learning from it. And I think many people do (I am not as cynical…!) And if anything, it probably ensures that papers by young researchers in unknown institutions get feedback they would have a hard time getting otherwise. What I really find troubling at the moment is the relatively recent trend of journals to "desk reject" about half of the manuscripts they receive. That gives considerable power to editors, and given that they have many papers on their desk, the 'signal' is even noisier. As an editor, I try to do all I can to implement the system as well as I can. But you are right that the system is noisy, and wasteful because one usually needs to go through several rounds of reviews before publishing. But it is not easy at all to come up with a good alternative… Academia also has some good apples. I know the recent news is depressing, but please keep some hope!
YES! This is the video I've been waiting for/wanting from you! Thank you!