Question 1 Peer Review and Replication Are Important for the Progression of Science True False

Much ink has been spilled over the "replication crisis" in the final decade and a half, including hither at Vox. Researchers have discovered, over and over, that lots of findings in fields like psychology, sociology, medicine, and economic science don't hold up when other researchers try to replicate them.

This conversation was fueled in function by John Ioannidis's 2005 article "Why Most Published Research Findings Are False" and by the controversy around a 2011 paper that used then-standard statistical methods to observe that people have precognition. But since and so, many researchers have explored the replication crisis from different angles. Why are inquiry findings so often unreliable? Is the problem simply that nosotros examination for "statistical significance" — the likelihood that similarly strong results could accept occurred past chance — in a dash-gratuitous style? Is information technology that null results (that is, when a study finds no detectable furnishings) are ignored while positive ones brand it into journals?

A recent write-up past Alvaro de Menard, a participant in the Defense Advanced Research Project's Bureau'south (DARPA) replication markets project (more than on this below), makes the case for a more depressing view: The processes that atomic number 82 to unreliable research findings are routine, well understood, predictable, and in principle pretty like shooting fish in a barrel to avoid. And yet, he argues, nosotros're however not improving the quality and rigor of social science research.

While other researchers I spoke with pushed back on parts of Menard's pessimistic have, they do agree on something: a decade of talking almost the replication crunch hasn't translated into a scientific process that'south much less vulnerable to it. Bad scientific discipline is still frequently published, including in top journals — and that needs to alter.

Most papers neglect to replicate for totally predictable reasons

Let'south take a pace back and explain what people mean when they refer to the "replication crisis" in scientific research.

When inquiry papers are published, they describe their methodology, so other researchers can re-create it (or vary information technology) and build on the original research. When another research team tries to deport a study based on the original to see if they detect the aforementioned result, that's an attempted replication. (Often the focus is not but on doing the verbal same thing, only approaching the same question with a larger sample and preregistered design.) If they find the same upshot, that'due south a successful replication, and evidence that the original researchers were on to something. Only when the attempted replication finds different or no results, that often suggests that the original research finding was spurious.

In an attempt to test just how rigorous scientific enquiry is, some researchers accept undertaken the chore of replicating inquiry that's been published in a whole range of fields. And every bit more and more of those attempted replications have come back, the results take been hit — it is not uncommon to find that many, many published studies cannot be replicated.

One 2015 attempt to reproduce 100 psychology studies was able to replicate simply 39 of them. A big international effort in 2018 to reproduce prominent studies plant that 14 of the 28 replicated, and an effort to replicate studies from summit journals Nature and Science constitute that xiii of the 21 results looked at could be reproduced.

The replication crisis has led a few researchers to enquire: Is at that place a mode to guess if a paper volition replicate? A growing body of research has constitute that guessing which papers will concur up and which won't is often just a affair of looking at the same elementary, straightforward factors.

A 2019 newspaper by Adam Altmejd, Anna Dreber, and others identifies some simple factors that are highly predictive: Did the study have a reasonable sample size? Did the researchers clasp out a result barely below the significance threshold of p = 0.05? (A paper tin can often claim a "significant" result if this "p" threshold is met, and many use various statistical tricks to push their paper across that line.) Did the study find an effect across the whole study population, or an "interaction effect" (such every bit an issue only in a smaller segment of the population) that is much less likely to replicate?

Menard argues that the trouble is not so complicated. "Predicting replication is piece of cake," he said. "There's no need for a deep dive into the statistical methodology or a rigorous exam of the information, no need to scrutinize esoteric theories for subtle errors — these papers accept obvious, surface-level bug."

A 2018 study published in Nature had scientists place bets on which of a pool of social science studies would replicate. They found that the predictions by scientists in this betting market were highly authentic at estimating which papers would replicate.

Colin F. Camerer et al./Nature

"These results propose something systematic about papers that fail to replicate," written report co-author Anna Dreber argued later the written report was released.

Additional research has established that you don't even demand to poll experts in a field to estimate which of its studies will agree up to scrutiny. A study published in August had participants read psychology papers and predict whether they would replicate. "Laypeople without a professional background in the social sciences are able to predict the replicability of social-science studies with above-chance accuracy," the study concluded, "on the footing of nothing more than unproblematic verbal report descriptions."

The laypeople were not as accurate in their predictions as the scientists in the Nature study, but the fact they were still able to predict many failed replications suggests that many of them have flaws that even a layperson can notice.

Bad science can still be published in prestigious journals and be widely cited

Publication of a peer-reviewed paper is not the final step of the scientific process. Subsequently a paper is published, other research might cite it — spreading any misconceptions or errors in the original newspaper. But research has established that scientists accept skilful instincts for whether a paper will replicate or not. And so, do scientists avoid citing papers that are unlikely to replicate?

This hitting chart from a 2020 study by Yang Yang, Wu Youyou, and Brian Uzzi at Northwestern University illustrates their finding that actually, at that place is no correlation at all betwixt whether a study will replicate and how often information technology is cited. "Failed papers circulate through the literature as quickly equally replicating papers," they argue.

Yang Yang, Wu Youyou, and Brian Uzzi/PNAS

Looking at a sample of studies from 2009 to 2017 that accept since been subject to attempted replications, the researchers discover that studies have near the aforementioned number of citations regardless of whether they replicated.

If scientists are pretty expert at predicting whether a newspaper replicates, how can information technology be the case that they are as likely to cite a bad newspaper as a good one? Menard theorizes that many scientists don't thoroughly check — or even read — papers one time published, expecting that if they're peer-reviewed, they're fine. Bad papers are published by a peer-review process that is not acceptable to catch them — and once they're published, they are not penalized for beingness bad papers.

The contend over whether we're making whatsoever progress

Hither at Vox, nosotros've written nearly how the replication crunch tin can guide usa to do improve science. And even so blatantly shoddy work is still being published in peer-reviewed journals despite errors that a layperson can see.

In many cases, journals effectively aren't held accountable for bad papers — many, like The Lancet, have retained their prestige even later on a long string of embarrassing public incidents where they published inquiry that turned out fraudulent or nonsensical. (The Lancet said recently that, after a study on Covid-nineteen and hydroxychloroquine this spring was retracted after questions were raised about the data source, the journal would change its information-sharing practices.)

Fifty-fifty outright frauds often take a very long fourth dimension to be repudiated, with some universities and journals dragging their feet and declining to investigate widespread misconduct.

That's discouraging and infuriating. It suggests that the replication crisis isn't one specific methodological reevaluation, but a symptom of a scientific system that needs rethinking on many levels. We tin can't simply teach scientists how to write better papers. We also demand to change the fact that those amend papers aren't cited more than often than bad papers; that bad papers are about never retracted even when their errors are visible to lay readers; and that in that location are no consequences for bad enquiry.

In some means, the culture of academia actively selects for bad research. Force per unit area to publish lots of papers favors those who tin put them together apace — and one way to be quick is to be willing to cut corners. "Over time, the most successful people will be those who can best exploit the arrangement," Paul Smaldino, a cognitive science professor at the University of California Merced, told my colleague Brian Resnick.

So we accept a system whose incentives keep pushing bad enquiry even as we understand more than about what makes for skilful research.

Researchers working on the replication crisis are more than divided, though, on the question of whether the last decade of work on the replication crunch has left usa meliorate equipped to fight these issues — or left us in the same place where we started.

"The future is vivid," concludes Altmejd and Dreber'southward 2019 paper about how to predict replications. "There will exist rapid accumulation of more than replication data, more outlets for publishing replications, new statistical techniques, and—nearly importantly—enthusiasm for improving replicability among funding agencies, scientists, and journals. An exciting replicability 'upgrade' in science, while perhaps overdue, is taking identify."

Menard, by contrast, argues that this optimism has not been borne out — none of our improved understanding of the replication crisis leads to more papers beingness published that actually replicate. The project that he's a office of — an effort to design a better model to predict which papers replicate run by DARPA in the Defense Section — has non seen papers grow whatsoever more likely to replicate over time.

"I oftentimes see the notion that after the replication crisis hitting there was some sort of peachy improvement in the social sciences, that people wouldn't even dream of publishing studies based on 23 undergraduates whatever more ... In reality in that location has been no discernible improvement," he writes.

Researchers who are more optimistic point to other metrics of progress. It'due south true that papers that neglect replication are yet extremely common, and that the peer-review process hasn't improved in a way that catches these errors. Simply other elements of the fault-correction process are getting better.

"Journals at present retract about i,500 articles annually — a virtually twoscore-fold increase over 2000, and a dramatic change even if you account for the roughly doubling or tripling of papers published per twelvemonth," Ivan Oransky at Retraction Watch argues. "Journals have improved," reporting more details on retracted papers and improving their process for retractions.

Other changes in mutual scientific practices seem to be helping too. For example, preregistrations — announcing how you lot'll conduct your assay before you do the study — pb to more zippo results being published.

"I don't think the influence [of public conversations about the replication crisis on scientific practise] has been cypher," statistician Andrew Gelman at Columbia University told me. "This crisis has influenced my ain inquiry practices, and I assume it's influenced many others also. And information technology'south my general impression that journals such as Psychological Science and PNAS don't publish as much junk as they used to."

There's some reassurance in that. But until those improvements translate to a higher percentage of papers replicating and a difference in citations for skillful papers versus bad papers, information technology'south a small victory. And it's a pocket-sized victory that has been hard-won. Later on tons of resource spent demonstrating the scope of the trouble, fighting for more than retractions, teaching better statistical methods, and trying to drag fraud into the open, papers even so don't replicate as much as researchers would hope, and bad papers are still widely cited — suggesting a large part of the problem still hasn't been touched.

We need a more sophisticated understanding of the replication crunch, not as a moment of realization after which nosotros were able to movement forward with college standards, but as an ongoing rot in the scientific process that a decade of piece of work hasn't quite stock-still.

Our scientific institutions are valuable, as are the tools they've built to assist us empathise the world. There's no crusade for hopelessness here, even if some frustration is thoroughly justified. Science needs saving, sure — but scientific discipline is very much worth saving.

floreswastures.blogspot.com

Source: https://www.vox.com/future-perfect/21504366/science-replication-crisis-peer-review-statistics

Share this post