An important paper recently published by an esteemed Stanford research team reported an unusual result: An experiment went wrong.
Usually, scientists seek to burnish their reputations by announcing positive news of a discovery that solves a problem or transforms how we view the world.
But this negative news — which revealed that earlier neuroscience research was flawed — can also be positive. It builds a stronger scientific foundation and helps restore public trust, according to a growing consensus of scientists and journal editors.
Faced with a disturbing flurry of experiments that can’t be reproduced, academia is ramping up efforts to “future proof” its research.
While headlines are dominated by fraud or research misconduct cases, including a scandal that led to the resignation of Stanford University President Dr. Marc Tessier-Lavigne, these instances are relatively rare. A bigger problem is experimentation that lacks robust design, methodology, analysis and interpretation of results — so arrives at the wrong conclusions.
“Our efforts highlight the importance of experimental rigor,” said Stanford postdoctoral neuroscientist Kif Liakath-Ali, who conducted the work with Nobel Laureate Thomas Südhof.
His revelation — that sometimes a negative can be a positive — came while he was trying to reproduce and build upon a 2017 study about the behavior of brain cells. He wanted to understand the regulation of brain cells, with major implications for memory, behavior and neurological disease. He discovered that the previous approach in the lab had killed cells, leading to “a skewing of results and biased conclusions,” he said.
It was a professional setback for Liakath-Ali, who had aimed to build on this research to make a new and meaningful discovery.
A junior scientist, he worried that his insight, based on almost two years of work, would not advance his career. Instead, Liakath-Ali has been honored by the School of Medicine’s new Program on Research Rigor and Reproducibility with an award for his integrity.
His finding has emboldened other research teams to come forward to describe their own failed attempts, he said. Although those teams had stayed silent, “they had seen the same thing.”
“That’s what good science is about,” said Dr. Steven Goodman, who leads Stanford’s Program on Research Rigor and Reproducibility. “It detected that some really important findings … were just wrong.”
“We want to reward how people do science,” he said, “and if they do it better than the last person.”
Science is famed for its “Eureka” moments. We love the tale of Scottish microbiologist Alexander Fleming, who came home from vacation to discover a mold producing penicillin, the world’s first antibiotic, growing inside a neglected Petri dish..
Real experimentation takes many twists and turns and doesn’t always deliver the expected outcome.
“I have not failed,” inventor Thomas Edison famously said. “I’ve just found 10,000 ways that won’t work.”
But modern science is competitive. In today’s “publish or perish” academic culture, careers are advanced by insights, so scientists are incentivized to announce only positive findings.
In the worst cases, this can foster fraudulent or sloppy practices. Tessier-Lavigne resigned his post after an independent review found multiple errors in five papers he had overseen, concluding that “multiple members of (his) labs over the years appear to have manipulated research data and/or fallen short of accepted scientific practices.”
Journals also favor papers that are “hot,” with impact that will be widely cited and elevate a journal’s reputation.
“A challenge for scientists has been that (experimental) repetitions are difficult to publish, no matter whether they are positive or negative,” said Südhof, who won the 2013 Nobel Prize in Physiology or Medicine. “This is bad because it creates a disincentive for repeating experiments.”
Journals are trying to change, said Holden Thorp, editor of the prominent journal Science, at last week’s Stanford conference on research integrity. “It’s a very, very challenging problem, because all of the emphasis is on novelty and ‘being first…We don’t take a lot of papers that say, ‘We tested this hypothesis and we found that it’s still correct.’”
But repetition may reveal problems, and “ensures that people have the full story,” said Emily Chenette, editor-in-chief of PLOS ONE, published by the Public Library of Science, which evaluates research on scientific validity, methodology and ethical standards — not perceived significance. PLOS ONE publishes a collection called “Missing Pieces,” which lists studies that present inconclusive, null findings or demonstrate failed replications of other published work.
A negative finding can suggest promising new directions, approaches and hypotheses. It may warn other investigators to steer clear, saving time and money, Chenette said.
“It has real life implications for people,” she said.
Early in the COVID-19 pandemic, use of the malaria drug hydroxychloroquine was spurred by anecdotal reports from China and France of patients who seemed to improve and laboratory findings of a possible antiviral effect. But a rigorous study found that the drug didn’t work — a discovery that saved many lives.
In the prestigious British journal Lancet, a doctor linked vaccines to autism, a claim that has led to clusters of resistance to inoculation. It was refuted by multiple studies, and a subsequent investigation showed his work to be bunk.
This week, in a surprise announcement, a precious 280-million-year-old fossilized lizard turned out to be mostly … black paint. An Italian team had hoped to make history by using high-tech tools — electron microscopy, spectroscopy and micro x-rays — to reveal the cellular structure of one of the world’s oldest reptiles. Instead, they found forgery. But their revelation could lead to a rethinking of ancient taxonomy.
Stanford’s Liakath-Ali sought to better understand how brain cells, called neurons, communicate via trillions of synapses — and how things go wrong. Synapses connect using a vast network of molecules, governed by genes whose function may change if subjected to stress, causing devastating ailments like schizophrenia, autism and other neurological disorders.
He based his work on a 2017 report by scientists from China’s Tsinghua University published in the journal Nature Neuroscience. They had found that when we learn something, brain cells change in a way that helps us remember it better. But brain cells’ regulatory mechanism can be altered by stress.
He took a closer look at the Chinese research and found fault with their technique, which caused the cells to be so stressed — “hammered,” he said — that they died. This skewed their results.
“Liakath-Ali did what no one else had done: He took the care to look at the cells,” said Goodman.
Nobel Laureate Südhof commended his perseverance.
“Science operates by a trial-and-error process in which scientists, like all other humans, also make mistakes,” he said. “To distinguish valid results from erroneous ones, it is necessary to repeat experiments independently.”
