For two decades, researchers using brain-imaging technology have sought to identify how a person’s brain structure and function are linked to a range of mental-health disorders, ranging from anxiety and depression to suicidal tendencies.
But a new paper published Wednesday in Nature raises the question of whether most of this research is actually yielding valid findings. Many such studies, the authors of the paper have found, involved less than two dozen participants, which is shy of the number required to produce reliable results.
“You need thousands of people,” said Scott Merek, a psychiatric researcher at the Washington University School of Medicine in St. Petersburg. Lewis and author of the paper. He described the discovery as a “gut punch” for typical studies using imaging to better understand mental health.
Studies using magnetic-resonance imaging technology typically outrage their findings with a cautious statement noting the small sample size. But registering participants can be time consuming and expensive, ranging from $ 600 to $ 2,000 per hour, Dr. Nico Dosenbeck, a neurologist at the University of Washington School of Medicine and another author on the paper. He added that the average number of subjects in mental-health-related studies using brain imaging is about 23.
But the Nature paper shows that data from just two dozen subjects are generally insufficient to be credible and could in fact give “massively inflated” conclusions, “said Dr. Dosenbach.
For their analysis, the researchers examined three of the largest studies using brain-imaging techniques to draw conclusions about brain formation and mental health. All three studies are ongoing: the Human Connectum project, which has 1,200 participants; Cognitive development of the adolescent brain, or ABCD, study with 12,000 participants; And the UK Biobank study, with 35,700 participants.
The authors of the Nature Paper looked at the subsets of data within those three studies to see if the smaller slices were misleading or “reproducible”, meaning the findings could be scientifically validated.
The ABCD study, for example, looked at whether, among other things, the gray matter thickness of the brain could be related to mental health and problem-solving ability. The authors of the Nature Paper looked at the smaller subsets within the larger study and found that the subsets produced results that were unreliable compared to the results obtained by the complete data set.
The authors, on the other hand, found that when results were generated from a sample size covering several thousand subjects, the findings were similar to those obtained from a complete data set.
The authors performed millions of calculations using different sample sizes and hundreds of brain areas explored in various major studies. Frequently, researchers have found that data subsets of less than a thousand people do not produce results consistent with a complete data set.
Dr. Merek said the paper’s findings apply “completely” outside of mental health. Other fields, such as genomics and cancer research, have their own calculations with small sample size limitations and have tried to improve the course, he noted.
“I think this is more about anthropology than about any one of those areas,” he said.