What next? Reproducibility in research

First thing: True reproducibility is basically impossible

Better: “reproducibility at dissemination / archiving” or “inspectability” (Gil et al. 2016)

How can we check reproducibility if no code is given?

But that doesn’t even matter, because we can’t have reproducibility if no one shares their code!

And this is no joke. Getting data on this is difficult, but the research that has been done shows that almost no one is sharing their code. The estimates range between fields in health science from zero to maybe five percent of published studies. The only area that is doing pretty well is bioinformatics, at about 60% of published studies.

We as researchers try to find our niche, make our own space in the research world. Sometimes its a real struggle to find that niche.. but guys! No one is doing this, no one is sharing their code! You start doing the simplest thing of sharing your code and you will be one of very very few people who do. And this isn’t a niche, this is a gaping hole in our modern scientific process. A huge hole.

Multiple benefits, from personal to philosophical

It’s a core principle of the scientific method: Verification

Learning more from others: For PhD students to senior researchers

So few are doing open science, this is a great niche!

Easier and quicker collaboration (aside from the learning part)

(especially when done through GitHub)

Finding better opportunities outside of academia

Strong institutional barriers, such as …

Lack of adequate awareness, support, infrastructure, training

Research culture values publications over all else

What would you spend your time on if we didn’t have this publication-obsession?

Strong personal barriers like …

Fear of being scooped or ideas being stolen

Overwhelmed with everything that we should do better

Strongly rewarded to get things done, not get things right.

Need to constantly stay updated

So… what you can do right now to be more open and reproducible?

Follow some core principles

Use open source tools wherever possible
Use plain text as often as possible
Upload and share publicly early and often (e.g. to GitHub or Zenodo)
Upload and share publicly as many things as possible

Teach others!

It’s also a great way to learn 😉 😉

References

Considine, E. C., G. Thomas, A. L. Boulesteix, A. S. Khashan, and L. C. Kenny. 2017. “Critical Review of Reporting of the Data Analysis Step in Metabolomics.” Metabolomics 14 (1). https://doi.org/10.1007/s11306-017-1299-3.

Evans, Sheridan, Ian A. Fladie, J. Michael Anderson, Daniel Tritz, and Matt Vassar. 2019. “Evaluation of Reproducible and Transparent Research Practices in Sports Medicine Research: A Cross-Sectional Study.” bioRxiv, ahead of print, September. https://doi.org/10.1101/773473.

Gil, Yolanda, Cédric H. David, Ibrahim Demir, et al. 2016. “Toward the Geoscience Paper of the Future: Best Practices for Documenting and Sharing Research from Data to Software to Provenance.” Earth and Space Science 3 (10): 388–415. https://doi.org/10.1002/2015ea000136.

Hughes, Taylor, Andrew Niemann, Daniel Tritz, Kryston Boyer, Hal Robbins, and Matt Vassar. 2019. “Transparent and Reproducible Research Practices in the Surgical Literature.” bioRxiv, ahead of print, October. https://doi.org/10.1101/779702.

Peng, Roger D., Francesca Dominici, and Scott L. Zeger. 2006. “Reproducible Epidemiologic Research.” American Journal of Epidemiology 163 (9): 783–89. https://doi.org/10.1093/aje/kwj093.

Rauh, Shelby Lynn, Bradley S. Johnson, Aaron Bowers, Daniel Tritz, and Benjamin Matthew Vassar. 2019. “Evaluation of Reproducibility in Urology Publications.” bioRxiv, ahead of print, September. https://doi.org/10.1101/773945.

Rauh, Shelby, Trevor Torgerson, Austin L. Johnson, Jonathan Pollard, Daniel Tritz, and Matt Vassar. 2019. “Reproducible and Transparent Research Practices in Published Neurology Research.” bioRxiv, ahead of print, September. https://doi.org/10.1101/763730.

Seibold, Heidi, Severin Czerny, Siona Decke, et al. 2021. “A Computational Reproducibility Study of PLOS ONE Articles Featuring Longitudinal Data Analyses.” PLOS ONE 16 (6): e0251194. https://doi.org/10.1371/journal.pone.0251194.