What next? Reproducibility in research

First thing: True reproducibility is basically impossible

Better: “reproducibility at dissemination / archiving” or “inspectability” (1)

How can we check reproducibility if no code is given?

Possible role models as research groups: Jeff Leek and Ben Marwick. Or Steno Aarhus’ GitHub account!

But that doesn’t even matter, because we can’t have reproducibility if no one shares their code!

And this is no joke. Getting data on this is difficult, but the research that has been done shows that almost no one is sharing their code. The estimates range between fields in health science from zero to maybe five percent of published studies. The only area that is doing pretty well is bioinformatics, at about 60% of published studies.

We as researchers try to find our niche, make our own space in the research world. Sometimes its a real struggle to find that niche.. but guys! No one is doing this, no one is sharing their code! You start doing the simplest thing of sharing your code and you will be one of very very few people who do. And this isn’t a niche, this is a gaping hole in our modern scientific process. A huge hole.

Multiple benefits, from personal to philosophical

It’s a core principle of the scientific method: Verification

Learning more from others: For PhD students to senior researchers

More exposure and visibility: More output to show and be seen

So few are doing open science, this is a great niche!

Easier and quicker collaboration (aside from the learning part)

Finding better opportunities outside of academia

Strong instutional barriers, such as …

Lack of adequate awareness, support, infrastructure, training

Research culture values publications over all else

What would you spend your time on if we didn’t have this publication-obsession?

Strong personal barriers like …

Fear of …

Fear of being scooped or ideas being stolen
Errors and public humiliation

Overwhelmed with everything that we should do better

Need to constantly stay updated

So… what you can do right now to be more open and reproducible?

Follow some core principles

Use open source tools wherever possible
Use plain text as often as possible
Upload and share publicly early and often (e.g. to GitHub or Zenodo)
Upload and share publicly as many things as possible
Archive to get a DOI/version for major milestones

Teach others!

It’s also a great way to learn 😉 😉

References

Gil Y, David CH, Demir I, Essawy BT, Fulweiler RW, Goodall JL, et al. Toward the geoscience paper of the future: Best practices for documenting and sharing research from data to software to provenance. Earth and Space Science. 2016 Oct;3(10):388–415.

Considine EC, Thomas G, Boulesteix AL, Khashan AS, Kenny LC. Critical review of reporting of the data analysis step in metabolomics. Metabolomics. 2017 Dec;14(1).

Rauh S, Torgerson T, Johnson AL, Pollard J, Tritz D, Vassar M. Reproducible and transparent research practices in published neurology research. bioRxiv. 2019 Sep;

Evans S, Fladie IA, Anderson JM, Tritz D, Vassar M. Evaluation of reproducible and transparent research practices in sports medicine research: A cross-sectional study. bioRxiv. 2019 Sep;

Rauh SL, Johnson BS, Bowers A, Tritz D, Vassar BM. Evaluation of reproducibility in urology publications. bioRxiv. 2019 Sep;

Hughes T, Niemann A, Tritz D, Boyer K, Robbins H, Vassar M. Transparent and reproducible research practices in the surgical literature. bioRxiv. 2019 Oct;

Peng RD, Dominici F, Zeger SL. Reproducible epidemiologic research. American Journal of Epidemiology. 2006 Mar;163(9):783–9.

Seibold H, Czerny S, Decke S, Dieterle R, Eder T, Fohr S, et al. A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses. Wicherts JM, editor. PLOS ONE. 2021 Jun;16(6):e0251194.