What next? Reproducibility in research

First thing: True reproducibility is basically impossible

Better: “reproducibility at dissemination / archiving” or “inspectability” (1)

How can we check reproducibility if no code is given?

Possible role models as research groups: Jeff Leek and Ben Marwick.

But that doesn’t even matter, because we can’t have reproducibility if no one shares their code!

And this is no joke. Getting data on this is difficult, but the research that has been done shows that almost no one is sharing their code. The estimates range between fields in health science from zero to maybe five percent of published studies. The only area that is doing pretty well is bioinformatics, at about 60% of published studies.

We as researchers try to find our niche, make our own space in the research world. Sometimes its a real struggle to find that niche.. but guys! No one is doing this, no one is sharing their code! You start doing the simplest thing of sharing your code and you will be one of very very few people who do. And this isn’t a niche, this is a gaping hole in our modern scientific process. A huge hole.

Not going to lie, there are very strong…

Institutional barriers

Lack of adequate awareness, support, infrastructure, training

Research culture values publications over all else

More traditional academics don’t understand or resist change

‘Business as usual’ is easier

You will encounter a lot of resistance, a lot of barriers and hardship.

At the institutional level, there is no real awareness of this, no support or infrastructure. You’re basically doing this on your own. Which probably isn’t that uncommon anyway.

Research culture and incentives pretty much only care about publishing journal articles. Creating software tools, meh. Making teaching materials to help other researchers, meh. Communicating your science to the public, meh. Doing actual science that might take years and not lead to any “hard papers”, meh.

We have a large portion of traditional academics who have benefitted from and succeeded in this system and are invested in continuing it. Probably because they don’t understand the scope of the problem or just resist change.

We have a system that favours each individual person repeating the same mistakes that others make because the system doesn’t allow for us to take the time to create tools and infrastructure that helps ourselves and others out.

Because business as usual is the easiest way in the short term. Our current scientific culture is just not prepared for this, for the rising modern analytic and computational era.

…and personal barriers

Fear of:
- Fear of being scooped or ideas being stolen
- Not being credited for ideas
- Errors and public humiliation
- Risk to reputation

Need to constantly stay updated
Finding better opportunities outside of academia

And there aren’t just institutional barriers. We as researchers have fears of being scooped, of embarrassment and humiliation for your methods being gasp wrong. Which is actually just part of science.

You also have to constantly stay updated, and that can be tiring.

And the last barrier, which may actually be a benefit, is that one reason you don’t see a lot of researchers sharing their code or being more reproducible is.. they end up getting picked up by industry and paid really well or decide to leave academia for the reasons I mentioned.

Just as an example, I found a Norwegian group who had a really inefficient workflow and decided to re-build their workflow to make use of programming, to be reproducible, to have a pipeline. I looked up the lead author as well as several other of the co-authors and guess what… many of them now work in really great companies as data scientists or software engineers, probably making a lot of money and having potentially a less stressful life.

So… what you can do right now?

What else can you do?

Find or start building a community of people using R
Start doing code reviews within your group
Start new projects or collaborations by:
- Using R, Quarto / R Markdown, Git, and GitHub
- Aiming to share and be reproducible

And teaching others is a great way to learn 😉

References

Gil Y, David CH, Demir I, Essawy BT, Fulweiler RW, Goodall JL, et al. Toward the geoscience paper of the future: Best practices for documenting and sharing research from data to software to provenance. Earth and Space Science. 2016 Oct;3(10):388–415.

Considine EC, Thomas G, Boulesteix AL, Khashan AS, Kenny LC. Critical review of reporting of the data analysis step in metabolomics. Metabolomics. 2017 Dec;14(1).

Rauh S, Torgerson T, Johnson AL, Pollard J, Tritz D, Vassar M. Reproducible and transparent research practices in published neurology research. bioRxiv. 2019 Sep;

Evans S, Fladie IA, Anderson JM, Tritz D, Vassar M. Evaluation of reproducible and transparent research practices in sports medicine research: A cross-sectional study. bioRxiv. 2019 Sep;

Rauh SL, Johnson BS, Bowers A, Tritz D, Vassar BM. Evaluation of reproducibility in urology publications. bioRxiv. 2019 Sep;

Hughes T, Niemann A, Tritz D, Boyer K, Robbins H, Vassar M. Transparent and reproducible research practices in the surgical literature. bioRxiv. 2019 Oct;

Peng RD, Dominici F, Zeger SL. Reproducible epidemiologic research. American Journal of Epidemiology. 2006 Mar;163(9):783–9.

Seibold H, Czerny S, Decke S, Dieterle R, Eder T, Fohr S, et al. A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses. Wicherts JM, editor. PLOS ONE. 2021 Jun;16(6):e0251194.