Reproducibility
Subject librarians can work with individuals, research groups, and departments to advise on many aspects of reproducibility and rigor. If you have any questions reach out to the subject librarian for your area.
Reproducibility is a core principle of scientific progress. Scientific claims should not gain credence because of the status or authority of their originator but by the replicability of their supporting evidence.
Services for reproducible research
The University Libraries support students and researchers across the University of Minnesota and have many experts and services that can support research reproducibility and rigor across the research lifecycle.
The library has subject librarians for each discipline who can
- help you find reporting guidelines,
- help you manage your data and understand data sharing requirements and options,
- find repositories for pre-registering studies and analysis plans, and
- answer more discipline specific questions.
We are also happy to present in classes and workshops on this topic.
Consult with a subject librarian
Methods
Find and use reporting guidelines
Reporting guidelines provide specific instructions for what you need to report about your methodology so others can evaluate and reproduce your work.
There are reporting guidelines for all type of qualitative and quantitative research.
Equator Network A comprehensive database of reporting guidelines and links to other resources relevant to research reporting.Research data management
Data management improves the consistency and rigor of your research data so that when you report your research others can understand and interpret it.
Manage your data Consult with a data management expert at the Libraries, or request a training session.Systematic review support
Librarians have extensive experience conducting systematic reviews, and the involvement of librarians has been shown to improve the quality and reproducibility of systematic reviews.
Systematic reviews and evidence synthesis Evidence synthesis reviews encompass systematic reviews, scoping reviews, meta-analyses, and evidence gap maps. These research methods include a reproducible and transparent methodology.Reporting and dissemination
Tools for pre-registration
Pre-registering your studies and analysis plans ensures that your study can be found and readers can differentiate exploratory from confirmatory research.
Clinicaltrials.gov ClinicalTrials.gov is a database of privately and publicly funded clinical studies conducted around the world. Open Science Framework (OSF) OSF is a free and open source project management tool that supports researchers throughout their entire project lifecycle.Reproducibility
The ability to verify results
Curating and sharing your research data, code, and materials, means others can reproduce your results.
Data Repository for the University of Minnesota (DRUM) DRUM is a publicly available collection of digital research data generated by U of M researchers, students, and staff. Anyone can search and download the data housed in the repository, instantly or by request.Definitions
Reproducibility
The ability of a researcher to duplicate the results of a prior study using the same materials and procedures as were used by the original investigator.
In this “replication crisis” era, reproducibility is the only thing that can be effectively guaranteed in a published study. Whether any claimed findings are indeed true or false can only be confirmed via additional studies, but reproducibility can be confirmed immediately.
Replicability
The ability of a researcher to duplicate the results of a prior study if the same procedures are followed but new data are collected.
Guidelines
- Transparency and Openness Promotion (TOP) Guidelines: The Transparency and Openness Promotion (TOP) Guidelines (COS, 2014) are a set of 8 standards intended to be adopted by academic journals to improve reproducibility through transparency. The TOP guidelines is endorsed by over 2900 journals and organizations.
- National Institute of Health (NIH): Rigor and Reproducibility: Principles and Guidelines for Reporting Preclinical Research: A set of 5 guidelines for reporting preclincal research.
- National Science Foundation (NSF): A Framework for Ongoing and Future National Science Foundation Activities to Improve Reproducibility, Replicability, and Robustness in Funded Research (PDF)
- American Statistical Association (ASA): Recommendations to Funding Agencies for Supporting Reproducible Research (PDF)
- Federation of American Societies for Experimental Biology: Enhancing Research Reproducibility (PDF)
Other campus resources
Liberal Arts Technologies and Innovation Services (LATIS)
LATIS provides training, consultation, and direct support for a variety of research methodologies and tools. Their goal is to help faculty and graduate students take steps towards more reproducible workflows and better managed data wherever they are in research life cycle.
LATIS also offers a workshop series on reproducible research tools that are free and open to graduate students and faculty.
Minnesota Supercomputing Institute (MSI)
Jupyter is an interactive electronic notebook for any computations in R, Python or other languages.
Stratus is a local research compute cloud environment that will enable you to create exact cloned images of your production workflows.
Galaxy enables extensive point-and-click genomics analyses within your browser, where all histories of data analysis are carefully logged, tracked, and easily reproduced from start to finish.
References
Broman, K., Cetinkaya-Rundel, M., Nussbaum, A., Paciorek, C., Peng, R., Turek, D., & Wickham, H. (2017). Recommendations to Funding Agencies for Supporting Reproducible Research. American Statistical Association.
Bollen, K., Cacioppo, J., Kaplan, R., Krosnick, J. A., & Olds, J. L. (2015). Social , Behavioral , and Economic Sciences Perspectives on Robust and Reliable Science. Report of the Subcommittee on Replicability in Science Advisory Committee to the National Science Foundation Directorate for Social, Behavioral, and Economic Sciences.