"Reproducibility is a core principle of scientific progress. Scientific claims should not gain credence because of the status or authority of their originator but by the replicability of their supporting evidence." - Open Science Collaboration

Definitions

Reproducibility

"The ability of a researcher to duplicate the results of a prior study using the same materials and procedures as were used by the original investigator" (Bollen et al, 2015)

“In this “replication crisis” era, reproducibility is the only thing that can be effectively guaranteed in a published study. Whether any claimed findings are indeed true or false can only be confirmed via additional studies, but reproducibility can be confirmed immediately.” (Broman, et al. 2017)

Replicability

“The ability of a researcher to duplicate the results of a prior study if the same procedures are followed but new data are collected.” (Bollen et al, 2015)

Guidelines

Services for reproducible research

University Libraries

The University Libraries support students and researchers across the University of Minnesota and have many experts and services that can support research reproducibility and rigor across the research lifecycle. The library has subject librarians for each discipline who can help you find reporting guidelines, help you manage your data and understand data sharing requirements and options, find repositories for pre-registering studies and analysis plans, and answer more discipline specific questions.  We are also happy to present in classes and workshops on this topic. 

Consultation Services

  • Subject librarians can work with individuals, research groups, and departments to advise on many aspects of reproducibility and rigor. If you have any questions reach out to the subject librarian for your area. 

Methods

Finding and Using Reporting Guidelines
: Reporting Guidelines provide specific instructions for what you need to report about your methodology so others can evaluate and reproduce your work. There are reporting guidelines for all type of qualitative and quantitative research. 

Research Data Management: Data Management improves the consistency and rigor of your research data so that when you report your research others can understand and interpret it. 

Systematic Review Support
: Librarians have extensive experience conducting systematic reviews and the involvement of librarians has been shown to improve the quality and reproducibility of systematic reviews. 

Reporting and Dissemination 

Tools for pre-registration: Pre-registering your studies and analysis plans ensures that your study can be found and readers can differentiate exploratory from confirmatory research. 

Reproducibility (the ability to verify results)

Data Curation and Sharing: Curating and sharing your research data, code, and materials, means others can reproduce your results. 

 

LATIS

Liberal Arts Technologies and Innovation Services (LATIS), housed in the College of Liberal Arts (CLA), supports researchers in the liberal arts and social sciences. We provide training, consultation, and direct support for a variety of research methodologies and tools. Our goal is to help faculty and graduate students take steps towards more reproducible workflows and better managed data wherever they are in research life cycle.

We offer a workshop series on reproducible research tools that are free and open to graduate students and faculty.


More information about our services is available at z.umn.edu/latisresearch.

Some specific services relevant for reproducibility:

  • Documentation: Tips and techniques to document and organize your workflow no matter what tool you are using. Some examples of tools we can help with  Markdown/LaTeX, Jupyter Notebooks, Qualitative Analysis Tools (NVivo, Atlas.TI), Quantitative Analysis Tools (R, SPSS, Stata)
  • Open Science: Prepare your data and code for sharing. We can help: 1. Review datasets for potential privacy/identification issues 2. Extract metadata (variable labels, value labels) from survey tools or statistical packages to create codebooks. 3. Work with the libraries to determine the best ways to make your data/materials available.
  • Version Control: Integrate version control systems (such as github) into your workflows.
  • Efficiency: Our research computing resources can help you run large data analyses efficiently using CLA's new cluster computing and scripting techniques for parallelization.
  • Automation: Learn tools to automate portions of your workflow, from data collection to data "wrangling" using tools such as R and Python.

Contact us at surveys@umn.edu or our Online Service Request Form.

 

Minnesota Supercomputing Institute

  • Jupyter is an interactive electronic notebook for any computations in R, Python or other languages. See MSI beta for access to MSI’s installation.
  • Stratus is a local research compute cloud environment available as part of MSI beta that will enable you to create exact cloned images of your production workflows.
  • MSI hosts their own enterprise version of Galaxy, that enables extensive point-and-click genomics analyses within your browser, where all histories of data analysis are carefully logged, tracked, and easily reproduced from start to finish.

 

References

Broman, K., Cetinkaya-Rundel, M., Nussbaum, A., Paciorek, C., Peng, R., Turek, D., & Wickham, H. (2017). Recommendations to Funding Agencies for Supporting Reproducible Research. American Statistical Association.

Bollen, K., Cacioppo, J., Kaplan, R., Krosnick, J. A., & Olds, J. L. (2015). Social , Behavioral , and Economic Sciences Perspectives on Robust and Reliable Science. Report of the Subcommittee on Replicability in Science Advisory Committee to the National Science Foundation Directorate for Social, Behavioral, and Economic Sciences.