What is reproducible research?
"Research is defined as reproducible when published results can be replicated using the documented data, code, and methods employed by the author or provider without the need for any additional information or needing to communicate with the author or provider. Reproducible research is generally seen as a marker of good experimental design. One way that researchers can ensure that their research is reproducible is by adhering to data standards and including methods in their research data metadata" (National Network of Libraries of Medicine, n.d).
What is the Reproducibility Crisis?
The reproducibility crisis, also known as the replicability or replication crisis, refers to a growing realisation that a surprising number of published and highly cited scientific studies fail to replicate when procedures are repeated. The concern is that a proportion of research findings are false positives due to a lack of transparency in research methods and analyses and a bias towards publishing “positive” findings. Open Science is a movement that aims to improve the robustness of research findings and increase our confidence in scientific research.
According to van der Zee and Reich (2018), "one challenge in defining Open Education Science is the great methodological diversity within the education fields, and our aim is to decribe a framework that can be interpreted and implemented across qualitative, quantitative, and design research" (p. 2).
How is ReproducibiliTea contributing to this solution?
Started by three early career researchers, ReproducibiliTea began at Oxford University as a journal club that focuses on promoting Open Science practices. The initiative aimed to allow researchers of all career stages to get familiar with and critically discuss seminal publications on the Open Science movement through weekly meetings. To date, there are 70 ReproducibiliTea journal clubs in institutions around the world.
The Singapore ReproducibiliTea Journal Club had its first meeting on 31 Oct 2019. The journal club also has a Twitter account that shares and connects with other researchers involved in the Open Science movement.
In this article, we find out more from Dr Alexa von Hagen and Dr Pierina Cheung, two of the organising committee members, on how Open Science practices can benefit educational research.
1. How did you come to know about or were introduced to Open Science practices?
Alexa: I first heard about the Open Science Movement during my PhD studies at Macquarie University in Australia. There was a group of postdoctoral researchers who organised weekly meetings to discuss recent publications questioning the way we do science and talking about ways to improve the reproducibility, transparency and accountability of our research practices. Now my primary source of learning about new publications and events on Open Science is Twitter.
Pierina: It got me interested when I heard about the failure to replicate experiments that you may consider classic in psychology. I also have collaborators who are supportive of adopting open science practices in research.
2. What kinds of Open Science practices do you adopt in your research?
Alexa: I have only very recently started to learn about Open Science practices and am slowly trying to implement them in my research. For instance, I am trying to pre-register all the new studies I work on to protect myself from questionable research practices such as p-hacking, harking or selective reporting. I am also trying to work on the reproducibility of my research by being more detailed in the descriptions of the methods I use and annotating my datasets in a way that can be used by other researchers in the future.
Pierina: I started pre-registering one of my studies 2- 3 years ago, and it was a very positive experience. So now I continue with this practice, and try to pre-register all my studies. I also share data and analysis code when I can, and post my work on preprint servers for others to download. Currently, I am looking into ways to incorporate open science practices (or at least to introduce the topic) when I teach this semester.
3. What do you think are the challenges or constraints facing educational researchers locally in adopting open science practices?
Alexa: My impression is that implementing changes in your workflow to adopt Open Science practices takes a lot of time at the beginning, although I am convinced that it improves the quality and integrity of your research and saves you time in the long run. However, to get started, you need to be able to make some extra time to learn and implement new procedures, such as pre-registrations, curation of datasets, annotating scripts, etc.
Pierina: One challenge is that educational research tends to involve a broader range of research methodologies than psychology. For example, qualitative research and secondary data analysis are quite common in educational research. Open science practices for these types of studies are in its infancy stage and may thus be harder for researchers to adopt in their research workflow. But there’s a growing body of resources for these types of studies that we can all draw on and learn about! Also, open science practices may look differently for qualitative research. I think it is important to discuss what open science means to each individual researcher. It’s also important to acknowledge that we are all new to this, and that it's not unusual to experience barriers or make mistakes as we change our ways of doing science!
4. How do you think local researchers can benefit from it?
Alexa: I believe that the Open Science Movement proposed nothing more than that we ‘just do science the right way’. So I don’t really see it as an option for researchers, but rather as a responsibility that we all need to commit to. In my own experience, learning about Open Science and trying to adopt transparent, reproducible practices in my research have increased my confidence in my research, as well as the control I feel I have over my workflow.
Alexa von Hagen is a Research Fellow at the Office of Educational Research, Centre for Research in Child Development (OER, CRCD) at the National Institute of Education (NIE). She has previous experience as a clinician and researcher in topics related to child bilingualism and learning difficulties. In addition, she has a special interest in open science practices and research synthesis methodologies, such as systematic reviews and meta-analyses.
Pierina Cheung is a Research Scientist at the Office of Educational Research, Centre for Research in Child Development (OER, CRCD). Her research examines the role of language for abstract thinking, with a focus on the acquisition of numerical knowledge. She is a member of the Society for the Improvement of Psychological Science and has experience integrating open science practices in her research.
Citations:
National Network of Libraries of Medicine. (n.d.). Reproducible research. Retrieved from https://nnlm.gov/data/thesaurus/reproducible-research
van der Zee, T., & Reich, J. (2018). Open Education Science. AERA Open, 4(3), 1-15. https://doi.org/10.1177/2332858418787466