Category Archives: Publishing

What we can learn from the ‘promise and pitfalls of preregistration’ meeting

Publishing, Supporting Open Research

Dr Mandy Wigdorowitz, Open Research Community Manager, Cambridge University Libraries

The promise and pitfalls of preregistration meeting was held at the Royal Society in March 2024. It was organised to address the utility of preregistration and initiate an interdisciplinary dialogue about its epistemic and pragmatic aims. The goal of the meeting was to explore the limitations associated with preregistration, and to conceive of a practical way to guide future research that can make the most of its implementation.

Preregistration is the practice of publicly declaring a study’s hypotheses, methods, and analyses before conducting a research study. Researchers are encouraged to be as specific as possible when writing preregistration plans, detailing every aspect of the research methodology and analyses, including, for instance, the study design, sample size, procedure for dealing with outliers, blinding and manipulation of conditions, and how multiple analyses will be controlled for. By doing so, researchers commit to a time-stamped study plan which will reduce the potential for flexibility in analysis and interpretation that may lead to biased results. Preregistration is a community-led response to the replication crisis and aims to mitigate questionable research practices (QRPs) that have come to light in recent years, some of which include HARKing (Hypothesising After Results are Known), p-hacking (the inappropriate manipulation of data analysis to enable a favoured result to be presented as statistically significant), and publication bias (the unbalanced publication of statistically significant findings or positive results over null and/or unexpected findings) (Simmons et al., 2011; Stefan & Schönbrodt, 2023).

The meeting brought together scholars and publishers from a range of disciplines and institutions to discuss whether preregistration has indeed lived up to these aims and whether and to what extent it has solved the problems it was envisioned to address.

It became clear that the problems associated with QRPs have not simply disappeared with the uptake and implementation of preregistration. From the perspective of meta-research, the success of preregistration appears to be largely disciplinary and legally dependent, with some disciplines mandating and normalising it (e.g., clinical trial registration in biomedical research), others greatly encouraging and (sometimes) requiring it (e.g., psychological science research), and others having no expectations about its use (e.g., economics research). The effectiveness of preregistration was shown to be linked to these dependencies, but also related to the quality and detail of the preregistration plan itself. Researchers are the arbiters of their research choices and if they choose to write vague or ambiguous preregistration plans, the problems that preregistration are assumed to address will inevitably persist.

Various preregistration templates exist (such as on the Open Science Framework, OSF) and some incentives for preregistration are recognised, such as the preregistration badges awarded by some journals, making it a systematic and straightforward exercise. In practice, however, it is not always the case that sufficient information is provided, and even in cases where preregistered plans are detailed, they are not always followed for various pragmatic or other (not always nefarious) reasons. As such, the research community are cautioned to not assume that preregistration equates to better or more trustworthy research. Rather, the preregistration plan needs to be critically reviewed as a standalone document in conjunction with the published study. This is important because preregistration plans that are usually deposited into repositories (e.g., OSF, National Library of Medicine’s Clinical Trials Registry) are seldom evaluated as entities of their own or against their corresponding research articles. Note that this is unlike registered reports which are a type of journal article that details a study’s protocol that does get peer reviewed before data is collected and if reviewed favourably, is given an in-principal acceptance regardless of the study outcomes.

Other discussions centred around the utility of preregistration in exploratory versus confirmatory research, whether preregistration can improve our theories, and how the process of conducting multiple but slightly varied analyses and selecting the most desired outcome (also referred to the ‘garden of forking paths’) affects the claims we make.

The overall sentiment from the meeting was that while preregistration does not solve all the issues that have arisen from QRPs, it ultimately leads to more transparency of the research process, accountability on the part of the researchers conducting the research, and it facilitates deeper engagement with one’s own research prior to any collection or analysis of data.

Since attending the meeting, I have taken away valuable insights that have made me critically reflect on my own research choices, and from a practice perspective, I have downloaded the OSF preregistration template and am documenting the plans for a research project.

Given the strides that have been taken toward improving the transparency, credibility and reproducibility of research, researchers at Cambridge need to consider whether preregistration plans should be included as another type of output that can be deposited on the institutional repository, Apollo. We have recently added Methods and preprints as output types which have broadened the options for sharing and which align with open research practices. Including preregistration could be a valuable and timely addition.  

References

Stefan, A. M., & Schönbrodt, F. D. (2023). Big little lies: a compendium and simulation of p-hacking strategies. Royal Society Open Science, 10(2), 220346. https://doi.org/10.1098/rsos.220346

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-1366. https://doi.org/10.1177/0956797611417632

Open Research for Inclusion – event round up

Humanities, Open Research at Cambridge Conference, Publishing, Supporting Open Research

Dr Mandy Wigdorowitz, Open Research Community Manager, Cambridge University Libraries

On Friday 17 November 2023, participants from across Cambridge and beyond gathered for a hybrid meeting on Open Research from different perspectives. Hosted by Cambridge University Libraries at Downing College, ‘Open Research for Inclusion: Spotlighting Different Voices in Open Research at Cambridge‘ drew attention to different areas of Open Research that have been at the forefront of recent discussions in Cambridge by showcasing the scope and breadth of open practices in typically under-represented disciplines and contexts. These included the Arts, Humanities, and Social Sciences, the GLAM sector (Galleries, Libraries, Archives, and Museums), and research from and about the Global South. A total of 84 in-person and 75 online attendees participated in the day-long event consisting of a keynote address, two talks, two panels, and a workshop.

The conference opened with a welcome address from Professor Anne Ferguson-Smith CBE FRS FMedSci, Pro-Vice-Chancellor for Research and International Partnerships and the Arthur Balfour Professor of Genetics. Professor Ferguson-Smith emphasised the significance and timeliness of the conference and how it underscores the importance of the Open Research movement. She encouraged attendees to be open to new ideas, approaches, and perspectives that center around Open Research and to celebrate the richness of diversity in research.

Our keynote speaker, Dr Siddharth Soni, Isaac Newton Trust Fellow at Cambridge Digital Humanities and affiliated lecturer at the Faculty of English, then addressed the audience with a talk on Common Ground, Common Duty: Open Humanities and the Global South, providing an account of how to think against neoliberal conceptions of ‘open’ and to reimagine what openness might look like if the Global South was viewed as a common ground space for building an open and international university culture. Dr Soni’s address set the tone for a rich, multi-layered exploration of Open Research on the day, urging attendees to think of open humanities as a form of knowledge that seeks to alter the form and content of knowledge systems rather than just opening Euro-American knowledge systems to global publics.

Dr Siddharth Soni Common Ground, Common Duty: Open Humanities and the Global South

The next talk was from Dr Stefania Merlo from the McDonald Institute for Archaeological Research and Dr Rebecca Roberts from the McDonald Institute for Archaeological Research and Fitzwilliam Museum who further explored the theme of the Global South in their practical perspective on how they managed the curation of digital archives for heritage management from their work on the projects: Mapping Africa’s Endangered Archaeological Sites and Monuments (MAEASaM) and Mapping Archaeological Heritage in South Asia (MAHSA). They reflected on the opportunities and challenges relating to the production and dissemination of information about archaeological sites and monuments in projects across Africa and South Asia as well as their experience working with and learning from local communities.

Dr Stefania Merlo and Dr Rebecca Roberts Open Data for Open Research – Reflections on the Curation of Digital Archives for Heritage Management in the Global South

An Open Research panel session was next which featured panellists with diverse backgrounds and expertise who addressed registrants’ pre-submitted and live questions. Some questions included the meaning of Open Research, its advantages and challenges, how Open Research can be engaged with by researchers (and in particular, early career researchers), and how it can be rewarded and embedded into the culture of research practices. There was engaging insights and debate amongst the panellists, led by Bertrand Russell Professor of Philosophy, Professor Alexander Bird. He shared the platform with Philosophy of Science Professor, Professor Anna Alexandrova, Psychiatry PhD student Luisa Fassi, Cambridge University Libraries (CUL) Interim Head of Open Research Services Dr Sacha Jones, Cambridge University Press & Assessment’s Research Data Manager Dr Kiera McNeice, and Cambridge’s Head of Research Culture Liz Simmonds.

Open Research Panel

Following lunch, a second panel of scholars working across the GLAM sector (Galleries, Libraries, Archives and Museums) took place. The panel was chaired by CUL’s Scholarly Communication Specialist, Dr Samuel Moore, and brought together experts who showcased their diverse work in this sector, from software development and museum practices to infrastructure and archiving support. The panel included Dr Mary Chester-Kadwell, CUL’s Senior Software Developer and Lead Research Software Engineer at Cambridge Digital Humanities, Isaac Newton Trust Research Associate in Conservation Dr Ayesha Fuentes from the Museum of Archaeology and Anthropology, Dr Agustina Martinez-Garcia, CUL’s Head of Open Research Systems, and Dr Amelie Roper, CUL’s Head of Research. Each panellist presented on a specialist area, including Open Research code and data practices in digital humanities, collections research, teaching and learning collections care, and Open Research infrastructure. A lively discussion followed from the presentations.

GLAM panel

In a workshop session, Tim Fellows, Product Manager for Octopus, outlined how Octopus is a free and alternative publishing model that can practically foster Open Research. The platform, funded by UKRI, is designed for researchers to share ‘micro publications’ that more closely represent how research is conducted at each stage of a project. In a demonstration of the platform, Tim Fellows showed how Octopus works, it’s design, user interface, and application all with the aim of aiding reproducibility, facilitating new ways of sharing research, and removing barriers to both publishing and accessing research. An in-depth discussion followed which centered on the ways the platform can be used as well as its uptake and application across various disciplines.

Tim Fellows Octopus.ac: Alternative Publishing Model to Foster Open Research

The final talk of the day was on Open Research and the coloniality of knowledge presented by Professor Joanna Page, Director of CRASSH and Professor of Latin American Studies. She discussed the topic with a specific focus on the questions of possession and access by outlining projects by three Latin American artists who have engaged with Humboldt’s legacy and the coloniality of knowledge. Using videos and imagery, Professor Page encouraged the audience to consider how they might identify where the principles of Open Research conflict with those of inclusion and cognitive justice, and what might be done to reconcile those ambitions across diverse cultures and communities. An engaging discussion ensued.

Professor Joanna Page Open Research and the Coloniality of Knowledge

A drinks reception brought the event to a close, allowing attendees a chance to mingle, network and continue the discussions. 

Special thanks to all speakers, attendees, and volunteers for making this event such a success. Stay tuned for information about our 2024 Open Research conference.

Rights retention built into Cambridge Self-Archiving Policy

Publishing, ,

We’re delighted to announce that the University of Cambridge has a new Self-Archiving Policy, which took effect from 1 April 2023.  The policy gives researchers a route to make the accepted version of their papers open access without embargo under a licence of their choosing (subject to funder requirements). We believe that researchers should have more control over what happens to their own work and are determined to do what we can to help them to do that.

This policy has been developed after a year-long rights retention pilot in which more than 400 researchers voluntarily participated. The pilot helped us understand the implications of this approach across a wide range of disciplines so we could make an informed decision. We are also not alone in introducing a policy like this – Harvard has been doing it since 2008, cOAlition S have been a catalyst for development of similar policies, and we owe a debt of gratitude to the University of Edinburgh for sharing their approach with us. 

Some of the issues that cropped up during the pilot were outlined by Samuel Moore, our Scholarly Communications Specialist, in an earlier post on the Unlocking Research blog.  The patterns we saw at that stage continued throughout the year-long pilot – there was no issue for most articles, but some publishers caused confusion through misinformation or by presenting conflicting licences for the researchers to sign. We do recognise that there are costs involved in high quality publishing, and we are willing to cover reasonable costs (while noting our concerns around inequities in scholarly publishing).   The fact is that some publishers are trying to charge the sector multiple times for the same content – subscription fees, OA fees, other admin fees – all while receiving free content courtesy of researchers that are usually funded by the taxpayer and charity funders. 

Many researchers and funders are understandably becoming firmer in their convictions that publicly funded research should be openly and publicly available. We are fortunate that at Cambridge we are in a position to support this through our support for diamond publishing initiatives (in which the costs of publishing are absorbed for example by universities and no fees are charged to the reader or the author), through read and publish agreements negotiated on behalf of the UK higher education sector and through payment of costs associated with publishing in fully open access venues. Rights retention gives researchers a back-up plan for when other routes are not available to them, e.g. when a journal moves unexpectedly out of a read and publish agreement or a publisher does not offer any publishing route that meets their funder requirements. 

This is not the end goal, we have work to do to reach an equitable approach to global scholarly publishing, and we can learn a lot especially from how South America approaches these issues. We welcome opportunities to work together with others around the world to create a more sustainable and equitable future for scholarly communications.

Read more about the new Cambridge Self-Archiving Policy on the Cambridge Open Access website.