Research integrity is the current discussion topic at many levels in the research sector. This week, the Council of the European Union will adopt conclusions on the European Open Science Cloud, including the Open Science agenda. To complement this, the League of European Research Universities (LERU) released its advice paper Open Science and its role in universities: A roadmap for cultural change, which discusses the  cultural change that is needed for universities – and other stakeholders – to embrace it.

This is hot on the heels of two events I spoke at last week. The first was “Towards cultural change in data management – data stewardship in practice” held at TU Delft in the Netherlands, and the second was Nurturing a Culture of Responsible Research in the Era of Open Science, coordinated by LERU and held at Geneva University’s Campus Biotech. Both events were attended by a mix of policy makers, researchers, librarians and administrators.

The slides and accompanying tweets for my TU Delft talk: The ‘end of the expert’: why science needs to be above criticism and the slides from my LERU talk: Institutional Framework and Responsibilities: Facing Open Science’s challenges and assuring quality of research are available.

My talks focused on questions of research integrity – that in this period of ‘post-truth’, where “Britain has had enough of experts” and a US lawyer “alone will decide what is and isn’t acceptable science” – science is very much under attack.

In this environment it is important that research remains above criticism, and that opens up the question of reproducibility of research. The late Professor Stephen Hawking noted “When public figures abuse scientific argument, citing some studies but suppressing others, to justify policies that they want to implement for other reasons, it debases scientific culture”.

We have been here before

This harks back to observations made by sociologist Robert K Merton in a 1942 essay “The Normative Structure of Science” who noted “Incipient and actual attacks upon the integrity of science” stating that “An institution under attack must re-examine its foundations, restate its objectives, seek out its rationale”. Now, 72 years later, the world is once again in a similar position.

Discussion of a ‘reproducibility crisis’ has been circulating for some years now. In 2016, Nature published the results of a survey of researchers asking about this issue, where 90% of the respondents said there was a ‘significant’ or ‘slight’ crisis. The study also asked whether people had been able to replicate results (others’ or their own) and whether they had published their inability to replicate the work.

Reproducibility studies – where specific studies are chosen and attempts were made to reproduce the results – support the argument that published research is not always repeatable. A reproducibility study of 100 psychology experimental and correlational studies showed a substantial decline in the replication effects: 97% of the original studies had significant results (p < .05), but only 36% of replications had significant results.

This issue is significant enough for governments and large bodies to take notice. A UK enquiry into Research Integrity that was halted for a few months last year during the general election has been revived. In December 2017, the US National Academies of Science established a committee on Reproducibility and Replicability in Science.

However, an opinion piece published in March in the Proceedings of the National Academies of Science put forward an alternative view, asking “Is science really facing a reproducibility crisis, and do we need it to?”. The piece tracks the recent increase in the frequency of use of the ‘crisis narrative’ in published research. It then asks how common fabricated, false, biased, and irreproducible findings are. The conclusion is there has not been a measurable increase, and that “Instead of inviting greater respect for and investment in research, [the crisis narrative] risks discrediting the value of evidence and feeding antiscientific agendas.”

Either way, this is a narrative that is experiencing a high level of interest currently.

Open Research to the rescue?

One of the proposed solutions to this issue is to increase transparency in the research process – opening up research so that each step of the research process is itself a research output that is citable and can be recognised. The term ‘Open Research’ or ‘Open Science’ (in Europe, meaning all types of research) or ‘Open Scholarship’ has many definitions. A small example includes from FOSTER science, also Wikipedia article on Open Science and in the Open innovation, open science, open to the world report.

Indeed, there are so many different definitions of Open Research/Science that now there is an attempt to define the definitions. One list attempting to collect the declarations and position statements on Open Research from around the world has over 90 entries. However, common to the majority of them is they are effectively based on Robert Merton’s 1942 work. The Mertonian norms of science are:

• Universalism
• Communalism
• Disinterestedness
• Organised scepticism

An example of how Open Research is being put forward as a solution is in the recent National Academies of Science report: “The Irreproducibility Crisis of Modern Science” which recommends: “Researchers should make their data available for public inspection after publication of their results”.

The European Commission Open Science Monitor has three core principles: Open Scholarly Communications, Open Access publications and Open Data. But this is easier said than done. While making underlying data available has been a requirement of some publishers and funders for some years now. Open Data is becoming less contentious, but other aspects of ‘openness’ are still new concepts to the majority of the research community.

There are plenty of actions individual researchers can do to make their work more open. Bianca Kramer and Jeroen Bosman have published 17 open science practices throughout the whole research workflow, including examples of tools that can help, as part of their 101 Innovations project.

As we have noted for some time at the Office of Scholarly Communication, making data and other non-traditional research outputs available is difficult. This requires training of our research community in how to research openly from the beginning of their research activity, rather than asking them to be open as an afterthought.

The institutional imperative

For all of the government enquiries, the funder requirements and the changing publisher processes, at some stage, research institutions need to be part of the process to implement change. In 2012, the Royal Society Science Policy Centre report Science as an open enterprise noted “Universities and research institutes should play a major role in supporting an open data culture”. In a 2016 paper in Royal Society Open Science titled The natural selection of bad science argued: “Improving the quality of research requires change at the institutional level”.

But institutions are slow to act, and reluctant to step outside the global esteem and reward systems that scaffold the research community. In 2017, a Harvard professor in a Nature column Faculty promotion must assess reproducibility referred to “The spectre of irreproducible research haunts the biomedical community”, and argued that “one group that must step up is that to which I belong: academic leadership”.

So are institutions meeting this challenge?

European moves towards Open Research

Well, some are. TU Delft is on the record for leading in this area  – their TU Delft Strategic Framework has implications for Open Science. Even more impressive is Utrecht University which “aims to operate at the forefront of Open Science” according to the University Strategic Plan 2016-2020.

Both universities, of course, are based in The Netherlands, which held the presidency of the European Union during the first half of 2016 with an agenda that included “Europe as an innovator and job creator”, achievable through “better alignment between academia and business through open access and better use of data”. This approach has clearly had deep impacts across the country.

Open Research in the UK

We are behind the curve in the Open Research space in the UK, but there are some moves in this direction.

The University of Reading has recently closed their consultation on their vision statement on Open Research. Discussion with the coordinators of this consultation underlined the  the question of language in this area. Because many of the terms used in scholarly communication are vernacular, interpretation of them varies (‘publish’ anyone?)

There is considerable confusion amongst the research community over questions of openness, even beyond the language question. It is very common for researchers to throw accusations against open access that reflect problems with the whole scholarly communication system, as I argued in 2015 with my research colleague Mary Anne Kennan in “Open access: the whipping boy for problems in scholarly publication“.

Cambridge is working towards a position statement on Open Research. When preparing for the accompanying consultation we are currently running at Cambridge, we considered the ‘Open Typology’ that was proposed by Sheila Corrall and Stephen Pinfield in their 2014 paper, “Coherence of ‘Open’ Initiatives in Higher Education and Research: Framing a Policy Agenda”. This breaks the areas of ‘open’ into three categories: Open Content, Open Systems and Open Development. Given the primary focus, at least initially, is a way of considering the overarching approach behind the Open Access and Research Data Management policy frameworks, the Cambridge consultation is focusing on Open Content and Systems. Open Development may come later but the conversations are not yet mature enough to include them at this stage.

The consultation is still underway, and to date attracting a strong response. The outcomes will be written up and shared after analysis.

However, while decidedly a step in the right direction, a position statement is only the beginning, as we have seen already over the past couple of years. The implementation is where the hard work begins. Change is slow in this space.

Published 31 May 2018
Written by Dr Danny Kingsley

Post script

I have spoken and written about the Open Research question at length. If you are interested, my keynote to the 2017 Conference on Open Access Scholarly Publishing – Is the tail wagging the dog? Perversity in academic rewards is available on video, as is my keynote to the Munin Open Access Conference in 2016 Reward, reproducibility and recognition in research – the case for going Open.

I have also written a series of blogs on the argument for Open Research:

• The case for Open Research: solutions?– 2 August 2016
• The case for Open Research: does peer review work?– 19 July 2016
• The case for Open Research: reproducibility, retractions & retrospective hypotheses– 14 July 2016
• The case for Open Research: the authorship problem– 12 July 2016
• The case for Open Research: the mis-measurement problem – 11 July 2016

There are also a couple of blogs about the Open Agenda at Cambridge:

• Perspectives on the Open future – 27 March 2018
• Could Open Research benefit Cambridge University researchers? – 3 August 2016