Category Archives: Open Research at Cambridge Conference

We are going OPEN – the Open Research experiment has begun!

Uncategorized, ,

There has been much discussion recently about the reproducibility crisis and about the growing distrust among the public in the quality of research. As illustrated in our ‘Case for Open Research’ series of blog posts, one of the main reasons for this is that researchers are currently rewarded for the number of papers they publish in high impact factor journals, and not necessarily for the quality of work that they are doing.

Indeed, Cambridge researchers clearly indicated that the lack of incentives to do anything other than publishing in these types of journals is one of the main blockers discouraging them from adopting a more open research practice.

Joining forces with the Wellcome Trust

The Office of Scholarly Communication started talking about these problem with the Open Research team at the Wellcome Trust. The Wellcome Trust are natural allies, as they have consistently led their researchers towards greater openness. They were one of the first funding bodies to introduce policies on Open Access and on data management and sharing. Now the Wellcome Trust is moving towards proactively supporting Open Research beyond enforcing their compliance requirements.

To promote immediate and transparent research sharing, they have recently launched the Wellcome Open Research platform which allows researchers to submit articles about virtually any research output and get published within a couple of days. The Wellcome Trust is now considering making Open Research one of their strategic priorities.

We quickly realised that we have a lot of shared interests, and joining forces to tackle the problem together made a lot of sense. We came up with the idea to launch the Open Research Pilot Project.

The Open Research Pilot – understanding the barriers to “openness”

We conceived the project as a two year experiment, which would allow us to gain an understanding of what is needed for researchers to share and get credit for all outputs of the research process. These include non-positive results, protocols, source code, presentations and other research outputs beyond the remit of traditional publications.

The Project aims to understand the barriers preventing researchers from sharing (including resource and time implications), as well as what the incentives are. The Project aims to utilise the new Wellcome Open Research publishing platform, together with other channels, to share these outputs.

The invitation to take part in the Pilot was sent to all researchers at Cambridge funded by the Wellcome Trust. Participating researchers had to commit to sharing of research outputs beyond traditional publications and to engage with the Project, by participating in Project meetings and contributing to Project publications.

Is ‘doing the right thing’ enough incentive?

Our biggest question was whether anyone would be willing to participate in the Pilot. We did not offer any incentive other than encouraging researchers to contribute to the greater good. The only support available to those who wanted to take part in the project was that offered by the Wellcome Trust and Cambridge Open Research team members, but there was no financial aid available to prospective participants. We thought that regardless of the outcome, that inviting researchers would be a good exercise to go through – we thought that if no one applied, we would have learnt that doing ‘the right thing’ was not a good enough motivator.

Thankfully, we received several fantastic applications from individual researchers and research groups who demonstrated great interest in and motivation for Open Research. We initially planned to work with two research groups, but given the quality of applications received and passion for Open Research expressed by the applicants, we decided to extend the scope of the project to four research groups. We have selected researchers doing different types of research, with the aim of learning about distinct problems in sharing that are experienced in diverse research disciplines:

  •       Dr Laurent Gatto –is  doing computational biology research, with a special focus on proteomics data. His interest is: How to effectively share research data and the code needed to reproduce them?
  •       Dr David Savage – is researching molecular pathogenesis of the consequences of obesity. His question is: What are the problems with sharing data coming from human participants?
  •       Dr Benjamin Steventon – is a developmental biologist generating and analysing large-scale imaging datasets. He would like to know: Are there image repositories allowing one to share large image datasets in a re-usable way?
  •       Dr Marta Costa and Dr Greg Jefferis (and others) – researchers leading the work on two collaborative projects: Connectomics and Virtual Fly Brain, which will create interactive tools to interrogate Drosophila neural network connections. They would like to understand: What are the issues with sharing complex interactive datasets? How to ensure long-term preservation of complex digital objects?

Motivations

So what motivated these researchers to apply for the project? We asked this question at the application stage and were positively surprised by the altruistic answers that we received. Our researchers were largely driven by a desire to improve the research process. We have seen responses like:

  • “Openness in research, from data and software to publication, is a central pillar of good research.”
  • “I am very concerned (disappointed as a scientist) by the current wave of ‘unreproducible’ and/or ‘irrelevant’ research, and am very passionate about contributing to improving scientific endeavour in this regard.”
  • “I am very enthusiastic about exploiting new ways of sharing my research output beyond the established peer-review journal system.”
  • “I believe that sharing research outputs fully, including data and code are essential to accelerate research, and I have benefitted from it in my own research.”

Summarising, researchers expressed a great desire for contributing to a cultural change. Researchers wanted to change the way in which research was disseminated and to increase research transparency and reproducibility.

Let’s get to work

We all met (the researchers, Wellcome Trust and Cambridge Open Research teams) on Friday 27 January to officially start the two year project. Each research group was appointed a facilitator – a dedicated member of the Cambridge Open Research team to support researchers during the Project. Research groups will meet with their facilitators on a monthly basis in order to discuss shareable research outputs and to decide on best ways to disseminate these outputs. Every six months all project members will meet together to discuss the barriers to sharing discovered and to assess the progress of the Project.

One of the main goals of the Project is to learn what the barriers and incentives are for Open Research and to share these findings with others interested in the subject to inform policy development. Therefore, we will be regularly publishing blog posts on the Unlocking Research blog and on the Wellcome Open Research blog with case studies describing what we have discovered while working together. There will be an update from each research group every six months. We will also be publicly sharing all main outputs of the Project.

We are all extremely excited about going “Open” and we suggest that anyone interested in the Open Research practice watches this space.

Published 08 February 2017
Written by Dr Marta Teperek
Creative Commons License

The art of software maintenance

Uncategorized, ,

When it comes to software management there are probably more questions than answers to problems – that was the conclusion of a recent workshop hosted by the Office of Scholarly Communication (OSC) as part of a national series on software sustainability, sharing and management, funded by Jisc. The presentations and notes from the day are available, as is a Storify from the tweets.

The goal of these workshops was to flesh out the current problems in software management and sharing and try to identify possible solutions. The researcher-led nature of this event provided researchers, software engineers and support staff with a great opportunity to discuss the issues around creating and maintaining software collaboratively and to exchange good practice among peers.

Whilst this might seem like a niche issue, an increasing number of researchers are reliant on software to complete their research, and for them the paper at the end is merely an advert for the research it describes. Stephen Eglen described this in his talk as an ‘inverse problem’ – papers are published and widely shared but it is very hard to get to the raw data and code from this end product, and the data and code are what is required to ensure reproducibility.

These workshops were inspired by our previous event in 2015, where Neil Chue Hong and Shoaib Sufi spoke with researchers at Cambridge about software licensing and Open Access. Since then the OSC has had several conversations with Daniela Duca at Jisc and together we came up with an idea of organising researcher-led workshops across several institutions in the UK.

Opening up software in a ‘post-expert world’

We began the day with a keynote from Neil Chue-Hong from the Software Sustainability Institute who outlined the difficulties and opportunities of being an open researcher in a ‘post-expert world’ (the slides are available here). Reputation is crucial to a researcher’s role and therefore researchers seek to establish themselves as experts. On the other hand, this expert reputation might be tricky to maintain since making mistakes is an inevitable part of research and discovery, which is poorly understood outside of academia. Neil introduced Croucher’s Law to help us understand this: everyone will make mistakes, even an expert, but an expert will be aware of this so will automate and share their work as much as possible.

Accepting that mistakes are inevitable in many ways makes sharing less intimidating. Papers are retracted regularly due to errors and Neil gave examples from a variety of disciplines and career stages where people were open about their errors so their communities were accepting of the mistakes. In fact, once you accept that we will all make mistakes then sharing becomes a good way to get feedback on your code and to help you fix bugs and errors.

This feeds into another major theme of the workshop which Neil introduced; that researchers need to stop aiming for perfect and adopt ‘good enough’ software practices for achievable reproducibility. This recognises that one of the biggest barriers to sharing is the time it takes to learn software skills and prepare data to the ‘best’ standards. Good enough practices mean accepting that your work may not be reproducible forever but that it is more important to share your code now so that it is at least partially reproducible now. Stephen Eglen built on this with his paper on ‘Towards standard practices for sharing computer code and programs in neuroscience’ which includes providing data, code, tests for your code and using licences and DOIs.

Both speakers and the focus groups in the afternoon highlighted that political work is needed, as well as cultural change, to normalise code sharing. Many journals now ask for evidence of the data which supports articles and the same standards should apply to software code. Similarly, if researchers ask for access to data when reviewing articles then it makes sense to ask for the code as well.

Automating your research: Managing software

Whilst sharing code can be seen as the end of the lifecycle of research software, writing code with the intention of sharing it was repeatedly highlighted as a good way to make sure it is well-written and documented. This was one of several ‘selfish’ reasons to share, where sharing also helps the management of software, through better collaboration, the ability to track your work and being able to use students’ work after they leave.

Croucher’s Law demonstrates one of the main benefits of automating research through software; the ability to track the mistakes to improve reproducibility and make fixing mistakes easier. There were lots of tools mentioned throughout the day to assist with managing software from the well-known version control and collaboration platform Github to the more dynamic such as Jupyter notebooks and Docker. As well as these technical tools there was also discussion of more straightforward methods to maintain software such as getting a code buddy who can test your code and creating appropriate documentation.

Despite all of these tools and methods to improve software management it was recognised by many participants that automating research through software is not a panacea; the difficulties of working with a mix of technical and non-technical people formed the basis of one of the focus groups.

Sustaining software

Managing software appropriately allows it to be shared but re-using it in the long- (or even medium) term means putting time into sustaining code and make sure it is written in a way that is understandable to others. The main recommendations from our speakers and focus groups to ensure sustainability were to use standards, create thorough documentation and embed extensive comments within your code.

As well as thinking about the technical aspects of sustaining software there was also discussion of what is required to motivate people to make their code re-usable. Contributing to a community seemed to be a big driver for many participants so finding appropriate collaborators is important. However larger incentives are needed and creating and maintaining software is not currently well-rewarded as an academic endeavour. Suggestions to rectify this included more software-oriented funding streams, counting software as an output when assessing academics, and creating a community of software champions to mirror the Data Champions scheme we recently started in Cambridge.

Next steps

This workshop was part of a national discussion around research software so we will be looking at outcomes of other workshops and wider actions the Office of Scholarly Communication can support to facilitate sharing and sustaining research software. Apart from Cambridge, five other institutions held similar workshops (Bristol, Birmingham, Leicester, Sheffield, and the British Library). As one of the next steps, all organisers of these events want to meet up to discuss the key issues raised by researchers to see what national steps should be taken to better support the community of researchers and software engineers and also to consider if there any remaining problems with software which could require a policy intervention.

However, following the maxim to ‘think global, act local’, Neil’s closing remarks urged everyone to consider the impact they can have by influencing those directly around them to make a huge difference to how software is managed, sustained and shared across the research community.

Published 29 January 2017
Written by Rosie Higman
Creative Commons License

‘Paperless research’ solutions – Electronic Lab Notebooks

Uncategorized, ,

The Office of Scholarly Communication started 2017 with a discussion about ‘going digital’ – on 13 January 2017 we organised an event at Cambridge University’s Department of Engineering to flesh out the problems preventing researchers from implementing Electronic Lab Notebook solutions. Chris Brown from Jisc wrote an excellent blog post with his reflections of the event* and agreed for us to re-blog it here.

For researchers working in laboratories the importance of recording experiments, results, workflows, etc in a notebook is engrained into you as a student. However, these paper-based solutions are not ideal when it comes to sharing and preservation. They pile on desks and shelves, vary in quality and often include printed data stuck in. To improve on this situation and resolve many of these issues, e-lab notebooks (ELNs) have been developed. Jisc has been involved in this work through funding projects such as CamELN and LabTrove in the past. Recently, interest in this area has been renewed with the Next Generation Research Environment co-design challenge.

On Friday 13 January I attended the E-Lab Notebooks workshop at the University of Cambridge, organised by Office of Scholarly Communication. Its purpose was to open up the discussion about how ELNs are being used in different contexts and formats, and the concerns and motivations for people working in labs. A range of perspectives and experience was given through presentations, group and panel discussions. The audience were mostly from Cambridge, but there was representation from other parts of the UK, as well as Denmark and Germany. A poll at the start showed that the majority of the audience were researchers (57%).

Institutional and researchers’ perspective on ELNs at Cambridge

The first part of the workshop focussed on the practitioners’ perspective with presentations from the School of Biological Sciences. Alastair Downie (Gurdon Institute) talked about their requirements for an ELN as well as anxieties and risks of adopting a particular system. Research groups currently use a variety of tools, such as Evernote and Dropbox, and often these are trusted more than ELNs. The importance of trust frequently came up during the day. Alastair conducted a survey to gather more detail on the use and requirements of ELNs and received an impressive 345 responses. Cost and complexity were given as the main reasons not to use ELNs. However, when asked for the most important features, cost was less important but ease of use was the most. Researchers want training, voice recognition and remote access. There is clear interest across the school at all levels, but it requires a push with guidance and direction.

Pic1Marko Hyvönen (Dept of Biochemistry) gave the PI perspective and the issues with an ELN for a biochemical lab. He reinforced what Alastair had said about ELNs. He showed how paper log books pile up, deteriorate over time and sometimes include printed information. They are hard to read and easy to destroy, a poor return on effort, often disappear and not searchable. It was interesting to hear about bad habits such as storing data in non-standardised ways, missing data, printing out Word documents and sticking them into the lab books.

With 99% of their data electronic many of the issues in the use of lab books generally are around data management and not ELNs. An ELN solution should be easy to use, cross platform, have a browser front end, be generic/adaptable, allow sharing of data and experiments, enforce Standard Operating Procedures when needed, have templates for standard work to minimise repetition, include inputting of data from phones and other non-specific devices. What they don’t want are the “bells and whistles” features they don’t use. Getting buy-in from people is the top issue to overcome in implementing an ELN.

Views on ELNs from outside the UK

Jan Krause from the École pPolytechnique Fédérale de Lausanne (EPFL) gave a non-UK perspective on ELNs. He described a study, as part of a national RDM project, where they separated ELNs (75 proprietary, 12 open source – 91 features) and Lab Info Management Systems (LIMS) (281 proprietary, 9 open source – 95 features) and compared their features. The two tools used mostly in Switzerland are SLims (commercial solution) and openBIS (homemade tool). To decide which tool to use they undertook a three phase selection process. The first selection was based on disciplinary and technical requirements. The second selection involved detailed analysis based on user requirements (interviews and evaluation weighted by feature) and price. The third selection was tendering and live demos.

Data storage, security and compliance requirements

When using and sharing data you need to make sure your data is safe and secure. Kieren Lovell, from the University Information Services, talked about how researchers should keep their data and accounts safe. Since he started in May 2015, all successful hacks on the university have been due to human error, such as unpatched servers, failures in processes, bad password management, and phishing. Even if you think your data and research isn’t important, the reputational damage of security attacks to the university is huge. He recommended that any research data is shared through cloud providers rather than email, never trust public wifi as is not secure so use Cambridge’s VPN service. If using a local machine you should encrypt your hard drive.

Pic2

Providers’ perspective

In the afternoon, presentations were from the providers’ perspective. Jeremy Frey, from the University of Southampton, talked about his experience of developing an open source ELN to support open and interdisciplinary science. He works on getting the people and technology to work together. It’s not just recording what you have done, you need to include the narrative behind what you do. This is critical for understanding and ELNs are one part of the digital ecosystem in the lab. The solution they’ve developed is LabTrove, partly funded by Jisc, which is a flexible open source web based solution. Allowing pictures to be added to the notes has really helped with accessibility and usability, such as dyslexia. Sustainability, as is often the case, came up and how a community is required to support such a system. It also needs to expand beyond Southampton. Finally, Jeremy used Amazon Echo to query the temperature within part of his lab. He hopes that this will be used more in the lab in the future when it can recognise each researcher’s voice.

In the next two presentations, it was over to the vendors to show the advantages of adopting RSpace (by Rory Macneil) and Dotmatics (by Dan Ormsby). The functionality on offer in these types of solutions is attractive for scientists and RSpace showed how it links to most common file stores. With any ELN, it should enhance researchers’ workflow and integrate with the tools they use.

Removing the barriers

After lunch there were three parallel focus group discussions. I attended the one on sustainability, something that comes up frequently in discussions, particularly when looking at open source or proprietary solutions. Each group reported back as follows:

Focus group 1: Managing the supplier lock in risk

Stories of use need to be shared. The PDF is not a great format for sharing. Vendors tell the truth like estate agents. Have to accept the reality that won’t have 100% exporting functionality so need to decide the minimum level. Determine specific users’ requirements.

Focus group 2: Sustainability of ELN solutions

What is the lifetime of an ELN? How long should everything be accessible? Various needs come from group and funder requirements, e.g. 10 years. There is concern if you are relying on one commercial solution as companies can die, so how can you guarantee the data will be available? Have exit policies and support standards and interoperability so data can be moved across ELNs. Broken links and file formats expiring is not just an ELN problem, but relates to the archiving of data in general. Should selection and support of an ELN be at group, department, institution or national level? This is difficult if it’s in one group as adopting any technical solution requires support in place. It requires institutional level support.

Focus group 3: Human element of ELN implementation

The biggest hurdle is culture change and showing the benefits of using an ELN. Training and technical support costs money and time. It would cost more initially but becomes more efficient. You can incentivise people by having champions. There are different needs in a large institution. You may join a lab and find the ELN is not adequate. Legal issues around sensitive data complicates matters. You need to believe it will save time. Long term solutions include using cloud base solutions, even MS Office, but what happens when people leave? Need support from higher level. Functionality should be based on user requirements. A start would be to set up a mailing list of people interested in ELNs.

Remaining barriers to wide ELN adoption

Finally, I chaired a panel session with all the presenters. Marta Teperek had kindly asked me to give a short presentation on what Jisc does as many researchers don’t know (in fact I was asked “what’s Jisc?” in the focus group) and to promote the Next Generation Research Environment co-design challenge. Following my presentation the discussion was prompted by questions from the audience and remotely via sli.do. Much of the discussion re-iterated what had been said in the presentations, such as the importance of an ELN that meets the requirements of researchers. It should allow integration with other tools and exporting of the data for use it other ELNs. Getting ELNs used within a department is often difficult so it does need institution level commitment and support. Without this ELNs are unlikely to be adopted within an institution, never mind nationally. One size does not fit all and we should not try to build an ELN that tries to satisfy the different needs of various disciplines. A modular system that integrates with the tools and systems already in use would be a better solution. Much of what was said tallied with the feedback received for the Next Generation Research Environment co-design challenge.

Closing remarks

Ian Bruno closed the workshop and he reiterated what was said in the panel discussion. I found the event extremely helpful and it provided lots of useful information to feed into the Next Generation Research Environment work. I’d like to thank Marta Teperek for inviting me to chair the panel and for all her hard work putting the event together with @CamOpenData. Marta has put together the tweets from the day into the following storify.  All notes and presentations from the event are now published in Apollo, the University of Cambridge’s research repository.

Follow-up actions at the University of Cambridge – give it a go!

Those of you who are interested in ELNs and who are based at the University of Cambridge might be interested in knowing that we are planning to do some trial access to Electronic Lab Notebooks (ELN). The purpose of this trial will be to test out several ELNs to decide on solutions which might best meet the requirements of the research community. A mailing list has been set up for people who are interested in being part of this pilot or would like to be involved in these discussions. If you would like to be added to the mailing list, please fill in the form here: https://lists.cam.ac.uk/mailman/listinfo/lib-eln

*Originally published by Jisc on 18 January 2017.

Published on 29 January 2017
Written by Chris Brown
Creative Commons License