Monthly Archives: July 2016

The case for Open Research: the authorship problem

This is the second in a blog series about why we need to move towards Open Research. The first post about the mis-measurement problem considered issues with assessment. We now turn our attention to problems with authorship. Note that as before this is a topic of research in itself – and there is a rich vein of literature to be mined here for the interested observer.

Hyperauthorship

In May last year a high energy physics paper was published with over 5,000 authors. Of the 33 pages in this article, the paper occupied nine with the remainder listing the authors. This paper caused something of a storm of protest about ‘hyperauthorship’ (a term coined in 2001 by Blaise Cronin).

Nature published a news story on it, which was followed a week later by similar stories decrying the problem. The Independent published  a story with the angle that many people are just coasting along without contributing. The Conversation’s take on the story looked at the challenge of effectively rewarding researchers. The Times Higher Education was a bit slower off the mark, in August publishing a story questioning whether mass authorship was destroying the credibility of papers.

This paper was featured in  a keynote talk given at this year’s FORCE2016 conference. Associate Professor Cassidy Sugimoto from the School of Informatics and Computing, Indiana University Bloomington spoke about ‘Structural Disruptions in the Reward System of Science’ (video here). She noted that authorship is the coin of the realm the pivot point of the whole of the scientific system and this has resulted in the growth of authors listed on a paper.

Sugimoto asked: What does ‘authorship’ mean when there are more authors than words in a document? This type of mass authorship raises concerns about fraud and attribution. Who is responsible if something goes wrong?

The authorship ‘proxy for credit’ problem

Of course not all of those 5,000 people actually contributed to the writing of the article – the activity we would normally associate with the word ‘authorship’. Scientific authorship does not follow the logic of literary authorship because of the nature of what is being written about.

In 1998 Biagioli (who has literally written the book on Scientific Authorship or at least edited it) in a paper called ‘The Instability of Authorship: Credit and Responsibility in Contemporary Biomedicine’ said that “the kind of credit held by a scientific author cannot be exchanged for money because nature (or claims about it) cannot be a form of private property, but belongs in the public domain”.

Facts cannot be copyrighted. The inability to write for direct financial remuneration in academia has implications for responsibility (addressed further down), but first let’s look at the issue of academic credit.

When we say ‘author’ what do we mean in this context? Often people are named as ‘authors’ on a paper because their inclusion will help to have the paper accepted, or it is a token thanks for providing the grant funding for the work. These are practices referred to as ‘gift authorship‘ where co-authorship awarded to a person who has not contributed significantly to the study.

In an attempt to stop some of the more questionable practices above, the International Committee of Medical Journal Editors (ICMJE) has defined what it means to be an author which says authorship should be based on:

  • a substantial contribution
  • drafting the work
  • giving final approval and
  • agreeing to be accountable for the integrity of the work.

The problem, as we keep seeing, is that authorship on a publication is the only thing that counts for reward. This means that ‘authorship’ is used as a proxy for crediting people’s contribution to the study.

Identifying contributions

Listing all of the people who had something to do with a research project as ‘authors’ on the final publication fails to credit different aspects of the labour involved in the research. In an attempt to address this, PLOS asks for the different contributions by those named on a paper to be defined on articles, with their guidelines suggesting categories such as Data Curation, Methodology, Software, Formal Analysis and Supervision (amongst many).

Sugimoto has conducted some research to find what this reveals about what people are contributing to scientific labour. In an analysis of PLOS data on contributorship, her team showed that in most disciplines the labour was distributed. This means that often the person doing the experiment is not the person who is writing up the work. (I should note that I was rather taken aback by this when it arose in interviews I conducted for my PhD).

It is not particularly surprising that in the Arts, Humanities and Social Sciences that the listed ‘author’ is most often the person who wrote the paper. However in Clinical Medicine, Biomedicine or Biology very few authors are associated with the task of writing.  (As an aside, the analysis found women are disproportionately likely to be doing the experimentation, and men are more likely to be authoring, conceiving experimentation or obtaining resources.)

So, would it not be better if rather than placing the only emphasis on authorship of journal articles in high impact journals, we were able to reward people for different contributions to the research?

And while everyone takes credit, not all people take responsibility.

Authorship – taking responsibility

It is not just the issue of the inability to copyright ‘facts of nature’ that makes copyright unusual in academia. The academic reward system works on the ‘academic gift principle’ – academics provide the writing, the editing and the peer review for free and do not expect payment. The ‘reward’ is academic esteem.

This arrangement can seem very odd to an outsider who is used to the idea of work for hire. But there are broader implications than what is perceived to be ‘fair’ – and these relate to accountability. It is much more difficult to sue a researcher for making incorrect statements than it is to sue a person who writes for money (like a journalist).

Let us take a short meander into the world of academic fraud. Possibly the biggest and certainly highly contentious case was Andrew Wakefield and the discredited (and retracted) claim that the MMR vaccine was associated with autism in children. This has been discussed at great length elsewhere – the latest study debunking the claim was published last year. Partly because of the way science is credited and copyright is handled, there were minimal repercussions for Wakefield. He is barred from practicing medicine in the UK, but enjoys a career on the talkback circuit in the US. Recently a film about the MMR claims, directed by Wakefield was briefly shown at the Tribeca film festival before protests saw it removed from the programme.

Another high profile case is Diedderik Stapel, a Dutch social psychologist who entirely fabricated his data over many years. Despite several doctoral students’ work being based on this data and over 70 articles having to be retracted there were no charges laid. The only consequence he faced was having his professorship stripped.

Sometimes the consequences of fraud are tragic. A Japanese stem cell researcher, Haruko Obokata, who fabricated her results had her PhD stripped from her. There were no criminal charges laid but her supervisor committed suicide and the funding for the centre she was working in was cut.  The work had been published in Nature which then retracted the work and wrote some editorial about the situation.

The question of scientific accountability is so urgent that there was a call last year to criminalise scientific misconduct in this paper. Indeed things do seem to be changing slowly and there have been some high profile cases where scientific fraud has resulted in criminal charges being laid. A former University of Queensland academic is currently facing fraud related charges over his fabricated results from a study into Parkinson’s disease and multiple sclerosis. This time last year, Dong-Pyou Han, a former biomedical scientist at Iowa State University in Ames, was sentenced to 57 months for fabricating and falsifying data in HIV vaccine trials. Han has also been fined US$7.2 million. In both the cases the issue is the misuse of grant funding rather than publication of false results.

The combination of great ‘reward’ from publication in high profile journals and little repercussion (other than having that ‘esteem’ taken away) has proven to be too great a temptation for some.

Conclusion

The need to publish in high impact journals has caused serious authorship issues –  resulting in huge numbers of authors on some papers because it is the only way to allocate credit. And there is very little in the way we reward researchers that adequately allows for calling researchers to take responsibility when something goes wrong, in some cases resulting in serious fraud.

The next instalment in this series will look at ‘reproducibility, retractions and retrospective hypotheses.

Published 12 July 2016
Written by Dr Danny Kingsley
Creative Commons License

The case for Open Research: the mis-measurement problem

Let’s face it. The biggest blockage we have to widespread Open Access is not researcher apathy, a lack of interoperable systems, or an unwillingness of publishers to engage (although these do each play some part) – it is the problem that the only thing that counts in academia is publication in a high impact journal.

This situation is causing multiple problems, from huge numbers of authors on papers, researchers cherry picking results and retrospectively applying hypotheses, to the reproducibility crisis and a surge in retractions.

This blog was intended to be an exploration of some solutions prefaced by a short overview of the issues. Rather depressingly, there was so much material the blog has had to be split up, with several parts describing the problem(s) before getting to the solutions.

Prepare yourself, this will be a bumpy ride. This first instalment looks at the reward system. The second instalment will consider authorship and credit. The third will look at reproducibility, retractions and retrospective hypotheses. The fourth asks if peer review is working. And the final blog will discuss some options for solving at least part of the problem.

I should note that this is not a comprehensive literature review. Every subheading of this blog series is a topic of considerable research on its own and there are many further examples available to the interested reader. I welcome debate, suggestions and links in the comments section of the blog(s).

Measurement for reward

The Journal Impact Factor

Let’s start with how researchers are measured. For decades academia has lived with the ‘Publish or Perish’ mantra which has spawned problems with poor publication practices. Today the pressure to be published in a high impact journal is stronger than ever.

A journal’s Impact Factor (JIF) averages the number of citations received by a journal in a given year divided by the number of articles published in the previous two years. For example, a journal’s JIF for a given year is calculated by taking the number of citations made that year to the articles published in the journal in the previous years and then dividing by the total number of articles (including reviews and other non-scholarly content) published in that journal in those years.

The JIF is compiled by Journal Citation Reports – which is owned by a commercial company Thompson Reuters. The company announced its sale for $3.5 billion today.

This blog will not dig in any depth into the issues with the way the JIF is calculated, although there are some serious ones (see a 2006 paper I coauthored on this topic). Neither will it explore the problem of how much the JIF is gamed – from self-citations to journals insisting on a certain number of citations to publications within the same journal. Sufficient to say that each year a number of journals are removed from the index due to this type of behaviour. The record to date was in 2013, a year which saw 66 journals struck from the list. By comparison only 18 were suppressed in the most recent report.

There have been many, many criticisms of the Journal Impact Factor and its effects on scholarship. But the criticisms put forward a decade ago to the month by PLOS still ring true. One of the issues, PLOS argued, was that because Thompson Reuters does not make public the process for choosing ‘citable’ article types, this means “science is currently rated by a process that is itself unscientific, subjective, and secretive”.

Indeed last week a news article in Science and a related news article in Nature put forward exactly the same criticism. The stories referred to a paper: “A simple proposal for the publication of journal citation distributions” posted on BioRxiv. This described some comparative research undertaken to look at whether a reanalysis of the data would provide the same results as Thompson Reuters. It didn’t. The work found the citation distributions were “so skewed that up to 75% of the articles in any given journal had lower citation counts than the journal’s average number”. The authors likened using the JIF to determine the impact of a given article to ‘guesswork‘.

Jon Tennant, in a 2015 blog stated that “The impact factor is one of the most mis-used metrics in the history of academia” and proposed an Open Letter template for researchers to “send to people in positions of power at different institutions, co-signed by as many academics as possible who believe in fairer and evidence-based assessment”. Tennant in turn references Stephen Curry’s 2012 blog which opened with the statement “The impact factor might have started out as a good idea, but its time has come and gone”.

There are many more, but I am sure you get the idea.

This is recognised as such a big problem that in 2012 the San Francisco Declaration on Research Assessment (DORA) was conceived with the intent to: ‘Put science into the assessment of research’. Over 12,000 individuals and over 700 organisations have signed the declaration to date supporting the call for a “need to assess research on its own merits rather than on the basis of the journal in which the research is published”.

If nothing else, there is clearly a problem with measuring the worth of something by considering the packaging and not the item itself. But the academy continues to use the JIF and criticisms continue to come thick and fast.

Clearly something is rotten in the state of Denmark.

Ditching the JIF

In Stephen Jay Gould’s seminal book The Mismeasure of Man where he debunks the science behind biological determinism, he criticises “the myth that science itself is an objective enterprise, done properly only when scientists can shuck the constraints of their culture and view the world as it really is”. This observation is true of any metrics we apply to the valuing of research outputs. They are not objective, and not an accurate view. Any measurement tool causes its own problems.

An example of a non-JIF type of measurement is the increased emphasis on ‘excellence’ by funders and governments (the Research Excellence Framework in the UK and Excellence in Research for Australia being two examples). But ‘excellence rhetoric’ is counterproductive to good research, according to one argument which concludes that ‘excellence’ is a “pernicious and dangerous rhetoric that undermines the very foundations of good research and scholarship”.

The insistence on excellence, it can be argued, have spawned problems with reproducibility and fraud. In other words, the same problems that the JIF has caused.

There have been many other suggestions for ways to measure researchers, such as the h-index which has its own set of issues, and the Eigenfactor Score – these are only two of a myriad of options. But as the system changes, so does researcher behaviour. A clear example was in Australia when the funding mechanism moved to a simple count of research papers rather than any assessment of the value of those papers. This resulted in a marked increase in the number of papers being produced and a concurrent decrease in the overall quality as described in ‘Modifying publication practices in response to funding formulas‘.

Clifford Lynch, the Executive Director of CNI noted in his welcome talk at the JISC-CNI event held at Wadham College, Oxford last week that using alternative metrics means we start running into issues about vendor lock-in and data confidentiality.

While alternate metrics might solve the ‘valuing the article rather than journal’ issue, they bring up problems of their own. In HEFCE’s 2015 report on metrics being used in assessment in the future noted that some indicators can be misused or ‘gamed’ – with journal impact factors, university rankings and citation counts put forward as three prominent examples. The report recommended that metrics should be updated in response to their potential effects. In deciding what metrics to use, the report recommended using the best possible data in terms of accuracy and scope, and that the data collection and analytical processes should be open and transparent to allow verification. It also suggested using a range of indicators.

Financial implications

What does this emphasis on particular publication outlets have to do with Open Access? Well a great deal as it happens. It is the big blocker to widespread change. As long as we continue with this emphasis we will not get any real traction with Open Access because it locks us into an old print paradigm of academia.

Much ink has been spilt over the cost of publication and the added cost of open access (some of it mine) which includes not just the cost of the article processing charges but the burden of administering multiple micropayments.

As I have said on numerous occasions (see here and here) funders paying for hybrid open access is expensive and has not resulted in journals flipping to gold (as a transition to fully Open Access environment) despite this being a stated aim of the process. It makes sense from a publisher’s perspective not to flip journals – why, when researchers are under pressure to publish in high impact journals, and there is a new revenue stream associated with that publishing, would you kill the proverbial goose?

Indeed, a paper earlier this year argued that “Open Access has the potential to become unsustainable for research communities if high-cost options are allowed to continue to prevail in a widely unregulated scholarly publishing market.”

The problem, it can be argued, is that the infrastructure underpinning open access is ‘path dependent’ a concept proposed in 1985 which explains how the set of decisions in the present is limited by the decisions one has made in the past, even though the contextual factors shaping the past decision no longer apply. Scholarly publishing is path-dependent, some authors argue “because it still heavily depends on a few players that occupy crucial nodes in the scientific information infrastructure. In the past, these players were scientific associations, but now these players are commercial publishing companies”.

As long as the current reward system remains, the crucial nodes will not change and we are stuck.

Conclusion

So that covers some of the problems with the way we measure our researchers, and some of the financial implications of this. The next blog in this series will cover some of the issues with authorship.

Published 11 July 2016
Written by Dr Danny Kingsley
Creative Commons License