Why Altmetric scores should never be used to measure the merit of scientific publications ( or ‘ how to tweet your way to honour and glory ’ )

Because journal impact factors are widely recognized as a seriously flawed means of assessing the merit of a scientific paper (Seglen 1997), and because it takes time before it is known how well cited a scientific paper will become, there is a demand for metrics that can quantify a paper’s impact rapidly after publication. One prominent recent development is that of ‘altmetrics’ which capitalize on dissemination of the work via social media. The company ‘Altmetric’ provides an articlelevel score, presented within a multicoloured badge that quantifies the extent to which the work has been picked up by various social and other media outlets, including Twitter, Facebook and blogs. This score is placed prominently alongside the abstract of every paper published in the majority of the main ecological journals. Although the Altmetric company’s website cautions that one should not read too much into these scores without digging ‘deeper into the numbers and looking at the qualitative data underneath’, it also emphasizes that ‘Altmetrics are becoming widely used in academia, by individuals (as evidence of influence for promotion and tenure and in applying for grants), institutions (for benchmarking a university’s overall performance)’, and that the Altmetric badges (showcasing the scores) ‘provide a quick and easy way of showcasing the value of your publishing program to internal and external stakeholders, such as funding institutions and editorial boards’. Indeed, increasing numbers of researchers are making use of the Altmetric scores of their work in their CVs and applications for jobs and tenure, at least when they reflect favourably on the author. If Altmetric scores are to be used as a reliable indicator of the merit of a scientific publication, then it is critical that they cannot be gamed, and that they are entirely independent of the actions of the author postpublication. To test if this is the case, I conducted a simple analysis on the first 100 papers published in the journal Ecology in 2015. For each paper I noted the Altmetric score presented alongside the paper’s abstract. Because Altmetric scores for most papers are determined in large part by how many Twitter users ‘tweeted’ about the paper, I then examined the tweets for that paper and recorded whether or not the paper had been tweeted about by its own authors, i.e., from a Twitter account that the author has primary control over (such as their personal Twitter account, or lab-group Twitter account). This analysis reveals that publications which were tweeted about by their own authors had Altmetric scores of 3.3 times greater than did the others when mean values were considered, or 4.0 times greater when median values were used (Table 1). There are two possible explanations for this outcome. The first is that through tweeting about their own work, the authors generated publicity for it that greatly elevated its Altmetric score. While it is noted on the Altmetric website that they ‘count one person as only one source’, each of the author’s ‘followers’ (who are obviously likely to be favourably inclined to the author) that re-tweet the author’s tweet, as well as the followers’ followers, are presumably all regarded as independent sources. This suggests that authors that tweet about their own work will greatly elevate their Altmetric scores simply on the basis of re-tweets especially if they have lots of loyal followers. The second possible explanation is that authors who maintain a Twitter account and who tweet about their own work are also on average better researchers whose work is more worthy of high Altmetric scores. For this to hold, it requires that those scientists whose work has the greatest impact are

Because journal impact factors are widely recognized as a seriously flawed means of assessing the merit of a scientific paper (Seglen 1997), and because it takes time before it is known how well cited a scientific paper will become, there is a demand for metrics that can quantify a paper's impact rapidly after publication.One prominent recent development is that of 'altmetrics' which capitalize on dissemination of the work via social media.The company 'Altmetric' provides an articlelevel score, presented within a multicoloured badge that quantifies the extent to which the work has been picked up by various social and other media outlets, including Twitter, Facebook and blogs.This score is placed prominently alongside the abstract of every paper published in the majority of the main ecological journals.Although the Altmetric company's website cautions that one should not read too much into these scores without digging 'deeper into the numbers and looking at the qualitative data underneath', it also emphasizes that 'Altmetrics are becoming widely used in academia, by individuals (as evidence of influence for promotion and tenure and in applying for grants), institutions (for benchmarking a university's overall performance)', and that the Altmetric badges (showcasing the scores) 'provide a quick and easy way of showcasing the value of your publishing program to internal and external stakeholders, such as funding institutions and editorial boards'.Indeed, increasing numbers of researchers are making use of the Altmetric scores of their work in their CVs and applications for jobs and tenure, at least when they reflect favourably on the author.
If Altmetric scores are to be used as a reliable indicator of the merit of a scientific publication, then it is critical that they cannot be gamed, and that they are entirely independent of the actions of the author postpublication.To test if this is the case, I conducted a simple analysis on the first 100 papers published in the journal Ecology in 2015.For each paper I noted the Altmetric score presented alongside the paper's abstract.Because Altmetric scores for most papers are determined in large part by how many Twitter users 'tweeted' about the paper, I then examined the tweets for that paper and recorded whether or not the paper had been tweeted about by its own authors, i.e., from a Twitter account that the author has primary control over (such as their personal Twitter account, or lab-group Twitter account).This analysis reveals that publications which were tweeted about by their own authors had Altmetric scores of 3.3 times greater than did the others when mean values were considered, or 4.0 times greater when median values were used (Table 1).
There are two possible explanations for this outcome.The first is that through tweeting about their own work, the authors generated publicity for it that greatly elevated its Altmetric score.While it is noted on the Altmetric website that they 'count one person as only one source', each of the author's 'followers' (who are obviously likely to be favourably inclined to the author) that re-tweet the author's tweet, as well as the followers' followers, are presumably all regarded as independent sources.This suggests that authors that tweet about their own work will greatly elevate their Altmetric scores simply on the basis of re-tweets especially if they have lots of loyal followers.The second possible explanation is that authors who maintain a Twitter account and who tweet about their own work are also on average better researchers whose work is more worthy of high Altmetric scores.For this to hold, it requires that those scientists whose work has the greatest impact are This work is licensed under a Creative Common Attribution 3.0 License.heavier users of Twitter relative to other scientists.To explore this, I used the latest (2015) Thompson-Reuters list of Highly Cited Ecologists, and determined (for all 131 of them) whether or not they maintained an active Twitter account (defined as an account that has sent at least 6 tweets in the past year).Note that unlike Thompson-Reuters' earlier Highly Cited lists, their most recent lists are not based on algorithms that favour only older researchers who might be more resistant to Twitter.Of these 131 researchers on the list, I found that 25 (or 19%) are active Twitter users.This is in line with the proportion of academics at large, and the adult population in general, that uses Twitter (Van Noorden 2014; Duggan 2015).As such, there is no evidence that those scientists whose work has the greatest impact disproportionately use Twitter, making this second explanation implausible.
Any metric system used for assessing the merit of a piece of scientific work needs to be set up to deliver the highest scores to the best work (by whatever criteria the work is being assessed) and the lowest scores to the poorest or least credible work, for example work claiming to support intelligent design or a climate change denialist viewpoint.To explore this, I obtained the Altmetric scores of four so-called 'scientific' papers supporting a climate change denialist viewpoint published in late 2014 and 2015, and compared these with the scores for the first 100 papers published in Ecology in 2015 (Table 2).All four denialist papers had Altmetric scores that were higher than 92% of all the papers published in Ecology, and all had scores that were several times higher than the median for the Ecology papers; one had an Altmetric score that was over double that of all 100 Ecology papers.This serves as clear evidence that very high scores can be attained for work that not only lacks scientific merit by any objective criteria but that actually undermines the scientific process.
While my analysis is exploratory, further interesting questions remain, particularly in relation to why some manuscripts that do not stand out in other ways nevertheless attract very high Altmetric scores.For example, many papers with very high scores also have long authorship lists, so the question emerges as to whether these high scores simply result from several authors all tweeting about the same paper, and with these tweets then being retweeted by the Twitter 'followers' of each of the authors.Another question is whether an author tweeting about the same paper multiple times can indirectly elevate its Altmetric score.As such, if an author tweets several times about the same paper, then it may be noticed (and thus retweeted) by more of their followers compared with if they tweeted about it only once.As an example, one ecological paper earlier this year was tweeted about over 90 times from just three Twitter accounts (i.e., accounts of two of the authors, plus an account dedicated to the project), and with the high Altmetric score that the paper obtained displayed prominently on the website of one of the authors.Even though the Altmetric algorithm counts each of the three Twitter accounts only once, the question remains as to the extent to which a Twitter-blitz of this type increases the number of followers that notice and thus retweet about the paper, and consequently, the impact that this has on its Altmetric score.
For any method of scoring any aspect of the merit of a piece of scientific work to be credible, it is essential that the score is entirely independent of the actions of its authors post-publication.This is clearly not the case with Altmetric scores; the simple action of an author tweeting about their own work will cause its score on average to increase between three-and four-fold.Note that this increase is just an average; the benefits accrued to an experienced Twitter user with a legion of loyal followers, or to an author of a paper that has several coauthors that also use Twitter, is likely to be several times that.Further, it is of concern that 'scientific' papers that lack credibility can nevertheless garner enormous Altmetric scores.While Altmetric scores are being very prominently displayed on journal websites where they cannot be ignored, and are finding their way into CVs and applications for funding, tenure and promotion, in their present form they do not provide a credible measure of any component of scientific merit.As such, high Altmetric scores, especially when associated with authors who regularly tweet about their own work, need to be treated with considerable caution.

Table 1 .
Analysis of Altmetric scores for the first 100 papers published in Ecology in 2015, broken down into those papers in which the authors did versus did not 'tweet' about their own publication from a Twitter account for which they have primary control.

Table 2 .
Altmetric scores for 4 recent (late 2014 and 2015) journal publications supporting a climate change denialist viewpoint, and for the first 100 papers published in Ecology in 2015.