Altmetrics
In scholarly and scientific publishing, altmetrics (stands for "alternative metrics") are non-traditional bibliometrics proposed as an alternative or complement to more traditional citation impact metrics, such as impact factor and h-index. The term altmetrics was proposed in 2010, as a generalization of article level metrics, and has its roots in the #altmetrics hashtag.
Although altmetrics are often thought of as metrics about articles,
they can be applied to people, journals, books, data sets,
presentations, videos, source code repositories, web pages, etc.
Altmetrics use public APIs across platforms to gather data with
open scripts and algorithms. Altmetrics did not originally cover citation counts,
but calculate scholar impact based on diverse online research output,
such as social media, online news media, online reference managers and
so on. It demonstrates both the impact and the detailed composition of the impact. Altmetrics could be applied to research filter, promotion and tenure dossiers, grant applications and for ranking newly-published articles in academic search engines.
Over time, the diversity of sources mentioning, citing, or
archiving articles has gone down. This happened because services ceased
to exist, like Connotea, or because changes in API availability. For
example, PlumX removed Twitter metrics in August 2023.
Adoption
The
development of web 2.0 has changed the research publication seeking and
sharing within or outside the academy, but also provides new innovative
constructs to measure the broad scientific impact of scholar work.
Although the traditional metrics are useful, they might be insufficient
to measure immediate and uncited impacts, especially outside the
peer-review realm.
Projects such as ImpactStory, and various companies, including Altmetric, Plum Analytics and Overton are calculating altmetrics. Several publishers have started providing such information to readers, including BioMed Central, Public Library of Science (PLOS),[21][22] Frontiers,[23] Nature Publishing Group,[24] and Elsevier.[25][26] The NIHR Journals Library also includes altmetric data alongside its publications.[27]
In 2008, the Journal of Medical Internet Research started to systematically collect tweets about its articles. Starting in March 2009, the Public Library of Science also introduced article-level metrics for all articles. Funders have started showing interest in alternative metrics, including the UK Medical Research Council. Altmetrics have been used in applications for promotion review by researchers. Furthermore, several universities, including the University of Pittsburgh are experimenting with altmetrics at an institute level.
However, it is also observed that an article needs little attention to jump to the upper quartile of ranked papers,
suggesting that not enough sources of altmetrics are currently
available to give a balanced picture of impact for the majority of
papers.
Important in determining the relative impact of a paper, a
service that calculates altmetrics statistics needs a considerably sized
knowledge base. The following table shows the number of artefacts,
including papers, covered by services:
Website |
Number of artefacts
|
Plum Analytics |
~ 52.6 million
|
Altmetric.com |
~ 28 million
|
ImpactStory |
~ 1 million
|
Overton |
~ 11 million
|
Categories
Altmetrics
are a very broad group of metrics, capturing various parts of impact a
paper or work can have. A classification of altmetrics was proposed by
ImpactStory in September 2012, and a very similar classification is used by the Public Library of Science:
- Viewed – HTML views and PDF downloads
- Discussed – journal comments, science blogs, Wikipedia, Twitter, Facebook and other social media
- Saved – Mendeley, CiteULike and other social bookmarks
- Cited – citations in the scholarly literature, tracked by Web of Science, Scopus, CrossRef and others
- Recommended – for example used by F1000Prime
Viewed
One
of the first alternative metrics to be used was the number of views of a
paper. Traditionally, an author would wish to publish in a journal with
a high subscription rate, so many people would have access to the
research. With the introduction of web technologies it became possible
to actually count how often a single paper was looked at. Typically,
publishers count the number of HTML views and PDF views. As early as
2004, the BMJ published the number of views for its articles, which was found to be somewhat correlated to citations.
Discussed
The
discussion of a paper can be seen as a metric that captures the
potential impact of a paper. Typical sources of data to calculate this
metric include Facebook, Google+, Twitter,
Science Blogs, and Wikipedia pages. Some researchers regard the
mentions on social media as citations. For example, citations on a
social media platform could be divided into two categories: internal and
external. For instance, the former includes retweets, the latter refers
to tweets containing links to outside documents.[42]
The correlation between the mentions and likes and citation by primary
scientific literature has been studied, and a slight correlation at best
was found, e.g. for articles in PubMed.
In 2008 the Journal of Medical Internet Research began publishing views and tweets.
These "tweetations" proved to be a good indicator of highly cited
articles, leading the author to propose a "Twimpact factor", which is
the number of Tweets it receives in the first seven days of publication,
as well as a Twindex, which is the rank percentile of an article's
Twimpact factor.
However, if implementing use of the Twimpact factor, research shows
scores to be highly subject specific, and as a result, comparisons of
Twimpact factors should be made between papers of the same subject area.
While past research in the literature has demonstrated a correlation
between tweetations and citations, it is not a causative relationship.
At this point in time, it is unclear whether higher citations occur as a
result of greater media attention via Twitter and other platforms, or
is simply reflective of the quality of the article itself.
Recent research conducted at the individual level, rather than
the article level, supports the use of Twitter and social media
platforms as a mechanism for increasing impact value.
Results indicate that researchers whose work is mentioned on Twitter
have significantly higher h-indices than those of researchers whose work
was not mentioned on Twitter. The study highlights the role of using
discussion based platforms, such as Twitter, in order to increase the
value of traditional impact metrics.
Besides Twitter and other streams, blogging has shown to be a
powerful platform to discuss literature. Various platforms exist that
keep track of which papers are being blogged about. Altmetric.com uses
this information for calculating metrics, while other tools just report
where discussion is happening, such as ResearchBlogging and Chemical
blogspace.
Recommended
Platforms may even provide a formal way of ranking papers or recommending papers otherwise, such as Faculty of 1000.
Saved
It
is also informative to quantify the number of times a page has been
saved, or bookmarked. It is thought that individuals typically choose to
bookmark pages that have a high relevance to their own work, and as a
result, bookmarks may be an additional indicator of impact for a
specific study. Providers of such information include science specific social bookmarking services such as CiteULike and Mendeley.
Cited
The
cited category is a narrowed definition, different from the discussion.
Besides the traditional metrics based on citations in scientific
literature, such as those obtained from Google Scholar, CrossRef, PubMed Central, and Scopus,
altmetrics also adopt citations in secondary knowledge sources. For
example, ImpactStory counts the number of times a paper has been
referenced by Wikipedia. Plum Analytics also provides metrics for various academic publications, seeking to track research productivity. PLOS is also a tool that may be used to utilize information on engagement.
Numerous studies have shown that scientific articles disseminated
through social media channels (i.e. Twitter, Reddit, Facebook, YouTube,
etc) have substantially higher biblometric scores (downlodas, reads and
citations) than articles not advertised through social media. In the
fields of plastic surgery, hand surgery and more, higher Altmetric scores are associated with better short-term bibliometrics.
Interpretation
While there is less consensus on the validity and consistency of altmetrics,
the interpretation of altmetrics in particular is discussed. Proponents
of altmetrics make clear that many of the metrics show attention or
engagement, rather than the quality of impacts on the progress of
science.
Even citation-based metrics do not indicate if a high score implies a
positive impact on science; that is, papers are also cited in papers
that disagree with the cited paper, an issue for example addressed by
the Citation Typing Ontology project.
Altmetrics could be more appropriately interpreted by providing
detailed context and qualitative data. For example, in order to evaluate
the scientific contribution of a scholar work to policy making by
altmetrics, qualitative data, such as who's citing online and to what extent the online citation is relevant to the policymaking, should be provided as evidence.
Regarding the relatively low correlation between traditional
metrics and altmetrics, altmetrics might measure complementary
perspectives of the scholar impact. It is reasonable to combine and
compare the two types of metrics in interpreting the societal and
scientific impacts. Researchers built a 2*2 framework based on the
interactions between altmetrics and traditional citations.
Further explanations should be provided for the two groups with high
altmetrics/low citations and low altmetrics/high citations.
Thus, altmetrics provide convenient approaches for researchers and
institutions to monitor the impact of their work and avoid inappropriate
interpretations.
Controversy
The usefulness of metrics for estimating scientific impact is controversial.
Research has found that online buzz could amplify the effect of other
forms of outreach on researchers' scientific impact. For the
nano-scientists that are mentioned on Twitter, their interactions with
reporters and non-scientists positively and significantly predicted
higher h-index, whereas the non-mentioned group failed.
Altmetrics expands the measurement of scholar impact for containing a
rapid uptake, a broader range of audiences and diverse research outputs.
In addition, the community shows a clear need: funders demand
measurables on the impact of their spending, such as public engagement.
However, there are limitations that affect the usefulness due to
technique problems and systematic bias of construct, such as data
quality, heterogeneity and particular dependencies.
In terms of technique problems, the data might be incomplete, because
it is difficult to collect those online research outputs without direct
links to their mentions (i.e. videos) and identify different versions of
one research work. Additionally, whether the API leads to any missing
data is unsolved.
As for systematic bias, like other metrics, altmetrics are prone
to self-citation, gaming, and other mechanisms to boost one's apparent
impact such as performing citation spam in Wikipedia. Altmetrics can be gamed: for example, likes and mentions can be bought.
Altmetrics can be more difficult to standardize than citations. One
example is the number of tweets linking to a paper where the number can
vary widely depending on how the tweets are collected.
Besides, online popularity may not equal to scientific values. Some
popular online citations might be far from the value of generating
further research discoveries, while some theoretical-driven or
minority-targeted research of great science-related importance might be
marginalized online.
For example, the top tweeted articles in biomedicine in 2011 were
relevant to curious or funny content, potential health applications, and
catastrophe. Altmetric state that they have systems in place to detect, identify and correct gaming.
Finally, recent research has shown Altmetrics reproduce gendered biases
found in disciplinary publication and citation practices: for example,
journal articles authored exclusively by female scholars score 27% lower
on average than exclusively male-authored outputs. At once, this same
research shows 0 attention scores are more likely for male-authored
articles.
Altmetrics for more recent articles may be higher because of the
increasing uptake of the social web and because articles may be
mentioned mainly when they are published.
As a result, it might not be fair to compare the altmetrics scores of
articles unless they have been published at a similar time. Researchers
has developed a sign test to avoid the usage uptake bias by comparing
the metrics of an article with the two articles published immediately
before and after it.
It should be kept in mind that the metrics are only one of the
outcomes of tracking how research is disseminated and used. Altmetrics
should be carefully interpreted to overcome the bias. Even more
informative than knowing how often a paper is cited, is which papers are
citing it. That information allows researchers to see how their work is
impacting the field (or not). Providers of metrics also typically
provide access to the information from which the metrics were
calculated. For example, Web of Science
shows which are the citing papers, ImpactStory shows which Wikipedia
pages are referencing the paper, and CitedIn shows which databases
extracted data from the paper.
Another concern of altmetrics, or any metrics, is how
universities or institutions are using metrics to rank their employees
make promotion or funding decisions, and the aim should be limited to measure engagement.
The overall online research output is very little and varied among different disciplines.
The phenomenon might be consistent with the social media use among
scientists. Surveys has shown that nearly half of their respondents held
ambivalent attitudes of social media's influence on academic impact and
never announced their research work on social media.
With the changing shift in open science and social media use, the
consistent altmetrics across disciplines and institutions will more
likely be adopted.
Ongoing research
The specific use cases and characteristics is an active research field in bibliometrics, providing much needed data to measure the impact of altmetrics itself. Public Library of Science has an Altmetrics Collection and both the Information Standards Quarterly and the Aslib Journal of Information Management recently published special issues on altmetrics. A series of articles that extensively reviews altmetrics was published in late 2015.[68][69][70]
There is other research examining the validity of one altmetrics[4][28] or make comparisons across different platforms.[60]
Researchers examine the correlation between altmetrics and traditional
citations as the validity test. They assume that the positive and
significant correlation reveals the accuracy of altmetrics to measure
scientific impact as citations.[60] The low correlation (less than 0.30[4])
leads to the conclusion that altmetrics serves a complementary role in
scholar impact measurement such as the study by Lamba (2020) [71]
who examined 2343 articles having both altmetric attention scores and
citations published by 22 core health care policy faculty members at
Harvard Medical School and a significant strong positive correlation
(r>0.4) was observed between the aggregated ranked altmetric
attention scores and ranked citation/increased citation values for all
the faculty members in the study. However, it remains unsolved that what
altmetrics are most valuable and what degree of correlation between two
metrics generates a stronger impact on the measurement. Additionally,
the validity test itself faces some technical problems as well. For
example, replication of the data collection is impossible because of the
instant changing algorithms of data providers.[72]