Evaluative altmetrics: is there evidence for its application to research evaluation?

Introduction Altmetrics have been demonstrated as a promising tool for analyzing scientific communication on social media. Nevertheless, its application for research evaluation remains underdeveloped, despite the advancement of research in the study of diverse scientific interactions. Methods This paper develops a method for applying altmetrics in the evaluation of researchers, focusing on a case study of the Environment/Ecology ESI field publications by researchers at the University of Granada. We considered Twitter as a mirror of social attention, news outlets as media, and Wikipedia as educational, exploring mentions from these three sources and the associated actors in their respective media, contextualizing them using various metrics. Results Our analysis evaluated different dimensions such as the type of audience, local attention, engagement generated around the mention, and the profile of the actor. Our methodology effectively provided dashboards that gave a comprehensive view of the different instances of social attention at the author level. Discussion The use of altmetrics for research evaluation presents significant potential, as shown by our case study. While this is a novel method, our results suggest that altmetrics could provide valuable insights into the social attention that researchers garner. This can be an important tool for research evaluation, expanding our understanding beyond traditional metrics.


. Introduction
The advent of social media has led to an unprecedented expansion of the spectrum of metrics used to study scientific impact and communication (Priem et al., 2012).The first firm call for the use of these metrics was made in 2010 with the Altmetrics Manifesto, which already pointed out that "broader impact metrics could also play a role in funding and promotion decisions" (Priem et al., 2010).This potential has always been in the spotlight and its realization has been viewed as the ultimate goal (Torres-Salinas et al., 2013).This revolution has led to a whole stream of research mainly focused on finding clear evidence of later citations in the early mentions of scientific publications on Twitter, Wikipedia and other social media (Costas et al., 2015).This correlation has only been found for Mendeley readers (Thelwall, 2018a), although transparency issues associated with Mendeley readership counts have hindered its study (Thelwall, 2018b).
Thus, there is an extensive altmetric landscape misaligned with traditional bibliometric indicators, which has proven useful for understanding how science is disseminated and discussed through these communication channels.Despite Twitter's well-known problem of volatility (Fang et al., 2022b), mentions made through this social network stand out due to their abundance, having the highest presence in altmetric data aggregators (Zahedi and Costas, 2018;Karmakar et al., 2021).This has given rise to a plethora of studies that analyze the coverage of scientific literature (Fang et al., 2020;Torres-Salinas et al., 2022), map its communities of interest and topics (Robinson-Garcia et al., 2018;Schalkwyk et al., 2020;Arroyo-Machado et al., 2021) or explore the different types of engagement (Fang et al., 2021(Fang et al., , 2022a)).While fewer efforts have been devoted to studying news mentions, there is no shortage of research that delves into the nature of such mentions.For example, similarities between news outlets have been mapped based on common mentions (Ortega, 2021) and the dominance of local research in the news has been studied (Yu et al., 2022).On the other hand, Wikipedia mentions, one of the earliest altmetric sources (Nielsen, 2007), have been explored through multiple perspectives, for example to analyze their citation patterns (Maggio et al., 2017) and the differences between linguistic editions (Lewoniewski et al., 2017) or to map its collaborative knowledge structure (Arroyo-Machado et al., 2020).
The evident potential of social media activity relating to science and its communication has led to the emergence of theoretical proposals for integration of evidence from different social media (Costas et al., 2020), ultimately giving rise to a new generation of social media metrics that aim to look beyond mention counts and better understand the diversity of interactions and attention around science (Díaz-Faes et al., 2019).While this new research stream is constantly evolving and growing with new proposals, no such breakthroughs have been made in the realm of research evaluation.This is no minor concern, given that the incorporation of metrics capable of reflecting social attention and knowledge transfer is currently on the political agenda (National Information Standards Organization, 2016; European Commission Directorate-General for Research Innovation et al., 2017Innovation et al., , 2019)).However, the only advances in this field are theoretical studies that rely on the empirical evidence provided by the ample altmetric literature to formulate recommendations for the use of these metrics in evaluation (Thelwall, 2018b(Thelwall, , 2020a,b),b).Indeed, there is a complete absence of proposals with a practical approach exploring tangible applications for this evaluative evidence and raising awareness of the possibilities of these social metrics.
One of the main reasons for this tardy interest in the development of practical applications in the field of scientific evaluation is the lack of evidence and consensus regarding their potential as metrics of social attention.Since their inception, these metrics have been seen not only as an alternative to traditional citation-based indicators, which reflect academic impact, but also as potential indicators of social attention by capturing the interest or influence of science outside the scientific realm (Thelwall et al., 2013;Bornmann, 2014).However, the question of what constitutes social attention has raised many doubts, with the early years pointing to the possibility of a fragmented nature made up of different dimensions to which this variety of metrics can contribute (Priem et al., 2012).This has resulted in it being referred to as "unknown attention" in the absence of such concreteness and evidence (Bornmann et al., 2019).Nevertheless, these areas The terms social attention and scientific impact are often used indi erently in the literature.But it is more appropriate to refer to attention as it denotes broad public interest while impact carries more transformative connotations (Sugimoto, ).
of attention have become increasingly delimited.For example, it is now clear that a patent citation differs from a Facebook mention in that the former reflects a commercial interest while the latter reflects a public interest (Thelwall, 2020a).Following this same premise, even applied proposals have arisen such as the InfluScience research project, which analyzes the social, media, political and educational influence of Spanish researchers through mentions on Twitter and in news outlets, policy reports and Wikipedia (InfluScience, 2022).Therefore, if altmetrics wish to move forward they must be nourished by historical and proven proposals in the field, such as those of evaluative informetrics (Moed, 2017).This means putting an end to "bean counting" (Ràfols, 2019) and importing multidimensional and contextual practices into the altmetrics domain.To this end, it is necessary to create concrete and precise evaluative frameworks that improve the meaning and usefulness of such indicators.Some proposals to give meaning to altmetrics have been oriented toward detecting audiences, identifying local attention, measuring engagement or studying the profiles of the actors.Thus, in light of the need to apply altmetrics in real decisionmaking contexts and the fact that there are already sufficiently tried and tested sources (Twitter, news, Wikipedia), in which the largest volume of altmetrics mentions are usually concentrated (Torres-Salinas et al., 2023), the general objective of this paper is to present a "portfolio" of significant case studies of altmetrics and university policy decision-making context based on the principles of evaluative bibliometrics established in the latest publication by Moed (2017).Its aim is to be of use in scientific policy and decision-making processes, especially in the context of universities (Torres-Salinas et al., 2016), which are increasingly concerned with the evaluation of social attention (Bornmann et al., 2019).More specifically, the following specific objectives have been established: • To individually explore the three altmetric sources (Twitter, news and Wikipedia) to propose practical solutions for identification of local attention and different audiences and obtain metrics that reflect engagement with mentions and characterize the profile of social actors.
• To conduct a research evaluation case study that highlights the potential of altmetrics at author level.

. Methodology
As this is an exploratory study that seeks to delve into the different types of altmetric evidence for evaluation at author level, we have used a small sample of publications associated with a controlled environment, but which captures the attention of the three social media studied (Twitter, news and Wikipedia).After reviewing the InfluScience2 ranking and the altmetric ranking of Spanish researchers and institutions, the disciplines of Environmental Science and Ecology were identified as the most suitable, because in addition to high altmetric attention scores they also have a balanced presence in the three sources studied.The decision was therefore made to use as the sample publications from the Environment/Ecology ESI field published by researchers https://ranking.influscience.eu/(accessed February , ).

FIGURE
Summary of the methodological process and transformation of altmetric data for author-level evaluation.
of the University of Granada (UGR), on the basis that it is one of the Spanish universities with the most experience and interest in this field.
The data retrieval process carried out on 7 February 2023 is summarized in Figure 1.Firstly, bibliographic records of the production of researchers at the University of Granada in the Environment/Ecology ESI field published between 2012 and 2021 were retrieved using InCites.A total of 1,959 publications were retrieved, of which 1,784 were articles (91%) and 135 were reviews (7%).It was not filtered according to any document type as the typologies that usually receive the most citations (articles and reviews) are not necessarily always those that also receive the most altmetric attention (Sugimoto et al., 2017).The authors were then disambiguated using the algorithm proposed by Caron and van Eck (2014), resulting in a total of 1,869 researchers affiliated with the University of Granada.Next, all the altmetric mentions of these publications from the three selected social media were retrieved by querying Altmetric.comwith the DOI of the publications, identifying 1,047 publications (53%) with at least one mention in any altmetric source.Subsequently, the metadata of these mentions were complemented using additional sources such as Twitter API and Wikimedia REST API.All the Python and R scripts with data processing are available at the following GitHub repository: https://doi.org/10.5281/zenodo.7740950.
Our proposal thus focuses on the exploration of altmetric indicators selected for their use like evaluative bibliometrics, which as a whole will be capable of offering a much clearer picture of the different kinds of attention that science receives.Accordingly, attention from Twitter is considered as social interest, attention from news as media and attention from Wikipedia as educational (Torres-Salinas et al., 2022).The consideration of Wikipedia as an educational source can be seen in the fact that it is one of the most used educational resources by students (Selwyn and Gorard, 2016), having a positive impact on their academic performance (Meseguer-Artola et al., 2020), so its mentions can be viewed as a reflection of educational interest.Although each of the social media is approached separately, in all cases we aim not only to identify the different types of audiences who make mentions of these publications, but also several metrics that allow us to characterize the mentions according to three dimensions: • Local attention: metrics associated with the number of local mentions.
• Mention-engagement: metrics associated with the mention that can reflect its outreach.
• Actor-profile: metrics associated with the actor who made the mention that can reflect the actor's influence and potential community.
At each stage of this proposal, the recommendations made by Thelwall (2020a) are taken into consideration to avoid common problems when dealing with altmetric data to measure social attention.Firstly, each social media source is considered separately to exploit their unique nature, as well as their specific attention, avoiding treating them all identically or recurring to aggregate indicators (Table 1).Secondly, a homogeneous sample of publications is used from the same research field, in a specific period and with authors who belong to the same institution.Thirdly, the altmetric sources chosen are those with the most mentions, with the aim of reducing the problem of zeros.More detailed and specific information can be found in the individual case sections.
. Evaluating social attention through twitter . .Twitter metadata retrieval and processing A total of 12,413 Twitter mentions of Environment/Ecology publications were retrieved from Altmetric.com.Firstly, the Twitter ./frma. .API was consulted to retrieve the metadata of the tweets (language, type of tweet and total favorites and retweets) and the tweeters (followers, friends and total tweets) that posted them.Due to inconsistencies in the Twitter data, after the query the tweet dataset was reduced to 11,952 tweets (96.29%) posted by 7,173 tweeters.
In the next step, the tweeters were classified to differentiate between different types of audiences.The tweeters were matched using the open dataset of Mongeon et al. (2022), which identifies nearly 500,000 researchers' Twitter accounts.As a result, a total of 2,541 researchers (35.42% of the total) were identified in our dataset.Secondly, we queried the API of Botometer v4 (Sayyadiharikandeh et al., 2020), the machine learning tool for identification of bot accounts in Twitter.This predictive model returns a score for each tweeter known as a botscore, which reflects how likely it is that a tweeter is a bot based on its activity patterns and the information it provides.The botscore ranges from 0 to 5, where 0 indicates behavior more associated with a non-automated account and 5 reflects bot-like behavior.The threshold is the most problematic issue, as there is no standard measure and it depends on the researchers (Yang et al., 2022).In this case, due to the distribution of the botscore (see Supplementary Figure 1) a threshold of four was set, thus detecting 289 of the tweeters (4.03%) as bots.The remaining 4,343 tweeters (60.55%) were classified as social audience.It is possible to identify more types of audiences on Twitter, such as journals (Nishikawa-Pacher, 2023), but in this case their presence is so small and in this initial approximation we only consider the three previous ones.
After classifying the tweeters by audience, the tweets published in Spanish were identified as local mentions.Next, the mentionengagement and actor-profile metrics were obtained.At the mention-engagement level the tweets and retweets were separated, a practice previously employed in altmetric studies to reduce all mentions to those that reflect the most evidence of social attention, i.e., non-retweets (Arroyo-Machado et al., 2021).Secondly, for each Because of the limitations in the location data of the tweeters' profiles (it is not provided by all tweeters and is a non-standardized field), the tweet language has been used as a proxy for the identification of local tweets.
tweet the number of retweets and favorites were used as metrics of engagement, both of which are not only widely used but have previously been shown to be useful in this regard (Fang et al., 2022a).It should be noted that both metrics do not apply to retweets, as they reflect the values corresponding to the original tweet that has been retweeted but not to the retweet per se.At the actor-profile level, the followers were used as a potential metric of the community reached by their activity, friends as a metric of the community with which they interact and total tweets as the volume of their activity in this social media.

. . Applying twitter for research evaluation
Table 2 summarizes all these metrics by audience type (bots, science or society) highlighting how each of them has different behavior.Bots have the highest average number of tweets (41,796.96)followed by friends (1,833.9),although they have the fewest average number of followers.Accordingly, despite their intense activity they impact a smaller number of tweeters than social actors, who have the largest community with an average of 4,811.68 followers.Most of the mentions also correspond to this latter group, with 6,562 tweets mentioning 6,595 publications.It is also noteworthy that for both the science and society audiences 66% of their mentions are retweets, although this value drops to 15.53% for bots.The latter are also the least engaged, while science is the most engaged with an average of 2.51 retweets and 7.59 favorites.
The value of these metrics can be seen at publication level, as they provide a better contextualization of the social attention captured through Twitter. Figure 2 shows how, apart from the number of times a paper is mentioned, there are multiple dimensions that give a broader view of social attention and the type of audience it has impacted.The paper by Seibold et al. (2021) has been marked in red and the paper by Lembrechts et al. (2020) in yellow to better exemplify this.In terms of the interest they have aroused in the science audience, both achieved a very similar number of mentions with the latter being slightly higher.However, the average number of times the tweet in which they are mentioned is marked as a favorite or retweeted is much higher in the first case, thus resulting in greater engagement in this community.This interpretation is inverted when looking at the society audience, as it is the second paper that remains behind in mentions but achieves a higher engagement, even though the potential community of the first paper is considerably higher.
These differences are also evident at author level, revealing different social attention profiles.Figure 3 shows the metrics of two researchers with an unequal number of mentions, the first of them being one of the most mentioned of the group of researchers analyzed.The percentage of retweets and the percentage of tweets in the local language are not very different, although both are slightly higher in the case of Castro.However, the differences become palpable when looking at the audiences.While Castro has a greater social audience, Reche has a greater audience within the scientific community.These differences in audience are reflected in the attention and outreach of the  tweets.Firstly, the potential community of Castro considered as the average number of followers of the actors mentioning his publications is much higher, as well as the average number of favorites received by the tweets mentioning his publications.This gap is also appreciable in the average number of retweets, although it is not as striking.Accordingly, apart from their differences in terms of mentions in raw numbers, these two authors have an impact on very different audiences and with equally different engagements.
. Evaluating media attention through news outlets . .News outlets metadata retrieval and processing In the case of Environment/Ecology publications, news mentions are the most extensive after Twitter, with a total of 1,513 mentions retrieved from Altmetric.com.However, it is necessary

FIGURE
Comparison of audience and social attention captured on Twitter of two authors from the University of Granada who publish in the field of Environment/Ecology (metrics upper limits correspond to the highest value of each metric for authors with five or more papers with mentions on Twitter).
to highlight a difference with respect to other sources: the actors are completely decentralized.Instead of having a single platform where all the actors move and interact, as for example in the case of tweeters on Twitter, each of the news outlets is an independent website.In fact, the tracking process by Altmetric.com is more biased than with other altmetric sources because it is based on a curated list of news outlets.Accordingly, despite the lack of transparency in the selection of media, Altmetric.comnot only offers greater coverage than other aggregators (Ortega, 2018), the accuracy of identification comes before exhaustiveness (Fleerackers et al., 2022), thus achieving highly useful mentions with limited noise.Aside from the strengths and weaknesses of this altmetric source, for metadata retrieval we decided to use the Site Traffic API, which returns a collection of metrics for a web domain related to its web traffic and classifies them by subject categories.
As was the case with Twitter, the initial number of mentions was reduced after the data was processed.This was due to problems with the Altmetric.comdata, as 139 mentions did not include the link to the news story and therefore the Site Traffic API query could not be queried.Similarly, for 14 of the domains it was impossible to retrieve information from the API, so the number of mentions finally analyzed was 1,324 (87.5%) made by 585 different websites.It is also noteworthy that 471 mentions included shortened URLs, a problem that had to be solved before querying the Site Traffic API.Once the data had been retrieved from the API, we made use of the subject category it offers for each website, which is associated with a classification consisting of more than one level.For example, the root category "health" has the subcategories "medicine" and "women's health, " among others.That is why in our case, given the volume of different subcategories in the study, the decision was made to reduce the subcategories to their 16 root categories.Furthermore, to make an even broader division these root categories were classified into two blocks: the root category "news and media" and the rest (e.g., arts https://zylalabs.com/api-marketplace/data/site$+$trac$+$api/ (accessed February , ).
In relation to the API metadata used, news outlets for which Spain was the country with the highest volume of traffic were identified as local.For the engagement metrics, the raw number of visits, the average number of pages of the website accessed by visitors and the average number of seconds spent on the website were selected.To characterize the news outlets, we used their position in the global ranking and their bounce rate, the latter being a well-known measure in the SEO and web industry that indicates the percentage of visitors who leave the website without clicking on it, indicating a lack of interactivity.Both metrics should be read with caution, as the lower the value, the higher the relevance.

. . Applying news outlets for research evaluation
Table 3 summarizes all metrics by audience type.Most of the websites belong to mainstream media, with 351 news outlets responsible for 710 of the mentions (53%), while specialist media account for 234 news outlets including 614 mentions (46%).While most of the mentions of local news outlets are from mainstream media, the science and education specialist media have the highest percentage of local mentions (21%).In terms of engagement metrics for mentions and characterization of news outlets, the mainstream media are the most prominent.However, this comparison varies when looking at certain categories, such as the 20 news outlets for "computers, electronics and technology" and the 18 for "finance, " which for some of the metrics have the best values.All these differences show how the media attention is highly different depending on the news outlets.
This multidimensionality in media attention is highlighted in Figure 4, which compares the news mentions of publications with the average bounce rate, visits and visit time of the news outlets that make such mentions, further differentiating between the two ./frma. .In both cases the mention counts are very similar, the former with a slight advantage in specialist media and the latter in mainstream media.However, while the latter is mentioned in mainstream media with a substantially higher number of visits and average visit time, in specialist media the differences are not very noticeable.Something similar occurs with the bounce rate.Accordingly, while both are mentioned in specialist media with very similar relevance, the mainstream media that pay attention to both are completely different, with those that mention the paper by Rodríguez-Barranco et al. (2021) being among the most relevant.At author level, these differences are even more noticeable and comprehensible.Figure 5 includes comparisons of two authors with a similar number of mentions, with Moleón having a slightly higher value.The audiences of the two authors are different; Cariñanos attracts more attention from the specialist media than the mainstream media, the exact opposite of Moleón.Similarly, the publications by Moleón have noticeably more local media attention than those of Cariñanos.This is also reflected in the average number of visits to the news outlets, which is higher in the case of Moleón, although the bounce rate is slightly lower.It can thus be concluded that while Cariñanos has more of a specialist and international media attention profile, Moleón attracts more attention from local and more visible news outlets.
. Evaluating educational attention through wikipedia . .Wikipedia metadata retrieval and processing In the case of Wikipedia, the mentions retrieved from Altmetric.comare the least numerous, with a total of 79 mentions.However, Wikipedia data is the most accessible of the three media studied as it is completely open and has multiple access points, ranging from data dumps to APIs (Arroyo-Machado et al., 2022).Similarly, the range of metrics that can be used to characterize the attention and engagement of mentions on Wikipedia is diverse.The Wikimedia REST API, .

FIGURE
Scatter plot of Environment/Ecology ESI field publications comparing the number of mentions on news outlets and the average indicators of visits, time and bounce.

FIGURE
Comparison of audience and media attention captured on news outlets for two authors from the University of Granada publishing on Environment/Ecology (metrics upper limits correspond to the highest value of each metric for authors with three or more papers with mentions on news outlets).
XTools API and ORES API have all been used to contextualize these mentions.
As in the previous cases, during the metadata retrieval process five mentions were eliminated as their pages were no longer available.To identify the audiences we used ORES, an artificial intelligence application which detects vandalism in edits of pages and can also predict a topic for the page according to a taxonomy.In this case, as was done for the news, we selected only the top level of this classification (Culture, Geography, History and Society and STEM).Moreover, since this option is not currently available in all languages, in those cases where a topic category could not be predicted because of the language, the English version of the page was used to identify the topic.Since not all pages have an English Wikipedia version, the language could not be identified for six of the pages.Meanwhile, for the identification of local mentions, all those coming from pages relating to edits of languages spoken in Spain were marked as such, in this case the Spanish and Catalan Wikipedia.Regarding the engagement metrics, the page views and number of edits of the Wikipedia pages that make the mentions were used.For the characterization of the pages, the number of words contained on the page, the number of other languages in which the page is translated and the total number of references made were used.These metrics have already demonstrated their potential in the past to capture attention and characterize the profile of Wikipedia pages (Mittermeier et al., 2021). .

. Applying wikipedia for research evaluation
Table 4 summarizes all metrics by audience type.Among the different audiences, the most prominent is STEM with a total of 44 pages (69%) and 51 mentions (69%).When analyzing the engagement of the different types of audiences, it is precisely this same audience that has the highest average number of page views, although the pages with the highest average number of edits are those of History and Society.This suggests that while the former attract the most interest, it is the latter that are the most active in terms of content.This is reinforced by the fact that the History and Society pages are the ones with the largest number of words, translations and references, indicating that they are pages with a high level of content and development of the topic they deal with.This profile is similar to that of the Culture pages, although they are less international.In general terms, the existence of different audiences with different profiles is once again appreciable.
The different perspectives of the proposed metrics for the contextualization of educational attention on Wikipedia and its main audiences are highlighted in Figure 6.Two publications have been highlighted to exemplify these differences, with the paper by Gottfried et al. (2012) marked in red and the letter by Cortés-Sánchez et al. (2019) in yellow.Both are mentioned by the two main audiences (History and Society, and STEM).The paper has more mentions in History and Society pages, which, despite not having a very high average number of words, are among the main ones in terms of translations and page views.Meanwhile, the letter receives the same number of mentions in STEM pages as the paper but it is mentioned in pages with a higher number of words, translations and, above all, page views.This shows that the attention received by both audiences is unequal, with greater relevance in the History and Society pages that mention the paper, as well as in the STEM pages that mention the letter.
Finally, two authors are compared again to demonstrate the potential of these metrics (Figure 7).Each of them has a similar number of mentions of their publications in Wikipedia, but their contextualization reveals relevant insights into this attention.Firstly, the audience of Fernández is mostly History and Society, with most mentions coming from non-local pages.The opposite holds true for Molero, whose mainly STEM audience has more local attention.Such international and local interest may be influential in terms of page views, development of content and translations, as Fernández is significantly superior in all these aspects.

. Discussion
This paper provides a framework for applying altmetrics in a meaningful way in the context of university evaluation for decision-making.Twitter has been used for the study of social attention, news for media and Wikipedia for education.Each social media source has been approached independently, but seeking in all cases to identify the different audiences, local attention, engagement metrics and characterization of the actors responsible for these mentions.One of the most relevant contributions is that it illustrates in a practical way how indicators can be combined with unique sources of information to allow contextualization of attention.More specifically, it shows how tools not commonly used in altmetric studies such as Botometer and Site Traffic API  can qualify and give meaning to metrics.In this sense, another valuable contribution is the link made between altmetrics and the practical postulates or principles of evaluative bibliometrics.This places the paper in line with the contributions of Henk F. Moed, who indicated the need for large informetric datasets to offer more robust metrics.He also mentioned the need for extra-informetric factors to determine the wider social context (Moed, 2017).We believe that this work sheds light on how context metrics can also be applied to altmetrics.This study is also unique in terms of its applied aspects, as it also follows the recommendations made by Thelwall (2020a) to avoid the well-known problems and overcome the main limitations when studying social attention.That is why each social media channel has been individually addressed and explored to take into account their strengths and weaknesses.This approach is diametrically opposed to the generation of aggregate metrics, which can provide an initial approximation of general social attention but also ignore many of the dimensions of the social media involved by establishing random criteria for their aggregation (Gumpenberger et al., 2016).Similarly, we also avoid evaluative malpractice by considering that what these metrics reflect is attention, contextualizing that attention, and ultimately providing useful and easily interpretable tools (Moed et al., 1985).Concerning these practical tools, our proposal advocates the generation of publication-level and authorlevel dashboards that offer a complete overview of the attention to be measured, clearly separating the different instances of social attention and always considering the research field in which they are evaluated.This is a point to consider in the development of applications that aim to integrate altmetrics into researcher portfolios, an unexploited domain with a limited number of proposals, one of the more noteworthy ones being ImpactStory (Priem and Piwowar, 2012).
Having demonstrated the potential and usefulness of altmetrics in the field of scientific evaluation, the proposed framework could be applied in evaluative contexts for decision-making by institutions and evaluation agencies.Firstly, continuing with the case study used in this paper, there are several possible scenarios for its potential use at institutional level at the University of Granada.It could be used by the Social Council of the University of Granada, a governing body that represents the interests of society and the relationship between the university and its surrounding environment.This university body could use altmetrics to evaluate the social attention and visibility of its researchers when awarding prizes and grants.Similarly, the International Research Projects Office (OFPI), the unit for advice, dissemination, promotion and management of international research projects at the University of Granada, could use altmetrics to evaluate the social attention of projects and prepare monitoring reports.Secondly, in terms of evaluation agencies the ANECA (National Agency for Quality Assessment and Accreditation of Spain) should be highlighted.Among its accreditations is the 6 Year Transfer, which recognizes the transfer of research activity to social agents and companies over a 6 year period.This accreditation provides an ideal framework for the use of altmetrics to facilitate the evaluation of applications.
This proposal and the related research are not free of limitations.Firstly, the well-known and inherent limitations of altmetrics stand out, especially the skewed distribution of data by social media.Secondly, news mentions may be biased, as Altmetric.comuses a curated list of news outlets for tracking.
machine learning techniques and datasets derived from their application have been used to detect audiences on Twitter and Wikipedia, meaning that some actors may not be accurately identified.Fourthly, the mention-engagement and actorprofile metrics are as of February 2023, so they may fail to contextualize the impact at the exact time the mention was made.Fifthly, while this methodology is fully reproducible, both the altmetrics data used and some of the complementary sources used are not open, which can be a barrier to largescale implementation, especially in the case of Twitter and news outlets.Finally, this proposal only considers a small subset of social attention metrics concerning three dimensions (local attention, mention-engagement and actor-profile), but it is possible to consider several more, such as self-dissemination, which is of particular interest in the case of Twitter (Ferreira et al., 2021).
We wish to conclude this paper by calling for the use of altmetrics in evaluative contexts.Although these indicators are a substantial improvement to bean counting, they must be used in addition to peer reviews as bibliometric evaluations have done since the 1980s, an approach that the European Union also encourages in its recommendations (European Commission Directorate-General for Research Innovation et al., 2017).It is time to move away from talking about alternative metrics (Buschman and Michalek, 2013) and incorporate them into the evaluation context by offering precise and practical solutions.
The percentage of publications with at least one altmetric mention is not far o the results of other studies (Arroyo-Machado et al.,) and is strongly influenced by the research field (Torres-Salinas et al.,).

FIGURE
FIGUREScatter plot of Environment/Ecology ESI field publications comparing the number of mentions on Twitter and the average indicators of favorites, likes and followers.
main audiences.Two papers have been highlighted to exemplify such differences, with the paper by Carlson et al. (2020) marked in red and the paper by Rodríguez-Barranco et al. (2021) in yellow.

FIGURE
FIGUREScatter plot of Environment/Ecology ESI field publications comparing the number of mentions on Wikipedia and the average indicators of views, words and translations.

FIGURE
FIGUREComparison of the audience and educational attention captured on Wikipedia of two authors from the University of Granada who publish on Environment/Ecology (metrics upper limits correspond to the highest value of each metric for authors with or more papers with mentions on Wikipedia).
TABLE External contextualizing sources used by social media and objective.
TABLE Twitter audience reached by the publications in the Environment/Ecology ESI field published by researchers at the University of Granada.
TABLE News outlets audience reached by the publications in the Environment/Ecology ESI field published by researchers at the University of Granada.
TABLE Wikipedia audience reached by the publications in the Environment/Ecology ESI field published by researchers at the University of Granada.