A Pragmatic Master List of Action Verbs for Bloom's Taxonomy

Bloom's Taxonomy is an approach to organizing learning that was first published in 1956. It is ubiquitous in UK Higher Education (HE), where Universities use it as the basis for teaching and assessment; Learning Outcomes are created using suggested verbs for each tier of the taxonomy, and these are then “constructively aligned” to assessments. We conducted an analysis to determine whether there is consensus regarding the presentation of Bloom's Taxonomy across UK HE. Forty seven publicly available verb lists were collected from 35 universities and textbooks. There was very little agreement between these lists, most of which were not supported by evidence explaining where the verbs came from. We were able to construct a pragmatic “master list” of action verbs by using a simple majority consensus method. We were also able to construct a master list of commonly recommended “verbs to avoid.” These master lists should be useful for anyone tasked with using Bloom's Taxonomy to write Learning Outcomes for assessment. However, our findings raise broader questions about the evidence base which underpins a common approach to teaching and assessment in UK HE and education generally.


INTRODUCTION
Learning Outcomes are a starting point for education at many levels in many countries. They are a statement of what students should be able to do by the end of their learning, and form the basis for how students are assessed (Biggs, 1996). In Higher Education (HE), in the United Kingdom (UK), it is a requirement of university accreditation to have learning outcomes mapped to levels of learning (QAA, 2014).
Part of the origin of the concept of Learning Outcomes is Bloom's Taxonomy. First published in 1956 (Bloom et al., 1956) and revised in 2002 (Krathwohl, 2002), the motivation for the taxonomy was a desire to define learning and assessment in an observable, measurable way. This was in contrast to the perceived practice of the time; using "nebulous terms" to characterize the aims of teaching, for example for learners to "understand" or to "comprehend" or to "internalize" knowledge. The genesis of the taxonomy can be summarized in this quote from page one of the original publication: "what does a student do who "really understands" which he does not do when he does not understand?" (Bloom et al., 1956, 1) Critical to writing effective Learning Outcomes is the use of specific and measurable verbs, avoiding verbs that are unobservable or unmeasurable and thus cannot be objectively assessed. For example a "good" Learning Outcome for an introductory research methods class might be to "list the main research methods used in [a discipline], " whereas a "bad" learning outcome might be to "know the main research methods. . . ." According to the original version of the taxonomy, as expertise develops, the learner moves through a series of hierarchical steps, from "Knowledge, " through "Comprehension, " "Application, " "Analysis, " "Synthesis, " and "Evaluation." Thus, by the end of a program of study we might want students to "evaluate the use of research method X to test hypothesis Y." The taxonomy was designed to form the basis for assessment as well as teaching. The original taxonomy contained numerous sample test items designed for use by teachers, mapped to the different levels of the taxonomy (Bloom et al., 1956), whereas the revised taxonomy was designed to emphasize more the use of the taxonomy in marking assessments, for example in scoring rubrics (Anderson, 1999).
The taxonomy has become near ubiquitous in educational theory and practice across many countries. A Google Scholar search for "Bloom's Taxonomy" (April 2020) returns over 29,000 results, and indicates that the original taxonomy (Bloom et al., 1956) has been cited over 34,000 times, with over 19,500 citations for the revised taxonomy (Krathwohl, 2002). A search for "Blooms taxonomy" "assessment" returns 21,700 results.
Criticism of Bloom's taxonomy has been published for decades (e.g., Stedman, 1973). Much of the criticism arises from the perception of the taxonomy as a simplistic, blunt instrument, particularly with regards to so-called higher-order learning and thinking (Ormell, 1974). Concerns have also been raised regarding the underlying epistemology and philosophy (Pring, 1971;Sockett, 1971). A major criticism of the taxonomy is that it is not aligned to current evidence about how, and why, people learn. For example, Bloom and co were clear that the taxonomy was hierarchical, that "the objectives in one class are likely to make use of and be built on the behaviors found in the preceding classes" (Bloom et al., 1956). Even at a basic level this is troublesome. For example, in the revised taxonomy "understanding" precedes "analysis" and "application." It could easily be argued that understanding comes from analysis and application rather than the other way round. The original taxonomy makes it clear that students are expected to perform better on assessments that are mapped to the lower tiers of the taxonomy, yet faculty show only a modest ability to map exam questions on Bloom's Taxonomy, even when the taxonomy is collapsed into three tiers (Karpen et al., 2017;Dempster and Kirby, 2018).
Bloom and co recognized many of these problems which would later be leveled as criticisms. For example, the overlap between the different classifications, and the fact that two students demonstrating the same observable behavior in an assessment may have arrived at that behavior in completely different ways, representing different types and even different levels of learning. One example quoted is as follows; "For example, two students solve an algebra problem. One student may be solving it from memory, having had the identical problem in class previously. The other student has never met the problem before and must reason out the solution by applying general principles, We can only distinguish between their behaviors as we analyze the relation between the problem and each student's background of experience." (Bloom et al., 1956) Despite these criticisms, the taxonomy remains near-ubiquitous in UK Higher Education, although the format of the taxonomy is often a considerably simplified version of the 216 page original. As we describe below, the websites of many UK universities contain an image of the hierarchical taxonomy in some form, normally a triangle, along with guidance about how to write Learning Outcomes based on the taxonomy. The forms in which the taxonomy appears vary considerably, but most include lists of verbs aligned to each step of the hierarchy. The verbs themselves appear to be derived, originally, from the subheadings of the tiers in the original and revised taxonomies. The verbs at the lower end of the hierarchy tend to be associated with assessments that might be used to test factual knowledge; "list, " describe, " "identify." Those at the higher end tend to be associated with assessments of "higher order thinking, " for example "appraise" or "evaluate." Given the age and ubiquity of the taxonomy, it seems reasonable to ask whether it is consistent. If multiple universities are basing teaching and assessment on learning outcomes mapped to the taxonomy; are they asking for the same thing? An analysis of the verb lists aligned with Blooms Taxonomy shown on 30 different educational websites from the USA found that there was very little agreement between the versions of Bloom's Taxonomy found, i.e., verbs which were suggested as belonging to one tier of the hierarchy on one version of the taxonomy were found, on a different list, to be associated with a different tier. The degree of disagreement between the different versions of Bloom's was considerable. Not a single verb was assigned to the same tier by all 30 lists. Three verbs (choose, relate, select) appeared in all six tiers, depending on which list was consulted (Stanny, 2016).
Here we repeat and expand the work of Stanny, in the context of UK Higher Education. Having found similar results, we also attempt to salvage something useful from the current inconsistencies of Blooms Taxonomy, by applying a pragmatic philosophy and research method. Pragmatic research prioritizes the undertaking of research that is practically useful (Feilzer, 2010), choosing the most appropriate methodology to address the research question(s) (Creswell, 2003). The knowledge that results from pragmatic research is valued for how useful it can be to address real world problems, that affect people (Duram, 2010). This methodology is often contrasted with approaches which prioritize other aspects of the research process, such as the definition of the epistemological position taken in a research activity.
In this paper then the research questions we seek to address are 1. How consistent is the presentation of Bloom's taxonomy to the UK Higher Education Sector by the websites of Universities and other stakeholders in the sector that present the taxonomy? 2. Can we identify a useful consensus position of the verbs identified within the taxonomy?
In line with the pragmatic approach, the primary stakeholders for whom we intend the findings to be useful are teaching staff responsible for writing Learning Outcomes, with follow-on value to their students and universities.

METHODS
A previous project (Ransome and Newton, 2017) identified the textbooks most commonly recommended to academics taking postgraduate certificates in Higher Education; the basic "teacher training" programmes currently used in UK HE. Of the six most commonly recommended books on general higher education, three included a version of Bloom's taxonomy and a verb list (Fry et al., 2003;Butcher et al., 2006;Biggs and Tang, 2011). Stanny (2016) identified verb lists using a simple Google search for the string "action words for Bloom"s taxonomy." To restrict our analysis to UK Higher Education we conducted a Google Search for the terms."ac.uk" and "Bloom's Taxonomy." We then included verb lists from university websites where the taxonomy was used as part of guidance for writing learning outcomes, or some other way or organizing or planning learning. This approach returned a total of 47 verb lists identified from 35 different sources (some sources included multiple lists). Of the 35 sources, 31 were UK Universities, 3 were the aforementioned textbooks and the final one was the UK Higher Education Academy, now called Advance HE, a professional body for academic teachers in UK Higher Education. We did not include search results that were about Bloom's taxonomy itself, for example research that cited the taxonomy. We only included verb lists that had six tiers from Blooms taxonomy, either the original or the revised (in addition to the 47 analyzed we also found 3 that combined the two 6-tier taxonomies into a 7-tier taxonomy, and two which used a five-tier list). Where a university linked to an external site with multiple lists, we transcribed only the first list.
Of the 47 lists, there was little consistency in terms of whether they used the original taxonomy, the revised taxonomy, a combination of the two, or a hybrid of the two. Thus, as in the work of Stanny (2016), we considered both the original and the revised taxonomy together. The 47 lists were transcribed into a single excel spreadsheet. Some sources included a list of "verbs to avoid" and these were also transcribed. The transcription of each list was rechecked by at least one author. Each source was examined to determine whether it directly cited the original version of Bloom's taxonomy (Bloom et al., 1956), or the revision (Krathwohl, 2002) or some other source explaining how the list of verbs was arrived at. This was also rechecked by at least one author.
Terms were rationalized into agreed verbs meanings between lists, for example "be familiar with" and "familiarize" were both rationalized to "familiar." This rationalization was not performed where both versions appeared in the same list (e.g., one list included "solve" and "solution") or where there appeared to be an error in the original list that could not be simply corrected (e.g., one list proposed the verb "or recount"). Unnecessary prefixes or suffixes were also removed, for example "have a good grasp of " was rationalized to "grasp." UK English was used throughout (e.g., memorize was changed to memorise). These changes were agreed by all three authors.

Unique Verbs
A total of 401 unique verbs were contained across the 47 lists. The full list of sources and verbs is shown in Appendix 1. Many verbs appeared in multiple lists and across multiple tiers of each list. Two hundred and fifty one unique verbs appeared in only one tier. These were distributed as follows; 43 for the Knowledge tier, 30 for Comprehension, 45 for Analysis, 54 for Application, 52 for Synthesis and 27 for Evaluation. Of the remaining 150 verbs, 71 were present in two tiers, 46 in three tiers, 24 in four tiers, 5 in five tiers. Four verbs (select, explain, relate, arrange) appeared across all six tiers of the taxonomies. Two of these (select and relate) also appeared across all six tiers in the analysis of US sites undertaken by Stanny, along with the verb choose (Stanny, 2016).

Unique Verbs Within and Across Tiers
To determine whether there is any consensus regarding the format of Bloom's Taxonomy, we examined the frequency with which one-tier verbs appeared within the tiers. Not one of the 251 one-tier verbs appeared in all 47 lists. The most common was "list, " which appeared in the "Knowledge" tier in 43 of the 47 lists. Only 10 of the 251 one-tier verbs appeared in more than half (24+) of the lists, and none of these were in the top two tiers of the taxonomy. In contrast, 214 (85%) of the one-tier verbs appeared in 5 or fewer of the lists, suggesting that most of the verbs which appeared in only one tier were very uncommon and potentially newer, perhaps explaining why they only appeared in one tier. Eyeballing the list appeared to confirm this-these verbs included terms like "tweet, " "google, " wiki build, " "film, " and "video blog"; terms which are anchored in a particular technology rather than the underlying learning. Considering that both the original taxonomies proposed some sort of overlap between tiers we relaxed the analysis to include verbs that were included across two tiers did not add much in terms of identifying consensus; only 6 (8%) of the 71 two-tier verbs were in more than half the lists.

Master List of Verbs
Due to the limited lack of agreement between lists, we applied a simple majority consensus method to the construction of a master list. From the 47 lists, we first identified verbs which appeared in >50% (24+) of the lists. From that list we then identified verbs for which 50% of their appearances were in one specific tier. The results are shown in Table 1.

Verbs to Avoid
Fourteen sources recommended a list of verbs to avoid. Many of those verbs were also recommended for use in Learning Outcomes by different lists. To identify a "master list" of verbs to avoid, we calculated the net difference between recommendations to avoid and recommendations to use. Verbs were included in the "master avoid" list when the net difference was >7. We also added in those five verbs which appeared across all six tiers of the taxonomy in our analysis and that of Stanny (2016). The results are also shown in Table 1. Compiled from 47 lists, verbs which appeared in more than half the lists, and were in the same tier from more than half of the lists in which they were included, are shown for that tier. Fourteen lists recommended verbs to avoid. Many of those verbs were also recommended for use in Learning Outcomes by different lists. Verbs were included in the "avoid" list here where the net difference between recommendations to avoid and recommendations to use was >7. Also included in the list of verbs to avoid are those which appeared across all 6 tiers of the taxonomy (on different lists).

Sources
Eight lists cited both the original and the revised taxonomy. Ten lists cited only the original, four lists cited only the revised, 25 lists did not offer a citation. None of the sources gave any other citations that explained where the verb lists came from although other works were often cited to support the linking of verb lists and learning outcomes to assessments (e.g., Biggs, 1996;Moon, 2004;Biggs and Tang, 2011).

DISCUSSION
The writing of Learning Outcomes based on Bloom's taxonomy is a common approach to organizing teaching and assessment. Verb lists based on the taxonomy are found on the websites of many UK Universities and in the textbooks recommended to academic staff as part of teacher training programmes in UK HE. Our findings demonstrate that there is not any one format representing "Bloom's Taxonomy, " echoing findings from other settings (Almerico and Baker, 2004;Stanny, 2016). Thus, the action verbs used by one university to plan learning at one tier of the hierarchy may be used to represent different tiers at other universities. For example, four verbs (select, explain, relate, arrange) appeared across all six tiers of the taxonomies. Two of these (select and relate) were the most commonly cited verbs, appearing 80 and 87 times, respectively. They also appeared across all six tiers in the analysis of US sites undertaken by Stanny, along with the verb choose (Stanny, 2016). As another example of basic problems with the current status of the taxonomy, two verbs (understand and know) commonly cited as problematic for writing specific learning outcomes even in the original taxonomy (Bloom et al., 1956), were actually recommended for use by five of the lists. At least one university with a proposed list of verbs to avoid, then recommended some of those same verbs for use in the taxonomy.
Does it matter that there is a lack of consistency between UK HE providers with regard to the verbs they use to map learning to the different levels? Part of the answer to this depends on whether the lists are actually used. Future work to answer this could include an analysis of whether the verb list proposed for use at a particular university actually maps on to the learning outcomes used at that University. The current analysis could also be developed through further discussion with subject experts to expand the master taxonomy devised here, and identify subjectspecific verbs and assessments.
It could be argued that having diversity within the sector is a good thing. The taxonomy was revised in 2001, demonstrating it has evolved over time. The existence of two versions of the taxonomy might also be thought to explain some of the heterogeneity in the verb lists. However, these arguments are undermined by the lack of any supporting evidence provided, by the University webpages, for the verb lists proposed. Twenty five lists did not offer a citation of Bloom's taxonomy and no other obvious citations were given, by any source, to support the hierarchical nature of the taxonomy or the verb lists contained within. Many universities directed their staff to external sources for the verb list; blogs and other informal sites, often with multiple colorful representations of the taxonomy, some including apps that map to the taxonomy and so clearly post-date either of the published versions of the taxonomy. This seems problematic given that a fundamental basis of UK Higher Education, in fact a requirement of university accreditation, is having learning outcomes mapped to levels of learning; this is one of the ways in which consistency can be achieved across the sector (QAA, 2014). Most of the sources used here were offering up Blooms Taxonomy in support of these levels and the writing of learning outcomes mapped to them, but did not cite either version of the published versions of Bloom's taxonomy and represented the taxonomy in very different ways, including many which merged the two versions together. This aforementioned diversity then is not evidence-based.
There is a broader question of whether the taxonomy accurately represents how we learn. A misalignment of learning science and the taxonomy was identified even when the taxonomy was first published in 1956. Bloom and co-wrote that, basically, there was no satisfactory, unifying, theory for how people learn, and that their taxonomy would make it easier for such a theory to be developed, even going so far as to state that "our method of ordering educational outcomes will make it possible to define the range of phenomena for which such a theory must account." (Bloom et al., 1956) In essence, they are saying "this is what learning looks like, now you have to explain how it happens." This is not the case now. There is an abundance of evidence from psychology, sociology, and neuroscience to explain how, and why, people learn, and what that looks like at the behavioral level (Bjork and Bjork, 2011;Dunlosky et al., 2013;Cowan, 2014;Freeman et al., 2014;Deslauriers et al., 2019) There is clearly a great deal that we do not know, but we propose that any attempt to classify learning outcomes should now be based on the science of learning, rather than the other way round. This could eventually lead to a revised, third version of the taxonomy that is grounded in an evidence-based understanding of how we learn.
There are many approaches used in education which are not supported by rigorous evidence. The use of some, such as the matching of teaching to so-called "Learning Styles, " have been directly contradicted by research evidence many years ago (Coffield et al., 2004;Pashler et al., 2008) and yet are still very popular in HE (Newton, 2015;Newton and Miah, 2017). We should therefore expect that Bloom's taxonomy will remain part of an approach to organizing learning in UK HE for the foreseeable future, despite the lack of evidence used to support the formats in which it is currently presented. Rather than simply complain about this, we offer up Table 1 as an approach to some sort of consensus regarding verbs to use, and of verbs to avoid. The broad consensus method used to generate the table allows for the fact that many of the verbs appeared in multiple tiers, a principle that is consistent with the principles of the revised taxonomy which proposes that the tiers, in particular the upper tiers, are not a fixed rigid hierarchy (Krathwohl, 2002). This principle is lost in the presentation of the taxonomy on the website of the UK universities analyzed here, where the verbs are simply presented in fixed lists and without reference to the supporting literature.
From a pragmatic perspective, for those wishing to (or required to) use Bloom's or any other taxonomy, we would advise careful inspection of the verbs in context before adopting any correspondence between verbs and any learning of a certain complexity. Given that verbs themselves can be used in different tiers we would further advise against establishing automatic correspondence between the isolated verb and HE level when designing or evaluating modules or programmes. We would echo the advice given by others that the best way to give meaning to a learning outcome is to identify the assessment type(s) that might map to that outcome (Ewell and Schneider, 2013) and that this is, itself, a test of whether one has written a useful Learning Outcome (Adelman, 2015). If an educator wishes to use any sort of hierarchical taxonomy to classify and map their outcomes, then we propose going further still and asking educators to identify assessments that would not be suitable for an outcome mapped to a specific level of the hierarchy.

DATA AVAILABILITY STATEMENT
All datasets presented in this study are included in the article/Supplementary Material.

AUTHOR CONTRIBUTIONS
PN designed the study, collected the data, checked the transcribed data, analyzed the data, and drafted the manuscript. AD and LP recollected the data for verification, checked the transcribed data, and provided critical revisions of the draft manuscript. All authors contributed to the article and approved the submitted version.