How is multi-criteria decision analysis supporting existing software for nature-based solutions decisions?

Introduction: Nature-based solutions (NbS) involve working with and enhancing nature to address social challenges, aiming to protect, manage and restore natural ecosystems. While recognition of their benefits and cost-effectiveness is improving, there remain challenges to their adoption. Since they are often multi-stakeholder problems and aim to achieve multiple objectives, Multiple Criteria Decision Analysis (MCDA) is a good methodology to assist with their planning and implementation. In addition, software tools that support the MCDA process while evaluating NbS help manage varying stakeholder preferences and facilitate collaboration between them. They also make complex information about NbS problems easier to visualise, express and interpret. Current analyses of NbS either do not focus on MCDA or when they do, they do not examine how software can support the implementation of MCDA in evaluating NbS.

Methods: This study thus conducted a systematic literature review via Web of Science and Google Scholar to analyse a sample of software that assess NbS using methods that support any part of the MCDA process. We analysed the technical aspects of the software, their scope of analysis and their capability to support complex decision-making for NbS selection, implementation and management.

Results and discussion: We found that the software assess a wide variety of scopes and often include a spatial GIS component too. They cater to different users, among which are public officials, private-sector businesses and consultants. They mainly assist with the planning and design of NbS, rather than their management. The software assess many different co-benefits of NbS—the most common ones being hydrology, carbon uptake, air quality and socio-economic factors. By suggesting NbS alternatives and describing their performance, they also support decisions by making NbS benefits more understandable and streamlined to decision-makers. Most of the software only allow the assessment of one alternative at a time, rather than supporting the comparison of multiple NbS. Software developers should justify and align their weighting and aggregation methods in documentation and should broaden the scope of their software's analysis in the future to include biodiversity, non-urban NbS and enhancement of stakeholder engagement and cooperation during the whole MCDA process, from problem formulation to provision of a decision recommendation.

1 Introduction

The International Union for Conservation of Nature (IUCN) defines nature-based solutions (NbS) as “Actions to protect, sustainably manage and restore natural or modified ecosystems that address societal challenges effectively and adaptively, simultaneously providing human wellbeing and biodiversity benefits” (Cohen-Shacham et al., 2016). NbS involve working with and enhancing nature to address social challenges, seeing as human wellbeing depends on the various services that healthy ecosystems produce (Seddon et al., 2020). Even if an NbS targets a specific societal challenge, it will often collectively address more than one challenge and thus have far-reaching co-benefits (Raymond et al., 2017; Penning et al., 2023). Climate resilience, improved biodiversity, flood control and soil health are key co-benefits of NbS (Osaka et al., 2021). NbS improve air and water quality and provide socio-economic benefits like tourism development, increasing job opportunities and lowering energy expenses through green infrastructure (Ommer et al., 2022).

The interest in NbS is increasing alongside recognition of their benefits and costeffectiveness. However, challenges to their adoption remain, such as limited knowledge and data about their (collective) benefits and uncertainty regarding how to plan, design, implement and maintain them (Voskamp et al., 2021). The multifunctional nature of NbS necessitates careful spatial planning which considers the three pillars of sustainable development: environmental, social and economic sustainability (Bousquet et al., 2023). Additionally, evaluating how NbS address multiple challenges simultaneously, like flooding, drought and biodiversity loss, is crucial to their planning, design and implementation (Penning et al., 2023). Furthermore, implementing NbS often involves multiple stakeholders whose differing preferences and interests need to be considered and reconciled (Saarikoski et al., 2016). Implementing NbS also requires assessing their performance and suitability within different environments (Sarabi et al., 2022). Multiple Criteria Decision Analysis (MCDA) is a suitable methodology to tackle such challenges, as it was specifically designed to evaluate single and multiple alternatives according to relevant criteria, while also involving stakeholders' preferences and technical information (Bousquet et al., 2023).

The MCDA methodology includes various tools and methods that help structure and formalise decision-making processes in a transparent and consistent manner. It allows for comparing different options or solutions (i.e., alternatives) based on certain criteria that can be weighted according to how a decision-maker perceives their importance (Langemeyer et al., 2016). MCDA features six stages: (i) problem definition; (ii) stakeholder analysis and engagement; (iii) definition of alternatives (where alternatives are various possible solutions to the problem); (iv) definition of criteria (where criteria are measures to evaluate the effectiveness of different alternatives); (v) weighting of criteria (when relevant); and (vi) aggregation to provide decision recommendations (Langemeyer et al., 2016; Cinelli et al., 2022b). The decision recommendations can be of four types: (1) ranking, or ordering alternatives from most to least preferred; (2) sorting, or assigning alternatives to preference-ordered classes; (3) clustering, or dividing alternatives into groups based on similarities or another measure; and (4) choice, selecting a preferred subset of alternatives (Cinelli et al., 2022b). MCDA is thus advantageous for assessing NbS problems because it involves multiple stakeholders and their differing preferences in a flexible and transparent manner (Sarabi et al., 2022). Furthermore, it can assess the varied ecological and socioeconomic co-benefits of NbS simultaneously within a single framework (Saarikoski et al., 2016; Penning et al., 2023).

Various studies evaluate NbS, by valuing ecosystem services and NbS in general. (Viti et al. 2022), for example, examine stated preference methods for assessing the non-tangible benefits of NbS, like ecosystem services, human wellbeing, recreation and economic benefits. (Wild et al. 2024) and (van Oijstaeijen et al. 2020) discuss tools that provide monetary valuations of NbS cobenefits. (Bockarjova et al. 2020) assess the willingness to pay for the ecosystem services provided by NbS. (Geukes et al. 2024) assess the values and indicators for assessing coastal NbS that decision-makers ideated during the decision-making process. (Voskamp et al. 2021) conduct a broader review of various tools for evaluating NbS, going beyond solely monetary valuation to any aid with designing, planning and implementing NbS. Such studies are very helpful to decisionmakers looking into the benefits of NbS, allowing them to assess a potential NbS project. However, these studies do not describe how such tools and valuation methods can support an MCDA-based approach to NbS.

There are very few studies which explicitly incorporate MCDA into the evaluation of NbS. (Blattert et al. 2017) assess the various criteria that exist for use in forest planning, providing ways to evaluate NbS alternatives used to address this challenge. Similarly, many MCDA-aligned frameworks have been proposed for evaluating NbS benefits. One such framework assesses NbS benefits by selecting indicators corresponding to the benefits different NbS provide, calculating the performance of each indicator and then aggregating these into a score of the NbS' performance (Watkin et al., 2019). (Calliari et al. 2019) construct a similar framework for implementing NbS, involving stakeholders in developing and evaluating the climate-resilience of NbS alternatives. There are also more spatially specific frameworks for NbS, which assess the suitability of specific areas for different NbS. These aid with planning and implementing NbS (Sarabi et al., 2022; Mubeen et al., 2021). (Bousquet et al. 2023) reviewed various NbS implementation case studies to assess how MCDA can assist with implementing NbS. (Ruangpan et al. 2021) also bring MCDA and NbS together to incorporate stakeholder preferences in implementing two NbS case studies. They outline a detailed set of criteria in a framework for assessing NbS benefits.

While these studies are highly relevant to bringing NbS and MCDA together, they do not assess how the tools support decision-making processes for NbS. Since NbS adoption is still lacking, a key step to improving their uptake is holistically valuing their advantages and disadvantages, including non-market and non-tangible benefits (Viti et al., 2022). NbS tools, such as software and databases, make an important contribution to evaluating them, as they can aid planning processes by selecting and evaluating NbS, simulating their implementation, calculating their costs and benefits, supporting stakeholder involvement and facilitating collaborative processes (Voskamp et al., 2021). However, current analyses of tools evaluating NbS broadly include databases, catalogues and repositories, without a specific focus on software (Voskamp et al., 2021). Software are very helpful for visualising the various benefits NbS provide, and can thus provide more information about them and encourage their uptake (Moreno-Calderón et al., 2020). Furthermore, NbS problems involve multiple stakeholders. Well-planned software are key for managing their varying preferences and including them in the decision-making process (Mustajoki and Marttunen, 2017). Lastly, software that have a user interface include a broad audience—they make complex information about NbS easier to visualise and interpret, and allow people who may not have programming expertise to assess NbS too. Current analyses of NbS software also do not examine their specific features, like the target users, weighting and aggregation methods, and which life-cycle phase of NbS they would best support (exploration, implementation, management). They thus do not study in detail how the software support the planning and implementation of NbS. Thus, assessing how software support an MCDA approach to evaluating the NbS benefits is, to the best of the authors' knowledge, a novel prospect that will be tackled in this study. The findings of this study aim to influence sustainability policy and practise by assessing how these software assist with NbS evaluation and decision-making. This will allow for real-time monitoring of their benefits, promotes adaptive management, and can also enhance inclusive and accessible decision-making with holistic stakeholder engagement, thus ensuring that equitable and tailored NbS are implemented.

Thus, this study aims to:

• Analyse a sample of software developed for NbS planning, assessment and management, to determine how they support decision-making according to the stages of the MCDA process;

• Assess the features of the software to determine how they support NbS-related decision-making;

• Introduce a structured way for planning and evaluating NbS which involves stakeholders and traceably assesses their benefits, aiming to contribute to NbS uptake and adaptive management based on life-cycle NbS benefits monitoring;

• Recommend future directions for developing software which more accurately assess NbS benefits and comprehensively support inclusive NbS decision-making.

This paper is structured as follows. The next section outlines the methods for conducting the literature review, including how articles were selected and screened, and the features of the included software that were analysed. Section 3 then examines the results of the analysis, assessing technical information about the software, its scope of analysis and how they correlate to the various stages of the MCDA process, and provides a brief discussion of the findings. Section 4 evaluates the software on their methodological quality, user-friendliness and contributions to software sustainability, before providing recommendations for software developers. Section 5 concludes the paper.

2 Methods

A systematic literature review (SLR) and a Google Scholar search were conducted to identify software that operationalise the MCDA process to plan and evaluate NbS, following the guidelines by (Siddaway et al. 2019). This method was chosen for its replicability, transparency and ability to provide an overview of the state of available software (Siddaway et al., 2019). This aligns with the study's objective to comprehensively identify, evaluate, and synthesise high-quality evidence on the NbS evaluation software.

An SLR was chosen over other reviews like scoping reviews and integrative reviews, for various reasons. Scoping reviews can map broad research areas and identify gaps, while the present study aims to answer more narrowly defined questions, such as assessing the methodological features of the studies, better addressed by an SLR. Similarly, although integrative reviews include diverse methodologies, the present study focuses on synthesising empirical findings within clearly defined inclusion/exclusion criteria, which is more consistent with the SLR methodology. The identification, screening and eligibility of studies were conducted following the Preferred Reporting Items of Systematic Reviews and Meta-Analyses (PRISMA) method (Moher et al., 2009).

2.1 Identification of search terms

Search terms for the literature review were divided into three categories: (1) those related to nature-based solutions; (2) those related to MCDA and decision support; and (3) those related to software and tools. They are summarised in Table 1. The search was conducted on the Web of Science database in March 2024 (Web Of Science. Clarivate 2024).

Table 1

Table 1. Search terms for systematic literature review.

No cutoff date was considered for the search due to the novelty of the field. The Google Scholar search was conducted in February 2024, using the search term “NbS software decision support.” The first ten pages were then screened to select potential candidate software.

2.2 Screening and eligibility

Papers were first screened by their titles and abstracts, and when deemed suitable for inclusion, the full text of the papers were screened. Papers meeting any of the six exclusion criteria described in Table 2 were excluded. The software were included if they were able to support any part of the MCDA process. Review papers were included if they mentioned multiple different software.

Table 2

Table 2. Description of exclusion criteria for inclusion in the NbS software analysis.

2.3 Analysis of software

The selected software were analysed based on how they aid the evaluation of NbS using an MCDA approach. The assessment features were shaped by the authors or selected from available research that supports complex decision-making. This included papers that focused specifically on NbS (Voskamp et al., 2021; Bousquet et al., 2023), as well as those that conceptualised the use of MCDA for various application areas (Cinelli et al., 2022b; Tsoukiàs, 2007; Nardo et al., 2008). Figure 1 further outlines the features that were analysed for each software. For a detailed description of the analysed features, refer to Supplementary Table 6. To improve transparency and reduce bias, the software were analysed by two researchers, who screened and coded the studies individually as per the below features and then collaborated to discuss the results and resolve any discrepancies in the analysis conducted.

Figure 1

Figure 1. Description of the features of the software that were analysed.

3 Results and discussion

3.1 Literature selection

The Web of Science search revealed 229 scholarly articles, while the Google Scholar Search yielded 7 software. No records dating before 2009 emerged in the search, and included records did not date before 2018. The screening process is shown in Figure 2, conducted as per the PRISMA method (Moher et al., 2009). A total of 23 software were included and analysed in this review. The PRISMA selection process and the list of selected software that were analysed according to various features are available in Supplementary Tables 3–5, and 7, respectively.

Figure 2

Figure 2. PRISMA figure showcasing the results of the literature review search.

Figure 3 presents a high-level conceptual diagram of the software analysed in this study. The figure describes how the software assist with NbS decision-making, how they evaluate NbS, and the types of results they provide, alongside examples of software that were analysed in the current study. Table 3 provides a more detailed description of each software included in the review.

Figure 3

Figure 3. Conceptual diagram of the software assessed in the review.

Table 3

Table 3. Software analysed in the present study.

3.2 Technical information about the software

Figure 4 summarises the technical characteristics of the 23 analysed tools. Most were web-based (74%, 17), while the remainder were downloadable Excel calculators (26%, 6); none were desktop applications (Figure 4A). Web-based applications enable online data storage and provide graphical interfaces—such as sliding bars, icons, and maps—that help stakeholders better communicate their preferences (Weistroffer and Li, 2016; Mustajoki and Marttunen, 2017). Downloadable Excel-based calculators, commonly used in the present study for monetary evaluations of NbS, allow offline storage of sensitive data and support statistical and aggregation functions, improving transparency and visualisation of MCDA processes (Weistroffer and Li, 2016). Excel also allows many options for presenting the results of a model, while allowing clear documentation of its structure, improving visualisation of the MCDA process and transparency around model-building (Hollman et al., 2017).

Figure 4

Figure 4. Technical information about NbS software: (A) NbS software platforms; (B) Data sources of NbS software; (C) Documentation available on NbS software; (D) Use of GIS in the NbS software (n = 23).

Regarding data sources, 52% of the tools integrated NbS evaluation data, 17% (4) combined some user-inputted data with datasets integrated in the software, and 31% (7) relied entirely on user-provided data (Figure 4B). Users commonly needed to input the type of hazard they wanted their NbS to address, for example, flooding or urban heat. Land cover type was also an important factor for many software, requiring users to indicate if the solution would be implemented on agricultural land, wetlands, forests, etc. Software also distinguished between NbS implemented in urban and rural areas. Often, more technical NbS software required the user to provide most of the input data. One example is Urban GreenUP, which requires detailed qualitative information on the social and political conditions of the user's organisation that would support or hinder NbS implementation (Croeser et al., 2021; Urban GreenUP, 2024). While such specific input data may be difficult to acquire, the software output is more tailored and insightful as the specificity of the data increases.

On the other hand, software like RECONECT or the NBS Benefits Explorer only request users to input the type of challenge they wish to address, the land-use type and whether the measure is urban or non-urban (RECONECT Measures Selector, 2024; CEO Water Mandate, 2024). These then indicate possible NbS alternatives without any suitability score or indication of the exact performance of the alternatives. Even though such software required limited input data and are less specific to the user's needs, they are easier to use and can provide a helpful preliminary assessment of NbS. Importantly, input data, as discussed here, do not indicate anything about NbS benefits or performance—they cater to user preferences and specifications, without evaluating NbS. Section 3.4.2 explores categories of analysis, which pertain to NbS benefits and performance.

Software documentation was generally accessible and current (87%) (Figure 4C). One tool (4%) had no documentation, while two (9%) had outdated documentation. Software documentation was not consistent—it did not commonly describe the methods used to develop software, often limited to a user manual and/or the input data sources for the software. Documentation should go beyond disclosing data sources and user instructions to also disclose its methods of development. For a few software, the opposite was true—the only documentation available was a journal article featuring more technical information on the software. Such documentation is not specific to different users, like developers interested in detailed methods as opposed to public officers using the software for a project. Software documentation helps users evaluate and compare different software and can also aid developers with identifying methods of development and thus should consider what information users are seeking about the software (Forward and Lethbridge, 2002). Documentation should be usable and readable (Aghajani, 2020). Usability means clear information on locating the documentation, locating different versions of the software, easy navigation of the documentation. Readability means not assuming prior knowledge of the users and ensuring documentation is concise and comprehensible (Aghajani, 2020). Thus, distinguishing between a user manual and the methods of development (but including both) increases transparency and user understanding of the software. The NBS Simulation Visualisation Tool analysed in the current study addresses this issue well (Urban Nature Labs, 2019). It differentiates between a technical user guide, a practitioner's user guide and a user manual. The technical guide describes data sources informing the models behind the tool. The practictioners' guide explains how to use the software for involving stakeholders in designing and implementing NbS. The final guide is a step-by-step user manual of the software. While the methods are not thoroughly detailed, the documentation is clear, comprehensive and relevant for different user groups (Urban Nature Labs, 2019).

Lastly, GIS integration into the software was also analysed (Figure 4D). A software had a GIS component if it provided results as a map. 22% of the software used GIS in this way, often indicating areas that were most suitable for implementing NbS or describing which areas benefit most from implementing NbS. UNOSAT's Decision Support System for Enhanced Disaster Risk Reduction is the only tool which focuses on mapping areas that are at risk from different climate hazards and indicates where NbS, in general, can be implemented, while the other tools focus on assessing the benefits a specific NbS alternative will provide at a particular place (UNOSAT, 2020). GIS is a highly useful tool for assessing NbS as it provides location-specific data on the benefits of NbS in specific areas. GIS is useful for assessing site suitability and addresses multiple criteria (through map layers), making it a vital tool for NbS software. Since NbS are often implemented in urbanised contexts where space is limited, GIS maps suitable areas and possible NbS benefits, helpful for decision-making and urban planning (Pacetti et al., 2022). GIS software store data in their map layers allowing users across time to access data specific to their project location. This greatly assists with the planning and design of the NbS (Baykal and Topal, 2022). Lastly, communicating spatial information about NbS benefits specific to stakeholders' local contexts helps them understand the importance of NbS in addressing climate change, improving their uptake (Veraart et al., 2024).

3.3 Scope of analysis of the software

The scope of analysis for the software is shown in Figure 5. Nearly all the software (95.6%, 22) were best aligned for public officials attempting to implement NbS in conservation or urban contexts (Figure 5A). A public official may be concerned with implementing urban NbS to create a climate-friendly living environment for their constituency. These software would help the officials to understand available NbS alternatives and methods of evaluating their impacts. Spatial tools could help them choose suitable areas to implement the NbS to maximise their impact. The abundance of tools providing a monetary valuation (43.4%, 10) of NbS benefits were helpful for businesses in the private sector or consultants, who may be concerned with the profits and costs associated with implementing NbS. For example, if a private firm is attempting to implement NbS (as part of sustainability commitments or corporate social responsibility efforts, perhaps), they may wish to examine whether NbS fit organisational budget constraints. Tools providing a monetary valuation of NbS benefits (and thus their impact) while assessing costs are thus helpful for businesses considering implementing NbS. Consultants would also benefit from such tools as they are generally hired to support businesses in determining viable, cost-effective solutions. Many software (56.5%, 13) were relevant for academics because they provide highly specific data on NbS benefits that are relevant for the literature and knowledge base on NbS. Academics conducting research on NbS benefits, for example, would require specific metrics to provide a clear analysis in their studies; thus, tools specifying exact amounts of carbon sequestration or pollutant removal caused by specific NbS are helpful researching the quantification of NbS benefits.

Figure 5

Figure 5. Scope of analysis for the analysed NbS software: (A) Target users for the software; (B) Application scales for the software; (C) Life-cycle phases where the software could be used (n = 23; software had multiple target users, application scales, and life-cycle phases where they could be used).

Most of the software were quite specific in their application scales: 39% (9) of them targeted a city-level radius, 22% of the tools focused on the street- or district-level, while 39% (9) applied to the project-level (Figure 5B). These last tools were either not location-specific or conducted monetary valuations. Only two tools assessed NbS at the country level, and six catered to the regional/provincial scales. Many software (47.8%, 11) address multiple application scales, allowing users to assess NbS at a city level or a street level within the same software. The urban bias amongst the software overlook rural contexts, which offer spacious natural environments suitable for NbS implementation, and provide benefits that also merit evaluation. Non-urban areas also feature a higher risk of not incorporating the preferences and local knowledge of rural stakeholders when designing and implementing NbS (Castelo et al., 2023). Rural socio-cultural interests and institutional contexts are particularly important to reconcile in co-creating NbS—however, opportunities for stakeholder collaboration remain limited, particularly with bringing together communities and local governments (Soini et al., 2023). Thus, focusing software on NbS co-creation and enhancing stakeholder cooperation in rural areas is a key gap to fill.

Figure 5C indicates that three tools addressed only the exploration stage, while 20 supported exploration alongside implementation and management. All the software thus addressed the exploration phase by suggesting NbS alternatives that would best address a particular climate hazard. GIS-equipped software tools identified optimal locations for NbS implementation. Such capabilities are very useful at the initial planning, design and creation phase. Software relevant only to the exploration stage suggested possible alternatives but lacked performance metrics or decision recommendations for NbS. The software relevant to the implementation and management stages covered both the exploration stage as well as the implementation and management stages. These software suggest certain NbS alternatives or where to implement them, relevant to exploration, but also allow life-cycle monitoring of NbS benefits by providing information about their performance based on measurable indicators and their suitability to the user's project specifications.

For example, the RECONECT Measures Selector focuses only on the exploration stage. It suggests a suite of relevant NbS measures based on input data like hazard type, land use, and urban/rural context, but does not evaluate performance or assist with selecting an appropriate measure. RECONECT helps users understand what NbS alternatives exist (exploration). However, it does not define criteria to measure how these NbS would perform over time or their long-term efficacy once implemented (RECONECT Measures Selector, 2024). It is thus only applicable to the exploration stage. In comparison, the UrbanPROOF Toolkit ranks alternatives using four criteria that users can weight as per their preferences. Thus, this software assists the user in choosing an NbS alternative by providing a score indicating its performance across measurable indicators or criteria—it suggests various NbS alternatives (exploration) and helps select a suitable solution based on a performance evaluation (implementation and management) (LIFE UrbanProof, 2018).

The emphasis on exploration reflects barriers to NbS uptake, including limited awareness, incentives, and institutional fragmentation (Voskamp et al., 2021; Pacetti et al., 2022). Quantifiable NbS benefit assessments remain scarce, hindering broader awareness of NbS benefits and thus adoption of these solutions (Viti et al., 2022). Consequently, developers prioritise tools that simplify design and implementation, namely through identifying available solutions and metrics for evaluating NbS impact. As NbS become more widespread, future software should evolve to support long-term performance monitoring, a critical step for adaptive management of NbS.

3.4 How the analysed software aid the MCDA process

3.4.1 MCDA stages that the software assist with

The six stages of the MCDA process were divided into two categories for analysis: problem formulation and decision aiding. The problem formulation stage includes the first four steps of the MCDA process (problem definition, stakeholder analysis and engagement, definition of alternatives and definition of criteria). Three software assisted with this stage (Figure 6). These suggested possible NbS alternatives without describing and/or aggregating the NbS' performance across different criteria.

Figure 6

Figure 6. MCDA stages that the NbS software assist with. n = 23.

The majority of the tools (20 out of 23) assisted with both the problem formulation and decision-aiding stages. The latter stage includes weighting, aggregation and the provision of decision recommendations. By offering different NbS alternatives and suggesting ways to evaluate their benefits, these software define alternatives and criteria for evaluating them and thus assist with problem formulation. These software then evaluate the benefits that NbS provide in some way. A smaller subset of those that evaluate NbS by comparing multiple alternatives (11 of the 20) then recommend which NbS alternative is most suitable based on its benefits or the user's project specifications, thus aiding decision-making.

The software mainly focus on the following MCDA stages: definition of alternatives, definition of criteria and development of decision recommendations. However, very few software focus on problem definition and stakeholder analysis and engagement. A crucial aspect of NbS problems is that they involve multiple stakeholders with differing preferences—although some of the analysed software can be used in group settings or for a live session, they do not actively include mechanisms to facilitate discussion and input from different stakeholders. A further exploration of how the tools aid stakeholder engagement is included in Section 4.2. More software should focus on the stakeholder analysis and engagement phase, assisting with addressing and reconciling the needs of different affected groups. It is still unclear how the software assessed in this study can involve many stakeholders simultaneously, allowing them to provide their inputs on alternatives, criteria and other aspects of the problem as is characteristic of the MCDA process.

The remaining MCDA weighting and aggregation stages are addressed in Section 3.4.3.

3.4.2 Key categories of benefits analysed by the software

The software assessed many different NbS co-benefits, differing greatly from each other. Figure 7 depicts the most common co-benefits (those that were assessed by at least five software). Notably, these categories are distinct from the input data desrcibed in Section 3.2—they do not describe user preferences, but rather the performance of an NbS alternative.

Figure 7

Figure 7. Benefits assessed by the NbS software: these are divided into three overarching domains, under which the specific categories are listed. Examples of metrics or indicators used to measure the categories of benefits are included too.

Software commonly assessed the disaster risk reduction benefits of the NbS, specifically with regard to hydrology and flood risk reduction (13 of the 23 software). Software also consistently assessed the environmental benefits of NbS. Eleven software assessed the NbS' potential for carbon uptake and sequestration. Air quality, assessed by 11 software, also arose frequently. Some software assessed the overall air quality benefits an NbS provides (e.g., Green Infrastructure Spatial Planning Model, Nature Smart Cities Business Model), while others looked into how NbS can reduce the spread of specific air pollutants like nitrogen oxides or ozone (e.g., i-Tree Canopy, NbS Simulation Visualisation Tool) (Meerow, 2019b; Nature Smart Cities, 2023; i-Tree Canopy, 2024; Urban Nature Labs, 2019). Lastly, 15 software assessed a wide variety of socioeconomic benefits.

For example, the i-Tree tools developed by the USDA Forest Service are a set of tools that assess the monetary benefits provided by trees in urban areas (MyTree, 2024; i-Tree Design, 2024; OurTrees, 2024; i-Tree Canopy, 2024; i-Tree Landscape, 2024). Five different tools assess the benefits at different geographic scales, from individual trees on a street to all trees in a particular district or region. These all contain more specific categories such as avoided runoff, amount of carbon sequestered/avoided, and amounts of specific air pollutants that the trees remove.

The focus on hydrological risk mitigation is highly relevant as NbS can reduce flood risk, surface run-off and erosion while also enhancing infiltration and water storage (Ruangpan et al., 2020). Some software distinguish between urban and rural areas, particularly relevant for the carbon and air quality data, which may differ based on vehicular and industrial activity. However, very few software assessed the biodiversity-related benefits of NbS. This is of concern considering that many other frameworks assessing NbS have focused on this category of benefits (Watkin et al., 2019; Ruangpan et al., 2021). Watkin et al.'s (2019) framework encompasses water, nature and people-related benefits of NbS. The nature-specific benefits include soil quality, air quality and biodiversity—the current software often overlook biodiversity. Focusing on improving biodiversity values through NbS can help to reduce disturbance to ecosystems and protect and enhance the condition of species (Ruangpan et al., 2021). It can also reduce land use change, a benefit that was assessed by many of the NbS software analysed in this study (Ruangpan et al., 2021). It can be challenging to incorporate biodiversity-related data into the software since it tends to be highly location-specific. However, as it is a key co-benefit and will further improve the effectiveness of NbS alternatives, software that have location-based data or include GIS should consider assessing NbS' biodiversity-related benefits in those areas to provide further nuance in evaluating them.

NbS often provide multiple co-benefits simultaneously. (Penning et al. 2023) outline that since ecosystems are vulnerable to hazards like flood and drought due to climate change, this has consequences for their biodiversity values. The combined effects of NbS on addressing these challenges is crucial for understanding the values they provide (Penning et al., 2023). (Ruangpan et al. 2021) also show that including only one type of co-benefit in a model could amplify the performance of grey infrastructure, or other solutions that may not align with stakeholders, compared to when multiple criteria are used for NbS evaluation.

3.4.3 MCDA weighting and aggregation methods

While 20 of the 23 software employ some form of aggregation, they do not specify the method and often only conduct the aggregation for a single alternative. Those that specify the type of MCDA weighting/aggregation methods often explicitly mention “multiple criteria decision analysis” in the software documentation.

Only five software used a weighting system. All of them employed the direct rating technique in various ways. Some allowed users to enter numbers for weights (e.g., percentages, numbers adding up to 10 or 100) while others featured a sliding bar to indicate the importance of different criteria. However, none of the developers justified the reason for choosing this system in their documentation. (Meerow 2019a) explained why they allowed users to set weights in the Green Infrastructure Spatial Planning Model (GISP) they developed. They outline two reasons—firstly, users may have different priorities compared to their original model and thus should be able to indicate their preferences through the weights. Secondly, as the original GISP weights were based on expert opinion, allowing users to indicate their own weights is inclusive of a wider user base (Meerow, 2019a,b). However, they do not explicitly mention direct rating or the reason that this method was most appropriate for the software.

The most common aggregation methods used by the software were the unweighted sum (14 software) or the additive weighted sum (four software). The former was largely employed by the software that conduct a monetary valuation of NbS. The latter featured in software that explicitly mentioned the MCDA process. Two software used different aggregation methods. One was HedgeDATE, which used the decision rules approach, basing decisions on simple “if…then” rules, and outlines preferences as examples of decisions (HedgeDATE, 2021; Greco et al., 2016). HedgeDATE suggests whether hedges and green walls should be implemented based on the conditions of a particular street, including the placement of buildings, the width of the street and the intended width of the hedge (Barwise et al., 2021; HedgeDATE, 2021). LaRiMiT was another software which employed a different aggregation method, the Analytic Hierarchy Process (AHP), to construct a ranking of NbS alternatives to mitigate landslide risk (LaRiMit, 2023; Capobianco et al., 2022). None of the software documentation provided a justification for their chosen aggregation methods.

Direct rating, commonly used in the analysed software for weighting, is user-friendly. It allows the software to address users' project specifications and can be implemented in various versatile ways. (Ruangpan et al. 2021) also recommend web-based tools and software for eliciting weights from non-expert stakeholders face-to-face, which is often challenging without such aids. Our findings align with Bousquet et al.'s (2023) review of NbS MCDA studies. They found that stakeholder involvement occurred mainly during the weighting phase, and that user interfaces (sliding bars, allowing users to enter numbers, etc.) helped decision-makers describe their preferences, but did not actually enhance cooperation between stakeholders with diverging preferences (Bousquet et al., 2023). Regarding aggregation, they similarly determined that studies either did not employ aggregation methods or did not mention which one was used (Bousquet et al., 2023). In the studies where aggregation methods were used, the most common one was a linear aggregation method like an additive weighted sum (Bousquet et al., 2023).

Weighting and aggregation are highly interdependent. If weights are used as measures of importance, the procedures for eliciting weights should correspond with an aggregation method from the outranking family. For example, the analytic hierarchy process (AHP) requires specific procedures for weight elicitation, which demand consideration at the weighting stage (Cinelli et al., 2022a). LaRiMiT, analysed in the present study, uses a method similar to direct rating to elicit user weights and yet aggregates using AHP, which is not an appropriate use of MCDA (LaRiMit, 2023; Capobianco et al., 2022). Various other software analysed conduct direct rating in combination with a weighted sum. This is also an incorrect use of MCDA, as weights for weighted sums should be elicited as compensation rates—this is not the case with direct rating (Cinelli et al., 2022a). Software developers who focus on MCDA approaches to valuing NbS should ensure that weighting methods are appropriately aligned with aggregation methods to ensure a smooth execution of the MCDA methodology.

3.4.4 Decision recommendations the software provide

The different types of decision recommendations provided by the software were also analysed (Figure 8). The authors define decision recommendations to be based on the aggregation of two or more criteria that, if relevant, were also weighted. We distinguish two types of decision recommendations—one based on a single alternative's performance, and the other based on comparing multiple alternatives. Under the second category, there are four further types of decision recommendations—ranking, sorting, clustering, and choice.

Figure 8

Figure 8. Decision recommendations provided by NbS software. n = 23.

Most of the software (56.5%), 13 did not allow the users to simultaneously compare different alternatives. These software provide an aggregated description of the performance of one NbS alternative—most commonly, they presented a sum of the monetarised benefits an NbS provides. Users could run the software multiple times to assess the performance of different alternatives, and then construct their own decision recommendation. These software can aid with hotspot analysis for NbS. By showcasing the performance of different alternatives on different criteria, they can provide information on which criteria frequently return high performance across all NbS alternatives, and vice versa.

Four software presented a ranking of the alternatives—usually a list of the alternatives and their ranking using a performance score or percentage. Often, the software simply presented a final ranking without showing the NbS' performance across the different criteria and indicators. The performance of each alternative on the different criteria should be indicated alongside the ranking to provide more detailed data and allow decision-makers to conduct further trade-offs and considerations.

Four software programs, three of which included a GIS component, provided a choice-based decision recommendation. The three GIS-based software indicated a subset of areas that would most benefit from NbS, and thus a set of alternatives that would be the ideal choice. The last one was HedgeDATE, which indicated whether or not to implement a particular alternative (HedgeDATE, 2021). Two software did not provide any decision recommendations—these focused on suggesting possible NbS alternatives based on the user's project specifications. They did not aggregate any criteria that evaluated NbS performance to recommend these alternatives and thus did not provide a decision recommendation (RECONECT Measures Selector, 2024; Climate App, 2024).

3.4.5 Uncertainty and sensitivity analysis

MCDA studies commonly conclude by conducting uncertainty and sensitivity analyses to test the accuracy and robustness of the model and see what influenced the decision outcome. However, uncertainty and sensitivity analyses were rarely described in the documentation of software analysed in the current study. Only three software mentioned that they conducted a sensitivity analysis, while none mentioned conducting an uncertainty analysis.

(Krieger and Grubert 2021), who developed a life-cycle costing tool for NbS conducted a sensitivity analysis (i-DST—Integrated Decision Support Tool, 2020). Their sensitivity analysis helps users understand the range of cost estimates for each criterion assessed, and the various drivers of uncertainty (Krieger and Grubert, 2021). They discovered that the level of maintenance required for an NbS and the discount rates have the greatest absolute impact on overall costs when they assessed how the various criteria interact to influence the output (Krieger and Grubert, 2021).

The developers of the Urban GreenUP tool, which assesses how organisational factors can influence the success of an NbS, also conducted a sensitivity analysis. Their model was largely stable against uncertainty, with limited changes in the rankings of NbS when they altered user priorities and input variables (Croeser et al., 2021). The rankings did see some change when the capabilities of the city for NbS implementation or the difficulty of implementing the NbS were altered (Croeser et al., 2021).

Lastly, WaterProof, which calculates the return on investment of an NbS implemented in a particular watershed, includes a sensitivity analysis in the tool (Rogéliz et al., 2022). The analysis was based on the results of the tool, namely the total discounted benefits and costs (Rogéliz et al., 2022). This is relevant as the tool is location-specific and requires highly specific quantitative input data from users and thus allows them to assess the robustness of the specific model developed through the tool.

Uncertainty and sensitivity analyses are crucial to complex, multi-criteria decision problems. Such problems incorporate two kinds of uncertainty—uncertainty about decision makers' preferences and knowledge, and uncertainties that arise from the MCDA model chosen (Mosadeghi et al., 2013). Uncertainty analysis focuses on addressing the former kind of uncertainty in inputs and how they impact the outputs of the model, while sensitivity analysis studies how specific sources of uncertainty contribute to the variability in the MCDA model outputs (Cinelli et al., 2021).

Since NbS problems require the input of the multiple stakeholders they may impact, they are prone to uncertain inputs. The analysed software use highly varied data sources and construct different models for evaluating NbS, thus facing uncertainty from different sources to different degrees. Conducting uncertainty and sensitivity analyses will contribute to understanding the influence of the assumptions made and the model inputs on the model results (Feizizadeh et al., 2014). It will reveal areas of uncertainty that future NbS evaluation software need to address, ensuring future software are more robust in their evaluations of NbS. Given that acceptance of NbS and their benefits is still slow, acknowledging and addressing gaps in their evaluation is crucial to ensuring a clearer understanding of their benefits (Viti et al., 2022). Software developers should consider including uncertainty and sensitivity analyses in their software and disclosing these procedures in documentation, such that their methods and further directions for software development are clear to users and other software developers.

4 Evaluation and recommendations

The current study provides a novel insight into how software can support decision-making processes involving NbS. It can assist decision-makers in better understanding NbS and describes tools that can help them inform their decisions during selection, development and monitoring of NbS. The current study assesses a variety of software that accomplish various functions from NbS siting to assessing their benefits in monetary, quantitative and qualitative terms. This section further evaluates the analysed software based on their methodological quality, user friendliness and contributions to software sustainability. It then provides some recommendations, synthesising the findings from Section 3 with the evaluations presented in this section.

4.1 Methodological quality of the software

The evaluation of the methodological quality of all the software considered the parameters outlined in Table 4. For the detailed analysis of all the included software, refer to Supplementary Table 8. At the time of conducting this evaluation, two of the software programs were no longer accessible through their links (the NbS Simulation Visualisation Tool [NbS SVT] and the Urban GreenUP tool). While documentation could be retrieved for Urban GreenUP, it was not possible to download or access documentation for the NbS SVT (Croeser et al., 2021; Urban GreenUP, 2024). Thus, for Urban GreenUP, the evaluation was conducted based solely on the documentation, while the evaluation could not be conducted for the NbS SVT. The methodological evaluation has been performed for 22 software programs. The authors have made all efforts to include updated links for software which released new versions after the data analysis for this study was completed (see Supplementary Table 7).

Table 4

Table 4. Parameters of methodological quality assessed for the software documentation.

The software programs are largely of good methodological quality, providing sufficient information about their development, data sources, usage, and sponsors. The programs largely disclose their data sources well, conduct fairly rigorous validation procedures and at least briefly describe the limitations of their methods of development. Limitations persist in disclosing ethical considerations and ensuring long-term software sustainability among the analysed software.

All the software disclosed their funding sources, creating transparency and clarity about their sponsors. Nearly all the software (19 of the 23) included a clear description of their user interface and features. These were frequently included in the user manual, along with step-by-step explanations of how to operate the software. Three software only partially described the features—both the Green Infrastructure Co-Benefits Valuation Tool and Urban GreenUP describe the methods of calculation and development without a clear explanation of how to use the features of the software (Green Infrastructure Co-Benefits Valuation Tool, 2019; Urban GreenUP, 2024). LaRiMiT's user manual has an incomplete description of the software, and updated documentation has not yet been uploaded (LaRiMit, 2023). However, the software has a simple, intuitive interface. Nearly all the software also disclosed the datasets and libraries they referenced. Many of the software, for example, the suite of i-Tree tools developed by the USDA Forest Service, GI-Val and the Green Infrastructure Co-Benefits Valuation tool, provide a detailed description of how these sources informed the methods used to calculate NbS benefits (i-Tree Tools, 2024; GI-Val, 2011; Green Infrastructure Co-Benefits Valuation Tool, 2019). This allows replication of the method for researchers or other stakeholders and also helps users understand more about the processes of calculating NbS benefits. The developer of the Green Infrastructure Spatial Planning Model (GISP) links the dataset they used to build the software, which is accessible and downloadable (Meerow, 2019b). Next, nearly all the software justify their purpose. Software that had a peer-reviewed study detailing their methods of development, such as WaterProof, GISP and HedgeDATE, usually provided a thorough description of the problem context in addition to describing the purpose of the software (Rogéliz et al., 2022; Meerow, 2019a; Barwise et al., 2021). Other software often miss this broader explanation of the problem context of NbS before describing the purpose of the software—including this would be important for users and practitioners to better understand how the software can assist with NbS selection, implementation and uptake, and what gaps they are filling.

Twelve of the software included at least a partial evaluation/validation. The most common method of validation was by using a case study. The analysed software applied their methods to a real-world case to test the reliability of the results, making for good-quality evaluations. WaterProof, which aims to calculate the ROI on NbS implemented in watersheds, conducted a particularly rigorous and well-planned validation process. Besides including a detailed case study to test the software, the developers also conducted user interviews when alpha and beta testing the software, alongside other systematic tests to ensure the software was functioning as intended (Rogéliz et al., 2022). User feedback was incorporated into the software. The HedgeDATE tool conducted a similar evaluation by seeking feedback from prospective users during a public workshop, aiming to incorporate this into the tool (Barwise et al., 2021).

Fifteen of the 23 software at least partially outlined the limitations of the software or the development method in detail. WaterProof (mentioned above) does not outline any limitations despite a detailed validation process and an extensive description of its methods of development (Rogéliz et al., 2022). The Klima Konform tool details the limitations of the method and the case study used to validate it, including incomplete data in the tree register used, differences in wind direction and the low likelihood of the selected heatwave event (Wollschläger et al., 2022; Urban Climate Simulations ENVI-met - Naumburg, n.d.). This allows developers and researchers to identify gaps in the methods and address these in future research, while users can be aware about limitations in the softwares' recommendations while making decisions regarding NbS.

None of the included software programs describe ethical considerations like data privacy, while a few describe long-term sustainability considerations. This could be because many tools do not request identifying input data from users. Those that include sensitive information, like monetary values, are Excel tools, allowing such data to be stored on users' desktops alone. There were limited ethical considerations when developing the models that inform the software, as these largely relied on secondary data and mathematical calculations. While the tools have limited ethical risks, disclosure about the storage of user data or similar should be included in the documentation or on the websites to improve transparency.

Most of the tools included limited discussion about their long-term sustainability. While some tools allowed for reporting technical issues or contacting the developers, there was largely no mention of planned improvements or maintenance of these tools. Only the i-Tree tools thoroughly incorporate technical sustainability considerations in the documentation, describing planned improvements to each tool, specific to the methods of calculation for each NbS benefit (i-Tree Tools, 2024). Community support methods are varied and useful, including a support page, a user forum and even virtual office hours with the developers. The importance of addressing software sustainability is further detailed in Section 4.3—however, it is crucial to describe planned improvements, provide forums for raising technical concerns and allow community support and contact with the developers to ensure stakeholder involvement in improving the tools and ensure user concerns are heard and addressed consistently.

4.2 User-friendliness of the software

Users have many considerations while choosing a particular software—thus, we assessed how user-friendly the present software are and how a potential user would consider selecting one for assessing NbS. For this, we assessed three parameters—the ease of navigating the software interface, the software's suitability for live focus groups for stakeholder engagement and whether they provide useful reports that capture the decision-making process. The complete analysis of user-friendliness can be found in Supplementary Table 7.

4.2.1 Ease of navigating the software interface

The software interfaces were largely easy to navigate. To assess the ease of navigating the user interface, the authors considered: (1) the ease of inputting data into the software; (2) the ease of finding information about the software; (3) the ease of interpreting the results provided by the software; and (4) how smoothly the software works (i.e., glitches, slow loading, etc.).

4.2.1.1 Ease of inputting data

The web-based software included intuitive visual elements like sliding bars or drop-down menus of options that ease inputting data into the software. For example, LaRiMiT's intuitive UI includes sliding bars for setting criteria weights (LaRiMit, 2023). Many of the software have integrated data sources to conduct their assessments of NbS and require limited, often qualitative input data from the user that is generally easy both to source and to input into the software. Data input was more difficult for two types of software. Firstly, various software, such as the tools within the i-Tree suite and the Nature Value Explorer, require users to either select locations or draw areas for NbS implementation on a map (i-Tree Tools, 2024; Nature Value Explorer, 2023). For users lacking technical expertise, it can be challenging to zoom, drag and control the map interface for location selection. With the i-Tree suite, alternatives such as inputting a specific address or uploading a shapefile can ease the process, but various tools still require the user to click on a specific area for siting the NbS (i-Tree Tools, 2024). Some of tools which required highly specific input data from the users pose challenges in terms of sourcing this data. Secondly, Excel-based tools are often comprised of multiple tabs requiring extensive user input—these can be difficult to go through and thoroughly understand. Such tools eased this process by highlighting cells needing user input and have an introductory sheet explaning how to navigate the different tabs within the tool. Lastly, WaterProof, a web-based software with highly specific input requirements from the user (such as costs of implementing different NbS), requires some degree of expertise to understand the tool and source the data. The meanings of various components of the software and how to use it are not entirely clear—one would need to thoroughly read the user manual, whereas other analysed web-based software were somewhat more intuitive and easier to understand simply by exploring the interface (WaterProof, 2022).

4.2.1.2 Ease of finding information about the software

Documentation was largely accessible and current, making it easy to find information about the softwares' development, data sources and user manuals. Many tools included both a shorter step-by-step guide on the webpage of the tool alongside a more detailed user manual. This helps users to quickly understand how to use the software, while also allowing them the option to seek more detailed information if required. Only the RECONECT and Urban GreenUP tools presented some challenges in finding additional information about the software. RECONECT is a project with a very wide scope—thus, little focus is placed on the measures selector tool assessed in this study compared to other aspects of the project, making it difficult to find information about it (RECONECT Measures Selector, 2024). Next, UrbanGreenUP has detailed information in an associated peer-reviewed paper and a short explanation of the Excel-based tool within the Excel sheet. However, the peer-reviewed paper is not linked in the sheet or on the website, which can make it difficult for users seeking this information (Croeser et al., 2021).

4.2.1.3 Ease of interpreting the results

The results provided by most of the software are quite easy to interpret. Many of the analysed software present monetised valuations of NbS benefits in a manner that is easy to read, interpret and understand. Given that a significant proportion of the software allow the assessment of only one NbS alternative at a time, this also makes it easier to closely examine and interpret the benefits provided by each alternative. Results indicating a ranking of alternatives with a score are similarly easy to interpret by helping users prioritise and identify suitable alternatives. Many web-based tools include aesthetically pleasing user interfaces, such as icons, tables or diagrams that assist with visualising the results too. Some software programs, however, outputted results that were more complicated to interpret. For example, the NbS Benefits Explorer outputs a detailed diagram of NbS alternatives, processes and benefits. It takes some time to understand the differences between the processes and the benefits, click through each process and link and understand the entire visualisation clearly (CEO Water Mandate, 2024). Next, the Green Life Cycle Costing tool is also slightly difficult to understand and visualise. The different Excel sheets within the tool are ordered such that cumulative results are followed by assumptions and other background information, only after which the results per alternative are presented. This can be confusing to navigate through (i-DST—Integrated Decision Support Tool, 2020).

4.2.1.4 Smoothness of functioning

The authors assessed, in this case, whether the tools functioned without glitches or errors. While some of the tools were quite slow to load due to processing high volumes of data, the tools all functioned smoothly and provided the results without errors once loaded. At the time of this analysis, the websites for two tools have been taken down—those of the NbS Simulation Visualisation Tool and the Green Infrastructure Co-Benefits Valuation tool.

The latter, an Excel-based tool that had been pre-downloaded, was included in the analysis. The former was a web-based tool which is now unavailable—it was analysed based on the prior information gathered for the initial analysis.

4.2.2 Contribution of the software to live group sessions

The authors also assessed whether the software programs were suitable for use in a live session, promoting stakeholder engagement. Many of the tools can be used in such settings, although they are not tailored to actively promote stakeholder engagement (as observed, the focus of the tools remains on exploring possible NbS alternatives and quantifying their benefits). The well-designed, intuitive interfaces of many of the tools can promote collective discussion around them during live sessions. Three tools, namely Urban GreenUP, the NbS Simulation Visualisation Tool (NbS SVT), and UrbanProof Toolkit seem particularly well-suited for NbS co-creation. Urban GreenUP's focus on organisational barriers to implementing NbS suggests it would be relevant for an organisation's planning team to collectively fill out after discussing their organisational culture and how it can help or hinder NbS implementation (Urban GreenUP, 2024). Second, the NbS SVT included a “practitioner's guide” specifically exploring using the tool for NbS co-creation. While this guide and the tool are no longer available, its relevance was noted during the prior analysis of the tool (Urban Nature Labs, 2019). The UrbanProof Toolkit is an MCDA tool, which can be used in a focus group to discuss weights, relevant alternatives, and the final ranking of alternatives (LIFE UrbanProof, 2018). However, many of the tools requiring highly specific inputs did not facilitate stakeholder engagement—for example, the i-Tree suite requires specific input data, such as locations of trees, their trunk widths and the condition of the tree. Such objective information does not leave much room for discussion and stakeholder engagement (i-Tree Tools, 2024). Similarly, tools which require a significant degree of expertise in understanding the required input data, such as CAVAT or WaterProof, do not account for differing knowledge levels of different stakeholder groups and are not very conducive to stakeholder engagement (CAVAT Calculator, 2024; WaterProof, 2022).

4.2.3 Contribution of the software to the decision-making process

The authors lastly assessed whether the assessed software provide useful reports that capture the decision-making process. This was linked to whether the tools assisted with only the exploration phase of NbS or also contributed to the implementation and management phases. Tools contrbibuting to the latter two phases generally produced very useful reports as they provided decision-making assistance, either by quantifying NbS benefits or by providing a relevant decision recommendation for selecting particular NbS alternatives. The three tools which contributed only to the exploration phase had less relevant reports, as they suggested a suite of NbS alternatives without any performance data or decision support. Of these, HedgeDATE is a rather basic tool which only suggests a yes/no result of whether to implement hedges. Its results are highly simple, lacking clear quantifications that could provide greater decision support (HedgeDATE, 2021).

The analysis has thus found that the software programs are largely very user-friendly—having up-to-date, accessible documentation, it is generally easy to understand how to navigate their interfaces, including inputting data, interpreting results and finding information about the tools' development. The tools do not feature any glitches or errors, and despite slow processing and loading times, provide relevant results. Though not specifically tailored for stakeholder engagement, the programs can be useful for co-creating NbS, aside from the few which require highly specific input data. Lastly, the tools also provide relevant and useful reports where they capture NbS benefits or provide tailored decision support.

4.3 Software sustainability

The software must be sustainable over the long-term. (Noman et al. 2024) create a software sustainability framework based on four pillars—environmental, social, technical and individual—which ensures software are practically applicable and relevant for use for generations to come. The analysed software are largely technically and individually sustainable. The software are technically sustainable because they are functional, efficient, and backed by relevant data sources that identify NbS benefits and clear methodologies for selecting NbS alternatives. The software are also individually sustainable, providing a good user experience. It was easy to understand how to use the software—even if there was no user manual, the web link or Excel sheet was generally straightforward to understand. The software also consider user privacy, evidenced in that monetary tools were usually downloadable Excel calculators. This ensured that no sensitive monetary information was stored on the Internet. Some of the software assessed in this review released updated versions whilst authoring this study, which shows that the developers are considering sustainability to ensure their software are useful and relevant for generations to come. As discussed regarding documentation, though, various software could ensure documentation is relevant to different users; thus, both users with limited expertise or those seeking more specific information on methods and data sources can have a more seamless and tailored experience.

However, the software can better address environmental and social sustainability. The software are socially sustainable in that they support decision-making processes for NbS and are generally transparent to users about their methods and data sources. However, as examined, software are not explicitly designed ensure dialogue and collaboration between stakeholders—a key feature of MCDA. An urban bias is noted amongst the software, with a limited focus on rural contexts. This is discussed further in the next section. Ensuring better stakeholder engagement and including different contexts would improve the social relevance of these software and ensure better social sustainability. The software contribute to environmental sustainability by helping to assess, evaluate and select appropriate NbS, improving the uptake of these solutions and thus contributing to climate resilience. However, there is little information on their resource use, energy consumption and emissions potential. Finding ways to reduce these or incorporate other environmental considerations into the software development is relevant for improving their environmental sustainability.

4.4 Recommendations for software developers

While the analysed software provide notable contributions for different users interested in assessing and selecting NbS, there is room for improvement for developers. This section proposes a qualitative analysis to outline recommendations for the developers, determining which of these are more cognitively demanding and technically complex. Recommendations are plotted in Figure 9 according to how frequently the weaknesses arose in this study on the y-axis and how much cognitive effort it would take for developers to address these issues on the x-axis.

Figure 9

Figure 9. Effort required to improve NbS software (x-axis) in relation to the frequency the weakness for each decision-making feature (y-axis).

4.4.1 Recommendation 1: alignment and justification of MCDA methods used

None of the developers have justified the MCDA weighting and/or aggregation methods used. Some software use misaligned weighting methods and MCDA methods, which calls into question the reliability of their results. Developers must thus ensure an appropriate use of MCDA methods and justify selecting them. Justifying the types of MCDA methods used will enhance understanding of what methods work best for analysing NbS. MCDA weighting and aggregation methods are heavily interlinked, and thus must be aligned when developing software to ensure reliable results. There are various tools that could aid developers with matching the type of decision-making problem with the MCDA weighting and aggregation methods. Some examples are the MCDA-Methods Selection Software (MSS) and the MCDA.it (Cinelli et al., 2022b; Watróbski et al., 2019). Both assist with selecting an appropriate MCDA method for the decision problem, selecting from a wide range of existing MCDA methods.

4.4.2 Recommendation 2: perform sensitivity analyses

Only four of the analysed software mentioned conducting a sensitivity analysis. Conducting a sensitivity analysis is crucial as it helps to identify the most influential sources of uncertainty and how these affect the model outputs and variability, providing directions for improvement. This process is not highly cognitively demanding—thus, developers should conduct sensitivity analyses on MCDA software and disclose the results of these analyses in the documentation. This ensures transparency and allows for improvement of the model by addressing influential sources of uncertainty.

4.4.3 Recommendation 3: perform uncertainty analyses

None of the analysed software mentioned conducting an uncertainty analysis. Uncertainty analysis requires including solutions that can model different forms of uncertainty simultaneously; for example, the effects on criteria performances and model structure (Cinelli et al., 2021). Specifically, multi-criteria models that assess various alternatives, as with the software analysed in the present study, contain various types of uncertainty. These are ambiguity (uncertainty in interpretations of the input data, such as decision-makers' preferences), stochasticity (uncertainty in the criteria weights, alternative evaluation or model parameters), and partial information (where decision-makers do not clearly express their preferences, or data is not complete or available; Pelissari et al., 2018). Such uncertainty can be addressed through scenarios, probability distributions or fuzzy sets. Scenario analysis involves developing different scenarios to test the model, including changing the weights and determining how the model responds under various situations of risk and uncertainty (Dias et al., 2012). Probability distributions can involve setting upper and lower bounds for probabilities for the decision-maker's preferences or for any event relevant to the decision problem (Dias et al., 2012). Fuzzy sets help assess the possibilities of certain events to deal with imprecisions in language (Dias et al., 2012). Developers should consider incorporating such methods into their software to better account for uncertainties in the decision-maker's preferences or other forms of uncertainty and disclose the results in their documentation.

4.4.4 Recommendation 4: develop user-specific documentation

The study identified gaps in the documentation of the software. Only one software provided comprehensive documentation tailored to different users. The others either had only a user manual (which made it difficult to find information about data sources or methods for developing the software), or only a highly technical peer-reviewed study explaining the methods and development of the software (which may not be accessible to users with limited expertise or those only wishing to understand how to operate the software). Documentation for the software must thus be tailored to different users, to improve the ease of use of the different software. This is crucial to ensuring that different users can find information about the software that they can understand as per their level of expertise. This will ensure the software are more user-friendly.

4.4.5 Recommendation 5: disclose performance data alongside decision recommendations

Software which provide a decision recommendation usually only disclose the ranking or the choice-based recommendation, without information on how individual alternatives were assessed and their performance against the various criteria. The software should disclose the performance data, showcasing how the NbS alternative performed on the different criteria assessed, as this helps users understand what informed the decision recommendation. It also provides users with additional data that can help them make further trade-offs and arrive at a different decision than the one recommended by the software. This may be demanding to integrate into the software, but if a decision recommendation is provided, such information is crucial to accompany it. An excellent example of this is the HELDA software. It allows users to define goals, criteria and alternatives and create a performance table outlining the performance of all the alternatives. Most importantly, HELDA creates varied visualisations of MCDA results. Graphs can be created to display performance data and a cobweb chart also allows users to assess this data visually. HELDA also draws up correlations of performance between different criteria [HELDA (Version 1.0), 2024; Mesa-Estrada, Personal communication, 8 April 2025]. This ensures that users have access to all data that informs their decisions and can visualise it in various ways should they wish to make further trade-offs, creating a more comprehensive and transparent decision-making process.

The highest-effort tasks outlined in Figure 9 require changes to the development of the software itself, which can be very demanding for developers.

4.4.6 Recommendations 6, 7 and 8: extend analyses of biodiversity, urban areas and cumulative benefits

The analysis found that the software feature an urban bias and provide a highly limited focus on evaluating NbS in rural areas. Additionally, biodiversity, despite being a key NbS co-benefit, is not featured frequently in the analysed software. Lastly, the software do not assess multiple co-benefits simultaneously, and thus do not consider the synergies between these co-benefits. Assessing biodiversity, non-urban areas and multiple co-benefits simultaneously can be highly challenging and requires tedious integration into the software.

However, including these data can greatly improve the recommendations the software provide and enhances the scope of the software analysis. As outlined, biodiversity is a key NbS co-benefit that demands evaluation and assessment. While its specificity to location makes it difficult to include an overarching criterion assessing biodiversity, spatially-specific software should consider including such a criterion in their analysis. A possible consideration would be sourcing data from the Integrated Biodiversity Assessment Tool (IBAT), which contains highly spatially-specific data on ecologically important species around the world. Similarly, the tool could be relevant when determining how to site NbS, as it could help practitioners determine if implementing NbS could assist with the conservation of key species in the area. However, its paywalled nature may make it inaccessible for developers creating open-access software, driving costs up and alienating users too.

Due to the lack of information around NbS benefits and implementation in rural areas and the difficulty in ensuring stakeholder collaboration, very few NbS are currently implemented in such areas. However, examining how NbS perform in rural areas and which NbS are most suitable there can ensure their implementation and co-benefits extend to non-urban communities too. Lastly, as co-benefits are interlinked and often interact to produce greater cumulative co-benefits, this should be quantified. Integrating this into the software could provide a clearer picture of the actual benefits provided by NbS.

4.4.7 Recommendation 9: enhancing stakholder engagement

The analysis revealed that the software largely assist with identifying criteria and alternatives, weighting, aggregation and providing decision recommendations. The software have limited focus on the initial stages of MCDA, which involves stakeholder engagement and collaboration. Software should address ways to involve multiple stakeholders and enhance engagement and cooperation between them throughout the MCDA process and not simply during the weighting phase. This is highly demanding as, to the best of the authors' knowledge, features that actively enhance stakeholders cooperation have not been included in any software so far. Thus, it is likely a currently unexplored area, which makes it difficult for developers to know how to integrate it into their software. A few possible ways stakeholders could be involved through the software: the results could be displayed per stakeholder group, as well as the final aggregated decision recommendation; stakeholders could be given the option to suggest criteria for evaluating the NbS or contribute their views on defining alternatives within the software. Incorporating such methods to more clearly visualise the preferences of each stakeholder group ensures the decision-making processes for NbS and similar environmental planning problems are truly inclusive.

5 Conclusions and limitations of the study

This study has assessed a sample of software that can aid MCDA-driven assessments of NbS. These software help with NbS planning, design, assessment and decision-making. They assessed a wide variety of NbS co-benefits, including hydrology, air quality, carbon and socio-economic factors. They catered to different user groups with a focus on public officials, private sector businesses and consultants and were able to assess various scopes, from city-level to street-level. Many software programs recognised NbS as spatial problems and used GIS to help with siting them appropriately. The software can assist MCDA in many ways, mainly by providing performance data and suggesting NbS alternatives, assisting with decision-making processes around NbS. The software were also user-friendly and of fairly good methodological quality; while incorporating some considerations of software sustainability. The present study is relevant for sustainability policy and practise. By assessing open-access software with accessible user platforms, the findings democratise the evaluation of NbS, contributing to participatory decision-making, inclusive of different users and contexts. The findings highlight software platforms that can assist with monitoring NbS and the benefits they provide once implemented. This can assist environmental managers with adaptive management of NbS and quick responses to ecological changes or threats within the implemented solutions. Thus, the findings are relevant for sustainability policy by ensuring proper monitoring and inclusive implementation of NbS.

However, the analysed software also present certain limitations. The software programs focus mainly on urban contexts, excluding rural areas. There is also limited focus on biodiversity-related NbS benefits and on the synergies created between multiple NbS co-benefits. We also identified that some of the software have misaligned weighting and MCDA methods, which affects the reliability of their results. Additionally, decision recommendations provided by the software are often not accompanied by performance data detailing how each software performed on the various criteria assessed. The software also largely did not include uncertainty and sensitivity analyses, which limits transparency about how uncertainty impacts the models informing the software. Lastly, the software do not enhance stakeholder engagement and cooperation, which is a key consideration for both NbS problems and MCDA. The recommendations provided intend for software developers to address these limitations and thus more holistically evaluate NbS and aid decision-making surrounding them.

The current study focuses on open-access software that did not require user expertise in programming. This ensures the software analysed are catered to users with limited financial resources and programming skills—and that analysis and understanding of NbS performance is more inclusive and freely accessible. Open-access software are accessible to both users and developers with freely accessible and modifiable code, which means that developers can also collaborate on the software to improve it based on user feedback (Heron et al., 2013). This builds a “development base” around the software and improves its long-term sustainability (Heron et al., 2013).

However, future studies should aim to include software behind a paywall too. Paywalled software are generally better regulated, which ensures that documentation is clearly explained and does not overestimate the knowledge of novice users, remains up-to-date with changes in the user experience and that there are official channels for the provision of technical support (Heron et al., 2013). The ever-changing nature of open-source software generally makes it difficult for these characteristics to be included, relying on community-based technical support and resulting in confusing or outdated software documentation. Paywalled software may address some of the limitations identified amongst open-access software, for example, including better documentation or even providing more specific data on NbS performance, assessing more of their benefits and incorporating stakeholder engagement.

Additionally, the present study is not registered in a protocol registry such as OSF (https://osf.io/), which reduces the transparency, accountability and proper recognition of the methodology as original. However, the authors took all measures amongst themselves to review and synthesise findings to resolve any discrepancies. Both authors screened and coded the studies individually as per the selected features and then discussed results collaboratively to resolve differences in the analyses conducted and synthesise findings. The present method for analysing the topic at hand was not identified in any protocol registries—thus, to the best of the authors' knowledge, the work has been prepared transparently and originally without replication of existing studies.

The present study focuses on the use of MCDA for addressing real-world NbS problems. MCDA is a valuable approach that can assist with the multi-stakeholder, multi-benefit nature of NbS planning problems through its focus on stakeholder engagement and assessing various solutions' performance on various criteria. The structured approach of MCDA is better visualised and operationalised through the assistance of software. However, there are also certain limitations of applying MCDA software to real-world NbS problems. Analysts may find it difficult to be aware about the ever-changing multitude of available MCDA methods, and then learn to appropriately integrate these into software. It is also challenging for them to identify relevant MCDA methods without burdening non-expert stakeholders who often provide key inputs into MCDA weighting and aggregation. Programme managers may find it challenging to maintain the functionality and connexion of the software for real-world use. Additionally, the first few stages of the MCDA process (such as stakeholder engagement and alternative definition) are rather unstructured, which can make it difficult for programme managers to maintain trust in the MCDA process at this stage compared to the more structured later stages where criteria and alternatives are defined and decision recommendations are given based on clearly-articulated preferences. Lastly, it will pose a challenge for programme managers to clarify with users and stakeholders that MCDA software aim to support and aid decision-making, and not to substitute decision-makers and stakeholders altogether. Stakeholders must lead the decision-making process and not be excluded during it—a particular challenge for programmers to incorporate into their software. Developers, programmers and analysts must be aware of these challenges when developing MCDA software, and when addressing the recommendations provided in Section 4. Software developers can further improve the analysis and uptake of NbS by widening the scope of the software's analysis, conducting uncertainty and sensitivity analyses and clarifying the MCDA methods they use to inform their software.

Author contributions

SM: Writing – review & editing, Formal analysis, Data curation, Visualization, Investigation, Writing – original draft. MC: Conceptualization, Validation, Supervision, Methodology, Writing – review & editing, Project administration.

Funding

The author(s) declared that financial support was not received for this work and/or its publication.

Conflict of interest

The author(s) declared that this work was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frsus.2025.1612404/full#supplementary-material

References

Aghajani, E. (2020). Software Documentation Automation and Challenges. Available online at: https://www.inf.usi.ch/lanza/Downloads/PhD/Agha2020c.pdf (Accessed September 15, 2024).

Google Scholar

Alves, A., Gersonius, B., Sanchez, A., Vojinovic, Z., and Kapelan, Z. (2018). Multi-criteria approach for selection of green and grey infrastructure to reduce flood risk and increase co-benefits. Water Resour. Manag. 32, 2505–2522. doi: 10.1007/s11269-018-1943-3

Crossref Full Text | Google Scholar

Arvanitou, E. M., Ampatzoglou, A., Chatzigeorgiou, A., Galster, M., and Avgeriou, P. (2017). A mapping study on design-time quality attributes and metrics. J. Syst. Softw. 127, 52–77. doi: 10.1016/j.jss.2017.01.026

Crossref Full Text | Google Scholar

Barwise, Y., Kumar, P., Tiwari, A., Rafi-Butt, F., McNabola, A., Cole, S., et al. (2021). The co-development of HedgeDATE, a public engagement and decision support tool for air pollution exposure mitigation by green infrastructure. Sustain. Cities Soc. 75:103299. doi: 10.1016/j.scs.2021.103299

Crossref Full Text | Google Scholar

Baykal, T. M., and Topal, T. Ü. (2022). The role of GIS-based thematic urban maps in determining the effectiveness of nature-based solutions. ACU Int. J. Soc. Sci. 8, 82–99. doi: 10.22466/acusbd.1196850

Crossref Full Text | Google Scholar

Blattert, C., Lemm, R., Thees, O., Lexer, M. J., and Hanewinkel, M. (2017). Management of ecosystem services in mountain forests: review of indicators and value functions for model based multi-criteria decision analysis. Ecol. Indic. 79, 391–409. doi: 10.1016/j.ecolind.2017.04.025

Crossref Full Text | Google Scholar

Bockarjova, M., Botzen, W. J. W., and Koetse, M. J. (2020). Economic valuation of green and blue nature in cities: a meta-analysis. Ecol. Econ. 169:106480. doi: 10.1016/j.ecolecon.2019.106480

Crossref Full Text | Google Scholar

Bousquet, M., Kuller, M., Lacroix, S., and Vanrolleghem, P. A. (2023). A critical review of multicriteria decision analysis practices in planning of urban green spaces and nature-based solutions. Blue-Green Syst. 5, 200–219. doi: 10.2166/bgs.2023.132

Crossref Full Text | Google Scholar

Calliari, E., Staccione, A., and Mysiak, J. (2019). An assessment framework for climate-proof nature-based solutions. Sci. Total Environ. 656, 691–700. doi: 10.1016/j.scitotenv.2018.11.341

PubMed Abstract | Crossref Full Text | Google Scholar

Capobianco, V., Uzielli, M., Kalsnes, B., Choi, J. C., Strout, J. M., von der Tann, L., et al. (2022). Recent innovations in the LaRiMiT risk mitigation tool: implementing a novel methodology for expert scoring and extending the database to include nature-based solutions. Landslides 19, 1563–1583. doi: 10.1007/s10346-022-01855-1

Crossref Full Text | Google Scholar

Castelo, S., Amado, M., and Ferreira, F. (2023). Challenges and opportunities in the use of nature-based solutions for urban adaptation. Sustainability 15:7243. doi: 10.3390/su15097243

Crossref Full Text | Google Scholar

Castro, C., and Rifai, H. (2021). Development and assessment of a web-based national spatial data infrastructure for nature-based solutions and their social, hydrological, ecological, and environmental co-benefits. Sustainability 13:11018. doi: 10.3390/su131911018

Crossref Full Text | Google Scholar

CAVAT Calculator (2024). London: London Tree Officers Association. Available online at: https://www.ltoa.org.uk/documents-1/capital-asset-value-for-amenity-trees-cavat/306-cavat-calculator (Accessed September 15, 2024).

Google Scholar

CEO Water Mandate (2024). NBS Benefits Explorer. Available online at: https://nbsbenefitsexplorer.net/tool (Accessed June 6, 2024).

Google Scholar

Cinelli, M., Burgherr, P., Kadziński, M., and Słowiński, R. (2022a). Proper and improper uses of MCDA methods in energy systems analysis. Decis. Support Syst. 163:113848. doi: 10.1016/j.dss.2022.113848

Crossref Full Text | Google Scholar

Cinelli, M., Kadziński, M., Miebs, G., Gonzalez, M., and Słowiński, R. (2022b). Recommending multiple criteria decision analysis methods with a new taxonomy-based decision support system. Eur. J. Oper. Res. 302, 633–651. doi: 10.1016/j.ejor.2022.01.011

Crossref Full Text | Google Scholar

Cinelli, M., Spada, M., Kim, W., Zhang, Y., and Burgherr, P. (2021). MCDA Index Tool: an interactive software to develop indices and rankings. Environ. Syst. Decis. 41, 82–109. doi: 10.1007/s10669-020-09784-x

PubMed Abstract | Crossref Full Text | Google Scholar

Climate App (2024). Adaptive Solutions. Available online at: https://www.climateapp.nl/ (Accessed September 15, 2024).

Google Scholar

Cohen-Shacham, E., Walters, G., Janzen, C., and Maginnis, S., (2016). Nature-Based Solutions to Address Global Societal Challenges. Gland: IUCN International Union for Conservation of Nature. doi: 10.2305/IUCN.CH.2016.13.en

Crossref Full Text | Google Scholar

Croeser, T., Garrard, G., Sharma, R., Ossola, A., and Bekessy, S. (2021). Choosing the right nature-based solutions to meet diverse urban challenges. Urban For. Urban Green. 65:127337. doi: 10.1016/j.ufug.2021.127337

Crossref Full Text | Google Scholar

Dias, L. C., Henggeler Antunes, C., and Rios Insua, D. (2012). Dealing with uncertainty in decision support systems: recent trends (2000–2011). Intell. Decis. Technol. 6, 245–264. doi: 10.3233/IDT-2012-0141

Crossref Full Text | Google Scholar

Feizizadeh, B., Jankowski, P., and Blaschke, T. (2014). A GIS based spatially-explicit sensitivity and uncertainty analysis approach for multi-criteria decision analysis. Comput. Geosci. 64, 81–95. doi: 10.1016/j.cageo.2013.11.009

PubMed Abstract | Crossref Full Text | Google Scholar

Forward, A., and Lethbridge, T. C. (2002). “The relevance of software documentation, tools and technologies: a survey,” in Proceedings of the 2002 ACM Symposium on Document Engineering (McLean, VA), 26–33. Available online at: https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=69017c8739ecb77bd80a3defe353b8af9ac1712c

Google Scholar

Geukes, H. H., van Bodegom, P. M., and van Oudenhoven, A. P. E. (2024). Setting the stage for decision-making on nature-based solutions for coastal climate adaptation. Ocean Coast. Manag. 247:106916. doi: 10.1016/j.ocecoaman.2023.106916

Crossref Full Text | Google Scholar

GI-Val (2011). The Mersey Forest. Available online at: https://merseyforest.org.uk/green-infrastructure-valuation-toolkit/ (Accessed June 6, 2024).

Google Scholar

Gogoll, J., Zuber, N., Kacianka, S., Greger, T., Pretschner, A., Nida-Rümelin, J., et al. (2021). Ethics in the software development process: from codes of conduct to ethical deliberation. Philos. Technol. 34, 1085–1108. doi: 10.1007/s13347-021-00451-w

Crossref Full Text | Google Scholar

Greco, S., Matarazzo, B., and Słowiński, R. (2016). “Decision rule approach,” in Multiple Criteria Decision Analysis: State of the Art Surveys, 2nd Edn, eds. S. Greco, M. Ehrgott, and J. R. Figueira (Berlin: Springer), 497–552. doi: 10.1007/978-1-4939-3094-4_13

Crossref Full Text | Google Scholar

Green Infrastructure Co-Benefits Valuation Tool (2019). Green Infrastructure Leadership Exchange. Available online at: https://giexchange.org/green-infrastructure-co-benefits-valuation-tool/ (Accessed September 15, 2024).

Google Scholar

Grosman-Rimon, L., and Wegier, P. (2024). With advancement in health technology comes great responsibility—ethical and safety considerations for using digital health technology: a narrative review. Medicine 103:e39136. doi: 10.1097/MD.0000000000039136

Crossref Full Text | Google Scholar

HedgeDATE (2021). Available online at: https://hedgedate.eps.surrey.ac.uk/ (Accessed June 6, 2024).

Google Scholar

HELDA (Version 1.0) (2024). Available online at: https://www.mcda-helmholtz.de/ (Accessed April 10, 2025).

Google Scholar

Heron, M. J., Hanson, V. L., and Ricketts, I. (2013). Open source and accessibility: advantages and limitations. J. Interact. Sci. 1, 1–10. doi: 10.1186/2194-0827-1-2

Crossref Full Text | Google Scholar

Hollman, C., Paulden, M., Pechlivanogloui, P., and McCabe, C. (2017). A comparison of four software programs for implementing decision analytic cost-effectiveness models. Pharmacoeconomics 35, 817–830. doi: 10.1007/s40273-017-0510-8

PubMed Abstract | Crossref Full Text | Google Scholar

i-DST—Integrated Decision Support Tool (2020). Golden, CO: Colorado School of Mines. Available online at: https://idst.mines.edu/download/ (Accessed September 15, 2024).

Google Scholar

i-Tree Canopy (2024). i-Tree Canopy—iTree Tools. Available online at: https://canopy.itreetools.org/ (Accessed September 15, 2024).

Google Scholar

i-Tree Design (2024). i-Tree Design—iTree Tools. Available online at: https://design.itreetools.org/ (Accessed September 15, 2024).

Google Scholar

i-Tree Landscape (2024). i-Tree Landscape—iTree Tools. Available online at: https://landscape.itreetools.org/ (Accessed September 15, 2024).

Google Scholar

i-Tree Tools (2024). Available online at: https://www.itreetools.org/ (Accessed September 15, 2024).

Google Scholar

Krieger, J., and Grubert, E. (2021). Life-cycle costing for distributed stormwater control measures on the gray-green continuum: a planning-level tool. J. Sustain. Water Built Environ. 7:4020019. doi: 10.1061/JSWBAY.0000933

Crossref Full Text | Google Scholar

Kuller, M., Farrelly, M., Marthanty, D. R., Deletic, A., and Bach, P. M. (2022). Planning support systems for strategic implementation of nature-based solutions in the global south: current role and future potential in Indonesia. Cities 126:103693. doi: 10.1016/j.cities.2022.103693

Crossref Full Text | Google Scholar

Kumar, S., Arzaghi, E., Baalisampang, T., Garaniya, V., and Abbassi, R. (2023). Insights into decision-making for offshore green hydrogen infrastructure developments. Process Saf. Environ. Prot. 174, 805–817. doi: 10.1016/j.psep.2023.04.042

Crossref Full Text | Google Scholar

Langemeyer, J., Gómez-Baggethun, E., Haase, D., Scheuer, S., and Elmqvist, T. (2016). Bridging the gap between ecosystem service assessments and land-use planning through multi-criteria decision analysis (MCDA). Environ. Sci. Policy 62, 45–56. doi: 10.1016/j.envsci.2016.02.013

Crossref Full Text | Google Scholar

LaRiMit (2023). Available online at: https://www.larimit.com/assessment/ (Accessed September 15, 2024).

Google Scholar

Lee, D., and Oh, K. (2019). The green infrastructure assessment system (GIAS) and its applications for urban development and management. Sustainability 11:3798. doi: 10.3390/su11143798

Crossref Full Text | Google Scholar

LIFE UrbanProof (2018). The URBANPROOF Toolkit—A Decision Support System for Assisting Municipalities in Adaptation Planning. Available online at: https://tool.urbanproof.eu/ (Accessed September 15, 2024).

Google Scholar

Meerow, S. (2019a). A green infrastructure spatial planning model for evaluating ecosystem service tradeoffs and synergies across three coastal megacities. Environ. Res. Lett. 14:125011. doi: 10.1088/1748-9326/ab502c

Crossref Full Text | Google Scholar

Meerow, S. (2019b). GISP Model. Available online at: https://sites.google.com/view/gispmodel (Accessed September 15, 2024).

Google Scholar

Moher, D., Liberati, A., Tetzlaff, J., and Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 6:e1000097. doi: 10.1371/journal.pmed.1000097

Crossref Full Text | Google Scholar

Moreno-Calderón, A., Tong, T. S., and Thokala, P. (2020). Multi-criteria decision analysis software in healthcare priority setting: a systematic review. Pharmacoeconomics 38, 269–283. doi: 10.1007/s40273-019-00863-9

PubMed Abstract | Crossref Full Text | Google Scholar

Mosadeghi, R., Warnken, J., Tomlinson, R., and Mirfenderesk, H. (2013). Uncertainty analysis in the application of multi-criteria decision-making methods in Australian strategic environmental decisions. J. Environ. Plan. Manag. 56, 1097–1124. doi: 10.1080/09640568.2012.717886

Crossref Full Text | Google Scholar

Mubeen, A., Ruangpan, L., Vojinovic, Z., Sanchez Torrez, A., and Plavšić, J. (2021). Planning and suitability assessment of large-scale nature-based solutions for flood-risk reduction. Water Resour. Manag. 35, 3063–3081. doi: 10.1007/s11269-021-02848-w

Crossref Full Text | Google Scholar

Mustajoki, J., and Marttunen, M. (2017). Comparison of multi-criteria decision analytical software for supporting environmental planning processes. Environ. Model. Softw. 93, 78–91. doi: 10.1016/j.envsoft.2017.02.026

Crossref Full Text | Google Scholar

MyTree (2024). MyTree—iTree Tools. Available online at: https://mytree.itreetools.org/#/ (Accessed September 15, 2024).

Google Scholar

Nardo, M., Saisana, M., Saltelli, A., Tarantola, S., Hoffman, A., and Giovannini, E. (2008). Handbook on Constructing Composite Indicators: Methodology and User Guide. Paris: OECD Publishing.

Google Scholar

Nature Smart Cities (2023). Antwerp: University of Antwerp. Available online at: https://www.uantwerpen.be/en/centres/environment-sustainable-development/research/projects/nature-smart-cities/ (Accessed September 15, 2024).

Google Scholar

Nature Value Explorer (2023). Available online at: https://natuurwaardeverkenner.be/#start (Accessed September 15, 2024).

Google Scholar

Noman, H., Mahoto, N., Bhatti, S., Rajab, A., and Shaikh, A. (2024). Towards sustainable software systems: a software sustainability analysis framework. Inf. Softw. Technol. 169:107411. doi: 10.1016/j.infsof.2024.107411

Crossref Full Text | Google Scholar

Ommer, J., Bucchignani, E., Leo, L. S., Kalas, M., Vranić, S., Debele, S., et al. (2022). Quantifying co-benefits and disbenefits of nature-based solutions targeting disaster risk reduction. Int. J. Disaster Risk Reduct. 75, 2212–4209. doi: 10.1016/j.ijdrr.2022.102966

Crossref Full Text | Google Scholar

Osaka, S., Bellamy, R., and Castree, N. (2021). Framing “nature-based” solutions to climate change. Wiley Interdiscip. Rev. 12:e729. doi: 10.1002/wcc.729

Crossref Full Text | Google Scholar

OurTrees (2024). OurTrees—iTree Tools. Available online at: https://ourtrees.itreetools.org/#/ (Accessed September 15, 2024).

Google Scholar

Pacetti, T., Cioli, S., Castelli, G., Bresci, E., Pampaloni, M., Pileggi, T., et al. (2022). Planning nature based solutions against urban pluvial flooding in heritage cities: a spatial multi criteria approach for the city of Florence (Italy). J. Hydrol. Reg. Stud. 41:101081. doi: 10.1016/j.ejrh.2022.101081

Crossref Full Text | Google Scholar

Pelissari, R., Oliveira, M. C., Abackerli, A. J., Ben-Amor, S., and Assumpção, M. R. P. (2018). Techniques to model uncertain input data of multi-criteria decision-making problems: a l iterature review. Int. Trans. Oper. Res. 28, 523–559. doi: 10.1111/itor.12598

Crossref Full Text | Google Scholar

Penning, E., Burgos, R. P., Mens, M., Dahm, R., and de Bruijn, K. (2023). Nature-based solutions for floods AND droughts AND biodiversity: do we have sufficient proof of their functioning? Overview review. Cambridge Prisms Water, 1, 1–17. doi: 10.1017/wat.2023.12

Crossref Full Text | Google Scholar

Raymond, C. M., Frantzeskaki, N., Kabisch, N., Berry, P., Breil, M., Nita, M. R., et al. (2017). A framework for assessing and implementing the co-benefits of nature-based solutions in urban areas. Environ. Sci. Policy 77, 15–24. doi: 10.1016/j.envsci.2017.07.008

Crossref Full Text | Google Scholar

RECONECT Measures Selector (2024). RECONECT. Available online at: https://www.webscada.nl/reconect/measures-and-tools/#/measures (Accessed June 6, 2024).

Google Scholar

Rogéliz, C. A., Vigerstol, K., Galindo, P., Nogales, J., Raepple, J., Delgado, J., et al. (2022). WaterProof—a web-based system to provide rapid ROI calculation and early indication of a preferred portfolio of nature-based solutions in watersheds. Water 14:3447. doi: 10.3390/w14213447

Crossref Full Text | Google Scholar

Ruangpan, L., Vojinovic, Z., Plavšic, J., Doong, D.-J., Bahlmann, T., Alves, A., et al. (2021). Incorporating stakeholders' preferences into a multi-criteria framework for planning large-scale nature-based solutions. Ambio 50, 1514–1531. doi: 10.1007/s13280-020-01419-4

PubMed Abstract | Crossref Full Text | Google Scholar

Ruangpan, L., Vojinovic, Z., Sabatino, S. D., Leo, L. S., Capobianco, V., Oen, A. M. P., et al. (2020). Nature-based solutions for hydro-meteorological risk reduction: a state-of-the-art review of the research area. Nat. Hazards Earth Syst. Sci. 20, 243–270. doi: 10.5194/nhess-20-243-2020

Crossref Full Text | Google Scholar

Saarikoski, H., Barton, D. N., Mustajoki, J., Keune, H., Gomez-Baggethun, E., and Langemeyer, J. (2016). “Multi-criteria decision analysis (MCDA) in ecosystem service valuation”, in OpenNESS Ecosystem Services Reference Book, EC FP7 Grant Agreement No. 308428, eds. M. Potschin, and K. Jax (Wageningen: Wageningen University and Research). Available online at: www.openness-project.eu/library/referencebook

Google Scholar

Sarabi, S., Han, Q., de Vries, B., and Romme, A. G. L. (2022). The nature-based solutions planning support system: a playground for site and solution prioritization. Sustain. Cities Soc. 78:103608. doi: 10.1016/j.scs.2021.103608

Crossref Full Text | Google Scholar

Seddon, N., Chausson, A., Berry, P., Girardin, C. A. J., Smith, A., Turner, B., et al. (2020). Understanding the value and limits of nature-based solutions to climate change and other global challenges. Philos. Trans. R. Soc. Lond. B Biol. Sci. 375:20190120. doi: 10.1098/rstb.2019.0120

PubMed Abstract | Crossref Full Text | Google Scholar

Siddaway, A. P., Wood, A. M., and Hedges, L. V. (2019). How to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annu. Rev. Psychol. 70, 747–770. doi: 10.1146/annurev-psych-010418-102803

PubMed Abstract | Crossref Full Text | Google Scholar

Soini, K., Anderson, C. C., Polderman, A., Teresa, C., Sisay, D., Kumar, P., et al. (2023). Context matters: co-creating nature-based solutions in rural living labs. Land Use Policy 133:106839. doi: 10.1016/j.landusepol.2023.106839

Crossref Full Text | Google Scholar

Tsoukiàs, A. (2007). On the concept of decision aiding process: an operational perspective. Ann. Oper. Res. 154, 3–27. doi: 10.1007/s10479-007-0187-z

Crossref Full Text | Google Scholar

UNOSAT (2020). Decision Support System for Enhanced Disaster Risk Reduction. Geneva: UNOSAT. Available online at: https://unosat-geodrr.cern.ch/dss/FJI/MCDA/ (Accessed September 15, 2024).

Google Scholar

Urban Climate Simulations ENVI-met - Naumburg (n.d.). Available online at: https://web.app.ufz.de/KlimaKonform/urbansimulation/Naumburg/ (Accessed September 15, 2024).

Google Scholar

Urban GreenUP (2024). NBS Selection Tool. Available online at: https://www.urbangreenup.eu/resources/nbs-selection-tool/nbs-selection-tool.kl (Accessed September 15, 2024).

Google Scholar

Urban Nature Labs (2019). NBS Simulation Visualisation Tool. Brussels: Urban Nature Labs. Available online at: https://unalab.eng.it/nbssvt_v4/home.jsp?city=eindhoven&scale=local&year=2030&indicator=default&impact=default&scenario=00&prevision2=5&view=professional#! (Accessed September 15, 2024).

Google Scholar

van Oijstaeijen, W., Silva, M. F. E., Back, P., Collins, A., Verheyen, K., de Beelde, R., et al. (2023). The Nature Smart Cities Business Model: a rapid decision-support and scenario analysis tool to reveal the multi-benefits of green infrastructure investments. Urban For. Urban Green. 84:127923. doi: 10.1016/j.ufug.2023.127923

Crossref Full Text | Google Scholar

van Oijstaeijen, W., van Passel, S., and Cools, J. (2020). Urban green infrastructure: a review on valuation toolkits from an urban planning perspective. J. Environ. Manage. 267:110603. doi: 10.1016/j.jenvman.2020.110603

PubMed Abstract | Crossref Full Text | Google Scholar

Veraart, J. A., van Oosten, C., Linderhof, V., Duku, C., Appelman, W. A. J., Groot, A. M. E., et al. (2024). Use of spatial information to remove barriers and to foster enablers of uptake of nature based solutions for food production and water resource management in Ghana and the Netherlands. Int. J. Water Gov. 11. doi: 10.59490/ijwg.11.2024.6741

Crossref Full Text | Google Scholar

Viti, M., Löwe, R., Sørup, J. D., Rasmussen, M., Arnbjerg-Nielsen, K., Mcknight, U. S., et al. (2022). Knowledge gaps and future research needs for assessing the non-market benefits of nature-based solutions and nature-based solution-like strategies. Sci. Total Environ. 841, 1–11. doi: 10.1016/j.scitotenv.2022.156636

PubMed Abstract | Crossref Full Text | Google Scholar

Voskamp, I. M., de Luca, C., Polo-Ballinas, M. B., Hulsman, H., and Brolsma, R. (2021). Nature-based solutions tools for planning urban climate adaptation: state of the art. Sustainability 13:6381. doi: 10.3390/su13116381

Crossref Full Text | Google Scholar

WaterProof (2022). Available online at: https://water-proof.org/ (Accessed September 15, 2024).

Google Scholar

Watkin, L. J., Ruangpan, L., Vojinovic, Z., Weesakul, S., and Torres, A. S. (2019). A framework for assessing benefits of implemented nature-based solutions. Sustainability 11:6788. doi: 10.3390/su11236788

Crossref Full Text | Google Scholar

Watróbski, J., Jankowski, J., Ziemba, P., Karczmarczyk, A., and Zioło, M. (2019). Generalised framework for multi-criteria method selection. Omega 86, 107–124. doi: 10.1016/j.omega.2018.07.004

Crossref Full Text | Google Scholar

Weistroffer, H. R., and Li, Y. (2016). “Multiple criteria decision analysis software,” in Multiple Criteria Decision Analysis, 2nd Edn, eds. S. Greco, M. Ehrgott, and J. R. Figueira (Berlin: Springer), 1301–1341. doi: 10.1007/978-1-4939-3094-4_29

Crossref Full Text | Google Scholar

Wild, T., Baptista, M., Wilker, J., Kanai, J. M., Giusti, M., Henderson, H., et al. (2024). Valuation of urban nature-based solutions in Latin American and European cities. Urban For. Urban Green. 91:128162. doi: 10.1016/j.ufug.2023.128162

Crossref Full Text | Google Scholar

Wollschläger, N., Zinck, F., and Schlink, U. (2022). Sustainable urban development for heat adaptation of small and medium sized communities. Land 11:1385. doi: 10.3390/land11091385

Crossref Full Text | Google Scholar

Yang, C., Liang, P., and Avgeriou, P. (2016). A systematic mapping study on the combination of software architecture and agile development. J. Syst. Softw. 111, 157–184. doi: 10.1016/j.jss.2015.09.028

Crossref Full Text | Google Scholar