The ongoing rapid urbanization of city regions urges governments to face and come up with innovative ideas to overcome urban challenges and support and improve the health of citizens and healthcare for patients, whereby digitalization is considered to be a useful method to reinforce efficiency and reduce costs (1). For example, the World Health Organization (WHO) Healthy Cities Network and associated national networks have established hundreds of member cities globally that aim to benefit from Internet of Things (IoT) to improve the health and well-being of citizens (2, 3). Although during the last few years smart cities have been established across Europe and other continents, limited research has been conducted to evaluate smart city interventions and their impact and other outcomes of embedded smart technologies for cities and citizens and their health and well-being (4). A smart city is a place where digital and telecommunication technologies (e.g., IoT services and applications for healthcare, active and healthy aging technologies, algorithms, and artificial intelligence methods for gather relevant knowledge from IoT devices) are used to benefit inhabitants and businesses (2). In a recent systematic review performed by (5), they showed that the most relevant applications supported by a smart city infrastructure with a significant impact on healthcare provision were applications for population surveillance, active aging, healthy lifestyles, disabled people, response to emergencies, better service organization, and socialization.

In this context, we present a case study that focuses on better service organization to reduce pollution to decrease the adverse effects that mobility behavior has on citizens health, which describes the potential benefits of using the MAFEIP (Monitoring and Assessment Framework for the European Innovation Partnership) tool to evaluate smart city projects and interventions by using a cost-effectiveness analysis that has been used extensively in the health economy sector. The MAFEIP tool was originally developed to assess the impact and cost-effectiveness of digital health interventions and is highly promising to be used for different digital interventions as well, such as smart city projects.

The purpose of the MAFEIP tool is to estimate the outcomes of a large variety of social and technological innovations, by providing an early assessment of the likelihood that interventions will achieve the anticipated impact. In addition, the MAFEIP tool also helps to identify what drives interventions' effectiveness or efficiency in order to guide further design, development, or evaluation. MAFEIP therefore represents a clear support to the decision-making process, also for smart city projects and interventions.

The MAFEIP tool rests on the Markov model principle, which is an analytical decision-making model developed by and for health economics (6–9). The main objective of the MAFEIP tool is to provide support in the decision-making process, including an ex ante analysis before a concrete intervention is implemented. The Markov model is able to tackle uncertainty in the real effects and costs, and its flexibility allows for the analysis of a large and heterogeneous range of interventions. The model uses the best evidence available from multiple sources, such as administration records, official databases, ad hoc information collected for projects' evaluation, or results from evaluations in similar interventions.

Furthermore, the MAFEIP tool provides a useful methodology for the early assessment of the likelihood that interventions will achieve the anticipated impact, while at the same time helping to identify what drives interventions' effectiveness or efficiency in order to guide further design, development, or evaluation. Thus, it can be helpful for a wide variety of stakeholders involved in the IoT-based health service applications in cities. First, health or social care providers can use the MAFEIP tool for codesigning technology-based solutions, while also using the evidence from real life pilots to assess the effectiveness, impact, and utility, to make a more informed decision to invest or to buy a specific technology-based solution. Second, the MAFEIP tool has been shown to be a valuable instrument in Health Technology and Intervention Assessment to inform policy decision making in relation to citizens' health and well-being (10, 11). The tool is able to analyze with more precision the value of the innovation for citizens and other stakeholders and support the systematic evaluation of properties, effects, and/or impacts of health technologies and interventions in different population target groups. Third, big companies, small and midsize enterprises, and startups can take advantage of MAFEIP's utility in assessing the potential impact of new business propositions for healthcare interventions and thus guiding the decision-making process for further technology developments. Last, the MAFEIP tool can be relevant to researchers, as it can be used to improve the quality and relevance of future research and to better serve the information needs of citizens, payers, and other decision makers by helping to identify gaps in evidence.

The MAFEIP monitoring framework is an online and free-to-use tool that rests on the principles of decision analytic modeling. The MAFEIP tool is based on a traditional Markov model, which is an approach that is common in health economic evaluations and mostly used to assess the impact of healthcare innovations in terms of health outcomes and resource use. Based on the data introduced in the tool, which may be (preliminary) data from clinical studies, expert opinions and user's own views, or data based on a randomized controlled trial, this model performs an incremental analysis of the impact of innovations. As a result, users need data on both the current care situation for the target population and the situation in which the intervention is used (12).

In order to run the model, users need to introduce to MAFEIP different parameters divided in four sections, namely, (1) model analysis, (2) costs associated with health states and intervention costs, (3) transition probabilities for moving between states with and without the intervention, and (4) utilities (also called quality-of-life weights) that are associated with each state. A value has to be selected for each input parameter in order to run the model. Subsequently, the model runs the analyses by itself and provides graphical and numerical outcomes of the analyses.

The results of the current case study showed that the intervention is more expensive than the non-intervention situation because of the initial investment. Nevertheless, if we look at the upcoming years, the difference between the non-intervention would be reduced because periodic costs would be smaller. At the same time, if the intervention succeeded in moving to states with less pollution, costs would also decrease, as states with less pollution have lower associated costs, especially if governments decide to tax greenhouse gas emissions. Figure 4 presents the cost evolution in both situations (intervention and non-intervention) for a 5-year period, where the intervention is more expensive than the non-intervention, but the difference between them is reduced. This can be seen on the incremental cost line. These results would change if intervention costs were lower. Figure 5 shows what would happen if initial investment costs were 50 MU instead of 500 MU. In that case, the intervention would initially be more expensive, but it would bring monetary savings afterward.

The other key results of the model are the effects derived from intervention, in this case the “usefulness” linked to the pollution level. Figure 6 shows how accumulative intervention effects increase faster than non-intervention effects (because there are more neighborhoods transiting to lower pollution levels), and consequently, the incremental effect is positive, and it increases.

The final result of the evaluation is given by the ICE, which informs the decision making. Whether the program is implemented or not will depend on where the point is located. Figure 7 gives an example of a scenario where the necessary initial intervention investment is 500 MU. The horizontal axis represents the incremental effect, and the vertical axis represents the incremental cost. The ICE is located on the top right quadrant. Therefore, the WTP for reduction of the level of pollution will be the key to decide if the intervention will be implemented or not. If the WTP is 700 MU per usefulness unit (gray line), the intervention would be acceptable because the ICE would be below the threshold. However, if the WTP is 400 MU (blue line), the intervention should not be implemented. Therefore, in this scenario, the decision would depend on the WTP of the administration in charge of developing the project, which may depend on the available budget, the priority attributed to the impacts pursued by the intervention, etc. WTP also depends on the citizens or private organizations who will pay for the intervention in a direct or indirect way (through taxes). It should be noted that this WTP is not static, and the participation of the main stakeholders may be needed to define the limits in which an intervention would be considered acceptable.

The simulation for Scenario 1 has been conducted under concrete parameters. However, these are just estimations, and they may deviate from reality. That is why the model can be run multiple times, demonstrating how results vary when certain parameters are updated. As seen previously, one change in the investment costs may modify the result of the accumulative costs. For example, for Scenario 2, the ICE would be in the bottom right quadrant, meaning the intervention is dominant. That means that it would be cheaper and more efficient than the current option, and therefore, it would be positive to implement it, regardless of WTP (Figure 8). On the other hand, Figure 9 brings a more pessimistic scenario in which the intervention would not be more effective than the current situation, and therefore, the final decision should be to not implement it. The different results would form the set of possible scenarios that could be reached by implementing a certain intervention, which would provide key information to the authorities in charge of making decisions.

Because of the improvement of healthcare delivery that makes healthy aging more possible than ever before, in addition to the growing world's population that has been moving toward urban areas over the past years, it has created the necessity to develop smart cities that are aware of the special needs of all their citizens (14, 15). Digital and smart technologies need to deliver affordable and easy-to-use solutions to a growing and aging population, thereby improving health and well-being of citizens (16). These developments, which are expected to keep happening within the next 40 years, bring great challenges and disadvantages associated with urban agglomerations. These challenges include several problems to be tackled from a policy point of view, such as resource scarcity, air pollution, decrease of citizen health, and traffic problems, among others (17). Thus, urban areas have increased their sizes, which has been made possible by a simultaneous upward shift in the urban technological frontier in order to allow cities to accommodate more inhabitants (18). The WHO Healthy Cities Network and associated national networks already have established hundreds of member cities globally that aim to benefit from IoT to improve the health and well-being of citizens, but assessing their full potential and impact is still scarce.

History shows that problems associated with urban agglomerations have been solved by different means related to creativity, human capital, and cooperation among stakeholders, which according to Kylili (18) can be called “smart” solutions. In this context, the European Union, as well as other international organizations, has put effort on devising a strategy to achieve urban growth in a smart way for its metropolitan areas, fostering development through information and communications technology–driven interventions, and improving citizens' health and well-being (14, 18). As a consequence, there has been an exponential growth of projects and other initiatives to develop and establish smart cities, creating a clear need for a reliable and valid methodology to evaluate these initiatives. The current innovative scenario demands a holistic methodological approach for assessing smart city projects that aim to improve health and well-being of citizens, which incorporates a combination of technologies and solutions that take into account progressive economic feasibility considerations (19). It is of utmost relevance for policy makers to adopt these methodologies, as they ensure the effectiveness of the interventions aiming at transitioning from the European cities as we know them into smart ones.

However, searches carried out have not identified any evaluation framework that is currently addressing this need. As we have shown in the present report, the MAFEIP tool is highly promising to support evidence-based decision making in the development and uptake of smart city interventions and technologies. MAFEIP goes beyond simple measurement with indicators such as those that we can find on many smart cities studies, to inform on the effects produced directly by an intervention and the expected impacts (ex ante evaluation). On their work, Caird (20) suggests that the selection of measurement indicators for evaluation purposes at project, program, and city levels and different geographical scales should be appropriate to smart city developments, with potential correspondences mapped between each level. The MAFEIP tool compares the costs and the efforts invested with the results to examine the profitability and the feasibility, giving smart cities interventions and technologies a chance to improve their efficiency and effectivity, scale up, and be able to improve decision making in this field.

Evaluation plays a fundamental role in the development of smart city projects, as it is what allows decision-makers to identify programs that obtain the expected results. Additionally, the ex ante evaluation analyzes the expected results of one or more interventions before their application, in order to decide whether their implementation is recommended. For each intervention, MAFEIP analyzes how the costs are related to its effects, taking into account the WTP. The results can be positive or negative, of greater or lesser degree, and with a possible differential effect depending on the actors. MAFEIP allows the choice between alternatives that are normally exclusive due to budgetary restrictions and cannot be applied simultaneously. Thus, it is essential to make use of the evaluation before, during, and at the end of each public program or intervention with a public impact, like all those that fall within the framework of smart cities.

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

NF, FF, and FL-V have conceptualized the study design, designed the methodology, conducted the analyses, and wrote the manuscript. All authors have approved the final version of the manuscript and have approved the content.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.