Front. Public HealthFrontiers in Public HealthFront. Public Health2296-2565Frontiers Media S.A.10.3389/fpubh.2022.892930Public HealthCase ReportCo-creation in a digital health living lab: A case studyFotisTheofanis1*KioskliKitty23SundaralingamAnand4FasihiAmer5MouratidisHaralambos21School of Sport and Health Sciences, University of Brighton, Brighton, United Kingdom2School of Computer Science and Electronic Engineering, Institute of Analytics and Data Science (IADS), University of Essex, Essex, United Kingdom3Trustilio B.V., Amsterdam, Netherlands4Oxford Respiratory Trials Unit, Nuffield Department of Medicine, University of Oxford, Oxford, United Kingdom5Kraydel Ltd., Belfast, United Kingdom
Edited by: Ann Borda, The University of Melbourne, Australia
Reviewed by: Timo Dietrich, Griffith University, Australia; Sonja Pedell, Swinburne University of Technology, Australia
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
Co-creation in healthcare, especially in developing digital health solutions, has been widely identified as a fundamental principle for person-centered technologies that could accelerate the adaptation of innovation. A Digital Health Living Lab based on community offers a sustainable and real-life environment to ideate, develop, and evaluate digital health solutions addressing the needs of multiple stakeholders. This article presents the experience of the School of Sport and Health Sciences at the University of Brighton in establishing a Digital Health Living Lab. In addition, we share a proposed step-by-step approach to establishing such a living lab in the community, supplemented by a case study of product development.
Innovation in digital services and products is mostly dependent on enhancing knowledge on a national and international scale, targeting to foster an ecosystem of complementary evidence (1). Therefore, it has become broadly accepted that the innovation process would be leveraged by including external stakeholders from the early stages to create a competitive advantage. Meanwhile, users are also encouraged to be involved. However, the feasibility of such involvement is debated in the literature (2). The living labs are user-centric innovation tools that have become very prominent in recent years (3) to fulfill this vision.
Over the last two decades, numerous initiatives, organizations, and institutes have sprung up worldwide as “living labs.” Meanwhile, regional and national governments, as well as international bodies (i.e., European Commission) have cautiously supported the concept of “living labs” and included it in their work programs (4–6). Overall, the phenomenon of a living lab mainly refers to and supports the involvement of multiple stakeholders for the (co-) creation, application, and evaluation of innovation services or products within a real-life setting (7, 8). Currently, there are numerous living labs worldwide, but a higher concentration is observed in Europe (9).
There is no standard definition of the concept of the living lab. However, according to the European Network of Living Labs (ENoLL) (10), these are defined as “user-centered, open innovation ecosystems based on systematic user co-creation approach, integrating research and innovation processes in real life communities and settings” (10). ENoLL's definition aligns with many definitions found in the literature. For example, the article by Leminen et al. (11) defines the living labs similarly as “physical regions or virtual realities in which stakeholders form public–private–people partnerships of firms, public agencies, universities, institutes, and users of products or services, all collaborating for creation, prototyping, validating, and testing of new technologies, services, products, and systems in real-life contexts.”
The living lab phenomenon embraces different contexts, for instance: the development of innovation activities driven by citizens aiming to improve everyday life, testing of technology-driven human-centric products from pharmaceutical companies, targeting to provide affordable and easy-to-use products to patients, or development of activities from NGOs, citizens, or other actors in developed societies (12). These differentiated actions may be initiated by various stakeholders (i.e., providers, users, and enablers), which impact the duration, focus, and outcomes of the innovation actions (11). Overall, the living labs offer a safe space for development, testing, and validation with co-creation in all stages, from conceptualization to commercialization (13). They are often characterized as testbeds for innovative solutions, systems, and products by providing a platform for collaboration (14, 15).
A living lab involves the following four main pillars:
Co-creation: Co-design by users and producers.
Exploration: Discovering emerging usages, behaviors, and market opportunities.
Experimentation: Implementing live scenarios within communities of users.
Evaluation: Assessment of concepts, products, and services according to socio-ergonomic, socio-cognitive, and socio-economic criteria.
It is worth noting that a core element of any living lab is sustainability, and there is a plethora of studies that have addressed this (16), but their perspectives differ. For example, some studies examine development and innovation activities that target to improve, in a sustainable way, the everyday life of citizens (12). At the same time, other studies explore transition labs that aim to accomplish change in sustainable development (17) or analyze the connection between living labs and sustainable innovation (13). Others investigate the role of processes, design, and practice in environmental transformation (18). Moreover, studies have also focused on sustainable development in smart city actions (19) and in entrepreneurship and urban development (20).
The existing literature gives a fruitful basis for understanding the potential and usefulness of living labs. This is due to their conceptualization and theorization (21), which further investigates the processes and methods followed (22), while also recording results from empirical studies (23–25). More specifically, Følstad (26) wrote the first review study, including 32 articles to establish theoretical foundations, methods, perspectives, and processes of a living lab. Later, Franz (27) developed an understanding of the phenomenon, which was socially centered. Schuurman et al. (5) reviewed 45 studies and concluded that practice and research in the living labs were still in the infant stages. Research in the existing literature from Leminen and Westerlund (28) established eight main research streams at the time. After reviewing 195 studies, Leminen et al. (6) aimed to understand the need around the emergence of the living labs movement. McLoughlin et al. (29) conducted a bibliometric analysis of 169 studies, while a more recent study by Westerlund et al. (30) performed topic modeling for 86 articles on the topic.
More recently, Hossain et al. (31) conducted a systematic review of 114 studies regarding living labs to gain an understanding of the main facets discussed in the developing literature. Notably, the study investigated the origin of a living lab and its key characteristics and paradigms, including contexts, stakeholder roles, main outcomes, challenges, and sustainability. It is notable that the literature in the living lab context has increased vastly since 2015, showing the urgency and advantages of the phenomenon. Scholarly studies discuss the living labs as infrastructures that could be utilized as novel tools for research opportunities to tackle needs and challenges in society (32). However, it is evident that the literature on living labs in the context of digital health and the linkage to innovation is still rather fragmented (33).
Traditional models of healthcare are experiencing significant pressure in the context of overwhelming strain on the existing systems due to the high demand for services from one side and limited funding from the other (34). As a result, an important opportunity for innovation in digital health has arisen. However, the market is currently leading in innovation in this space, but there is significant risk in bringing healthcare products and/or services that are not evidence-based to be consumed directly by the masses. “Disruption” is often proclaimed as the mark of any worthy innovation. However, this adopts a rather irresponsible view. In many cases, it is this irresponsible view that results in tensions between technologists and healthcare professionals (35).
The difficult issue of evidence-based digital health often rears its head, and it is a challenge not just for the technologists who are operating in a rather unfamiliar space but for the academics too. The bridge between academia and industry that aims to create links between the two sides is on the rise, but it is still not fully robust. Although the relationship seems simplistic, they function in parallel. Filling the gap through a fully developed collaboration in research between the two can boost the economy and growth by preparing a much-needed workforce with the industry's required skills and products that are developed and evaluated through evidence-based academic methodologies (36). Given that digital health interventions are often sitting at the intersection of biomedical, behavioral, and computer science, not to mention the design and user experience components, technologists feel that the classical evaluation models do not do them justice. Enhancement by collaboration from different disciplines is urgently needed (37).
The benefits to be gained from the participation of end-user groups, local health and government organizations, voluntary sector organizations, technologists, and Academic Health Science Networks (AHSNs) cannot be overstated. As outlined in Greenhalgh et al. systematic review of the challenges related to adapting new technologies, there are many obstacles to sustaining technological change, many of them resulting from the complex adaptive systems that provide healthcare (38). From this review, the Non-adoption, Abandonment, Scale-up, Spread, and Sustainability (NASSS) framework was derived as a tool to explore and identify sustainable adoption and applicability of technological innovations in healthcare and social care. The living lab model provides an opportunity to explore this complexity and shed light on the wider system which is targeting to embed the change with evidence-based models. Genuine stakeholder participation can be used as a tool for the ideation, development, and evaluation of digital health solutions toward optimizing the conditions within the system and refining the technology to match the requirements of the system. As a result, it will create a sustained and scalable technological change with rewards that are realized and provide benefits to all involved actors.
In this study, we present the setting up of a community-based Digital Health Living Lab (DHLL).
Living lab environment
The lab was the culmination of a national scheme, the Leading Places program, that had brought together the University of Brighton with Brighton and Hove City Council (39, 40). The Brighton project aimed to help develop strategies in self-managed care for older adults, for example, in the context of medication administration, self-monitoring and self-awareness, and self-management of the emotional impact of multiple comorbidities. Therefore, the team focused on a series of interventions for groups of people living in assisted sheltered accommodation to find ways to prevent or delay entry into more intensive and expensive care programs. The lab was developed as a response to address the difficult issue of evidence-based digital health supporting self-managed care and acting as a tool to be used by various stakeholders (41).
The site chosen to establish the DHLL was a retirement housing scheme recommended by the city council as it meets the requirements in terms of residents' demographics and caring needs. Retirement housing schemes are complexes where senior citizens live independently. They are specifically designed for those over 55, providing easy access, and being efficient and ergonomic. They are maintained by the local council, which provides on-site managers responsible for the safety and day-to-day running of the complex and catering to residents' needs. The site is comprised of 108 flats split into three building blocks, seven stories each, and is linked by a ground floor corridor and communal rooms (Figure 1). Residents were invited to attend a launching event where the DHLL team presented the concept and aims of this initiative and invited them to consider participating by registering their interest and sharing their contact details, consenting to the DHLL contacting them in future, along with basic demographic data including gender and age. Regarding the resident's profile, 81% are 65 years and older, with 58 and 42% female and male subjects, respectively, and 70% with recorded disabilities.
Digital health living lab.
Step-by-step approach
The literature describes different approaches and methodologies for living labs in other disciplines and sciences (i.e., environmental, green energy, and smart cities). Here, we present our experience and propose an approach directly related to setting up Digital Health Living Lab in the community (42–46).
One of the first decisions is the location that will “host” the living lab. According to our experience, this depends on the focus of the activities in relation to the main stakeholder. Considering that the citizen is always at the center of any activity, the site might be a building block: For example, the sheltered accommodation for our Empowercare project where the building block met the demographic requirements (main stakeholder citizen). If the main stakeholder is the local council, the DHLL can be a neighborhood where the diversity of civic life is more dependent on demographics (i.e., it may be related to environmental factors). It can be based on a community space (or even university dorms) where a group of citizens meet regularly. It can also be at a hospital ward when the main aim is disease orientated.
Once the location is determined, the next step is to create participants' profiles through interviews and understand stakeholder needs through co-production workshops. In Table 1, we present the different stakeholders and their needs from the DHLL, followed by a further Power/interest matrix of stakeholders (Figure 2). Our approach to systematically identify the relevant stakeholders was informed by the study of Manzini (47) and followed the steps for stakeholder selection (48) as described in the AgriLink Living Lab Toolbox (49). Applying these guidelines, we started by defining stakeholders, individuals, and organizations relevant to the living lab residents. The criterion of relevancy was based on the “position” of a citizen within the health and social care system in the UK and was defined as the type of relationship that affected directly or indirectly the residents' lives. As such, we developed a list of stakeholders (Table 1) where the relationship might be direct (i.e., carers, neighbors, and DHLL staff) or indirect (i.e., general health practitioners, local council, and commissioners). The list was also informed by the residents themselves through 1-2-1 interviews exploring who and which individuals and organizations they perceived as affecting their health and wellbeing.
Scoping DHLL stakeholders' needs.
Citizens of the DHLL • Wellbeing • To live as independently as possible with support as needed • Health and social care that is tailored to them • Choices and the ability to make decisions around health and wellbeing • Seen as an asset, not a burden (what can they do to help!)
Carers/next of kin • Their loved one is being cared for kindly and compassionately • Promote independence but ensure safety and reassurance • What can they do to help?
Staff of DHLL (i.e., site managers of the building) • Happy residents and happy staff • Eager to help but need education and training • Do not want additional responsibilities or workload
General health practitioners •Improved health outcomes • Reduced demand on their services • Do not want additional responsibilities or workload
Secondary care • Improved health outcomes • Reduced demand on their services • Don't want additional responsibilities or workload
Local government implementation plans •Mandate to work on prevention and self-management, better integration of health and social care, and incorporate aspects of local digital roadmap.
Local council/municipality •Improved health outcomes • Reduced demand on services • Needing to improve specific targets (falls/out-of-hours provision/ medication compliance) • Mandate to work on prevention and self-management, better integration of health and social care, and incorporate aspects of local digital roadmaps • Do not want additional responsibilities, workload, or financial commitments • Incorporate digital health as a cornerstone of initiative • Raise their profile through an effective and positive campaign
Commissioners • Improved health outcomes • Reduced demand on services • Needing to improve specific targets (falls/out-of-hours provision/medication compliance) • Mandate to work on prevention and self-management, better integration of health and social care, and incorporate aspects of local digital roadmaps • Incorporate digital health as a cornerstone of initiative • Do not want additional responsibilities, workload, or financial commitments
Academic institution •Mandate to work more collaboratively with civic authorities through the “Leading Places” project • Raise their profile through an effective and positive campaign • Incorporate digital health as a cornerstone of initiative • Research and publications • Ways to incorporate project in the curriculum of health science students
Academic networks •Mandate to work more collaboratively with NHS, academic institutions, and industry • Raise their profile through an effective and positive campaign
Digital innovators • Learning and training through working alongside service users • Innovation • Raise their profile through an effective and positive campaign • Commercialization
Voluntary sector •Keen to be involved and offer their perspective on health and social care • Raise their profile through an effective and positive campaign • May seek or offer funds depending on specific organization and nature of the relationship
Power/interest matrix of stakeholders.
Upon the completion of the stakeholder needs identification, we recommend prioritizing these needs reflecting the importance of the stakeholder in relation to the timeframe. In Table 2, we present an example of the prioritization of the needs of the DHLL stakeholders.
Prioritization of stakeholders' needs.
Timing (to address)
Now
Later
Activity
High priority
• To live as independently as possible with support as needed in a safe manner with reassurance • Choice and the power to make decisions around their own health and care needs with plans that are tailored to them • Seen as an asset, not a burden (what can they do to help!) • Their loved one is being cared for kindly and compassionately • Do not want additional responsibilities or workload, or financial commitments
• Improved health outcomes and wellbeing • Reduced demand on their services • Raise their profile through an effective and positive campaign (commonality between multiple stakeholders raises its significance) • Incorporate Digital Health as a cornerstone of initiative • Incorporate project in the curriculum of health science students
Low priority
• Needing to improve specific targets (falls/out-of-hours provision/ medication compliance) • Keen to be involved and offer their perspective on health and social care • May seek or offer funds depending on specific organization and nature of the relationship
• Mandate to work on prevention and self-management, better integration of healthcare and social care, and incorporate aspects of local digital roadmaps more collaboratively with civic authorities through the “Leading Places” project, academic institutes, and industry • Research and publications • Learning and training through working alongside service users • Innovation • Commercialization
The next steps include regular project management activities, such as time framing (i.e., GANTT chart), risk analysis, and register, followed by the actual implementation of the activities (i.e., testing, evaluation, etc.).
Projects
Since the establishment of the DHLL, we have utilized it in several cross-disciplinary research projects, with two more notable recent European funded (Table 3). Within these projects, we had the chance to test and evaluate different technologies, including wearables (Figure 3) and smart glasses (Figure 4).
DHLL projects.
Project
Funding body
Aim/content
EMPOWERCARE (EMPOWERing individuals & communities to manage their own CARE)
Interreg 2 seas
The project aims to create resilient communities and reduce individual frailty and loneliness by developing an approach using research-based solutions and technology to address gaps in the care of the target groups of those aged 65+ and those aged 50+ with at least one chronic condition. The DHLL is utilized as the tool providing a suitable environment to work along with citizens, carers, and healthcare professionals, co-creating, trialing and evaluating digital health solutions that would be able to be implemented across borders and healthcare systems.
AI4HealthSec (A Dynamic and Self-Organized Artificial Swarm Intelligence Solution for Security and Privacy Threats in Healthcare ICT Infrastructures
HORIZON2020
AI4HEALTHSEC proposes a state-of-the-art solution that improves the detection and analysis of cyber-attacks and threats on Healthcare ICT infrastructures and increases the knowledge on the current cyber security and privacy risks within the digital Healthcare ecosystem and among the involved Health operators. As such, it is the first time a DHLL is being utilized as a pilot testing site for the development security framework
Activity tracker.
Smart glasses.
Case study
Sharing our experience working in the DHLL with industrial partners, here we present a case demonstrating how it can contribute to the development of a digital product, in this case, the Kraydel Konnect, throughout different stages of its maturity.
Konnect is an established, easy-to-use home communication system. The system connects the user's TV with a communication hub using the HDMI port on the TV, and this hub connects to the internet via the User's home Wi-Fi network or mobile 4G signals. The hub enables video calls by connecting with standard video-calling platforms (i.e., Vonage and Zoom) for TV-based video calling.
Konnect's user interface is designed as a carousel system (i.e., the user cycles through options on their TV) which the user navigates through by simply responding with a “yes” or “no” from the specially designed remote control. Kraydel has several user-related features, such as the ability to respond to questionnaires and surveys through the TV, which can be used to gather user-related insights such as wellbeing assessments. Callers can use the Konnect app on their smartphones (available for iOS and Android) to make video calls to the user's TV. The system also offers so-called sofa-to-sofa communication. For example, any individual (relative, carer, and healthcare professional) who has a Konnect unit can make calls to the user's TV from their own Konnect unit as long as the user allows access in advance. Users can also upload photos from their phones and upload videos through a dashboard that can be viewed on the TV.
Kraydel's aim was to work with residents of the DHLL and through co-production workshops to share the first prototypes developed in their labs and receive end user's input toward the further development and finalization of these early versions of their devices. As such initial prototypes (Figures 5, 6) of the Konnect units have been used by residents in the DHLL, providing valuable input and feedback for further adjustments.
Kraydel Konnect first prototype.
Kraydel Konnect remote control first prototype.
The testing and development continued with further co-creation workshops, focus groups, and individual interviews (47–49) (Figure 7), ensuring the updated hardware design and the user interface is easy and straightforward and do not provoke technophobia (Figure 8). The co-creation workshops took the shape of 1-2-1 sessions between residents and the developers' devices, supported by user experience and design thinking professionals, exploring technical aspects (shape, materials, colors, usage, and dexterity). At the same time, the focus groups were utilized to gather qualitative feedback on usability feasibility and applicability from both the residents and additional stakeholders (in this case, DHLL staff, residents' carers, and friends). In total, the project included two 1-2-1 sessions as described earlier and six focus groups.
Co-creation workshop.
Kraydel Konnect updated version.
The current Konnect version is also a wellbeing monitoring system that uses onboard sensors (for room temperature and physical movement in the TV room) and has Bluetooth capability with a wide range of third-party devices that are in the process of being integrated. Remote secure cloud for storage and processing of data and Application Programming Interfaces (APIs) connecting local devices to Konnect are already established for digital thermometers and pulse oximeters, and heart rate and to feed data to service provider personnel for remote assessment (Figures 9A, B).
(A, B) Kraydel Konnect current version.
As discussed earlier, existing literature (31) highlights the benefits of various types of living labs, like our DHLL.
Lessons learned
In this case study, we share lessons learned from this development as well as some unique advantages linked to such a community-based DHLL:
Real-life environment. During our testing and evaluation activities, it became apparent that conducting these in participants' own homes provided a more original experience as reflected by their feedback.
Costs. As discussed in the introduction, establishing an LL, for example, in a university environment or a municipality building, is accompanied by costs related to the actual room, facilities, maintenance, utility bills, and staff. On the contrary, our experience showed that establishing a DHLL in the community has the advantage of lower costs. It can be set up in an existing site without any additional costs for extra physical spaces and their accompanying expenses, such as utility bills and/or maintenance as these are covered by the local council. In our case, the used spaces for any of the scheduled activities are either the communal spaces or smaller rooms where we conduct interviews or focus groups.
Sustainability. Further to the maintenance of the physical spaces of living labs, their sustainability requires the investment of personnel and time for traveling and transport, bringing participants together and keeping them engaged. In our case, an important advantage is a fact that the community-based DHLL is “self-sustained.” The group of participants engages in the labs' activities in their own familiar spaces (homes and communal areas) without the need to travel to the university or to another site to continue networking with fellow residents even when there are no active projects. As a result, there is no need for continuous presence or visits of academic staff. In addition, the presence of on-site managers provides the advantage of fast recruitment and resuming of activities once a new project starts through the dissemination of any required material and invitations.
A benefit for any SMEs utilizing such a DHLL is their opportunity to showcase their solutions directly to end users but also stakeholders, including commissioners and decision-makers.
Limitations
Setting and maintaining a living lab can be accompanied by many limitations, but in this case study, we have an opportunity to share our experience reflecting on a single development outside of having explored a summative project evaluation. Based on this experience, the team identified certain limitations that, if considered going forward, would expand the DHLL benefits.
One of the limitations is the demographics of the residents, where, although there is a diverse group of citizens living on this site, these might not represent the diverse community beyond the living lab. A way to mitigate this limitation would be by including more sites from areas across the council that would include hard-to-reach and/or vulnerable populations (i.e., minorities and learning disabilities).
Another limitation of this single development comes from the voluntary commitment of the residents. This results in working with residents that might already be tech-savvy and eager to contribute to such testing. This may exclude valuable input from no digitally literate citizens that hesitate to volunteer in such testing. A future solution could be to increase the number of living labs and provide incentives to potential participants.
Conclusion
In this single-case study, the DHLL proved to be an open innovation ecosystem as it brought together multiple stakeholders sourcing ideas for a small business enterprise, contributing effectively to the user-centered development of the described digital health solution. Following the approach shared in our article, we believe that establishing DHLLs in the community and engaging with the right stakeholders can be a streamlined and straightforward process with the subsequent benefits described.
Data availability statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.
Ethics statement
Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.
Author contributions
All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.
Funding
The research conducted for this article was triggered by TF, KK, and HM involvement in the project A Dynamic and Self-Organized Artificial Swarm Intelligence Solution for Security and Privacy Threats in Healthcare ICT Infrastructures (AI4HEALTHSEC) under grant agreement no. 883273.
The authors are grateful for the financial support of this project which has received funding from the European Union's Horizon 2020 research and innovation programme. The authors wish to thank the Brighton & Hove City council for their valuable contribution to establishing the Digital Health Living Lab.
Conflict of interest
Author AF is employed by Kraydel Ltd. KK is employed by Trustilio B.V. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Author disclaimer
The views expressed in this article represent only the views of the authors and not of the European Commission or the partners in the project, as mentioned earlier.
ReferencesBallonPSchuurmanD. Living labs: concepts, tools and cases. Info. (2015) 17:1–10. 10.1108/info-04-2015-0024SchaffersHGarcia GuzmanJMerzC. An Action Research Approach to Rural Living Labs Innovation. In: Collaboration and the Knowledge Economy: Issues, Applications, Case Studies, IOS Press. (2008).García-GuzmánJFernández del CarpioAde AmescuaAVelascoM. A process reference model for managing living labs for ICT innovation: A proposal based on ISO/IEC 15504. Comput Stand Interf. (2013) 36:33–41. 10.1016/j.csi.2013.07.004AlmirallELeeMWarehamJ. Mapping living labs in the landscape of innovation methodologies. Technol Innov Manage Rev. (2012) 2:12–8. 10.22215/timreview/603SchuurmanD. Bridging the gap between open and user innovation? Exploring the value of living labs as a means to structure user contribution and manage distributed innovation (Doctoral Thesis). University of Ghent: Belgium. (2015).LeminenSNiitamoVPWesterlundM. A brief history of Living Labs: From Scattered Initiatives to Global Movement. In: Proceedings of the Open Living Labs Research Day, (2017) 42–58.DutilleulBBirrerFMensinkW. Unpacking European living labs: analysing innovation's social dimensions. Central Eur J Public Policy. (2010) 4:60–85.BallonPVan HoedMSchuurmanD. The effectiveness of involving users in digital innovation: Measuring the impact of living labs. Telemat Inform. (2018) 35:1201–14. 10.1016/j.tele.2018.02.003McPheeCSantonenTShahANazariA. Reflecting on 10 Years of the TIM Review. Technol Innov Manage Rev. (2017) 7:5–20. 10.22215/timreview/1087ENoLL. (2018). Bill Mitchell, Father of the Living Lab Concept, Passed Away This Weekend. Available online at: http://www.openlivinglabs.eu/news/call-papers-living-labs-openlivinglab-days-2018 (accessed December 15, 2021).LeminenSWesterlundM. Towards innovation in Living Labs networks. Int J Product Develop. (2012) 17:43–55. 10.1504/IJPD.2012.051161NyströmALeminenSWesterlundMKortelainenM. Actor roles and role patterns influencing innovation in living labs. Ind Market Manage. (2014) 43:483–95. 10.1016/j.indmarman.2013.12.016BuhlJGeiblerJEchternachtLLinderM. Rebound effects in Living Labs: Opportunities for monitoring and mitigating re-spending and time use effects in user integrated innovation design. J Clean Prod. (2017) 151:592–602. 10.1016/j.jclepro.2017.03.001AlmirallEWarehamJ. Living Labs: arbiters of mid- and ground-level innovation. Technol Analy Strat Manag. (2011) 23:87–102. 10.1080/09537325.2011.537110LeminenSNyströmAWesterlundMA. typology of creative consumers in living labs. J Eng Technol Manage. (2015) 37:6–20. 10.1016/j.jengtecman.2015.08.008BakiciTAlmirallEWarehamJ. A smart city initiative: the case of Barcelona. J Knowl Econ. (2012) 4:135–48. 10.1007/s13132-012-0084-9NevensFFrantzeskakiNGorissenLLoorbachD. Urban Transition Labs: co-creating transformative action for sustainable cities. J Clean Prod. (2017) 50:111–22. 10.1016/j.jclepro.2012.12.001BulkeleyHCoenenLFrantzeskakiNHartmannCKronsellAMaiL. Urban living labs: governing urban sustainability transitions. Curr Opin Environ Sustain. (2016) 22:13e17. 10.1016/j.cosust.2017.02.003LeminenSRajahonkaMWesterlundM. Towards third generation living lab networks in cities. Technol Innov Manage Rev. (2017) 7:21–35. 10.22215/timreview/1118RodriguesMFrancoM. Importance of living labs in urban Entrepreneurship: A Portuguese case study. J Clean Prod. (2018) 180:780–9. 10.1016/j.jclepro.2018.01.150SchuurmanDTõnuristP. Innovation in the public sector: exploring the characteristics and potential of living labs and innovation labs. Technol Innov Manage Rev. (2017) 7:7–14. 10.22215/timreview/1045DekkerRFranco ContrerasJMeijerA. The living lab as a methodology for public administration research: a systematic literature review of its applications in the social sciences. Int J Public Admin. (2020) 43:1207–17. 10.1080/01900692.2019.1668410CriadoJDiasTSanoHRojas-MartínFSilvanAFilhoA. Public innovation and living labs in action: a comparative analysis in post-new public management contexts. Int J Public Admin. (2021) 44:451–64. 10.1080/01900692.2020.1729181McGannMWellsTBlomkampE. Innovation labs and co-production in public problem solving. Public Manag Rev. (2019) 23:297–316. 10.1080/14719037.2019.1699946OlejniczakKBorkowska-WaszakSDomaradzka-WidłaAParkY. Policy labs: the next frontier of policy design and evaluation. Policy Politics. (2020) 48:89–110. 10.1332/030557319X1557923042010823599727FølstadA. Living labs for innovation and development of information and communication technology: a literature review. Electron J Virt Org Netw. (2008) 10:99e131.FranzY. Designing social living labs in urban research. Info. (2015) 17:53–66. 10.1108/info-01-2015-0008LeminenSWesterlundMA. framework for understanding the different research avenues of living labs. Int J Technol Market. (2016) 11:399–415. 10.1504/IJTMKT.2016.079731McLoughlinSMaccaniGPrendergastDDonnellanBLeroN. Living labs: a bibliometric analysis. In: Hawaii International Conference on System Sciences (HICSS). (2018). 10.24251/HICSS.2018.563WesterlundMLeminenSRajahonkaM. A Topic modelling analysis of living labs research. Technol Innov Manage Rev. (2018) 8:40–51. 10.22215/timreview/1170HossainMLeminenSWesterlundMA. systematic review of living lab literature. J Clean Prod. (2019) 213:976–88. 10.1016/j.jclepro.2018.12.257DietrichTGuldagerJLykPVallentin-HolbechLRundle-ThieleSMajgaardG. Co-creating virtual reality interventions for alcohol prevention: living lab vs. Co-design. Front Public Health. (2021) 9. 10.3389/fpubh.2021.63410233791269FuglsangLHansenAMergelIRøhnebækM. Living labs for public sector innovation: an integrative literature review. Administrative Sciences. (2021) 11:58–65. 10.3390/admsci11020058AlderwickH. Is the NHS overwhelmed?BMJ. (2022) 376:o51. 10.1136/bmj.o5135017162SundaralingamAFotisT. Making the Case for Responsible Innovation. The J mHealth. (2019) 6:25–6.AhmedFFattaniMTAliSREnamRN. Strengthening the bridge between academic and the industry through the academia-industry collaboration plan design model. Front Psychol. (2022) 13:875940. 10.3389/fpsyg.2022.87594035734456TribertiSBrivioE. User-centered design approaches and methods for P5 eHealth. In: Pravettoni, G., Triberti, S. (eds) P5 eHealth: An Agenda for the Health Technologies of the Future. Cham: Springer. (2020). 10.1007/978-3-030-27994-3_9GreenhalghTWhertonJPapoutsiCLynchJHughesGA'CourtC. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res. (2017) 19:e367. 10.2196/jmir.877529092808Brighton.ac.uk. (2017). Leading the way in regional partnership working. Available online at: https://www.brighton.ac.uk/business-services/strategic-partnerships/leading-the-way-in-regional-partnership-working.aspx (accessed March 5, 2022).Local.gov.uk. (2017). Leading Places. Available online at: https://www.local.gov.uk/topics/devolution/leading-places/leading-places-phase-1/brighton (accessed February 5, 2022).HIBrighton,. (2019). Home | HI Brighton. Available online at: https://www.hibrightongroup.com/ (accessed March 7, 2022).StåhlbröstAHolstM. The Living Lab Methodology Handbook. (2013). Available online at: http://www.ltu.se/cms_fs/1.101555!/file/LivingLabsMethodologyBook_web.pdf (accessed February 15, 2022).AversanoPBaccarneBSchuurmanD. Implementation Living Lab Methodology. (2016). Available online at: https://cordis.europa.eu/docs/projects/cnect/5/632905/080/deliverables/001-D33ImplementationLivingLabMethodology.pdf (accessed February 20, 2022).HigginsAKleinS. Introduction to the living lab approach. In: Y. H. Tan, N. Björn-Andersen, S. Klein, and B. Rukanova (Eds), Accelerating Global Supply Chains with IT-Innovation. Berlin: Springer, Heidelberg. (2011) p. 31–36. 10.1007/978-3-642-15669-4_2MalmbergKVaittinenIEvansPSchuurmanDStåhlbröstAVervoortK. Living Lab Methodology Handbook. Zenodo. (2017).Dell'EraCLandoniP. Living lab: A methodology between user-centred design and participatory design. Creativ Innov Manage. (2014) 23:137–154. 10.1111/caim.1206134402413ManziniE. Design, When Everybody Designs: An Introduction to Design for Social Innovation. Cambridge: MIT Press. (2015). 10.7551/mitpress/9873.001.0001SteenKvan BuerenE. Urban Living Labs: A Living Lab Way of Working. Amsterdam:AMS. (2017).LaneA,. (2018). AgriLink Living Lab Toolbox [Ebook]. The Open University. Available online at: https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwiWuYvs9eH4AhXSN8AKHVecBA4QFnoECBUQAQ&url=https%3A%2F%2Fwww.open.edu%2Fopenlearncreate%2Fmod%2Foucontent%2Folink.php%3Fid%3D173545%26targetdoc%3DAgriLink%2BLiving%2BLab%2BToolbox&usg=AOvVaw0F50jtnPX8RmO166M2rYtf (accessed June 29, 2022).