AI-supported approach for human-building interaction implemented at furniture scale

Human-Building Interaction (HBI) relies on sensor-actuator networks that are increasingly supported by Artificial Intelligence (AI). This paper presents a novel AI-supported Design-to-Robotic-Production-Assembly and -Operation (D2RPA&O) approach for reconfigurable furniture. It involves a multidisciplinary approach that relies on the integration of various domains such as architecture, robotics, computer, and material science. It contributes to the advancement of HBI by employing spatial reconfiguration relying on AI and lightweight material design, which is of relevance, particularly when the furniture consists of non-identical but similar components that are re − / configured in a variety of possible combinations.


Introduction
Human-Building Interaction (HBI) relying on sensor-actuator networks is increasingly supported by Artificial Intelligence (AI) at various scales (Alavi et al., 2019).In the case study presented in this paper, the novel AI-supported approach relies on Design-to-Robotic-Production-Assembly and -Operation (D2RPA&O) methods that are employed for developing reconfigurable furniture.These methods integrate robotics in building processes and buildings (Bier and Knight, 2014;Robotic Building and Bier, 2018;Pillan et al., 2020) by (a) computationally designing and robotically constructing building components (D2RP&A), and (b) by operating those components interactively (D2RO).While some of the methods and techniques used by the involved disciplines in the furniture design process are only developed conceptually, others are virtually and/ or physically prototyped.

State-of-the-art
HBI goes back to notions of Interactive Architecture (IA) relying on intelligence integrated into the built environment that has been defined already in the 70s as 'architecture machines' and 'intelligent environments' adapting to their users' needs (Negroponte, 1975;Pask, 1975).Later, Ambient Intelligence (AmI) was defined as a built environment imbued with computing hardware and software technology to benefit its occupant(s) (Zelkha et al., 1998).More recently, architecture concerned with interaction between human and non-human agents has Bier et al. 10.3389/fcomp.2024.1295014Frontiers in Computer Science 02 frontiersin.orgproliferated at various scales ranging from furniture to urban scale (Negroponte, 1975;Pask, 1975;Zelkha et al., 1998;Fox and Kemp, 2009;Fox, 2010;Oosterhuis and Bier, 2013;Bier and Knight, 2014;Robotic Building and Bier, 2018;Alavi et al., 2019;Pillan et al., 2020).At furniture scale, examples such as The Big Data (2016) furniture and Media Block Chair (2012) are relevant as they demonstrate furniture reacting to people's movement by changing color. 1 Such furniture takes advantage of location-based context-aware services and Internet connectivity that are ubiquitously available.Big Data furniture, for instance, communicates with users by changing colors in response to users' movement.The movement and behavior of users as well as environmental data patterns are shared on an online database that is used to modulate user experience.
While these examples rely on sensor-actuator networks and involve some level of intelligence they do not involve AI relying on Computer Vision (CV)2 for spatial reconfiguration as in the presented case study.With AI progressing and evolving in time through various stages of innovation and practical application, spatial reconfiguration is challenged to integrate multimodal technologies and improve HBI.

Contribution
The presented case study implemented with MSc students at Technical University (TU) Delft integrates AI-supported multimodal technologies into an alternative design approach relying on D2RPA&O methods.It addresses both the technical as well as architectural aspects from the early stages of the design process.It introduces as main features spatial reconfiguration relying on object recognition and lightweight material design.These are of relevance, particularly when the furniture consists, as in this case, of non-identical but similar furniture components that are re−/configured in a variety of possible combinations.

Implementation
Implemented with a group of architecture students tutored by experts from architecture, robotics, and computer science, as well as interaction and structural design, the project focused on the development of furniture components that offer reconfigurable indoor seating areas for accommodating various activities such as workshops (involving collaborative work sessions), events (requiring a stage and seating for the audience), group and individual studying activities next to temporary exhibitions taking place simultaneously or in short sequence.Hence spatial reconfiguration was paramount.Furthermore, the furniture was supposed to be taking structural, functional, environmental, and assembly considerations into account and have degrees of porosity, where the degree and distribution of porosity, i.e., density would facilitate both passive (structural strength, physical comfort, etc.) and active behaviors (interaction, etc.).
The approach involved D2RPA&O methods with the aim to improve spatial experience, process-and material-efficiency as well as embed intelligence in building processes and buildings by (a) computationally and robotically optimizing design and material production, (b) AI-supporting dis−/re−/assembly process, and (c) embedding sensor-actuators that enable interaction with users and the physically built environment.While these procedural steps were all implemented, they achieved various degrees of completion.For instance, the material production was implemented on a fragment, the AI model was tested on a synthetic data set, and the interaction was developed only conceptually at this stage.Future work will involve completing all steps in order to test the proposed varied and multiple use of the space in the library by allowing users to modify the configuration of furniture to meet their changing needs.

D2RPA&O
The D2RPA&O process relied on a Voronoi-based design that facilitates spatial reconfiguration by detaching and reattaching furniture components, i.e., booths to create various functional configurations.The challenge for users to identify availability of components and possible reconfiguration modes ranging from dispersed to clustered configurations that are serving varying activities has been addressed by means of interactive lighting, color coding, and AI-supported object recognition (Figure 1).Furthermore, the challenge for users to move components from one location to another has been addressed by implementing a highly porous, lightweight material design.All aspects were developed at conceptual level and achieved various degrees of implementation ranging from design to virtual and physical prototyping.

Conceptual design
The need for spatial reconfiguration has been identified by investigating user demands that were captured through a series of behavioral observations in the library space.These included the analysis of current occupancy patterns in terms of number of occupants and type of activities performed over one week.Additional unstructured observations were implemented ahead of the study by the students as they were themselves users of the library.Patterns of stagnation and movement were observed to understand how long users spend in various sections of the library and how users interact with others and various concerns were identified as for instance the reduced reconfigurability options of the furniture and lack of insight into available furniture meeting requirements for various activities and their location.
The occupancy pattern of the library showed large variability in terms of number of occupants during the year (with strong fluctuations in exam vs. vacation periods) and activities.However, even if the largest number of occupants were single users, the space was also used by groups of people for joint activities and one of the problems was the lack of furniture that could accommodate and adapt to user demands during respective activities.Reconfigurability and adaptability emerged therefore as the key objective of the presented design solution.
Reconfiguration has various challenges in terms of function, form, and requirements for lightweight.While precedents such as Media Block Chair presented interactive features, Cloud City, Office Excavate, FRGM: 1, and Voronoi Helmet3 showcased the potential of Voronoibased design at building, component, and material scale.Hence, these approaches were adopted and enhanced using AI.

Computational design
Aspects of computational design were addressed by generating a Voronoi-based design at macro (i.e., building), meso (i.e., component), and micro (i.e., material) scales.This implies that the Voronoi-based material design of components and the assembly of components into a larger structure to facilitate spatial reconfiguration is scalable.
In order to optimize production and assembly of the furniture components, functional, structural, and material design studies were implemented.To optimize functional requirements in relation to activity patterns, reconfiguration by dis−/ reassembly of furniture components was chosen.To structurally optimize the Voronoi-based furniture components, a Karamba3D simulation that provided insight into the forces and deformations of the components, both for the isolated wireframe as for the surface elements (Figure 2) was implemented.The Voronoi-based material design with various cell densities is mapped onto the frame and surface elements according to the optimization results.The variable densities pattern is efficient in terms of the material used versus the resulting degree of stiffness of the object and weight reduction.
The pattern is calculated based on the stress distribution (Figure 3) and the load cases that are used are the combination of the loads from the overall model and the forces on the component due to its own weight.The main aim of this study was to improve the mechanical performance of the components while keeping them as lightweight as possible.After all, excessive weight would considerably impact their movability.
The considered material is thermoplastic elastomer (TPE) with a density of 1.1 g/cm3.The paneling geometry is thickened to 10 mm and based on this volume distributed loads are applied to the surfaces of the panels.Loads applied are line loads based on the weight of the panels.The resulting principal stress concentrations are translated into

Interaction design
The primary goal to create a user-centered spatial experience and provide effective and adaptive spatial configurations to meet current user needs in the library includes consideration for accommodating changing use (single or group use, with various levels of privacy) and user-furniture interaction through responsive lighting in order to communicate users the potential availability of furniture components (Figure 1).
The user-light interaction system was conceptually designed to provide visual comfort and information to users.The use of indoor lighting for communicating with users while providing appropriate indoor visual conditions was developed and tested in previous work (Ackerly and Brager, 2013;Meerbeek et al., 2016;Luna-Navarro et al., 2020) and has been applied to this case study.While Light-emitting Diodes (LEDs) are integrated into the furniture to provide comfortable direct lighting, various, changing colors are used to communicate if it is available.Using an active system of pressure sensors integrated into the seats, the furniture responds to various types of use.Outward lighting signals availability, whereas inward lighting illuminates the occupied components while in use.
The challenges related to the integration of micro-controllers, sensors and actuators, i.e., LEDs into the furniture components involved ensuring that the physical and virtual subcomponents communicate data to the cloud and data is processed fast enough to allow for smooth uninterrupted interaction.This has been, however, only developed conceptually in this study, with the physical prototyping focusing on the materialization of variable stiffness with cushioning based on structural and ergonomic considerations.

Material design and robotic prototyping
Component and material design was Voronoi-based and involved consideration with respect to stability, lightweight, and comfort requiring variable stiffness and 3D printing without support. 4The variable stiffness concept developed in previous research (Hidding et al., 2019) has been implemented in this case study in order to create cushioning for the seating areas while keeping the structural integrity of the furniture components, i.e., booths intact.The cushioning principle was prototyped by robotically milling into Expanded Polystyrene (EPS) that in this case was emulating the support surface, which has high stiffness.On top of the support surface is the cushioning surface with low stiffness and is implemented by 3D printing with Thermoplastic Polyurethane (TPU).The imprint of the Voronoi-based pattern designed to serve as cushioning surface was milled out of the EPS so the 3D printed part could be fitted onto the surface (Figure 4).
The variable stiffness design of the cushion is based on structural analysis, where the stiffness is controlled by the cell sizes implying that the smaller the cells, the denser the material and, therefore, the stiffer the structure.The design of the cells with their varying size and distribution depends on the structural analysis as well as the requirements for printing without support.In this context, the material design and prototyping approach relied on optimized material distribution based on structural optimization ensuring minimized material use and reduced production time.When compared with other production methodologies, the robotic prototyping of the Voronoibased components has several major advantages.Firstly, it allows the production of complex geometries by linking the parametric design with the production data of the robotic equipment (D2RP).In the case of robotic 3D printing, the angles of the geometry have to be optimized 4 This implies that the angles of the faces of the Voronoi cells have to be within 3D printing tolerances with printing angles between −45 to +45 degrees in relation to the printing bed. to be within the angle constraints of −45 to +45 degrees for printing without support material (Hidding et al., 2019).Also, the size of the components and subcomponents are optimized to fit in the bounding box determined by the reachability of the robotic arm.The printability of the material, TPE, and its recyclability have to be considered as well.

Computer vision and artificial intelligence
Computer Vision (CV) as an Artificial Intelligence (AI) module is used for the identification of (a) detached furniture components and (b) the places where the components are supposed to be reattached (Figure 5).Specifically, a Deep Learning (DL) model 5 is trained from scratch on a synthetic dataset.The input and output of the model are images, pointing for the input at a specific detached, i.e., stand-alone 5 Deep learning is a machine learning method, which is based on artificial neural networks that are built using principles of organization in the biological neural networks of animal brains (https://news.mit.edu/2017/explained-neural-networks-deep-learning-0414).
component, and for the output indicating its position and orientation in the constellation.

Task description
The CV task was initially cast as a 3D puzzle reconstruction, where all the components are considered unique pieces of the constellation having their own individual base configuration.The problem could be formulated as a visual placement recognition for machine vision, given adequate training data with various configurations of missing pieces in the constellation (Figure 5).Even though an exciting CV challenge, the 3D puzzle reconstruction appeared out of the scope for this student project.The problem has then been reformulated to object recognition which is a well-known computer vision task with available model architectures.The components are labelled with ID numbers that are the keys to the correct placement in the constellation.Thus, the challenge is to predict the object's ID number given an input image of the component taken by the user.Material design (left) and prototyped fragment (middle and right) using robotic subtractive and additive methods.to the specific configuration for this problem, no pre-trained model was available for effective deployment, hence, the students trained an object detection model from scratch to meet the requirements for this project.For the training set, Voronoi-based components are randomly distributed in a 3D model of the library (Figure 6), and around 3,000 images were generated from the scattered objects from different viewpoints.80% of the data was used for training and 20% for evaluation.The synthetic dataset is curated by simulating a realistic scenario where the images are taken from real objects at the library.A generalization gap is, however, expected due to the distribution mismatch caused by lighting, shadowing, and occlusion, which are present in a real setting.This gap is increased by the differences between the 3D model and the 3D-printed elements.

Dataset
The differences between the digital files and physical objects after 3D printing could be significant enough for the AI to perform poorly on photographs of these components.Creating 3D models that have the 3D-printed layers in the models could help to create training images that more closely resemble the produced components.Labelled photographs of the produced components could also be used to re-train the model in the case of poor performance.This has been, however, not the focus of this study and will be implemented in the next step.

Model and results
The Yolo object detection model (Redmon et al., 2016) is trained on the created dataset to predict the object's ID number for the Voronoi-based components with acceptable accuracy and confidence.An online demo 6 and screenshots from the demo (Figure 6) show randomly placed components that are detected in real time using the trained object detection model.The probability of specific components being present within the bounding box is shown in the right text box next to the box (Figure 6 right).These probabilities are used to classify an object as a true positive (correctly detected) or false positive (wrongly detected).Setting a confidence level as a hyperparameter for measuring the precision score for the object detection model.In this case, a confidence level of 0.95 from 1.00 has 6 Link to RB-page: http://www.roboticbuilding.eu/project/human-robot-interaction-for-d2ra/.
been achieved.There is a trade-off between precision and recall, therefore for such a high confidence level false negatives are increasing.
3 Discussion and future work Presented case study outlines D2RPA&O strategies for AI-supported furniture reconfiguration as part of a larger HBI approach.Developed computational, interaction, material design, robotic production, and AI-supported assembly approaches (Figure 7) facilitate furniture adaptation to various functions and spatial requirements by allowing users to modify furniture configurations to meet changing needs in the multi-functional space of the TU Delft library.
In the presented case study, individual furniture components were re−/ arranged to create various spatial configurations by moving, de−/ re−/ attaching, and/ or nesting together components.In this context, two main configurations, clustered (Figure 5) and dispersed (Figure 6), were considered.Additional solutions for individual cluster configurations will be investigated in the next iteration to identify the limitations of the system in terms of number of furniture components and their possible recombination in relation to the available space.Furthermore, by physically prototyping whole components and using photographs of the produced components to re-train the AI model the feasibility of this approach for real-life applications will be demonstrated.
In furniture components integrated interactive lighting was designed to facilitate user engagement via color-and intensity-changing LED lights that adjust to users' preferences and communicate various states (un−/ available) and moods (open/ closed).In this context, HBI relied on the integration of technology, design, and user-centered approaches to create spaces that are responsive to the needs and behaviors of their occupants.It involved developing an understanding of the needs, preferences, and behaviors of the people who occupy the space.However, in order to make reconfiguration accessible to people of all abilities the robotization of the system is considered in the future.

Conclusion
While employed AI-supported approach presented in this paper is not new, its integration with D2RPA&O methods advances The evaluation and testing were implemented by computationally designing the structure, simulating the whole process and prototyping a fragment at 1:1 scale.Since the focus was on the integration of various research areas involving architecture, computer science, interaction, and structural design, the study met its objectives as all aspects were considered although D2RO remained at conceptual and computational design levels, while D2RP&A reached virtual and physical prototyping levels (Figures 7, 8).
The use of lightweight material design for physical dis−/ reassembly and the use of lights to provide user-building interaction strategies by communicating to users which furniture components and when they are available were conceptually and to some degree physically explored, though, tests with users still need to be conducted in order to evaluate user response and satisfaction with proposed interaction strategy.Also, the use of AI on physically prototyped components needs verification and operability by users with variable physical abilities needs consideration, which will be addressed in the future.
The collaborative approach required the integration of all aspects from the very beginning of the process.While architecture focused on function and form meeting structural and material design requirements, the interaction aspects relied on the integration of CV.The project, however, did not get to the point of testing the CV approach on a physically prototyped furniture component missing to establish the feasibility of the CV to adequately operate in situ, i.e., under realistic conditions.In future, the feedback from the performance of the CV on the physically prototyped furniture component will be used to update the training of the algorithm.For Diagram showing the workflow of AI-supported assembly using an app on a phone to identify the components and their base placement location in real-time.Diagram showing D2RP process reaching prototyping level.

FIGURE 1
FIGURE 1 Red movable Voronoi-based component (left) and available component lighted green (left and right).

FIGURE 2
FIGURE 2 Structural design at component scale involved analysis of tension and compression (left and left-middle) and deformation and stiffness (right-middle and right).

FIGURE 3
FIGURE 3Structural design at material scale without and with optimization (left and right, respectively) showing variation in density of Voronoi cells that are represented in varying colors on x and y coordinates in meters.

A
synthetic training and validation dataset is created to develop a deep learning model for the object detection task.Due

FIGURE 5
FIGURE 5Object and placement recognition for the missing component (left), input image with the missing part (right).

FIGURE 6
FIGURE 6Random camera and components distribution within the 3D model of library (top view left), an example input image from the setting (middle), and object recognition (right) implemented via an app on the phone.

FIGURE 8
FIGURE 8 research purposes the data set used in this study can be obtained by contacting the corresponding author. legitimate