Interactive Simulations of Biohybrid Systems

2 In this article we present approaches to interactive simulations of biohybrid systems. 3 These simulations are comprised of two major computational components: (1) Agent-based 4 developmental models that retrace organismal growth and unfolding of technical scaffoldings, 5 and (2) interfaces to explore these models interactively. Simulations of biohybrid systems allow us 6 to fast forward and experience their evolution over time based on our design decisions involving 7 the choice, conﬁguration and initial states of the deployed biological and robotic actors as well 8 as their interplay with the environment. We brieﬂy introduce the concept of swarm grammars, 9 an agent-based extension of L-systems for retracing growth processes and structural artefacts. 10 Next, we review an early augmented reality prototype for designing and projecting biohybrid 11 system simulations into real space. In addition to models that retrace plant behaviours, we specify 12 swarm grammar agents to braid structures in a self-organising manner. Based on this model, 13 both robotic and plant-driven braiding processes can be experienced and explored in virtual 14 worlds. We present an according user interface for use in virtual reality. As we present interactive 15 models concerning rather diverse description levels, we only ensured their principal capacity for 16 interaction but did not consider efﬁciency analyses beyond prototypic operation.We conclude this 17 article with an outlook on future works on melding reality and virtuality to drive the design and 18 deployment of biohybrid systems.

considered in the scope of one or many simulations. Serving the simulated development of a biohybrid 48 system in-situ not only challenges the systems' engineers in terms of computational efficiency-the in-situ 49 projection also needs to be supported by an accessible user interface which considers the intricacies of 50 biohybrid systems as well as the complexities of their physical environment. 51 In this article, we present our ongoing efforts towards according technologies at the intersection of 52 biohybrid systems and their human users. Our goal is to simulate biohybrid systems in realtime and to make 53 these simulations interactive. Prototyping, planning, and deployment of biohybrid system configurations 54 represent the immediate use cases for the corresponding realtime interactive simulations. Accordingly, our 55 approach considers realtime-capable simulation models of plant growth and dynamics as well as robotic 56 interactions. Generative models such as L-Systems and generic agent-based modelling approaches paved 57 the way for the models we devised for interactive biohybrid simulations. We briefly survey these preceding 58 works in Section 1. Next, we introduce our interactive modelling approach for biohybrid systems in Section 59 2. More specifically, we adjust a swarm grammar representation to incorporate various developmental 60 behaviours of plants such as lignification, phototropism and shade avoidance. We also utilise the agent-61 based swarm grammar approach to develop futuristic models of robotic units braiding scaffolding structures 62 as currently worked on in the biohybrids research community. In Section 3, we present an augmented 63 reality (AR) prototype for the design of biohybrid systems. The specific challenges introduced by the 64 augmented reality setting, such as remodelling real-world lighting conditions or limited input capabilities Step 1 Step 2 Step 3  instance, an efficient implementation allows the user to generate strings and interpret them visually fast 161 enough as to explore the model space and adjust the concrete instances' parameters for the growth within 162 interactively selected regions of interest (Onishi et al. (2003)). This idea was resumed by Hamon et al.

163
(2012), who made it possible not only to let the L-System grow based on contextual cues such as collisions 164 but also to change the L-System formalism interactively, on-the-fly. for instance by Shaker et al. (2014). The authors demonstrate the concept in the context of a digger agent that leaves corridor trails with chambers at random points in a subsurface setting. They distinguish 172 between an uninformed, "blind" agent and one that is more aware of the built environment only places new 173 chambers that do not overlap with previously existing ones. Figure 2 shows this representative constructive 174 agent-based example. In our implementation, both agent types had a chance of 10% of changing their 175 direction and of 5% of creating a chamber at each step. The chamber dimensions were randomly chosen 176 between 2 to 5 times of a single corridor cell.

SWARM GRAMMARS
In l-systems and other grammatical developmental representations, neighbourhood topologies and 178 neighbourhood constraints (who informs whom and how?) are mostly embedded in production rule 179 sets. This is different for the agent-based approach, which also explains the need for awareness about 180 the built environment to achieve coordinated constructions (see Figure 2). Overall, the quality of agent-181 generated artefacts greatly depends on the ingenuity and complexity of their behavioural programme and 182 on the simulated environment.

183
An important advantage over grammatical representations such as l-systems is the simplicity of extending 184 agent-based systems. Sensory information, behavioural logic or the repertoire of actions can be easily and 185 directly changed. Dependencies to other agents or the environment can be designed relative to the agent 186 itself and the topology among interaction partners can evolve arbitrarily based on a modelled, possibly 187 dynamic environment and arbitrary preceding multi-modal interactions. The inherent flexibility of agents 188 (due to threefold design of sensing/processing/acting) facilitates the resulting system to be interactive not 189 only with respect to a modelled environment but also to user input that is provided on-the-fly. are very simple, so-called reactive agents that interact merely spatially. Due to their simplicity, they lend 198 themselves well for a primary agent model to be extended by the l-system concept of generative production.

199
As a result, SGs augment boids to leave trails in space and to differentiate and proliferate as instructed by a 200 set of production rules.

201
Formally speaking, a swarm grammar SG = {SL, ∆} consists of a system SL that is comprised of 202 an axiom α and a set of production rules P , and a set of agent types or agent specifications ∆. Each  each direction. SL 1 only deploys agent specification A, SL 2 and SL 3 only specification B, and SL 3 also 213 lets its agents reproduce with a probability of 1% at each step. To this date, most swarm grammar implementations incorporate the flocking model by Reynolds (1987), 215 where simple local reactive acceleration rules of spatially represented agents drive the flight formation of 216 agent collectives. Hence, next to attributes of the agents' display, e.g. their shape, scale and colour, the 217 agent specifications δ also consider the parameterisation of the agents' fields of view and the coefficients 218 that determine their accelerations with respect to their perceived neighbourhoods. In particular, these 219 coefficients weigh several different acceleration "urges". These include one that drives an agent to the 220 geometric centre of its peers (cohesion), one that adjusts its orientation and speed towards the average 221 velocity of its peers (alignment), one that avoids peers that are too close (separation) as well as some 222 stochasticity. The field of view that determines the agents' neighbourhood perception is typically realised 223 by a viewing angle and by testing proximity (within a maximal perception distance, potentially triggering 224 uneasy closeness).

225
Over the years, swarm grammars have evolved in different directions, some implementations featuring 226 agents that individually carry the production rules along in order to rewrite them based on local needs or 227 store/retrieve them alongside the agent's other data (von Mammen and Edenhofer (2014)). This modeling which yields plausible outcomes of the plants' evolution. By means of swarm grammars, we can specify 242 arbitrary agent properties, behaviours and production rules to drive a computational developmental model.

243
Those processes that are observed and described at the level of a plant individual represent an adequate 244 level of abstraction for realising interactive biohybrid system simulations. As an optimisation step, groups 245 of individuals might be subsumed and be calculated as single meta-agents (von Mammen and Steghöfer 246 (2014)) but individual plants are the basic unit of abstraction as their influence on the biohybrid system 247 matters. Therefore, to let interactive swarm grammars retrace biological growth and dynamics more closely, 248 we started incorporating various behavioural processes exhibited by different plants to different degrees.

249
Among the most common behaviours are the growth of a plant, its movement, orientation towards light, 250 avoidance of shadow, bending, and lignification (for a general introduction, see for instance Stern et al.

251
( 2003)). In the following paragraphs, we shed light on our implementations of these behaviours and the 252 underlying, abstract models.  (2006)). Later, these components were unified and arbitrary 256 living agents or inanimate building blocks were placed by the simulated agents based on local interaction 257 rules rather than grammatical production rules (von Mammen and Jacob (2009)). In order to retrace the 258 dynamics of plant physiology, we decided to follow the original approach, keep the tip of our abstract plant 259 model separate from the stem's segments and assign very clear capabilities to these primary and secondary 260 data objects. In order to support the dynamics arising from interdependencies between the stem's segments, 261 we introduced a hierarchical data structure to traverse the segments in both directions, also considering 262 branches. This traversal is required to retrace the transport of water, sugar and other nutrients but also to 263 provide a physical, so-called articulated body structure. Figure 4 shows a swarm grammar with rewrite  In our abstract plant model, the tip determines the direction of growth by moving upwards (gravitropism) 270 and in accordance with the lighting situation (phototropism and shade avoidance). Growth is primarily 271 realised by repeatedly adding segments to the plant's body that are registered as children in the hierarchy.

272
The conceptual translation of biological growth to this additive process follows the original swarm grammar 273 model which is shown in Figure 3. Secondarily, the segments grow in diameter, increasing the transport  plane P XY and the unit vector e Z in Z direction, a bending target direction d t i is calculated using Equation

295
(1). Then, incorporating the stiffness and the integration time step, we compute the new orientation, a 296 quaternion R i of the segment as the linear quaternion interpolation lerp between its current orientation and 297 the influence by all of its children as summarised in Equation (2), whereas the function rotation yields a 298 quaternion oriented towards a given vector. In Figure 5(a) the process of lignification is depicted by means 299 of a branching swarm grammar. A simple colouring scheme is directly mapped to the hierarchy to illustrate 300 the age and the degree of lignification of the respective segments. In Figure 5(b) the growth target of a 301 swarm grammar is slightly shifted to the left. In this way, the plant bends based on its own weight.

Phototropism and Shadow Avoidance 303
According to the basic swarm grammar implementation, we expressed branching processes as production 304 rules. Currently, exceeding a given nutritional value triggers the respective rules. Other conditions, 305 for instance relating to the achieved form or considering pruning activities by a gardener, may trigger 306 productions just the same. Phototropism, i.e. the urge to grow towards light, is realised as follows. A set of 307 light sources is iterated and, if the respective light is activated, a raycast, i.e. a projected line between the 308 two objects, reveals whether the light shines on a given segment or not: The raycast may not collide with 309 other objects and the angle between the light source and the segment may not exceed the angle of radiation.

310
In this case, the distance vector to the light source d l i down-scaled by some constant c ∈ [0, 1], the stability t=52 t=90 t=120 t=140 t=180 t=240

330
The advantage of the present model over other approaches lies in its interactivity. This means, due to 331 its algorithmic simplicity, its incremental growth procedure and the underlying data model, it works in 332 realtime. As a consequence, the model can be utilised in interactive simulations in which human users 333 can seed plants, place obstacles, light sources, scaffolds or robots to tend the plants. We deem this a 334 critical aspect of simulations of biohybrid systems due to their inherent complexity: The great numbers of 335 interacting agents, their ability to self-organise and to reach complex system regimes, also due to constant 336 interaction with the potentially dynamic and partially self-referential environment, makes it mandatory to 337 develop a notion of a specific system's configuration's impact before deployment. In the following section, 338 we present interfaces that can harness interactive simulation models for designing and planning biohybrid 339 systems.

ROBOT GARDENS AR
We previously presented an early prototype of an augmented reality system for designing and exploring

435
Braids are comprised of multiple threads that are pairwise interwoven. In order to create such a structure, 436 a simple algorithm can be formulated, where a specific thread is identified based on its relative position 437 to its neighbour threads and folded to cross them. Figure 11(a) shows our first approach to retrace such a 438 centralised algorithm. In an open, biohybrid system, the agents-whether robots or biological organisms-439 need to act autonomously and in a self-organised fashion. Therefore, we created an according behavioural 440 description that can be performed by each agent individually and globally results in a braided structure. In 441 Figure 11(b), the latter, self-organised approach is shown in the context of two braiding swarm grammar 442 agents: If a neighbour is close enough, they start rotating around the axis between the two. Braids across 443 several threads can, for instance, be achieved by (1)  towards the opposite end of the flock while the others continue as they were. As this multi-agent braid 446 takes considerable effort in terms of velocity regulation and parameter calibration as seen in Figure 11(c),

447
we relied on a two-agent braiding function for our early experiments. In order to guide the braid agents, 448 the user can place objects such as the plate in Figure 12(a) in VR space, which lets the agents deflect.

449
Alternatively, as shown in Figure 12(b), so-called braiding volumes (in analogy to "breeding volumes" 450 used by von Mammen and Jacob (2007)) can be deployed to enclose agents within specific spaces. These 451 braiding volumes can be placed seamlessly as seen in the next paragraphs to provide arbitrary target spaces.

453
In a first set of experiments, we asked students with computer science background as well as architecture but one result is already emerging. It took the testers very little time (roughly 2 to 3 minutes) to familiarise 457 themselves with the various aspects of the user interface. Therefore, we assume that it offered a very 458 shallow learning curve despite the inherent complexity of the combined task of navigation, transformation, 459 placement and simulation control. We could also learn that both student groups were intrigued by the 460 autonomy of the braiding agents but due to a lack of explanation, they could not fully retrace the individual 461 agents' behaviours in different situations. Clearly this is one of the aspects we aim at working next. Figure   462 13 shows examples of braided structures that were captured during the experiments. During the subsequent 463 interviews, especially the architecture students and one architecture professor stressed that they foresaw 464 great potential for biohybrid systems in design and construction and that they are convinced that research 465 towards according simulations and user interfaces is crucial for its realisation.