<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD Journal Publishing DTD v2.3 20070202//EN" "journalpublishing.dtd">
<article article-type="research-article" dtd-version="2.3" xml:lang="EN" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">
<front>
<journal-meta>
<journal-id journal-id-type="publisher-id">Front. Robot. AI</journal-id>
<journal-title>Frontiers in Robotics and AI</journal-title>
<abbrev-journal-title abbrev-type="pubmed">Front. Robot. AI</abbrev-journal-title>
<issn pub-type="epub">2296-9144</issn>
<publisher>
<publisher-name>Frontiers Media S.A.</publisher-name>
</publisher>
</journal-meta>
<article-meta>
<article-id pub-id-type="publisher-id">828732</article-id>
<article-id pub-id-type="doi">10.3389/frobt.2022.828732</article-id>
<article-categories>
<subj-group subj-group-type="heading">
<subject>Robotics and AI</subject>
<subj-group>
<subject>Original Research</subject>
</subj-group>
</subj-group>
</article-categories>
<title-group>
<article-title>Where Does It Belong? Autonomous Object Mapping in Open-World Settings</article-title>
<alt-title alt-title-type="left-running-head">Langer et al.</alt-title>
<alt-title alt-title-type="right-running-head">Object Mapping in Open-World Settings</alt-title>
</title-group>
<contrib-group>
<contrib contrib-type="author" corresp="yes">
<name>
<surname>Langer</surname>
<given-names>Edith</given-names>
</name>
<xref ref-type="aff" rid="aff1">
<sup>1</sup>
</xref>
<xref ref-type="corresp" rid="c001">&#x2a;</xref>
<uri xlink:href="https://loop.frontiersin.org/people/1362156/overview"/>
</contrib>
<contrib contrib-type="author">
<name>
<surname>Patten</surname>
<given-names>Timothy</given-names>
</name>
<xref ref-type="aff" rid="aff1">
<sup>1</sup>
</xref>
<xref ref-type="aff" rid="aff2">
<sup>2</sup>
</xref>
<uri xlink:href="https://loop.frontiersin.org/people/794283/overview"/>
</contrib>
<contrib contrib-type="author">
<name>
<surname>Vincze</surname>
<given-names>Markus</given-names>
</name>
<xref ref-type="aff" rid="aff1">
<sup>1</sup>
</xref>
<uri xlink:href="https://loop.frontiersin.org/people/244651/overview"/>
</contrib>
</contrib-group>
<aff id="aff1">
<sup>1</sup>
<institution>Vision for Robotics Laboratory</institution>, <institution>Automation and Control Institute</institution>, <institution>TU Wien</institution>, <addr-line>Vienna</addr-line>, <country>Austria</country>
</aff>
<aff id="aff2">
<sup>2</sup>
<institution>Robotics Institute</institution>, <institution>University of Technology Sydney</institution>, <addr-line>Sydney</addr-line>, <addr-line>NSW</addr-line>, <country>Australia</country>
</aff>
<author-notes>
<fn fn-type="edited-by">
<p>
<bold>Edited by:</bold> <ext-link ext-link-type="uri" xlink:href="https://loop.frontiersin.org/people/263416/overview">Reza Sabzevari</ext-link>, Robert Bosch, Germany</p>
</fn>
<fn fn-type="edited-by">
<p>
<bold>Reviewed by:</bold> <ext-link ext-link-type="uri" xlink:href="https://loop.frontiersin.org/people/1133744/overview">Huimin Lu</ext-link>, National University of Defense Technology, China</p>
<p>
<ext-link ext-link-type="uri" xlink:href="https://loop.frontiersin.org/people/1017260/overview">Jen Jen Chung</ext-link>, ETH Z&#xfc;rich, Switzerland</p>
</fn>
<corresp id="c001">&#x2a;Correspondence: Edith Langer, <email>langer@acin.tuwien.ac.at</email>
</corresp>
<fn fn-type="other">
<p>This article was submitted to Robot and Machine Vision, a section of the journal Frontiers in Robotics and AI</p>
</fn>
</author-notes>
<pub-date pub-type="epub">
<day>19</day>
<month>04</month>
<year>2022</year>
</pub-date>
<pub-date pub-type="collection">
<year>2022</year>
</pub-date>
<volume>9</volume>
<elocation-id>828732</elocation-id>
<history>
<date date-type="received">
<day>03</day>
<month>12</month>
<year>2021</year>
</date>
<date date-type="accepted">
<day>21</day>
<month>02</month>
<year>2022</year>
</date>
</history>
<permissions>
<copyright-statement>Copyright &#xa9; 2022 Langer, Patten and Vincze.</copyright-statement>
<copyright-year>2022</copyright-year>
<copyright-holder>Langer, Patten and Vincze</copyright-holder>
<license xlink:href="http://creativecommons.org/licenses/by/4.0/">
<p>This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.</p>
</license>
</permissions>
<abstract>
<p>Detecting changes such as moved, removed, or new objects is the essence for numerous indoor applications in robotics such as tidying-up, patrolling, and fetch/carry tasks. The problem is particularly challenging in open-world scenarios where novel objects may appear at any time. The main idea of this paper is to detect objects from partial 3D reconstructions of interesting areas in the environment. In our pipeline we first identify planes, consider clusters on top as objects, and compute their point-pair-features. They are used to match potential objects and categorize them robustly into static, moved, removed, and novel objects even in the presence of partial object reconstructions and clutter. Our approach dissolves heaps of objects without specific object knowledge, but only with the knowledge acquired from change detection. The evaluation is performed on real-world data that includes challenges affecting the quality of the reconstruction as a result of noisy input data. We present the novel dataset <italic>ObChange</italic> for quantitative evaluation, and we compare our method against a baseline using learning-based object detection. The results show that, even with a targeted training set, our approach outperforms the baseline for most test cases. Lastly, we also demonstrate our method&#x2019;s effectiveness in real robot experiments.</p>
</abstract>
<kwd-group>
<kwd>object detection</kwd>
<kwd>object matching</kwd>
<kwd>object mapping</kwd>
<kwd>open-world detection</kwd>
<kwd>autonomous robot</kwd>
<kwd>point-pair-features</kwd>
</kwd-group>
</article-meta>
</front>
<body>
<sec id="s1">
<title>1 Introduction</title>
<p>Industrial tasks such as fetching objects, mobile manipulation, patrolling, or supporting humans in robot assembly require an understanding of objects relevant for the task in relation to the environment. When asking people what they wish robots could do at home, cleaning, tidying up, and picking up items from the floor are top priorities (<xref ref-type="bibr" rid="B9">Bugmann and Copleston, 2011</xref>; <xref ref-type="bibr" rid="B10">Cakmak and Takayama, 2013</xref>; <xref ref-type="bibr" rid="B3">Bajones et al., 2018</xref>). While there are solutions for vacuuming or mowing the lawn, maintaining order is largely unsolved. To work towards the open challenges, several competitions have been started, for example, the ICRA 2018 &#x201c;Tidy Up My Room&#x201d; Challenge<xref ref-type="fn" rid="fn1">
<sup>1</sup>
</xref> or the WRS RoboCup@Home (<xref ref-type="bibr" rid="B40">Okada et al., 2019</xref>) tidy-up task.</p>
<p>The tidy-up task is complex because a robot must operate in an unstructured and dynamic environment where it needs to localize known as well as unknown objects. To determine where objects belong to, the robot needs to have knowledge about the intended storage locations for these objects, e.g., in a knowledge base or another form of reference (<xref ref-type="bibr" rid="B52">Tenorth and Beetz, 2013</xref>). The focus of this paper is on the perception system for the tidy-up task and related applications, for which we refer to object detection and matching as the <italic>object mapping</italic> task.</p>
<p>Many approaches exist that detect objects in an environment by identifying changes between two visits based on camera data. A common method is to perform differencing between two single views using either color images (<xref ref-type="bibr" rid="B22">Furukawa et al., 2020</xref>; <xref ref-type="bibr" rid="B49">Sakurada et al., 2020</xref>) or RGB-D frames (<xref ref-type="bibr" rid="B1">Alimi et al., 2012</xref>; <xref ref-type="bibr" rid="B37">Mason and Marthi, 2012</xref>). Both modalities are affected by inaccuracies of view alignment. An alternative pathway is to reconstruct the environment, which has the benefit of providing a 3D object map to store object locations (<xref ref-type="bibr" rid="B2">Ambrus et al., 2014</xref>; <xref ref-type="bibr" rid="B17">Fehr et al., 2017</xref>). The major difficulty is to first create a consistent map and then align to this map at later visits given uncertainties in robot localization, view direction, or odometry. Options to handle this are to request users to give alignment cues (<xref ref-type="bibr" rid="B18">Finman et al., 2013</xref>) or assume sufficient robot accuracy (<xref ref-type="bibr" rid="B6">Bj&#xf6;rkman and Kragic, 2010</xref>; <xref ref-type="bibr" rid="B2">Ambrus et al., 2014</xref>; <xref ref-type="bibr" rid="B50">Song et al., 2015</xref>).</p>
<p>Object detection is the prerequisite to find matches. It reduces the object mapping task to the problem of comparing the locations of objects in the environment at two time instances. This definition is generic and independent of the specific robot task. For example, if the task is tidy up, a comparison is performed between the present situation and a <italic>reference map</italic>. If the task is patrolling, the observant robot will use all object detections to create a <italic>present object map</italic> for the new time instance. For fetching an object, the knowledge where this object was last seen, i.e., in the present object map, is used to retrieve it and, if not found, to start a search that may include information where the object has been found before.</p>
<p>Similarly to the object mapping task, the object rearrangement task as introduced by Batra et al. (<xref ref-type="bibr" rid="B4">Batra et al., 2020</xref>) also deals with the goal of transforming the current state of the environment into a target state assuming a closed world. This does not represent the real world that must consider objects that appear, and are therefore unknown, or disappear. Today most approaches assume a given and fixed set of objects, e.g., <xref ref-type="bibr" rid="B8">Bore et al. (2018</xref>) and <xref ref-type="bibr" rid="B54">Weihs et al. (2021</xref>). To develop more general methods, the task of open-world object detection is recently defined by <xref ref-type="bibr" rid="B28">Joseph et al. (2021</xref>). Objects from unknown classes need to be identified and then learned when label information becomes available. As a recent example, <xref ref-type="bibr" rid="B29">Kim et al. (2021</xref>) tackles the first aspect. They propose a method that generates class-agnostic object proposals in an open-world setting, but without classification. These approaches operate on small image patches and do not yet generalize to robotics applications in the 3D world.</p>
<p>Towards this goal, we present an approach that copes with all possible cases of static, moved, removed, and novel objects in different room settings. We partition a room into local horizontal surfaces, which is motivated by the fact that objects are typically found on furniture such as tables or shelves. Furthermore, it is infeasible for daily use to repeatedly and exhaustively scan an entire room. Tasks rather need to check if the object is at a specific location or surface. Finally, local surfaces can be easily extended to include other structures such as vertical surfaces to locate a broader variety of objects including pictures, switches, or door handles. This concept of local surfaces can be easily extended to multiple rooms. At the core of our approach is a comparison function to match detected objects to previously seen instances. To achieve this we represent surfaces where the objects reside as a 3D point cloud in the reference map. To autonomously create the surface partitioning, we exploit semantic segmentation. Finally, local surfaces enable high-quality reconstructions of every plane, which enhances the matching of detected objects using state-of-the-art methods such as Point Pair Features (PPF) (<xref ref-type="bibr" rid="B15">Drost et al., 2010</xref>). PPF is computationally cheap and runs on CPU only, which plays a considerable role for approaches running on mobile robots.</p>
<p>Our approach is evaluated on the <italic>ObChange</italic> (<italic>Ob</italic>ject <italic>Change</italic>) dataset, which extends prior work in <xref ref-type="bibr" rid="B31">Langer et al. (2020</xref>) with better local reconstructions. It encompasses multiple visits to five rooms with a total of 219 annotated objects. Taking all possible comparisons of visits per room into account, this leads to 961 objects for detection and matching. We report the results achieved on <italic>ObChange</italic> compared to a baseline using a learning-based detection approach as well as highlight possible failures and remaining challenges. Furthermore, we show the performance of our proposed approach using a fully autonomous system working in a real indoor environment.</p>
<p>To summarize, our contributions are:<list list-type="simple">
<list-item>
<p>&#x2022; A procedure that uses semantic segmentation and surface fitting to reliably detect objects and that robustly handles all cases encompassed in an open-world, that is, static, moved, removed, and novel objects.</p>
</list-item>
<list-item>
<p>&#x2022; An object mapping approach that does not rely on a trained classifier or pre-defined 3D models and, thus, works in an open-world setting by leveraging information extracted from a 3D representation.</p>
</list-item>
<list-item>
<p>&#x2022; Presenting the <italic>ObChange</italic> dataset and an evaluation of different detection methods to categorize objects into the four cases.</p>
</list-item>
<list-item>
<p>&#x2022; An evaluation on a fully autonomous robot that performs experiments in a real environment.</p>
</list-item>
</list>
</p>
<p>The remainder of the paper is organised as follows. <xref ref-type="sec" rid="s2">Section 2</xref> discusses related work for object mapping in open-world settings. <xref ref-type="sec" rid="s3">Section 3</xref> details our approach and <xref ref-type="sec" rid="s4">Section 4</xref> presents the experiments with <italic>ObChange</italic>, real experiments with a mobile robot, and a discussion of the results followed by the conclusion in <xref ref-type="sec" rid="s5">Section 5</xref>.</p>
</sec>
<sec id="s2">
<title>2 Related Work</title>
<p>Object mapping and determining object relations across different time instances requires their detection and association. Our work focuses on the perceptual part involved for this task. Object detection in RGB images, as well as 3D data, is an active field in computer vision and robotics. While most of the single frame, learning-based detectors are limited by their training dataset, some methods working on 3D data use only geometric properties (<xref ref-type="bibr" rid="B51">Tateno et al., 2015</xref>; <xref ref-type="bibr" rid="B21">Furrer et al., 2018</xref>) or additional semantics (<xref ref-type="bibr" rid="B24">Grinvald et al., 2019</xref>). Although these methods are useful in open-world settings, this section reviews work related to object detection via scene differencing in 3D, which is closer to our approach as it also uses a reference map. Additionally, this section explores image-based methods that tackle the open-world assumption and discusses available datasets useful for the object mapping scenario.</p>
<p>For object change detection, <italic>scene differencing</italic> based on 3D data is a common approach (<xref ref-type="bibr" rid="B18">Finman et al., 2013</xref>; <xref ref-type="bibr" rid="B2">Ambrus et al., 2014</xref>; <xref ref-type="bibr" rid="B17">Fehr et al., 2017</xref>; <xref ref-type="bibr" rid="B32">Langer et al., 2017</xref>) because the only prerequisite are two aligned reconstructions of the environment. The methods deal with alignment inaccuracies and falsely detected objects by requesting human help (<xref ref-type="bibr" rid="B18">Finman et al., 2013</xref>), filter detected objects with morphological operations (<xref ref-type="bibr" rid="B17">Fehr et al., 2017</xref>; <xref ref-type="bibr" rid="B32">Langer et al., 2017</xref>), or limit the trajectory of the robot while creating 3D maps (<xref ref-type="bibr" rid="B2">Ambrus et al., 2014</xref>). Approaches based on scene differencing are in general not able to detect replaced objects.</p>
<p>Object detection based on <italic>learning-based</italic> approaches such as YOLO or Mask R-CNN are very popular, yet are limited to a closed world and show weakness by assigning unknown objects mistakenly a learned class with high confidence. Learning-based open-world object classification is an emerging research field (<xref ref-type="bibr" rid="B45">Pidhorskyi et al., 2018</xref>; <xref ref-type="bibr" rid="B34">Liu et al., 2019</xref>; <xref ref-type="bibr" rid="B7">Boccato et al., 2020</xref>; <xref ref-type="bibr" rid="B44">Perera et al., 2020</xref>), while the extension to open-world object detection is only recently defined (<xref ref-type="bibr" rid="B28">Joseph et al., 2021</xref>). Only few works exist for focusing on object detection, which deal with estimating uncertainty and therefore being able to distinguish unknown objects (<xref ref-type="bibr" rid="B39">Miller et al., 2018</xref>, <xref ref-type="bibr" rid="B38">2019</xref>). Currently, they are not capable of gradually extending their knowledge when new classes emerge, which is essential to be useful in real-world applications. Incrementally extending the knowledge of a trained detector leads to the problem of catastrophic forgetting (<xref ref-type="bibr" rid="B20">French, 1999</xref>; <xref ref-type="bibr" rid="B30">Kirkpatrick et al., 2017</xref>), which is the challenge of maintaining robust performance on known classes as new classes are learned.</p>
<p>Only a few suitable <italic>datasets</italic> exist that not only support object change detection but also the categorization of different change types (novel, removed, etc.). Similarly, not many datasets provide information about object associations between two recordings. The datasets can be separated into synthetic frames-wise annotated datasets (<xref ref-type="bibr" rid="B42">Park et al., 2021</xref>; <xref ref-type="bibr" rid="B54">Weihs et al., 2021</xref>) and real-world datasets where the 3D map is annotated (<xref ref-type="bibr" rid="B53">Wald et al., 2019</xref>; <xref ref-type="bibr" rid="B31">Langer et al., 2020</xref>). Based on the task definitions from Batra et al.(<xref ref-type="bibr" rid="B4">Batra et al., 2020</xref>), Weihs et al. (<xref ref-type="bibr" rid="B54">Weihs et al., 2021</xref>) introduced a new dataset with object rearrangements in a virtual environment for studying how robots explore their environment. They set up two different versions of the rearrangement task. In the easier setting, the robot sees the current and the goal state at the same time, leading to perfectly aligned observations. In the advanced version, the robot must explore the environment in the target state first, and after the objects are moved, bring them back to their target location. A limitation of their task definition is that only objects present in the goal state can be out of place in the current state, thus assuming a closed world. ChangeSim (<xref ref-type="bibr" rid="B42">Park et al., 2021</xref>) is a synthetic dataset of warehouse scenes with different illumination and dusty air levels acquired with a drone. This is curated to support online detection approaches that work directly on frames. Therefore, first a correct pairing of frames from two different time instances must be found before change detection can be computed. For change detection, they define the following categories: new, removed, replace, rotated, or static. The most important category for object mapping, moved, is not defined. This makes it impossible to differentiate between a removed or a moved object. Wald et al. (<xref ref-type="bibr" rid="B53">Wald et al., 2019</xref>). introduce a real-world dataset acquired with a handheld device. Similar to Weihs et al. (<xref ref-type="bibr" rid="B54">Weihs et al., 2021</xref>), the dataset is designed for object instance re-localization. Unfortunately no novel or removed objects are considered. Another drawback is that mainly large items such as furniture are labeled and not objects that can be manipulated by a service robot. A real-word robotic dataset for change detection is acquired by Langer et al. (<xref ref-type="bibr" rid="B31">Langer et al., 2020</xref>). Objects that are labeled have been selected from the YCB object set (<xref ref-type="bibr" rid="B11">Calli et al., 2017</xref>). This dataset is only used for change detection and we extend it to provide the necessary reconstructions and ground truth for matching cases for open-world settings.</p>
<p>Only a few works exist that can be applied to the object mapping task. Bore et al. (<xref ref-type="bibr" rid="B8">Bore et al., 2018</xref>) detect objects based on the change detection approach of Ambrus et al. (<xref ref-type="bibr" rid="B2">Ambrus et al., 2014</xref>) and additionally track the movement of detected objects over time by defining a two-stage movement model, which is limited to a closed world. Also the baselines introduced by Weihs et al. (<xref ref-type="bibr" rid="B54">Weihs et al., 2021</xref>) work with a closed-world assumption, where at both time instances the same objects occur. The best performing method evaluated on their proposed dataset achieves a success rate of 8.2% when exploring the goal and current state subsequently and 17.9% when seen simultaneously. This is in line with the observation from <xref ref-type="bibr" rid="B42">Park et al. (2021</xref>), that pairing frames is a non-trivial task. It has to be noted that they propose an end-to-end learned approach for a robotic system without any knowledge of the environment. Finman et al. (<xref ref-type="bibr" rid="B18">Finman et al., 2013</xref>) discover objects through differencing of reconstructions. The focus of their work is on learning segmentation methods to re-discover objects in future visits, which are then used to segment the whole environment at the next visit. The result is used to find segments that overlap with the object. The biggest limitation of their work is the need of well aligned room reconstructions, where they rely on human input to define the overlapping parts between two reconstructions. This overlapping part is then aligned using ICP, which tends to fail if big parts in the overlapping area have changed. The approach closest to our work is by Song et al. (<xref ref-type="bibr" rid="B50">Song et al., 2015</xref>). By assuming that the robot stays in the same environment, their goal is to determine a global instance-based labeling and to further recognize individual objects. The method requires a high-quality reconstruction that is generated by an RGB-D camera array. Based on the full semantic labeling generated by a crowd-sourcing marketplace, objects are either classified as movable or non-movable. Non-movable objects are considered as background and used to align frames from different timestamps. For the remaining environment parts, a SIFT descriptor is computed and used for matching. Their approach heavily relies on the quality of manually labeled instance segmentation for the whole environment. In contrast, we propose a fully autonomous system using a surface concept to more efficiently partition an environment for different object mapping applications.</p>
</sec>
<sec id="s3">
<title>3 Object Mapping Using Local Surfaces for Matching</title>
<p>This section formally defines the problem of object mapping in arbitrary environments in <xref ref-type="sec" rid="s3-1">Section 3.1</xref>. An overview of the perception components for this task is given in <xref ref-type="sec" rid="s3-2">Section 3.2</xref>. Finally, the details for reconstruction, object detection, and object matching are described in <xref ref-type="sec" rid="s3-5">Sections 3.3&#x2013;3.5</xref>.</p>
<sec id="s3-1">
<title>3.1 Problem Definition</title>
<p>The goal of this work is to detect objects in an environment and to further assign each object a category depicting its relationship to previous detections. The focus is on objects, which are detachable from the surface they are placed on and can be manipulated by a service robot. To remain task-independent, we compare objects present in an environment at time <italic>t</italic>
<sub>0</sub> and objects detected at a later time <italic>t</italic>
<sub>1</sub>. We refer to objects detected at <italic>t</italic>
<sub>0</sub> as models and denote the set of detections as <inline-formula id="inf1">
<mml:math id="m1">
<mml:mi mathvariant="script">M</mml:mi>
</mml:math>
</inline-formula>. Objects detected at <italic>t</italic>
<sub>1</sub> are referred to as candidates and the set is denoted <inline-formula id="inf2">
<mml:math id="m2">
<mml:mi mathvariant="script">C</mml:mi>
</mml:math>
</inline-formula>. Detected objects are matched across the time instances, then categorized into static, moved, removed, and novel; see also <xref ref-type="table" rid="T1">Table 1</xref>. A static object is a candidate <inline-formula id="inf3">
<mml:math id="m3">
<mml:mi>c</mml:mi>
<mml:mo>&#x2208;</mml:mo>
<mml:mi mathvariant="script">C</mml:mi>
</mml:math>
</inline-formula> that has a matching model <inline-formula id="inf4">
<mml:math id="m4">
<mml:mi>m</mml:mi>
<mml:mo>&#x2208;</mml:mo>
<mml:mi mathvariant="script">M</mml:mi>
</mml:math>
</inline-formula> and where the distance between <italic>c</italic> and <italic>m</italic> is less than a threshold <italic>d</italic>. This threshold is selected depending on the uncertainty in robot localisation, the reconstruction, object detection, object placement, etc. The value may be different depending on specific applications. We use <italic>d</italic> &#x3d; 20&#xa0;cm throughout the paper. A moved object is a candidate that has a matching model but where the distance between the objects is greater than <italic>d</italic>. An object is considered removed if it exists in <inline-formula id="inf5">
<mml:math id="m5">
<mml:mi mathvariant="script">M</mml:mi>
</mml:math>
</inline-formula> but has no matching candidate in <inline-formula id="inf6">
<mml:math id="m6">
<mml:mi mathvariant="script">C</mml:mi>
</mml:math>
</inline-formula>. Novel objects are any candidates in <inline-formula id="inf7">
<mml:math id="m7">
<mml:mi mathvariant="script">C</mml:mi>
</mml:math>
</inline-formula> that have no matching model in <inline-formula id="inf8">
<mml:math id="m8">
<mml:mi mathvariant="script">M</mml:mi>
</mml:math>
</inline-formula>. The set of models <inline-formula id="inf9">
<mml:math id="m9">
<mml:mi mathvariant="script">M</mml:mi>
</mml:math>
</inline-formula> is the union of all static, moved, and removed objects while the intersection of these must be empty. Likewise, the set of candidates <inline-formula id="inf10">
<mml:math id="m10">
<mml:mi mathvariant="script">C</mml:mi>
</mml:math>
</inline-formula> is the union of static, moved, and novel objects while their intersection must be empty.</p>
<table-wrap id="T1" position="float">
<label>TABLE 1</label>
<caption>
<p>Definition of the different categories an object gets assigned to when comparing an environment at two different timestamps.</p>
</caption>
<table>
<thead valign="top">
<tr>
<th align="left">Object category</th>
<th align="center">Description</th>
</tr>
</thead>
<tbody valign="top">
<tr>
<td align="left">Static</td>
<td align="left">Object did not move or only less than a distance <italic>d</italic> at time <italic>t</italic>
<sub>0</sub> compared to <italic>t</italic>
<sub>1</sub>
</td>
</tr>
<tr>
<td align="left">Moved</td>
<td align="left">Object is detected at time <italic>t</italic>
<sub>0</sub> and at time <italic>t</italic>
<sub>1</sub>, but at different locations</td>
</tr>
<tr>
<td align="left">Removed</td>
<td align="left">Object is detected at time <italic>t</italic>
<sub>0</sub> but not at time <italic>t</italic>
<sub>1</sub>
</td>
</tr>
<tr>
<td align="left">Novel</td>
<td align="left">Object is detected at time <italic>t</italic>
<sub>1</sub> but not at time <italic>t</italic>
<sub>0</sub>
</td>
</tr>
</tbody>
</table>
</table-wrap>
</sec>
<sec id="s3-2">
<title>3.2 System Overview</title>
<p>An overview of the proposed perception system for autonomous object change detection and mapping is given in <xref ref-type="fig" rid="F1">Figure 1</xref>. The approach is composed of two phases: setup of the reference map (blue), which needs to be preformed only one time, and every run through the environment to visit all or a subset of surfaces (green and orange). If the larger structure of the room or the main surfaces (such as furniture) are significantly moved, phase one must be repeated to generate a new reference map.</p>
<fig id="F1" position="float">
<label>FIGURE 1</label>
<caption>
<p>System overview of our approach. The setup of the reference is performed only once.</p>
</caption>
<graphic xlink:href="frobt-09-828732-g001.tif"/>
</fig>
<p>This work leverages the concept of surfaces to focus object comparisons, which ultimately leads to improved object detection. Technically, this is achieved by first combining the reconstruction of the room and semantic segmentation to create a list of relevant surfaces; see <xref ref-type="sec" rid="s3-3">Section 3.3</xref>. Then a more detailed scan of every surface is performed to improve the reconstruction. This high-quality reconstruction of each surface is used to detect objects (see <xref ref-type="sec" rid="s3-4">Section 3.4</xref>), which are stored in a database as the reference object map for future change detection requests. When the robot revisits rooms and surfaces for its specific task, for example, tidying up a kitchen counter, a new detailed reconstruction is generated. Objects are extracted and then matched to those in the reference. The matching process is performed in three different ways to handle different cases as outlined in <xref ref-type="sec" rid="s3-5">Section 3.5</xref>.</p>
<p>In this work we use only change as a cue for segmentation, which is fully applicable to open-world settings. If change occurs multiple times, then a heap of objects may become disentangled; otherwise, the heap will be considered as a single object. In the following sections, we refer to each detection as an object, both for single standing items or heaps of objects.</p>
</sec>
<sec id="s3-3">
<title>3.3 Reconstruction of the Indoor Environment and Plane Extraction</title>
<p>The first step for detecting objects is to identify the regions where objects are commonly located, in other words, the surfaces. Similar to our previous work (<xref ref-type="bibr" rid="B31">Langer et al., 2020</xref>), the search space for objects is reduced according to the assumption that objects are most often placed on horizontal planes in home environments (<xref ref-type="bibr" rid="B6">Bj&#xf6;rkman and Kragic, 2010</xref>; <xref ref-type="bibr" rid="B36">Marton et al., 2010</xref>). To extract horizontal planes, the environment is reconstructed using <italic>Voxblox</italic> (<xref ref-type="bibr" rid="B41">Oleynikova et al., 2017</xref>). This method runs on CPU only and is tightly coupled with ROS (<xref ref-type="bibr" rid="B47">Quigley et al., 2009</xref>), both great qualities when working with a robot. The coarse reconstruction of the environment, which is a result of the voxelized representation, is used to geometrically search for planes. To do so, the reconstruction is transformed into a point cloud by extracting the centroids of all voxels. In addition, <italic>SparseConvnet</italic> (<xref ref-type="bibr" rid="B23">Graham et al., 2018</xref>) is applied to the reconstruction to retrieve a semantic label of each point and consequently to exclude non-relevant regions for the plane search; all points are removed that are not assigned any of the following classes: cabinet, bed, chair, sofa, table, bookshelf, counter, desk, shelves, nightstand, other props, other structure, and other furniture. Since we focus on horizontal planes, only points with a normal facing upward are retained. Finally, for each semantic class the remaining points are downsampled and input to RANSAC (<xref ref-type="bibr" rid="B19">Fischler and Bolles, 1981</xref>) to fit to a plane. Each iteration generates one plane and these points are removed from the input to enable further plane fitting. The loop ends when the extracted plane consists of less than a certain number of points.</p>
<p>For each plane, descriptive information such as the plane coefficients, convex hull points, and centroid are stored in a database. Additionally, waypoints for the robot to navigate to when inspecting the plane are computed. Waypoints are equally distributed positions around the plane at a fixed distance to the edge of the convex hull. The pose of each waypoint is described by its position and an orientation that faces the center of the plane. All the surface information is used for subsequent visits.</p>
</sec>
<sec id="s3-4">
<title>3.4 Reconstruction of the Surface and Object Detection</title>
<p>Once the global reconstruction of the environment is created, a higher-quality local reconstruction is generated for each surface to enable more precise object detection. In this work we use <italic>ElasticFusion</italic> (<xref ref-type="bibr" rid="B55">Whelan et al., 2015</xref>) for the local surface reconstruction. It uses both photometric and geometric pose estimation, which is configured using a relative ICP/RGB tracking weight parameter. While <italic>ElasticFusion</italic> is more precise than <italic>Voxblox</italic>, it still suffers specific failure cases that need to be addressed for robust operation on a mobile robot. Firstly, viewpoints focusing on large planar and low-textured surfaces have too few features to track the camera pose, which results in misalignment (see <xref ref-type="fig" rid="F2">Figure 2</xref> [top]). Secondly, changing lighting conditions resulting in over- and underexposured images is problematic for registration as can be seen in <xref ref-type="fig" rid="F2">Figure 2</xref> (middle). Another source of error are geometric symmetries as well as low depth disparity. <xref ref-type="fig" rid="F2">Figure 2</xref> (bottom) shows an example reconstruction using RGB and ICP registration but suffers from duplicated and misaligned objects or smeared objects.</p>
<fig id="F2" position="float">
<label>FIGURE 2</label>
<caption>
<p>Each row shows a surface of one of the environments in our dataset. The first column is the result of ElasticFusion using RGB and ICP information to estimate the camera pose, the second row shows results using only ICP, and the last column shows the result of fusing the robot poses with the estimated camera poses from ElasticFusion (only ICP).</p>
</caption>
<graphic xlink:href="frobt-09-828732-g002.tif"/>
</fig>
<p>To countermeasure those real-world problems, we propose a computationally simple solution. Our idea is to assist the tracking method whenever the estimated trajectory begins to significantly diverge from the reported robot odometry data. Clearly, odometry data has inaccuracies; thus, using it directly is not sensible. Whenever the registration fails completely (e.g., <xref ref-type="fig" rid="F2">Figure 2</xref> [top]) the estimated pose and the odometry pose differ significantly. Resetting the estimated pose to the odometry data is not feasible either because this can lead to smeared reconstructions. Our approach is to blend the poses from the camera tracking and odometry data to repair drift and misalignment error in the running reconstruction. For each frame, before running <italic>ElasticFusion</italic>&#x2019;s frame processing, we recompute the pose given the last estimated pose and the robot pose from odometry data. To this end we first compute the difference between the last estimated pose <italic>E</italic> and the current robot pose <italic>P</italic>.</p>
<p>More specifically, the poses are represented by transformation matrices:<disp-formula id="e1">
<mml:math id="m11">
<mml:mi>E</mml:mi>
<mml:mo>&#x3d;</mml:mo>
<mml:mfenced open="[" close="]">
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>R</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>E</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo stretchy="false">&#x7c;</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold">t</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>E</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:mfenced>
<mml:mo>,</mml:mo>
<mml:mi>P</mml:mi>
<mml:mo>&#x3d;</mml:mo>
<mml:mfenced open="[" close="]">
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>R</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>P</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo stretchy="false">&#x7c;</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold">t</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>P</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:mfenced>
<mml:mo>&#x2208;</mml:mo>
<mml:mi mathvariant="double-struck">S</mml:mi>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="double-struck">E</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
</mml:msub>
</mml:math>
<label>(1)</label>
</disp-formula>with rotation components <inline-formula id="inf11">
<mml:math id="m12">
<mml:msub>
<mml:mrow>
<mml:mi>R</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>E</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>,</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi>R</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>P</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x2208;</mml:mo>
<mml:mi mathvariant="double-struck">S</mml:mi>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="double-struck">O</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
</mml:msub>
</mml:math>
</inline-formula> and translation components <inline-formula id="inf12">
<mml:math id="m13">
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold">t</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>E</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>,</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold">t</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>P</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x2208;</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="double-struck">R</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
</mml:msup>
</mml:math>
</inline-formula>. Next the matrix is computed:<disp-formula id="e2">
<mml:math id="m14">
<mml:mi>D</mml:mi>
<mml:mo>&#x3d;</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mi>P</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mo>&#x2212;</mml:mo>
<mml:mn>1</mml:mn>
</mml:mrow>
</mml:msup>
<mml:mi>E</mml:mi>
<mml:mo>&#x3d;</mml:mo>
<mml:mfenced open="[" close="]">
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>R</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>D</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo stretchy="false">&#x7c;</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold">t</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>D</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:mfenced>
<mml:mo>.</mml:mo>
</mml:math>
<label>(2)</label>
</disp-formula>
</p>
<p>Given this, we compute the angular difference:<disp-formula id="e3">
<mml:math id="m15">
<mml:mi>&#x3d5;</mml:mi>
<mml:mo>&#x3d;</mml:mo>
<mml:mo stretchy="false">&#x7c;</mml:mo>
<mml:mi>arccos</mml:mi>
<mml:mfenced open="(" close=")">
<mml:mrow>
<mml:mfrac>
<mml:mrow>
<mml:mtext>trace</mml:mtext>
<mml:mfenced open="(" close=")">
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>R</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>D</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:mfenced>
<mml:mo>&#x2212;</mml:mo>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:mfrac>
</mml:mrow>
</mml:mfenced>
<mml:mo stretchy="false">&#x7c;</mml:mo>
<mml:mo>,</mml:mo>
</mml:math>
<label>(3)</label>
</disp-formula>to derive the mixing term:<disp-formula id="e4">
<mml:math id="m16">
<mml:mi>&#x3bb;</mml:mi>
<mml:mo>&#x3d;</mml:mo>
<mml:mtext>max</mml:mtext>
<mml:mfenced open="(" close=")">
<mml:mrow>
<mml:mtext>min</mml:mtext>
<mml:mfenced open="(" close=")">
<mml:mrow>
<mml:mn>1.0</mml:mn>
<mml:mo>,</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mfenced open="(" close=")">
<mml:mrow>
<mml:mfrac>
<mml:mrow>
<mml:mi>&#x3d5;</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>z</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>r</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:mfrac>
</mml:mrow>
</mml:mfenced>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msup>
</mml:mrow>
</mml:mfenced>
<mml:mo>,</mml:mo>
<mml:mtext>min</mml:mtext>
<mml:mfenced open="(" close=")">
<mml:mrow>
<mml:mn>1.0</mml:mn>
<mml:mo>,</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mfenced open="(" close=")">
<mml:mrow>
<mml:mfrac>
<mml:mrow>
<mml:mo stretchy="false">&#x7c;</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold">t</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>D</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo stretchy="false">&#x7c;</mml:mo>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>z</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>t</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:mfrac>
</mml:mrow>
</mml:mfenced>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msup>
</mml:mrow>
</mml:mfenced>
</mml:mrow>
</mml:mfenced>
<mml:mo>,</mml:mo>
</mml:math>
<label>(4)</label>
</disp-formula>where <italic>z</italic>
<sub>
<italic>r</italic>
</sub> and <italic>z</italic>
<sub>
<italic>t</italic>
</sub> are constant scaling factors. This allows the modified <italic>E</italic>&#x2032; to be used as replacement for <italic>E</italic> through a linear combination of the poses:<disp-formula id="e5">
<mml:math id="m17">
<mml:msup>
<mml:mrow>
<mml:mi>E</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mo>&#x2032;</mml:mo>
</mml:mrow>
</mml:msup>
<mml:mo>&#x3d;</mml:mo>
<mml:mi>&#x3bb;</mml:mi>
<mml:mi>P</mml:mi>
<mml:mo>&#x2b;</mml:mo>
<mml:mfenced open="(" close=")">
<mml:mrow>
<mml:mn>1</mml:mn>
<mml:mo>&#x2212;</mml:mo>
<mml:mi>&#x3bb;</mml:mi>
</mml:mrow>
</mml:mfenced>
<mml:mi>E</mml:mi>
<mml:mo>.</mml:mo>
</mml:math>
<label>(5)</label>
</disp-formula>
</p>
<p>The more the two poses disagree (i.e., the estimated pose diverges from the pose measured by the robot), the more the odometry pose is used in hope for future agreement. Clearly, if odometry is less accurate, the results will degrade. However, it still prevents ElasticFusion from completely failing in difficult scenes. In our experiments we found good results with <italic>z</italic>
<sub>
<italic>r</italic>
</sub> &#x3d; 0.2 and <italic>z</italic>
<sub>
<italic>t</italic>
</sub> &#x3d; 0.2, which is used for all experiments.</p>
<p>Integrating the tracked camera and robot poses generates high-quality reconstructions in real-world scenarios from the robot trajectory. This adaption is used to create a reconstruction for each extracted plane that is stored in the database before each is processed separately. For each plane, its parameters and waypoints derived in the setup stage are queried from the database. Based on this information, the robot navigates around the plane while the camera is directed to the center of the plane. Before <italic>ElasticFusion</italic> transforms the camera stream into a reconstruction, the depth images are pre-processed by cropping them such that only the plane is retained. Cropping the depth input prevents <italic>ElasticFusion</italic> from trying to align the background (e.g., walls) at the expense of reconstruction accuracy of objects placed on the plane. Benefiting from the local surface concept, trajectories to create plane reconstructions are comparably short and result in precise reconstructions. From these generated plane reconstructions, objects are extracted by removing the points of the plane according to the known plane parameters. The parameters <italic>&#x3b1;</italic>
<sub>max</sub>, to allow deviation from a perfect horizontal plane, and <italic>d</italic>
<sub>
<italic>plane</italic>
</sub>, which defines the maximum distance of inlier points to the detected plane, have to be chosen to take into account small inaccuracies. The remaining points within the convex hull are clustered using Euclidean distance. All points up to 0.3&#xa0;m above the plane are considered; however, this value can be chosen depending on the application. We use a minimally shrinked convex hull to reduce the number of false positive detections such as walls or arm rests. At this stage, we do not try to separate objects in clutter and treat each cluster as an object.</p>
</sec>
<sec id="s3-5">
<title>3.5 Object Matching and Categorization</title>
<p>To support high-level robot tasks, objects are assigned one of the four categories, which is an indicator for what action should be performed with the object. For example, a static object should be left alone while a moved object should be returned to its original location. Performing the categorization requires the determination of which objects are present in different visits and additionally finding those that are matched. The following subsections explain the different stages of matching, which are also depicted in <xref ref-type="fig" rid="F1">Figure 1</xref>.</p>
<sec id="s3-5-1">
<title>3.5.1 Local Matching</title>
<p>For each detected object, a check is performed to determine if it is still approximately located at the same position. If there exists a model and a candidate within a distance less than <italic>d</italic>, they are considered a potential match. A confidence score for the match is computed by aligning their point clouds with ICP, which is suitable in this case as their close proximity provides a good initial registration. Two scores are then computed if ICP converges: one for the model <italic>S</italic>
<sub>
<italic>m</italic>
</sub> and one for the candidate <italic>S</italic>
<sub>
<italic>c</italic>
</sub> due to the object potentially not being symmetric. The model and the candidate are a match if min (<italic>S</italic>
<sub>
<italic>m</italic>
</sub>, <italic>S</italic>
<sub>
<italic>c</italic>
</sub>) &#x3e; <italic>&#x3c4;</italic>
<sub>
<italic>icp</italic>
</sub> for a given threshold <italic>&#x3c4;</italic>
<sub>
<italic>icp</italic>
</sub>.</p>
<p>Formally, we consider the model point cloud <inline-formula id="inf13">
<mml:math id="m18">
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="script">P</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>m</mml:mi>
</mml:mrow>
</mml:msub>
</mml:math>
</inline-formula> and candidate point cloud <inline-formula id="inf14">
<mml:math id="m19">
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="script">P</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>c</mml:mi>
</mml:mrow>
</mml:msub>
</mml:math>
</inline-formula>. For each point <inline-formula id="inf15">
<mml:math id="m20">
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold">p</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>m</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x2208;</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="script">P</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>m</mml:mi>
</mml:mrow>
</mml:msub>
</mml:math>
</inline-formula>, a set of points <inline-formula id="inf16">
<mml:math id="m21">
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="script">Q</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>m</mml:mi>
<mml:mo>,</mml:mo>
<mml:mi>c</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x2282;</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="script">P</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>c</mml:mi>
</mml:mrow>
</mml:msub>
</mml:math>
</inline-formula> is determined as the collection of all corresponding points in <inline-formula id="inf17">
<mml:math id="m22">
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="script">P</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>c</mml:mi>
</mml:mrow>
</mml:msub>
</mml:math>
</inline-formula> that have a distance to <bold>p</bold>
<sub>
<italic>m</italic>
</sub> less than the inlier threshold <italic>&#x3c4;</italic> after alignment. A score is then computed for <bold>p</bold>
<sub>
<italic>m</italic>
</sub> and each <inline-formula id="inf18">
<mml:math id="m23">
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold">p</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>c</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x2208;</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="script">Q</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>m</mml:mi>
<mml:mo>,</mml:mo>
<mml:mi>c</mml:mi>
</mml:mrow>
</mml:msub>
</mml:math>
</inline-formula>, which is composed of the geometric and color similarity. Given the point normals, <bold>n</bold>
<sub>
<italic>m</italic>
</sub> and <bold>n</bold>
<sub>
<italic>c</italic>
</sub>, the geometric score is given by:<disp-formula id="e6">
<mml:math id="m24">
<mml:msub>
<mml:mrow>
<mml:mi>s</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>g</mml:mi>
<mml:mi>e</mml:mi>
<mml:mi>o</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x3d;</mml:mo>
<mml:mfenced open="{" close="">
<mml:mrow>
<mml:mtable class="cases">
<mml:mtr>
<mml:mtd columnalign="left">
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold">n</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>m</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x22c5;</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold">n</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>c</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mspace width="1em"/>
</mml:mtd>
<mml:mtd columnalign="left">
<mml:mtext>if&#x2009;</mml:mtext>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold">n</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>m</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x22c5;</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold">n</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>c</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x2265;</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi>&#x3c4;</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>g</mml:mi>
<mml:mi>e</mml:mi>
<mml:mi>o</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>,</mml:mo>
</mml:mtd>
</mml:mtr>
<mml:mtr>
<mml:mtd columnalign="left">
<mml:mn>0</mml:mn>
<mml:mspace width="1em"/>
</mml:mtd>
<mml:mtd columnalign="left">
<mml:mtext>otherwise</mml:mtext>
<mml:mo>.</mml:mo>
</mml:mtd>
</mml:mtr>
</mml:mtable>
</mml:mrow>
</mml:mfenced>
</mml:math>
<label>(6)</label>
</disp-formula>
</p>
<p>The color score is computed as:<disp-formula id="e7">
<mml:math id="m25">
<mml:msub>
<mml:mrow>
<mml:mi>s</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>c</mml:mi>
<mml:mi>o</mml:mi>
<mml:mi>l</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x3d;</mml:mo>
<mml:mfenced open="{" close="">
<mml:mrow>
<mml:mtable class="cases">
<mml:mtr>
<mml:mtd columnalign="left">
<mml:mn>0</mml:mn>
<mml:mspace width="1em"/>
</mml:mtd>
<mml:mtd columnalign="left">
<mml:mtext>if&#x2009;</mml:mtext>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold-italic">&#x3ba;</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>m</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x2296;</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold-italic">&#x3ba;</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>c</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x2265;</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi>&#x3c4;</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>c</mml:mi>
<mml:mi>o</mml:mi>
<mml:mi>l</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mtd>
</mml:mtr>
<mml:mtr>
<mml:mtd columnalign="left">
<mml:mn>1</mml:mn>
<mml:mo>&#x2212;</mml:mo>
<mml:mfrac>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold-italic">&#x3ba;</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>m</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x2296;</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold-italic">&#x3ba;</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>c</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>&#x3c4;</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>c</mml:mi>
<mml:mi>o</mml:mi>
<mml:mi>l</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:mfrac>
<mml:mspace width="1em"/>
</mml:mtd>
<mml:mtd columnalign="left">
<mml:mtext>otherwise</mml:mtext>
<mml:mo>,</mml:mo>
</mml:mtd>
</mml:mtr>
</mml:mtable>
</mml:mrow>
</mml:mfenced>
</mml:math>
<label>(7)</label>
</disp-formula>where <bold>
<italic>&#x3ba;</italic>
</bold>
<sub>
<italic>m</italic>
</sub> and <bold>
<italic>&#x3ba;</italic>
</bold>
<sub>
<italic>c</italic>
</sub> are the color values of the points <bold>p</bold>
<sub>
<italic>m</italic>
</sub> and <bold>p</bold>
<sub>
<italic>c</italic>
</sub> in LAB-space and &#x2296; is the CIEDE2000 color difference (<xref ref-type="bibr" rid="B35">Luo et al., 2001</xref>). <italic>&#x3c4;</italic>
<sub>
<italic>geo</italic>
</sub> and <italic>&#x3c4;</italic>
<sub>
<italic>col</italic>
</sub> are thresholds. The similarity score between <bold>p</bold>
<sub>
<italic>m</italic>
</sub> and <bold>p</bold>
<sub>
<italic>c</italic>
</sub> in the correspondence set is the weighted combination of the geometric and color scores:<disp-formula id="e8">
<mml:math id="m26">
<mml:msub>
<mml:mrow>
<mml:mi>s</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>m</mml:mi>
<mml:mo>,</mml:mo>
<mml:mi>c</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x3d;</mml:mo>
<mml:mfenced open="{" close="">
<mml:mrow>
<mml:mtable class="cases">
<mml:mtr>
<mml:mtd columnalign="left">
<mml:mn>0</mml:mn>
<mml:mspace width="1em"/>
</mml:mtd>
<mml:mtd columnalign="left">
<mml:mtext>if&#x2009;</mml:mtext>
<mml:msub>
<mml:mrow>
<mml:mi>s</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>g</mml:mi>
<mml:mi>e</mml:mi>
<mml:mi>o</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x3d;</mml:mo>
<mml:mn>0</mml:mn>
<mml:mo>&#x2228;</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi>s</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>c</mml:mi>
<mml:mi>o</mml:mi>
<mml:mi>l</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x3d;</mml:mo>
<mml:mn>0</mml:mn>
</mml:mtd>
</mml:mtr>
<mml:mtr>
<mml:mtd columnalign="left">
<mml:mi>w</mml:mi>
<mml:msub>
<mml:mrow>
<mml:mi>s</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>g</mml:mi>
<mml:mi>e</mml:mi>
<mml:mi>o</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x2b;</mml:mo>
<mml:mfenced open="(" close=")">
<mml:mrow>
<mml:mn>1</mml:mn>
<mml:mo>&#x2212;</mml:mo>
<mml:mi>w</mml:mi>
</mml:mrow>
</mml:mfenced>
<mml:msub>
<mml:mrow>
<mml:mi>s</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>c</mml:mi>
<mml:mi>o</mml:mi>
<mml:mi>l</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mspace width="1em"/>
</mml:mtd>
<mml:mtd columnalign="left">
<mml:mtext>otherwise</mml:mtext>
<mml:mo>,</mml:mo>
</mml:mtd>
</mml:mtr>
</mml:mtable>
</mml:mrow>
</mml:mfenced>
</mml:math>
<label>(8)</label>
</disp-formula>where <inline-formula id="inf1114">
<mml:math id="m1119">
<mml:mi>w</mml:mi>
</mml:math>
</inline-formula> balances the contribution of the geometric and color similarity.</p>
<p>The overall fitness score for the model is defined as:<disp-formula id="e9">
<mml:math id="m27">
<mml:msub>
<mml:mrow>
<mml:mi>S</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>m</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x3d;</mml:mo>
<mml:mfrac>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">&#x7c;</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="script">P</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>m</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo stretchy="false">&#x7c;</mml:mo>
</mml:mrow>
</mml:mfrac>
<mml:munder>
<mml:mrow>
<mml:mo>&#x2211;</mml:mo>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold">p</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>m</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x2208;</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="script">P</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>m</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:munder>
<mml:msubsup>
<mml:mrow>
<mml:mi>s</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>m</mml:mi>
<mml:mo>,</mml:mo>
<mml:mi>c</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mo>&#x2a;</mml:mo>
</mml:mrow>
</mml:msubsup>
<mml:mo>,</mml:mo>
</mml:math>
<label>(9)</label>
</disp-formula>where <inline-formula id="inf19">
<mml:math id="m28">
<mml:msubsup>
<mml:mrow>
<mml:mi>s</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>m</mml:mi>
<mml:mo>,</mml:mo>
<mml:mi>c</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mo>&#x2a;</mml:mo>
</mml:mrow>
</mml:msubsup>
</mml:math>
</inline-formula> is the best similarity score for <bold>p</bold>
<sub>
<italic>m</italic>
</sub> given all the scores computed for the points in its correspondence set <inline-formula id="inf20">
<mml:math id="m29">
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="script">Q</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>m</mml:mi>
<mml:mo>,</mml:mo>
<mml:mi>c</mml:mi>
</mml:mrow>
</mml:msub>
</mml:math>
</inline-formula>. Likewise, the overall fitness score for the candidate is computed:<disp-formula id="e10">
<mml:math id="m30">
<mml:msub>
<mml:mrow>
<mml:mi>S</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>c</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x3d;</mml:mo>
<mml:mfrac>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">&#x7c;</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="script">P</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>c</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo stretchy="false">&#x7c;</mml:mo>
</mml:mrow>
</mml:mfrac>
<mml:munder>
<mml:mrow>
<mml:mo>&#x2211;</mml:mo>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold">p</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>c</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x2208;</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="script">P</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>c</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:munder>
<mml:msubsup>
<mml:mrow>
<mml:mi>s</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>c</mml:mi>
<mml:mo>,</mml:mo>
<mml:mi>m</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mo>&#x2a;</mml:mo>
</mml:mrow>
</mml:msubsup>
<mml:mo>.</mml:mo>
</mml:math>
<label>(10)</label>
</disp-formula>
</p>
<p>A match is recorded if at least one of the scores <italic>S</italic>
<sub>
<italic>m</italic>
</sub> or <italic>S</italic>
<sub>
<italic>c</italic>
</sub> is greater than the matching threshold <italic>&#x3c4;</italic>
<sub>
<italic>icp</italic>
</sub>. If only a subset of the points match, it is necessary to examine if either or both of the model and candidate need to be split into independent objects. A split based on the overlapping points is insufficient due to possible over or under segmentation of the objects. Therefore, to generate more precise object boundaries we perform region growing based on color and normal similarity where only points that contributed to the score are seed points. For all seed points, points within a certain radius are added if color and normal are similar enough compared to the seed point. The maximum allowed color difference <italic>rg</italic>
<sub>
<italic>col</italic>
</sub> is computed with the CIEDE2000 formula using the color values in the LAB-space of both points. The dot product from the normal vectors is used to check if the angle difference is not greater than <italic>rg</italic>
<sub>
<italic>geo</italic>
</sub>. Points that fulfil both criteria are added and act as new seed points. If no more points can be added, the region growing stops. All points in the result that belong to the model or candidate are categorized as static. The remaining points form a new object in the respective set.</p>
</sec>
<sec id="s3-5-2">
<title>3.5.2 Semi-local Matching</title>
<p>ICP alignment detects static objects in case of close proximity between the two instances. For other objects in the environment, where the initialization is poor, a more robust matching scheme is required that is independent of the pre-alignment. We propose to match objects by using the PPF global descriptor. PPF is a simple learning-free descriptor, nevertheless top-ranked in pose estimation challenges (<xref ref-type="bibr" rid="B26">Hoda&#x148; et al., 2018</xref>). In our system, PPF is computed for each unmatched object and the pipeline returns for each candidate between zero and ten hypotheses for each model. For each hypothesis we compute a confidence score for the model and the candidate as formulated in <xref ref-type="disp-formula" rid="e9">Eq. 9</xref> and <xref ref-type="disp-formula" rid="e10">Eq. 10</xref> and then compute the average. Only the best fitting hypothesis from each model is retained per candidate. Note that planar and small objects are filtered before applying PPF matching. All objects are downsampled to achieve a unified point density with a voxel size <italic>v</italic> and only objects with more than <italic>obj</italic>
<sub>min</sub> points and where less than <italic>&#x3c4;</italic>
<sub>
<italic>plane</italic>
</sub> of the points can be explained by a plane model are kept. Otherwise their geometric characteristic is too generic and would result in many false matches.</p>
<p>More concretely, the matching problem is simplified by eliminating objects that match with a high certainty. To unravel the hypothesis we use a bipartite maximum matching graph (<xref ref-type="bibr" rid="B16">Edmonds, 1965</xref>). The nodes on one side are the models and on the other side the candidates. A connecting edge exists if the model hypothesis for a candidate fulfils min (<italic>S</italic>
<sub>
<italic>m</italic>
</sub>, <italic>S</italic>
<sub>
<italic>c</italic>
</sub>) &#x3e; <italic>&#x3c4;</italic>
<sub>
<italic>high</italic>
</sub>, where <italic>&#x3c4;</italic>
<sub>
<italic>high</italic>
</sub> is a fixed threshold. The weight of an edge is the average of the two scores. The maximum weighted matching of the graph is then computed. For each model-candidate pair in the graph solution we compute the spatial distance and categorize it either as static or moved. These models and candidates are considered as fully matched and are not processed any further.</p>
<p>With the reduced set of models and candidates, a new graph is built where the condition for an edge is relaxed. An edge is created if min (<italic>S</italic>
<sub>
<italic>m</italic>
</sub>, <italic>S</italic>
<sub>
<italic>c</italic>
</sub>) &#x3e; <italic>&#x3c4;</italic>
<sub>min</sub> and avg (<italic>S</italic>
<sub>
<italic>m</italic>
</sub>, <italic>S</italic>
<sub>
<italic>c</italic>
</sub>) &#x3e; <italic>&#x3c4;</italic>
<sub>
<italic>low</italic>
</sub>. With <italic>&#x3c4;</italic>
<sub>
<italic>low</italic>
</sub> &#x226a; <italic>&#x3c4;</italic>
<sub>
<italic>high</italic>
</sub> it is possible to match models and candidates in heaps. In such cases it is often infeasible to achieve a very high confidence because of objects that are clustered together and increase the number of points used to normalize the score. This concept also helps to overcome deficits arising from incomplete reconstructions caused, for example, by few viewpoints or occlusions. The extracted matching results from the graph are then processed the same way as described in <xref ref-type="sec" rid="s3-5-1">Section 3.5.1</xref>: starting from the matched points region growing is performed to extract all points from the reconstruction belonging to the matched model/candidate. These points are then categorized as static or moved depending on their spatial distance. Remaining points are considered as an additional model/candidate.</p>
<p>For all unmatched candidates, new hypotheses with the unmatched models are computed. The matching process restarts by building a graph with the relaxed edge condition. This procedure is repeated until no more matches are found.</p>
</sec>
<sec id="s3-5-3">
<title>3.5.3 Global Matching</title>
<p>The final matching procedure considers objects that have been moved between different surfaces. This is performed by collecting all models and candidates from all surfaces that were not matched in the local or semi-local checks. Technically, the same approach as described in <xref ref-type="sec" rid="s3-5-2">Section 3.5.2</xref> is applied but now all objects from all surfaces are pooled together to perform global matching. The PPF descriptor is the basis for hypotheses creation and computed confidence scores are used as edge weights for a maximum weight bipartite graph. In the case that there is no match for a candidate or a model in the entire environment, the candidate is considered new or a model is considered removed.</p>
<p>The advantage of our approach is that the robot does not need to learn object models in a cumbersome process, but inherently extends its knowledge through change detection. <xref ref-type="fig" rid="F3">Figure 3</xref> shows an example of the clutter dissolving capabilities of our system on an example from the real-world dataset. At timestep <italic>t</italic>
<sub>0</sub> a pitcher and a mug are detected next to each other. They are treated as one object. At timestep <italic>t</italic>
<sub>1</sub> the pitcher is a single standing object, while the mug belongs to a pile together with the bowl. First the pitcher is matched and with that information the mug is separated from the pitcher in <italic>t</italic>
<sub>0</sub>. Now the separated mug is matched with the mug in <italic>t</italic>
<sub>1</sub>. The result of the matching process is that all heaps are disentangled and instead of three object clusters the robot is now aware of five individual objects.</p>
<fig id="F3" position="float">
<label>FIGURE 3</label>
<caption>
<p>Example showing how objects are separated using PPF matching. Objects next to each other having the same shadow color are recognized as one object by the robot. After objects are matched they have the same shadow color in both recordings.</p>
</caption>
<graphic xlink:href="frobt-09-828732-g003.tif"/>
</fig>
</sec>
</sec>
</sec>
<sec id="s4">
<title>4 Experiments</title>
<p>This section evaluates the performance of our approach for detecting and categorizing object changes. It is compared to two variants of a learning-based method as baselines and quantitative experiments are conducted with the newly created <italic>ObChange</italic> dataset, consisting of recordings from a real autonomous robot (see <xref ref-type="sec" rid="s4-1">Section 4.1</xref>). Additionally, we qualitatively demonstrate the applicability of our method in real-world scenarios with online experiments using a real robot (see <xref ref-type="sec" rid="s4-2">Section 4.2</xref>). Finally, we discuss the indication of our experimental results and the consequent open research questions in <xref ref-type="sec" rid="s4-3">Section 4.3</xref>. For reproducibility we give an overview of all parameters used for the dataset and real-world evaluation in <xref ref-type="table" rid="T2">Table 2</xref>.</p>
<table-wrap id="T2" position="float">
<label>TABLE 2</label>
<caption>
<p>Parameters used for evaluation.</p>
</caption>
<table>
<thead valign="top">
<tr>
<th align="left">Method</th>
<th align="center">Parameter</th>
<th align="center">Value</th>
</tr>
</thead>
<tbody valign="top">
<tr>
<td align="left">Local plane extraction</td>
<td align="left">Maximum angle <italic>&#x3b1;</italic>
<sub>max</sub> between plane normal and upward-directed axis</td>
<td align="center">5&#xb0;</td>
</tr>
<tr>
<td align="left">Local plane extraction</td>
<td align="left">Inlier distance threshold <italic>d</italic>
<sub>
<italic>plane</italic>
</sub> for plane model</td>
<td align="center">0.015&#xa0;m</td>
</tr>
<tr>
<td align="left">Matching score</td>
<td align="left">Inlier threshold <italic>&#x3c4;</italic> for radius search</td>
<td align="center">0.01&#xa0;m</td>
</tr>
<tr>
<td align="left">Matching score</td>
<td align="left">Color threshold for point-wise matching <italic>&#x3c4;</italic>
<sub>
<italic>col</italic>
</sub>
</td>
<td align="center">20</td>
</tr>
<tr>
<td align="left">Matching score</td>
<td align="left">Dot product threshold between two normal vecots <italic>&#x3c4;</italic>
<sub>
<italic>geo</italic>
</sub> for point-wise matching</td>
<td align="center">0.95</td>
</tr>
<tr>
<td align="left">Matching score</td>
<td align="left">Linear weight factor <inline-formula id="inf1115">
<mml:math id="m1120">
<mml:mi>&#x03C9;</mml:mi>
</mml:math>
</inline-formula> for combined score</td>
<td align="center">0.7</td>
</tr>
<tr>
<td align="left">Object filtering</td>
<td align="left">Voxel size <italic>v</italic> for object downsampling</td>
<td align="center">0.005&#xa0;&#x2009;m</td>
</tr>
<tr>
<td align="left">Object filtering</td>
<td align="left">Minimum number of object points <italic>obj</italic>
<sub>min</sub>
</td>
<td align="center">200</td>
</tr>
<tr>
<td align="left">Object filtering</td>
<td align="left">Proportion of object points to count as plane <italic>&#x3c4;</italic>
<sub>
<italic>plane</italic>
</sub>
</td>
<td align="center">0.9</td>
</tr>
<tr>
<td align="left">Region growing</td>
<td align="left">Point inlier radius <italic>r</italic>
</td>
<td align="center">0.01&#xa0;m</td>
</tr>
<tr>
<td align="left">Region growing</td>
<td align="left">Maximum allowed angle between normal vectors of neighboring points <italic>rg</italic>
<sub>
<italic>geo</italic>
</sub>
</td>
<td align="center">5&#xb0;</td>
</tr>
<tr>
<td align="left">Region growing</td>
<td align="left">Maximum allowed color difference between neighboring points <italic>rg</italic>
<sub>
<italic>col</italic>
</sub>
</td>
<td align="center">15</td>
</tr>
<tr>
<td align="left">Local matching</td>
<td align="left">Minimum score for candidate match <italic>&#x3c4;</italic>
<sub>
<italic>icp</italic>
</sub>
</td>
<td align="center">0.7</td>
</tr>
<tr>
<td align="left">Semi-local matching</td>
<td align="left">Low score threshold <italic>&#x3c4;</italic>
<sub>
<italic>low</italic>
</sub> for graph edge</td>
<td align="center">0.4</td>
</tr>
<tr>
<td align="left">Semi-local matching</td>
<td align="left">High score threshold <italic>&#x3c4;</italic>
<sub>
<italic>high</italic>
</sub> for graph edge</td>
<td align="center">0.8</td>
</tr>
<tr>
<td align="left">Semi-local matching</td>
<td align="left">Minimum score for model and candidate <italic>&#x3c4;</italic>
<sub>min</sub>
</td>
<td align="center">0.2</td>
</tr>
<tr>
<td align="left">PPF</td>
<td align="left">Distance sampling rate as defined in <xref ref-type="bibr" rid="B15">Drost et al. (2010</xref>)</td>
<td align="center">0.025</td>
</tr>
<tr>
<td align="left">PPF</td>
<td align="left">Orientation sampling rate as defined in <xref ref-type="bibr" rid="B15">Drost et al. (2010</xref>)</td>
<td align="center">5</td>
</tr>
</tbody>
</table>
</table-wrap>
<sec id="s4-1">
<title>4.1 Evaluation on the Robotic Dataset ObChange</title>
<p>Since no suitable dataset to evaluate object mapping exists, we extend the dataset from <xref ref-type="bibr" rid="B31">Langer et al. (2020</xref>). Its description is given in <xref ref-type="sec" rid="s4-1-1">Section 4.1.1</xref>. <xref ref-type="sec" rid="s4-1-2">Section 4.1.2</xref> introduces the metrics to measure the performance for the evaluation. In order to analyze our approach quantitatively, we compare it to two learning-based baseline using state-of-the-art methods as outlined in <xref ref-type="sec" rid="s4-1-3">Section 4.1.3</xref>. In <xref ref-type="sec" rid="s4-1-4">Section 4.1.4</xref> we show overall performance and discuss in detail the results and open challenges in <xref ref-type="sec" rid="s4-1-5">Section 4.1.5</xref>.</p>
<sec id="s4-1-1">
<title>4.1.1 The ObChange Dataset</title>
<p>For the quantitative evaluation of the object mapping task, we create the <italic>ObChange</italic> dataset, which consists of RGB and depth images acquired with the onboard Asus camera mounted on the head of the Human Support Robot from Toyota (<xref ref-type="bibr" rid="B57">Yamamoto et al., 2019</xref>). Additionally, the transformation matrices between different coordinate frames are recorded. The dataset is recorded for five different rooms or parts of rooms, namely a big room, a small room, a living area, a kitchen counter, and an office desk. For each room a version in a clean state exists that is used to extract surfaces. Each room is visited by the robot at least five times while between each run a subset of objects from the YCB dataset (<xref ref-type="bibr" rid="B11">Calli et al., 2017</xref>) is re-arranged in the room. For each setup between three and 17 objects are placed. Furthermore, furniture and permanent background objects are slightly rearranged compared to the reference. These rearrangements are small so that the moved furniture does not interfere with the robot&#x2019;s navigation and that the moved objects are considered irrelevant in a tidy-up task. We use all rooms and all different recordings, which include YCB objects, in total 26 recordings.</p>
<p>The <italic>ObChange</italic> dataset extension from the data used in <xref ref-type="bibr" rid="B31">Langer et al. (2020</xref>) is for the reconstructions necessary for object matching. For each room a 3D semantically labeled reconstruction is created, which is used to identify horizontal planes as described in <xref ref-type="sec" rid="s3-3">Section 3.3</xref>. For each detected surface, images from the recorded stream where the surface is visible are extracted, depth images are masked according to the plane parameters, and <italic>ElasticFusion</italic> is used to reconstruct the area. Only with the combination of odometry pose and camera tracking pose as in <xref ref-type="disp-formula" rid="e4">Eq. 4</xref>, suitable reconstructions for all surfaces are achieved. In this dataset, the robot drives exhaustively through an environment, leading to surfaces of interest being seen several times. Unfortunately, <italic>ElasticFusion</italic> cannot handle non-continuous input data. To overcome this problem, we create several reconstructions and merge them using ICP. We visually check the results and have adjusted them manually if needed. This manual step is not needed in the real world where the robot moves around the surface only once. The collection of all surface reconstructions together with their point-wise labeling of all YCB objects form the <italic>ObChange</italic> dataset. It is available at <ext-link ext-link-type="uri" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="https://doi.org/10.48436/y3ggy-hxp10">https://doi.org/10.48436/y3ggy-hxp10</ext-link>.</p>
<p>Compared to our previous work (<xref ref-type="bibr" rid="B31">Langer et al., 2020</xref>) we are not only interested in detecting all objects, but also to assign them to one of the four categories: static, moved, removed, and novel. To create a more meaningful evaluation of the possible categories, we compare each recording with all other recordings of the same room, leading to 961 objects in total. For each comparison the objects of both recordings are counted, meaning that static and moved objects are counted at <italic>t</italic>
<sub>0</sub> and <italic>t</italic>
<sub>1</sub>. <xref ref-type="table" rid="T3">Table 3</xref> gives an overview of the data for each room.</p>
<table-wrap id="T3" position="float">
<label>TABLE 3</label>
<caption>
<p>Overview of the dataset used for the quantitative evaluation. Except for the first two columns the numbers state the sum over all possible comparisons per room.</p>
</caption>
<table>
<thead valign="top">
<tr>
<th align="left">Rooms</th>
<th align="center">&#x23;Visits</th>
<th align="center">&#x23;Surfaces</th>
<th align="center">&#x23;Objects</th>
<th align="center">&#x23;Static</th>
<th align="center">&#x23;Moved</th>
<th align="center">&#x23;Removed</th>
<th align="center">&#x23;Novel</th>
</tr>
</thead>
<tbody valign="top">
<tr>
<td align="left">Big Room</td>
<td align="char" char=".">6</td>
<td align="char" char=".">12</td>
<td align="char" char=".">425</td>
<td align="char" char=".">28</td>
<td align="char" char=".">260</td>
<td align="char" char=".">50</td>
<td align="char" char=".">87</td>
</tr>
<tr>
<td align="left">Small Room</td>
<td align="char" char=".">5</td>
<td align="char" char=".">5</td>
<td align="char" char=".">208</td>
<td align="char" char=".">72</td>
<td align="char" char=".">62</td>
<td align="char" char=".">25</td>
<td align="char" char=".">49</td>
</tr>
<tr>
<td align="left">Living Area</td>
<td align="char" char=".">5</td>
<td align="char" char=".">11</td>
<td align="char" char=".">108</td>
<td align="char" char=".">0</td>
<td align="char" char=".">54</td>
<td align="char" char=".">30</td>
<td align="char" char=".">24</td>
</tr>
<tr>
<td align="left">Office Desk</td>
<td align="char" char=".">5</td>
<td align="char" char=".">4</td>
<td align="char" char=".">100</td>
<td align="char" char=".">12</td>
<td align="char" char=".">28</td>
<td align="char" char=".">24</td>
<td align="char" char=".">36</td>
</tr>
<tr>
<td align="left">Kitchen Counter</td>
<td align="char" char=".">5</td>
<td align="char" char=".">2</td>
<td align="char" char=".">120</td>
<td align="char" char=".">10</td>
<td align="char" char=".">66</td>
<td align="char" char=".">14</td>
<td align="char" char=".">30</td>
</tr>
<tr>
<td align="left"/>
<td align="char" char=".">26</td>
<td align="char" char=".">34</td>
<td align="char" char=".">961</td>
<td align="char" char=".">122</td>
<td align="char" char=".">470</td>
<td align="char" char=".">143</td>
<td align="char" char=".">226</td>
</tr>
</tbody>
</table>
</table-wrap>
</sec>
<sec id="s4-1-2">
<title>4.1.2 Metrics</title>
<p>In <italic>ObChange</italic> only YCB objects change, i.e., are novel, moved, or removed. All other objects are static and therefore irrelevant for change detection. Given that, we apply the following metrics:<list list-type="simple">
<list-item>
<p>&#x2022; <italic>Detected objects</italic>: The number and percentage of detected YCB objects on surfaces.</p>
</list-item>
<list-item>
<p>&#x2022; <italic>Correctly categorized objects</italic>: The number and percentage of YCB objects that are correctly categorized as static, moved, removed, or novel.</p>
</list-item>
<list-item>
<p>&#x2022; <italic>False positives</italic>: The number of background objects that are not categorized as static plus the number of objects that are correctly classified, but parts of it have an incorrect category assigned.</p>
</list-item>
</list>
</p>
</sec>
<sec id="s4-1-3">
<title>4.1.3 Baseline</title>
<p>To the best of our knowledge, no other work exists to categorize objects into static, moved, removed, and novel by comparing two time instances. Thus, in order to compare our approach for solving object mapping, we extend the recent work of Oliveira et al. (<xref ref-type="bibr" rid="B13">de Oliveira et al., 2021</xref>) for the task. Their proposed method detects objects with Yolov3 (<xref ref-type="bibr" rid="B48">Redmon and Farhadi, 2018</xref>) trained on the COCO dataset (<xref ref-type="bibr" rid="B33">Lin et al., 2014</xref>) in each frame. These object detections are used to create a 3D object map by temporal and spatial associations. We utilize this 3D object map to perform object matching. Thus, based on their work we develop two baselines. Instead of Yolov3 we use Mask R-CNN(<xref ref-type="bibr" rid="B25">He et al., 2017</xref>), another state-of-the-art object detector, because it is not only readily available for the established COCO dataset but also for the YCBV dataset (<xref ref-type="bibr" rid="B56">Xiang et al., 2017</xref>). We refer to the baselines as <italic>COCO-baseline</italic> and <italic>YCBV-baseline</italic>.</p>
<p>For both baselines we create 3D object maps for all dataset recordings using the standard parameters mentioned in <xref ref-type="bibr" rid="B13">de Oliveira et al. (2021</xref>) with the following exceptions: (1) use Mask R-CNN instead of Yolov3, (2) decrease the distance threshold for the bounding box center distance to 0.0001 (which equals to 30 pixels) and decrease the spatial association to 0.3&#x2009;m for better results. The RGB images from the recordings and the estimated poses from <italic>ElasticFusion</italic>, which are more accurate than pure robot poses, constitute the input for 3D object maps. For the <italic>COCO-baseline</italic> we do not include detected objects from the following COCO categories in the 3D object map because they cover classes not relevant for indoor object change detection: person, vehicle, outdoor, animal, furniture (except potted plant), and appliance. The example in <xref ref-type="fig" rid="F4">Figure 4</xref> shows the detected objects with their assigned classes overlayed on the reconstruction.</p>
<fig id="F4" position="float">
<label>FIGURE 4</label>
<caption>
<p>3D object map of detected objects using Mask R-CNN trained on COCO created with the approach from <xref ref-type="bibr" rid="B13">de Oliveira et al. (2021</xref>). Besides some background objects, all YCB objects are detected, except for the gelatin box (marked with cyan rectangle).</p>
</caption>
<graphic xlink:href="frobt-09-828732-g004.tif"/>
</fig>
<p>Based on the generated 3D object maps, we compute the changes between two visits to a room using the assigned labels from the object detector. An object with a class label that exists in the visit at time <italic>t</italic>
<sub>0</sub>, but not at time <italic>t</italic>
<sub>1</sub>, is considered removed and vice versa as novel. If there is exactly one object of a specific object class in both visits, it is either categorized as static or moved depending on the distance between the object&#x2019;s bounding box centers compared to the threshold <italic>d</italic>. In the case that several objects from the same class are detected, we find an association by utilizing the feature vector of the second last layer of Mask R-CNN. This is inspired by Qiu et al. (<xref ref-type="bibr" rid="B46">Qiu et al., 2020</xref>) who extract the features from the last layer as instance-level features. However, in our experience features from the last layer are already too class-specific, whereas the second last layer is more suitable for instance comparisons. Because an object is usually detected in several frames, the feature vector is extracted for each. To find the best matching instances from the same class within two visits, we compute for each object at <italic>t</italic>
<sub>0</sub>,the dot-product between all its feature vectors with all feature vectors of all objects from <italic>t</italic>
<sub>1</sub>, then match the object that achieves the highest value. Depending on the distance between the two matched objects, they are either categorized as static or moved.</p>
<p>The baseline using Mask R-CNN trained on the YCBV dataset (<xref ref-type="bibr" rid="B56">Xiang et al., 2017</xref>) is evaluated to analyze the performance when provided a tailored training set in comparison to the more general COCO dataset. Park et al. (<xref ref-type="bibr" rid="B43">Park et al., 2019</xref>) provide the weights for Mask R-CNN, which they trained for their pipeline to participate in the BOP challenge (<xref ref-type="bibr" rid="B26">Hoda&#x148; et al., 2018</xref>) on the YCBV dataset.</p>
</sec>
<sec id="s4-1-4">
<title>4.1.4 Evaluation</title>
<p>Our evaluation is based on manually labeled data. In <italic>ObChange</italic> all YCB objects are point-wise labeled in all plane reconstructions for all the recordings. Besides the object point indices, the object name is stored as ground truth data. The categories (static, moved, removed, novel) are extracted given two recordings and the ground truth: If the object name exists only in one of the two recordings, the object is novel or removed. If it occurs in both recordings, we compute the centroid of both objects and, depending on the Euclidean distance, assign the category static or moved.</p>
<p>Our method works directly on the plane reconstructions and therefore the resulting object points match directly to the labeled data, and no further processing is needed. We consider an object as detected if at least 50% of the points in the result overlap with the labeled points. Each point in the result has one of the four categories assigned. The object matching stage may erroneously assign different labels to data points from the same object due to imprecise region growing. Therefore, we use the maximum voted category per object and compare it against the ground truth. If the categories match, the object is correctly categorized. Otherwise, it counts as a wrongly categorized object. For the categories static and moved, the correct association of the two involved objects must be given to be counted as correct. For example, it is not enough that an object at time <italic>t</italic>
<sub>0</sub> and <italic>t</italic>
<sub>1</sub> is categorized as moved; also, the association that the object at <italic>t</italic>
<sub>0</sub> moved to the location of the object at <italic>t</italic>
<sub>1</sub> must be given. The sum of false positives combines two different failure cases: (1) a YCB object where parts of it are wrongly classified. An object is therefore counted as correctly classified and at the same time it is a false positive. (2) Points in the result that are categorized as moved, removed, or novel but do not belong to any YCB object are from a static background object. We cluster the points and each cluster is counted as one false positive object. Equally to <xref ref-type="bibr" rid="B31">Langer et al. (2020</xref>), this metric is an approximation because no ground truth labeling for background objects exists.</p>
<p>Both baselines, in contrast, cannot be evaluated point-wise. Each detected object is described by a single 3D location and a label. Therefore, for each object in the ground truth, the centroid is computed and the closest object in the result is identified. If the distance is less than 10&#xa0;cm, we count it as a detected object; otherwise, it is a false positive. Further, we check for each detected object if there is a nearby ground truth object with the same category, which was not already matched. If so, the detected object is correctly categorized. The associations for moved and static detected objects must correspond to the ground truth. For the evaluation of the <italic>YCBV-baseline</italic>, we remove objects from the ground truth that do not appear in the YCBV dataset and are therefore not used for training.</p>
<p>We provide an additional evaluation based on the fact that if a static or moved model-candidate pair is not detected, a subsequent failure may occur. For example, if a moved mug is detected at <italic>t</italic>
<sub>0</sub>, but not at <italic>t</italic>
<sub>1</sub>, it can never be categorized correctly as moved. Therefore, we re-evaluate the methods based on an adapted ground truth. Only detected objects are considered when re-computing the categorization for the ground truth objects. The correct categorization for the mug in the previously mentioned example would then be <italic>removed</italic>. This way we evaluate the categorization process stand-alone and independent of the preceding detection performance.</p>
</sec>
<sec id="s4-1-5">
<title>4.1.5 Results</title>
<p>A summary of the performance on <italic>ObChange</italic> of our proposed method as well as the two baselines is given in <xref ref-type="table" rid="T4">Table 4</xref>. Our method detects 91.8% of all the labeled objects in the dataset while achieving the lowest number of false positive objects. Inspecting the detection rates of the two baselines, the result is surprising. Applying Mask R-CNN trained on the COCO dataset outperforms Mask R-CNN trained on the YCBV dataset by a significant margin although the ground truth objects in <italic>ObChange</italic> are selected from the YCB objects. Recently, Dhamija et al. (<xref ref-type="bibr" rid="B14">Dhamija et al., 2020</xref>) investigated state-of-the-art object detectors and their performance in open-world settings. They showed that all methods detect objects from classes not presented during training with high confidence, despite the fact that object detectors should only detect objects from known classes. This could explain the good performance of <italic>COCO-baseline</italic>, although trained on different classes. We conjecture that the detection performance of <italic>YCBV-baseline</italic> is only about 43% because of two reasons: (1) although the objects used in <italic>ObChange</italic> are from the official YCB object set, some have a slightly different texture than the objects used in the YCBV dataset and (2) the training data is captured from a certain distance range, while the robot did not always get as close to the objects in the dataset. Therefore, we assume that the object detector overfits to the training data, which is a huge problem when applying it to images that are dissimilar to the training dataset. The detection rate could be increased by using a lower confidence threshold; however, this would also increase the number of false positives.</p>
<table-wrap id="T4" position="float">
<label>TABLE 4</label>
<caption>
<p>Results of the baseline trained on COCO and YCBV compared to the results of our method evaluated on <italic>ObChange</italic> and averaged per room.</p>
</caption>
<table>
<thead valign="top">
<tr>
<th colspan="6" align="left">Result using Mask R-CNN trained on COCO dataset</th>
</tr>
<tr>
<th align="left">Rooms</th>
<th align="center">&#x23;Total objects</th>
<th align="center">&#x23;Detected objects</th>
<th align="center">&#x23;Correctly categorized</th>
<th align="center">&#x23;Correctly categorized in detected objects</th>
<th align="center">&#x23;False positives</th>
</tr>
</thead>
<tbody valign="top">
<tr>
<td align="left">Big Room</td>
<td align="center">425</td>
<td align="center">340 (80.0%)</td>
<td align="center">117 (27.5%)</td>
<td align="center">130 (38.2%)</td>
<td align="center">218</td>
</tr>
<tr>
<td align="left">Small Room</td>
<td align="center">208</td>
<td align="center">164 (78.8%)</td>
<td align="center">102 (49.0%)</td>
<td align="center">106 (64.6%)</td>
<td align="center">10</td>
</tr>
<tr>
<td align="left">Living Area</td>
<td align="center">108</td>
<td align="center">72 (66.7%)</td>
<td align="center">61 (56.5%)</td>
<td align="center">65 (90.3%)</td>
<td align="center">20</td>
</tr>
<tr>
<td align="left">Office Desk</td>
<td align="center">100</td>
<td align="center">64 (64.0%)</td>
<td align="center">50 (50.0%)</td>
<td align="center">55 (85.9%)</td>
<td align="center">64</td>
</tr>
<tr>
<td align="left">Kitchen Counter</td>
<td align="center">120</td>
<td align="center">88 (73.3%)</td>
<td align="center">50 (41.7%)</td>
<td align="center">57 (64.8%)</td>
<td align="center">22</td>
</tr>
<tr>
<td align="left">Overall Performance</td>
<td align="center">961</td>
<td align="center">728 (75.8%)</td>
<td align="center">380 (39.5%)</td>
<td align="center">413 (56.7%)</td>
<td align="center">334</td>
</tr>
<tr>
<td colspan="6" align="left">
<bold>Result using Mask R-CNN trained on YCBV dataset</bold>
</td>
</tr>
<tr>
<td align="left">
<bold>&#xa0;Rooms</bold>
</td>
<td align="center">
<bold>&#x23;Total objects</bold>
</td>
<td align="center">
<bold>&#x23;Detected objects</bold>
</td>
<td align="center">
<bold>&#x23;Correctly categorized</bold>
</td>
<td align="center">
<bold>&#x23;Correctly categorized in detected objects</bold>
</td>
<td align="center">
<bold>&#x23;False positives</bold>
</td>
</tr>
<tr>
<td align="left">&#xa0;Big Room</td>
<td align="center">215</td>
<td align="center">95 (44.2%)</td>
<td align="center">48 (22.3%)</td>
<td align="center">75 (78.9)</td>
<td align="center">219</td>
</tr>
<tr>
<td align="left">&#xa0;Small Room</td>
<td align="center">100</td>
<td align="center">32 (32.0%)</td>
<td align="center">14 (14.0%)</td>
<td align="center">18 (56.3)</td>
<td align="center">75</td>
</tr>
<tr>
<td align="left">&#xa0;Living Area</td>
<td align="center">56</td>
<td align="center">20 (35.7%)</td>
<td align="center">12 (21.4%)</td>
<td align="center">18 (90.0)</td>
<td align="center">22</td>
</tr>
<tr>
<td align="left">&#xa0;Office Desk</td>
<td align="center">40</td>
<td align="center">24 (60.0%)</td>
<td align="center">18 (45.0%)</td>
<td align="center">24 (100)</td>
<td align="center">36</td>
</tr>
<tr>
<td align="left">&#xa0;Kitchen Counter</td>
<td align="center">100</td>
<td align="center">48 (48.0%)</td>
<td align="center">24 (24.0%)</td>
<td align="center">34 (70.8)</td>
<td align="center">49</td>
</tr>
<tr>
<td align="left">&#xa0;Overall Performance</td>
<td align="center">511</td>
<td align="center">219 (42.9%)</td>
<td align="center">116 (22.7%)</td>
<td align="center">
<bold>169 (77.2%)</bold>
</td>
<td align="center">401</td>
</tr>
<tr>
<td colspan="6" align="left">
<bold>Result of our approach</bold>
</td>
</tr>
<tr>
<td align="left">
<bold>&#xa0;Rooms</bold>
</td>
<td align="center">
<bold>&#x23;Total objects</bold>
</td>
<td align="center">
<bold>&#x23;Detected objects</bold>
</td>
<td align="center">
<bold>&#x23;Correctly categorized</bold>
</td>
<td align="center">
<bold>&#x23;Correctly categorized in detected objects</bold>
</td>
<td align="center">
<bold>&#x23;False positives</bold>
</td>
</tr>
<tr>
<td align="left">&#xa0;Big Room</td>
<td align="center">425</td>
<td align="center">419 (98.6%)</td>
<td align="center">286 (67.3%)</td>
<td align="center">290 (69.2%)</td>
<td align="center">66</td>
</tr>
<tr>
<td align="left">&#xa0;Small Room</td>
<td align="center">208</td>
<td align="center">183 (88.0%)</td>
<td align="center">131 (63.0%)</td>
<td align="center">136 (71.6%)</td>
<td align="center">13</td>
</tr>
<tr>
<td align="left">&#xa0;Living Area</td>
<td align="center">108</td>
<td align="center">92 (85.2%)</td>
<td align="center">75 (69.4%)</td>
<td align="center">78 (84.48%)</td>
<td align="center">10</td>
</tr>
<tr>
<td align="left">&#xa0;Office Desk</td>
<td align="center">100</td>
<td align="center">88 (88.0%)</td>
<td align="center">76 (76.0%)</td>
<td align="center">78 (88.6%)</td>
<td align="center">24</td>
</tr>
<tr>
<td align="left">&#xa0;Kitchen Counter</td>
<td align="center">120</td>
<td align="center">100 (83.3%)</td>
<td align="center">49 (40.8%)</td>
<td align="center">56 (56.0%)</td>
<td align="center">28</td>
</tr>
<tr>
<td align="left">&#xa0;Overall Performance</td>
<td align="center">961</td>
<td align="center">
<bold>882 (91.8%)</bold>
</td>
<td align="center">
<bold>617 (64.2%)</bold>
</td>
<td align="center">638 (72.3%)</td>
<td align="center">
<bold>141</bold>
</td>
</tr>
</tbody>
</table>
</table-wrap>
<p>From the detected objects 64.3% are correctly classified with our approach. This is 1.5&#xd7; more than <italic>COCO-baseline</italic> and 3&#xd7; more than <italic>YCB-baseline</italic>. The difference between the results evaluated on the original ground truth and the adapted ground truth (<italic>&#x23;Correctly categorized in detected objects</italic>) is small for our approach because we achieve a high detection rate. For <italic>YCBV-baseline</italic>, the difference is significant. Comparing the adapted ground truth, <italic>YCBV-baseline</italic> slightly outperforms our approach in terms of correctly categorized objects (given that a substantially lower number of objects is detected in the first place). However, it has significantly more false positives even though it is specifically trained for the objects in the dataset.</p>
<p>The performance for each YCB object used in the evaluation is shown in <xref ref-type="fig" rid="F5">Figure 5</xref> for our approach and the baselines. It shows that the smallest object in the dataset, the large marker, cannot be localized by any method. However, our method was able to localize other small objects such as the screwdrivers. It can be seen that our method has difficulties detecting a plate because many points can be explained by the supporting plane model and are within the distance threshold <italic>d</italic>
<sub>
<italic>plane</italic>
</sub>. Interestingly, for all objects from the YCB dataset the localization performance of <italic>COCO-baseline</italic> is superior compared to <italic>YCBV-baseline</italic>. The lemon is the only object where our approach performs significantly worse in categorizing than the baseline. The reason is that the lemon is small as well as part of a pile in most scenes. The wrong categorizations of the master chef can are the result of incomplete object reconstructions and the confusion with the pitcher, which has very similar appearance.</p>
<fig id="F5" position="float">
<label>FIGURE 5</label>
<caption>
<p>Overview of YCB objects used in the dataset. The performance for each object is shown for our approach (first bar), <italic>COCO-baseline</italic> (second bar) and <italic>YCBV-baseline</italic> (third bar). If the object is not in the YCBV dataset, no bar for <italic>YCBV-baseline</italic> is visualized.</p>
</caption>
<graphic xlink:href="frobt-09-828732-g005.tif"/>
</fig>
</sec>
</sec>
<sec id="s4-2">
<title>4.2 Robot Experiments</title>
<p>For the real-world robot experiments we use a Toyota Human Support Robot in one room with nine surfaces and conduct three runs with different object changes. <xref ref-type="fig" rid="F6">Figure 6</xref> presents the reconstruction of the room using <italic>Voxblox</italic>. It also shows the surfaces that are automatically extracted by exploiting semantic labeling as outlined in <xref ref-type="sec" rid="s3-3">Section 3.3</xref>.</p>
<fig id="F6" position="float">
<label>FIGURE 6</label>
<caption>
<p>Reconstruction of the room used for the real robot experiments. The detected planes are highlighted in turquoise.</p>
</caption>
<graphic xlink:href="frobt-09-828732-g006.tif"/>
</fig>
<p>To create the waypoints for each surface, robot locations are generated at a distance of 20&#xa0;cm to the convex hull around the plane oriented towards the plane center and evenly distributed every 30&#xa0;cm around the circumference. To create the reference map (used for all three runs), the robot visits each surface and stores the detected objects in the database. Twenty out of 21 objects placed on the extracted planes are detected. The object locations then change three times for the three runs at times <italic>t</italic>
<sub>1</sub> to <italic>t</italic>
<sub>3</sub>.</p>
<p>
<xref ref-type="table" rid="T5">Table 5</xref> presents the results. For each run we give the numbers of the objects assigned to the four categories compared to the ground truth. Missed objects are those that were not detected at all and therefore result in incorrectly categorized objects. The table also shows the number of false positive objects as defined in <xref ref-type="sec" rid="s4-1-2">Section 4.1.2</xref>. The difference in the numbers of the last column between our approach and the ground truth stems from the sum of objects that were not detected (<italic>&#x23;Missed</italic>) and from wrongly matched objects. In summary, most of the objects are detected (65 of 70) and correctly classified (61 of 65). Including the removed objects, 65 of 74 objects are correctly categorized (last column in the table). For each run the detection and matching part took approximately 10&#xa0;min on a standard laptop.</p>
<table-wrap id="T5" position="float">
<label>TABLE 5</label>
<caption>
<p>Results of matching objects from the robot experiments. The rows with GT refer to ground truth for the respective runs 1 to 3.</p>
</caption>
<table>
<thead valign="top">
<tr>
<th align="left">Run</th>
<th align="center">&#x23;Static</th>
<th align="center">&#x23;Moved</th>
<th align="center">&#x23;Removed</th>
<th align="center">&#x23;Novel</th>
<th align="center">&#x23;Missed</th>
<th align="center">&#x23;FPs</th>
<th align="center">&#x23;Correct</th>
</tr>
</thead>
<tbody valign="top">
<tr>
<td align="left">1</td>
<td align="char" char=".">17</td>
<td align="char" char=".">2</td>
<td align="char" char=".">1</td>
<td align="char" char=".">4</td>
<td align="char" char=".">1</td>
<td align="char" char=".">0</td>
<td align="char" char=".">24</td>
</tr>
<tr>
<td align="left">GT</td>
<td align="char" char=".">18</td>
<td align="char" char=".">2</td>
<td align="char" char=".">1</td>
<td align="char" char=".">4</td>
<td align="char" char=".">0</td>
<td align="char" char=".">0</td>
<td align="char" char=".">25</td>
</tr>
<tr>
<td align="left">2</td>
<td align="char" char=".">12</td>
<td align="char" char=".">5</td>
<td align="char" char=".">2</td>
<td align="char" char=".">3</td>
<td align="char" char=".">1</td>
<td align="char" char=".">0</td>
<td align="char" char=".">22</td>
</tr>
<tr>
<td align="left">GT</td>
<td align="char" char=".">13</td>
<td align="char" char=".">6</td>
<td align="char" char=".">2</td>
<td align="char" char=".">3</td>
<td align="char" char=".">0</td>
<td align="char" char=".">0</td>
<td align="char" char=".">24</td>
</tr>
<tr>
<td align="left">3</td>
<td align="char" char=".">5</td>
<td align="char" char=".">10</td>
<td align="char" char=".">1</td>
<td align="char" char=".">3</td>
<td align="char" char=".">3</td>
<td align="char" char=".">2</td>
<td align="char" char=".">19</td>
</tr>
<tr>
<td align="left">GT</td>
<td align="char" char=".">8</td>
<td align="char" char=".">12</td>
<td align="char" char=".">1</td>
<td align="char" char=".">4</td>
<td align="char" char=".">0</td>
<td align="char" char=".">0</td>
<td align="char" char=".">25</td>
</tr>
</tbody>
</table>
</table-wrap>
<p>
<xref ref-type="fig" rid="F7">Figure 7</xref> gives details about the matches. In run 1 all objects are matched correctly. In run 2, the Pringles can is misclassified when comparing the room at time <italic>t</italic>
<sub>0</sub> and <italic>t</italic>
<sub>2</sub>. At <italic>t</italic>
<sub>0</sub> the Pringles can is close to a tea box, which is correctly categorized as moved, but region growing fails to stop at the object boundaries. Therefore, the two objects are assigned the same label. This leads to the error that the Pringles can at <italic>t</italic>
<sub>2</sub> is wrongly labeled as new. At time <italic>t</italic>
<sub>3</sub> the keyboard is partially occluded when the robot moves around the table. Consequently the reconstruction is only a planar surface, which is not considered as an object. As a consequence the keyboard from time <italic>t</italic>
<sub>0</sub> is matched to the keyboard of a laptop (labeled as M11 in <xref ref-type="fig" rid="F7">Figure 7</xref>). The other wrongly categorized objects result from the inability to split the orange and blue object on the table in the middle of the room. Although for both objects PPF found the correct match and the confidence is the highest compared to other possible matches, it is still too low to accept the match. The reason is the low geometric overlap because the two objects are only partially reconstructed at <italic>t</italic>
<sub>3</sub> but almost a full model exists from <italic>t</italic>
<sub>0</sub>.</p>
<fig id="F7" position="float">
<label>FIGURE 7</label>
<caption>
<p>Results of real-world robot experiments. Each row shows the comparison of the reference room (first column) with the state of the same room at a different time (second column). Each detected object is marked with an ellipse and labeled with M (moved), R (removed), or N (novel). Correct categorizations are colored in green, wrong ones in red. Objects, which were matched, have the same number assigned. For simplicity static objects are not shown and missed objects are marked with a red cross.</p>
</caption>
<graphic xlink:href="frobt-09-828732-g007.tif"/>
</fig>
</sec>
<sec id="s4-3">
<title>4.3 Discussion</title>
<p>Based on the evaluation results using ObChange and the robot experiments, in the following sections we highlight the findings on remaining open challenges due to factors such as robot localisation, covering surfaces with views, the detection of small objects, partially occluded objects, and quality of reconstructions.</p>
<sec id="s4-3-1">
<title>4.3.1 Robot Localization Error</title>
<p>Precise robot localization is necessary for most change detection applications. It supports the pairing of frames as well as the creation of clean reconstructions. In our experience the performance of dense visual SLAM methods in environments with areas of little visual and geometric features greatly benefit from integrating odometry data assuming state-of-the-art localization. Our solution to integrate odometry data to the <italic>ElasticFusion</italic> framework could reconstruct all surfaces in <italic>ObChange</italic> because the robot drift is insignificant. In case this cannot be guaranteed, the recent work by Houseago et al. (<xref ref-type="bibr" rid="B27">Houseago et al., 2019</xref>) shows how to integrate odometry data and dense visual SLAM in the presence of drift. Although their method could be integrated in our system, we opted for a computationally cheap approach. This is important when working with mobile robots, where limited resources have to be shared among different system components.</p>
</sec>
<sec id="s4-3-2">
<title>4.3.2 Search Space</title>
<p>Detecting objects is often a trade-off between reducing the search space and not missing objects (examples are shown in <xref ref-type="fig" rid="F8">Figure 8</xref>), while at the same time the chance to detect false positives increases with a less restricted search space. The concept of surfaces reduces the search space significantly but may lead to missed objects if the surface is occluded or if it is not planar. We decrease the number of false positive detections by utilizing a shrinked convex hull of the surface, but this may cause objects at the border of a surface to not be detected.</p>
<fig id="F8" position="float">
<label>FIGURE 8</label>
<caption>
<p>Examples of missed objects. Extracted planes are visualized in red. First image: two objects in black rectangle. Only the yellow cup is detected because the couch seat is too curved. Second and third image: only few object points are within the convex hull of the plane and therefore classified as noise and rejected. Note, that in the third example the reconstruction is very sparse because the robot did not get close enough to the surface.</p>
</caption>
<graphic xlink:href="frobt-09-828732-g008.tif"/>
</fig>
</sec>
<sec id="s4-3-3">
<title>4.3.3 Detection of Small Objects</title>
<p>The detection of small objects is a well-known problem. Especially when working with 3D maps, actions taken to reduce the effect of noise and misalignments are the reason for filtering small objects. Our evaluation shows that neither the learning-based approaches nor the surface concept approach are capable of reliably detecting small objects.</p>
</sec>
<sec id="s4-3-4">
<title>4.3.4 Occlusion</title>
<p>Occlusion poses a significant challenge for object matching because parts of an object are missing. The same accounts for incomplete reconstruction due to view limitations. Generally, ambiguities due to truly similar object appearance or partial reconstructions (see <xref ref-type="fig" rid="F9">Figure 9</xref> first two examples) need to be counter-measured by taking into account as much information as possible, for example, color PPF (<xref ref-type="bibr" rid="B12">Choi and Christensen, 2012</xref>) could provide better a priori hypotheses at the cost of higher computational efforts.</p>
<fig id="F9" position="float">
<label>FIGURE 9</label>
<caption>
<p>Examples of hypotheses generated based on PPF.</p>
</caption>
<graphic xlink:href="frobt-09-828732-g009.tif"/>
</fig>
</sec>
<sec id="s4-3-5">
<title>4.3.5 Object Matching Verification</title>
<p>A verification step to check if a match is physically plausible, similar to <xref ref-type="bibr" rid="B21">Furrer et al. (2018</xref>) or <xref ref-type="bibr" rid="B5">Bauer et al. (2022</xref>), could help in some cases to reject inconsistent matches. However, this method of verification does not work in all cases; approaches often fail when objects are partially visible or symmetric. For example, a tube was detected at <italic>t</italic>
<sub>0</sub> (<italic>tube</italic>
<sub>0</sub>) and at <italic>t</italic>
<sub>1</sub> (<italic>tube</italic>
<sub>1</sub>), but was only half visible in the latter. The object matching and registration is ambiguous and resulted in aligning <italic>tube</italic>
<sub>1</sub> to the top of <italic>tube</italic>
<sub>0</sub>. Projecting <italic>tube</italic>
<sub>0</sub> into the <italic>t</italic>
<sub>1</sub>-recording resulted in half of the tube being below the supporting plane, which is physically implausible. This triggered a rejection even though the matched objects are the same. More generally, the fact that detected objects can actually be a heap poses challenges in geometric and semantic verification.</p>
</sec>
<sec id="s4-3-6">
<title>4.3.6 Reconstruction Quality</title>
<p>Clearly, the quality of the surface reconstruction and therefore the 3D object models impacts the performance of object matching applications. While <italic>ElasticFusion</italic> tends to create reconstructions with smoothed normals, it also connects spatially close objects with additional points having a continuous color gradient. While smooth reconstructions are appealing and useful for many applications, we would rather prefer sharp reconstructions, especially when it comes to processing steps such as matching or region growing which favor distinctive geometric features.</p>
</sec>
</sec>
</sec>
<sec id="s5">
<title>5 Conclusion</title>
<p>In this paper we tackled the core perception capabilities for open-world operation in the context of the object mapping task. This was defined as comparing the locations of objects in the present visit (present object map) to a previously stored reference map. Without loss of generality, we proposed to create a 3D reference map of the environment. Comparing complete room reconstructions is impractical for many application scenarios; therefore, we presented a concept where only local surfaces are reconstructed. This has the benefit that local comparisons are performed more rapidly and that these local planes are reconstructed more accurately. The latter is critical to enhance object detection and matching results. We showed that semantic segmentation methods are suitable to autonomously provide a high-level partitioning into relevant horizontal surfaces.</p>
<p>The key step of object mapping is to compare object detections and match objects from two different time instances. A main contribution of this work is the perception of all possible cases of static, moved, removed, and novel objects. To evaluate the novel approach, we compared against two baselines using state-of-the-art methods. For a quantitative evaluation we presented the <italic>ObChange</italic> dataset consisting of five different rooms, an extension of the dataset in <xref ref-type="bibr" rid="B31">Langer et al. (2020</xref>), for the object mapping task. Results indicated that the proposed method significantly improved over the baselines in terms of detected objects as well as the accuracy of categorization. Furthermore, we conducted real-world experiments with an autonomous mobile robot to demonstrate our developed capabilities in a realistic setting. The experiments and evaluation give clear hints on the remaining open challenges such as improving the reconstruction quality in terms of accuracy as well as completeness. Another open challenge is the detection of small objects and objects that are partially hidden, where in both cases multiple views and using cues that there might be another object and creating a close-up view might be a future solution to work on.</p>
<p>Future work is also needed to realize the application scenarios such as tidy up. While we focused on the key aspect of perceiving all object changes, object pose estimation needs to be further integrated with grasping. Extending the reference map to a knowledge base may help to keep track of where objects have been seen in the past and might be found when requested by the user (<xref ref-type="bibr" rid="B52">Tenorth and Beetz, 2013</xref>). Over time, the more changes happen, the better the environment is segmented and explained. Finally, given we can detect novel objects, it would be beneficial to merge partially detected objects into a complete 3D model as proposed in <xref ref-type="bibr" rid="B21">Furrer et al. (2018</xref>).</p>
</sec>
</body>
<back>
<sec id="s6">
<title>Data Availability Statement</title>
<p>The dataset generated for this study is available at <ext-link ext-link-type="uri" xlink:href="https://doi.org/10.48436/y3ggy-hxp10">https://doi.org/10.48436/y3ggy-hxp10</ext-link>.</p>
</sec>
<sec id="s7">
<title>Author Contributions</title>
<p>EL, TP, and MV conceived the idea of the presented work, contributed to the analysis of the results, and to the writing of the manuscript. EL implemented the system and conducted the experiments.</p>
</sec>
<sec id="s8">
<title>Funding</title>
<p>The research leading to these results has received funding from the Austrian Science Fund (FWF) under grant agreement Nos. I3969-N30 (InDex), I3967-N30 (BURG) and from the Austrian Research Promotion Agency (FFG) under grant agreement 879878 (K4R).</p>
</sec>
<sec sec-type="COI-statement" id="s9">
<title>Conflict of Interest</title>
<p>The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.</p>
</sec>
<sec sec-type="disclaimer" id="s10">
<title>Publisher&#x2019;s Note</title>
<p>All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors, and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.</p>
</sec>
<ack>
<p>We thank Markus Leitner for his help in setting up the proposed pipeline on the robot.</p>
</ack>
<fn-group>
<fn id="fn1">
<label>1</label>
<p>
<ext-link ext-link-type="uri" xlink:href="http://juxi.net/challenge/tidy-up-my-room/">http://juxi.net/challenge/tidy-up-my-room/</ext-link>
</p>
</fn>
</fn-group>
<ref-list>
<title>References</title>
<ref id="B1">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Alimi</surname>
<given-names>P.</given-names>
</name>
<name>
<surname>Meger</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Little</surname>
<given-names>J. J.</given-names>
</name>
</person-group> (<year>2012</year>). &#x201c;<article-title>Object Persistence in 3D for Home Robots</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE International Conference on Robotics and Automation: Workshop on Semantic Perception, Mapping and Exploration</conf-name>. </citation>
</ref>
<ref id="B2">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Ambrus</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>Bore</surname>
<given-names>N.</given-names>
</name>
<name>
<surname>Folkesson</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Jensfelt</surname>
<given-names>P.</given-names>
</name>
</person-group> (<year>2014</year>). &#x201c;<article-title>Meta-rooms: Building and Maintaining Long Term Spatial Models in a Dynamic World</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems</conf-name>, <fpage>1854</fpage>&#x2013;<lpage>1861</lpage>. <pub-id pub-id-type="doi">10.1109/iros.2014.6942806</pub-id> </citation>
</ref>
<ref id="B3">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Bajones</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Fischinger</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Weiss</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Wolf</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Vincze</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>de la Puente</surname>
<given-names>P.</given-names>
</name>
<etal/>
</person-group> (<year>2018</year>). <article-title>Hobbit: Providing Fall Detection and Prevention for the Elderly in the Real World</article-title>. <source>J. Robotics</source> <volume>2018</volume>, <fpage>1</fpage>&#x2013;<lpage>20</lpage>. <pub-id pub-id-type="doi">10.1155/2018/1754657</pub-id> </citation>
</ref>
<ref id="B4">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Batra</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Chang</surname>
<given-names>A. X.</given-names>
</name>
<name>
<surname>Chernova</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Davison</surname>
<given-names>A. J.</given-names>
</name>
<name>
<surname>Deng</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Koltun</surname>
<given-names>V.</given-names>
</name>
<etal/>
</person-group> (<year>2020</year>). <article-title>Rearrangement: A challenge for Embodied AI</article-title>. <source>arXiv preprint arXiv:2011.01975</source>. </citation>
</ref>
<ref id="B5">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Bauer</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Patten</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Vincze</surname>
<given-names>M.</given-names>
</name>
</person-group> (<year>2022</year>). &#x201c;<article-title>Sporeagent: Reinforced Scene-Level Plausibility for Object Pose Refinement</article-title>,&#x201d; in <conf-name>IEEE/CVF Winter Conference on Applications of Computer Vision</conf-name>, <fpage>654</fpage>&#x2013;<lpage>662</lpage>. <pub-id pub-id-type="doi">10.1109/wacv51458.2022.00027</pub-id> </citation>
</ref>
<ref id="B6">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Bj&#xf6;rkman</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Kragic</surname>
<given-names>D.</given-names>
</name>
</person-group> (<year>2010</year>). &#x201c;<article-title>Active 3D Scene Segmentation and Detection of Unknown Objects</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE International Conference on Robotics and Automation</conf-name>, <fpage>3114</fpage>&#x2013;<lpage>3120</lpage>. </citation>
</ref>
<ref id="B7">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Boccato</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Patten</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Vincze</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Ghidoni</surname>
<given-names>S.</given-names>
</name>
</person-group> (<year>2020</year>). &#x201c;<article-title>In the Depths of Hyponymy: A Step towards Lifelong Learning</article-title>,&#x201d; in <conf-name>Proceedings of the International Conference on Autonomic and Autonomous Systems</conf-name>, <fpage>103</fpage>&#x2013;<lpage>109</lpage>. </citation>
</ref>
<ref id="B8">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Bore</surname>
<given-names>N.</given-names>
</name>
<name>
<surname>Ekekrantz</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Jensfelt</surname>
<given-names>P.</given-names>
</name>
<name>
<surname>Folkesson</surname>
<given-names>J.</given-names>
</name>
</person-group> (<year>2018</year>). <article-title>Detection and Tracking of General Movable Objects in Large Three-Dimensional Maps</article-title>. <source>IEEE Trans. Robotics</source> <volume>35</volume>, <fpage>231</fpage>&#x2013;<lpage>247</lpage>. </citation>
</ref>
<ref id="B9">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Bugmann</surname>
<given-names>G.</given-names>
</name>
<name>
<surname>Copleston</surname>
<given-names>S. N.</given-names>
</name>
</person-group> (<year>2011</year>). &#x201c;<article-title>What Can a Personal Robot Do for You?</article-title>,&#x201d; in <conf-name>Proceedings of the Conference Towards Autonomous Robotic Systems</conf-name>, <fpage>360</fpage>&#x2013;<lpage>371</lpage>. <pub-id pub-id-type="doi">10.1007/978-3-642-23232-9_32</pub-id> </citation>
</ref>
<ref id="B10">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Cakmak</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Takayama</surname>
<given-names>L.</given-names>
</name>
</person-group> (<year>2013</year>). &#x201c;<article-title>Towards a Comprehensive Chore List for Domestic Robots</article-title>,&#x201d; in <conf-name>Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction</conf-name>, <fpage>93</fpage>&#x2013;<lpage>94</lpage>. <pub-id pub-id-type="doi">10.1109/hri.2013.6483517</pub-id> </citation>
</ref>
<ref id="B11">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Calli</surname>
<given-names>B.</given-names>
</name>
<name>
<surname>Singh</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Bruce</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Walsman</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Konolige</surname>
<given-names>K.</given-names>
</name>
<name>
<surname>Srinivasa</surname>
<given-names>S.</given-names>
</name>
<etal/>
</person-group> (<year>2017</year>). <article-title>Yale-cmu-berkeley Dataset for Robotic Manipulation Research</article-title>. <source>Int. J. Robotics Res.</source> <volume>36</volume>, <fpage>261</fpage>&#x2013;<lpage>268</lpage>. <pub-id pub-id-type="doi">10.1177/0278364917700714</pub-id> </citation>
</ref>
<ref id="B12">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Choi</surname>
<given-names>C.</given-names>
</name>
<name>
<surname>Christensen</surname>
<given-names>H. I.</given-names>
</name>
</person-group> (<year>2012</year>). &#x201c;<article-title>3D Pose Estimation of Daily Objects Using an RGB-D Camera</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems</conf-name>, <fpage>3342</fpage>&#x2013;<lpage>3349</lpage>. <pub-id pub-id-type="doi">10.1109/iros.2012.6386067</pub-id> </citation>
</ref>
<ref id="B13">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>de Oliveira</surname>
<given-names>F. D. B.</given-names>
</name>
<name>
<surname>da Silva</surname>
<given-names>M. R.</given-names>
</name>
<name>
<surname>Ara&#xfa;jo</surname>
<given-names>A. F. R.</given-names>
</name>
</person-group> (<year>2021</year>). <article-title>Spatio-temporal Data Association for Object-Augmented Mapping</article-title>. <source>J. Intell. Robot Syst.</source> <volume>103</volume>, <fpage>1</fpage>&#x2013;<lpage>19</lpage>. <pub-id pub-id-type="doi">10.1007/s10846-021-01445-8</pub-id> </citation>
</ref>
<ref id="B14">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Dhamija</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Gunther</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Ventura</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Boult</surname>
<given-names>T.</given-names>
</name>
</person-group> (<year>2020</year>). &#x201c;<article-title>The Overlooked Elephant of Object Detection: Open Set</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision</conf-name>, <fpage>1021</fpage>&#x2013;<lpage>1030</lpage>. <pub-id pub-id-type="doi">10.1109/wacv45572.2020.9093355</pub-id> </citation>
</ref>
<ref id="B15">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Drost</surname>
<given-names>B.</given-names>
</name>
<name>
<surname>Ulrich</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Navab</surname>
<given-names>N.</given-names>
</name>
<name>
<surname>Ilic</surname>
<given-names>S.</given-names>
</name>
</person-group> (<year>2010</year>). &#x201c;<article-title>Model Globally, Match Locally: Efficient and Robust 3D Object Recognition</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition</conf-name>, <fpage>998</fpage>&#x2013;<lpage>1005</lpage>. <pub-id pub-id-type="doi">10.1109/cvpr.2010.5540108</pub-id> </citation>
</ref>
<ref id="B16">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Edmonds</surname>
<given-names>J.</given-names>
</name>
</person-group> (<year>1965</year>). <article-title>Maximum Matching and a Polyhedron with 0, 1-vertices</article-title>. <source>J. Res. Natl. Bur. Stand. B</source> <volume>69</volume>, <fpage>55</fpage>&#x2013;<lpage>56</lpage>. <pub-id pub-id-type="doi">10.6028/jres.069b.013</pub-id> </citation>
</ref>
<ref id="B17">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Fehr</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Furrer</surname>
<given-names>F.</given-names>
</name>
<name>
<surname>Dryanovski</surname>
<given-names>I.</given-names>
</name>
<name>
<surname>Sturm</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Gilitschenski</surname>
<given-names>I.</given-names>
</name>
<name>
<surname>Siegwart</surname>
<given-names>R.</given-names>
</name>
<etal/>
</person-group> (<year>2017</year>). &#x201c;<article-title>TSDF-based Change Detection for Consistent Long-Term Dense Reconstruction and Dynamic Object Discovery</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE International Conference on Robotics and Automation</conf-name>, <fpage>5237</fpage>&#x2013;<lpage>5244</lpage>. <pub-id pub-id-type="doi">10.1109/icra.2017.7989614</pub-id> </citation>
</ref>
<ref id="B18">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Finman</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>Whelan</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Kaess</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Leonard</surname>
<given-names>J. J.</given-names>
</name>
</person-group> (<year>2013</year>). &#x201c;<article-title>Toward Lifelong Object Segmentation from Change Detection in Dense RGB-D Maps</article-title>,&#x201d; in <conf-name>Proceedings of the European Conference on Mobile Robots</conf-name>, <fpage>178</fpage>&#x2013;<lpage>185</lpage>. <pub-id pub-id-type="doi">10.1109/ecmr.2013.6698839</pub-id> </citation>
</ref>
<ref id="B19">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Fischler</surname>
<given-names>M. A.</given-names>
</name>
<name>
<surname>Bolles</surname>
<given-names>R. C.</given-names>
</name>
</person-group> (<year>1981</year>). <article-title>Random Sample Consensus</article-title>. <source>Commun. ACM</source> <volume>24</volume>, <fpage>381</fpage>&#x2013;<lpage>395</lpage>. <pub-id pub-id-type="doi">10.1145/358669.358692</pub-id> </citation>
</ref>
<ref id="B20">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>French</surname>
<given-names>R.</given-names>
</name>
</person-group> (<year>1999</year>). <article-title>Catastrophic Forgetting in Connectionist Networks</article-title>. <source>Trends Cognitive Sciences</source> <volume>3</volume>, <fpage>128</fpage>&#x2013;<lpage>135</lpage>. <pub-id pub-id-type="doi">10.1016/s1364-6613(99)01294-2</pub-id> </citation>
</ref>
<ref id="B21">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Furrer</surname>
<given-names>F.</given-names>
</name>
<name>
<surname>Novkovic</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Fehr</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Gawel</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Grinvald</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Sattler</surname>
<given-names>T.</given-names>
</name>
<etal/>
</person-group> (<year>2018</year>). &#x201c;<article-title>Incremental Object Database: Building 3D Models from Multiple Partial Observations</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems</conf-name> (<publisher-name>IEEE</publisher-name>), <fpage>6835</fpage>&#x2013;<lpage>6842</lpage>. <pub-id pub-id-type="doi">10.1109/iros.2018.8594391</pub-id> </citation>
</ref>
<ref id="B22">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Furukawa</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Suzuki</surname>
<given-names>K.</given-names>
</name>
<name>
<surname>Hamaguchi</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>Onishi</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Sakurada</surname>
<given-names>K.</given-names>
</name>
</person-group> (<year>2020</year>). &#x201c;<article-title>Self-supervised Simultaneous Alignment and Change Detection</article-title>,&#x201d; in <conf-name>Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems</conf-name>, <fpage>6025</fpage>&#x2013;<lpage>6031</lpage>. <pub-id pub-id-type="doi">10.1109/iros45743.2020.9340840</pub-id> </citation>
</ref>
<ref id="B23">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Graham</surname>
<given-names>B.</given-names>
</name>
<name>
<surname>Engelcke</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>van der Maaten</surname>
<given-names>L.</given-names>
</name>
</person-group> (<year>2018</year>). &#x201c;<article-title>3D Semantic Segmentation with Submanifold Sparse Convolutional Networks</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE/CVF International Conference on Computer Vision</conf-name>, <fpage>9224</fpage>&#x2013;<lpage>9232</lpage>. <pub-id pub-id-type="doi">10.1109/cvpr.2018.00961</pub-id> </citation>
</ref>
<ref id="B24">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Grinvald</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Furrer</surname>
<given-names>F.</given-names>
</name>
<name>
<surname>Novkovic</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Chung</surname>
<given-names>J. J.</given-names>
</name>
<name>
<surname>Cadena</surname>
<given-names>C.</given-names>
</name>
<name>
<surname>Siegwart</surname>
<given-names>R.</given-names>
</name>
<etal/>
</person-group> (<year>2019</year>). <article-title>Volumetric Instance-Aware Semantic Mapping and 3d Object Discovery</article-title>. <source>IEEE Robot. Autom. Lett.</source> <volume>4</volume>, <fpage>3037</fpage>&#x2013;<lpage>3044</lpage>. <pub-id pub-id-type="doi">10.1109/lra.2019.2923960</pub-id> </citation>
</ref>
<ref id="B25">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>He</surname>
<given-names>K.</given-names>
</name>
<name>
<surname>Gkioxari</surname>
<given-names>G.</given-names>
</name>
<name>
<surname>Doll&#xe1;r</surname>
<given-names>P.</given-names>
</name>
<name>
<surname>Girshick</surname>
<given-names>R.</given-names>
</name>
</person-group> (<year>2017</year>). &#x201c;<article-title>Mask R-CNN</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE International Conference on Computer Vision</conf-name>, <fpage>2961</fpage>&#x2013;<lpage>2969</lpage>. <pub-id pub-id-type="doi">10.1109/iccv.2017.322</pub-id> </citation>
</ref>
<ref id="B26">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Hoda&#x148;</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Michel</surname>
<given-names>F.</given-names>
</name>
<name>
<surname>Brachmann</surname>
<given-names>E.</given-names>
</name>
<name>
<surname>Kehl</surname>
<given-names>W.</given-names>
</name>
<name>
<surname>Glent Buch</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Kraft</surname>
<given-names>D.</given-names>
</name>
<etal/>
</person-group> (<year>2018</year>). &#x201c;<article-title>BOP: Benchmark for 6D Object Pose Estimation</article-title>,&#x201d; in <conf-name>European Conference on Computer Vision (ECCV)</conf-name>. </citation>
</ref>
<ref id="B27">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Houseago</surname>
<given-names>C.</given-names>
</name>
<name>
<surname>Bloesch</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Leutenegger</surname>
<given-names>S.</given-names>
</name>
</person-group> (<year>2019</year>). &#x201c;<article-title>KO-fusion: Dense Visual SLAM with Tightly-Coupled Kinematic and Odometric Tracking</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)</conf-name>, <fpage>4054</fpage>&#x2013;<lpage>4060</lpage>. <pub-id pub-id-type="doi">10.1109/icra.2019.8793471</pub-id> </citation>
</ref>
<ref id="B28">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Joseph</surname>
<given-names>K. J.</given-names>
</name>
<name>
<surname>Khan</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Khan</surname>
<given-names>F. S.</given-names>
</name>
<name>
<surname>Balasubramanian</surname>
<given-names>V. N.</given-names>
</name>
</person-group> (<year>2021</year>). &#x201c;<article-title>Towards Open World Object Detection</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition</conf-name>. <pub-id pub-id-type="doi">10.1109/cvpr46437.2021.00577</pub-id> </citation>
</ref>
<ref id="B29">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Kim</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Lin</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Angelova</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Kweon</surname>
<given-names>I. S.</given-names>
</name>
<name>
<surname>Kuo</surname>
<given-names>W.</given-names>
</name>
</person-group> (<year>2021</year>). <article-title>Learning Open-World Object Proposals without Learning to Classify</article-title>. <source>CoRR abs/2108.06753</source>. </citation>
</ref>
<ref id="B30">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Kirkpatrick</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Pascanu</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>Rabinowitz</surname>
<given-names>N.</given-names>
</name>
<name>
<surname>Veness</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Desjardins</surname>
<given-names>G.</given-names>
</name>
<name>
<surname>Rusu</surname>
<given-names>A. A.</given-names>
</name>
<etal/>
</person-group> (<year>2017</year>). <article-title>Overcoming Catastrophic Forgetting in Neural Networks</article-title>. <source>Proc. Natl. Acad. Sci. USA</source> <volume>114</volume>, <fpage>3521</fpage>&#x2013;<lpage>3526</lpage>. <pub-id pub-id-type="doi">10.1073/pnas.1611835114</pub-id> </citation>
</ref>
<ref id="B31">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Langer</surname>
<given-names>E.</given-names>
</name>
<name>
<surname>Patten</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Vincze</surname>
<given-names>M.</given-names>
</name>
</person-group> (<year>2020</year>). &#x201c;<article-title>Robust and Efficient Object Change Detection by Combining Global Semantic Information and Local Geometric Verification</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems</conf-name>, <fpage>8453</fpage>&#x2013;<lpage>8460</lpage>. <pub-id pub-id-type="doi">10.1109/iros45743.2020.9341664</pub-id> </citation>
</ref>
<ref id="B32">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Langer</surname>
<given-names>E.</given-names>
</name>
<name>
<surname>Ridder</surname>
<given-names>B.</given-names>
</name>
<name>
<surname>Cashmore</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Magazzeni</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Zillich</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Vincze</surname>
<given-names>M.</given-names>
</name>
</person-group> (<year>2017</year>). &#x201c;<article-title>On-the-fly Detection of Novel Objects in Indoor Environments</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE International Conference on Robotics and Biomimetics</conf-name>, <fpage>900</fpage>&#x2013;<lpage>907</lpage>. <pub-id pub-id-type="doi">10.1109/robio.2017.8324532</pub-id> </citation>
</ref>
<ref id="B33">
<citation citation-type="book">
<person-group person-group-type="author">
<name>
<surname>Lin</surname>
<given-names>T.-Y.</given-names>
</name>
<name>
<surname>Maire</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Belongie</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Hays</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Perona</surname>
<given-names>P.</given-names>
</name>
<name>
<surname>Ramanan</surname>
<given-names>D.</given-names>
</name>
<etal/>
</person-group> (<year>2014</year>). &#x201c;<article-title>Microsoft COCO: Common Objects in Context</article-title>,&#x201d; in <source>Proceedings of the ECCV</source> (<publisher-loc>Berlin, Germany</publisher-loc>: <publisher-name>Springer</publisher-name>), <fpage>740</fpage>&#x2013;<lpage>755</lpage>. <pub-id pub-id-type="doi">10.1007/978-3-319-10602-1_48</pub-id> </citation>
</ref>
<ref id="B34">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Liu</surname>
<given-names>Z.</given-names>
</name>
<name>
<surname>Miao</surname>
<given-names>Z.</given-names>
</name>
<name>
<surname>Zhan</surname>
<given-names>X.</given-names>
</name>
<name>
<surname>Wang</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Gong</surname>
<given-names>B.</given-names>
</name>
<name>
<surname>Yu</surname>
<given-names>S. X.</given-names>
</name>
</person-group> (<year>2019</year>). &#x201c;<article-title>Large-scale Long-Tailed Recognition in an Open World</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition</conf-name>, <fpage>2537</fpage>&#x2013;<lpage>2546</lpage>. <pub-id pub-id-type="doi">10.1109/cvpr.2019.00264</pub-id> </citation>
</ref>
<ref id="B35">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Luo</surname>
<given-names>M. R.</given-names>
</name>
<name>
<surname>Cui</surname>
<given-names>G.</given-names>
</name>
<name>
<surname>Rigg</surname>
<given-names>B.</given-names>
</name>
</person-group> (<year>2001</year>). <article-title>The Development of the CIE 2000 Colour-Difference Formula: CIEDE2000</article-title>. <source>Color Res. Appl.</source> <volume>26</volume>, <fpage>340</fpage>&#x2013;<lpage>350</lpage>. <pub-id pub-id-type="doi">10.1002/col.1049</pub-id> </citation>
</ref>
<ref id="B36">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Marton</surname>
<given-names>Z.-C.</given-names>
</name>
<name>
<surname>Pangercic</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Blodow</surname>
<given-names>N.</given-names>
</name>
<name>
<surname>Kleinehellefort</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Beetz</surname>
<given-names>M.</given-names>
</name>
</person-group> (<year>2010</year>). &#x201c;<article-title>General 3D Modelling of Novel Objects from a Single View</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems</conf-name>, <fpage>3700</fpage>&#x2013;<lpage>3705</lpage>. <pub-id pub-id-type="doi">10.1109/iros.2010.5650434</pub-id> </citation>
</ref>
<ref id="B37">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Mason</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Marthi</surname>
<given-names>B.</given-names>
</name>
</person-group> (<year>2012</year>). &#x201c;<article-title>An Object-Based Semantic World Model for Long-Term Change Detection and Semantic Querying</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems</conf-name>, <fpage>3851</fpage>&#x2013;<lpage>3858</lpage>. <pub-id pub-id-type="doi">10.1109/iros.2012.6385729</pub-id> </citation>
</ref>
<ref id="B38">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Miller</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Dayoub</surname>
<given-names>F.</given-names>
</name>
<name>
<surname>Milford</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>S&#xfc;nderhauf</surname>
<given-names>N.</given-names>
</name>
</person-group> (<year>2019</year>). &#x201c;<article-title>Evaluating Merging Strategies for Sampling-Based Uncertainty Techniques in Object Detection</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE International Conference on Robotics and Automation</conf-name>, <fpage>2348</fpage>&#x2013;<lpage>2354</lpage>. <pub-id pub-id-type="doi">10.1109/icra.2019.8793821</pub-id> </citation>
</ref>
<ref id="B39">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Miller</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Nicholson</surname>
<given-names>L.</given-names>
</name>
<name>
<surname>Dayoub</surname>
<given-names>F.</given-names>
</name>
<name>
<surname>S&#xfc;nderhauf</surname>
<given-names>N.</given-names>
</name>
</person-group> (<year>2018</year>). &#x201c;<article-title>Dropout Sampling for Robust Object Detection in Open-Set Conditions</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE International Conference on Robotics and Automation</conf-name>, <fpage>3243</fpage>&#x2013;<lpage>3249</lpage>. <pub-id pub-id-type="doi">10.1109/icra.2018.8460700</pub-id> </citation>
</ref>
<ref id="B40">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Okada</surname>
<given-names>H.</given-names>
</name>
<name>
<surname>Inamura</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Wada</surname>
<given-names>K.</given-names>
</name>
</person-group> (<year>2019</year>). <article-title>What Competitions Were Conducted in the Service Categories of the World Robot Summit?</article-title> <source>Adv. Robotics</source> <volume>33</volume>, <fpage>900</fpage>&#x2013;<lpage>910</lpage>. <pub-id pub-id-type="doi">10.1080/01691864.2019.1663608</pub-id> </citation>
</ref>
<ref id="B41">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Oleynikova</surname>
<given-names>H.</given-names>
</name>
<name>
<surname>Taylor</surname>
<given-names>Z.</given-names>
</name>
<name>
<surname>Fehr</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Siegwart</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>Nieto</surname>
<given-names>J.</given-names>
</name>
</person-group> (<year>2017</year>). &#x201c;<article-title>Voxblox: Incremental 3D Euclidean Signed Distance fields for On-Board MAV Planning</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems</conf-name>, <fpage>1366</fpage>&#x2013;<lpage>1373</lpage>. <pub-id pub-id-type="doi">10.1109/iros.2017.8202315</pub-id> </citation>
</ref>
<ref id="B42">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Park</surname>
<given-names>J.-M.</given-names>
</name>
<name>
<surname>Jang</surname>
<given-names>J.-h.</given-names>
</name>
<name>
<surname>Yoo</surname>
<given-names>S.-M.</given-names>
</name>
<name>
<surname>Lee</surname>
<given-names>S.-K.</given-names>
</name>
<name>
<surname>Kim</surname>
<given-names>U.-h.</given-names>
</name>
<name>
<surname>Kim</surname>
<given-names>J.-H.</given-names>
</name>
</person-group> (<year>2021</year>). &#x201c;<article-title>ChangeSim: towards End-To-End Online Scene Change Detection in Industrial Indoor Environments</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems</conf-name>, <fpage>8555</fpage>&#x2013;<lpage>8562</lpage>. <pub-id pub-id-type="doi">10.1109/iros51168.2021.9636350</pub-id> </citation>
</ref>
<ref id="B43">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Park</surname>
<given-names>K.</given-names>
</name>
<name>
<surname>Patten</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Vincze</surname>
<given-names>M.</given-names>
</name>
</person-group> (<year>2019</year>). &#x201c;<article-title>Pix2Pose: Pixel-wise Coordinate Regression of Objects for 6D Pose Estimation</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE/CVF International Conference on Computer Vision</conf-name>, <fpage>7668</fpage>&#x2013;<lpage>7677</lpage>. <pub-id pub-id-type="doi">10.1109/iccv.2019.00776</pub-id> </citation>
</ref>
<ref id="B44">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Perera</surname>
<given-names>P.</given-names>
</name>
<name>
<surname>Morariu</surname>
<given-names>V. I.</given-names>
</name>
<name>
<surname>Jain</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>Manjunatha</surname>
<given-names>V.</given-names>
</name>
<name>
<surname>Wigington</surname>
<given-names>C.</given-names>
</name>
<name>
<surname>Ordonez</surname>
<given-names>V.</given-names>
</name>
<etal/>
</person-group> (<year>2020</year>). &#x201c;<article-title>Generative-discriminative Feature Representations for Open-Set Recognition</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition</conf-name>, <fpage>11814</fpage>&#x2013;<lpage>11823</lpage>. <pub-id pub-id-type="doi">10.1109/cvpr42600.2020.01183</pub-id> </citation>
</ref>
<ref id="B45">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Pidhorskyi</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Almohsen</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>Adjeroh</surname>
<given-names>D. A.</given-names>
</name>
<name>
<surname>Doretto</surname>
<given-names>G.</given-names>
</name>
</person-group> (<year>2018</year>). <article-title>Generative Probabilistic Novelty Detection with Adversarial Autoencoders</article-title>. <source>arXiv preprint arXiv:1807.02588</source>. </citation>
</ref>
<ref id="B46">
<citation citation-type="book">
<person-group person-group-type="author">
<name>
<surname>Qiu</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Yang</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Wang</surname>
<given-names>X.</given-names>
</name>
<name>
<surname>Tao</surname>
<given-names>D.</given-names>
</name>
</person-group> (<year>2020</year>). &#x201c;<article-title>Hallucinating Visual Instances in Total Absentia</article-title>,&#x201d; in <source>Proceeding of the ECCV</source> (<publisher-loc>Berlin, Germany</publisher-loc>: <publisher-name>Springer</publisher-name>), <fpage>264</fpage>&#x2013;<lpage>282</lpage>. <pub-id pub-id-type="doi">10.1007/978-3-030-58558-7_16</pub-id> </citation>
</ref>
<ref id="B47">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Quigley</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Conley</surname>
<given-names>K.</given-names>
</name>
<name>
<surname>Gerkey</surname>
<given-names>B.</given-names>
</name>
<name>
<surname>Faust</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Foote</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Leibs</surname>
<given-names>J.</given-names>
</name>
<etal/>
</person-group> (<year>2009</year>). &#x201c;<article-title>ROS: an Open-Source Robot Operating System</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE International Conference on Robotics and Automation, Workshop on Open Source Software</conf-name>, <conf-loc>Kobe, Japan</conf-loc>, <fpage>5</fpage>. <comment>Vol. 3</comment>. </citation>
</ref>
<ref id="B48">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Redmon</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Farhadi</surname>
<given-names>A.</given-names>
</name>
</person-group> (<year>2018</year>). <article-title>Yolov3: An Incremental Improvement</article-title>. <source>arXiv preprint arXiv:1804.02767</source>. </citation>
</ref>
<ref id="B49">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Sakurada</surname>
<given-names>K.</given-names>
</name>
<name>
<surname>Shibuya</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Wang</surname>
<given-names>W.</given-names>
</name>
</person-group> (<year>2020</year>). &#x201c;<article-title>Weakly Supervised Silhouette-Based Semantic Scene Change Detection</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)</conf-name>, <fpage>6861</fpage>&#x2013;<lpage>6867</lpage>. <pub-id pub-id-type="doi">10.1109/icra40945.2020.9196985</pub-id> </citation>
</ref>
<ref id="B50">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Song</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Zhang</surname>
<given-names>L.</given-names>
</name>
<name>
<surname>Xiao</surname>
<given-names>J.</given-names>
</name>
</person-group> (<year>2015</year>). <article-title>Robot in a Room: Toward Perfect Object Recognition in Closed Environments</article-title>. <source>CoRR, abs/1507.02703</source>. </citation>
</ref>
<ref id="B51">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Tateno</surname>
<given-names>K.</given-names>
</name>
<name>
<surname>Tombari</surname>
<given-names>F.</given-names>
</name>
<name>
<surname>Navab</surname>
<given-names>N.</given-names>
</name>
</person-group> (<year>2015</year>). &#x201c;<article-title>Real-time and Scalable Incremental Segmentation on Dense Slam</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems</conf-name> (<publisher-loc>Hamburg, Germany</publisher-loc>: <publisher-name>IEEE</publisher-name>), <fpage>4465</fpage>&#x2013;<lpage>4472</lpage>. <pub-id pub-id-type="doi">10.1109/iros.2015.7354011</pub-id> </citation>
</ref>
<ref id="B52">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Tenorth</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Beetz</surname>
<given-names>M.</given-names>
</name>
</person-group> (<year>2013</year>). <article-title>KnowRob: A Knowledge Processing Infrastructure for Cognition-Enabled Robots</article-title>. <source>Int. J. Robotics Res.</source> <volume>32</volume>, <fpage>566</fpage>&#x2013;<lpage>590</lpage>. <pub-id pub-id-type="doi">10.1177/0278364913481635</pub-id> </citation>
</ref>
<ref id="B53">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Wald</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Avetisyan</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Navab</surname>
<given-names>N.</given-names>
</name>
<name>
<surname>Tombari</surname>
<given-names>F.</given-names>
</name>
<name>
<surname>Nie&#xdf;ner</surname>
<given-names>M.</given-names>
</name>
</person-group> (<year>2019</year>). &#x201c;<article-title>RIO: 3D Object Instance Re-localization in Changing Indoor Environments</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE/CVF International Conference on Computer Vision</conf-name>, <fpage>7658</fpage>&#x2013;<lpage>7667</lpage>. <pub-id pub-id-type="doi">10.1109/iccv.2019.00775</pub-id> </citation>
</ref>
<ref id="B54">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Weihs</surname>
<given-names>L.</given-names>
</name>
<name>
<surname>Deitke</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Kembhavi</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Mottaghi</surname>
<given-names>R.</given-names>
</name>
</person-group> (<year>2021</year>). &#x201c;<article-title>Visual Room Rearrangement</article-title>,&#x201d; in <conf-name>Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition</conf-name>, <fpage>5922</fpage>&#x2013;<lpage>5931</lpage>. <pub-id pub-id-type="doi">10.1109/cvpr46437.2021.00586</pub-id> </citation>
</ref>
<ref id="B55">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Whelan</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Leutenegger</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Salas-Moreno</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>Glocker</surname>
<given-names>B.</given-names>
</name>
<name>
<surname>Davison</surname>
<given-names>A.</given-names>
</name>
</person-group> (<year>2015</year>). &#x201c;<article-title>ElasticFusion: Dense SLAM without a Pose Graph</article-title>,&#x201d; in <conf-name>Proceedings of Robotics: Science and Systems</conf-name>. <pub-id pub-id-type="doi">10.15607/rss.2015.xi.001</pub-id> </citation>
</ref>
<ref id="B56">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Xiang</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Schmidt</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Narayanan</surname>
<given-names>V.</given-names>
</name>
<name>
<surname>Fox</surname>
<given-names>D.</given-names>
</name>
</person-group> (<year>2017</year>). <article-title>PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes</article-title>. <source>arXiv preprint arXiv:1711.00199</source>. </citation>
</ref>
<ref id="B57">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Yamamoto</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Terada</surname>
<given-names>K.</given-names>
</name>
<name>
<surname>Ochiai</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Saito</surname>
<given-names>F.</given-names>
</name>
<name>
<surname>Asahara</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Murase</surname>
<given-names>K.</given-names>
</name>
</person-group> (<year>2019</year>). <article-title>Development of Human Support Robot as the Research Platform of a Domestic Mobile Manipulator</article-title>. <source>ROBOMECH J.</source> <volume>6</volume>, <fpage>1</fpage>&#x2013;<lpage>15</lpage>. <pub-id pub-id-type="doi">10.1186/s40648-019-0132-3</pub-id> </citation>
</ref>
</ref-list>
</back>
</article>