Abstract
The 4th industrial revolution started in 2016 and referred to a new phase in the industrial revolution. One of the most significant technological evolvements during the 4th industrial revolution is Augmented Reality (AR) technology. AR superimposes interactional virtual objects/images to real environments. Because of the interaction and see-through characteristics, AR is better applied to engineering than Virtual Reality (VR). The application of AR in civil infrastructure can avoid artificial mistakes, improve efficiency, and saves budget. This article reviews AR applications in civil infrastructure, focusing on research studies in the latest five years (2016–2020) and their milestone developments. More than half of the AR research and implementation studies have focused on the construction domain in the last five years. Researchers deploy AR technologies in on-site construction to assist in discrepancy checking, collaborative communication, and safety checking. AR also uses building information models (BIMs) to produce detailed 3D structural information for visualization. Additionally, AR has been studied for structural health monitoring (SHM), routine and damage detection, energy performance assessment, crack inspection, excavation, and underground utility maintenance. Finally, AR has also been applied for architecture design, city plan, and disaster prediction as an essential part of smart city service. This article discusses the challenges of AR implementation in civil infrastructure and recommends future applications.
Introduction
The 4th industrial revolution is generally believed to start in 2016 (More Ickson and ben Dhaou, 2019). However, the exact starting point is still debatable because it is still an ongoing event. Different countries also tend to have different definitions of the starting time. For instance, German is widely accepted as one of the first countries entering the 4th industrial revolution since 2011 (Pitsis et al., 2020).
Caudell and Mizell proposed the official term “'Augmented Reality” in 1990 (). Milgram and Kishino (1994a) defined the concept of “Virtuality Continuum” and clarified the concept of Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), and Augmented Virtuality (AV). Figure 1 shows the classification of these four terms. The real environment is the physical surroundings that the user stands in. The virtual environment is computer-generated objects/information made up of laser/light. With VR, a user completely immerses in an entirely virtual environment and sees no more real objects in the surroundings (; Zyda, 2005); while with MR, a user sees both objects in the real surroundings and virtual images (Milgram and Kishino, 1994b; Ohta and Tamura, 2014; Speicher et al., 2019). MR composes of AR and AV. AR displays the real environment enhanced by interactive virtual objects (Milgram and Kishino, 1994a; ). AV is to project and control real objects in a virtual world (Ternier et al., 2012; Nahon et al., 2015).
FIGURE 1
Figure 2 shows the trend of AR implementation in civil infrastructure. The data in Figure 2 are from the number of research articles on the Web of Science (2020). In the last five years (2016–2020), AR has undergone rapid development and implementation in civil infrastructure, compared with the early years (Petrillo et al., 2018; ; ; Thompson, 2019). A general and comprehensive overview of the AR implementations in civil infrastructure is needed to invoke more profound AR applications. This article provides a timely and general review of recent AR studies in civil infrastructure and a new application classification.
FIGURE 2
Figure 3 shows the distribution of the existed publications in different countries/regions (same data with Figure 2.). 40% of the studies are from the United States and China, 24 and 16%, respectively. Figure 3 shows that the United States and China are leading the 4th industrial revolution in the AR area (More Ickson and ben Dhaou, 2019; Pitsis et al., 2020). This conclusion is consistent with recent research studies (Petrillo et al., 2018; ). claimed that the United States and China had introduced initiatives albeit with different objectives to support the modernization of the industrial production sector. Petrillo et al. (2018) stated that the economic opportunities of Industry 4.0 are vast and affect the entire economies and countries, mainly including the United States, China, Germany, and the United Kingdom by 2030.
FIGURE 3
There are two kinds of classification methods for AR implementation in civil infrastructure (Wang et al., 2013): by subjects (building structures, bridges, and railways) or by application scenarios (architecture design, structural design, construction, SHM, and others) (Shin and Dunston, 2008). This article deploys the classification of different AR technologies by application scenarios. Figure 4 shows the percentage of AR applications in civil infrastructure between 2016 and 2020, divided into five significant branches: construction, building information model (BIM), structural health monitoring (SHM) and damage detection, underground utilities, and smart city management. The percentage is calculated by dividing the corresponding category's publication amount in the last five years by the same period's total publication number. The statistics ended on December 31st, 2020. The total publication number in this period is 95. Nearly half of AR implementations in civil infrastructure have focused on the on-site construction (45%), followed by SHM and damage detection (20%), BIM (19%), underground (10%), and smart city management (6%). A comprehensive review of AR studies in civil infrastructure education can be found in , but it was not relevant as a topic for this article.
FIGURE 4
Researchers should consider comprehensively when selecting the suitable AR platform (Julier et al., 2000; Jahn et al., 2018; Mahmood et al., 2018). There are two major categories of AR systems in general. One is the prototypes developed in labs, and the other one is the commercial AR devices in the market. The prototypes are typically designed for solving specific research problems efficiently but typically lack versatility. Unless a researcher decides to design a particular AR system for a specific research project, a suitable commercial AR device is usually a better choice (Roitman et al., 2017; Wang and Dunston, 2013). When selecting an AR device, all the parameters are classified into four major categories, general properties, sensors, computational capacities, and display capacities. The general properties include weight, price, and product durability (e.g., water resistance, working temperature range, working humidity range, and drop safe distance). Sensors include head-tracking, eye-tracking, depth sensor, GAM (gyroscope, accelerometer, and magnetometer), and GPS sensor. Researchers should also consider processing units, RAM, and storage to fulfill the computational requirements. Connectivity (e.g., Wi-Fi, Bluetooth, and USB) and display capacities (e.g., FoV and camera) are also important considerations. The choice of a suitable AR device is a complect work. Xu et al. (2020) summarized all the parameters and compared sixteen AR devices regarding these parameters. Xu and Doyle et al. (2020) also provide an empirical formula for civil engineers to select a suitable AR device. applied the AR device selection procedure and connected AR with sensors in a shaking table experiment. Hossain et al. (2020) used a similar selection procedure and conducted AR modal shape analysis.
In the following sections, the authors reviewed the milestone AR implementations in construction, BIM, SHM and damage detection, underground utilities, and smart city management, respectively. Following the review, this article discusses the current progress and technical challenges of AR implementation in civil infrastructure. Finally, this article concludes and predicts the future AR development direction in civil infrastructure.
Construction
One of the salient features of AR technology is the combination of virtual and real objects. Traditionally, VR was deployed for the construction process visualization (Sacks et al., 2013; Zhao and Lucas, 2015). The pith of VR implementation in construction is the detailed 3D-CAD modeling of both the project and the surroundings (Sampaio and Martins, 2014). Detailed environment information input is also required for VR implementations. However, the requirements of detailed models become impractical with the increasing complexity of the construction procedure (). With AR technology, the virtual design blueprint and real construction site can be combined and illustrated to engineers and workers in real time, resulting in higher precision and efficiency (). AR does not require detailed modeling of the surroundings or the existing project process. Applying AR in the lifecycle construction site planning can help keep the project within budget and to avoid process mistakes or complement time.
In this study, the authors classified all the AR implementations in construction nowadays into two major research areas to simplify the acknowledgment for the readers, as shown in Figure 5. The two major research areas are monitoring project progress and assisting workers. Studies of project progress monitoring can be divided into two categories, discrepancy checks and quality control. The second research area, the assistance for workers, includes collaborative visualization, telecommunication, and site safety.
FIGURE 5
Monitoring Project Progress
The significant practical construction challenges of the AR applications are the complicated and arduous outdoor environment, for example, lighting conditions and terrain variation. The monitoring project progress research has been conducted for both hardware and software development.
Discrepancy Check
A primary AR application in construction is the discrepancy check (Son et al., 2015), that is, checking the difference between on-site structures and design models. The first systematic AR implementation for discrepancy check is MARSIFT (). However, MARSIFT is a promising concept rather than a viable application. Since the development of MARSIFT, AR implementation in civil construction has been increasing, both in theoretic studies (pose estimation, scene understanding (), and accurate 3D reconstruction of the real environment ()) and on-site applications. There are mainly two practical AR implementations in the discrepancy check area, SMART-ARMOR and D4AR.
SMART-ARMOR was developed based on UM-AR-GPS-ROVER (). UM-AR-GPS-ROVER’s ability to display virtual mechanicals in an outdoor construction site has been verified with a field experiment. In 2008, ; ; presented ARVISCOPE-ROVER as an updated version of UM-AR-GPS-ROVER. The Global Positioning System (GPS) and orientation tracking technologies were applied in ARVISCOPE to create a dynamic simulation of the models' continuous animations. In the following years, ; Kamat et al., 2011 evolved the ARVISCOPE-ROVER system to the more accurate and lighter SMART-ARMOR system. SMART-ARMOR extended the system to a more standard AR development environment by separating AR logic from the application logic part. SMART-ARMOR is a milestone holistic and applicable outdoor construction application of AR technologies.
D4AR was developed by ; ; ; , applying the registration method to solve the discrepancy check problem. Later, D4AR was updated to the HD4AR system (), enabling the application of available camera-equipped mobile devices. In 2014, bottomed on the D4AR system, manual occlusion management, and “X-ray” metaphors (Zollmann et al., 2012; Karsch et al. (2014) combined a set of AR tools to build a valuable platform, ConstructAide, to monitor project progress. ConstructAide combined unordered photo collections and 3D models to obtain construction progress.
Along with the theoretical development, researchers also conducted field experiments. For instance, by solving the alignment and plumbness problem, the discrepancy check of steel pipe modules assembly was solved by Nahangi, 2015; Nahangi et al., 2015 and Jeanclos et al., 2018; Kwiatek et al., 2018; Kwiatek et al., 2019; Li et al., 2018, capable of extending more general applications.
Quality Control
Quality control is another critical AR application in construction. Zhou et al. (2017) inspected the on-site segment displacement during tunneling construction compared with the baseline model to monitor construction quality. connected AR devices and sensors, taking advantage of both the displaying function of AR technology and sensors' inspecting function. Figure 6 shows the design procedure of connecting AR devices and sensors. Figure 7 shows an illustrative experiment that conducted to verify the AR-sensor connection.
FIGURE 6
FIGURE 7
Assisting Workers
High-efficiency collaborative visualization and communication among multiple users can be realized by applying AR technologies.
Collaborative Visualization
developed an AR application, ARVita, for the collaborative visualization of multiple users wearing HMDs. Using ARVita, all the uses can interact with the dynamic visuals of the real-time construction processes. Soman et al. (2017) developed an application to share AR images between designers and the field site. developed a prototype distributed AR visualization collaborative system, DARCC, which was tested in a bridge deck rehabilitation project to show the effectiveness.
Telecommunication
Telecommunication can provide high-performance guidance for workers. In 2003, invented an AR helmet to provide an online welding assistant to improve welders working conditions. designed an AR interface to teleoperate remote cranes by retrieving information from the field and enhancing the decision-making processes.
Site Safety
Another significant application is to guarantee site safety by visualizing complex workplace situations with AR devices. (Li et al., 2018). Kim et al. (2017) developed an AR system to provide workers safety information to avoid hazards. Liu et al., 2018 deployed AR technologies to give the workers alarms of dangers from falling.
The authors are committed to the implementation of AR to increase the safety of workers. The authors used AR to mark the bridge components for the inspectors, as shown in Figure 8. The authors are also designing an AR application using eye-tracking data to draw a cognitive map for Federal Railroad Administration (FRA) to increase railroader safety during field inspections.
FIGURE 8
In the future, AR construction research trends are real-time monitoring and the human factor. A construction site is a dynamic and fast-changing scenario. Hence, to effectively monitor real-time site information, the fast-response requirements for AR algorithms and devices are essential. Real-time monitoring has become a hot topic for AR construction. Zollmann et al. (2012) developed an application using overview and detail techniques to visualize real-time dynamic data for construction sites with AR. furthered the ARVISCOPE-ROVER system's study to visualize 3D AR animations of active engineering operations. developed a framework driven by data to collect real-time field data and visualize dynamic 3D simulations of engineering activities. Liu et al. (2018) developed a rapid construction site information acquisition method to conduct dynamic site layout planning. Besides overcoming technical difficulties, performance-based decisions are being considered. , Wang and Dunston (2006), Wang and Dunston (2008), and analyzed the feasibility of considering human factors in AR applications to develop visualization tools for construction tasks.
Most research and devices were in the experiment and prototype phase yet not applied to practical site construction. The main obstacle to adopting AR technologies during construction was the high expense and immature devices (Manuel Davila Delgado et al., 2020). Developing rapidly in recent years, AR devices suitable for construction are already commercially available. An eligible AR device can be selected by appropriate performance requirements for a different construction site. For example, the RealWear HMT-1Z1 should be chosen for high-risk (e.g., potentially explosive areas) and high-value construction for its safety and high price. At the same time, Google Glass Enterprise Edition 2 is fit for daily civil infrastructure for its lightweight. Besides HMDs and handheld displays, the combination of unmanned aerial vehicle (UAV) and AR technologies to conduct holistic site planning and large-scale terrain scanning is a significant trend. Wen and Kang (2014) enhanced and prototyped a UAV with AR technologies to provide site-level image transmission. Zollmann et al. (2014) used UAV to conduct 3D reconstruction of site information, providing on-site visualization on mobile. Yan et al. (2019) combined UAV and AR to run landscape simulations. Commercial AR-UAVs for construction could be the next important component of the AR market.
Another challenge is workforce training. Since AR technology is new to most construction workers, efficient training is necessary to implement AR technology in a real construction project. The workers will need to understand basic AR knowledge and AR device operating methods.
Other Implementations
Building Information Models
Many researchers classify BIM as an application in construction because of the extensive implementations of BIM in construction. Nonetheless, given the significance, AR-enhanced BIM is classified as a separate category in this article. Combined with AR technologies' visualization potential, BIM can perform as a functional decision support system (; Machado and Vilela, 2020). A holistic bibliometric literature review of BIM between 2005 and 2015 can be found in Santos et al. (2017) and , concluding that AR-enhanced BIM is one of the main trends of practical BIM application. This section mainly focused on the research trend after 2015.
BIM is a functional tool for structural information management. Combined with AR technologies, visualization and intuition are enhanced for displaying information to engineers. The merging of AR and BIM maximized both the AR's visualization advantage and BIM's information management advantage (Wang et al., 2015). In 2014, MeĹľa et al. (2014) conducted studies on information storage and regeneration and presented a prototype software system tested in a real project. In 2018, conducted a diagnosis and performance assessment for existing buildings, finding that information management is crucial for AR-enhanced BIM. In 2020, proposed an innovative bridge maintenance system using AR-enhanced BIM to provide bridge maintenance information for engineers.
(Hossain et al., 2020) used AR technology to overlap model shapes onto the real specimen in a shaking table experiment, as shown in Figure 9. The model shape of the specimen was changed and displayed in real time, with the real specimen moving.
FIGURE 9
Providing systematic information management, defects, and rework during the construction process can be minimized, hence guaranteeing project schedule and cost budget, verified by a quantitative analysis conducted by Hwang et al. (2019). Kim et al. (2015) combined BIM and 3D laser scanning technology to perform quality monitoring for concrete elements. Schickert et al. (2018) conducted nondestructive testing for the concrete specimen with BIM framework and AR-enhanced visualization.
Many AR-enhanced BIM applications are now available for holistic lifecycle management and specialized construction. For example, Tang et al. (2016) proposed a conceptual workflow with the BIM-GIS application for the prefabrication design to save construction time, increase efficiency, and reduce energy consumption; suggested engagement of a BIM application, BIM-IKBMS, for the building management after construction.
For the application of standard software, compared 122 BIM software applications for project management. Considering that each software has pros and cons, and different software can complete similar targets, suggested selecting BIM software should consider supply chain partners and the project's lifecycle. reviewed commercially available BIM software for historic buildings and suggested that the BIM applications for handheld AR devices, A360 (), iVisit3D (), BIMx (), and , best fit AR technologies. Kopsida and Brilakis (2017) evaluated different software designed for AR-enhanced BIM applications, concluding that a gap still exists between AR-enhanced BIM technologies and practical on-site applications. Rahimian et al. (2020) proposed an AR-enhanced BIM framework to integrate structural elements' visualization information using Unify.
The authors are developing the interface of finite element models (FEMs) in AR for hybrid experiments (Figure 10.) The user can interface with a hologram of one virtual FEM experiment using a shaking table input created by the computer. The user can define the mass, stiffness, and height of each story in the interface. In a hybrid experiment, some structural components' movement is calculated in the computer, while others are tested using a real shaking table (Figure 11). The AR interface can superpose the virtual components onto the real ones, showing a whole structure with consistent and intuitive dynamic moving. More complicated BIM models will be developed based on this prototype.
FIGURE 10
FIGURE 11
Among all the HMD devices, Microsoft HoloLens 2 has the enormous potential to be the game-changer for the AR-enhanced BIM application. First, HoloLens 2 is embedded with a robust depth sensor. Hence, it is capable of 3D monitoring. Second, the programming platform of HoloLens 2, Unity, is relatively completed and widely accepted by engineers. Last, HoloLens 2 is compatible with most general civil infrastructural models, for example, Autodesk 3D model and Trimble SketchUp model. A detailed comparison of the available AR devices is given in Xu et al., 2020.
Life Cycle Management
AR technologies can also be used in the energy performance assessment of a building. Traditional energy performance assessment for a building is time-consuming and unwieldy. The application of AR technologies can save assessing time, reduce operation difficulty, and enable engineers to monitor real-time energy abnormity. In 2005, Malkawi and Srinivasan (2005) presented an interactive and immersive AR model showing an indoor temperature field by HMD modified in 2009 using a mobile robot to achieve robust visualization. conducted an energy performance assessment by a new approach, EPAR, using unordered digital and thermal imagery. Experiments have verified the robustness of EPAR. Based on the EPAR models, developed new algorithms to conduct rapid automated analysis. Wang et al. (2013) developed a prototype hybrid LIDAR system for rapid 3D thermal monitoring. Liu and Seipel (2015) provided another novel algorithm with quadrilateral features registration (e.g., windows) for building's thermal monitoring.
Structural Health Monitoring and Damage Detection
AR-based SHM is more suitable for routine inspection and monitoring than other tasks, for example, special inspection, testing, and maintenance (Peres et al., 2018). AR technologies are mainly used in the infrastructure SHM and crack inspection.
The safety of bridges is of great significance during heavy transportation today. More and more bridges are reaching the designed service year corresponding to the increasing safety requirements, leading to massive SHM and detection necessities (Moreu, 2015; ). Traditional manual monitoring methods can hardly fulfill the rapidly increasing monitoring tasks. To improve monitoring efficiency, conducted SHM for highway bridges using a context-aware computing framework. The framework automatically identified the component of interest and retrieved relevant information from the database. Computer vision techniques are functional for crack detection during SHM. Wu (2015) developed a novel self-adaptive image-processing algorithm for the crack detection of road pavements. Lattanzi et al. (2016) conducted seismic damage detection for bridge columns. Computer vision techniques were used to estimate the peak displacement during the earthquake by damage observations from images captured by cameras.
SHM and damage detection are essential for critical infrastructure during disaster and post-disaster. In Section 2, the application of AR technologies for discrepancy check during construction has been introduced. Similar platforms can be applied for post-disaster damage detection. By comparing the post-disaster real structure and previously stored AR building information, structural damage can be observed (Kamat and El Tawil, 2007). Tedeschi and Benedetto (2017) designed an automatic recognition system to inspect pavements for potholes, longitudinal–transversal cracks, and fatigue cracks for pavements. developed 3D digital twin models with related damage records for typical bridge structures as a database to predict a similar structure's reliable future performance. Napolitano et al. (2019) developed a novel AR framework for data visualization during the structural inspection.
With the improvement of the display, cameras, and depth sensors of HMDs, conducting SHM and damage detection with HMDs has been studied. Yamaguchi et al. (2019) developed an AR crack inspection support system for concrete structures with HoloLens. Maharjan et al. (2019) adopted HoloLens to conduct bridge inspection in a controlled laboratory environment and found that applying AR technologies in SHM leads to higher safety and a smaller workload for field inspectors.
Napolitano et al., 2019 used AR to conduct structural health monitoring, as shown in Figure 12. AR technology helped to capture the real environment (the building information) in the field (Figure 12A) and reanchoring the image in a new position (Figure 12B).
FIGURE 12
The authors developed an AR application named “Time Machine Measure (TMM)” to assist in SHM. Traditionally, researchers need to use markers to measure over time if an object's moving distance needs to be measured. TMM provides a time machine function using the real environment's spatial mapping, restoring its position at a previous timestamp. Therefore, researchers can directly measure the moving distance of an object without tedious and time-consuming marking procedures. Figure 13 shows an illustration of measurement with TMM. In Figure 13A, the green cube is the original position of the cube storage (Position-1), and the blue one is the position after a move (Position-2). The blue cube is real, while the green one is an invoked virtual image marking the original position. With this function, the moving distance can be conveniently measured as 0.91 m. Figure 13B shows that the cube is moved again to Position-3 (the cyan cube). Similarly, the cyan cube is real, while the blue and green ones are invoked virtual images. Figure 13B shows that the moving distance from Position-1 to Position-3 is 1.88 m. This measurement assistant AR application simplifies the SHM work for field inspections.
FIGURE 13
During SHM and damage detection, functional depth sensor and high-resolution cameras are requisite to HMDs. The only devices meeting both requirements are HoloLens and ThirdEye Gen X2 (Xu et al., 2020). The larger 52° FOV makes HoloLens a more popular device for researchers. However, studies of HoloLens applied in SHM are still in the experimental phase. More on-site field experiments should be conducted to verify reliability. Besides HMDs, a novel AR projector, LF2, could also be a game-changer for damage detection if the reliability under strong sunlight can be improved. As a projector, LF2 is not limited by FOV, processing, or storage capacity.
Underground Utilities
Underground utilities encompass underground geological structures (e.g., basements, foundations, garages, tunnels, and subways) and pipelines (e.g., water, sewage, gas, electricity, and telecommunications) (). The application of AR technologies superimposed virtual underground information to the real aboveground environment, preventing accidental utility strikes during excavation and improving efficiency in maintenance (Kaddioui et al., 2019).
The major challenge is the localization problem (Pereira et al., 2020), that is, how to guarantee the accuracy of the superimposition to deploy AR technologies. Traditionally, abstaining location information required merging data from blueprints, historical records, dead reckoning, and aboveground sensors, which are time-consuming and lack accuracy (Li et al., 2015). The leading solutions today to solve the localization problem consist of utilizing the GPS and ground penetrating radar (GPR) (Kaddioui et al., 2019; Ortega et al., 2019). proposed an analytical approach compiled with GPS to incorporate virtual underground infrastructure into site photos to provide more invisible information. Pereira et al. (2018); Pereira et al. (2019) provided a novel GPR system integrated with AR to solve the localization problem. The proposed GPR system was capable of arbitrary-route scanning with high fidelity.
With the development of HMDs and handheld devices, the information traditionally stored in office-based desktop computers can be shown directly to engineers in the field. Webster et al. (1996) first developed an AR HMD system for architectural construction. The system had the see-through characteristic and could show the columns occluded by walls and rebars inside the columns. combined the HMD device with GPS and CAD models to superimpose real-time virtual underground utilities over the real environment on an excavation site. Schall et al. (2010) designed a novel mobile AR device, VIDENTE, to maintain, plan, and survey underground infrastructure using spatial interaction and visualization techniques. Improved from Vesp'R (Schall et al., 2009), VIDENTE was a relatively complete and functional AR device, getting positive feedback from utility companies. Roberts et al. (2002) developed an AR see-through system compatible with HMDs and handheld laptops, solving the Tracker Technology positioning problem.
Bentley Systems developed a relatively mature and available prototype application for the AR-enhanced underground utility visualization software. The application combined AR, GPR, and BIM and can measure the distance between underground objectives.
Smart City
The official definition of “smart city” has not been universally accepted. However, a common understanding is conducting city management by collecting information from residents and the IoT (Kyriazis et al., 2013; Poslad et al., 2015; ). AR is playing a big part in smart city service, for example, public transport (Novi Sad, Serbia) based on the IoT devices installed on buses (Pokric et al., 2014). and proposed a novel algorithm to monitor and preserve the high-quantity and low-cost assets on highways (e.g., traffic signals and light poles). Besides transportation, AR technologies have also been deployed in the city plan and architecture design.
Early in 1999, Thomas et al. (1999) designed an AR platform, TINMITH2, to visualize the architecture design in real surroundings. Due to computation limits, the architecture was depicted with single lines. Wang (2009) considered using AR technologies for architecture design in 2009. However, due to the technology limit, a practical 3D rendering prototype application was not completed. The primary technology limit is the transition from legally approved 2D design drawing to well-structured integrated 3D models readily for AR systems. The maturity of 3D BIM design promoted the application of AR in architecture design. For example, in 2013, interpreted 2D engineering drawings to the 3D real environment directly with an animated sliding plane and model clipping technique. The mobile AR application developed by VTT Technical Research Center of Finland (VTT, 2020) was the first practical application, helping plan a hotel in Billnäs ironworks site at Raseborg, Kämp Tower in Jätkäsaari, Helsinki, and a wind generator farm in Pörtom, Närpes (Woodward, 2015).
Traditionally, existing structures and natural objects need to be modeled in 3D CAD during city planning to illustrate the relative position and height relation, which is time-consuming and money-costing. AR is a powerful tool for both landscape and height design during a city plan. In 2011, to preserve historic buildings’ good landscapes and provide a reasonable basis for city height regulations, Yabuki et al. (2011) developed a feasible and effective system deploying ARToolKit to measure the maximum invisible height of each building. A typical example illustrating the AR city plan is conducted in Christchurch, New Zealand. A mobile AR application, CityViewAR (no longer available), was used to assist the post-quake city plan and reconstruction of destroyed buildings for the first time (Lee et al., 2012).
Satellites are widely implemented to obtain data for building smart cities. For example, DSCOVR satellite (Deep Space Climate Observatory satellite), launched on February 11th, 2015, can monitor the real-time solar wind in the Unites States, hence to the accuracy forecast weather alerts (NOAA, 2015). The authors developed an AR application, SMAC, to monitor and control the satellites, including the position (latitude and altitude), velocity, and other data collected by the satellites, as shown in Figure 14. Until now, SMAC can access three satellites, SHARC, DSCOVR, and the International Space Station. SMAC can assist smart city management with access to more satellite data, such as traffic monitoring data and communication network data.
FIGURE 14
HoloLens and Unity platform were further investigated among all the available AR devices. developed a toolkit by Unity to conduct water hazard prediction by monitoring real-time precipitation and hydrological information. tested HoloLens by 21 participants in an in-house workshop, showing the potential of using HoloLens in AR-assisted architecture design. As an essential part of the smart city, the rapid development of applying HoloLens in city plan and architecture design is foreseeable.
Discussion
The authors generate a synthesis comparison table to summarize the current progress and technical challenges for each AR implementation category in civil infrastructure. As discussed in Sections Construction and Other Implementations, AR implementations in civil infrastructure can be divided into construction, BIM, SHM, underground utilities, and smart city management. Implementations in construction are further divided into monitoring project progress and assisting workers. With this comparison table, readers can generate a thorough understanding of AR implementations in civil infrastructure today and compare their study to the frontiers in the corresponding area. References in Table 1 are the typical studies in the correlated category. Comparably, AR technologies in discrepancy check, quality control, BIM, and SHM are more mature. Multiple prototypes have been designed, and small-scale field experiments have been conducted.
TABLE 1
| AR implementations in civil infrastructure | Current progress | Technical/practical challenges | References | ||
|---|---|---|---|---|---|
| Construction | Monitoring project progress | Discrepancy check | Prototype devices and software have been developed. Small-scale experiments have been conducted | Outdoor environment Occlusion problem | : Kamat et al. (2011): Karsch et al. (2014) |
| Quality control | Typical field implementations | Accuracy requirement Occlusion problem | Zhou et al. (2017) | ||
| Assisting workers | Collaborative visualization | Theory | Communication stability | Soman et al. (2017) | |
| Telecommunication | Experiments | Communication stability Occlusion problem | |||
| Site safety | Experiments | Accuracy requirement Communication efficiency | Kim et al. (2017): Liu et al. (2018) | ||
| Building information model (BIM) | Multiple prototype devices and software have been developed | Accuracy requirement Occlusion problem latency problem | Machado and Vilela (2020)Hwang et al. (2019): : Santos et al. (2017): | ||
| Structural health monitoring (SHM) and damage detection | Multiple prototype devices and software have been developed | Accuracy requirement Occlusion problem latency problem | : Xu et al. (2020): Yamaguchi et al. (2019)Napolitano et al. (2019): | ||
| Underground utilities | Small-scale experiments | Localization problem. Accuracy requirement Occlusion problem | Kaddioui et al. (2019): Ortega et al. (2019): Pereira et al. (2020) | ||
| Smart City | Concepts | Accuracy requirement Occlusion problem Communication stability Privacy concern | Lee et al. (2012): NOAA (2015): Woodward (2015) | ||
Synthesis comparison table of AR implementations in civil infrastructure.
There are three fundamental technologies in AR implementations, display technology, tracking registration technology, and human–computer interaction technology (Zhang et al., 2020). In the three fundamental AR technologies, five significant technical challenges hinder the AR development in civil infrastructure today are as follows:
1)Occlusion problem. Occlusion problem is one of the two major fundamental technical issues in the various AR implementations in civil infrastructure. Occlusion problem exists in every mobile AR application and HMD, limiting the augmented range and accuracy. The standard solution for the occlusion problem is to pre-map the surroundings to avoid solving the real-time occlusion problem. However, researchers have proposed more acceptable methods for the real-time occlusion problem (Holynski and Kopf, 2018). Applying recent theories to AR devices can motivate more AR implementations, although the occlusion problem still has a long way to the perfect solution.
2)Latency problem. Another major fundamental challenge for AR technology is the latency problem, that is, the time-delay problem. The time delay of the virtual environment calculation can cause a displacement between the real and virtual environments. Two display methods exist when merging the real and virtual environments, optical see-through and video see-through. The optical see-through method displays the real environment through a glass with virtual objects showing on the glass (e.g., Microsoft HoloLens
3)The see-through video method captures the real environment by a camera and shows both the images in the real environment and the virtual objects on the same monitor screen. The synchronization problem of the video see-through display method can be solved by adding a time delay for the real environment on purpose to offset the virtual environment (Kanbara et al., 2000). However, time delay cannot be eliminated entirely for the optical see-through method but can only be minimized by accelerating the virtual environment's calculation (Kiyokawa et al., 2003).
4)Outdoor environments. Today, most AR devices, including both prototypes and commercial devices, use lasers to overlap virtual images onto the real environment. Lasers work well and stable in indoor environments. However, the temperature and luminance changes in outdoor environments can be a challenge to most AR devices. For instance, Microsoft HoloLens 2, as a powerful and comprehensive AR device in the market, still inherits the intrinsic disadvantages of the working environment.
a)HoloLens uses visible light to track the user's location and the surroundings, hence requiring a steady 500–1000 lux luminance for best performance. If the environment is too bright, the cameras can get saturated, and nothing can be seen. If the environment is too dark, the cameras cannot pick up enough information, and nothing is seen (Microsoft, 2019). 500 lux luminance is the light of typical indoor environments, for example, offices, groceries, and laboratories. 1000 lux luminance is the light of bright indoor environments, for example, detailed mechanical workshops and operation theaters.
b)HoloLens operates with the best performance in the environments within the temperature range of 10–27°C (50–81°F) and the humidity range of 8–90% relative humidity. In the authors' experiments, HoloLens can operate in environments out of these ranges. However, implementations out of the recommended range may lead to shorter battery life or device damage.
c)Localization problem is also a challenge for an outdoor environment. AR devices need to scan the surroundings using a depth sensor to generate a detailed spatial map so that virtual images can be displayed in an accurate position. However, in an outdoor environment, the surroundings often lack adequate references like an indoor wall. Some AR devices choose to deploy a GPS sensor to solve the localization problem.
5)Accuracy requirement. To generate realistic virtual images, the positions of the virtual images have to be very accurate. A relatively large position error will lead users to doubt the reality of the AR images and cause motion sickness, dizziness, and nausea.
6)Communication stability and efficiency requirement. AR devices are designed to be able to communicate with each other. This function is beneficial for assisting workers and providing remote directions. In this scenario, the stability and efficiency of the communication are vital. Unstable communication and large latency can cause danger to workers. The speed and quality of remote communications typically depend on the wireless network. Therefore, a sound and reliable local wireless network is necessary for AR communication.
Frontier computer vision and graphics research provide more satisfying solutions for the five previously mentioned AR technical challenges. Today, frontier computer vision research includes four aspects: data collection, low-/mid-level processing, high-level processing, and application (Xu et al., 2020). The frontier data collection technologies try to design a good data collection strategy, for example, layout plan to adverse environmental noise. The frontier low-/mid-level processing technologies exploit GPUs and parallel computing to reduce computational cost and improve accuracy. The frontier high-level processing technologies are combined with BIM to solve the problem of lacking prior models. Finally, all the frontier computer vision technologies lack implementations and tests in real projects, which have become one of the most considerable weaknesses of computer vision and graphics research.
In another article, the authors classified sixteen AR devices available in the market in the last five years regarding civil infrastructure implementation capabilities. The authors compared AR devices’ performances in weight, price, product durability, sensors, connectivity, computational capacities, and display capacities, including the parameters that civil engineers are most interested in, for example, camera resolution, processing unit, and FoV. According to the classification and ranking of that study, Google Glass Enterprise Edition 2, Microsoft HoloLens 2, and Everysight Raptor have the highest combined capabilities to become popular in civil infrastructure. Google Glass Enterprise Edition 2 has the largest FoV, which is essential in construction. Everysight Raptor has the second largest FoV among all the AR devices and the second largest camera resolution. Microsoft HoloLens 2 has all-sided sensors (including depth sensor), a high-resolution display, the third largest FoV, and relatively balanced calculation and storage performances. All the three AR devices have significant potential in the future civil infrastructure implementations.
Conclusion
This article reviewed the research field of AR applications in civil infrastructure, mainly focused on research studies in the latest five years and milestone developments. While several other introductory articles have been written on AR applications in civil engineering, this article is more comprehensive and up-to-date. The most extensive research branch is construction with AR, followed by BIM, SHM, underground utilities, and smart city management. Given that quite a few prototype devices and applications have been designed and tested today, few devices are commercially practical for massive on-site applications. Fit devices are recommended for each application branch. Generally, HoloLens has become the most promising AR HMD for civil infrastructure because of its all-sided sensors (including depth sensor), high-resolution display, larger FOV, and balanced calculation and storage performance.
There are three central research tendencies in the future. First, theoretical breakthroughs in AR technology research are vital, including further studies of the occlusion and latency problems. Second, the weight and price of updated version AR devices should be reduced and improve performance. Finally, a unified AR function “toolbox” platform and benchmark models should be established and well accepted in both the academic and engineering community. The complicated applications and applications can be compared and combined to display the respective superiority. Unity has become a more general platform for AR application development. In the future, researchers should conduct more field experiments to verify the prototypes. With the help from the industry, the verified AR applications will then be implemented more extensively. For a civil engineer who is a novice in AR, the authors suggested the engineer starting the AR studies with relatively more mature platforms, for example, Unity 3D. Besides the references discussed in this article, the authors also recommend reading the Unity manual (Unity, 2021) and starting with the programmed MRTK SDKs (The Mixed Reality Toolkit (MRTK), 2020).
Statements
Ethics statement
Written informed consent was obtained from the individuals for the publication of any potentially identifiable images or data included in this article.
Author contributions
JX collected the data, organized the citations, and drafted the manuscript. FM is the leader of the project and coordinated the access to data, resources, and analysis. FM reviewed the manuscript composition, sources, and the final version. FM secured the funding to conduct this research, access to AR resources, codes, and experiments.
Funding
This research’s financial support is provided in part by the Air Force Research Laboratory (AFRL, Grant number FA9453-18-2-0022) and the New Mexico Consortium (NMC, Grant number 2RNA6.) The conclusions of this research solely represent those of the authors.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
1
AbandaF. H.VidalakisC.OtiA. H.TahJ. H. M. (2015). A critical analysis of building information modelling systems used in construction projects. Adv. Eng. Softw.90, 183–201. 10.1016/j.advengsoft.2015.08.009
2
AbdiradH. (2017). Metric-based BIM implementation assessment: a review of research and practice. Architectural Eng. Des. Manage.13 (1), 52–78. 10.1080/17452007.2016.1183474
3
AgueroM.MaharjanD.del Pilar RodriguezM.MascarenasD. D. L.MoreuF. (2020). Design and implementation of a connection between augmented reality and sensors. Robotics9 (3), 9010003. 10.3390/robotics9010003
4
AhmedS. (2018). A review on using opportunities of augmented reality and virtual reality in construction project management. Organization, technology and management in construction. Int. J.10 (1), 1839–1852. 10.2478/otmcj-2018-0012
5
AiteanuD.HillersB.GraserA. (2003). “A step forward in manual welding: demonstration of augmented reality helmet,” in The second IEEE and ACM international symposium on mixed and augmented reality, Tokyo, TKY, 27 October 2003 (IEEE), 309–310. 10.1109/ISMAR.2003.1240734
6
AkhavianR.BehzadanA. H. (2012). An integrated data collection and analysis framework for remote monitoring and planning of construction operations. Adv. Eng. Inform.26 (4), 749–761. 10.1016/j.aei.2012.04.004
7
AkulaM.SandurA.KamatV. R.PrakashA. (2015). Context-aware framework for highway bridge inspections. J. Comput. Civ. Eng.29 (1), 04014027. 10.1061/(ASCE)CP.1943-5487.0000292
8
AoyamaK. (2019). “Novel display using percutaneous electrical stimulation for virtual reality,” in Human interface and the management of information (international conference on human-computer interaction), Orlando, FL, July 26–31, 2019 (New York, NY: Springer), 3–14. 10.1007/978-3-030-22649-7_1
9
Artlantis Studio (n.d.) (2019). iVisit3D. Available at: https://atl.artlantis.com/en/ivisit3d.
10
Augmented 3D (2020). Available at: https://rb.gy/wbc6vp.
11
Autodesk (n.d.) (2016). A360. Artlantis Studio. Available at: https://www.autodesk.com/products/a360/overview.
12
BaeH.Golparvar-FardM.WhiteJ. (2013). High-precision vision-based mobile augmented reality system for context-aware architectural, engineering, construction and facility management (AEC/FM) applications. Visual. Eng.1 (1), 3. 10.1186/2213-7459-1-3
13
BalaliV. (2015). Image-based recognition, 3D localization, and retro-reflectivity evaluation of high-quantity low-cost roadway assets for enhanced condition assessment. Champaign, IL: University of Illinois at Urbana-Champaign.
14
BarazzettiL.BanfiF. (2017). “Historic BIM for mobile VR/AR applications,” in Mixed reality and gamification for cultural heritage. New York, NY: Springer International Publishing, 271–290. 10.1007/978-3-319-49607-8_10
15
BayodeA.van der PollJ. A.RamphalR. R. (2019). “4th industrial revolution: challenges and opportunities in the south African context,” in 17th JOHANNESBURG Int’l conference on science, engineering, technology and waste management (SETWM-19), Johannesburg, South Africa, November 18–19, 2019 (Potchefstroom, South Africa: North West University), 341–347. 10.17758/EARES8.EAP1119285
16
BehzadanA. H. (2008). ARVISCOPE: georeferenced visualization of dynamic construction processes in three-dimensional outdoor Augmented Reality. PhD thesis. Ann Arbor (MI): University of Michigan.
17
BehzadanA. H.KamatV. R. (2009a). Automated generation of operations level construction animations in outdoor Augmented Reality. J. Comput. Civ. Eng.23 (6), 405–417. 10.1061/(ASCE)0887-3801200923
18
BehzadanA. H.KamatV. R. (2009b). Interactive augmented reality visualization for improved damage prevention and maintenance of underground infrastructure. Construct. Res. Cong. Build. a Sustain. Fut., 1214–1222. 10.1061/41020(339)123
19
BehzadanA. H.KamatV. R. (2011). Integrated information modeling and visual simulation of engineering operations using dynamic augmented reality scene graphs. J. Inf. Tech. Construction (ITcon)16 (17), 259–278.
20
BehzadanA. H.KamatV. R. (2005). “Visualization of construction graphics in outdoor augmented reality,” in Proceedings of the winter simulation conference 2005. Orlando, FL, 23 January 2006 (IEEE), 1914–1920. 10.1109/WSC.2005.1574469
21
BehzadanA. H.TimmB. W.KamatV. R. (2008). General-purpose modular hardware and software framework for mobile outdoor augmented reality applications in engineering. Adv. Eng. Inform.22 (1), 90–105. 10.1016/j.aei.2007.08.005
22
BerrymanD. R. (2012). Augmented reality: a review. Med. Ref. Serv. Q.31 (2), 212–218. 10.1080/02763869.2012.670604
23
BobylevN. (2009). Mainstreaming sustainable development into a city's Master plan: a case of Urban Underground Space use. Land Use Policy26 (4), 1128–1137. 10.1016/j.landusepol.2009.02.003
24
BrunoS.De FinoM.FatigusoF. (2018). Historic building information modelling: performance assessment for diagnosis-aided information modelling and management. Autom.Constr.86, 256–276. 10.1016/j.autcon.2017.11.009
25
BurdeaG. C.CoiffetP. (2003). Virtual reality technology. New York, NY: John Wiley & Sons.
26
CarmignianiJ.FurhtB.AnisettiM.CeravoloP.DamianiE.IvkovicM. (2011). Augmented reality technologies, systems and applications. Multimed Tools Appl.51 (1), 341–377. 10.1007/s11042-010-0660-6
27
CasasJ. R.MoughtyJ. J. (2017). Bridge damage detection based on vibration data: past and new developments. Front. Built Environ.3, 4. 10.3389/fbuil.2017.00004
28
CattariN.CutoloF.D’amatoR.FontanaU.FerrariV. (2019). “Toed-in vs parallel displays in video see-through head-mounted displays for close-up view,” in IEEE Access, Pisa, PI, October 31, 2019 (IEEE), 159698–159711.
29
ChenK.LuW.PengY.RowlinsonS.HuangG. Q. (2015). Bridging BIM and building: from a literature review to an integrated conceptual framework. Int. J. Project Manage.33 (6), 1405–1416. 10.1016/j.ijproman.2015.03.006
30
ChiH.-L.ChenY.-C.KangS.-C.HsiehS.-H. (2012). Development of user interface for tele-operated cranes. Adv. Eng. Inform.26 (3), 641–652. 10.1016/j.aei.2012.05.001
31
CôtéS.TrudelP.SnyderR.GervaisR. (2013). “An augmented reality tool for facilitating on-site interpretation of 2D construction drawings,” in Proceedings of the 13th international conference on construction applications of virtual reality. London, United Kingdom, October 30–31, 2013 (CONVR), 316–323.
32
DaiF.LuM.KamatV. R. (2011). Analytical approach to augmenting site photos with 3D graphics of underground infrastructure in construction engineering applications. J. Comput. Civ. Eng.25 (1), 66–74. 10.1061/(ASCE)CP.1943-5487.0000072
33
DangN.-S.KangH.-R.LonS.ShimC.-S. (2018). “3D digital twin models for bridge maintenance,” in Proceedings of 10th international conference on short and medium span bridges, Quebec,QC, July 13–August 3, 200373, 1–9.
34
DangN.ShimC. (2020). BIM-based innovative bridge maintenance system using augmented reality technology. CIGOS 2019 Innov. Sustain. Infrastruct.54, 1217–1222. 10.1007/978-981-15-0802-8_195
35
DiaoP.-H.ShihN.-J. (2019). Trends and research issues of augmented reality studies in architectural and civil engineering education-A review of academic journal publications. Appl. Sci.9 (9), 1840. 10.3390/app9091840
36
DongS.BehzadanA. H.ChenF.KamatV. R. (2013). Collaborative visualization of engineering processes using tabletop augmented reality. Adv. Eng. Softw.55, 45–55. 10.1016/j.advengsoft.2012.09.001
37
DongS.KamatV. R. (2010). “Robust mobile computing framework for visualization of simulated processes in augmented reality,” in Proceedings of the 2010 winter simulation conference, Baltimore, MD, December 5–8, 2010 (IEEE), 3111–3122. 10.1109/WSC.2010.5679004
38
DunstonP. S.WangX. (2011). An iterative methodology for mapping mixed reality technologies to AEC operations. J. Inf. Tech. Constr. (ITcon)16 (30), 509–528.
39
DunstonP. S.WangX. (2005). Mixed reality-based visualization interfaces for architecture, engineering, and construction industry. J. Constr. Eng. Manage.131 (12), 1301–1309. 10.1061/(asce)0733-9364(2005)131:12(1301)
40
ErgünO.AkınŞ.Dinoİ. G.SurerE. (2019). “Architectural design in virtual reality and mixed reality environments: a comparative analysis,” in IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, OX, March 23–27, 2019 (IEEE), 10.1109/VR.2019.8798180
41
FathiH.BrilakisI. (2016). Multistep explicit stereo camera calibration approach to improve euclidean accuracy of large-scale 3D reconstruction. J. Comput. Civ. Eng.30 (1), 04014120. 10.1061/(asce)cp.1943-5487.0000454
42
FeemanS. M.WrightL. B.SalmonJ. L. (2018). Exploration and evaluation of CAD modeling in virtual reality. Computer-Aided Des. Appl.15 (6), 892–904. 10.1080/16864360.2018.1462570
43
FengC. (2015). Camera marker networks for pose estimation and scene understanding in construction automation and robotics. PhD thesis. Ann Arbor (MI): University of Michigan.
44
GangS.-M.ChoiH.-W.KimD.-R.ChoungY.-J. (2016). A study on the construction of the unity 3D engine based on the webGIS system for the hydrological and water hazard information display. Proced. Eng.154, 138–145. 10.1016/j.proeng.2016.07.431
45
GhaffarianHoseiniA.ZhangT.NwadigoO.GhaffarianHoseiniA.NaismithN.TookeyJ.et al (2017). Application of nD BIM integrated knowledge-based building management system (BIM-IKBMS) for inspecting postconstruction energy efficiency. Renew. Sustain. Energ. Rev.72, 935–949. 10.1016/j.rser.2016.12.061
46
GiangN. K.LeaR.BlackstockM.LeungV. C. M. (2016). “On building smart city IoT applications: a coordination-based perspective,” in Proceedings of the 2nd international workshop on smart, Vancouver,BC, December 2016 (ACM), 1–6. 10.1145/3009912.3009919
47
Golparvar-FardM.BalaliV.De La GarzaJ. M. (2015). Segmentation and recognition of highway assets using image-based 3D point clouds and semantic texton forests. J. Comput. Civil Eng.29 (1), 04014023. 10.1061/(ASCE)CP.1943-5487.0000283
48
Golparvar-FardM.BohnJ.TeizerJ.SavareseS.Peña-MoraF. (2011a). Evaluation of image-based modeling and laser scanning accuracy for emerging automated performance monitoring techniques. Autom. Constr.20 (8), 1143–1155. 10.1016/j.autcon.2011.04.016
49
Golparvar-FardM.Peña-MoraF.SavareseS. (2011b). Integrated sequential as-built and as-planned representation with D4AR tools in support of decision-making tasks in the AEC/FM industry. J. Constr. Eng. Manage.137 (12), 1099–1116. 10.1061/(ASCE)CO.1943-7862.0000371
50
Golparvar-FardM.HamY. (2014). Automated diagnostics and visualization of potential energy performance problems in existing buildings using energy performance augmented reality models. J. Comput. Civ. Eng.28 (1), 17–29. 10.1061/(asce)cp.1943-5487.0000311
51
Golparvar-FardM.Peña-MoraF.ArboledaC. A.LeeS. (2009a). Visualization of construction progress monitoring with 4D simulation model overlaid on time-lapsed photographs. J. Comput. Civ. Eng.23 (6), 391–404. 10.1061/(asce)0887-3801(2009)23:6(391)
52
Golparvar-FardM.Peña-MoraF.SavareseS. (2009b). D4AR–a 4-dimensional augmented reality model for automating construction progress monitoring data collection, processing and communication. J. Inf. Tech. Constr. (ITcon)14 (13), 129–153.
53
Graphisoft (2011). BIMx. Available at: https://en.wikipedia.org/wiki/BIMx.
54
HamY.Golparvar-FardM. (2012). “Rapid 3D energy performance modeling of existing buildings using thermal and digital imagery,” in Construction challenges in a flat world:construction research congress proceedings of the 2012 construction Research congress, West Lafayette, IN, May 21–23, 2012 (ASCE), 991–1000. 10.1061/9780784412329.100
55
HammadA.GarrettJ. H.KarimiH. A. (2002). “Potential of mobile augmented reality for infrastructure field tasks,” in Applications of advanced technologies in transportation, Cambridge, MA, August 5–7, 2002 (ASCE), 425–432. 10.1061/40632(245)54
56
HammadA.WangH.MudurS. P. (2009). Distributed augmented reality for visualizing collaborative construction tasks. J. Comput. Civ. Eng.23 (6), 418–427. 10.1061/(ASCE)0887-3801200923
57
HolynskiA.KopfJ. (2018). Fast depth densification for occlusion-aware augmented reality. ACM Trans. Graphics37, 1–11. 10.1145/3272127.3275083
58
HossainM.HansonJ.-W.MoreuF. (2020). Real-time theoretical and experimental dynamic mode shapes for structural analysis using augmented reality. Top. Modal Anal. Test., 351–356. 10.1007/978-3-030-47717-2_36
59
HwangB.-G.ZhaoX.YangK. W. (2019). Effect of BIM on rework in construction projects in Singapore: status quo, magnitude, impact, and strategies. J. Constr. Eng. Manage.145 (2), 04018125. 10.1061/(ASCE)CO.1943-7862.0001600
60
JahnG.NewnhamC.van den BergN.BeanlandM. (2018). “Making in mixed reality,” in ACADIA 2018 Proceedings of the 38th Annual Conference of the Association for Computer Aided Design in Architecture (ACADIA), Mexico, MX, October 18–20, 2018, 88–97. Available at: http://papers.cumincad.org/cgi-bin/works/BrowseTreefield=seriesorder=AZ/Show?acadia18_88.
61
JeanclosN.SharifM.-M.LiS. K.KwiatekC.HaasC. (2018). “Derivation of minimum required model for augmented reality based stepwise construction assembly control,” in Workshop of the european group for intelligent computing in engineering, Waterloo,ON, May19, 2018 (Springer), 336–358. 10.1007/978-3-319-91635-4_18
62
JulierS.BaillotY.LanzagortaM.BrownD.RosenblumL. (2000). “Bars: battlefield augmented reality system,” in NATO symposium on information processing techniques for military systems. Washington, DC, Available at: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.1.5064
63
KaddiouiA.ShahrourI.El OirrakA. (2019). “Uses of augmented reality for urban utilities management,” in MATEC Web of Conference, Lille,NO, January 2019, 295, 02009. 10.1051/matecconf/201929502009
64
KamatV. R.El-TawilS. (2007). Evaluation of augmented reality for rapid assessment of earthquake-induced building damage. J. Comput. Civ. Eng.21 (5), 303–310 . 10.1061/(ASCE)0887-3801200721
65
KamatV. R.MartinezJ. C.FischerM.Golparvar-FardM.Peña-MoraF.SavareseS. (2011). Research in visualization techniques for field construction. J. Constr. Eng. Manage.137 (10), 853–862. 10.1061/(ASCE)CO.1943-7862.0000262
66
KanbaraM.OkumaT.TakemuraH.YokoyaN. (2000). “A stereoscopic video see-through augmented reality system based on real-time vision-based registration,” in Proceedings IEEE Virtual Reality, 255–262. 10.1109/vr.2000.840506
67
KarschK.Golparvar-FardM.ForsythD. (2014). ConstructAide: analyzing and visualizing construction sites through photographs and building models. ACM Trans. Graph.33 (6), 1–11. 10.1145/2661229.2661256
68
KimK.KimH.KimH. (2017). Image-based construction hazard avoidance system using augmented reality in wearable device. Autom. Constr.83, 390–403. 10.1016/j.autcon.2017.06.014
69
KimM.-K.ChengJ. C. P.SohnH.ChangC.-C. (2015). A framework for dimensional and surface quality assessment of precast concrete elements using BIM and 3D laser scanning. Autom. Const.49, 225–238. 10.1016/j.autcon.2014.07.010
70
KiyokawaK.BillinghurstM.CampbellB.WoodsE. (2003). “An occlusion capable optical see-through head mount display for supporting co-located collaboration,” in The Second IEEE and ACM International Symposium on Mixed and Augmented Reality (Institute of Electrical and Electronics Engineers Inc.), 133–141. 10.1109/ISMAR.2003.1240696
71
KopsidaM.BrilakisI. (2017). “BIM registration methods for mobile augmented reality-based inspection,” in EWork and EBusiness in architecture, engineering and construction: ECPPM 2016: proceedings of the 11th European conference on product and process modelling (ECPPM 2016), Cambridge, United Kingdom, September 21, 2016, 7–9.
72
KwiatekC.LiS. K.SharifM. M.JeanclosN.HaasC. T.WalbridgeS. (2018). “Facilitating the communication of rework information to craft workers using an augmented reality process,” in ISARC 2018-35th international symposium on automation and robotics in construction and international AEC/FM hackathon: the future of building things, Waterloo, ON, July 12, 2018. 1–8. 10.22260/isarc2018/0071
73
KwiatekC.SharifM.LiS.HaasC.WalbridgeS. (2019). Impact of augmented reality and spatial cognition on assembly in construction. Autom.in Constr.108, 102935. 10.1016/j.autcon.2019.102935
74
KyriazisD.VarvarigouT.WhiteD.RossiA.CooperJ. (2013). “Sustainable smart city IoT applications: heat and electricity management & eco-conscious cruise control for public transportation,” in 2013 IEEE 14th International Symposium on" A World of Wireless, Mobile and Multimedia Networks (WoWMoM), Madird. MA, June 4–7, 2013 (IEEE),
75
LattanziD.MillerG. R.EberhardM. O.HaraldssonO. S. (2016). Bridge column maximum drift estimation via computer vision. J. Comput. Civ. Eng.30 (4), 04015051. 10.1061/(ASCE)CP.1943-5487.0000527
76
LeeG. A.DunserA.KimS.BillinghurstM. (2012). “CityViewAR: a mobile outdoor AR application for city visualization,” in 2012 IEEE international symposium on mixed and augmented reality-arts. Media, and Humanities (ISMAR-AMH), Atlanta, GA, November 5–8 2012 (IEEE), 57–64. 10.1109/ISMAR-AMH.2012.6483989
77
LiS.CaiH.KamatV. R. (2015). Uncertainty-aware geospatial system for mapping and visualizing underground utilities. Autom. Constr.53, 105–119. 10.1016/j.autcon.2015.03.011
78
LiX.YiW.ChiH.-L.WangX.ChanA. P. C. (2018). A critical review of virtual and augmented reality (VR/AR) applications in construction safety. Autom. Constr.86, 150–162. 10.1016/j.autcon.2017.11.003
79
LiuC.-W.WuT.-H.TsaiM.-H.KangS.-C. (2018). Image-based semantic construction reconstruction. Autom. Constr.90, 67–78. 10.1016/j.autcon.2018.02.016
80
LiuF.SeipelS. (2015). Infrared-visible image registration for augmented reality-based thermographic building diagnostics. Visualization Eng.3 (1), 16. 10.1186/s40327-015-0028-0
81
MachadoR. L.VilelaC. (2020). Conceptual framework for integrating BIM and augmented reality in construction management. J. Civil Eng. Manage.26 (1), 83–94. 10.3846/jcem.2020.11803
82
MaharjanD.AgueroM.LippittC.MoreuF. (2019). Infrastructure stakeholders’ perspective in development and implementation of new structural health monitoring (SHM) technologies for maintenance and management of transportation infrastructure. MATEC web of conferences271, 01010. 10.1051/matecconf/201927101010
83
MahmoodA.ButlerB.ZenH.JenningsB. (2018). “Key early verticals, challenges and limitations in implementation of augmented reality,” in Encyclopedia of computer graphics and games (Berlin, Germany: Springer).
84
MalkawiA. M.SrinivasanR. S. (2005). A new paradigm for human-building interaction: the use of CFD and augmented reality. Autom. Constr.14 (1), 71–84. 10.1016/j.autcon.2004.08.001
85
Manuel Davila DelgadoJ.OyedeleL.BeachT.DemianP. (2020). Augmented and virtual reality in construction: drivers and limitations for industry adoption. Loughborough, United: American Society of Civil Engineers.
86
MežaS.TurkŽ.DolencM. (2014). Component based engineering of a mobile BIM-based augmented reality system. Autom. Constr.42, 1–12. 10.1016/j.autcon.2014.02.011
87
Microsoft (2019). Environment considerations for HoloLens. Available at: https://rb.gy/kb5xay.
88
MilgramP.KishinoF. (1994a). “A taxonomy of mixed reality visual displays. IEICE Transact. Inform. Sys.E77-D (12), 1–15.
89
MilgramP.KishinoF. (1994b). “A taxonomy of mixed reality visual displays. IEICE Transact. Inform. Sys.77 (12), 1321–1329.
90
More IcksonM.ben DhaouS. (2019). “Responding to the challenges and opportunities in the 4 th Industrial revolution in developing countries,” in Proceedings of the 12th international conference on theory and practice of electronic governance, Melbourne, Australia, Feburary 22, 2019, 244–253. 10.1145/3326365.3326398
91
MoreuF. (2015). Framework for risk-based management and safety of railroad bridge infrastructure using wireless smart sensors (WSS). Champaign, IL: University of Illinois at Urbana-Champaign.
92
NahangiM. (2015). Development of transformations between designed and built structural systems and pipe assemblies. Waterloo, ON: University of Waterloo.
93
NahangiM.YeungJ.HaasC. T.WalbridgeS.WestJ. (2015). Automated assembly discrepancy feedback using 3D imaging and forward kinematics. Autom. Const.56, 36–46. 10.1016/j.autcon.2015.04.005
94
NahonD.SubileauG.CapelB. (2015). “Never Blind VR” enhancing the virtual reality headset experience with augmented virtuality,” in 2015 IEEE Virtual RealityVR, Arles, FC, March 23–27 2015, 347–348.
95
NapolitanoR.LiuZ.SunC.GlisicB. (2019). Combination of image-based documentation and augmented reality for structural health monitoring and building pathology. Front. Built Environ.5, 1–14. 10.3389/fbuil.2019.00050
96
NOAA (2015). NOAA satellite and information service deep space climate observatory (DSCOVR), Silver Spring, MD: NOAA
97
OhtaY.TamuraH. (2014). Mixed reality: merging real and virtual worlds. Berlin, Germany: Springer Publishing Company
98
OrtegaS.WendelJ.SantanaJ.MurshedS.BoatesI.TrujilloA.et al (2019). Making the invisible visible-strategies for visualizing underground infrastructures in immersive environments. IJGI8 (3), 152–173. 10.3390/ijgi8030152
99
PereiraM.BurnsD.OrfeoD.FarrelR.HustonD. R.XiaT. (2018). “New GPR system integration with augmented reality based positioning,” in Proceedings of the 2018 on great lakes symposium on VLSI, Boston,MA, May 2018 (ACM), 341–346. 10.1145/3194554.3194623
100
PereiraM.BurnsD.OrfeoD.ZhangY.JiaoL.HustonD.et al (2020). “3-D multistatic ground penetrating radar imaging for augmented reality visualization,” in IEEE transactions on geoscience and remote sensing, Princeton, PU, February 6, 2020, 5666–5675.
101
PereiraM.OrfeoD.EzequelleW.BurnsD.XiaT.HustonD. R. (2019). “Photogrammetry and augmented reality for underground infrastructure sensing, mapping and assessment,” in International conference on smart infrastructure and construction 2019 (ICSIC) driving data-informed decision-making, London, United Kingdom, July 5, 2019 (ICE), 169–175. 10.1680/icsic.64669.169
102
PeresF. F. F.ScheerS.FariaÉ. F. d. (2018). A taxonomy of tasks in dam cracks surveillance for augmented reality application. IJAERS5 (10), 179–184. 10.22161/ijaers.5.10.24
103
PetrilloA.De FeliceF.CioffiR.ZomparelliF. (2018). “Fourth industrial revolution: current practices, challenges, and opportunities,” in Digital transformation in smart manufacturing. Editors CioffiR.De FeliceF.PetrilloSoskiceA. (London, United Kingdom: IntechOpen), 1–20.
104
PitsisT. S.BeckmanS. L.SteinertM.OviedoL.MaischB. (2020). Designing the future: strategy, design, and the 4th industrial revolution-An introduction to the special issue. Calif. Manage. Rev.62 (2), 5–11. 10.1177/0008125620907163
105
PokricB.KrcoS.PokricM. (2014). “Augmented reality based smart city services using secure IoT infrastructure,” in 2014 28th international conference on advanced information networking and applications workshops. Victoria, BC, May 13–16, 2014 (IEEE), 803–808. 10.1109/WAINA.2014.127
106
PosladS.MaA.WangZ.MeiH. (2015). Using a smart city IoT to incentivise and target shifts in mobility behaviour-Is it a piece of pie?Sensors15 (6), 13069–13096. 10.3390/s150613069
107
RahimianF. P.SeyedzadehS.OliverS.RodriguezS.DawoodN. (2020). On-demand monitoring of construction projects through a game-like hybrid application of BIM and machine learning. Autom. Constr.110, 103012. 10.1016/j.autcon.2019.103012
108
RobertsG. W.EvansA.DodsonA. H.DenbyB.CooperS.HollandsR. (2002). “The use of augmented reality, GPS and INS for subsurface data visualization,” in FIG XXII international congress, Washington, DC, April 19–26, 2002, 1–12.
109
RoitmanL.ShragerJ.WinogradT. (2017). A comparative analysis of Augmented Reality technologies and their marketability in the consumer electronics segment. J. Biosens. Bioelectron.08 (1), 1000236. 10.4172/2155-6210.1000236
110
SacksR.PerlmanA.BarakR. (2013). Construction safety training using immersive virtual reality. Constr. Manage. Econ.31 (9), 1005–1017. 10.1080/01446193.2013.828844
111
SampaioA. Z.MartinsO. P. (2014). The application of virtual reality technology in the construction of bridge: the cantilever and incremental launching methods. Autom. Constr.37, 58–67. 10.1016/j.autcon.2013.10.015
112
SantosR.CostaA. A.GriloA. (2017). Bibliometric analysis and review of building information modelling literature published between 2005 and 2015. Autom. Constr.80, 118–136. 10.1016/j.autcon.2017.03.005
113
SchallG.MendezE.KruijffE.VeasE.JunghannsS.ReitingerB.et al (2009). Handheld augmented reality for underground infrastructure visualization. Pers Ubiquit Comput.13 (4), 281–291. 10.1007/s00779-008-02045
114
SchallG.SchmalstiegD.JunghannsS. (2010). “VIDENTE-3D visualization of underground infrastructure using handheld augmented reality,” in Geohydroinformatics: integrating GIS and water engineering (Berlin, Germany: Springer).
115
SchickertM.KochC.BonitzF. (2018). “Prospects for integrating augmented reality visualization of nondestructive testing results into model-based infrastructure inspection,” in NDE/NDT for Highways and Bridges SMT 2018, Brazil, August 27, 2018.
116
ShinD. H.DunstonP. S. (2008). Identification of application areas for augmented reality in industrial construction based on technology suitability. Autom. Const.17 (7), 882–894. 10.1016/j.autcon.2008.02.012
117
SomanR. K.BirchD.WhyteJ. K. (2017). “Framework for shared visualization and real-time information flow to the construction site,” in 24th International Workshop on Intelligent Computing in Engineering, London, United Kingdom, July 12, 2017. 10.24928/jc3-2017/0273
118
SonH.BoschéF.KimC. (2015). As-built data acquisition and its use in production monitoring and automated layout of civil infrastructure: a survey. Adv. Eng. Inform.29 (2), 172–183. 10.1016/j.aei.2015.01.009
119
SpeicherM.HallB. D.NebelingM. (2019). “What is mixed reality?,” in Proceedings of the 2019 CHI conference on human factors in computing systems, Ann Arbor, MI, May 17, 2019, 15, 1–15. 10.1145/3290605.3300767
120
TangS.TangL.WuZ.ZhengW.ChenC. (2016). “A conceptual workflow for BIM based prefabrication design visualization with augmented reality,” in Proceedings of the 16th international conference on construction applications of virtual reality, Hong Kong,HK, December 2016, 376–384.
121
TedeschiA.BenedettoF. (2017). A real-time automatic pavement crack and pothole recognition system for mobile Android-based devices. Adv. Eng. Inform.32, 11–25. 10.1016/j.aei.2016.12.004
122
TernierS.KlemkeR.KalzM.van UlzenP.SpechtM. (2012). ARLearn: augmented reality meets augmented virtuality. J. Univ. Comput. Sci.18 (15), 2143–2164.
123
The Mixed Reality Toolkit (2020). Available at: https://rb.gy/93r4nf.
124
ThomasB.PiekarskiW.GuntherB. (1999). “Using augmented reality to visualise architecture designs in an outdoor environment,” in International Journal of Design Computing Special Issue on Design Computing on the Net (DCNet), Mawson Lakes, SA1 (4), 2.
125
ThompsonC. (2019). Inventor Alex Kipman’s grand vision for how holograms will change our lives. Washington, DC: Smithsonian MagazineAvailable at: https://www.smithsonianmag.com/innovation/alex-kipman-grand-visions-how-holograms-change-lives-180973489/.
126
Unity (2021). Unity user manual (2019.4 LTS). Available at: https://docs.unity3d.com/Manual/index.html
127
VTT (2020). Building and infrastructure. Available at: https://rb.gy/2itv2g.
128
WangC.ChoY. K.GaiM. (2013a). As-is 3D thermal modeling for existing building envelopes using a hybrid LIDAR system. J. Comput. Civil Eng.27 (6), 645–656. 10.1061/(ASCE)CP.1943-5487.0000273
129
WangJ.HouL.WangY.WangX.SimpsonI. (2015). “Integrating augmented reality into building information modeling for facility management case studies,” in Building information modeling: applications and practices (Reston, Virginia: American Society of Civil Engineers (ASCE)). 279–304. 10.1061/9780784413982
130
WangX. (2009). Augmented reality in architecture and design: potentials and challenges for application. Int. J. Archit. Comput.7 (2), 309–326. 10.1260/147807709788921985
131
WangX.DunstonP. S. (2006). Compatibility issues in augmented reality systems for AEC: an experimental prototype study. Autom. Constr.15 (3), 314–326. 10.1016/j.autcon.2005.06.002
132
WangX.DunstonP. S. (2013). Tangible mixed reality for remote design review: a study understanding user perception and acceptance. Visual. Eng.1 (1), 8. 10.1186/2213-7459-1-8
133
WangX.DunstonP. S. (2008). User perspectives on mixed reality tabletop visualization for face-to-face collaborative design review. Autom. Constr.17 (4), 399–412. 10.1016/j.autcon.2007.07.002
134
WangX.KimM. J.LoveP. E. D.KangS. C. (2013b). Augmented reality in built environment: classification and implications for future research. Autom.Constr.32, 1–13. 10.1016/j.autcon.2012.11.021
135
Web of Science (2020). Available at: https://rb.gy/va0yqz.
136
WebsterA.FeinerS.MacIntyreB.MassieW.KruegerT. (1996). Augmented reality in architectural construction, inspection, and renovation. Proc. ASCE Third Congress on Computing in Civil Engineering (New York, NY), 913–919.
137
WenM. C.KangS. C. (2014). Augmented reality and unmanned aerial vehicle assist in construction management. In Proc. ASCE Third Congress on Computing in Civil Engineering1, 1570–1577.
138
WoodwardC. (2015). Mobile augmented reality for city planning. Smart City, 28–29.
139
WuL. (2015). Applications of computer vision technologies of automated crack detection and quantification for the inspection of civil infrastructure systems. Orlando, FL: University of Central Florida.
140
XuJ.DoyleD.MoreuF. (2020). State of the art of augmented reality (AR) capabilities for civil infrastructure applications (in submission). Autom. Const.1 (3), 17,
141
XuS.WangJ.ShouW.NgoT.SadickA. M.WangX. (2020). Computer vision techniques in construction: a critical review. Arch. Comput. Methods Eng.1, 3. 10.1007/s11831-020-09504-3
142
YabukiN.MiyashitaK.FukudaT. (2011). An invisible height evaluation system for building height regulation to preserve good landscapes using augmented reality. Autom. Constr.20 (3), 228–235. 10.1016/j.autcon.2010.08.003
143
YamaguchiT.KandaM.ShibuyaT.YasojimaA. (2019). “Crack inspection support system for concrete structures using head mounted display in mixed reality space,” in 2019 58th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE). Hiroshima, OA, September 10–13, 2019 (IEEE). 791–796. 10.23919/SICE.2019.8859876
144
YanL.FukudaT.YabukiN. (2019). Integrating UAV development technology with augmented reality toward landscape tele-simulation.
145
ZhangY.ZhangH.ChenX. (2020). Augmented reality technology: research situation and key technologies. Lecture Notes Electr. Eng.588, 986–1004. 10.1007/978-981-32-9437-0_102
146
ZhaoD.LucasJ. (2015). “Virtual reality simulation for construction safety promotion,” in Int. J. Inj. Cont. Saf. Promot.22 (1), 57–67. 10.1080/17457300.2013.861853
147
ZhouY.LuoH.YangY. (2017). Implementation of augmented reality for segment displacement inspection during tunneling construction. Autom. Constr.82, 112–121. 10.1016/j.autcon.2017.02.007
148
ZollmannS.HoppeC.KlucknerS.PoglitschC.BischofH. (2014). “Augmented reality for construction site monitoring and documentation,” in Proceedings of the IEEE, Graz, AT, January20, 2014 (IEEE), 137–154.
149
ZollmannS.KalkofenD.HoppeC.KlucknerS.BischofH.ReitmayrG. (2012). “Interactive 4D overview and detail visualization in augmented reality,” in 2012 IEEE international symposium on mixed and augmented reality (ISMAR), Atlanta, GA, November 5–8, 2012 (IEEE).
150
ZydaM. (2005). From visual simulation to virtual reality to games. Computer38 (9), 25–32. 10.1109/MC.2005.297
Summary
Keywords
augmented reality, civil infrastructure, construction, building information models, structural health monitoring
Citation
Xu J and Moreu F (2021) A Review of Augmented Reality Applications in Civil Infrastructure During the 4th Industrial Revolution. Front. Built Environ. 7:640732. doi: 10.3389/fbuil.2021.640732
Received
12 December 2020
Accepted
17 February 2021
Published
02 June 2021
Volume
7 - 2021
Edited by
Branko Glisic, Princeton University, United States
Reviewed by
David Lattanzi, George Mason University, United States
Rebecca Napolitano, Pennsylvania State University (PSU), United States
Dryver R Huston, University of Vermont, United States
Updates
Copyright
© 2021 Xu and Moreu.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Fernando Moreu, fmoreu@unm.edu
This article was submitted to Structural Sensing, a section of the journal Frontiers in Built Environment
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.