A Review of Augmented Reality Applications in Civil Infrastructure During the 4th Industrial Revolution

The 4th industrial revolution started in 2016 and referred to a new phase in the industrial revolution. One of the most significant technological evolvements during the 4th industrial revolution is Augmented Reality (AR) technology. AR superimposes interactional virtual objects/images to real environments. Because of the interaction and see-through characteristics, AR is better applied to engineering than Virtual Reality (VR). The application of AR in civil infrastructure can avoid artificial mistakes, improve efficiency, and saves budget. This article reviews AR applications in civil infrastructure, focusing on research studies in the latest five years (2016–2020) and their milestone developments. More than half of the AR research and implementation studies have focused on the construction domain in the last five years. Researchers deploy AR technologies in on-site construction to assist in discrepancy checking, collaborative communication, and safety checking. AR also uses building information models (BIMs) to produce detailed 3D structural information for visualization. Additionally, AR has been studied for structural health monitoring (SHM), routine and damage detection, energy performance assessment, crack inspection, excavation, and underground utility maintenance. Finally, AR has also been applied for architecture design, city plan, and disaster prediction as an essential part of smart city service. This article discusses the challenges of AR implementation in civil infrastructure and recommends future applications.


INTRODUCTION
The 4th industrial revolution is generally believed to start in 2016 (More Ickson and ben Dhaou, 2019). However, the exact starting point is still debatable because it is still an ongoing event. Different countries also tend to have different definitions of the starting time. For instance, German is widely accepted as one of the first countries entering the 4th industrial revolution since 2011 (Pitsis et al., 2020).
Caudell and Mizell proposed the official term "'Augmented Reality" in 1990 (Berryman, 2012). Milgram and Kishino (1994a) defined the concept of "Virtuality Continuum" and clarified the concept of Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), and Augmented Virtuality (AV). Figure 1 shows the classification of these four terms. The real environment is the physical surroundings that the user stands in. The virtual environment is computer-generated objects/information made up of laser/light. With VR, a user completely immerses in an entirely virtual environment and sees no more real objects in the surroundings (Burdea and Coiffet, 2003;Zyda, 2005); while with MR, a user sees both objects in the real surroundings and virtual images (Milgram and Kishino, 1994b;Ohta and Tamura, 2014;Speicher et al., 2019). MR composes of AR and AV. AR displays the real environment enhanced by interactive virtual objects (Milgram and Kishino, 1994a;Carmigniani et al., 2011). AV is to project and control real objects in a virtual world (Ternier et al., 2012;Nahon et al., 2015). Figure 2 shows the trend of AR implementation in civil infrastructure. The data in Figure 2 are from the number of research articles on the Web of Science (2020). In the last five years (2016-2020), AR has undergone rapid development and implementation in civil infrastructure, compared with the early years (Petrillo et al., 2018;Aoyama, 2019;Cattari et al., 2019;Thompson, 2019). A general and comprehensive overview of the AR implementations in civil infrastructure is needed to invoke more profound AR applications. This article provides a timely and general review of recent AR studies in civil infrastructure and a new application classification. Figure 3 shows the distribution of the existed publications in different countries/regions (same data with Figure 2.). 40% of the studies are from the United States and China, 24 and 16%, respectively. Figure 3 shows that the United States and China are leading the 4th industrial revolution in the AR area (More Ickson and ben Dhaou, 2019; Pitsis et al., 2020). This conclusion is consistent with recent research studies (Petrillo et al., 2018;Bayode et al., 2019). Bayode et al. (2019) claimed that the United States and China had introduced initiatives albeit with different objectives to support the modernization of the industrial    production sector. Petrillo et al. (2018) stated that the economic opportunities of Industry 4.0 are vast and affect the entire economies and countries, mainly including the United States, China, Germany, and the United Kingdom by 2030.
There are two kinds of classification methods for AR implementation in civil infrastructure (Wang et al., 2013): by subjects (building structures, bridges, and railways) or by application scenarios (architecture design, structural design, construction, SHM, and others) (Shin and Dunston, 2008). This article deploys the classification of different AR technologies by application scenarios. Figure 4 shows the percentage of AR applications in civil infrastructure between 2016 and 2020, divided into five significant branches: construction, building information model (BIM), structural health monitoring (SHM) and damage detection, underground utilities, and smart city management. The percentage is calculated by dividing the corresponding category's publication amount in the last five years by the same period's total publication number. The statistics ended on December 31st, 2020. The total publication number in this period is 95. Nearly half of AR implementations in civil infrastructure have focused on the on-site construction (45%), followed by SHM and damage detection (20%), BIM (19%), underground (10%), and smart city management (6%). A comprehensive review of AR studies in civil infrastructure education can be found in Diao and Shih (2019), but it was not relevant as a topic for this article.
Researchers should consider comprehensively when selecting the suitable AR platform (Julier et al., 2000;Jahn et al., 2018;Mahmood et al., 2018). There are two major categories of AR systems in general. One is the prototypes developed in labs, and the other one is the commercial AR devices in the market. The prototypes are typically designed for solving specific research problems efficiently but typically lack versatility. Unless a researcher decides to design a particular AR system for a specific research project, a suitable commercial AR device is usually a better choice (Roitman et al., 2017;Wang and Dunston, 2013). When selecting an AR device, all the parameters are classified into four major categories, general properties, sensors, computational capacities, and display capacities. The general properties include weight, price, and product durability (e.g., water resistance, working temperature range, working humidity range, and drop safe distance). Sensors include head-tracking, eye-tracking, depth sensor, GAM (gyroscope, accelerometer, and magnetometer), and GPS sensor. Researchers should also consider processing units, RAM, and storage to fulfill the computational requirements. Connectivity (e.g., Wi-Fi, Bluetooth, and USB) and display capacities (e.g., FoV and camera) are also important considerations. The choice of a suitable AR device is a complect work. Xu et al. (2020) summarized all the parameters and compared sixteen AR devices regarding these parameters. Xu and Doyle et al. (2020) also provide an empirical formula for civil engineers to select a suitable AR device. Aguero et al. (2020) applied the AR device selection procedure and connected AR with sensors in a shaking table experiment. Hossain et al. (2020) used a similar selection procedure and conducted AR modal shape analysis.
In the following sections, the authors reviewed the milestone AR implementations in construction, BIM, SHM and damage detection, underground utilities, and smart city management, respectively. Following the review, this article discusses the current progress and technical challenges of AR implementation in civil infrastructure. Finally, this article concludes and predicts the future AR development direction in civil infrastructure.

CONSTRUCTION
One of the salient features of AR technology is the combination of virtual and real objects. Traditionally, VR was deployed for the construction process visualization (Sacks et al., 2013;Zhao and Lucas, 2015). The pith of VR implementation in construction is the detailed 3D-CAD modeling of both the project and the surroundings (Sampaio and Martins, 2014). Detailed environment information input is also required for VR implementations. However, the requirements of detailed models become impractical with the increasing complexity of the construction procedure (Feeman et al., 2018). With AR technology, the virtual design blueprint and real construction site can be combined and illustrated to engineers and workers in real time, resulting in higher precision and efficiency (Ahmed, 2018). AR does not require detailed modeling of the surroundings or the existing project process. Applying AR in the lifecycle construction site planning can help keep the project within budget and to avoid process mistakes or complement time.
In this study, the authors classified all the AR implementations in construction nowadays into two major research areas to simplify the acknowledgment for the readers, as shown in Figure 5. The two major research areas are monitoring project progress and assisting workers. Studies of project progress monitoring can be divided into two categories, discrepancy checks and quality control. The second research area, the assistance for workers, includes collaborative visualization, telecommunication, and site safety.

Monitoring Project Progress
The significant practical construction challenges of the AR applications are the complicated and arduous outdoor environment, for example, lighting conditions and terrain variation. The monitoring project progress research has been conducted for both hardware and software development.

Discrepancy Check
A primary AR application in construction is the discrepancy check (Son et al., 2015), that is, checking the difference between on-site structures and design models. The first systematic AR implementation for discrepancy check is MARSIFT (Hammad et al., 2002). However, MARSIFT is a promising concept rather than a viable application. Since the development of MARSIFT, AR implementation in civil construction has been increasing, both in theoretic studies (pose estimation, scene understanding (Feng, 2015), and accurate 3D reconstruction of the real environment (Fathi and Brilakis, 2016)) and on-site applications. There are mainly two practical AR implementations in the discrepancy check area, SMART-ARMOR and D4AR.
SMART-ARMOR was developed based on UM-AR-GPS-ROVER (Behzadan and Kamat, 2005). UM-AR-GPS-ROVER's ability to display virtual mechanicals in an outdoor construction site has been verified with a field experiment. In 2008, Behzadan, 2008;Behzadan et al., 2008;Behzadan and Kamat, 2009a presented ARVISCOPE-ROVER as an updated version of UM-AR-GPS-ROVER. The Global Positioning System (GPS) and orientation tracking technologies were applied in ARVISCOPE to create a dynamic simulation of the models' continuous animations. In the following years, Dong and Kamat, 2010;Kamat et al., 2011 evolved the ARVISCOPE-ROVER system to the more accurate and lighter SMART-ARMOR system. SMART-ARMOR extended the system to a more standard AR development environment by separating AR logic from the application logic part. SMART-ARMOR is a milestone holistic and applicable outdoor construction application of AR technologies.  Golparvar-Fard et al. (2011b), applying the registration method to solve the discrepancy check problem. Later, D4AR was updated to the HD4AR system (Bae et al., 2013), enabling the application of available camera-equipped mobile devices. In 2014, bottomed on the D4AR system, manual occlusion management, and "X-ray" metaphors (Zollmann et al., 2012;Karsch et al. (2014) combined a set of AR tools to build a valuable platform, ConstructAide, to monitor project progress. ConstructAide combined unordered photo collections and 3D models to obtain construction progress.
Along with the theoretical development, researchers also conducted field experiments. For instance, by solving the alignment and plumbness problem, the discrepancy check of steel pipe modules assembly was solved by Nahangi, 2015;Nahangi et al., 2015 andJeanclos et al., 2018;Kwiatek et al., 2018;Kwiatek et al., 2019;Li et al., 2018, capable of extending more general applications.

Quality Control
Quality control is another critical AR application in construction. Zhou et al. (2017) inspected the on-site segment displacement during tunneling construction compared with the baseline model to monitor construction quality. Aguero et al., 2020 connected AR devices and sensors, taking advantage of both the displaying function of AR technology and sensors' inspecting function. Figure 6 shows the design procedure of connecting AR devices and sensors. Figure 7 shows an illustrative experiment that Aguero et al., 2020 conducted to verify the AR-sensor connection.

Assisting Workers
High-efficiency collaborative visualization and communication among multiple users can be realized by applying AR technologies. Dong et al. (2013) developed an AR application, ARVita, for the collaborative visualization of multiple users wearing HMDs. Using ARVita, all the uses can interact with the dynamic visuals of the real-time construction processes. Soman et al. (2017) developed an application to share AR images between designers and the field site. Hammad et al. (2009) developed a prototype distributed AR visualization collaborative system, DARCC, which was tested in a bridge deck rehabilitation project to show the effectiveness.

Telecommunication
Telecommunication can provide high-performance guidance for workers. In 2003, Aiteanu et al. (2003) invented an AR helmet to provide an online welding assistant to improve welders working conditions. Chi et al. (2012) designed an AR interface to teleoperate remote cranes by retrieving information from the field and enhancing the decision-making processes.

Site Safety
Another significant application is to guarantee site safety by visualizing complex workplace situations with AR devices. . Kim et al. (2017) developed an AR system to provide workers safety information to avoid hazards. Liu et al., 2018 deployed AR technologies to give the workers alarms of dangers from falling. The authors are committed to the implementation of AR to increase the safety of workers. The authors used AR to mark the bridge components for the inspectors, as shown in Figure 8. The authors are also designing an AR application using eyetracking data to draw a cognitive map for Federal Railroad Administration (FRA) to increase railroader safety during field inspections.
In the future, AR construction research trends are real-time monitoring and the human factor. A construction site is a dynamic and fast-changing scenario. Hence, to effectively monitor real-time site information, the fast-response requirements for AR algorithms and devices are essential.
Real-time monitoring has become a hot topic for AR construction. Zollmann et al. (2012) developed an application using overview and detail techniques to visualize real-time dynamic data for construction sites with AR. Behzadan and Kamat (2011) furthered the ARVISCOPE-ROVER system's study to visualize 3D AR animations of active engineering operations. Akhavian and Behzadan (2012) developed a framework driven by data to collect real-time field data and visualize dynamic 3D simulations of engineering activities. Liu et al. (2018) developed a rapid construction site information acquisition method to conduct dynamic site layout planning. Besides overcoming technical difficulties, performance-based   (2006), Wang and Dunston (2008), and Dunston and Wang (2011) analyzed the feasibility of considering human factors in AR applications to develop visualization tools for construction tasks.
Most research and devices were in the experiment and prototype phase yet not applied to practical site construction. The main obstacle to adopting AR technologies during construction was the high expense and immature devices (Manuel Davila Delgado et al., 2020). Developing rapidly in recent years, AR devices suitable for construction are already commercially available. An eligible AR device can be selected by appropriate performance requirements for a different construction site. For example, the RealWear HMT-1Z1 should be chosen for high-risk (e.g., potentially explosive areas) and high-value construction for its safety and high price. At the same time, Google Glass Enterprise Edition 2 is fit for daily civil infrastructure for its lightweight. Besides HMDs and handheld displays, the combination of unmanned aerial vehicle (UAV) and AR technologies to conduct holistic site planning and large-scale terrain scanning is a significant trend. Wen and Kang (2014) enhanced and prototyped a UAV with AR technologies to provide site-level image transmission. Zollmann et al. (2014) used UAV to conduct 3D reconstruction of site information, providing on-site visualization on mobile. Yan et al. (2019) combined UAV and AR to run landscape simulations. Commercial AR-UAVs for construction could be the next important component of the AR market.
Another challenge is workforce training. Since AR technology is new to most construction workers, efficient training is necessary to implement AR technology in a real construction project. The workers will need to understand basic AR knowledge and AR device operating methods.

Building Information Models
Many researchers classify BIM as an application in construction because of the extensive implementations of BIM in construction. Nonetheless, given the significance, ARenhanced BIM is classified as a separate category in this article. Combined with AR technologies' visualization potential, BIM can perform as a functional decision support system (Chen et al., 2015;Machado and Vilela, 2020). A holistic bibliometric literature review of BIM between 2005 and 2015 can be found in Santos et al. (2017) and Abdirad (2017), concluding that AR-enhanced BIM is one of the main trends of practical BIM application. This section mainly focused on the research trend after 2015.
BIM is a functional tool for structural information management. Combined with AR technologies, visualization and intuition are enhanced for displaying information to engineers. The merging of AR and BIM maximized both the AR's visualization advantage and BIM's information management advantage . In 2014, Meža et al. (2014) conducted studies on information storage and regeneration and presented a prototype software system tested in a real project. In 2018, Bruno et al. (2018) conducted a diagnosis and performance assessment for existing buildings, finding that information management is crucial for AR-enhanced BIM. In 2020, Dang and Shim (2020) proposed an innovative bridge maintenance system  using AR-enhanced BIM to provide bridge maintenance information for engineers. (Hossain et al., 2020) used AR technology to overlap model shapes onto the real specimen in a shaking table experiment, as shown in Figure 9. The model shape of the specimen was changed and displayed in real time, with the real specimen moving.
Providing systematic information management, defects, and rework during the construction process can be minimized, hence guaranteeing project schedule and cost budget, verified by a quantitative analysis conducted by Hwang et al. (2019). Kim et al. (2015) combined BIM and 3D laser scanning technology to perform quality monitoring for concrete elements. Schickert et al. (2018) conducted nondestructive testing for the concrete specimen with BIM framework and AR-enhanced visualization.
Many AR-enhanced BIM applications are now available for holistic lifecycle management and specialized construction. For example, Tang et al. (2016) proposed a conceptual workflow with the BIM-GIS application for the prefabrication design to save construction time, increase efficiency, and reduce energy consumption; GhaffarianHoseini et al. (2017) suggested engagement of a BIM application, BIM-IKBMS, for the building management after construction.
For the application of standard software, Abanda et al. (2015) compared 122 BIM software applications for project management. Considering that each software has pros and cons, and different software can complete similar targets, Abanda et al. (2015) suggested selecting BIM software should consider supply chain partners and the project's lifecycle. Barazzetti and Banfi (2017) reviewed commercially available BIM software for historic buildings and suggested that the BIM applications for handheld AR devices, A360 (Autodesk (n.d.), 2016), iVisit3D (Artlantis Studio (n.d.), 2019), BIMx (Graphisoft, 2011), and Augmented 3D (2020), best fit AR technologies. Kopsida and Brilakis (2017) evaluated different software designed for AR-enhanced BIM applications, concluding that a gap still exists between AR-enhanced BIM technologies and practical on-site applications. Rahimian et al. (2020) proposed an AR-enhanced BIM framework to integrate structural elements' visualization information using Unify.
The authors are developing the interface of finite element models (FEMs) in AR for hybrid experiments (Figure 10.) The user can interface with a hologram of one virtual FEM experiment using a shaking table input created by the computer. The user can define the mass, stiffness, and height of each story in the interface. In a hybrid experiment, some structural components' movement is calculated in the computer, while others are tested using a real shaking table (Figure 11). The AR interface can superpose the virtual components onto the real ones, showing a whole structure with consistent and intuitive dynamic moving. More complicated BIM models will be developed based on this prototype.
Among all the HMD devices, Microsoft HoloLens 2 has the enormous potential to be the game-changer for the AR-enhanced BIM application. First, HoloLens 2 is embedded with a robust depth sensor. Hence, it is capable of 3D monitoring. Second, the programming platform of HoloLens 2, Unity, is relatively completed and widely accepted by engineers. Last, HoloLens 2 is compatible with most general civil infrastructural models, for example, Autodesk 3D model and Trimble SketchUp model. A detailed comparison of the available AR devices is given in Xu et al., 2020.

Life Cycle Management
AR technologies can also be used in the energy performance assessment of a building. Traditional energy performance FIGURE 10 | AR application-finite element models.  (2015) provided another novel algorithm with quadrilateral features registration (e.g., windows) for building's thermal monitoring.

Structural Health Monitoring and Damage Detection
AR-based SHM is more suitable for routine inspection and monitoring than other tasks, for example, special inspection, testing, and maintenance (Peres et al., 2018). AR technologies are mainly used in the infrastructure SHM and crack inspection. The safety of bridges is of great significance during heavy transportation today. More and more bridges are reaching the designed service year corresponding to the increasing safety requirements, leading to massive SHM and detection necessities (Moreu, 2015;Casas and Moughty, 2017). Traditional manual monitoring methods can hardly fulfill the rapidly increasing monitoring tasks. To improve monitoring efficiency, Akula et al., 2015 conducted SHM for highway bridges using a context-aware computing framework. The framework automatically identified the component of interest and retrieved relevant information from the database. Computer vision techniques are functional for crack detection during SHM. Wu (2015) developed a novel self-adaptive image-processing algorithm for the crack detection of road pavements. Lattanzi et al. (2016) conducted seismic damage detection for bridge columns. Computer vision techniques were used to estimate the peak displacement during the earthquake by damage observations from images captured by cameras.
SHM and damage detection are essential for critical infrastructure during disaster and post-disaster. In Section 2, the application of AR technologies for discrepancy check during construction has been introduced. Similar platforms can be applied for post-disaster damage detection. By comparing the post-disaster real structure and previously stored AR building information, structural damage can be observed (Kamat and El Tawil, 2007). Tedeschi and Benedetto (2017) designed an automatic recognition system to inspect pavements for potholes, longitudinal-transversal cracks, and fatigue cracks for pavements. Dang et al. (2018) developed 3D digital twin models with related damage records for typical bridge structures as a database to predict a similar structure's reliable future performance. Napolitano et al. (2019) developed a novel AR framework for data visualization during the structural inspection.
With the improvement of the display, cameras, and depth sensors of HMDs, conducting SHM and damage detection with HMDs has been studied. Yamaguchi et al. (2019) developed an AR crack inspection support system for concrete structures with HoloLens. Maharjan et al. (2019) adopted HoloLens to conduct bridge inspection in a controlled laboratory environment and found that applying AR technologies in SHM leads to higher safety and a smaller workload for field inspectors.
Napolitano et al., 2019 used AR to conduct structural health monitoring, as shown in Figure 12. AR technology helped to capture the real environment (the building information) in the field ( Figure 12A) and reanchoring the image in a new position ( Figure 12B).
The authors developed an AR application named "Time Machine Measure (TMM)" to assist in SHM. Traditionally, researchers need to use markers to measure over time if an object's moving distance needs to be measured. TMM provides a time machine function using the real environment's spatial mapping, restoring its position at a previous timestamp. Therefore, researchers can directly measure the moving distance of an object without tedious and time-consuming marking procedures. Figure 13 shows an illustration of measurement with TMM. In Figure 13A, the green cube is the original position of the cube storage (Position-1), and the blue one is the position after a move (Position-2). The blue cube is real, while the green one is an invoked virtual image marking the original position. With this function, the moving distance can be conveniently measured as 0.91 m. Figure 13B shows that the cube is moved again to Position-3 (the cyan cube). Similarly, the cyan cube is real, while the blue and green ones are invoked virtual images. Figure 13B shows that the moving distance from Position-1 to Position-3 is 1.88 m. This measurement assistant AR application simplifies the SHM work for field inspections.
During SHM and damage detection, functional depth sensor and high-resolution cameras are requisite to HMDs. The only devices meeting both requirements are HoloLens and ThirdEye Gen X2 (Xu et al., 2020). The larger 52°FOV makes HoloLens a more popular device for researchers. However, studies of HoloLens applied in SHM are still in the experimental phase. More on-site field experiments should be conducted to verify reliability. Besides HMDs, a novel AR projector, LF2, could also be a game-changer for damage detection if the reliability under strong sunlight can be improved. As a projector, LF2 is not limited by FOV, processing, or storage capacity.

Underground Utilities
Underground utilities encompass underground geological structures (e.g., basements, foundations, garages, tunnels, and subways) and pipelines (e.g., water, sewage, gas, electricity, and telecommunications) (Bobylev, 2009). The application of AR technologies superimposed virtual underground information to the real aboveground environment, preventing accidental utility strikes during excavation and improving efficiency in maintenance (Kaddioui et al., 2019).
The major challenge is the localization problem (Pereira et al., 2020), that is, how to guarantee the accuracy of the superimposition to deploy AR technologies. Traditionally, abstaining location information required merging data from blueprints, historical records, dead reckoning, and aboveground sensors, which are time-consuming and lack accuracy (Li et al., 2015). The leading solutions today to solve the localization problem consist of utilizing the GPS and ground penetrating radar (GPR) (Kaddioui et al., 2019;Ortega et al., 2019). Dai et al. (2011) proposed an analytical approach compiled with GPS to incorporate virtual underground infrastructure into site photos to provide more invisible information. Pereira et al. (2018); Pereira et al. (2019) provided a novel GPR system integrated with AR to solve the localization problem. The proposed GPR system was capable of arbitrary-route scanning with high fidelity.
With the development of HMDs and handheld devices, the information traditionally stored in office-based desktop computers can be shown directly to engineers in the field. Webster et al. (1996) first developed an AR HMD system for architectural construction. The system had the see-through characteristic and could show the columns occluded by walls and rebars inside the columns. Behzadan and Kamat (2009b) combined the HMD device with GPS and CAD models to superimpose real-time virtual underground utilities over the real environment on an excavation site. Schall et al. (2010) designed a novel mobile AR device, VIDENTE, to maintain, plan, and survey underground infrastructure using spatial interaction and visualization techniques. Improved from Vesp'R (Schall et al., 2009), VIDENTE was a relatively complete and functional AR device, getting positive feedback from utility companies. Roberts et al. (2002) developed an AR see-through system compatible with HMDs and handheld laptops, solving the Tracker Technology positioning problem.
Bentley Systems developed a relatively mature and available prototype application for the AR-enhanced underground utility visualization software. The application combined AR, GPR, and BIM and can measure the distance between underground objectives.

Smart City
The official definition of "smart city" has not been universally accepted. However, a common understanding is conducting city management by collecting information from residents and the IoT (Kyriazis et al., 2013;Poslad et al., 2015;Giang et al., 2016). AR is playing a big part in smart city service, for example, public transport (Novi Sad, Serbia) based on the IoT devices installed on buses (Pokric et al., 2014). Golparvar-Fard et al. (2015) and Balali (2015) proposed a novel algorithm to monitor and preserve the high-quantity and low-cost assets on highways (e.g., traffic signals and light poles). Besides transportation, AR technologies have also been deployed in the city plan and architecture design. Early in 1999, Thomas et al. (1999) designed an AR platform, TINMITH2, to visualize the architecture design in real surroundings. Due to computation limits, the architecture was depicted with single lines. Wang (2009) considered using AR technologies for architecture design in 2009. However, due to the technology limit, a practical 3D rendering prototype application was not completed. The primary technology limit is the transition from legally approved 2D design drawing to well-structured integrated 3D models readily for AR systems. The maturity of 3D BIM design promoted the application of AR in architecture design. For example, in 2013, Côté et al. (2013) interpreted 2D engineering drawings to the 3D real environment directly with an animated sliding plane and model clipping technique. The mobile AR application developed by VTT Technical Research Center of Finland (VTT, 2020) was the first practical application, helping plan a hotel in Billnäs ironworks site at Raseborg, Kämp Tower in Jätkäsaari, Helsinki, and a wind generator farm in Pörtom, Närpes (Woodward, 2015).
Traditionally, existing structures and natural objects need to be modeled in 3D CAD during city planning to illustrate the relative position and height relation, which is time-consuming and money-costing. AR is a powerful tool for both landscape and height design during a city plan. In 2011, to preserve historic buildings' good landscapes and provide a reasonable basis for city height regulations, Yabuki et al. (2011) developed a feasible and effective system deploying ARToolKit to measure the maximum invisible height of each building. A typical example illustrating the AR city plan is conducted in Christchurch, New Zealand. A mobile AR application, CityViewAR (no longer available), was used to assist the post-quake city plan and reconstruction of destroyed buildings for the first time (Lee et al., 2012).
Satellites are widely implemented to obtain data for building smart cities. For example, DSCOVR satellite (Deep Space Climate Observatory satellite), launched on February 11th, 2015, can monitor the real-time solar wind in the Unites States, hence to the accuracy forecast weather alerts (NOAA, 2015). The authors developed an AR application, SMAC, to monitor and control the satellites, including the position (latitude and altitude), velocity, and other data collected by the satellites, as shown in Figure 14. Until now, SMAC can access three satellites, SHARC, DSCOVR, and the International Space Station. SMAC can assist smart city management with access to more satellite data, such as traffic monitoring data and communication network data.
HoloLens and Unity platform were further investigated among all the available AR devices. Gang et al. (2016) developed a toolkit by Unity to conduct water hazard prediction by monitoring realtime precipitation and hydrological information. Ergün et al. (2019) tested HoloLens by 21 participants in an in-house workshop, showing the potential of using HoloLens in ARassisted architecture design. As an essential part of the smart city, the rapid development of applying HoloLens in city plan and architecture design is foreseeable.

DISCUSSION
The authors generate a synthesis comparison table to summarize the current progress and technical challenges for each AR implementation category in civil infrastructure. As discussed in Sections Construction and Other Implementations, AR implementations in civil infrastructure can be divided into construction, BIM, SHM, underground utilities, and smart city management. Implementations in construction are further divided into monitoring project progress and assisting workers. With this comparison table, readers can generate a thorough understanding of AR implementations in civil infrastructure today and compare their study to the frontiers in the corresponding area. References in Table 1 are the typical studies in the correlated category. Comparably, AR technologies in discrepancy check, quality control, BIM, and SHM are more mature. Multiple prototypes have been designed, and smallscale field experiments have been conducted.
There are three fundamental technologies in AR implementations, display technology, tracking registration technology, and human-computer interaction technology . In the three fundamental AR technologies, five significant technical challenges hinder the AR development in civil infrastructure today are as follows: 1) Occlusion problem. Occlusion problem is one of the two major fundamental technical issues in the various AR implementations in civil infrastructure. Occlusion problem exists in every mobile AR application and HMD, limiting the augmented range and accuracy. The standard solution for the occlusion problem is to pre-map the surroundings to avoid solving the real-time occlusion problem. However, researchers have proposed more acceptable methods for the real-time occlusion problem (Holynski and Kopf, 2018). Applying recent theories to AR devices can motivate more AR implementations, although the occlusion problem still has a long way to the perfect solution. 2) Latency problem. Another major fundamental challenge for AR technology is the latency problem, that is, the time-delay problem. The time delay of the virtual environment calculation can cause a displacement between the real and virtual environments. Two display methods exist when merging the real and virtual environments, optical see-through and video see-through. The optical see-through method displays the real environment through a glass with virtual objects showing on the glass (e.g., Microsoft HoloLens 3) The see-through video method captures the real environment by a camera and shows both the images in the real environment and the virtual objects on the same monitor screen. The synchronization problem of the video see-through display method can be solved by adding a time delay for the real environment on purpose to offset the virtual environment (Kanbara et al., 2000). However, time delay cannot be eliminated entirely for the optical see-through method but can only be minimized by accelerating the virtual environment's calculation (Kiyokawa et al., 2003). 4) Outdoor environments. Today, most AR devices, including both prototypes and commercial devices, use lasers to overlap virtual images onto the real environment. Lasers work well and stable in indoor environments. However, the temperature and luminance changes in outdoor environments can be a challenge to most AR devices. For instance, Microsoft HoloLens 2, as a powerful and comprehensive AR device in the market, still inherits the intrinsic disadvantages of the working environment.
a) HoloLens uses visible light to track the user's location and the surroundings, hence requiring a steady 500-1000 lux luminance for best performance. If the environment is too bright, the cameras can get saturated, and nothing can be seen. If the environment is too dark, the cameras cannot pick up enough information, and nothing is seen (Microsoft, 2019). 500 lux luminance is the light of typical indoor environments, for example, offices, groceries, and laboratories. 1000 lux luminance is the light of bright indoor environments, for example, detailed mechanical workshops and operation theaters. b) HoloLens operates with the best performance in the environments within the temperature range of 10-27°C (50-81°F) and the humidity range of 8-90% relative humidity. In the authors' experiments, HoloLens can operate in environments out of these ranges. However, implementations out of the recommended range may lead to shorter battery life or device damage. c) Localization problem is also a challenge for an outdoor environment. AR devices need to scan the surroundings using a depth sensor to generate a detailed spatial map so that virtual images can be displayed in an accurate position.   However, in an outdoor environment, the surroundings often lack adequate references like an indoor wall. Some AR devices choose to deploy a GPS sensor to solve the localization problem. 5) Accuracy requirement. To generate realistic virtual images, the positions of the virtual images have to be very accurate.
A relatively large position error will lead users to doubt the reality of the AR images and cause motion sickness, dizziness, and nausea. 6) Communication stability and efficiency requirement. AR devices are designed to be able to communicate with each other. This function is beneficial for assisting workers and providing remote directions. In this scenario, the stability and efficiency of the communication are vital. Unstable communication and large latency can cause danger to workers. The speed and quality of remote communications typically depend on the wireless network. Therefore, a sound and reliable local wireless network is necessary for AR communication. Frontier computer vision and graphics research provide more satisfying solutions for the five previously mentioned AR technical challenges. Today, frontier computer vision research includes four aspects: data collection, low-/mid-level processing, high-level processing, and application (Xu et al., 2020). The frontier data collection technologies try to design a good data collection strategy, for example, layout plan to adverse environmental noise. The frontier low-/mid-level processing technologies exploit GPUs and parallel computing to reduce computational cost and improve accuracy. The frontier high-level processing technologies are combined with BIM to solve the problem of lacking prior models. Finally, all the frontier computer vision technologies lack implementations and tests in real projects, which have become one of the most considerable weaknesses of computer vision and graphics research.
In another article, the authors classified sixteen AR devices available in the market in the last five years regarding civil infrastructure implementation capabilities. The authors compared AR devices' performances in weight, price, product durability, sensors, connectivity, computational capacities, and display capacities, including the parameters that civil engineers are most interested in, for example, camera resolution, processing unit, and FoV. According to the classification and ranking of that study, Google Glass Enterprise Edition 2, Microsoft HoloLens 2, and Everysight Raptor have the highest combined capabilities to become popular in civil infrastructure. Google Glass Enterprise Edition 2 has the largest FoV, which is essential in construction. Everysight Raptor has the second largest FoV among all the AR devices and the second largest camera resolution. Microsoft HoloLens 2 has all-sided sensors (including depth sensor), a high-resolution display, the third largest FoV, and relatively balanced calculation and storage performances. All the three AR devices have significant potential in the future civil infrastructure implementations.

CONCLUSION
This article reviewed the research field of AR applications in civil infrastructure, mainly focused on research studies in the latest five years and milestone developments. While several other introductory articles have been written on AR applications in civil engineering, this article is more comprehensive and up-to-date. The most extensive research branch is construction with AR, followed by BIM, SHM, underground utilities, and smart city management. Given that quite a few prototype devices and applications have been designed and tested today, few devices are commercially practical for massive on-site applications. Fit devices are recommended for each application branch. Generally, HoloLens has become the most promising AR HMD for civil infrastructure because of its all-sided sensors (including depth sensor), high-resolution display, larger FOV, and balanced calculation and storage performance.
There are three central research tendencies in the future. First, theoretical breakthroughs in AR technology research are vital, including further studies of the occlusion and latency problems. Second, the weight and price of updated version AR devices should be reduced and improve performance. Finally, a unified AR function "toolbox" platform and benchmark models should be established and well accepted in both the academic and engineering community. The complicated applications and applications can be compared and combined to display the respective superiority. Unity has become a more general platform for AR application development. In the future, researchers should conduct more field experiments to verify the prototypes. With the help from the industry, the verified AR applications will then be implemented more extensively. For a civil engineer who is a novice in AR, the authors suggested the engineer starting the AR studies with relatively more mature platforms, for example, Unity 3D. Besides the references discussed in this article, the authors also recommend reading the Unity manual (Unity, 2021) and starting with the programmed MRTK SDKs (The Mixed Reality Toolkit (MRTK), 2020).

ETHICS STATEMENT
Written informed consent was obtained from the individuals for the publication of any potentially identifiable images or data included in this article.

AUTHOR CONTRIBUTIONS
JX collected the data, organized the citations, and drafted the manuscript. FM is the leader of the project and coordinated the access to data, resources, and analysis. FM reviewed the manuscript composition, sources, and the final version. FM secured the funding to conduct this research, access to AR resources, codes, and experiments.