Augmented Reality for Enabling Smart Nuclear Infrastructure

Introduction

The recent emergence of commercially-available augmented reality headsets opens up many opportunities for enabling smart infrastructure. Augmented reality headsets are capable of projecting holograms onto the field of view of a user. An example of an augmented reality headset can be found in Figure 1. This particular augmented reality headset is called the HoloLens from Microsoft [Microsoft, (n.d)]. Contemporary augmented reality headsets typically contain a variety of sensor, communication interfaces, and peripherals. Examples of sensors include RGB/depth imagers, microphones, and inertial measurement units. They often come equipped with Bluetooth and 802.11 networking capabilities. Peripherals can include features such as stereo sound. Augmented reality headsets also often contain some on-board processing capabilities. Some augmented reality headsets have even been certified as protective eyewear (Bardeen, 2017). Interaction with augmented reality headsets is accomplished using voice commands, tracking the gaze of the user, and with gesture recognition. These input modalities make augmented reality headsets attractive for industrial applications because they leave the hands free to perform manual work. Augmented reality headsets also allow for the placement of virtual monitors in any location where the ambient lighting is appropriate for viewing holograms. This capability is convenient in situations where the utility of a monitor offers an advantage, but it is not practical to have a physical monitor due to factors such as space, power, and legal constraints (Surur, 2017). The features of augmented reality headsets open the door to creating truly intelligent industrial infrastructure. This work represents the first known example of a comprehensive framework for smart industrial infrastructure that leverages augmented reality technology. The focus of this work is on nuclear industrial applications, which are particularly demanding.

FIGURE 1

Figure 1. An example of an augmented reality headset [Microsoft HoloLens, Microsoft, (n.d)].

Augmented reality is typically implemented using either a handset or a headset. Augmented reality technologies can support planning, construction, and maintenance of nuclear facilities (nuclear power plants). By superimposing computer generated objects on the real environment and tracking their three-dimensional physical positions, augmented reality allows nuclear facility workers to intuitively interact with the physical and virtual worlds. Specifically, augmented reality technology can effectively navigate workers to a workplace while showing dangerous areas to avoid, and overlaying physical parameters of critical components and environment (e.g., dose rate) on the workers' field-of-view while in the nuclear facilities (Ishii et al., 2007).

Tracking and display techniques that measure the three-dimensional physical coordinates of workers and allows visualization of such information are the enabling technology for augmented reality to become an effective tool for nuclear facilities maintenance and safety tasks. Marker-based tracking technology, which measures the position between the markers (placed on the target object) and the sensors (imagers), has been extensively studied for applications in nuclear facilities (Shimoda et al., 2004, 2005; Ishii et al., 2007, 2014). It was found that traditional square markers were not suitable for nuclear power plants because a large marker size was required for long-distance tracking, which is a common requirement in nuclear facilities. A hybrid circular marker design, which consists of circles of various sizes, was proposed and successfully tested for long- and close-distance tracking in nuclear power plants (Ishii et al., 2007). In addition, to account for the common occurrence of pipe structures in nuclear facilities, a new line marker (barcode) was proposed to further reduce the marker size for practical tracking (Shimoda et al., 2005). Because nuclear facilities are usually very large (tens of meters in length), to expand the tracking field-of-view, a multi-camera tracking system with a novel stereo, hybrid markers-based tracking algorithm was developed. This system provides a 3x wider angle of view than a single camera (Ishii et al., 2007) for wide area tracking (Ishii et al., 2014). Also, various off-the-shelf display devices are studied for their effectiveness and feasibility for use with augmented reality technology in nuclear facilities (Ishii et al., 2007). Recently, a holistic, mixed reality system with a colored finger-tip-based tracking and gesture recognition method coupled with a video-based, see-through head-mounted display was proposed and experimentally studied to improve the effectiveness and reduce intervention time for maintenance and repair tasks in nuclear power plants (Abate et al., 2013).

Augmented reality has also been explored as an emerging communication tool for planning, worker training, and evacuation of nuclear facilities (Yim and Seong, 2010; Tsai et al., 2013; Johnsen and Mark, 2014; Sørensen, 2014). In particular, augmented reality-enhanced instructions are used for more effective training and the optimum amount of information for a trainee to learn more effectively was determined for training a novice nuclear facility operator (Yim and Seong, 2010). In another study, a practice/simulated evacuation from a nuclear power plant was performed. It was found that augmented reality-enhanced escape guidelines required less escape time and had a higher evacuation success rate than regular electronic maps/guidelines (Tsai et al., 2013).

Based on the literature review above, research to date on using augmented reality for nuclear applications has been restricted to specific, individual tasks. To date no one has proposed a comprehensive framework for integrating human operators with emerging smart nuclear infrastructure in a comprehensive manner. A general Internet-of-Things (IoT) augmented reality framework that was considered during the development of this framework was the Ventana project (Beltran et al., 2016). However, the Ventana framework does not address the requirements and concerns associated with the nuclear industry. The work contained herein aims to fill this gap by presenting a framework for using augmented reality to seamlessly integrate a human operator with all aspects of smart nuclear infrastructure. As a precursor to this work, virtual reality technologies combined with depth imagers were explored for tracking nuclear materials in a mock glovebox for the purpose of visualizing criticality safety information associated with the state of the nuclear materials in the environment (Embry et al., 2015). In addition the use of vibro-haptic human-machine interfaces was explored to interface humans with distributed sensors used to monitor the state-of-health of infrastructure (Mascareñas et al., 2014). This work takes the lessons learned from these earlier results and applies them to synthesize a framework for the use of augmented reality to enable comprehensive smart nuclear infrastructure.

Challenges Associated With Smart Nuclear Infrastructure

Enabling smart nuclear infrastructure requires taking into account stringent constraints associated with safety, security, and documentation. In order to design an experimental demonstration of the framework for smart nuclear infrastructure, the development team included managers, and professionals intimately familiar with managing the day-to-day operations in high-value nuclear facilities. These nuclear facility professionals pointed out two key tasks executed within high-value, critical nuclear infrastructure that were of high importance and would help inform the development of the framework. The tasks were executing and documentation of nuclear material moves, and providing operators working in gloveboxes with online assessments of nuclear criticality safety.

First, we consider the challenges associated with nuclear material moves. One of the largest challenges associated with nuclear material moves is documentation of the material move both during the planning phase as well as during execution. Documentation is in general a challenging task associated with executing work in a nuclear facility. Operators need to be provided with documentation associated with the materials and equipment located inside a nuclear facility, as well as facilitating the documentation of work as it is being carried out. Access to documentation and the ability to modify documentation is critical in nuclear facilities because it aids in providing nuclear criticality safety information to operators so they do not violate criticality safety postings and potentially cause a criticality incident that could lead to loss of life. Documentation is further complicated when work must take place within a glovebox. In this case a worker must scan their hands with a radiation monitor every time they remove their hands from the gloves. This process can take on the order of 30 s and is not efficient from a time management point-of-view. Furthermore, every entry and exit from glovebox gloves provides an opportunity for a glove tear which could result in personnel contamination. As a result, it is not uncommon to solve the problem of documentation by assigning one technician whose main responsibility is work documentation for every two technicians actually working in the glovebox. The documentation technician may also be responsible for calling out written procedures to be followed by the technicians performing the work in the glovebox. Access to up-to-date documentation associated with a nuclear facility is also important for operations such as planning a nuclear material move to ensure that criticality safety regulations are followed, as well as for maintaining continuity-of-knowledge of who possessed a nuclear material at any given time. In some cases security requirements stipulate that two persons must be in custody of a nuclear material during a material move. Many of these documentation challenges are present during nuclear material move tasks. Current documentation techniques can involve the use of sensors such as scales, identification tag reading technologies (e.g., barcodes reader) operator observations/handwritten notes and access control instrumentation. Documentation of the wastes produced by these facilities is also important. In some cases regulations make it necessary to capture video documentation showing how nuclear waste containers are filled, and what they are filled with. The demonstration of the comprehensive augmented reality smart infrastructure framework presented here will focus on improving the process of documenting work in nuclear facilities. Specifically it will focus on documenting a material move in a nuclear facility, which is an operation that relies heavily on access to documentation of the move, as well as tools to account for the owner, location, and mass of the material.

An additional challenge associated with smart nuclear infrastructure is the need to provide glovebox operators with on-the-fly nuclear criticality safety assessments of their workspace as they execute work. If nuclear criticality safety postings are violated during work, the consequence can be the costly shutdown of unique facilities (Malone and Smit, 2017). The ability to provide glovebox workers with on-the-fly nuclear criticality safety assessments would have the potential to greatly enhance the ability of operators to complete their work safety. Demonstrations of techniques to perform online-nuclear criticality safety assessments will be shown in this work.

In addition to the comprehensive demonstration of a nuclear material move using augmented reality headsets, and the demonstration of augmented reality for nuclear criticality safety assessments, this work also provides proof-of-concept demonstrations for the periphery technologies deemed important during the course of this work for enabling smart nuclear infrastructure.

It must be made clear, the main contribution of this work is a comprehensive framework for enabling the next generation of smart nuclear infrastructure. It is not to try and solve any specific problem associated with nuclear operations. At this time there are too many administrative barriers in place with respect to safety and security that prevent adoption of the wearable computing technology in high-value nuclear infrastructure. It is anticipated it will be 3–10 years before these issues are addressed. By the time the administrative challenges associated with smart nuclear infrastructure have been addressed, sensing, communications, and augmented reality technologies will have matured to a point at which detailed solutions to problems using technology available today will most likely be of little value. At this point it has been determined that research on a comprehensive, strategic, framework providing guidance on how augmented reality technologies can be integrated with other Internet of Things sensors and devices to solve challenges associated with operations in smart nuclear facilities is most appropriate. The goal of this work is to generate this framework, and pass on the lessons learned during the development of that framework to professionals in both nuclear materials handling as well as wearable computing in order to inform future research directions in the development of smart nuclear infrastructure. Nuclear material operations tend to be labor intensive so there is a strong argument to make sure a framework for smart nuclear infrastructure has a strong component of wearable computing (e.g., augmented reality headset) built in upfront. The development of this framework will help future researchers focusing on specific problems to see how their work fits into the big picture of smart nuclear infrastructure.

Smart Nuclear Infrastructure Design Overview

The smart nuclear infrastructure developed to assist in the execution of the mock material move is shown in Figure 2. The smart nuclear infrastructure consists of a network of sensors that communicate with a central server. The augmented reality headset then communicates with the central server to obtain information from the sensors. The networking for the smart nuclear infrastructure is currently implemented using the Robot Operating System (ROS) (MoveIt, (n.d); Quigley et al., 2009). ROS was used because facilitates the low-level sensor integration. The sensors include near field communication (NFC) tag readers, and a scale. A smart cart was also developed for moving nuclear material canisters. The smart cart is outfitted with an NFC tag reader so canisters, operators, and safety/security information can be associated with it.

FIGURE 2

Figure 2. The layout of the prototype network of components used for the next generation of smart nuclear infrastructure.

In an actual nuclear facility many other types of sensors might also be included in this network including radiation monitors for detecting material contamination and thermal imagers for detecting thermal hazards. In addition, the canisters, rooms, gloveboxes, tools, and equipment in the facility would be instrumented with NFC tags and QR codes to allow for unique identification capabilities. The framework described in this paper is flexible enough to accommodate the inclusion of all of these sensors.

The operator is equipped with an augmented reality headset and an NFC operator ID tag. The augmented reality headset is used to provide the operator with feedback from the smart nuclear infrastructure network. This feedback currently takes the form of text-based updates, holograms, and video. It is also possible to implement audio and speech feedback.

The NFC tag is used to provide an authentication capability, so the operator can be associated with the operations he/she performs in the smart nuclear facility. In practice it may be necessary to use a more robust operator authentication method, but the use of an NFC tag is sufficient for demonstrating the concept. In an actual nuclear facility the operator's NFC tag identification could also be used for access control to facilities, equipment, and materials.

Figure 3 shows how the operator might typically interact with a smart nuclear infrastructure sensor. Sensors would typically be instrumented with an NFC reader for operator authentication. The operator would first scan their NFC operator ID tag with the reader. Next, the operator would scan the NFC identification tag associated with the item to be measured. Finally, the sensor would be used to make a measurement, which would include the item ID, location, and the operator as metadata.

FIGURE 3

Figure 3. Example showing how a user would interact with the system and the data that is transported on the network as the system is used.

Figure 4 shows the smart cart developed for transporting materials within the facility. The cart features an NFC reader to associate material canisters with the cart. By including the NFC reader, the cart will be able to keep track of which operators are using it, which canisters have been “owned” by the cart, and the cart can check to ensure safety and security regulations associated with the cart are being complied with.

FIGURE 4

Figure 4. The smart cart used to electronically document the movement of nuclear materials. That smart cart features an NFC tag reader, microprocessor, and wireless networking capabilities.

It must be stressed that the sensors and specific software packages used in this work are only representative of the class of possible technologies that could be used to implement smart nuclear infrastructure. Technologies for augmented reality headsets, embedded computing, imaging, machine learning, indoor localization, and communication are all constantly improving and are expected to be significantly more powerful by the time safety and security requirements are in place for smart nuclear infrastructure. For example, it is anticipated that within the next decade, 5G communications technologies such as LiFi will be mature and incorporated into future augmented reality headsets to enable secure, high-bandwidth communications inside of facilities (Haas, 2018). Secure communication technology such as LiFi could have a significant impact on the adoption of wearable computing such as augmented reality headsets in high-value nuclear facilities. The framework described in this paper has been designed to be flexible enough to accommodate advances in augmented reality, sensors, and communications technology.

Software Framework for Smart Nuclear Infrastructure

A software framework has been developed to implement the AR interface for the smart nuclear infrastructure as shown in Figures 5, 6. Currently component-based software engineering tools/languages are used to program AR headsets (Unity n.d.)¹, [Microsoft, (n.d)], (Vuforia n.d.)², so a component-based software framework was developed to match the programming tools.

FIGURE 5

Figure 5. The software framework for the system. These are the primary components that the user interacts with.

FIGURE 6

Figure 6. The supporting components for the software framework. The components are used to supply and document relevant information for the software components that the user interacts with.

The smart nuclear infrastructure AR software framework consists of two types of components, primary components, and support components. The operator would ordinarily have direct access to, and interact with, the primary components (Figure 5). These primary components consist of modules that the user would call up to perform specific tasks such as collecting video documentation of loading a canister with nuclear material. Daemons are primary component processes that run in the background, but bring up ambient information associated with the environment. For example, a daemon might be responsible for automatically bringing up the safety information associated with a room.

The support components (Figure 6) can be thought of as software that facilitates the operation of the primary components. Examples of such components include databases (and associated large-scale analysis), networking capabilities, and modeling/simulation/optimization. Typically, the software associated with the support components will run on a machine external to the AR headset and interfaces with the AR headset through the wireless network. The ROS software framework being used to enable network communications for the sensor is an example of a support component. The support component software will in many cases require computational and memory resources that are greater than the computational and memory resources available on an individual augmented reality headset.

Currently the software framework is designed in such a way that all components are arranged in a hierarchical manner. Each component has an Application Programming Interface (API) associated with it that is meant to only be accessed by its parent's components. Adopting this convention improves the maintainability and modularity of the code.

Requirements Overview and Demonstration Description for Material Moves

To demonstrate how augmented reality can be used to enable a smart nuclear facility, this work focuses on the task of performing nuclear material moves. Ensuring nuclear materials are moved in a safe and secure manner is a vitally important operation in certain high-consequence nuclear facilities. It is a common operation that requires careful planning for considerations such as nuclear criticality safety and it is an operation that must be thoroughly documented. The augmented reality system for smart nuclear infrastructure will be demonstrated in the context of a mock nuclear material move in a mock nuclear facility using mock nuclear materials. The mock material move will consist of taking mock nuclear material in a primary canister, and then transferring it to a second canister for movement to a different room where it will ultimately be placed inside a mock safe. The movement of the material will take place using a smart-cart. During the mock material move the operator will be wearing an augmented reality headset which will provide the operator with feedback from the smart nuclear infrastructure. The headset (along with other sensors) will also be used to provide data to document the move within a facility database.

Demonstration of Augmented Reality-Based Smart Nuclear Infrastructure System during the Execution of a Mock Material Move

An example implementation of the proposed framework for smart nuclear infrastructure is now presented. The framework is demonstrated in the context of performing a mock material move in a mock nuclear facility. Figures 7–9 are photos taken from the perspective of the operator while wearing the augmented reality headset, and interacting with the mock smart nuclear facility. Upon activating the AR headset, the user is presented with a main menu hologram. The choices associated with the main menu correspond to the primary software components. Typically, the operator would first engage in authenticating themselves as the user of the system. In this case the operator would select the “Operators” button with gaze and tap gestures. Selecting the “Operator” button brings up a holographic menu of operators to choose from. The operator then selects themselves and undergoes an appropriate authentication protocol. This ensures the user's actions are associated with the proper records in the database therefore documenting the actions in the smart nuclear facility. At this point, the holographic gray box in the lower left hand corner would have changed to show a picture of the operator along with their name.

FIGURE 7

Figure 7. If the operator applies a tap gesture to the info button a holographic display is made visible in the augmented reality headset showing information associated with the canister.

FIGURE 8

Figure 8. Orthorectified view of the menu visible to the operator.

FIGURE 9

Figure 9. The holographic display can be used to display video of the process of loading the canister which is important for documentation requirements in certain nuclear facilities.

Next, the operator would need to connect the wireless communications capability to interface with the smart nuclear infrastructure. The operator would return to the main menu and select the “Settings” button. Then the operator would scroll down to the “ROS Daemon Settings” button. This button would allow the operator to set the IP address of the ROS server and connect to it.

Once the connection is made, the operator can interact with the smart nuclear infrastructure in the facility. In this case, the operator locates the canister containing the material they need to move to a different location. Note the QR code located on the canister. The AR headset recognizes the canister and brings up a holographic “Info” button, that can be selected with gestures to summon information related to the canister as shown in Figure 7.

When the “Info” button is selected, a hologram is displayed that provides access to a wide range of information associated with the canister. This information could include material properties, safety information, chain-of-custody, and even more comprehensive documentation such as the ability to bring up a video showing how the canister was loaded, as shown in Figures 7, 8. Video documentation of the loading procedure for canisters can be a requirement for some operations in the nuclear industry.

The fields in the information holograms are programmed to allow the operator to use speech-to-text commands to enter or edit the documentation associated with the canister as needed. The ability to perform documentation in a hands-free manner is a big advantage for work in the nuclear industry because current practice for documenting work in some nuclear facilities still relies on hand-written notes. An orthorectified view of the menu visible to the operator is shown in Figure 8.

Once the operator has examined the holographic documentation of the canister, the operator can decide to proceed with the material move. In this case the next step is for the operator to remove some of the mock nuclear material from the canister and measure its mass before it goes into a new canister to be transported to the new room in the mock facility. The operator goes through the procedure outlined in Figure 3 to authenticate themselves and associate the new canister with the measurement. Then, the operator directs the scale to measure the mass of the mock nuclear material (Figure 10).

FIGURE 10

Figure 10. Mock nuclear material is taken from the canister and placed on a networked balance. The measurement from the balance associated with operator and balance information is displayed in the augmented reality headset.

After the scale makes the mass measurement of the mock nuclear materials, a text update is provided in the upper-left corner of the operator's vision that provides information relevant to the measurement and confirms the measurement has been documented in the smart nuclear infrastructure system. Once the measurement has been made, the operator moves the mock nuclear material in to a new canister for transport.

Once the new canister is loaded with mock nuclear material and placed inside a secondary containment canister, the mock material is ready to be transferred to the smart cart. The operator once again goes through the protocol in Figure 3 to associate the user and the canister with the smart cart using the on-board NFC tag reader. Once the smart cart has detected that the operator and the canister are associated with it, update text associated with the transfer is presented to the operator in the upper-left corner of their field of view.

The mock nuclear material can now be transported to room 2 on the smart cart (Figure 11).

FIGURE 11

Figure 11. The new canister with the mock nuclear material being moved with the smart cart to a new location in the facility.

When the operator arrives at room 2 they will find that the door is instrumented with a QR code. When the augmented reality headset recognizes a QR code, it can display a holographic “Info” button that summons the detailed information regarding the contents of the room (Figure 12). This information could include nuclear criticality safety information, safety protocols associated with the equipment in the room, or information regarding the current state of the room.

FIGURE 12

Figure 12. The doors of the mock facility have been instrumented with QR codes. The augmented reality headset can recognize the QR codes. A holographic interface can provide information to the operator regarding the contents of the room if they air-tap the info.

Once the operator, smart cart, and canister are inside room two, the operator can access the mock safe to store the nuclear material canister. Prior to placing the canister inside the safe, the operator first uses the protocol shown in Figure 3 to associate the canister and the operator with the safe. Once this association is complete, update text is shown in the upper-left corner of the operator's field-of-view, verifying the association has completed successfully. Now, the operator places the canister inside the mock safe.

Finally, the operator closes the safe. The door of the safe is also instrumented with a QR code. When the augmented reality headset recognizes the QR code, a holographic “Info” button is presented to the operator. The operator can select this “Info” button to gain additional information about the safe in the same manner as for the canister and room. At this point, the demonstration of the augmented reality system for interacting with smart nuclear infrastructure to execute a mock nuclear material move is complete.

Enhanced Glovebox Operations Using Augmented Reality

A demonstration of the framework is now made in the context of using augmented reality for providing operators in gloveboxes with information associated with nuclear criticality safety and to enable hands-free documentation. A mock glovebox was constructed for this demonstration. The mock glovebox consists of a table with a cardboard box on top. Two representative nuclear material canisters are placed on top of the box in order to bring them to working height as would be common for many gloveboxes. A unique QR code is taped to each canister. A polycarbonate sheet is placed on top of the table and in front of the box and canisters. The purpose of the polycarbonate sheet is to demonstrate that augmented reality technologies can still operate in the presence of a transparent barrier as would be found in a glovebox. The polycarbonate sheet is small enough to allow the operator to place their arms around the sheet in order to interact with the canister. The glovebox demonstration setup can be seen in Figure 13.

FIGURE 13

Figure 13. Mock glovebox setup for demonstrating how augmented reality headsets can aid glovebox technicians with nuclear criticality safety, information retrieval, and documentation.

We used a commercially available tracking software (Vuforia n.d.)² to identify the QR codes on each canister and track their poser and position (Figure 14). Once the canisters are detected a cylindrical hologram is overlaid on each canister in order to provide the user feedback that the system has in fact detected the canisters. A holographic button is also associated with each canister based on the tracking software output, that can allow the user to bring up information from a database that is associated with each canister (Figure 15). The current tracking software is robust enough to tolerate some occlusion of the QR code Figure 16. For the purpose of this demonstration, it is assumed that each canister is filled with some amount of a mock nuclear material. A predefined nuclear criticality safety rule is used to check to make sure the proximity of the canisters stays beyond a predefined distance as the operator is working and moving canisters (Figure 17). In the event that that the canisters pass into the proximately threshold the color of the holographic cylinder overlaid on the canisters turns blue (Figure 18). The blue color is reminiscent of the blue flash that is caused by Cherenkov radiation when nuclear materials go critical (Clayton, 2010). It must be stressed that the administrative criticality limits associated with the use of an augmented reality human-machine interface would be very conservative in order to ensure a warning is produced long before and actual nuclear material becomes critical. Once the canisters are moved sufficiently far from one another the canisters return to their original neutral color to indicate to the operator they are once again in compliance with nuclear criticality safety postings (Figure 19).

FIGURE 14

Figure 14. QR code on canister allows for tracking of position and pose of canister. A holographic cylinder is overlaid on the canister to provide feedback to the user that the canister is detected and being tracked.

FIGURE 15

Figure 15. A holographic button is associated with the canister. The button works despite the presence of the polycarbonate sheet.

FIGURE 16

Figure 16. The tracking algorithm used to follow the canister is robust to occlusion of the QR code.

FIGURE 17

Figure 17. The operator brings the canisters closer and farther from one another as they are working.

FIGURE 18

Figure 18. Once the canisters pass a predefined proximity threshold, the holographic canisters turn blue to give the operator notice they have violated the criticality safety postings.

FIGURE 19

Figure 19. Once the canisters are separated beyond the proximity threshold the color of the holographic canister returns to neutral.

Next we demonstrate the ability to retrieve and fill-out documentation such as a “slip printout” used to track nuclear materials in a hands-free manner using the augmented reality headset. First, the operator places the cursor on the holographic button associated with one of the canisters using the motion of their head (Figure 20). Once the cursor is on the holographic button they use the voice command “select.” To actuate the button. At that time the documation comes into the operator's field of view (Figure 21). Once the documentation is in view the operator can select a given field using the cursor and the “select” voice command. The operator can then use voice recognition capabilities in order to place data in the field. Figure 22 provides an example of selecting the “Identification Number” associated with the canister and changing it using voice recognition Figure 22. Figure 23 shows how both the “Identification Number” field and the “Material Name” field have both been changed in a hands-free manner using the cursor and voice commands/voice recognition. An orthorectified view of an example slip printout is shown in Figure 24 and is representative of what the operator would see when wearing the augmented reality headset. This demonstration shows that augmented reality can be used to enable hands-free documentation retrieval as well as modification in environments representative of a glovebox. Operators can interact with documentation without taking their hands out of their gloves.

FIGURE 20

Figure 20. The holographic button associated with the cursor is actuated using the cursor and a “select” voice command.

FIGURE 21

Figure 21. The documentation associated with the canister comes into the operator's field of view.

FIGURE 22

Figure 22. The operator selects a field of view using the cursor and “select” voice command. Once the field of view is selected the contents of the field are changed using voice recognition.

FIGURE 23

Figure 23. Changes to two fields in the documentation have been made in a hands-free manner using the voice recognition capabilities.

FIGURE 24

Figure 24. An orthorectified view of a representative slip printout visible to the operator through the augmented reality headset.

Additional Applications for Augmented Reality in Smart Nuclear Infrastructure

The execution of this work revealed exciting opportunities for the use of augmented reality headsets for smart nuclear infrastructure. The ability of augmented reality headsets to create 3D maps of the environment is particularly interesting because it provides a means to perform indoor localization for wireless sensor networks that the augmented reality headset interacts with. Augmented reality headsets also provide a means to enable persistent surveillance of industrial facilities. If all operators in a nuclear facility were outfitted with augmented reality headsets, the headsets themselves could be collecting sensor data in the background without any operator intervention. This background data could be fed to a database where machine learning algorithms search for relevant patterns in the data. This solution would have many advantages over robotic surveillance solutions because robots must be provided power, must be able to navigate the facility, and can have significant safety concerns associated with them. The use of networks consisting of human operators wearing augmented reality headsets can have significant impacts on persistent facility monitoring. These additional applications for augmented reality are specifically mentioned because they were deemed to be of high impact for operations and training in nuclear infrastructure.

Augmented Reality for Indoor Localization

The execution of this work revealed exciting opportunities for the use of augmented reality headsets for smart nuclear infrastructure. The ability of augmented reality headsets to create 3D models of the environment using algorithms for solving the Simultaneous Localization and Mapping (SLAM) problem (Cadena et al., 2016), is particularly interesting because it provides a means to perform indoor localization. The primary purpose of these 3D models is to aid in the placement of holograms in the environment in such a way that they minimally overlap with the real environment thus enhancing the illusion that the holograms are actually present in the environment. However, in the course of working with augmented reality headsets it has become clear that electro-optic-based SLAM techniques have great potential for solving the indoor localization problem that often arises when deploying wireless sensor networks indoors. When a wireless sensor network is located outdoors GPS can typically be used for the purpose of localization. Unfortunately GPS typically does not work indoors, thus adding severe limitations on the ability to accurately localize objects indoors. Significant research investments have been made in developing technologies for indoor localization that make use of physical mechanisms such as acoustics/ultrasonic and electromagnetic waves (Chen et al., 2016; Zafari et al., 2018). Unfortunately these technologies often suffer from limitations associated with accuracy and noise in the environment.

A demonstration of the use of an augmented reality headset for indoor localization was created. This demonstration is executed using one operator wearing augmented reality headset that is networked to a base-station computer. The headsets initially use a static QR code displayed by the base station computer (Figure 25) as a landmark to determine the starting location of the operator (Figure 26). The updated locations of the operator are then determined using the SLAM-based updates provided by the headset and visualized on the 3D model of the facility. Examples of the operator being localized can be found in Figures 27, 28. The facility model location is calibrated relative to the location of the static QR code. The localization procedure described here can be extended to multiple operators. Indoor localization based on SLAM such as that used here has a large number of potential applications for smart infrastructure. It could be used to localize sensors, tools, equipment, and personnel. It could be used to track personnel for documentation purposes as well as to enable studies regarding the efficiency of how work is performed.

FIGURE 25

Figure 25. The operator first anchors their initial position and pose by gazing at a QR code.

FIGURE 26

Figure 26. Once the operator gazes at the QR code the operator's position and pose are recorded in the map.

FIGURE 27

Figure 27. As the operator moves throughout the facility their location relative to the map also changes.

FIGURE 28

Figure 28. Another instance of the operator moving throughout the facility while their position is captured on the map.

There is still much potential research that could be undertaken with regards to SLAM-based indoor localization techniques and sensor hardware that could be used to enable the next-generation of smart infrastructure.

Continuous Infrastructure Monitoring

Augmented reality headsets also provide a means to enable persistent surveillance of industrial facilities. If all operators in a nuclear facility were outfitted with augmented reality headsets, the headsets themselves could be collecting sensor data in the background without any operator intervention. This background data could be fed to a database where machine learning algorithms search for relevant patterns in the data that could inform decision-making. This solution would have many advantages over robotic surveillance solutions (David and Miller, 2017) because robots must be provided power, must be able to navigate the facility, and can have significant safety concerns associated with them. The use of networks consisting of human operators wearing augmented reality headsets can have significant impacts on persistent facility monitoring. For example, an augmented reality headset is integrated with a thermal imager that is pointed in the same direction as the field-of-view as the operator could have great application for persistent facility monitoring. The measurements from the thermal imager are overlaid on a subset of the field-of-view of the operator using the AR headset (Figure 29). This information provides the operator with information regarding whether devices are energized or not. This information is useful not only to provide the immediate operator with safety information, but can also be combined with the localization information to provide facility managers with an update on the operational state of the facility. For example, if the insulation on a piece of pipe was damaged, it could be observed by the AR headset/thermal imager of an operator in the vicinity performing other work. The AR headset in combination with machine-learning algorithms could recognize the off-normal condition and alert managers to the problem who could then schedule maintenance personnel to repair the damaged insulation. If additional information were needed to better characterize the problem, the AR headset could provide audio-visual instructions to the operator to help the headset collect the relevant information (images, video, point-clouds) for decision makers to help schedule maintenance personnel.

FIGURE 29

Figure 29. Screenshot from augmented reality headset with thermal imager overlay indicating soldering iron is heated up.

Interfacing Augmented Reality With Simulation Tools

Augmented reality also allows a human operator to be more intimately interfaced with powerful computational and memory resources beyond those present at the AR headset itself. One example of this that is relevant to enabling smart nuclear facilities is running nuclear criticality safety analysis on new work configurations. Figure 30 shows a 3rd person view of mock nuclear work station featuring two containers. It could be desirable to perform a nuclear criticality safety analysis of this work configuration. In order to facilitate this analysis, an augmented reality application has been developed to interface the Monte Carlo N-Particle transport code MCNP (Monte Carlo Methods, Codes, and Applications Group–Los Alamos National Laboratory, 2013) to the data collected by the AR headset and the operator. This application allows the operator to use the augmented reality headset to generate MCNP input files based on user input and the holographic feedback provided by the augmented reality headset. In this case the human operator uses hand gestures to specify the geometry of the actual containers of interest. The pink holograms are overlaid above the actual canisters in order to provide the operator with feedback that indicates they have correctly specified the geometry that is used to generate the MCNP input file that could be sent to a more powerful remote computer running MCNP. The results from the MCNP could then be transmitted back to the augmented reality headset where they could be presented to the operator in a manner that respects the context of the work that is to be performed. Future research could consider techniques to further automate the generation of MCNP input files using the 3D environmental models captured by the augmented reality headset. Interfacing augmented reality with high-fidelity radiation transport codes is particularly useful for glovebox criticality safety applications as well as for high-fidelity training.

FIGURE 30

Figure 30. 3rd-person view of a human operator using an augmented reality headset to determine the geometry for MCNP input files in-situ.

Augmented reality tools can also be integrated with facility-level information provided by a database. The operator can both download as well as upload data to the database and this information could be used to perform tasks such as specifying the MCNP input file information as previously discussed. This information could also help enable tools such as a tool to help operators plan nuclear materials moves in such a way that they are safe and do not violate any nuclear criticality safety limits. A demonstration application of such a material move planning tool has been created. This application takes information regarding the geometry of the interconnected gloveboxes in a facility, along with the information regarding the nuclear criticality limits associated with each glovebox and the properties of the material that is desired to be moved, and suggests a possible safe path to move the material using a technique based on the A^* path planning algorithm (Hart et al., 1968). The proposed safe path is visualized using a magenta line. Future research could focus on integrating this path-planning tool with facility-level database information and probabilistic risk-assessment algorithms to generate paths that make use of the most-up-to-date facility information and are also robust against human errors in execution as well as in data entry.

This work also illustrates the need for robust authentication of the operator using any given augmented reality headset. Many applications are security critical, and for this reason there will be a need to have a high confidence of which operator is using a given augmented reality headset. Such authentication will help ensure that operators are not performing undesirable actions such as stealing resources, or performing work incorrectly.

Interfacing Bar Code Scanners to an Augmented Reality Headset for Agile Auditing

Having access to information is critical in nuclear facilities, for executing daily operations related to management and decisions. One of the ways we currently address the problem of quickly accessing information has been with the use barcodes and barcode scanners. Barcode scanning allows user to send a search input into a database to send and receive information. For many years barcode scanning has provided: a fast and efficient search mechanism to get information quickly, and it has allowed improvements in automation. This research includes the development of an Augmented Reality application to integrate the barcode scanner as a USB HID device to be connected to an augmented reality headset. This application allows the user to read barcodes in order to get more visual information. The software has a simple GUI design, containing a 3D rectangle that moves with the head motion, a smaller white 3D rectangle that shows the decoded string of the barcode. This software has been tested with different barcode scanners to see which one provided the most efficiency. The second contribution of this work includes the implementation of a transmission control protocol (TPC) network socket connection between the barcode being scanned and a remote database on a distant server. This stream socket connection between the object being scanned and the server allows inspectors to obtain real-time information from the server, and also store information of the inspection in the server. The final objective of this work is to augment the human perception of their environment to be faster and more accurate with the use of augmented reality and barcodes in the context of nuclear facilities operations and management. Future works include the implementation of additional safety features with augmented description of the canisters, display of internal contents that can be sorted and comprehend by the inspector, and other implementations that enable real-time display of information across time and space not previously possible.

The proposed application consists of the integration of three main assets now linked through the augmented auditing of the nuclear storage through bar-code scanning and AR. Figure 31 summarizes the three components included in the proposed framework. The inspector conducts the auditing using a bar-code that collects the information of a given canister in the storage environment. This bar-code information is linked with a server through a transmission control protocol (TCP) socket that streams information in two directions to the inspector on real-time. The information of the inspection can be saved in the database, and the information from the database can be visualized by the inspector in real-time. This real-time auditing enables the inspector to modify their decisions during the inspection informed by information now available on real-time through the new application. Figure 31 shows the proposed solution architecture.

FIGURE 31

Figure 31. Solution architecture.

The design of the app is processed in Unity, and the assembly and compilation is carried out with Visual Studio [Microsoft, (n.d)]. The deployment in Hololens enables the real-time access to the linked database which can be accessed by the inspector. The inspector can then access Safety Data Sheets (SDS) that are linked through the stream socket in front of the canister with valuable information in PDF format. When the PDF information is updated in the server the inspector can receive the most updated PDF through AR.

The new application enables the inspector to scan the bar codes and access the information automatically. The application enables the following inspection abilities:

1. Bar-code scanning of the nuclear canister (hands free).

2. The inspector can scan the code of the canister and identified the specific canister from a remote database. The access to the canister information is conducted remotely through a socket connection from a remote server, where the barcode sends the information collected from the code (Figure 32).

4. The inspector can also select accessing the database transmit PDF properties of the canister to the inspector by clicking with their fingers the audited canister.

5. The inspector can observe the properties of the canister as they are updated from the server remotely through the socket.

6. The inspector can open and display the information during the audit. This information has arrived to the interface of the inspection through the socket, so the database allows the inspector to read the most up to date information of the specific canister.

FIGURE 32

Figure 32. Bar-code scanning (hands-free).

Conclusions

In this work, we present a framework for using augmented reality to enable the next generation of smart nuclear infrastructure. Nuclear infrastructure is among the most highly regulated infrastructure in existence. Nuclear infrastructure features a variety of hazards (e.g., radiation, chemicals, mechanical, electrical). It also features stringent security considerations that may need to conform with the stipulations of international non-proliferation treaties. Nuclear infrastructure also commonly includes systems to keep rigorous documentation of work and chain-of-custody information of the materials held within. For these reasons, solutions created for nuclear facilities can easily be adapted to other forms of infrastructure because other forms of infrastructure generally only feature a subset of these constraints and hazards.

A prototype of the Augmented-Reality based system for enabling smart nuclear infrastructure has been implemented and demonstrated in a surrogate environment. The development of this prototype can be used to help guide the development of a system that can be deployed in actual nuclear infrastructure.

Author Contributions

The work was led by and significant software and writing contribution was provided by the lead author DaM. JM performed the final execution of the smart nuclear infrastructure prototype demonstration. Indoor localization was implemented by MK. ES performed the work on the databases and interfacing the sensor nodes for the overall system demonstration. BeB also did significant work on sensor node integration. JaT implemented the interface between the AR headset and MCNP. BrB and BK worked on the glovebox criticality and hands-free documentation demonstration. FM, DiM, MA, and RF implemented the bar-code enabled auditing system using the augmented reality headset. JuT and AW provided technical expertise in the operations of nuclear facilities. The remaining co-authors primarily contributed to the prototype of the system smart nuclear infrastructure.

Funding

Funding for this work was provided by the Los Alamos National Laboratory FY2017 Pathfinder fund, the Los Alamos National Laboratory Nuclear Criticality Safety Division, the Science Undergraduate Laboratory Internship (SULI) program and the Community College Initiative (CCI) program.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We would like to thank Yijing Watkins for helping demonstrate the ability to perform indoor localization using augmented reality. At the time of this work Los Alamos National Laboratory was operated by Los Alamos National Security LLC, for the National Nuclear Security Administration of the U.S. Department of Energy, under DOE Contract DE-AC52-06NA25396.

Footnotes

1. ^n.d. Unity. Available online at: https://unity3d.com/ (accessed December 15, 2017).

2. ^n.d. Vuforia. Available online at: https://www.vuforia.com/ (accessed December 15, 2017).

References

Abate, A. F., Narducci, F., and Ricciardi, S. (2013). “Mixed reality environment for mission critical systems servicing and repair,” in International Conference on Virtual, Augmented, and Mixed Reality (Las Vegas, NV). doi: 10.1007/978-3-642-39420-1_22

CrossRef Full Text | Google Scholar

Bardeen, L. (2017). Thyssenkrupp Transforms the Delivery of Home Mobility Solutions With Microsoft HoloLens. Microsoft: Windows Blogs. 4 24. (2017). Available online at: https://blogs.windows.com/devices/2017/04/24/thyssenkrupp-transforms-the-delivery-of-home-mobility-solutions-with-microsoft-hololens/#0sIr61S7TscF5IJV.97 (accessed December 17, 2017).

Beltran, S., Durkan, A., Fitzpatrick, E., Gauthier, T., and Ospina, J. (2016). Ventana IoT. Edited by Boston University. Available online at: https://github.com/VentanaIoT/Ventana (accessed December 19, 2017).

Cadena, C., Carlone, L., Carrillo, H., Latif, Y., Scaramuzza, D., Neira, J., et al. (2016). Past, present and future of simultaneous localization and mapping: towards the robust-perception age. IEEE Trans. Robot. 32, 1309–32. doi: 10.1109/TRO.2016.2624754

CrossRef Full Text | Google Scholar

Chen, C., Yan, C., Hung-Quoc, L., Yi, H., and Ray Liu, K. J. (2016). “High accuracy indoor localization: a WiFi-based approach,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (Shanghai). Available online at: https://ieeexplore.ieee.org/document/7472878/

Google Scholar

Clayton, E. D. (2010). Anomalies of Nuclear Criticality. Richland, WA: Pacicif Northwest National Laboratory. doi: 10.2172/972533

CrossRef Full Text | Google Scholar

David, L., and Miller, G. (2017). Review of robotic infrastructure systems. J. Infrastructure Syst. 23:04017004. doi: 10.1061/(ASCE)IS.1943-555X.0000353

CrossRef Full Text

Embry, K., Hengartner-Cuellar, A., Ross, H., and Mascarenas, D. (2015). “Tracking the dynamics of nuclear materials inside a glovebox,” in IMAC XXXIII a Conference and Exhibition on Structural Dynamics (Orlando, FL).

Haas, H. (2018). LiFi is a paradigm-shifting 5G technology. Rev. Phys. 3, 26–31. doi: 10.1016/j.revip.2017.10.001

CrossRef Full Text | Google Scholar

Hart, P. E., Nilsson, N. J., and Raphael, B. (1968). A formal basis for the heuristic determination of minimum cost paths. IEEE Trans. Syst. Sci. Cyber. 4, 100–107. doi: 10.1109/TSSC.1968.300136

CrossRef Full Text | Google Scholar

Ishii, H., Bian, Z., Fujino, H., Sekiyama, T., Nakai, T., Okamoto, A., et al. (2007). “Augmented reality applications for nuclear power plant maintenance work,” in International Symposium on Symbiotic Nuclear Power Systems or 21st Century (ISSNP) (Tsuruga).

Google Scholar

Ishii, H., Yan, W., Yang, S. F., Shimoda, H., and Izumi, M. (2014). “Wide area tracking method for augmented reality supporting nuclear power plant maintenance work,” in Progress of Nuclear Safety for Symbiosis and Sustainability, eds H. Yoshikawa, and Z. Zhang (Tokyo: Springer), 283–289. doi: 10.1007/978-4-431-54610-8_29

CrossRef Full Text | Google Scholar

Johnsen, T., and Mark, N. K. (2014). “Virtual and augmented reality in the nuclear plant lifecycle perspective,” in Progress of Nuclear Safety for Symbiosis and Sustainability, eds H. Yoshikawa, and Z. Zhang (Tokyo: Springer), 243–255. doi: 10.1007/978-4-431-54610-8_25

CrossRef Full Text | Google Scholar

Malone, P., and Smit, R. J. (2017). Safety lapses undermine nuclear warhead work at Los Alamos. The Washington Post. Available online at: https://www.washingtonpost.com/world/national-security/safety-lapses-undermine-nuclear-warhead-work-at-los-alamos/2017/06/17/87f051ee-510d-11e7-b064-828ba60fbb98_story.html?utm_term=.29c58ee7fd60 (accessed June 13, 2019).

Mascareñas, D., Plont, C., Brown, C., Cowell, M., Jameson, N. J., Block, J. T., et al. (2014). A vibro-haptic human–machine interface for structural health monitoring. Struct. Health Monit. 13, 671–685. doi: 10.1177/1475921714556569

CrossRef Full Text | Google Scholar

Microsoft (n.d). Microsoft Hololens. Available online at: https://www.microsoft.com/en-us/hololens (accessed December 17, 2017).

Microsoft (n.d). Visual Studio. Available online at: https://www.visualstudio.com/ (accessed December 15, 2017).

Monte Carlo Methods, Codes, Applications Group–Los Alamos National Laboratory. (2013). A General Monte Carlo N-Particle (MCNP) Transport Code. Available online at: https://laws.lanl.gov/vhosts/mcnp.lanl.gov/index.shtml (accessed April 15, 2018).

Google Scholar

MoveIt (n.d). Robot Operating System. Available online at: https://moveit.ros.org/ (accessed September 20, 2018).

Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., et al. (2009). “ROS: an open-source robot operating system,” in ICRA Workshop on Open Source Software (Kobe).

Google Scholar

Shimoda, H., Ishii, H., Yamazaki, Y., and Yoshikawa, H. (2004). A support system for water system isolation task of nuclear power plant by using augmented reality and RFID. Hozengaku 3, 30–37.

Google Scholar

Shimoda, H., Maeshima, M., Nakai, T., Bian, Z., Ishii, H., and Yoshikawa, H. (2005). “Development of a tracking method for augmented reality applied to nuclear plant maintenance work,” in Proceedings of ISOFIC (Tongyeong-si).

Google Scholar

Sørensen, S. (2014). “Augmented reality for improved communication of construction and maintenance plans in nuclear power plants,” in Progress of Nuclear Safety for Symbiosis and Sustainability, eds H. Yoshikawa, and Z. Zhang (Tokyo: Springer), 269–274. doi: 10.1007/978-4-431-54610-8_27

CrossRef Full Text | Google Scholar

Surur (2017). HoloLens used to Beat the Laptop Travel Ban (Video). mspoweruser. 6:19. Available online at: https://mspoweruser.com/hololens-used-beat-laptop-travel-ban-video/ (accessed December 17, 2017).

Tsai, M. K., Liu, P. H. E., and Yau, N. J. (2013). Using electronic maps and augmented reality-based training materials as escape guidelines for nuclear accidents: an explorative case study in Taiwan. Br. J. Educ. Technol. 44, E18–21. doi: 10.1111/j.1467-8535.2012.01325.x

CrossRef Full Text | Google Scholar

Yim, H. B., and Seong, P. H. (2010). Heuristic guidelines and experimental evaluation of effective augmented-reality based instructions for maintenance in nuclear power plants. Nuclear Eng. Des. 240, 4096–102. doi: 10.1016/j.nucengdes.2010.08.023

CrossRef Full Text | Google Scholar

Zafari, F., Gkelias, A., and Leung, K. K. (2018). A Survey of Indoor Localization Systems and Technologies. Available online at: https://arxiv.org/pdf/1709.01015.pdf

Google Scholar