Intelligent railroad inspection and monitoring
Yu Qian- Department of Civil and Environmental Engineering, University of South Carolina, Columbia, SC, United States
Railways are essential to the global transportation infrastructure, providing eco-friendly and economical solutions for the movement of freight and passengers. Inspecting and maintaining extensive rail networks timely poses significant challenges. My group and collaborators have focused on automated railroad inspection technologies, emphasizing the use of deep learning and computer vision to overcome the limitations of traditional manual inspections. Our research introduces groundbreaking real-time inspection methods, leveraging a specialized dataset of railroad components for enhanced instance segmentation models, achieving unprecedented accuracy and inference speeds. The developed computer vision systems efficiently detect track components and their changes over time, and also quantify rail surface defects. Additionally, our work extends to improving railroad crossing safety, utilizing deep learning frameworks for the detection of unusual pedestrian behaviors and object identification, aimed at reducing crossing incidents and improving emergency response times. Our future research directions aim to further refine the cost-effectiveness and autonomy of railroad inspection systems. Through these innovations, we hope to aid in the inspection and maintenance of railroads, offering practical solutions for railroad and other civil engineering applications.
1 Introduction
Railways are crucial to the worldwide transportation network, fueling economic development and facilitating efficient travel. However, monitoring these vast rail systems presents considerable challenges. The extensive railroad networks, traversing varied and sometimes inaccessible terrains, require timely and comprehensive maintenance efforts. Factors such as extreme weather conditions, routine wear and tear, and the risks of vandalism or sabotage continually threaten the integrity of railway tracks. Moreover, the complexity of conducting track inspections necessitating advanced tools and skilled professionals. As of 2024, the United States operate an extensive railway infrastructure, with around 220,000 km allocated for freight and an additional 34,000 km for passenger services (Robinson et al., 2023). The Federal Railroad Administration (FRA) emphasizes the importance of regular, comprehensive track inspections to uphold safety and operational efficiency (FRA, 2018a). Key track elements like spikes, bolts, and clips are especially susceptible to damage from continuous use and environmental changes. Failure to identify such defects can lead to serious accidents and considerable economic losses. In 2018 alone, track-related defects resulted in 546 incidents, incurring over $97 million in damages, including significant incidents involving missing or damaged components causing around $10 million in losses (FRA, 2018b). The FRA mandates frequent inspections to promptly identify and address these issues. However, traditional inspection methods remain labor-intensive and costly. Inspections, particularly for missing components, are often performed manually, a process that is both expensive and inefficient.
Over the past 2 decades, remarkable progress has been achieved in automating track inspection systems. Early methods provided innovative solutions for detecting missing fasteners and rail defects, though they often faced limitations in terms of resolution, component variety, and computational demands. The advent of convolutional neural networks (CNNs) has revolutionized this field, leveraging vast datasets and increased computing power to enhance object detection and segmentation capabilities. Yet, despite these advancements, the challenge of real-time, on-site inspections has remained, with many systems requiring post-processing of data or substantial computing resources.
2 Our recent effort in developing real-time railroad inspection methods
In response to these challenges, our team has focused on advancing real-time railroad inspection techniques. While various datasets exist for training computer vision models on general object detection and segmentation, specific datasets for railroad track images have been scarce. Guo et al. (2021a) took a significant step by creating the first publicly available dataset of railroad components, comprising 1,000 images to aid the application of advanced deep learning models in track inspections. This initiative led to the development of a pixel-level detection system for track components, utilizing an enhanced instance segmentation model. This system achieved real-time instance segmentation with remarkable accuracy, especially reaching an inference speed over 30 FPS with a single GPU for the first time in the field of railroad engineering. Guo et al. (2021b) further accelerated the inspection speed by introducing hybrid activation functions to better allocate limited computational resources. The rapid evolution of deep learning and computer vision has unlocked new possibilities for automated track inspections, addressing the limitations of earlier systems that relied on data transfer for post-processing or required substantial computing power. Recent innovations include a portable system by Tang and Qian, (2024a), which combines the latest YOLOv8 object detection model with a custom template matching algorithm. This setup not only identifies track components like spikes, bolts, and clips but also detects missing ones with unprecedented speed and efficiency, offering a cost-effective solution for enhancing rail safety. Tang and Qian, (2024b) further refined the inspection process by introducing a high-speed model inference pipeline. This new approach, leveraging parallel processing and advanced computing techniques, significantly increases the detection speed, making it suitable for high-speed inspection requirements.
The complexity of rail inspections lies not only in identifying present components but also in detecting absent ones to facilitate maintenance planning. The variability in baseplate types and fastening systems, along with their installation patterns, necessitates a flexible approach to detection. Tang et al. (2024c) introduced a Cascade R-CNN model with Predefined Proposal Templates representing a significant advancement in adapting to new track configurations not previously included in training datasets, enhancing the versatility and applicability of the proposed system. Rail surface defect (RSD) is another major concern for railway safety, contributing to a significant portion of rough vehicle-track interaction and even derailments. To address RSD, Guo et al. (2021c) quantified RSD areas automatically based on Mask-RCNN. Later, Wu et al. (2022) developed a hybrid deep learning architecture for detecting RSDs with a novel rail boundary guidance network. This approach, along with other advancements such as the implementation of advanced models like RailFormer (Guo et al., 2024), demonstrates a significant improvement in detecting RSDs at various scales. The introduction of an all-in-one YOLO framework for multi-task track component inspection by Wu et al. (2023) marks a further step towards enhancing inspection efficiency, offering a comprehensive solution that encompasses both track components and adjacent areas through UAV imagery. Figure 1 gives an example of processing a raw image to identify different track components and quantify rail surface defects, respectively.
Figure 1. Example of track component and rail surface defect detection. (A) Raw Track Image, (B) Track Component Detection, and (C) Rail Surface Defect Detection.
These innovations represent a leap forward in railroad inspection technology, offering more efficient, accurate, and cost-effective solutions for maintaining rail safety. By leveraging the latest in computer vision and deep learning, researchers are setting new standards in the field, with potential applications extending beyond railroad inspections to other areas of civil engineering and maintenance.
3 Our recent effort in railroad crossing safety and connected community
The FRA report to congress “National Strategy to Prevent Trespassing on Railroad Property” highlights that trespassing is the leading cause of railway-related fatalities in the United States (FRA 2018c). Surpassing even vehicle-train collisions, the toll includes over 1,230 pedestrian rail trespass incidents (both fatalities and injuries) in 2022, with 675 deaths and 555 injuries reported. This issue, alongside the staggering $43 billion financial impact from 2012 to 2016, underscores the urgent need for innovative solutions (FRA, 2024).
Addressing pedestrian behavior at railway crossings presents significant challenges due to the nuanced differences between normal and potentially hazardous actions. The pioneering work by Jiang et al. (2022) introduced a deep learning framework capable of detecting unusual pedestrian behaviors through video analysis and skeleton tracking. This method marks a key step towards understanding and mitigating risks at crossings. Further advancements came from Song et al. (2023), who developed a GAN-based framework for analyzing pedestrian behavior without the need of location-specific adjustments, enhancing its applicability across various settings.
However, dangers at railway crossings are not limited to intentional or unintentional pedestrian actions. A notable incident on 27 June 2022, involved an Amtrak train colliding with a dump truck in Missouri, causing four deaths and injuring about 150 passengers (Jonathan, 2022). Tang et al. (2023) responded to the need for broader detection capabilities with the RC-SAFE Network, a system designed to identify any foreign object at crossings, extending beyond just pedestrian monitoring. Figure 2 provides an example of continuously monitoring of a crossing and the pedestrian and vehicle can be detected and segmented in real time.
Figure 2. Example of continuously object detection at a railroad grade crossing.
Congestion at railway crossings not only causes delays but can also impede emergency response times. Research conducted in Columbia, SC, revealed that all surveyed first responders had faced delays at crossings, sometimes up to 40 min. Early attempts to predict crossing clearance times used computer vision to estimate vehicle queues (Jiang et al., 2021; Guo et al., 2022a), while Guo et al. (2022b) refined this approach with the DTDNet, a CNN designed to count vehicles accurately under various conditions.
To further aid emergency response, Wu et al. (2024) introduced a vehicle dispatching algorithm that dynamically updates optimal routes based on train movements and responder locations, potentially reducing response times by up to 61.6%. This innovation promises significant benefits for community safety and emergency dispatch efficiency.
4 Our future research
Railways play a crucial role in the transportation ecosystem, and our research is dedicated to enhancing railroad safety through the development of advanced, efficient inspection and monitoring systems. Given the increasing demands of track inspection, fueled by a vast network and the growing need for secure and timely freight and passenger transport, our future efforts will pivot towards two primary objectives: cost-effectiveness and autonomy.
Firstly, we aim to pioneer lightweight yet precise computer vision models that can be integrated into cost-effective edge-computing platforms. Despite the availability of numerous inspection systems in the market, their adoption is often limited by high costs. Our objective is to create systems that are both affordable and user-friendly, thereby improving their accessibility and utility for railroad operators.
Secondly, we are committed to exploring autonomous systems capable of conducting inspections and monitoring tasks with minimal human intervention. Track inspection is a labor-intensive activity, requiring inspectors to juggle multiple tasks and process diverse data streams simultaneously. By developing systems that can autonomously handle specific inspection tasks, we aspire to alleviate the workload of human inspectors, enabling them to concentrate on other critical aspects of their role.
This approach not only promises to revolutionize how railroads maintain safety and efficiency but also aligns with the broader goal of advancing transportation technology to meet the challenges of the future.
Author contributions
YQ: Writing–review and editing, Writing–original draft.
Funding
The author(s) declare financial support was received for the research, authorship, and/or publication of this article. We have received funding from the Federal Railroad Administration (FRA), Grant numbers: 693JJ619C000009, 693JJ620C000021, and 693JJ621C000011; the National Academy of Science (NAS) IDEA Program, grant number: RS-361 and RS-412; the University Transportation Center for Railway Safety (UTCRS), and the Center for Connected Multimodal Mobility (C2M2).
Acknowledgments
I sincerely appreciate the Federal Railroad Administration (FRA), the National Academy of Sciences (NAS) through its IDEA program, the University Transportation Center for Railway Safety (UTCRS), and the Center for Connected Multimodal Mobility (C2M2) for their generous funding support. This research benefited significantly from the collaboration and insights provided by our industry partners: BNSF, CSX, Norfolk Southern, MxV Rail, Ensco, the City of Columbia, and the South Carolina Railroad Museum, whose assistance and guidance have been invaluable. I also wish to express my heartfelt appreciation to the dedicated and talented students and collaborators who have contributed to both past and current projects. Their hard work and creativity have been instrumental in advancing our research endeavors. Sharing this journey with such exceptional individuals and organizations has been profoundly rewarding. It is with sincere thanks that I acknowledge their contributions to our work.
Conflict of interest
The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
FRA (2018a). Track and Rail and Infrastructure Integrity Compliance Manual. Retrieved from https://railroads.dot.gov/sites/fra.dot.gov/files/fra_net/17940/CM%20Vol%20II%20Ch1%202018.pdf.
FRA (2018b). Train accidents by cause from FRA F 6180.54. Retrieved from https://safetydata.fra.dot.gov/OfficeofSafety/publicsite/Query/inccaus.aspx.
FRA (2018c). National Strategy to Prevent Trespassing on Railroad Property Report to Congress. Retrieved from https://railroads.dot.gov/elibrary/national-strategy-prevent-trespassing-railroad-property.
FRA (2024). Office of safety analysis dataset. Available at: https://safetydata.fra.dot.gov/OfficeofSafety/default.aspx.
Guo, F., Jiang, Z., Wang, Y., Chen, C., and Qian, Y. (2022b). Dense traffic detection at highway-railroad grade crossings. IEEE Trans. Intelligent Transp. Syst. 23, 15498–15511. doi:10.1109/TITS.2022.3140948
Guo, F., Liu, J., Qian, Y., and Xie, Q. (2024). Rail surface defect detection using A transformer-based network. J. Industrial Inf. Integration 38, 100584. doi:10.1016/j.jii.2024.100584
Guo, F., Qian, Y., Rizos, D., Suo, Z., and Chen, X. (2021c). Automatic rail surface defects inspection based on Mask R-CNN. Transp. Res. Rec. J. Transp. Res. Board 2675 (11), 655–668. Transportation Research Board of the National Academies. doi:10.1177/03611981211019034
Guo, F., Qian, Y., and Shi, Y. (2021b). Real-time railroad track components inspection based on the improved YOLOv4 framework. Automation Constr. 125, 103596. doi:10.1016/j.autcon.2021.103596
Guo, F., Qian, Y., Wu, Y., Leng, Z., and Yu, H. (2021a). Automatic railroad track components inspection using real-time instance segmentation. Computer-Aided Civ. Infrastructure Eng. 36 (3), 362–377. doi:10.1111/mice.12625
Guo, F., Wang, Y., and Qian, Y. (2022a). Computer vision-based approach for smart traffic condition assessment at the railroad grade crossing. Adv. Eng. Inf. 51, 101456. doi:10.1016/j.aei.2021.101456
Jiang, Z., Guo, F., Qian, Y., Wang, Y., and Pan, W. (2021). A deep learning-assisted mathematical model for decongestion time prediction at railroad grade crossings. Neural Comput. Appl. 34, 4715–4732. doi:10.1007/s00521-021-06625-z
Jiang, Z., Song, G., Qian, Y., and Wang, Y. (2022). A deep learning framework for detecting and localizing abnormal pedestrian behaviors at grade crossings. Neural Comput. Appl. 34, 22099–22113. doi:10.1007/s00521-022-07660-0
Jonathan, F. (2022). "4 dead, multiple injured after an Amtrak train hits a truck and derails in Missouri. Washington, D.C.: National Public Radio. Available at: https://www.npr.org/2022/06/27/1107993121/amtrak-train-collides-with-truck-derails-in-missouri.
Robinson, R., Nguyen, L., Moore, W. H., Culotta, K., Hocevar, H., Kimmel, S., et al. (2023). Transportation statistics annual report 2023. United States: Department of Transportation. Bureau of Transportation Statistics. doi:10.21949/1529944
Song, G., Qian, Y., and Wang, Y. (2023). Analysis of abnormal pedestrian behaviors at grade crossings based on semi-supervised generative adversarial networks. Appl. Intell. 53, 21676–21691. doi:10.1007/s10489-023-04639-9
Tang, Y., and Qian, Y. (2024b). High-speed railway track components inspection framework based on YOLOv8 with high-performance model deployment. High-speed Railw. doi:10.1016/j.hspr.2024.02.001
Tang, Y., Wang, Y., and Qian, Y. (2023). Railroad crossing surveillance and foreground extraction network: weakly supervised artificial-intelligence approach. Transp. Res. Rec. J. Transp. Res. Board 2677, 525–538. Transportation Research Board of the National Academies. doi:10.1177/03611981231159406
Tang, Y., Wang, Y., and Qian, Y. (2024a). Edge-computing oriented real-time missing track components detection. Transp. Res. Rec. J. Transp. Res. Board. Transportation Research Board of the National Academies. doi:10.1177/03611981241230546
Tang, Y., Wang, Y., and Qian, Y. (2024c), " Railroad missing components detection via cascade R-CNN with predefined proposal templates (CR-PPT)" Computer-Aided Civil and Infrastructure Engineering. (under review)
Wu, X., Chen, Y., and Qian, Y. (2024). Integrating railroad crossing blockage information in first responder dispatching route planning: a case study in South Carolina. ASCE J. Transp. Eng. Part A Syst. 150 (4), 04024008. doi:10.1061/JTEPBS.TEENG-7925
Wu, Y., Qin, Y., Qian, Y., Chen, P., Xu, F., Zhao, W., et al. (2023). Automatic railroad track components inspection using hybrid deep learning framework. IEEE Trans. Instrum. Meas. 72, 1–15. doi:10.1109/TIM.2023.3265636
Keywords: railroad, track, inspection, computer vision, artificial intelligence
Citation: Qian Y (2024) Intelligent railroad inspection and monitoring. Front. Built Environ. 10:1389092. doi: 10.3389/fbuil.2024.1389092
Received: 20 February 2024; Accepted: 21 March 2024;
Published: 12 April 2024.
Edited by:
Xinbao Yu, University of Texas at Arlington, United StatesReviewed by:
Hailu Yang, University of Science and Technology Beijing, ChinaCopyright © 2024 Qian. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Yu Qian, yuqian@sc.edu