Enhanced YOLOv5s + DeepSORT method for highway vehicle speed detection and multi-sensor verification

Abstract

Addressing the need for vehicle speed measurement in traffic surveillance, this study introduces an enhanced scheme combining YOLOv5s detection with Deep SORT tracking. Tailored to the characteristics of highway traffic and vehicle features, the dataset data augmentation process was initially optimized. To improve the detector’s recognition capabilities, the Swin Transformer Block module was incorporated, enhancing the model’s ability to capture local regions of interest. loss was employed as the loss function for the vehicle detection network, accelerating model convergence and achieving higher regression accuracy. The Mish activation function was utilized to reduce computational overhead and enhance convergence speed. The structure of the Deep SORT appearance feature extraction network was modified, and it was retrained on a vehicle re-identification dataset to mitigate identity switches due to obstructions. Subsequently, using known references in the image such as lane markers and contour labels, the transformation from image pixel coordinates to actual coordinates was accomplished. Finally, vehicle speed was measured by computing the average of instantaneous speeds across multiple frames. Through radar and video Multi-Sensor Verification, the experimental results show that the mean Average Precision (mAP) for target detection consistently exceeds 90%. The effective measurement distance for speed measurement is around 140 m, with the absolute speed error generally within 1–8 km/h, meeting the accuracy requirements for speed measurement. The proposed model is reliable and fully applicable to highway scenarios.

1 Introduction

Intelligent Transportation Systems (ITS) have been widely applied to practical traffic scenarios such as highways, urban roads, tunnels, and bridges. This integration owes much to the convergence of various technologies, including pattern recognition, video image processing, and network communication [1, 2]. Vehicle speed is a crucial parameter that directly reflects the state of traffic [3, 4]. Meanwhile, in highly complex traffic monitoring scenarios and under special weather conditions, intelligent transportation monitoring systems face numerous significant challenges. In addressing the issue of vehicle speeding, the measurement of vehicle speed can provide vital data for traffic management authorities. Accurate measurement of vehicle target speed is one of the challenges faced by traffic monitoring systems.

Traditional vehicle speed detection primarily utilizes inductive loop detection, laser detection, and radar detection. These methods are well-developed and commonly used in traffic systems. However, traditional detection methods have the following disadvantages: (1) the required equipment is expensive; (2) the equipment is installed under the road surface, leading to high subsequent maintenance costs and maintenance not only affects traffic but also damages road structure. Video-based vehicle speed detection leverages numerous traffic video monitoring devices, significantly overcoming the high costs and difficult maintenance issues associated with traditional speed detection methods. The vehicle speed detection system can be categorized into two types: one type focuses on accurate speed monitoring systems (such as speed camera applications) [5, 6], and the other type, though less precise, can be used to estimate traffic speed (such as traffic camera application scenarios) [7, 8]. This classification system takes into account the intrinsic parameters of the camera (such as sensor size and resolution, focal length), as well as extrinsic parameters (such as the camera’s position relative to the road surface, drone-based cameras, etc.), and the number of cameras (monocular, stereo, or multiple cameras).

Through these parameters, the actual scene on the image plane can represent one or multiple lanes, as well as the relative position of vehicles to the camera, ultimately yielding one of the most critical variables: the ratio of pixels to road segment length, i.e., the road length each pixel represents. Due to the perspective projection model, this ratio is directly proportional to the square of the camera’s distance, implying that measurements over long distances have poor accuracy. Accurate estimation of the camera’s intrinsic and extrinsic parameters is required to provide measurements in the actual coordinate system. The most common approach is soft calibration, which involves calibrating intrinsic parameters in a verification laboratory or using sensor and lens characteristics, and estimating the rigid transformation between the camera and the road surface using manual [9, 10] or automatic [11] methods.

Hard calibration involves estimating both the intrinsic and extrinsic parameters of the camera, which can be done either manually [12] or automatically [13–15]. In certain limited scenarios, some details of camera calibration may be overlooked, such as the exact position of the camera, anchoring systems, gantries. Since cameras are mostly static (except for drone cameras), vehicle detection is most often addressed by modeling the background [16–18]. Other methods are feature-based, such as detecting vehicle license plates [19, 20] or other characteristics [21–23].

Recently, learning-based approaches have become increasingly popular for recognizing vehicles in images [24, 25]. The ability to track vehicles with smooth and stable trajectories is a key issue in handling vehicle speed detection. Vehicle tracking can be divided into three different categories: The first category is feature-based [26–28], where tracking originates from a set of features of the vehicle (such as optical flow). The second category focuses on tracking the centroid of a vehicle’s blob or bounding box [29, 30]. The third category concentrates on tracking the entire vehicle [31, 32] or its specific parts (such as the license plate [33, 34]).

The prerequisite for speed measurement is the effective assessment of distance. In monocular vision systems, the estimation of vehicle distance typically relies on specific constraints and methods. These include: (1) Flat road assumption and homography-based methods, which assume that the road is flat and apply a mathematical transformation known as homography [35, 36], helping in mapping the view of a scene from one perspective to another, which is crucial for estimating distances in 2D images; (2) Detection of lines and specific areas [37, 38]. By detecting lines and specific areas, designed detection lines and areas can be overlaid on the real-world view, providing a reference scale for measuring distances; (3) Use of prior knowledge about object dimensions, utilizing the known dimensions of certain objects to estimate distances. For instance, knowing the standard sizes of license plates ([39, 40]) or the average dimensions of vehicles [41] can assist in calibrating distance measurements. However, these monocular methods have limitations, which are addressed in stereo vision systems. In stereo vision systems [42], two cameras are used to capture the same scene from slightly different angles, similar to human binocular vision. This setup allows for more accurate depth perception and distance estimation, as it mimics the way.

Currently, speed detection is primarily divided into macroscopic traffic flow speed and individual vehicle speed. Macroscopic traffic flow speed detection is based on a specific road section, using the length of the section and travel time to estimate the average speed of the segment [43, 44]. Individual vehicle speed detection focuses on the micro-level speed of the vehicle itself, presenting greater technical challenges. This process requires prior knowledge of the camera’s frame rate or accurate timestamps for each image to calculate the time between measurements. Utilizing consecutive or non-consecutive [45] images to estimate speed is a key factor impacting accuracy. In summary, whether in traffic flow speed or individual vehicle speed detection, factors such as the method of image capture (continuous or non-continuous), frame rate, timestamp accuracy, and the integration of various measurement data need to be carefully considered. The selection method and precision of these factors directly affect the accuracy of speed estimation.

In summary, vision-based vehicle speed detection involves the entire process of camera calibration, distance estimation, and speed estimation. However, the calibration process for monocular vision cameras is complex, the accuracy of distance estimation is relatively poor, and the precision of individual vehicle speed estimation needs improvement. Currently, there are few instances of rapidly detecting and stably tracking vehicle instantaneous speeds solely through video recognition technology, which limits the broader application of video recognition technologies in the field of traffic safety. Therefore, this study introduces an enhanced scheme that combines YOLOv5s detection with Deep SORT tracking, targeting the need for vehicle speed measurement in traffic monitoring. The dataset data expansion process is preliminarily optimized based on the characteristics of highway traffic and vehicle features. The Swin Transformer Block module is introduced to improve the detector’s recognition capabilities and enhance the model’s ability to capture areas of interest. The loss is employed as the loss function for the vehicle detection network to accelerate model convergence and achieve higher regression precision. The Mish activation function is used to reduce computational costs and improve convergence speed. Modifications are made to the structure of the Deep SORT appearance feature extraction network, and it is retrained on the vehicle re-identification dataset to mitigate identity switches caused by obstacles. Subsequently, known references in the image, such as lane markings and contour labels, are used to complete the conversion from image pixel coordinates to actual coordinates through maximum likelihood estimation, maximum posterior estimation, and non-linear least squares methods. Finally, vehicle speed is measured by calculating the average of instantaneous speeds over multiple frames. The algorithm can detect and track vehicle targets without prior camera parameters and calibration, extract known reference information such as lane lines and contour labels, and automatically convert pixel coordinates to actual coordinates in traffic monitoring scenes, as well as automatically measure vehicle speeds, the algorithm framework as shown in Figure 1. Accurate estimation of vehicle speed can support the detection of traffic accidents and incidents, offering scientific technical means for active safety management in intelligent transportation systems.

FIGURE 1

2 Improved YOLOv5s + DeepSORT algorithm for highway vehicle detection and tracking

2.1 Construction of vehicle target dataset

2.1.1 Characteristics of highway traffic scenarios

FIGURE 2

The existence of these four types of issues makes large public datasets such as COCO and VOC unsuitable for the perspectives captured by highway cameras, leading to a large number of false positives and missed detections of target vehicles.

2.1.2 Data preparation

Given the relatively uniform types of motor vehicles in highway scenarios, vehicles are generally classified into three categories: Car, Bus, and Truck. Car mainly refer to passenger vehicles with seating for fewer than seven people; Bus mainly include commercial buses, public transport buses, etc.; Truck primarily refer to small, medium, and large trucks, trailers, and various types of special-purpose vehicles as shown in Table 1. By collecting datasets from different scenes on highways and manually labeling them using the labelImg tool, a dataset in YOLO format was ultimately created.

The specific process includes: (1) Data Collection: Collect representative image data covering various scenes and angles of target categories. (2) Data Division: Divide the dataset into training, validation, and test sets, typically in a certain ratio, to ensure the independence and generalizability of the data. (3) Bounding Box Annotation: Annotate each target object with a bounding box, usually represented by a rectangle, including the coordinates of the top-left and bottom-right corners. Category Labeling: Assign corresponding category labels to each target object, identifying the category to which the object belongs. During dataset annotation, rectangular bounding boxes encompassing the entire vehicle are marked, with each side fitting closely to the vehicle. Annotation is not performed when the occlusion exceeds 50%, the vehicle type is indistinguishable, or the size is below 10*10 pixels. Furthermore, in cases where vehicles are truncated, the truncation is not considered to affect the overall annotation. Trucks used for transportation are uniformly annotated, without separately marking the vehicles on them.

2.1.3 Data augmentation

To enhance the accuracy and generalization capability of model training, data augmentation techniques are employed, tailored to the characteristics of highway traffic environments and vehicle features. These techniques include Mosaic, Random_perspective, Mixup, HSV, Flipud, Fliplr, as shown in Figure 3.

FIGURE 3

2.2 Optimization of object detection network

In response to the identified issues with YOLOv5 in highway vehicle detection, the following optimizations were made to enhance the accuracy of vehicle detection: (1) Incorporating the Swin Transformer Block module to improve the model’s ability to capture information from local areas of interest; (2) Utilizing loss as the loss function for the vehicle detection network to accelerate model convergence and achieve higher regression accuracy; (3) Adopting the Mish activation function to reduce computational overhead and increase convergence speed.

2.2.1 Introduction of swin transformer block

To address the shortcomings of traditional YOLOv5 in traffic object detection, the Swin Transformer Block module is introduced for optimization.

The Swin Transformer network [46], proposed in 2021, is a Transformer network enhanced with a local self-attention mechanism. It has stronger dynamic computation capabilities compared to convolutional neural networks, with enhanced modeling capacity, and can adaptively compute both local and global pixel relationships, making it highly valuable for widespread use.

The core modules of the Transformer Block overall architecture are the Window-based Multi-Head Self-Attention layer (W-MSA) and the Shifted Window-based Multi-Head Self-Attention layer (SW-MSA). By restricting attention computation within a window, the network not only introduces the locality of convolution operations but also saves computational resources, resulting in good performance.

This article proposes the integration of the Swin Transformer Block structure into the backbone feature extraction network and neck feature fusion, utilizing the efficient self-attention mechanism module to fully explore the potential of feature representation. The improved YOLOv5 network incorporating the Swin Transformer Block module is shown in Figure 4, named SwinTransYOLOv5 network.

FIGURE 4

2.2.2 Improvement of loss function

YOLOv5s employs loss as the bounding box regression loss function to evaluate the distance between the predicted bounding box (PB) and the ground truth bounding box (GT), as shown in Eq. 1.

In the formula, represents the intersection over union of PB and GT, is the area of the smallest rectangular box containing both PB and GT, is the union of PB and GT, and is the loss. The advantage of loss is its scale invariance, meaning the similarity between PB and GT is independent of their spatial scale. The problem with Loss is that when either PB or GT completely encompasses the other, Loss degenerates entirely into loss. Because it heavily relies on the term, this results in slow convergence during actual training and lower accuracy of the predicted bounding boxes. To address these issues, loss also considers the overlapping area of PB and GT, the distance between their centroids, and their aspect ratios, as shown in Eq. 2.

In the formula, and represent the centroids of PB and GT, denotes the Euclidean distance, is the length of the shortest diagonal of the smallest enclosing box of PB and GT, represents a positive balance parameter, and indicates the consistency of the aspect ratio of PB and GT. The definitions of and are as follows in Eq. 3.

In the formula, , and , respectively represent the width and height of GT and PB.

Compared to the loss used in YOLOv5s, loss incorporates penalty terms for the distance between the centers of PB and GT, as well as their aspect ratios in the loss function. This ensures faster convergence of the predicted bounding boxes during training and yields higher regression localization accuracy. In This article, loss is adopted as the loss function for the vehicle detection network.

2.2.3 Activation function

Changing the activation function can significantly enhance recognition performance. Activation functions are categorized into saturated and non-saturated types. The primary advantages of using non-saturated activation functions are twofold [47]: firstly, they effectively address the vanishing gradient problem, which becomes more severe with saturated activation functions; secondly, they can accelerate the convergence speed. After comparing the pros, cons, and characteristics of various activation functions without significantly increasing computational load, as shown in Table 2, the Leaky ReLU activation function in YOLOv5 was replaced with the Mish activation function.

TABLE 1

Type	Example
Car
Bus
Truck

Dataset categorization.

TABLE 2

	Sigmoid	tanh	ReLU	Leaky ReLU	Mish
Function graphs
Function Formula
Advantages	Can restrict the output to be between (0, 1), facilitating the completion of classification tasks	①Can restrict the output to be between (−1, 1), facilitating the completion of classification tasks	Linear: Saves computational resources and shortens convergence time	①Linear	①Linear
		②Zero-Centered		②Gradient non-saturation, no neuron death	②Gradient non-saturation, no neuron death
					③The network’s convergence is the best among the five activation functions
Disadvantages	①The output is not zero-centered, leading to a zigzag pattern in gradient descent	②Gradient saturation, Gradient vanishing	Neuron Death: The left side of the ReLU function is completely flat. When the neuron’s -value is negative, the output is 0, and the gradient is also 0, making it impossible to alter the weight value through the gradient, leaving unchanged	The network’s convergence is not advantageous compared to the latest networks	Relatively higher computational cost
	②Gradient saturation, Gradient vanishing	③Non-linear
	③Non-linear, involves exponential operations, consuming more resources during computation

Comparison of common activation functions.

2.3 Optimization of deep SORT for vehicle tracking

The multi-object online tracking algorithm SORT [48] (Simple Online and Realtime Tracking) utilizes Kalman filtering and Hungarian matching, using the between tracking and detection results as the cost matrix, to implement a simple, efficient, and practical tracking paradigm. However, the SORT algorithm’s limitation lies in its association metric being effective only when the uncertainty in state estimation is low, leading to numerous identity switches and tracking failures when the target is occluded. To address this issue, Deep SORT [49] combines both motion and appearance information of the target as the association metric, improving tracking failures caused by the target’s disappearance and reappearance.

2.3.1 Tracking processing and state estimation

The Mahalanobis distance is used to evaluate the predicted Kalman state and the new state, as shown in Eq. 4.

represents the motion matching degree between the j detection and the i trajectory, where is the covariance matrix of the observation space at the current moment predicted by the Kalman filter for the trajectory, is the predicted observation of the trajectory at the current moment, and is the state of the j detection.

While Mahalanobis distance is a good measure of association when the target’s motion uncertainty is low, it becomes ineffective in practical situations like camera movement, leading to a large number of mismatches. Therefore, we integrate a second metric. For each BBox detection, we compute an appearance feature descriptor. We create a gallery to store the descriptors of the latest 100 trajectories and then use the minimum cosine distance between the i and j trajectories as the second measure, as shown in Eq. 6.

Can be represented using a threshold function, as shown in Eq. 7.

Mahalanobis distance can provide reliable target location information in short-term predictions, and the cosine similarity of appearance features can recover the target ID when the target is occluded and reappears. To make the advantages of both measures complementary, a linear weighting approach is used for their combination, as shown in Eqs 8, 9.

The strategy of cascaded matching is used to improve matching accuracy, mainly because when a target is occluded for a long time, the uncertainty of Kalman filtering greatly increases, leading to a dispersion of continuous prediction probabilities. Assuming the original covariance matrix is normally distributed, continuous predictions without updates will increase the variance of this normal distribution, so points far from the mean in Euclidean distance may obtain the same Mahalanobis distance value as points closer in the previous distribution. In the final stage, the authors use IOU association from the previous SORT algorithm to match unconfirmed and unmatched trajectories. This can alleviate significant changes caused by abrupt appearance shifts or partial occlusions. However, this approach may also connect some newly generated trajectories to older ones.

2.3.2 Deep appearance features

The original algorithm uses a residual convolutional neural network to extract the appearance features of the target, training the model on a large-scale pedestrian re-identification dataset for pedestrian detection and tracking. Since the original algorithm was only used for the pedestrian category and the input images were scaled to 128 × 64, which does not match the aspect ratio of vehicle targets, this article improves the network model by adjusting the input image size to 128 × 128, as shown in Table 3. The adjusted network is then re-identification trained on the vehicle re-identification dataset VeRi [50].

3 Vehicle speed measurement

3.1 Model assumptions

All locations in road monitoring images can be mapped to the plane of the world coordinate system through camera calibration, as shown in Figure 5. However, the precise measurement of vehicle speed depends not only on camera calibration but also significantly on the vehicle’s trajectory. To better implement vehicle speed measurement, the speed model assumes the following: (1) In highway scenarios, the road is relatively flat without significant undulations, meeting the condition of ; (2) In highway monitoring scenarios, the movement of vehicles between each frame is linear, allowing for the measurement of vehicles moving in both straight and non-straight paths using the proposed speed measurement method; (3) In highway video surveillance, the time interval between each frame is the same, facilitating the calculation of vehicle speed after obtaining the exact vehicle position using the interval between frames.

FIGURE 5

3.2 Model design and implementation

Based on the assumptions and establishment of the aforementioned speed model, the specific process of speed detection is implemented. Firstly, using the YOLO object detection algorithm, the coordinates of the top-left corner of the image detection box are obtained. By determining the length and width of the detection box, the coordinates of the center of the bottom edge of the box can be obtained. This ensures that the measured vehicle speed is closer to the actual speed. For every target vehicle in each frame of the video stream, a set of vector relations can be obtained, as shown in Eq. 10.

Here, represents the center coordinates of the bottom edge of the vehicle target detection box in the current video frame; represents the center coordinates of the bottom edge of the vehicle target detection box in the previous frame; is the time interval between the two frames; represents the tracked trajectory points.

represents the pixel distance between adjacent frames, and calculating the speed requires mapping the pixel coordinates to world coordinates. The current common method involves camera calibration, but camera calibration requires knowledge of the camera’s focal length, height, internal parameters, etc., and the calibration process can be cumbersome.

Maximum Likelihood Estimation (MLE) is an important and widely used method for estimating quantities. MLE explicitly uses a probability model with the goal of finding a system occurrence tree that can produce observed data with a high probability. MLE is a representative of a class of system occurrence tree reconstruction methods based entirely on statistics. Given a set of data, if we know it is randomly taken from a certain distribution, but we don't know the specific parameters of this distribution, that is, “the model is determined, but the parameters are unknown.” For example, we know the distribution is a normal distribution, but we don’t know the mean and variance; or it’s a binomial distribution, but we don’t know the mean. MLE can be used to estimate the parameters of the model. The objective of MLE is to find a set of parameters that maximize the probability of the model producing the observed data, as shown in Eq. 11.

Here, represents the observed sequence data, and is the likelihood function, which denotes the probability of the observed data occurring under the parameter . Assuming each observation is independent, as shown in Eq. 12.

In Bayesian statistics, Maximum A Posteriori (MAP) Estimation refers to the mode of the posterior probability distribution. MAP estimation is used to estimate the values of quantities that cannot be directly observed in experimental data. It is closely related to the classical method of Maximum Likelihood Estimation (MLE), but it uses an augmented optimization objective that further considers the prior probability distribution of the quantity being estimated. Therefore, MAP estimation can be seen as a regularized form of MLE, as shown in Eqs 15, 16.

The Least Squares Method (also known as the Method of Least Squares) is a mathematical optimization technique. It finds the best function match for data by minimizing the sum of the squares of the errors. The Least Squares Method can be used to easily obtain unknown data, ensuring that the sum of the squares of the errors between these obtained data and the actual data is minimized. The Least Squares Method can also be used for curve fitting, and other optimization problems can be expressed using this method by minimizing energy or maximizing entropy. Using the Least Squares Method to estimate the mapping relationship, the mapping parameters are obtained, as shown in Eqs 17, 18.

Where is a nonlinear function, and is the covariance matrix.

Then, the Gauss-Newton method is used to solve for ψ(x), as shown in Eq. 19:

For the sum of errors, we investigate the i term, also performing a second-order Taylor expansion, followed by differentiation. We first calculate its first-order derivative (gradient) and second-order derivative.

First-order derivative, as shown in Eqs 20, 21.

Where is the element in the column of the row of the Jacobian matrix, thus the first-order derivative can also be expressed in the following form, as shown in Eq. 22.

Second-order derivative, as shown in Eq. 23.

Observing the result of the second-order derivative, the terms and are elements of the Jacobian matrix. When the iterative point is far from the target point, both the error and its second-order derivative are small and can be ignored. Therefore, the second-order derivative can be expressed in the following form, as shown in Eq. 24.

Therefore, after the second-order expansion, can be written in the following form, as shown in Eq. 25:

Similarly, by differentiating it and setting the derivative equal to zero, Eq. 26:

Let then, as shown in Eq. 27:

3.3 Vehicle speed measurement

Through prior estimation, and can be mapped to the world coordinate system, representing the actual distance moved by the target vehicle from the previous frame to the current frame, as shown in Eq. 28. is measured in meters and is the Euclidean norm of , representing the physical distance moved by the target vehicle in the world coordinate system from time to . The speed of the vehicle target can be measured using as Eq. 29. Here, is the time between two frames, measured in seconds, and is considered constant, being the reciprocal of the frame rate. For highway surveillance videos, which typically have a frame rate of 25 fps, = 1/25.

Assuming a vehicle’s trajectory contains m frame trajectory points, meaning in the first m frames of the video, the vehicle’s speed between each adjacent pair of frames is , then according to Eq. 29, v1, v2, vm⁻¹ as shown in Eqs 30–32:

Therefore, the average driving speed of the target vehicle in the first m frames is as shown in Eq. 33. The detection of the target vehicle’s speed is achieved by calculating the average of the instantaneous speeds over multiple frames.

4 Model training and evaluation metrics selection

4.1 Experimental environment and model training

Experimental setup and hardware environment for the dataset: System Type: Windows 10 64-bit Operating System, Memory: 64GB, GPU: NVIDIA GeForce RTX3080ti, 24 GB Graphics Card. Software environment: The auxiliary environment includes CUDA V11.2, OpenCV4.5.3. This article tested different corresponding datasets for various traffic scenarios. The dataset established in This article comprises a total of 30,000 images, including a diverse collection from different scenes, angles, and times.

During training, 80% of the dataset was used for training, while 20% of the data was reserved for testing. Data augmentation was applied in this study, which involved random scaling, cropping, and arrangement of images using the Mosaic method. Random rotation (parameter set to 0.5), random exposure (parameter set to 1.5), and saturation (parameter set to 1.5) were employed to enrich the training data. The learning rate was initially set to 0.001, and the maximum number of training iterations was set to 50,000. To optimize model convergence, the learning rate was adjusted to 0.0005 after 40,000 iterations. The input images to the network were resized to a resolution of 416 × 416, and a batch size of 8 was used during training to ensure efficient network processing. The convergence of the model’s training loss and mAP (mean Average Precision) can be observed in Figure 6. It shows that the model converged around 3,000 iterations, and as the loss decreased, mAP also reached a high level.

FIGURE 6

Convolutional Neural Networks (CNNs) are capable of extracting key features from image objects. The detected objects are classified into three categories: Car, Truck, and Bus. The unique features of each class can be observed in Figure 7, where each class of object exhibits distinct characteristics within the convolutional network. These distinct features are used for classification and detection purposes.

FIGURE 7

4.2 Selection of evaluation metrics

To verify the effectiveness of the model’s detection, several typical metrics in the field of object detection and classification were selected for evaluation. For distracted driving behavior detection and classification, the focus is on detection precision and recall rate, as well as classification accuracy. Therefore, the model is evaluated using precision, recall, and F1_Score.

AP (Average Precision) is the average accuracy and a mainstream evaluation metric for object detection models. To correctly understand AP, it is necessary to use three concepts: Precision, Recall, and (Intersection over Union). measures the degree of overlap between two areas, specifically the overlap rate between the target window generated by the model and the originally marked window, which represents the detection accuracy . The calculation formula is shown in Eq. 34. In an ideal situation, equals 1, indicating a perfect overlap.

Precision and Recall in object detection: Assuming a set of images containing several targets for detection, Precision represents the proportion of targets detected by the model that are actual target objects, while Recall represents the proportion of all real targets detected by the model. TP (True Positive) denotes samples correctly identified as positive, TN (True Negative) denotes samples correctly identified as negative, FP (False Positive) denotes samples incorrectly identified as positive, and FN (False Negative) denotes samples incorrectly identified as negative. The calculation of Precision and Recall values relies on the formulas shown in Eqs 35, 36.

After calculating values using the formula, a PR (Precision-Recall) curve can be plotted. The AP (Average Precision) is the mean of Precision values on the PR curve. To achieve more accurate results, the PR curve is smoothed, and the area under the smoothed curve is calculated using integral methods to determine the final AP value. The calculation formula is shown as Eq. 37.

The F1-Score, also known as the F1 measure, is a metric for classification problems, often used as the final metric in multi-class problems. It is the harmonic mean of precision and recall. For the F1-Score of a single category, the calculation formula is as shown in Eq. 38.

Subsequently, calculate the average value for all categories, denoted as F1. The calculation formula is shown in Eq. 39.mAP (mean Average Precision) involves calculating the AP (Average Precision) for all categories and then computing the mean. The calculation formula is shown in Eq. 40.

5 Results and discussion

5.1 Evaluation of object detection model results

Based on the aforementioned evaluation metrics, the trained object detection models are tested and assessed using the test sets from the datasets. The algorithm shows good statistical accuracy for different vehicle types, with APs of Car, Bus, Truck being 93.58, 91.26, 90.05 respectively, mAP at 92.42, and F1_Score at 97. This is primarily due to the high visibility in tunnel and roadbed sections, where target features are more distinct, resulting in a more accurate model. Overall, the model’s detection accuracy for buses is lower than for other categories, mainly because the sample size for buses is significantly smaller than for other categories. However, with a mean Average Precision (mAP) exceeding 90%, it demonstrates that the proposed model is reliable and fully applicable to highway scenarios.

5.2 Evaluation of speed estimation results

5.2.1 Selection of optimal fitting model

Based on the data distribution, This article selects 7 video points for fitting analysis with 7 sets of linear and nonlinear data. This curve relationship is not intuitively obvious but requires statistical testing. The optimal fitting model is chosen by comparing the degree of fit and its significance. The Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) are two commonly used indicators for assessing model fitness, with smaller values indicating a better-fitting model. Therefore, before selecting a model, it is necessary to assess the AIC and BIC values for each model, including dependent and independent variables. Additionally, the goodness of fit and p-value are also key parameters for evaluating the quality of the fit. As the data distribution within the range of road video surveillance is essentially similar in terms of distance calibration, a random surveillance point is selected for the fitting analysis of the 7 formulas, with results as shown in Table 4.

From Table 4, it is evident that apart from linear fitting, the goodness of fit for all other methods is greater than 0.8. Among them, the fitting shows the best performance, hence is chosen as the formula for distance-speed fitting.

To obtain the best fitting parameters for , employing Maximum Likelihood Estimation, Maximum A Posteriori Estimation, and Non-linear Least Squares method for parameter estimation on the distance calibration data from 7 video points. The parameters are evaluated using AIC, BIC, , and p-value, with the evaluation results presented in Table 5; Figure 8.

FIGURE 8

From the above table, it is clear that for the parameter estimation of the 7 video points, Maximum Likelihood Estimation shows the best performance, followed by Maximum A Posteriori Estimation, and lastly Non-linear Least Squares method, as indicated by AIC, BIC, , and p-value.

5.2.2 Speed estimation results

To evaluate the measurement results of the speed estimation method, based on radar and video multi-sensor fusion technology, the results measured by millimeter-wave radar are taken as the true speed values. The verification experiment was conducted in the Shimen Tunnel on the Hanping Expressway in Shaanxi China, where radar and video integration devices were installed at 150-m intervals, totaling seven units, to achieve holographic perception of traffic flow states within a 1050-m range, obtaining detailed information on coordinates, lane positions, and speeds for different lanes and vehicle types. Vehicle speeds detected by millimeter-wave radar and video were extracted using timestamps and target IDs. The comparison between the measured results and the true speed values, along with the overall experimental results and performance analysis, are shown in Table 6.

From Table 6, it is observed that the vehicle speed measurement method based on video, as discussed in This article, shows relatively good performance in scenarios with high overall speeds on highways. The minimum root mean square error is 2.0635, and the maximum is 9.2797. The main reasons for the larger deviation between the measured speeds and the actual values are environmental conditions, such as lighting and line shape. The coefficient of determination ranges from a minimum of 0.68259 to a maximum of 0.97730. The variation in the goodness of fit is for the same reasons as the minimum mean square error. Additionally, to further evaluate the speed tracking performance of this method, the vehicle speed measurement data from 7 video locations are manually divided into Front section, Middle section, Back section, and End section, for a comprehensive analysis of the overall tracking effect in these four segments, as seen in Figure 9.

FIGURE 9

As depicted in Figure 9, the effective measurement distance of this method is around 140 m, with the absolute speed error generally within 1–8 km/h, meeting the accuracy requirements for speed measurement. This method has certain advantages in distance detection, especially in tunnel scenarios, where a camera spacing of 150 m allows for continuous tracking of vehicle trajectories and speeds based on video. For further analysis of speed tracking differences within the 150 m detection range, it's divided into The first half and The second half. The first half data shows a minimum significance level of 0.4261, indicating small differences in speed tracking, reflecting stable tracking performance. The second half data has a minimum significance level of 0.0179, indicating some fluctuations in speed in the End section of The second half, but the absolute speed error still shows good precision.

6 Conclusion

References

• 1.

  WanYHuangYBucklesB. Camera calibration and vehicle tracking: highway traffic video analytics. Transp Res C Emerg Technol (2014) 44:202–13. 10.1016/j.trc.2014.02.018

  • CrossRef
  • Google Scholar
  • View reference in article

• 2.

  KarońGMikulskiJ. Selected problems of transport modelling with ITS services impact on travel behavior of users. In: 2017 15th International Conference on ITS Telecommunications (ITST); 29-31 May 2017; Warsaw, Poland. IEEE (2017). p. 1–7. 10.1109/ITST.2017.7972231

  • CrossRef
  • Google Scholar
  • View reference in article

• 3.

  WangYYuCHouJChuSZhangYZhuY. ARIMA model and few-shot learning for vehicle speed time series analysis and prediction. Comput Intell Neurosci (2022) 2022:1–9. 10.1155/2022/2526821

  • CrossRef
  • Google Scholar
  • View reference in article

• 4.

  JiaSPengHLiuS. Urban traffic state estimation considering resident travel characteristics and road network capacity. J Transportation Syst Eng Inf Tech (2011) 11:81–5. 10.1016/S1570-6672(10)60142-0

  • CrossRef
  • Google Scholar

• 5.

  JavadiSDahlMPetterssonMI. Vehicle speed measurement model for video-based systems. Comput Electr Eng (2019) 76:238–48. 10.1016/j.compeleceng.2019.04.001

  • CrossRef
  • Google Scholar
  • View reference in article

• 6.

  DahlMJavadiS. Analytical modeling for a video-based vehicle speed measurement framework. Sensors (Switzerland) (2020) 20(1):160. 10.3390/s20010160

  • CrossRef
  • Google Scholar
  • View reference in article

• 7.

  KhanASarkerDMSZRayamajhiS. Speed estimation of vehicle in intelligent traffic surveillance system using video image processing. Int J Sci Eng Res (2014) 5(12):1384–90. 10.14299/ijser.2014.12.003

  • CrossRef
  • Google Scholar
  • View reference in article

• 8.

  WicaksonoDWSetiyonoB. Speed estimation on moving vehicle based on digital image processing. Int J Comput Sci Appl Math (2017) 3(1):21–6. 10.12962/j24775401.v3i1.2117

  • CrossRef
  • Google Scholar
  • View reference in article

• 9.

  LuSWangYSongH. A high accurate vehicle speed estimation method. Soft Comput (2020) 24:1283–91. 10.1007/s00500-019-03965-w

  • CrossRef
  • Google Scholar
  • View reference in article

• 10.

  LiuCHuynhDQSunYReynoldsMAtkinsonS. A vision-based pipeline for vehicle counting, speed estimation, and classification. IEEE Trans Intell Transportation Syst (2021) 22:7547–60. 10.1109/TITS.2020.3004066

  • CrossRef
  • Google Scholar
  • View reference in article

• 11.

  BhardwajRTummalaGKRamalingamGRamjeeRSinhaP. AutoCalib: automatic traffic camera calibration at scale. ACM Trans Sen Netw (2018) 14(3-4):1–27. 10.1145/3199667

  • CrossRef
  • Google Scholar
  • View reference in article

• 12.

  QiminXXuLMingmingWBinLXianghuiS. A methodology of vehicle speed estimation based on optical flow. In: Proceedings of 2014 IEEE International Conference on Service Operations and Logistics, and Informatics; 08-10 October 2014; Qingdao, China. IEEE (2014). p. 33–7. 10.1109/SOLI.2014.6960689

  • CrossRef
  • Google Scholar
  • View reference in article

• 13.

  SchoepflinTNDaileyDJ. Dynamic camera calibration of roadside traffic management cameras for vehicle speed estimation. IEEE Trans Intell Transportation Syst (2003) 4:90–8. 10.1109/TITS.2003.821213

  • CrossRef
  • Google Scholar
  • View reference in article

• 14.

  HanJHeoOParkMKeeSSunwooM. Vehicle distance estimation using a mono-camera for FCW/AEB systems. Int J Automotive Tech (2016) 17:483–91. 10.1007/s12239-016-0050-9

  • CrossRef
  • Google Scholar
  • View reference in article

• 15.

  SochorJJuranekRSpanhelJMarsikLSirokyAHeroutAet alComprehensive data set for automatic single camera visual speed measurement. IEEE Trans Intell Transportation Syst (2019) 20:1633–43. 10.1109/TITS.2018.2825609

  • CrossRef
  • Google Scholar
  • View reference in article

• 16.

  LinH-YLiK-JChangC-H. Vehicle speed detection from a single motion blurred image. Image Vis Comput (2008) 26:1327–37. 10.1016/j.imavis.2007.04.004

  • CrossRef
  • Google Scholar
  • View reference in article

• 17.

  CelikTKusetogullariH. Solar-powered automated road surveillance system for speed violation detection. IEEE Trans Ind Elect (2010) 57:3216–27. 10.1109/TIE.2009.2038395

  • CrossRef
  • Google Scholar
  • View reference in article

• 18.

  NguyenTTPhamXDSongJHJinSKimDJeonJW. Compensating background for noise due to camera vibration in uncalibrated-camera-based vehicle speed measurement system. IEEE Trans Veh Technol (2011) 60:30–43. 10.1109/TVT.2010.2096832

  • CrossRef
  • Google Scholar
  • View reference in article

• 19.

  EslamiHRaieAAFaezK. Precise vehicle speed measurement based on a hierarchical homographic transform estimation for law enforcement applications. IEICE Trans Inf Syst (2016) E99.D:1635–44. 10.1587/transinf.2015EDP7371

  • CrossRef
  • Google Scholar
  • View reference in article

• 20.

  FamouriMAzimifarZWongA. A novel motion plane-based approach to vehicle speed estimation. IEEE Trans Intell Transportation Syst (2019) 20:1237–46. 10.1109/TITS.2018.2847224

  • CrossRef
  • Google Scholar
  • View reference in article

• 21.

  LiJChenSZhangFLiEYangTLuZ. An adaptive framework for multi-vehicle ground speed estimation in airborne videos. Remote Sens (Basel) (2019) 11:1241. 10.3390/rs11101241

  • CrossRef
  • Google Scholar
  • View reference in article

• 22.

  KoyuncuHKoyuncuB. Vehicle Speed detection by using Camera and image processing software. Int J Eng Sci (Ghaziabad) (2018) 7:64–72. 10.9790/1813-0709036472

  • CrossRef
  • Google Scholar
  • View reference in article

• 23.

  KimJ-HOhW-TChoiJ-HParkJ-C. Reliability verification of vehicle speed estimate method in forensic videos. Forensic Sci Int (2018) 287:195–206. 10.1016/j.forsciint.2018.04.002

  • CrossRef
  • Google Scholar
  • View reference in article

• 24.

  SochorJJuránekRHeroutA. Traffic surveillance camera calibration by 3D model bounding box alignment for accurate vehicle speed measurement. Computer Vis Image Understanding (2017) 161:87–98. 10.1016/j.cviu.2017.05.015

  • CrossRef
  • Google Scholar
  • View reference in article

• 25.

  PalubinskasGKurzFReinartzP. Model based traffic congestion detection in optical remote sensing imagery. Eur Transport Res Rev (2010) 2:85–92. 10.1007/s12544-010-0028-z

  • CrossRef
  • Google Scholar
  • View reference in article

• 26.

  DoǧanSTemizMSKülürS. Real time speed estimation of moving vehicles from side view images from an uncalibrated video camera. Sensors (2010) 10(5):4805–24. 10.3390/s100504805

  • CrossRef
  • Google Scholar
  • View reference in article

• 27.

  LiSYuHZhangJYangKBinR. Video-based traffic data collection system for multiple vehicle types. IET Intell Transport Syst (2014) 8:164–74. 10.1049/iet-its.2012.0099

  • CrossRef
  • Google Scholar

• 28.

  JeyabharathiDDejeyDD. Vehicle tracking and speed measurement system (VTSM) based on novel feature descriptor: diagonal hexadecimal pattern (DHP). J Vis Commun Image Represent (2016) 40:816–30. 10.1016/j.jvcir.2016.08.011

  • CrossRef
  • Google Scholar
  • View reference in article

• 29.

  AgrawalSCTripathiRK. An image processing based method for vehicle speed estimation. Int J Scientific Tech Res (2020) 9:1241–6.

  • Google Scholar
  • View reference in article

• 30.

  BiswasDSuHWangCStevanovicA. Speed estimation of multiple moving objects from a moving UAV platform. ISPRS Int J Geoinf (2019) 8(6):259. 10.3390/ijgi8060259

  • CrossRef
  • Google Scholar
  • View reference in article

• 31.

  LeeJRohSShinJSohnK. Image-based learning to measure the space mean speed on a stretch of road without the need to tag images with labels. Sensors (Switzerland) (2019) 19:1227. 10.3390/s19051227

  • CrossRef
  • Google Scholar
  • View reference in article

• 32.

  DongHWenMYangZ. Vehicle speed estimation based on 3D ConvNets and non-local blocks. Future Internet (2019) 11(6):123. 10.3390/fi11060123

  • CrossRef
  • Google Scholar
  • View reference in article

• 33.

  LuvizonDCNassuBTMinettoR. A video-based system for vehicle speed measurement in urban roadways. IEEE Trans Intell Transportation Syst (2017) 18:1–12. 10.1109/TITS.2016.2606369

  • CrossRef
  • Google Scholar
  • View reference in article

• 34.

  YangLLiMSongXXiongZHouCQuB. Vehicle speed measurement based on binocular stereovision system. IEEE Access (2019) 7:106628–41. 10.1109/ACCESS.2019.2932120

  • CrossRef
  • Google Scholar
  • View reference in article

• 35.

  BlankenshipKDiamantasS. Detection, tracking, and speed estimation of vehicles: a homography-based approach. IMPROVE (2022) 1:211–8. 10.5220/0011093600003209

  • CrossRef
  • Google Scholar
  • View reference in article

• 36.

  Fernández LlorcaDHernández MartínezAGarcía DazaI. Vision-based vehicle speed estimation: a survey. IET Intell Transport Syst (2021) 15:987–1005. 10.1049/itr2.12079

  • CrossRef
  • Google Scholar
  • View reference in article

• 37.

  KimHJ. Vehicle detection and speed estimation for automated traffic surveillance systems at nighttime. Tehnicki Vjesnik (2019) 26:091448. 10.17559/TV-20170827091448

  • CrossRef
  • Google Scholar
  • View reference in article

• 38.

  AshrafMHJabeenFAlghamdiHZiaMSAlmutairiM. HVD-net: a hybrid vehicle detection network for vision-based vehicle tracking and speed estimation. J King Saud Univ - Comp Inf Sci (2023) 35:101657. 10.1016/j.jksuci.2023.101657

  • CrossRef
  • Google Scholar
  • View reference in article

• 39.

  PalSKPramanikAMaitiJMitraP. Deep learning in multi-object detection and tracking: state of the art. Appl Intelligence (2021) 51:6400–29. 10.1007/s10489-021-02293-7

  • CrossRef
  • Google Scholar
  • View reference in article

• 40.

  JiaoLZhangFLiuFYangSLiLFengZet alA survey of deep learning-based object detection. IEEE Access (2019) 7:128837–68. 10.1109/ACCESS.2019.2939201

  • CrossRef
  • Google Scholar
  • View reference in article

• 41.

  KhosraviHDehkordiRAAhmadyfardA. Vehicle speed and dimensions estimation using on-road cameras by identifying popular vehicles. Scientia Iranica (2022) 29. 10.24200/sci.2020.55331.4174

  • CrossRef
  • Google Scholar
  • View reference in article

• 42.

  HuangLZheTWuJWuQPeiCChenD. Robust inter-vehicle distance estimation method based on monocular vision. IEEE Access (2019) 7:46059–70. 10.1109/ACCESS.2019.2907984

  • CrossRef
  • Google Scholar
  • View reference in article

• 43.

  JamshidnejadADe SchutterB. Estimation of the generalised average traffic speed based on microscopic measurements. Transportmetrica A: Transport Sci (2015) 11:525–46. 10.1080/23249935.2015.1026957

  • CrossRef
  • Google Scholar
  • View reference in article

• 44.

  SarkarNCBhaskarAZhengZMiskaMP. Microscopic modelling of area-based heterogeneous traffic flow: area selection and vehicle movement. Transp Res Part C Emerg Technol (2020) 111:373–96. 10.1016/j.trc.2019.12.013

  • CrossRef
  • Google Scholar
  • View reference in article

• 45.

  AppathuraiASundarasekarRRajaCAlexEJPalaganCANithyaA. An efficient optimal neural network-based moving vehicle detection in traffic video surveillance system. Circuits Syst Signal Process (2020) 39:734–56. 10.1007/s00034-019-01224-9

  • CrossRef
  • Google Scholar
  • View reference in article

• 46.

  LiuZLinYCaoYHuHWeiYZhangZet alSwin transformer: hierarchical vision transformer using shifted windows. Proc IEEE Int Conf Comp Vis (2021) 10012–22. 10.1109/ICCV48922.2021.00986

  • CrossRef
  • Google Scholar
  • View reference in article

• 47.

  ZhangQZhangMChenTSunZMaYYuB. Recent advances in convolutional neural network acceleration. Neurocomputing (2019) 323:37–51. 10.1016/j.neucom.2018.09.038

  • CrossRef
  • Google Scholar
  • View reference in article

• 48.

  BewleyAGeZOttLRamosFUpcroftB. Simple online and realtime tracking. In: Proceedings - International Conference on Image Processing, ICIP; 17-20 September 2017; Beijing, China. IEEE (2016). p. 3464–8. 10.1109/ICIP.2016.7533003

  • CrossRef
  • Google Scholar
  • View reference in article

• 49.

  WojkeNBewleyAPaulusD. Simple online and realtime tracking with a deep association metric. In: Proceedings - International Conference on Image Processing, ICIP; 17-20 September 2017; Beijing, China. IEEE (2017). p. 3645–9. 10.1109/ICIP.2017.8296962

  • CrossRef
  • Google Scholar
  • View reference in article

• 50.

  LiuXLiuWMeiTMaH. A deep learning-based approach to progressive vehicle re-identification for urban surveillance. In: Lecture notes in computer science including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics. Berlin, Germany: Spinger (2016). p. 869–84. 10.1007/978-3-319-46475-6_53

  • CrossRef
  • Google Scholar
  • View reference in article