Multiple object detection and tracking are important for security and surveillance [1]. Recently, there is increased usage of unmanned aerial vehicles (UAVs) or drones for aerial surveillance due to the efficiency of capturing remote scenes [2]. The image resolution for objects located far from the camera is low. Further, blurring and noise can degrade image quality. Various studies have been conducted on visual detection and tracking of moving objects. Methods based on background subtraction and frame difference were studied in [3–8]. Gaussian mixture modeling (GMM) was used to analyze background and target regions in [3–5]. Scale invariant feature transform (SIFT) was adopted to extract foreground objects from the background scene in [6, 7]. In [8], background was subtracted under Gaussian mixture assumption followed by morphological filters. In [9], hypothesis fitting was adopted for vehicle detection. Long-range moving objects were detected by background subtraction in [10].

In addition to the above-mentioned research, many studies have been conducted on multiple-target tracking [11–14]. High maneuvering, closely located multiple targets, heavy clutter (false alarm), and low detection probability are often considered difficulties to overcome [11]. Kalman filtering provides a solution for estimating the state of the target in real time. It is known to be optimal under independent Gaussian noise assumption [12, 13]. When multiple measurements are detected in a frame, data association is required to assign the measurements to the established tracks [14].

Vison-based target tracking with UAVs has been researched in [15–22]. Correlation filtering-based tracking was studied in [15]. Closely located-object tracking was performed with feature matching and multi-inertial sensing data in [16]. Pedestrians were tracked by template-matching in [17, 18]. Small animals were tracked with a freely moving camera in [19]. A moving ground target was tracked in dense obstacles areas with UAVs in [20]. Multiple moving targets are tracked by Kalman filtering and IMM filtering in [21] and [22], respectively. In [23], position and velocity were estimated with multiple sensor fusion.

In this paper, we address the detection and tracking of multiple moving vehicles obtained by a UAV, which captures videos above the road from a long distance. To achieve this goal, we propose to integrate two systems: one is detection based on image processing, which manipulates pixels in the stationary scene, and the other is tracking based on Kalman filtering, which estimates dynamic states (position and velocity) of targets in the time domain. Object detection is performed through frame differencing and thresholding, morphological filtering, and removing false alarms based on the minimum size of vehicles. Two frames separated by a constant interval are subtracted and turned into a binary image by thresholding, which can represent the difference between the current and the past frames. Two morphological operations, erosion and dilation, are applied to the binary image to extract regions of interest (ROIs), which are the disjoint alternative areas. The ROIs are the candidate areas of moving vehicles. False ROIs are removed by the minimum ROI size, which is based on the true size of moving vehicles. The centroids of ROIs are the results of detection. The centroids represent the measured positions of targets, which will be processed in the next tracking stage. Tracking is performed by Kalman filtering to estimate the state of the target. A nearly constant velocity (NCV) model is assumed for the dynamic state of the target [11]. A position gating process excludes the measurement outside the validation region. The nearest measurement-to-track association scheme assigns one measurement to one track [21]. The tracks are initialized by a two-point differencing initialization method with a maximum speed gating [11]. In the experiments, three moving cars are captured at a long distance by a UAV. The camera built in a UAV points directly downward to the road. Root-mean-squared errors (RMSEs) are obtained for position and velocity in x and y directions. Experimental results show that the location and the speed are well detected and tracked with good accuracy by the proposed method.

The remainder of the paper is organized as follows. Object detection is discussed in Section 2. Multiple target tracking is presented in Section 3. Section 4 demonstrates experimental results. The conclusion follows in Section 5.

A block diagram of object detection is described in Figure 1. First, frame difference is obtained between the current and the past frames. A thresholding step follows to generate a binary image as:

where I(m, n; k) and I(m, n; k−k_d) are the kth frame and the (k−k_d)th frame, respectively, k_d is a constant interval for frame differencing, θ_T is a thresholding value, andM and N are pixel sizes in x and y directions, respectively. Two morphological operations, erosion and dilation, are applied sequentially to the binary image to segment moving vehicles. Erosion and dilation filtering, respectively, are defined as [24]:

where E_l = {e + l|e ∈ E} and D_l = {d + l|d ∈ D}; E and D are the structuring elements for erosion and dilation, respectively. It is noted that l is an integer value less than the image size. Morphological filters extract ROIs, which are the disjoint alternative areas. At last, false ROIs are removed by the minimum ROI size as:

where O_i is the ith ROI and θ_s is the minimum size of ROI. The center of each ROI is considered as a measurement for target tracking in the next section.

3.1 System Modeling

The dynamic state of the target is modeled as a NCV model; the target’s maneuvering is modeled by the uncertainty of the process noise, which is assumed to follow the Gaussian distribution. The following is the discrete state equation of a target:

where x_t(k) = [x_t(k) v_tx(k) y_t(k) v_ty(k)]^T is the state vector at frame k, x_t(k) and y_t(k) are the positions in x and y directions, respectively, v_tx(k) and v_ty(k) are the velocities in x and y directions, respectively, T denotes matrix transposition, Δ is the sampling time, and v(k) is a process noise vector, which is Gaussian white noise with the covariance matrix Q=diag([σx2 σy2]), and N_T is the number of targets. A measurement vector is composed of two components in x and y directions. The following is the measurement equation:

where w(k) is a measurement noise vector, which is Gaussian white noise with the covariance matrix R=diag([rx2 ry2]).

3.2 Two-Point Differencing Initialization with Velocity Gating

The initial state of each target is calculated by the two-point differencing method with a maximum speed gating [14]. The initial state and covariance matrix of target t become:

The state is confirmed as an initial state of a track if the following speed gating is satisfied:

3.3 Prediction and Filter Gain

The state and covariance predictions are iteratively computed with the initial values as:

where x̂_t(k|k−1) and P_t(k|k−1), respectively, are the state and the covariance prediction for target t and frame k. The residual covariance S_t(k) and the filter gain W_t(k), respectively, are obtained as:

3.4 Data Association

Data association is the process of assigning multiple measurements to established tracks. Measurement gating is a preprocess that reduces the number of candidate measurements for data association. The measurement gating is performed by chi-square hypothesis testing with the assumption of the Gaussian measurement residuals [11]. Measurements become valid for target t at frame k if they exist in the validation region as

where z_m(k) is the mth measurement vector at frame k, γ is the gating size, and m_t(k) is the number of candidate measurements for target t at frame k. The nearest neighbor rule associates a measurement with track t as:

3.5 State Estimate and Covariance Update

The state estimate and the covariance matrix of targets are updated as follows:

If m_t(k) is equal to zero, i.e., no measurement is associated with target t at frame k, they merely become the predictions of the state and the covariance as:

3.6 Performance Evaluation

Several metrics are used for performance evaluation, position error, velocity error, and RMSE of position and velocity. The position error of target t at frame k is obtained as:

where xttrue(k) and yttrue(k) are the ground truth of position of target t at frame k, in x and y directions, respectively. The ground truth of positions of targets is obtained manually at each scene. The RMSE of position is obtained as:

where K_t(s) and K_t(f) are the first and last frame where target t is estimated. The velocity error of target t at frame k is obtained in both x and y directions as:

where the ground truth of the velocity is approximated as:

The RMSE of velocity is obtained as:

4.1 Scenario Description

A drone (Phantom 4 Advanced) captures three moving vehicles at 30 frames per second. A total of 372 frames are captured for more than 12 seconds. The size of one frame is 4096×2160 pixels; the frame is reduced to 20% size, 820×432 pixels, for efficient image processing. The drone hovers (stays still in the air) while capturing video sequences at a height of 92 m above the road. The camera points directly downwards while capturing video sequences. One pixel corresponds to 0.1344 m. There are three cars moving, with different initialization positions and maneuverings. The first car (Car 1) is located at the center of the scene from the first frame and moves to the right until it disappear after the 155th frame. The second car (Car 2) moves to the left from the first frame to the 289th frame. The third car (Car 3) moves in the same direction as the second car from the 74th to the 372nd frame. Figure 2(a) shows Cars 1 and 2 at the first frame, Figure 2(b) shows Cars 1–3 at the 150th frame, and Figure 2(c) shows Car 3 at the 300th frame. In the figures, the red circles indicate the moving vehicles.

4.2 Detection of Moving Objects

A total of 368 frames from the 5th frame to the 372nd frame is processed to detect moving objects, since k_d in Eq. (1) is set at 4, θ_T in Eq. (1) is set at 30, and E and D in Eqs. (2) and (3) are set at [1]_9×9 and [1]_15×15, respectively. The parameters for thresholding and morphological operations are determined heuristically when better results are produced; θs in Eq. (4) is set at 90 considering the true size of moving vehicles. The first row in Figure 3 shows the detection process of Figure 2(a). The second and third rows are of Figure 2(b) and (c), respectively. Figure 3(a) and (b), respectively, are the frame differencing and thresholding to generate a binary image. Figure 3(c) shows the morphological filtering (erosion and dilation) result; the centroids of the segmented areas are marked with an asterisk (*). The ROI window is a rectangle that includes disjoint detected areas. Figure 3(d) shows the segmented areas by the ROI windows after removing false ROIs according to Eq. (4).

Figure 4 shows all centroids for 368 frames. Table 1 shows the detection rate of three cars. Average detection rate is around 96%. The low detection rate of Car 3 is mainly due to the tree occlusion and the same background color (white sidewalk) as the target. A total of 10 false alarms are detected, and two of them are caused by the split measurements from one car.

4.3 Multiple Target Tracking

The centroids in Figure 4 become input measurements for tracking. The sampling time Δ in Eq. (6) is 0.033 s since the frame rate is 30 fps. The standard deviation of process and the measurement noise are set at σ_x = σ_y = 30 m/s² and r_x = r_y = 1.5 m, respectively; γ for the gating test is set at 4; and σ_x and σ_y are conventionally set between 0.5a_max and a_max where a_max is the maximum acceleration of the target [14]; and r_x and r_y are set at half the length of the vehicle. The validation region in Eq. (14) is chi-square distributed with the degree of freedom is equal to the dimension of measurement vectors, thus the probability mass corresponds to 86.5% when γ is set at 4. The maximum speed for speed gating of two-point differencing initialization is set at 30 m/s. The initial-state vectors of Cars 1 and 2 are initialized by the measured position in the 5th and 6th frames; the initial state of Car 3 is initialized by the 74th and 75th frames. Cars 1, 2, and 3 are estimated until 155th, 289th, and 372nd frame for 150, 284, and 298 frames, respectively, including initialization. Figure 5(a)–(c) are the results of tracking Cars 1–3, respectively. No false tracks are established, and no track termination scheme is applied.

Figure 6(a) shows the ground truth of the position of Car 1 from the first to the 155th frame. Figures 6(b) and c are the position vectors of Cars 2 and 3, respectively, from the first frame to the 289th frame, and from the 74th frame to the 372nd frame. Figure 7(a)–(c) shows the approximated ground truth of the velocity in x and y directions of Car 1, 2, and 3, obtained by Eq. (24); δis set at 26 when the least average velocity RMSE is produced. Figures 8 and 9 show the position in Eq. (21) and velocity errors in Eq. (23), respectively. Large position errors in x direction are mainly caused by measurement errors. High speed generates greater measurement errors. Therefore, measurement errors during detection are bigger in the direction of the car’s movement. Velocity errors are generated during tracking. It is noted that the velocity error differences in x and y directions are small compared with those of position errors.

The sampling time is set differently in the following experiments. Figure 10 shows the tracking result with Δ = 0.1 s, that is, measurements are acquired every third frame from the original data. Cars 1 and 2 are initialized by the 5th and 8th frames and Car 3 is initialized by the 74th and 77th frames. Cars 1, 2, and 3 are estimated until 155th, 287th, and 371th frame, respectively, thus Cars 1, 2, and 3 are estimated for 50, 94, and 99 frames, respectively, including initialization. Figure 11 is the tracking results with every sixth frame (Δ = 0.2 s) updated. Cars 1 and 2 are initialized by the 5th and 11th frames and Car 3 is initialized by the 77th and 83rd frames. Cars 1, 2, and 3 are estimated for 25, 47, and 49 frames, respectively, until 155th, 298th, and 371st frame. The relative computational times are 1, 0.51, and 0.34 for Δ = 0.033, Δ = 0.1, and Δ = 0.2 s, respectively.

Table 2 shows the RMSE of position in Eq. (22). The large position RMSEs of Cars 2 and 3 are due to the high maneuvering of the circular motion. The average position RMSE is bigger over many sampling times, as shown in Table 2. Finer sampling steps can reduce position errors in Car 1 and 2, costing more computational resources. Table 3 shows the RMSE of velocity in Eq. (25). The velocity RMSE is also bigger for Cars 2 and 3 than Car 1. It is shown that the average velocity RMSE decreases as the sampling time increases.

In this paper, multiple moving objects are captured over long distances by a UAV. The locations of objects are detected using a background scene. Three targets are tracked with Kalman filtering. Experimental results show that the proposed algorithm detects and tracks moving objects with good accuracy. Position and velocity errors are obtained for different maneuvering targets and sampling times. It is shown that the proposed system can adjust the sampling time to build a more efficient system in computational resources. This property is favorable for drone applications because the computational resources in a drone are very limited. This study also proposes new applications for drones, such as traffic control and smart car as well as security and defense. Tracking a large number of targets with high maneuvering remains for future study.

This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (Grant Number: 2017R1D1 A3B03031668).

Table 1. Detection rates.

Table 2. Position RMSE.

Table 3. Velocity RMSE.