Title Author Keyword ::: Volume ::: Vol. 18Vol. 17Vol. 16Vol. 15Vol. 14Vol. 13Vol. 12Vol. 11Vol. 10Vol. 9Vol. 8Vol. 7Vol. 6Vol. 5Vol. 4Vol. 3Vol. 2Vol. 1 ::: Issue ::: No. 4No. 3No. 2No. 1

Development of Sand Volume Estimator for Under-Struck Excavator Bucket Using Single Camera

In-Hwan Kim, Dong-Woo Lim, and Jin-Woo Jung

Department of Computer Science and Engineering, Dongguk University, Seoul, Korea
Correspondence to: Correspondence to: Jin-Woo Jung, (jwjung@dongguk.edu)
Received December 20, 2017; Revised November 21, 2018; Accepted December 15, 2018.
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
Abstract

To support the intelligence of construction environment, it is important to measure the workload of excavator in real time. But, previous studies are expensive to implement or can not processed in real time. In this paper, an image-based method to estimate the workload of the excavator bucket especially for the state of under-struck is proposed by assuming the shape of bucket and the shape of sand inside the bucket as geometric models. By analyzing the relation between single camera image and actual bucket geometry, the volume of sand which is proportional to the excavator workload is estimated. The experimental results show 93.5% accuracy even though only some part of sand region is unseen.

Keywords : Sand volume estimator, Excavator bucket modeling, Single camera
1. Introduction

It is not easy to measure in real time the amount of sand which an excavator load in working environment [1, 2]. Previous method to measure this workload in real environment was to measure the charge of weight of dump truck. In order to make this weight change, it is needed to stop the truck in a certain area. However, this method is inefficient in terms of time because it decreases the working efficiency by stopping the truck. This paper proposes a new method which can estimate automatically the volume of sand in a bucket by capturing an image from a single camera in the excavator and analyzing the image specifically focused on the under-struck state.

2. Background

Depending on the amount of sand piled up in a bucket of excavator, states could be divided like Figure 1. It is called as struck state if the volume of bucket and that of sand are equal. It is called as under-struck sate if the volume of sand is less than struck state. If the volume of sand is larger than struck state, it is called as heaped state. This paper focuses on under-struck state. The purpose of this paper is to estimate the amount of sand from the bucket image based on a single camera image processing [3, 4].

This paper assumes A1–A4 to estimate the volume of under-struck state.

1. A1: The shape of bucket is modeled as a combination of half cylinder and right triangular prism.

2. A2: The bucket diameter ‘2a’, bucket width ‘b’ and bucket teeth length ‘c’ are given by the excavator company.

3. A3: When the image is captured, the upper side of bucket is horizontal to the ground.

4. A4: In Figure 2, the pixel locations of points (P1 to P8) and the highest vertical line (U) can be calculated with image processing [5, 6].

3. Single Camera-based Sand Volume Estimation Algorithm for Under-Struck State

Under-struck state means that the highest line of sand U is below the line (Figure 3). This state could be divided into three cases in order to reduce the complexity of the formula. (Figure 2) The condition dividing the under-struck state is as follows:

1. L1: line segment $AB¯$

2. L2: parallel line segment with $AB¯$ passing through O

3. L3: Parallel line segment with $AB¯$ passing through D

4. If (U < L3) state ← Under-Struck-Case1

5. Else if (U < L2) state ← Under-Struck-Case2

6. Else if (U < L1) state ← Under-Struck-Case3

Experimental information

stateGrain size (mm)Original weight (g)Actual volume (mL)Estimated volume (mL)Estimated weight (g)Error ratio (%)
case1639.324.522.035.110.6
639.324.523.136.96.1

case26104.96059.194.69.8
6104.96065.2104.30.5

case36153.69594.1150.52.0
6153.695105.9169.410.3

### 3.1 Determination of Under-Struck State Parameters

Specific parameters are required to estimate the volume in case of Under-Struck State. There are parameters that need to be known by default such as lw, $lw′,lw⁗,lh′,lh‴$, and so on. These basic parameters can be obtained using each point (P1 ~ P8, U) through image processing.

In

$lw=(α2-α1)2+(β2-β1)2,$$lw⁗=(α8-α7)2+(β8-β7)2,$$lh⁗=(α7-α1)2+(β7-β1)2-(lw-lw⁗2)2,$$lh‴=(α5-α1)2+(β5-β1)2-(lw-lw‴2)2.$

The unit of parameters lw, $lw⁗,lh′,lh‴$ are number of pixels. In order to calculate the volume, metric unit such as mm is needed instead of number of pixels. This conversion is possible by multiplying the actual pixel size of image sensor.

### 3.2 Calculation of Angle between the Camera Centre Line $ZF¯$ and the Bucket Line $AB¯$

We can not always take the picture at the same angle. That changes the parameter values of bucket. This paper use the expression of the angle between the camera centre line and the bucket to solve this problem. This is called θC. This section will describe the process of calculating θC.

In Figure 6, lets assume that

$r2=kr1.$

By the similarity between ΔEDD′ and ΔEC′F,

$2a cos θc:r1=d:(r1+r2)=d:(1+k)r1,$$k=d2a cos θC-1.$

In

$L1=2a sin θc-kr1=2asinθc-(d2a cos θc-1) r1.$

By the similarity between ΔFZY and ΔFC′B,

$d:f=L1:l2,$$l2=fd [2a sin θc-(d2a cos θc-1) r1],$$d:f=(1+k)r1:l1=dr12a cos θc:l1,$$l1=fr12a cos θc.$

$lh‴$ is the height of bucket in

$lh‴=l1+l2=fd(2a sin θc+r1),$$r1=dlh‴f-2a sin θc,$$l1=dlh‴2a cos θc-f tan θc,$$l2=lh‴+f tan θc-dlh2a cos θc.$

l1 and l2 are decided based on the centerline of the camera image. Lets assume that the ratio of l2 and l1 is m.

$l2=ml1,$$lh‴=(1+m)l1,$$l1=lh‴1+m.$

By Eq. (15) and Eq. (19),

$lh‴=2af(m+1) sin θc(m+1)d-2a cos θc.$

By Figure 5 and

$b:d=lw:f,$$d=bflw.$

By Eq. (20) and Eq. (22),

$lh‴=2alwf(m+1) sin θc(m+1)bf-2alw cos θc,$$θc=cos-1 (4a2lw2(l‴h2+(m+1)2f2)-b2l‴h2(m+1)2f22alw(l‴h2+f2(m+1)2)×f(m+1)-l‴h2(m+1)bf2alw(l‴h2+f2(m+1)2)).$

### 3.3 Calculation of Angle θB between $AB¯$ and $BD¯$

In Figure 3, θB is fixed angle so it can be obtained by Eq. (25).

$θB=tan-1 (c2a).$

### 3.4 Calculation of Height h between the Bucket line $AB¯$ and $DH¯$

θy can be obtained by using θC and θB.

$θy=π2-θC+θB.$

In Figure 7, lets assume that

$r2′=k′r1′.$

The value k′ can be obtained by using the similarity of ΔEB′H′ and ΔEC′F.

$h cos θy:r1′=d′:(r1′+r2′)=d′:(1+k′)r1′,$$k′=d′hcosθy-1.$

By the similarity between ΔFEC′ and ΔFY Z

$d′:f=d′r1′h cos θy;l2′,$$l2′=fr1′h cos θy.$

By the similarity between ΔFHC′ and ΔFXA.

$d′:f=L1′:l1′,$$L1′=k′r1′-h sin θy=(d′h cos θy-1) r1′-h sin θy,$$l1′=fd′ [(d′h cos θy-1) r1′-h sin θy].$

In

$lh′=l2′-l1′=fd′(h sin θy+r1′),$$r1′=d′lh′f-h sin θy.$

By Eq. (34) and Eq. (36),

$l1′=d′lh′h cos θy-lh′-f tan θy.$

Let’s assume that

$l2′=m′l1′,$$lh′=(m′-1)l1′,$$l1′=lh′m′-1.$

In

$b:d′=lw⁗:f,$$d′=bflw⁗.$

By Eq. (37), Eq. (40), and Eq. (42),

$lh′=lw⁗(m′-1)fh sin θy(m′-1)bf-lw⁗m′h cos θy,$$h=lh′(m′-1)bflw⁗(m′-1)f sin θy+lh′lw⁗m′ cos θy,$$hl=h-ho,$$ho=a sin θB,$$hl=lh′(m′-1)bflw⁗(m′-1)f sin θy+lh′lw⁗m′ cos θy-a sin θB.$

### 3.5 Estimation of Under-Struck State Volume

3.5.1 Estimation of sand volume for Under-struck-case 1 condition

The volume can be estimated by sector OXY and ΔOY X. Here, V (x) means volume made by expanding the area x along bucket width b.

$V=V(sector OXY)-V(ΔOYX),$$θz=cos-1 (hla).$

By Eq. (49)

$sector OXY=a2 cos-1 (hla),$$XY¯=2a2-hl2,$$ΔOYX=hla2-hl2,$$V=a2b cos-1 (hla)-bhla2-hl2.$
3.5.2 Estimation of sand volume for under-struck-case 2 condition

In under-struck-case 2, the formula for calculating the volume can be obtained by subtracting sector BOY and ΔYOC from sector BDO and adding ΔXDC. See

$V=V(semicircle)-V(sector BOY)-V(ΔYOC)+V(ΔXDC).$

First, to obtain sector BOY, we need to find u1 and u2.

$u1=θB,$$u2=π2-cos-1 (hla),$$sector BOY=a22 (θB+π2-cos-1 (hla)).$

To find the area of ΔXDC, you need to find the base length and height. In

$2acos θB:ho+a sin θB=v2:a sin θB-hl,$$2a:ho+a sin θB=v3:a sin θB-hl.$

By Eq. (46),

$v2=a sin θB-hlsin θB cos θB,$$v3=a-hlsin θB,$$ΔXDC=12v3v22-v32=12tan θB (a sin θB-hlsin θB)2.$

In

$v4=a-v3.$

In

$ΔYOC=12hl (a2-hl2+hlcos θBsin θB),$

The total volume can be obtained by summing each areas.

$V=πa2b2-a2b2 (θB+π2-cos-1 (hla))-bhl2 (a2-hl2+hlcos θBsin θB)+b2tan θB (a sin θB-hlsin θB)2.$
3.5.3 Estimation of sand volume for under-struck-case 3 condition

The equation for obtaining the volume in under-struck-case 3 can be obtained by adding ΔY′YO and ΔXY′D in the semicircle-subtracted sector BY O. See

$V=V(semicircle)-V(sector BYO)+V(ΔY′YO)+V(ΔXY′D).$

In

$2a:2a sin θB=2a-v5:2a sin θB-h,$$2acos θB:2a sin θB=v6:2a sin θB-h,$$v5=hsin θB,$$v6=2a sin θB-hsin θB cos θb,$$ΔXY′D=12tan θB (2a sin θB-hsin θB)2.$

In

$v8=(a-v5)2-(a sin θB-h)2,$$v7=a2-(a sin θB-h)2-v8,$$v7+v8=a2-(a sin θB-h)2.$

In

$θB-u3=cos-1 (a2-(a sin θB-h)2a),$$u3=θB-cos-1 (a2-(a sin θB-h)2a),$$sector BYO=a22 (θB-cos-1 (a2-(a sin θB-h)2a)).$

In

$ΔY′YO=12(a sin θB-h)(a2-(a sin θB-h)2-a sin θB-htan θB),$$V=πa2b2+b2tan θB (2asinθB-hsin θB)2-a2b2 (θB-cos-1 (a2-(a sin θB-h)2a))+b2(a sin θB-h)(a2-(a sin θB-h)2-a sin θB-htan θB).$
4. Experimental Results

Several experiments were performed to confirm the accuracy of the algorithm presented in this paper. The volume expression contains a parameter called focal length. This focal length parameter may or may not appear in the H/W specification of commercial camera. Here, we assumed that the focal length is not given and try to find by experiments using pre-known lw. The estimated focal length was 19.0011 mm as Table 1 when we use SPC-B900W camera. The size of image sensor in the camera was 21.12 mm × 11.88 mm.

The experiment to find the focal length was done by taking a picture of an object with pre-known size at a pre-known distance. Then the focal length is found through the number of image pixels and pre-known size and distance of the object.

The experiment to find the density of a ball was done by finding the volume and weight of a fixed number of 6 mm balls. The density is determined by dividing the weight by the measured volume.

The sand used in the experiment is a type of ball with 6mm diameter. The method to estimate the volume is as follows: The original weight is obtained through an electronic scale. Actual Volume is the volume measured when pouring water into the beaker and then pouring balls there. Estimated volume is obtained through the proposed algorithm. Estimated weight is obtained by multiplying the estimated volume by the average 1.6g/ml density obtained in

The experimental results are shown in Table 3. The error rate of case 1 was 10.6% and 6.1%, the error rate of case 2 was 9.8% and 0.5%, and the error rate of case 3 was 2.0% and 10.3%. Most of errors may be from the uneven surface of the sand region made by the fixed size of ball.

5. Conclusions

In this paper, we proposed a novel method to estimate the amount of sands in the excavator bucket based on a single camera by using the image processing technique and mathematical modeling of bucket. For each of three under-struck states, a closed form of mathematical solution to estimate the sand volume of excavator bucket was implemented. The experimental results show that the error rate is within 10.6% and the minimum error rate is 0.5% in a case of under-struck-case 2.

Acknowledgments

This research was partially supported by the MIST (Ministry of Science and ICT), Korea, under the national program for excellence in SW supervised by the IITP (Institute for Information & Communications Technology Promotion) (2016-0-00017) and partially supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (2015R1D1A1A09061368) and supported by the KIAT (Korea Institute for Advancement of Technology) grant funded by the Korea Government (MOTIE : Ministry of Trade Industry and Energy) (No. N0001884, HRD program for Embedded Software R&D).

Conflict of Interest

Figures
Fig. 1.

Three representative states of sands in the bucket. (a) under-struck state, (b) struck state, and (c) heaped state.

Fig. 2.

Bucket image.

Fig. 3.

Bucket modeling.

Fig. 4.

Three cases for under-struck state.

Fig. 5.

Default parameters for under-struck state.

Fig. 6.

Camera geometry for calculating θC.

Fig. 7.

Camera geometry for calculating h.

Fig. 8.

Under-struck-case1.

Fig. 9.

Under-struck-case2.

Fig. 10.

ΔXDC.

Fig. 11.

ΔYOC.

Fig. 12.

Under-struck-case3.

Fig. 13.

ΔXY′D.

Fig. 14.

Sector BYO-1.

Fig. 15.

Sector BYO-2.

Fig. 16.

ΔY′YO.

Fig. 17.

Under-struck-case1 image.

Fig. 18.

Under-struck-case2 image.

Fig. 19.

Under-struck-case3 image.

TABLES

### Algorithm 3.1

Volume_Estimator

 INPUT P1, …, P6 : Six trapezoidal points of the bucket U : The uppermost edge point of the sand region in photographed image V1 : Center point of $P1P2¯$ V2 : Center point of $P3P6¯$ V3 : Center point of $P4P5¯$ C : Intersection point of line segment $V1V3¯$ and the center line of the image D1 : Intersection of $P1P3¯$ and $P2P6¯$ D2 : Intersection of $P1P4¯$ and $P2P5¯$ OUTPUT V : The volume of sand accumulated in the bucket FUNCTION Volume_Estimator(P1, …, P6, U, V1, V2, V3, C, D1, D2) 1 { 2 $m←V1C¯V3C¯,m′←V1C¯UC¯,lw←P1P2¯,lh′←V1U¯,lh″←V1V3¯,lw⁗←P7P5¯$ 3 $θC←cos-1 (f(m+1)4a2lw2{lh2+(m+1)2f2}-b2lh2(m+1)2f2-lh2(m+1)bf2alw(lh2+f2(m+1)2))$ 4 $h←lh′(m′-1)bflw⁗(m′-1)f sin θy+lh′lw⁗m′ cos θy$ 5 $h←lh′(m′-1)bflw⁗(m′-1)f sin θy+lh′lw⁗m′ cos θy$ 6 IF(h ≥ h0 + a sin θB) THEN 7 state ← Under-Struck-Case1 8 ELSE IF (h0 + a sin θB > h ≥ h0) THEN 9 stae ← Under-Struck-Case2 10 ELSE IF (h < h0) THEN // h0 = a sin θB 11 state ← Under-Struck-Case3 12 END IF 13 IF (state=Under-Struck-Case1) THEN 14 $V←a2b cos-1 (hla)-bhla2-hl2$ 15 ELSE IF (state=Under-Struck-Case2) THEN 16 $V←πa2b2-a2b2 (θB+π2-cos-1 (hla))-bhl2 (a2-hl2+hlcos θBsin θB)+b2tan θB (a sin θB-hlsin θB)2$ 17 ELSE IF (state=Under-Struck-Case3) THEN 18 $V←πa2b2-b2tan θB (2a sin θB-hsin θB)2-a2b2 (θB-cos-1 (a2-(a sin θB-h)2a))+b2(a sin θB-h)(a2-(a sin θB-h)2-a sin θB-htan θB)$ 19 END IF 20 RETURN V 21 }

### Table 1

Focal length experiment

Length between object and camera lens (mm)Object length (mm)Image pixel length (pixel)Focal length (mm)
45011344119.3181
40011349919.4301
35011355919.0456
30011365219.0407
25011378319.0553
20011396318.7487
150113125818.3690
Average focal length19.0011

### Table 2

Density experiment of object

Number of ballsVolume (mL)Weight (g)Density (g/mL)
101.252.01.6
202.54.01.6
303.756.01.6
4058.01.6
Average density1.6

### Table 3

Experimental information

stateGrain size (mm)Original weight (g)Actual volume (mL)Estimated volume (mL)Estimated weight (g)Error ratio (%)
case1639.324.522.035.110.6
639.324.523.136.96.1

case26104.96059.194.69.8
6104.96065.2104.30.5

case36153.69594.1150.52.0
6153.695105.9169.410.3

References
1. Ahn, SH, Kim, SK, and Lee, KH (2016). Development of a fleet management system for cooperation among construction equipment. Journal of the Korea Society of Civil Engineers. 36, 573-586. https://doi.org/10.12652/Ksce.2016.36.3.0573
2. David, F, Petr, P, Miroslav, M, and Milan, M 2016. Scanning of trucks to produce 3D models for analysis of timber loads., Proceedings of 17th International Carpathian Control Conference (ICCC), Tatranska Lomnica, Slovakia, Array, pp.194-199. https://doi.org/10.1109/CarpathianCC.2016.7501092
3. Vrublova, D, Kapica, R, and Jurman, J (2012). Methodology devising for bucket-wheel excavator surveying by laser scanning method to determine its main geometrical parameters. Geodesy and Cartography. 38, 157-164. https://doi.org/10.3846/20296991.2012.757438
4. Won, JU, Chung, YS, Kim, WS, You, KH, Lee, YJ, and Park, KH (2002). A single camera based method for cubing rectangular parallelepiped objects. Journal of KIISE: Computing Practices and Letters. 8, 562-573.
5. Baek, YH, and Moon, UR (2006). Color edge detection using variable template operator. International Journal of Fuzzy Logic and Intelligent System. 6, 116-120. https://doi.org/10.5391/IJFIS.2006.6.2.116
6. Xiong, Xing, and Choi, Byung-Jae (2013). Comparative analysis of detecting algorithms for corner and blob features in image processing. International Journal of Fuzzy Logic and Intelligent System. 13, 284-290.
Biographies

In-Hwan Kim has been under M.S. candidate course at Dongguk university, Korea, since 2016. His current research interests include robotics and intelligent human-robot interaction.

E-mail : dlsghks199@naver.com

Dong-Woo Lim has been under M.S. candidate course at Dongguk university, Korea, since 2018. His current research interests include intelligent human-robot interaction and image processing.

E-mail : aehddn@gmail.com

Jin-Woo Jung received the B.S. and M.S. degrees in electrical engineering from Korea Advanced Institute of Science and Technology (KAIST), Korea, in 1997 and 1999, respectively and received the Ph.D. degree in electrical engineering and computer science from KAIST, Korea in 2004. Since 2006, he has been with the Department of Computer Science and Engineering at Dongguk University, Korea, where he is currently a Professor. During 2001–2002, he worked as visiting researcher at the Department of Mechano-Informatics, University of Tokyo, Japan. During 2004–2006, he worked as researcher in Human-friendly Welfare Robot System Research Center at KAIST, Korea. During 2014, he worked as visiting scholar at the Department of Computer and Information Technology, Purdue University, USA. His current research interests include human behavior recognition, multiple robot cooperation and intelligent human-robot interaction.

E-mail : jwjung@dongguk.edu

December 2018, 18 (4)