In response to rapid market growth, various robots have recently been developed, and are being used as service and industrial robots. Among these robots, humanoids have a human-like appearance and structure and can be utilized for the same jobs as humans, such as lunar exploration robots and guide assistants that can interact with humans. Although these robots resemble humans, they have varying degrees of freedom, depending on the type of motor and joint. Depending on the manufacturer, certain robotic systems exhibit limited flexibility relative to humans owing to cost-saving measures, such as reductions in the number of motors used for efficient motion. The NAO robot is a specific example of this phenomenon. Unlike human arms, which utilize six degrees of freedom to articulate motion, the NAO robot is equipped with only four joints per arm, which limits its ability to replicate the full range of human movements. In situations where robots are intended to perform tasks by emulating human behavior in real time, mapping robot-to-human actions has become a critical concern. However, challenges arise while attempting to reconcile the degrees of freedom inherent in human and robotic systems. Specifically, the mapping process may result in a mismatch between the freedoms afforded to human and robotic agents. Consequently, the utilization of such mapped information may lead to unnatural or erroneous robotic behaviors. To overcome the problem of mismatch between human–robot degrees of freedom, some studies have been conducted, such as inverse kinematics [1–3], human–robot 3D modeling mapping [4], co-ordinate system transformation [5], and robotic joint prediction methods [6, 7]. However, matching the degrees of freedom between humans and robots using inverse kinematics requires precise Denavit–Hartenberg (DH) parameters that incorporate accurate structural information. Measuring the components of DH parameters, such as arm length, end-effector position, and angle, requires manual work, and any inaccuracies or incorrect application of these parameters can result in a large error or omission of information when converting from a human arm with six degrees of freedom to an NAO robot arm with four degrees of freedom through Jacobian and rotation matrix methods. To address this issue, the inverse kinematic method needs to be modified to ensure the accuracy of the mapping process. The human–robot 3D model mapping approach in [4] requires the robot to perform additional computations, such as the zero-movement point (ZMP), and because mapping is performed for fixed motion, continuous motion over time is restricted. Studies addressing the degrees-of-freedom problem by transforming human motion data into a coordinate system [5, 6] exhibit limitations in scale owing to platform dependency, and can only be applied to a robot platform with a converted coordinate system. Finally, neural-network-based motion imitation studies [7] require motion capture equipment and additional input components, such as bend and twist angles, in addition to joint angles, which increases the computation multifold depending on the number of imitation joints because each joint requires one neural network. To overcome these problems, this paper proposes a technique for a robot to accurately predict human motion over time while minimizing the degrees-of-freedom restrictions by imitating human motion using a single artificial neural network (ANN). The proposed method aims to enhance the efficiency of robot control processors by reducing kinematic calculations while simultaneously ensuring that no information is omitted. Using flexible learning through ANNs, robots can effectively predict and imitate human movements in various scenarios and movement types.

The remainder of this paper is organized as follows: Section 2 examines the previous studies related to the proposed technology, and Section 3 describes the proposed technical method. The experimental results for the proposed technology are described in Section 4, and the scope for future studies is discussed in Section 5.

Humanoids have human-like structures and can perform the same tasks as humans. Human motion imitation and accurate motion control have been extensively studied to make the work performance of humanoids more natural. For the precise motion control of human-motion-imitating robots, the problem of mismatch between human–robot degrees of freedom must be solved. This is because, even if they are designed similar to humans, robot joints have variable degrees of freedom based on the configuration of the motor and the direction of rotation of the joints. In circumstances in which the degrees of freedom of the human and robot differ, motion information in a specific direction is lost, resulting in different motions. To solve these problems, a technology that modifies human motions to fit robots is required. Accordingly, various studies have been conducted, including inverse kinematics [1–3], human–robot mapping [4], coordinate system transformation [5, 6], and robot joint prediction using neural networks [7].

Classically, this problem has been solved using inverse kinematics and various studies on human–robot motion, including geometric inverse kinematics [1], adaptive neuro-fuzzy system methods [2], and roll-pitch-yaw (RPY) conversion methods [3]. Shahverdi and Masouleh [1] studied a robot that imitates human motion and used DH parameters and inverse kinematics to convert Kinect [8] human motion data into robot motion in a robot operating system [9] framework. Mukherjee et al. [2] used the inverse kinematics approach of adaptive neuro-fuzzy inference systems as a human upper-body motion imitation technique to compute robot motion to fit the degrees of freedom of humanoids. Koenemann and Bennewitz [3] used an Xsens MVN motion capture system with individually mounted inertial sensors, mapped human motion to the robot, and applied the end-effector position of the hand and foot to the inverse kinematics to verify the imitation of human motion. This inverse-kinematics-based method has the disadvantage of requiring a precise understanding of structural factors, such as the robot arm length, in addition to the joint angle of the robot, as well as sophisticated calculations directly using DH parameters, rotation matrices, and Jacobian matrices.

A previous study used 3D model mapping, a technique for human–robot joint mapping, to overcome the problem of mismatch between human and robot degrees of freedom [4]. Wang et al. [4] used Kinect human motion data to generate a 3D human-shaped HUMROB model for robot motion imitation, followed by the Gaussian mixture model and expectation-maximization methods to compute robot motion by mapping the HUMROB model to NAO robots. However, for robots, additional work, such as ZMP, is required for optimization, and imitation is performed only for a fixed posture, limiting continuous operation.

The coordinate system transformation of the human–robot degrees of freedom addresses this issue. Filiatrault and Cretu [5] calculated robotic motion by transforming the coordinate system of human motion data into a robotic spatial coordinate system using the RPY angle transformation in real time. Yavsan and Ucar [6] obtained the robot arm joint angle using a transformation algorithm to match the Kinect human upper-body motion data to the robot coordinate system and used an extreme learning machine approach to categorize and imitate six humanarm motions. However, the transformation algorithm can only be applied to a converted robot platform, thereby limiting its scalability owing to platform dependency.

A previous study used neural networks, for example, the multilayer feed-forward (MLFF) neural network, to predict robot motion using human motion data in human–robot motion imitation [7]. Aleotti et al. [7] applied the MLFF neural network to imitate human arm motion using six degrees-of-freedom robotic arms, and the ShapeTape motion capture system [8] captured human arm motion data. The neural network method is imitated using the output values of similar motion by inputting human motion into neural networks as data. Multilayer neural networks use a neural network for each joint to predict robot joint angles through multiple neural networks. However, the bend and twist angles are used, in addition to the joint angles, as inputs through the equipment, and several neural networks are computed independently, which makes them time-consuming and less efficient. Balmik et al. [10] adopted a 1D convolutional neural network model to calculate the center of mass after converting human joint angles into robot joint angles. The conversion process was based on the method described by Zhang et al. [11]. Specifically, the study [11] resolved the issue of mismatched degrees of freedom between humans and robots by applying a Savitzky–Golay filter during the joint mapping process. However, the conversion still requires manual calculations for each joint, and the method may become inflexible when the platform of the robot is altered.

This paper proposes a method for developing a motion imitation robot based on ANNs with minimal degrees-of-freedom restrictions, which addresses the issue of degrees-of-freedom inconsistency by utilizing only joint angles rather than the input variables required for inverse kinematics, such as joint angles, lengths, and location information. This enables more efficient processing without compromising accuracy. In addition, the developed motion imitation robot can be adapted to various robot platforms, overcoming the limitations associated with manual processing or using neural networks for individual angle data.

In this section, we describe the proposed motion imitation robot, which is illustrated schematically in Figure 1. The process begins with the input of human joint data, as shown in Figure 1, which serves as the foundational input for the ANN. This network was designed to analyze human joint data and predict the corresponding robot joint data, effectively allowing the robot to mimic human actions. The predicted joint data are then transmitted to the robot interface, which executes the imitated motions.

Figure 1 illustrates the architecture of the ANN used in this study. The network consists of several fully connected layers. The input layer receives human joint data, and consists of N units tailored to capture the nuances of human motion. This is followed by two intermediary layers: the first contains six units, and the second comprises 12 units, enhancing the capability of the network to process complex patterns. Both layers employ the rectified linear unit activation function to introduce nonlinearity into the learning process, aiding in the effective modeling of joint behaviors. The architecture culminates in an output layer with M units, each corresponding to a specific robot joint, thereby producing the final output data that direct the movements of the robot.

The Euler angle vector [α, β, γ] represents the joint angle of the human motion data x, which is the input data. α represents the rotation of the x-axis, β represents the rotation of the y-axis, and γ represents the rotation of the z-axis. The right human shoulder angle (h1-right) is expressed as [α_(h1−right), β_(h1−right), γ_(h1−right)], the right human elbow angle (h2-right) is expressed as [α_(h2−right), β_(h2−right), γ_(h2−right)], the left human shoulder angle (h1-left) is expressed as [α_(h1−left), β_(h1−left), γ_(h1−left)], and the left human elbow angle (h2-left) is expressed as [α_(h2−left), β_(h2−left), γ_(h2−left)]. The right robot shoulder angle (r1-right) of the robot angle vector, the output of the ANN, is [α_(r1−right), β_(r1−right)], the right robot elbow angle (r2-right) is [α_(r2−right), γ_(r2−right)], the left robot shoulder angle (r1-left) is [α_(r1−left), β_(r1−left)], and the left robot elbow angle (r2-left) is [α_(r2−left), γ_(r2−left)].

The proposed framework uses six human joint angles for the right arm and 12 for the upper body as input values for each behavior in one frame. The input data were fed into the proposed ANN shown in Figure 1. This ANN calculated and predicted four robot right-arm joint angles and eight robot upper-body joint angles. The weights for each node were updated through the chain rule of backpropagation using a learning rate of 1 × 10⁻⁶, the stochastic gradient descent optimizer, and the loss function of the mean absolute error. The learning process was repeated 10,000 times to optimize the accuracy of the model. During training, the loss value was recorded as 5.3942 at the 5,000 epoch, 1.2032 at the 10,000 epoch, and 1.1422 at the 15,000 epoch in Figure 2. Convergence was observed at approximately 10,000 epochs, and training was stopped after 10,000 epochs to prevent overfitting.

In the proposed technique, motion imitation was primarily aimed at imitating the human right arm and upper body. In Figure 1, for the imitation of right-arm motion, the six-dimensional data [α_(h1−right), α_(h2−right), β_(h1−right), β_(h2−right), γ_(h1−right), γ_(h2−right)] were input by combining the right human shoulder and arm angle data as an input to the ANN, and the four-dimensional Euler angle [α_(r2−right), α_(r1−right), β_(r1−right), γ_(r2−right)] for the shoulder and elbow joints of the right arm of the robot was predicted.

In Figure 1, for human upper-body motion, the 12-dimensional data [α_(h1−right), α_(h2−right), β_(h1−right), β_(h2−right), γ_(h1−right), γ_(h2−right), α_(h1−left), α_(h2−left), β_(h1−left), β_(h2−left), γ_(h1−left), γ_(h2−left)] were input by combining both shoulder and elbow data, and the eight-dimensional Euler angle [α_(r2−right), α_(r1−right), β_(r1−right), γ_(r2−right), α_(r2−left), α_(r1−left), β_(r1−left), γ_(r2−left)] was predicted for the robot upper body, that is, both sides of the shoulder and elbow of the robot.

Through a series of processes, the robot transformed human motions into motions suitable for itself; thus, a human-motion-imitating robot system was implemented.

This section describes the experimental results of the technical verification of the motion imitation robot based on the proposed ANN, which minimizes the degrees-of-freedom restrictions. The experimental setup consisted of a SoftBank Nao robot [13] that was evaluated in the virtual environment of CoppeliaSim [14]. The training phase was conducted on hardware that consisted of an Intel Core i7-8750H CPU @ 2.20 GHz and a NVIDIA GeForce GTX 1050 Ti, whereas the software environment comprised Python 3 and PyTorch 1.10.0.

First, as shown in Figure 3, the datasets collected for human motion imitation consisted of human motion data and robot motion answer data. The left side of the human motion data in Figure 3 is the actual human movement data sourced from the Carnegie Mellon University (CMU) Motion Capture (Mocap) data [12], which is a free public database. The CMU Mocap dataset is an open dataset curated by the university. The dataset was captured using the Vicon motion capture system, which consists of 12 MX-40 cameras that track the entire human body via 40–60 markers. This collection encompasses a diverse range of active human behaviors, including walking, running, and dancing, and has emerged as one of the most extensively employed datasets for research purposes. In this study, the joint angles of a robot were forecasted by utilizing the joint data of human behavior from the CMU dataset as input without any preliminary processing procedures. The two data points on the right side of Figure 3 represent the robot motion data. The answer data were collected using manual measurements and inverse kinematic methods. The answer data in the manual measurement dataset were created by manually modifying the motion of the robot in CoppeliaSim such that the robot imitated human motion. The inverse kinematics answer data were created using CoppeliaSim’s inverse kinematics system to control the movement of the robot based on human motion data. CMU Mocap data 02–02, 13–14, and 141–16 corresponding to walking, jumping, and greeting motions, respectively, were used for the human motion data of the two datasets. The human motion data comprised 919 frames: 299 walking frames, 230 jumping frames, and 300 greeting frames. In addition, a manual measurement dataset and an inverse kinematics dataset, each of which has 919 frames in total, were constructed for the robot answer data. The robot answer data consisted of only the right-arm motion and upper-body motion during walking, jumping, and greeting. With respect to motion, the right arm of the robot has four joints and the upper body of the robot has eight joints.

4.1 ANN-based Right-Arm and Upper-Body Motion Imitation Performance

The performance of the model, in which the robot learned to imitate human motion through the two aforementioned datasets, was verified using 10-fold cross-validation. The learning performance was validated after randomly inputting 919 motion frames and splitting them into 800 training and 119 testing motion frames. The first step was to perform 10-fold cross-validation on the 800 training motion frames, followed by learning using the 800 training frames and testing on the 119 testing motion frames.

The performance results presented in Tables 1 and 2 for the manual measurements and inverse kinematics were obtained using the proposed model. The manual measurement and inverse kinematics datasets were used as the ground-truth dataset to train and evaluate the performance of the proposed model. Table 1 presents the experimental results of the 10-fold cross-validation and the test data of the neural network trained with the manual measurement and inverse kinematics datasets for the right arm of the robot. The average absolute error is the outcome of computing the combined error of the answer data and robot prediction data. Table 1 confirms the average error values for the 10-fold cross-validation and the manually measured answer values in the test set for the human right arm, which were 3.2450° and 2.9743°, respectively. The 10-fold cross-validation was conducted, resulting in an average of 4.2357° for the inverse kinematic answer value for the human right arm, whereas the test set error was 3.1399°. The results of learning with the inverse kinematics dataset showed a slightly larger error value than those with the manual measurement dataset. However, the small errors for both datasets verified the right-arm motion imitation performance of the robot.

Table 2 lists the experimental results of the 10-fold cross-validation and the test data of the neural network trained with the manual measurement and inverse kinematics datasets for the upper-body motion of the robot. The average absolute error is the outcome of computing the combined error of the answer data and robot prediction data. According to Table 2, using the manually measured answer, the human upper-body average errors for the 10-fold cross-validation and test sets were 5.1018° and 5.3845°, respectively. For the inverse kinematics answer for the upper-body, the average error values of the 10-fold cross-validation and test sets were 4.8150° and 4.6146°, respectively, as listed in Table 2. The error value for the right-arm motion was larger in both datasets for the robot’s upper-body learning neural network, although the small differences in the angle confirmed the ability of the robot to better imitate human upper-body motion.

Figures A1–A5 illustrate the differences in joint angles between the actual and predicted motions across all activities. On average, the robot can imitate human motion with an error range from 2° to 5°. Notably, during the “hello” motion, which exhibited significant discrepancies, particularly between frames 150 and 200, the training was adapted to ensure smooth transitions. This was achieved by seamlessly blending the motions to avoid abrupt actions, thereby effectively handling outliers in the answer data and improving the fidelity of the motion imitation by the robot.

After training, the loss graphs in Figure 3 were analyzed to assess the accuracy of the learning process of the model. Figures 2(a)–2(c) display the loss graphs for the walking, jumping, and greeting motions, respectively, using the inverse kinematics dataset. Similarly, Figures 2(d)–2(f) show the loss graphs for the same corresponding motions using the manually measured dataset. All the training results demonstrated a convergence of loss values toward zero, indicating effective learning across the datasets.

4.2 Results of Robot Motion Imitation Experiment

Learning was performed using an ANN with 800 randomly built data points, and testing was conducted with 119 test data points. The experimental results of the joint values of the robot imitation motion observed during the test in the CoppeliaSim— a virtual environment—were compared with those predicted by the neural network. Through the proposed ANN, the robot aims to imitate human motion as closely as possible. The robot imitated the right-arm and upper-body motions using the model described in Section 4.1. Figures 4 and 5 show the robot demonstrating motions with its right arm and upper body, where panels (a), (b), and (c) represent the walking, jumping, and greeting imitation motions, respectively. The picture on the left side of each figure represents the human motion, whereas the picture on the right side shows the results of the imitation motion by the robot. The signal graphs in Figures 4 and 5 show each joint angle of the robot according to time, and Columns 1, 2, and 3 represent the walking, jumping, and greeting motions of the robot joints, respectively. Figures 4(d)–4(f) show the inverse kinematics answer data graphs, and panels (g), (h), and (i) in Figures 4 and 5 show the motion prediction result graphs that the robot learned from the inverse kinematics dataset. Panels (j), (k), and (l) in Figures 4 and 5 represent the manually measured answer data graphs, whereas panels (m), (n), and (o) in Figures 4 and 5 show the motion prediction result graphs learned by the robot using the manual measurement dataset. Generally, the pattern observed with respect to the change in the angle in each answer graph was similar to that observed in the robot prediction result graph. A detailed analysis of each imitation experiment is as follows.

The experiment conducted on right-arm movement in Figure 4 demonstrates that the proposed model can predict and imitate human behavior based on both the inverse kinematics and manually measured datasets. As an example, Figure 4(c) shows the greeting behavior, and Figure 4(f) and 4(I) illustrate the complex movements between frames 25 and 150, which were predicted and imitated by the robot based on the inverse kinematics dataset. Similarly, Figure 4(l) and 4(o), which show experiments on the manually measured dataset, displayed human-like movements between frames 25 and 200. In the upper-body movement experiment shown in Figure 5, the robot successfully imitated human behavior, including the joint angles that increased during walking, as depicted in Figure 5(a). Both the inverse kinematics and manually measured datasets resulted in human-like movements that were predicted by the robot between frames 25 and 140, as shown in Figure 5(d) and 5(g) and Figure 5(j) and 5(m), respectively.

The experimental results demonstrate that the proposed model successfully enables the robot to imitate human behavior in both repetitive and complex movements, as shown in Figures 4 and 5. The robot could predict and replicate human actions for specific point movements, such as walking and running, and for complex movements, such as greeting behavior. These results confirm the effectiveness of the proposed model in minimizing restrictions on the degrees of freedom while achieving human-like behavior imitation. A video demonstration of the performance of the robot can be found at https://youtu.be/aEsFlhsqt3o.

In the ANN verified through 10-fold cross-validation, there was almost no difference between the answer and prediction graphs, and the four joints of the robot can predict human motion. In addition, not only the right-arm motion of the robot but also the upper-body motion was studied, and the results were encouraging. Through a series of experiments, this study demonstrates that the proposed motion imitation robot based on an ANN smoothly imitates human motion, confirming its superiority.

In this paper, we proposed a technology for controlling motion imitation robots based on ANNs with minimal restrictions on the degrees of freedom. The proposed method uses ANNs to enable a robot to imitate the same motion as a human, without requiring complex formula computations. The ANN demonstrated a low error rate, and the degrees-of-freedom limitations were minimized using 10-fold cross-validation, thereby demonstrating the good performance of the proposed technology through experiments on robot motions in a virtual environment.

The proposed technology aims to achieve technological advancements in real-world applications by enabling precise and natural robotic motion. However, the current study has certain limitations. The scalability of this method for different robotic platforms or various types of motions has not been fully explored. Although the experiments were conducted in a virtual environment, challenges may arise when applying this method to physical robots or systems with different hardware configurations. Additionally, the efficiency and effectiveness of the method in real-time motion imitation, particularly in dynamic environments, must be fully tested.

Future studies can address these limitations by investigating how the proposed method can be adapted for use on other robotic platforms and motion types, including robots with more complex morphologies. Expanding the study from upper-body to full-body motion poses a significant challenge, particularly for bipedal robots that rely solely on joint angles for balance control. To address these challenges, future studies may seek to minimize degrees-of-freedom issues while exploring model-based approaches to facilitate imitation learning across various robot types.

Furthermore, incorporating lower-body joint angles and techniques, such as the ZMP, can enhance the stability of bipedal walking, allowing for full-body motion analysis. Real-time motion imitation in dynamic environments will also be a key area of exploration, helping bring this technology closer to practical applications. Ultimately, such advancements could pave the way for novel methodologies, including reinforcement learning, to further advance the boundaries of robotic research.

Figures A1–A5 illustrate the differences in joint angles between the actual and predicted motions across all activities.

Table 1. Average error of 10-fold cross-validation and test set validation for right-arm motion imitation (unit: degree).

Table 2. Average error of 10-fold cross-validation and test set validation for upper-body motion imitation (unit: degree).