In the fields of machine learning and data analysis, the analysis of the relation between the input and output variables and the prediction of the output variables from newly given data have been some of the most relevant research topics [1]. In the studies dealing with the associated regression problem, two essential and conceptually diversified development strategies have been introduced: global and local learning techniques [2]. Global learning [3] assists in fitting a sophisticated model in terms of the structure to all the observation data patterns. Accordingly, the model that has been learned globally is valid throughout the input space. However, local learning [4, 5] focuses on and fits a relatively simple model to a subset of observations that are present only in a neighborhood of the query point [2]. In this scenario, the model identified by local learning pertains to some limited region of the entire input space, where the query point and its neighbors are positioned.

Local learning techniques, such as k-nearest neighbors (k-NN) classifiers [6], linear regression interpolation, and local interpolation [7], have been successfully applied to various learning tasks (refer to [8–13]).

A key issue in local learning techniques is the method to define a subset of all the data patterns to be used, to estimate the parameters of the local model at a related query point.

In this study, we are focused on developing a simple method to determine the neighbors of the given query point, without confining to any adaptive technique. Here, we develop a new nonparametric locally weighted regression (LWR), which is applicable within a region defined by the k-NN approah and in which the weight of each nearest neighbor is determined by fuzzy clustering. Typically, locally weighted learning (LWL) [14] is used to build a local model and estimate the coefficients.

The k-NN method, the first component of the proposed model, is the one of the most well-known non-parametric models. Local learning or instance-based learning (IBL) is applied here. In IBL, the learning phase is deferred until a new data pattern is provided. In the k-NN approach, the output value of a query point is estimated by taking the average of the target values corresponding to the k-NN of the query. LWR is the second functional component of the proposed model.

Although it is very difficult to determine an appropriate structure for a global model, it is unnecessary to do so for a local model in LWR. Therefore, LWR is useful in the modeling of complex non-linear systems [15]. The proposed model, referred to as non-parametric LWR is a regression model, is available within an area in which the k-NN of a query point is formed, and its coefficients are estimated using the weighted least square estimation (WLSE) with L₂ regularization. In this study, we apply the relative distance kernel of fuzzy clustering to complete the weight allocation. The distance kernel function is used to calculate the activation levels of the data patterns in fuzzy C-means (FCM) clustering [16].

This paper is organized as follows. First, in Section 2, we discuss the LWR based on the k-NNs exploited in the regression problem. The problem of local regression based on the k-NNs is further elaborated in Section 3. In Section 4, we present the experimental results. The conclusions are summarized in Section 5.

The k-NN approach is a representative method of nonparametric modeling. k-NN learning is one of the lazy learning (i.e., IBL) techniques that defer the decision of the parameters of a model until a query point is provided newly. When a query instance is given, a set of similar patterns is determined among the already obtained training patterns set and is used to estimate the output of the new instance [17]. There are two well-known advantages of lazy learning: effective usage of local information available in the input space and short training time [1].

The key assumption behind the k-NN approach is that the data patterns close to a newly provided query point may have similar target values used in the prediction task. The neighborhood of a query point, x, is defined by the k-NN method based on some distance function. The Euclidean distance or its generalized weighted version is commonly used as a distance function.

In the detailed description of the algorithm, we use the notations listed in Table 1.

The set of indices of the selected neighborhoods for a query is defined as follows:

Notation L[j] is used to represent the jth element of set L.

After deciding the neighborhood of a query point x, we estimate the target value at x, which is commonly encountered in classification problems, as follows:

where g(yL[i],c)={1,if yL[i]=c,0,if yL[i]≠c.

In case of function approximation, the k-NN method estimates the target value using a weighted average computed within some varying neighborhood [18] as Eq. (3).

where u_i = K(x_L[i], x).

It is known that the output determined by locally weighted averages could be significantly biased at the boundaries of the local region [19]. To overcome this limitation of the conventional k-NN learning, we apply the LWR. Although a linear model has two representative benefits—simplicity and easiness of use—it has a critical drawback: a relatively high bias. Specifically, if the target function is highly complex that a linear function cannot well approximate it, then a linear regression may present a poor modeling performance. Local linear regression was introduced from the viewpoint that any complex function can be well approximated by a simple linear function over an appropriate subspace in the input space [20].

The LWR is derived from the standard linear regression. The importance of a relevant instance is increased and that of an irrelevant one is decreased by weighting the data [19].

After determining the neighbors of a given query point, x, the weight of each neighbor is calculated [21]. This value indicates the importance of a particular neighbor to the training of the local regression. The weight function is determined based on the similarity between the given query point, x, and its neighbors.

We define the similarity (i.e., the kernel function) between two points, yielding the weight value of each neighbor as follows. The similarity is derived from the partition matrix defined in [23].

The kernel function is generally defined as any explicit functional form (e.g., Gaussian and ellipsoidal). However, the kernel function used in this study is a result of fuzzy clustering called as the fuzzy activation function. The kernel function called as the relative distance function is expressed as

where l ∈ L, x_l is one of the nearest neighbors, and p ∈ (1, ∞) denotes the fuzzification coefficient used in the FCM.

The fuzzification coefficient, p, determines the fuzziness of the partition produced in the clustering. When the fuzzification coefficient approaches 1, the resulting partition asymptotically approaches a Boolean (0 – 1) partition of the data. However, the fuzziness of the partition increases with the increase in the values of the fuzzification coefficient.

The output value of the query point, x, is estimated by the weighted average of the already given output values, y_i, of the neighbors of x.

where ŷ denotes the estimated output value for the newly given point, x.

Here, we present the development of a methodology to design the local regression defined within an area in which several neighbors of a given query point are positioned.

The local models being used in this study involve a constant and a linear regression. A linear regression might exhibit a high modeling bias when dealing with nonlinear systems. A local linear regression focuses on the operating region only (i.e., the local area enclosed by the nearest neighbors); therefore, it can compensate for the high bias of the overall global linear regression.

First, when a query point is provided, the local area should be determined by the nearest neighbors. The “k” nearest neighbors of the given query point are determined in terms of the Euclidean distance or its variant.

The obtained nearest neighbors determine the local area where the local model becomes available (suitable). If a new query point is provided, then in return, a new local area is defined.

After determining the local area, the local model has to be identified. To estimate its parameters, the least square estimation (LSE) is used. In this case, WLSE is more aligned with the k-NN rationale than the typical LSE.

To implement the local model within an area defined by the nearest neighbors of the query point, we substitute the kernel function (i.e., Gaussian function) with the similarity function defined in Eq. (14).

When a new data point is newly provided, the already implemented local model related to the old query point is discarded and a new one is designed based on the information associated with the new given query point. Specifically, whenever a query point is provided newly, the coefficients of the local regression are re-estimated.

As noted, the local models could be either constant or linear. Both these options were subjects of this study.

3.1 Constants

A constant local model is defined as follows:

where x is a vector of the input variables.

We modify the local objective function to be minimized,

where s denotes the learning effect coefficient (or the so-called linguistic modifier) of the kernel function, K(x_L[i], x), defined in Eq. (4).

This learning effect coefficient “s” controls the shape of the weight function. In Figure 1, we illustrate the variation in the shape of the weight function as a function of the values of “s.” As shown the increase in n, s reduces the width of the kernel function, whereas the decrease in n increases the width.

There are two extreme cases: s approaches 0 and infinity. In the first case (i.e., s → 0), the kernel function can cover the entire universe of the discourse, whereas when s → ∞, it can be considered as a singleton [24].

The optimized coefficients of the local model (6) can be calculated as follows:

where Ws=[K (xL[1],x)s0⋯00K (xL[2],x)s⋯0⋮⋮⋱⋮00⋯K (xL[k],x)s], 1 = [1 1 ⋯ 1]^T, y = [y_L[1]y_L[2] ⋯ y_L[k]]^T, and L[j] denotes the jth element of the set of L nearest neighbors.

In contrast with the global learning, in Eq. (8), only the nearest neighbors are used to estimate the values of this coefficient.

For the constant local model, the WLSE is used to estimate this value, and the shape parameter is equal to 1; the resulting local model is equivalent to the radial basis function neural network (RBFNN), where the radial basis functions are defined by FCM.

The optimized coefficient has the form

3.2 Linear Regression

The local model with a linear regression is defined as follows:

We modify the local objective function to be minimized, which is now expressed as

where x̃_i = [1 x_i] = [1 x_i1x_i2 ⋯ x_im], a_q = [a_q0a_q1 ⋯ a_qm]^T,

The optimal coefficients of the local model (10) that minimize this local objective function are calculated in the form

where I ∈ ℝ^(m+1)×(m+1) denotes the identity matrix.

The pseudo code of the proposed model is as follows.

Step 1. The k nearest neighbors of the query point are determined.

Step 2. The weight values are calculated by Eq. (4).

Step 3. The coefficient of a local model is calculated by Eq. (8) or (12) (when a constant model is used, Eq. (8) is used to estimate the coefficient, and in the case of a linear function, we use Eq. (12) to estimate the coefficients).

Step 4. The final output of the local model is calculated by Eq. (10).

We conducted several experiments to validate the generalization performance of the proposed model and compared its behavior with those of other models already introduced in the literature. Synthetic low-dimensional data are dealt with in the first subsection. In the second subsection, some selected machine learning datasets from a machine learning repository (http://www.ics.uci.edu/~mlearn/MLSummary.html) are used to validate the proposed model. We use a 10-fold cross-validation technique to determine the important structural parameters. First, all the data patterns are divided into 10 folds, and each fold is then selected as a test dataset to validate the generalization ability in consecutive order. The remaining folds are used in the training phase to estimate the coefficients of the local model with several structural parameters.

We summarize the results of the experiments in terms of the mean and standard deviation of the performance index.

The important structural parameters are summarized in Table 2. In this study, the number of nearest neighbors (k), value of the fuzzification coefficient (p), and shape parameter (s) are determined by conducting some cross-validation experiments.

The performance index of a model is defined as the root mean square error (RMSE) as follows:

4.1 Low-Dimensional Synthetic Data

A polynomial with two input variables is defined as follows:

The input variables, x and y, were generated from the evenly distributed over the input range x, y ∈ [−5, 5]. Table 3 summarizes the modeling performance (RMSE) of the proposed model according to the appropriate structural parameters of k, p, s, and O.

We note that the number of nearest neighbors that defines the local area is irrelevant to the performance of both the constant local model and the linear local model.

Let us consider the relevance of some other design parameters, such as the fuzzification coefficient (p) and the shape parameter (s), to the performance of the proposed models.

Figure 2 shows the performance index of the proposed models (constant local model and linear local model) according to the fuzzification coefficient (p) and the shape parameter (s).

As shown in Figure 2(a) and 2(b), the increase in the values of the shape parameter (s) and the decrease in the values of the fuzzification coefficient improve the performance of the proposed models under the condition that the number of nearest neighbors is fixed. When p = 2, as shown in Figure 2(c) and 2(d), a low value of the shape parameter (s) and an increase in the number of nearest neighbors lead to the worsening of the performance of the model. When the shape parameter is fixed (s = 1), as illustrated in Figure 2(e) and 2(f), we note that large values of the fuzzification coefficient (p) and numerous nearest neighbors worsen the performance.

Table 4 summarizes and compares the performance of the proposed model with the already studied models, i.e., linear regression, second-order polynomial, fuzzy k-NN smoothing approach, and local regression learning using a typical LSE.

4.2 Machine Learning Dataset

The experiments conducted with several datasets obtained from a machine learning repository are discussed in this subsection. To validate the modeling performance of the proposed model, we performed experiments with nine machine learning datasets, whose detailed information is specified in Table 5.

Table 6 summarizes the comparison results of the proposed model and the already studied models that were implemented in WEKA [25] in terms of the performance index. In Table 6, EPI denotes the performance index of the test dataset.

From Table 6, we can see that the proposed model with a linear function is an appropriate model whose average rank is 2.67. The proposed model with a linear function has the best results in six of the nine datasets among all the models in terms of the performance index (i.e., RMSE).

In this paper, we introduce a new local model based on the LWR. The proposed model can be considered as the expanded version of the k-NN approach, which is a representative of non-parametric models and the WLSE. When a query point is provided, it invokes lazy learning. First, the local area is defined in the input space, in which the k-NNs of the given query point in terms of the distance are enclosed. Second, the weights of the nearest neighbors are calculated using some predefined similarity function. Finally, the coefficients of the local regression are determined by the WLSE. As a new query is provided, the already constructed local regression is not available to the query. Therefore, a new local regression, which is available within the new local area surrounded by the new neighborhood of the newly given query point, is invoked. Based on the experimental results summarized in Table 6, local models reconstructed whenever new queries are arrived exhibit better performance than the global models. Therefore, the proposed local model can be considered as an effective approach to deal with complex system modeling. From the experimental results, it is validated that the local regression, which is reconstructed whenever a new query is given, performs better than the typical global models.

Table 1. Basic notations.

Table 2. Structural parameters of the proposed model.

Table 3. Performance index (RMSE) of the proposed model versus the polynomial order of the basic model and the size of the neighborhood.

Table 4. Results of the comparative analysis.

Table 5. Specification of the machine learning datasets.

Table 6. Results of the comparative analysis.