Conventionally, a radial basis function (RBF) network is constructed by obtaining cluster centers of basis function by maximum likelihood learning. This paper proposes a novel learning algorithm for the construction of radial basis function using sensitivity analysis. In training, the number of hidden neurons and the centers of their radial basis functions are determined by the maximization of the output's sensitivity to the training data. In classification, the minimal number of such hidden neurons with the maximal sensitivity will be the most generalizable to unknown data. Our experimental results show that our proposed sensitivity-based RBF classifier outperforms the conventional RBFs and is as accurate as support vector machine (SVM). Hence, sensitivity analysis is expected to be a new alternative way to the construction of RBF networks.
As one of the most popular neural network models, radial basis function (RBF) network attracts lots of attentions on the improvement of its approximate ability as well as the construction of its architecture. Bishop (1991) concluded that an RBF network can provide a fast, linear algorithm capable of representing complex non-linear mappings. Park and Sandberg (1993) further showed that RBF network can approximate any regular function. In a statistical sense, the approximate ability is a special case of statistical consistency. Hence, Xu, Krzyzak, and Yuille (1994) presented upper bounds for the convergence rates of the approximation error of RBF networks, and proved constructively the existence of a consistent estimator point-wise and L2 convergence rates of the best consistent estimator for RBF networks. Their results can be a guide to optimize the construction of an RBF network, which includes the determination of the total number of radial basis functions along with their centers and widths.
There are three ways to construct an RBF network, namely, clustering, pruning and critical vector learning. Bishop, 1991 and Xu, 1998 follow the clustering method, in which the training examples are grouped and then each neuron is assigned to a cluster. The pruning method, such as Chen et al., 1991 and Mao, 2002, creates a neuron for each training example and then to prune the hidden neurons by example selection. The critical vector learning method, exemplified by Scholkopf, Sung, Burges, Girosi, Niyogi, and Poggio (1997) constructs an RBF with the critical vectors, rather than cluster centers.
Moody and Darken (1989) located optimal set of centers using both the k-means clustering algorithm and learning vector quantization. The drawback of this method is that it considers only the distribution of the training inputs, yet the output values influence the positioning of the centers. Bishop (1991) introduced the Expectation–Maximization (EM) algorithm to optimize the cluster centers with two steps: obtaining initial centers by clustering and optimization of the basis functions by applying the EM algorithm. Such a treatment actually does not perform a maximum likelihood learning but a suboptimal approximation. Xu (1998) extended the model for mixture of experts to estimate basis functions, output neurons and the number of basis functions all together. The maximum likelihood learning and regularization mechanism can be further unified to his established Bayesian Ying Yang (BYY) learning framework (Xu, 2004a, Xu, 2004b and Xu, 2004c), in which any problem can be decomposed into Ying space or invisible domain (e.g., the hidden neurons in RBFs), and Yang space or visible domain (e.g., the training examples in RBFs), and the invisible/unknown parameters can be estimated through harmony learning between these two domains.
Chen et al. (1991) proposed orthogonal least square (OLS) learning to determine the optimal centers. The OLS combines the orthogonal transform with the forward regression procedure to select model terms from a large candidate term set. The advantage of employing orthogonal transform is that the responses of the hidden layer neurons are decorrelated so that the contribution of individual candidate neurons to the approximation error reduction can be evaluated independently. However, the original OLS learning algorithm lacks generalization and global optimization abilities. Mao (2002) employed OLS to decouple the correlations among the responses of the hidden units so that the class separability provided by individual RBF neurons can be evaluated independently. This method can select a parsimonious network architecture as well as centers providing large class separation.
The common feature of all the above methods is that the radial basis function centers are a set of the optimal cluster centers of the training examples. Schokopf et al. (1997) calculated support vectors using a support vector machine (SVM), and then used these support vectors as radial basis function centers. Their experimental results showed that the support-vector-based RBF outperforms conventional RBFs. Although the motivation of these researchers was to demonstrate the superior performance of a full support vector machine over either conventional or support-vector-based RBFs, their idea of critical vector learning is worth borrowing.
This paper proposes a novel approach to determining the centers of RBF networks based on sensitivity analysis. The remainder of this paper is organized as follows: In Section 2, we describe the concepts of sensitivity analysis. In Section 3, the most critical vectors are obtained by OLS in terms of sensitivity analysis. Section 4 contains our experiments and Section 5 offers our conclusions.
The conventional approach to constructing an RBF network is to search for the optimal cluster centers among the training examples. This paper proposes a novel approach to RBF construction that uses critical vectors selected by sensitivity analysis. Sensitivity is defined as the expectation of the square of output deviation caused by the perturbation of RBF centers. In training, orthogonal least square incorporated with a sensitivity measure is employed to search for the optimal critical vectors. In classification, the selected critical vectors will take on the role of the RBF centers. Our experimental results show that our proposed RBF classifier performs better than the conventional RBFs and C4.5. The sensitivity-based RBF can achieve the same level of accuracy as SVM, but strikes a balance between critical vector learning and robustness against noisy data.
Our future work includes the development of a decomposition method for large scale classification problems, as well as the optimization of RBF widths integrated into sensitivity analysis.
