Support vector regression (SVR) has become very promising and popular in the field of machine learning due to its attractive features and profound empirical performance for small sample, nonlinearity and high dimensional data application. However, most existing support vector regression learning algorithms are limited to the parameters selection and slow learning for large sample. This paper considers an adaptive particle swarm optimization (APSO) algorithm for the parameters selection of support vector regression model. In order to accelerate its training process while keeping high accurate forecasting in each parameters selection step of APSO iteration, an optimal training subset (OTS) method is carried out to choose the representation data points of the full training data set. Furthermore, the optimal parameters setting of SVR and the optimal size of OTS are studied preliminary. Experimental results of an UCI data set and electric load forecasting in New South Wales show that the proposed model is effective and produces better generalization performance.
Recently, support vector regression (SVR), which was developed by Vapnik [20], has become very promising and popular in the field of machine learning due to its attractive features and profound empirical performance for small sample, nonlinearity and high dimensional data application [1], [2] and [3]. The main technique of SVR is to use the principle of structural risk minimization (SRM) by constructing an optimal regression hyper-plane in the hidden feature space and solving the unique solution of the accordingly dual quadratic programming problem.
In the SVR, the model for forecasting is generated from the learning process with the training data set. Then, SVR has been successfully applied to solve forecasting problems in many areas, such as financial time series forecasting [4], short term wind speed prediction [5], face recognition [6], electric load prediction [7], and so on. However, these empirical results indicated that the largest problems encountered in building up the SVR are how to select the three parameters (C, ε, and δ2) and improve the slow learning for large sample. To solve the above problems, many researchers have given some parameter setting algorithms [8], [9], [10] and [11]. Particle swarm optimization has become a popular parameters selection algorithm [12], Lin et al. utilize particle swarm optimization (PSO) for parameter determination and feature selection of support vector machines (SVM) [13], Huang and Dun present an optimization mechanism that hybridized PSO and SVM to improve the classification accuracy with an appropriate feature subset and SVM's parameters [14]. Considering that the computation complexity is O(K × N2) (K is the number of iteration), parameter setting algorithms will lead to slow learning in large-scale training data set. This paper aims to present a model for solving the above problem.
In the large sample learning problem, the training data set contain much redundant information generally. The redundant data not only are useless for SVR learning, but also could lead to low computational efficiency and low accuracy potentially. Thus, discarding the redundant information of training data set can accelerate learning process of SVR's parameters selection. Inspired by that not all of these training data are equally important for a specific forecasting problem, only the support vectors determine the final SVR model. Better computation performance and generalization ability can be achieved by choosing the optimal training subset (OTS) containing support vectors. Therefore, the learning process can be fast and accurate by using the APSO algorithm and OTS selection method.
By combining the optimal training subset reconstruction method with APSO, here the author presents a new parameter selection algorithm for SVR, called APSO-OTS-SVR. Some improved techniques in the optimization framework are presented in order to simplify the APSO iteration learning algorithm and increase the learning speed. Based on APSO-OTS-SVR, forecasting models for an UCI data set and electric load forecasting in New South Wales are proposed. Compared with three SVR models, the experimental results show that APSO-OTS-SVR provides a parameters selection and better generalization performance at higher learning speed.
The rest of the study is organized as follows. Section 2 proposes APSO-OTS-SVR, and the main steps of it are also given in this section. The experiment design of the forecasting model is given in Section 3. Section 4 presents the experimental results. The final conclusion is drawn in Section 5.
The critical problem encountered in building up the SVR is how to select the three parameters (C, ε, and σ). In an iteration optimization algorithm, the SVR model is learned by solving a quadratic programming problem after the parameter selection of SVR, and its complexity is O(N2) (N is the number of training set). SVR learning process suffers from slow training speed in the case of large-scale training data, and over-fitting is caused probably [31].
Based on the above reason, this paper has proposed a parameters selection framework for the SVR model under OTS. The framework inherently has a concentration capable of solving the complexity problem O(K × N2) suffered by SVR model and avoiding the over-fitting of large-scale training set (K is the number of iteration). The contribution in this paper has been threefold, namely the specification of the parameters selection framework for the SVR model under OTS, the simplification iteration process by combining with APSO and OTS, and the relation study between the parameters setting of SVR and the size of OTS. The experimental results demonstrate the applicability and superiority of the proposed APSO-OTS-SVR model. However, it is fair to say that much remains to be done in the way of model construction.
