Improvements on support vector machine and single hidden layer feedforward network base algorithms for binary classification

Abstract

Support vector machine (SVM) employs the linear Hinge loss function for its misclassification error that exhibits noise and outlier sensitivity, and also class imbalance learning is not inherently incorporated into its optimization problem. SVM guarantees a global optimum solution obtained by solving a quadratic programming problem(QPP) which comes at a high computational cost. Twin support vector machines (TWSVM) is an SVM variant that, unlike SVM, finds two non-parallel proximal class hyperplanes which are closest to their corresponding class samples and at least unit relative distance away from the opposite class. Theoretically, with a reduced training cost of approximately four times of SVM, TWSVM also improves the generalization performance of SVM. Single hidden layer feedforwards networks (SLFNs) are another category of generalization efficient classification algorithms that consist of only one hidden layer. Random vector Functional-link networks (RVFL) and extreme learning machines (ELM) are two efficient classes of SLFN algorithms which are non-iterative closed-form solution-based approaches. RVFL and ELM are capable of handling some issues of the traditional artificial neural networks(ANNs), such as slow convergence rate and the presence of local minima. Borrowing the underlying philosophy from TWSVM, the recently proposed twin extreme learning machines(TELM) for binary classification construct two non-parallel hyperplanes in the ANN feature space. Similar to TWSVM, TELM requires solving two QPPs of class sizes. Keeping generalization efficiency as the primary criterion, in this research, improvements in respect to noise and outlier sensitivity, class imbalance learning and/or training computational complexity are suggested on some SVM and SLFN-based classification methods. An extensive study on the loss functions of SVM-based classification methods is carried out and SVM models with some robust loss functions are proposed.