WEIGHT INITIALIZATION AND ACTIVATION FUNCTIONS FOR SIGMOIDAL FEEDFORWARD ARTIFICIAL NEURAL NETWORKS

Abstract

Artiand#64257;cial neural networks ANN are known for their parallel structure inspired from human brains which are highly non-linear, complex and hugely parallel information processing system. ANNs can be utilized to solve a wide variety of tasks including function approximation, regression, classiand#64257;cation, density analysis etc.. For training of the Artiand#64257;cial Neural Networks, the architectural decision about the size of hidden layer for a single hidden layer, in particular is decided by exploratory experiments.Activation, Squashing,Transfer function is one of the key components in neural networks besides neurons, weights and learning algorithm. It maps the possibly inand#64257;nite input domain to an in general, prespeciand#64257;ed and or and#64257;nite output range. The number of the activation functions is inand#64257;nite and it is the choice of researcher to choose a particular activation function. Some of the frequently used activation functions are log-sigmoidal and tangent hyperbolic which have uni-modal derivatives, that is, these have a derivative which has a single local maxima. A study is required to and#64257;nd new activation functions as the study for and#64257;nding new activation functions has not been performed to the extent desired. New activation functions have been proposed in this thesis. One is a class of bi-modal derivative activation functions parameterized by a parameter which gives two maxima of equal values for non-zero value of the parameter, of the activation function deriva- tive. Another function proposed in the study is the skewed derivative activation function that is, whose derivative is shifted from the y-axis. All the proposed functions satisfy the requirements of the universal approximation theorems for ANNs.