Event Abstract

Back to Event

Usefulness of a neural network having the logistic function as the activation function of its output unit

Yoshifusa Ito1*, Hiroyuki Izumi2 and Cidambi Srinivasan3
  • 1 Aichi Medical University, Department of Physiology, Japan
  • 2 Aichi Gakuin University, Faculty of Policy Studies, Japan
  • 3 University of Kentucky, Department of Statistics, United States

Posterior probabilities are used as Bayesian discriminant functions. It is well known that a neural network can learn a posterior probability [9]. In the two category case, the logit transform of a posterior probability is a log ratio of two probability measures shifted by a log ratio of two prior probabilities. Since the logit function is monotone, the logit transform of a posterior probability is also a Bayesian discriminant function (see [1]). Though there is no simple way of obtaining a Bayesian discriminant function of the latter type as an output of a neural network, it can be obtained as the inner potential by training a neural network. If a neural network has the logistic function as the activation function of the output unit, the inner potential of the unit is the logit transform of the output. If the two probability distributions are from the exponential family and the network outputs the posterior probability, the inner potential has thus a simple form. Using this simplicity, Funahashi [2] has proposed a neural network which may approximate a Bayesian discriminant function, where he has supposed that the probability distributions are normal. We have shown that a network based on the algorithm given in [3] can in fact learn Bayesian discriminant functions [4], though Funahashi has not shown that his network can learn a Bayesian discriminant function. We have also shown that the network can be extended to multi-category cases [5]. We show that the neural network with the output unit having the logistic activation function is versatile, mainly because the inner potential of the output unit of a trained neural network is approximately a log ratio of two probability measures. The network has a capability of converting a Bayesian discriminant function to the corresponding Mahalanobis discriminant function. If the probability distributions are normal, the conversion can be done simply by shifting the inner potential of the output unit by a constant, because the logarithm of the p.d.f. of a normal distribution is a linear function of the square of the Mahalanobis distance. The network can also estimate the constant, the size of the sift [6]. Even if the distributions of the teacher signals are not normal, their distributions can be converted to normal individually, by applying the law of large numbers and the central limit theorem, in a way that keeps the means and variances of the distributions. Hence, the network can be used to obtain a Mahalanobis discriminant function, even when the distributions of the original teacher signals are not normal [7]. Furthermore, the network can realize the algorithm of Khasminskii et al. [8] for estimating Markov chains because the posterior probabilities at the respective steps can be obtained by shifting the inner potential of the output unit. These algorithms can be extended to multi-category cases, if several neural networks are simultaneously used [5,7].

References

1. R.O. Duda and P.E. Hart: Pattern classification and scene analysis, Joh Wiley & Sons, New York, 1973.
2. K. Funahashi: Multilayer neural networks and Bayes decision theory. Neural Networks, 11, 209-213, 1998.
3. Y. Ito: Simultaneous approximations of polynomials and derivatives and their applications to neural networks. Neural Computation 20, 2757-2791, 2008.
4. Y. Ito and C. Srinivasan. Bayesian decision theory on three-layer neural networks, Neurocomputing, 63, 209-228, 2005.
5. Y. Ito, C. Srinivasan and H. Izumi: Multi-category Bayesian decision by neural networks. Proceedings of ICANN 2008, LNCS 5163, Springer, 21-30, 2008.
6. Y. Ito, H. Izumi and C. Srinivasan: Learning of Maharanobis discriminant functions by a neural network. ICONIP 2009 I, LNCS 5863, 417-424, 2009.
7. Y. Ito, H. Izumi and C. Srinivasan: Learning Mahalanobis Discriminant Functions by a Neural Network (in preparation).
8. R. Khasminskii, B. Lazareva and J. Stapleton: Some procedures for state estimation of a hidden Markov chain with two states, Statistical decision theory and related topics, Eds. S.S. Gupta and J. Berger, Springer Verlag, 477-487, 1994.
9. D.W. Ruck, S.K. Rogers, M. Kabrisky, M.E. Oxley, B.W. Suter: The multilayer perceptron as an approximation to a Bayes optimal discriminant function. IEEE Transactions on Neural Networks 1, 296-298, 1990.

Keywords: Neural Network, Learning, activation function, logistic function, Bayesian, mahalanobis, deterministic function

Conference: Neuroinformatics 2015, Cairns, Australia, 20 Aug - 22 Aug, 2015.

Presentation Type: Poster, to be considered for oral presentation

Topic: General neuroinformatics

Citation: Ito Y, Izumi H and Srinivasan C (2015). Usefulness of a neural network having the logistic function as the activation function of its output unit. Front. Neurosci. Conference Abstract: Neuroinformatics 2015. doi: 10.3389/conf.fnins.2015.91.00013

Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters.

The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated.

Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed.

For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions.

Received: 07 Apr 2015; Published Online: 05 Aug 2015.

* Correspondence: Prof. Yoshifusa Ito, Aichi Medical University, Department of Physiology, Nagakute-shi, Aichi-ken, 480-1195, Japan, ito@aichi-med-u.ac.jp