(Return to ICNN97 Homepage) (Return to ICNN'97 Agenda)
ICNN'97 FINAL ABSTRACTS
LM: LEARNING & MEMORY
ICNN97 Learning & Memory Session: LM1A Paper Number: 169 Oral
Uniform approximation and gamma networks
Irwin W. Sandberg and Lilian Xu
Keywords: gamma network Uniform approximation nonlinear input-output maps
Abstract:
We briefly describe recent results concerning the approximation of nonlinear input-output maps, and show that for any single-variable shift-invariant causal uniformly-fading-memory map G there is a focused gamma network that approximates G uniformly arbitrarily well.
Keywords: Nonlinear Input-Output Maps, Gamma Networks, Uniform Approximation, Uniform Fading Memory.
_____
ICNN97 Learning & Memory Session: LM1B Paper Number: 419 Oral
Asymptotical analysis of modular neural network
Lin-Cheng Wang, Nasser M. Nasrabadi and Sandor Der
Keywords: modular neural network data representation asymptotical performance
Abstract:
Modular neural networks have been used in several applications because of their superiority over a single neural network in terms of faster learning, proper data representation, and feasibility of hardware implementation. This paper presents an asymptotical performance analysis showing that the performance of a modular neural network is always better than or as good as that of a single neural network when both neural networks are optimized. The minimum mean square error (MSE) that can be achieved by a modular neural network is also obtained.
_____
ICNN97 Learning & Memory Session: LM1C Paper Number: 422 Oral
Recognition algorithm using evolutionary learning on the random neural networks
Jose Aguilar and Adriana Colmenares
Keywords: evolutionary learning random neural networks pattern recognition
Abstract:
_____
ICNN97 Learning & Memory Session: LM1D Paper Number: 141 Oral
Dynamics of distance between patterns for higher order random neural networks
Hiromi Miyajima and Shuji Yatsuki
Keywords: hugher order random neural network dynamical properties dynamics of distance
Abstract:
Higher order neural networks which hold the weighted sum of products of input variables, have been proposed as a new concept. In some applications using them, it is shown that they are superior in ability to the traditional neural networks. But, little is known about the fundamental property and possibility of these models. In the previous paper, we have shown dynamical properties, dynamics of the activities for states, for higher order random neural networks (HORNNs) with the digital {-1,1}-, {0,1}- and the analog {-1,1}-, {0,1}-models. This paper describes dynamics of a distance between two states in full detail for HORNNs with the digital {0,1}-model using a statistical method . Further, comparison between the digital {-1,1}- and {0,1}-models is made.
_____
ICNN97 Learning & Memory Session: LM1E Paper Number: 414 Oral
Principal components via cascades of block-layers
Francesco Palmieri and Michele Corvino
Keywords: principal components cascades of block-layers constarained connectivity
Abstract:
We study the behaviour of the cascade of linear neural networks having constrained connectivity of Hebbian and anti-Hebbian synapses. We derive a formula for the number of layers necessary to obtain a sufficiently close approximation to the principal components at the final output. Results of simulations confirm the analyses.
_____
ICNN97 Learning & Memory Session: LM2A Paper Number: 383 Oral
Bayesian geometric theory of learning algorithms
Huaiyu Zhu
Keywords: bayesian geometric theory learning algorithms objective evaluation
Abstract:
The problem of objective evaluation of learning algorithms is analyzed under the principles of coherence and covariance. The theory of Bayesian information geometry satisfies these principles and encompasses most of the commonly used learning criteria. Implications to learning theory are discussed.
_____
ICNN97 Learning & Memory Session: LM2B Paper Number: 251 Oral
Note on effective number of parameters in nonlinear learning systems
Jianchang Mao and Anil Jain
Keywords: nonlinear learning system effective number of parameters feedforward neural network
Abstract:
Moody's notion of effective number of parameters in a nonlinear learning system has been used for studying the generalization ability of feedforward neural networks. It is more meaningful than the number of free parameters in a nonlinear learning system because the former explains explicitly how the generalization error is related to the expected training set error. In this paper, we extend Moody's model to include a more general noise model. We show that the addition of noise in both sampling points in test data and observations increases the deviation of the expected test set mean-squared-error (MSE) from the expected training set MSE, and also increases the effective number of parameters. Our extention makes less restrictive assumptions about the data generation process than in the original Moody's notion. Monte Carlo experiments have been conducted to verify our extension, and to demonstrate the role of the weight-decay regularization in improving the generalization ability of feedforward networks.
_____
ICNN97 Learning & Memory Session: LM2C Paper Number: 413 Oral
New bounds for correct generalization
Davide Mattera, Francesco Palmieri.
Keywords: correct generalization number of training examples neural network architecture
Abstract:
The ability to perform generalization is one of the most important properties of connectionist networks and has been the subject of a considerable research effort, mostly through experimental studies. Theoretical rather than experimental analysis of generalization is important if we wish to obtain results that are representative of a complete class of networks.
One of the most effective techniques for tackling the analysis of the generalization is provided by Vapnik's theory of minimization of the empirical risk. The theory is very general ( in particular it is valid for any probability density on X) and very appealing, even though the bounds so far proposed in the literature are generally considered to be rather crude. However, there are indication that the general results may not be so crude on small sample sizes if compared to known asymptotic results.
From a such perspective, improvements of the bounds that the theory provides can close the gap between the theoretical generality and the applicative usefulness.
In this paper we report new results for the binary functions case on the scenario where the trained system (e.g., neural network) may exhibit, after learning, a small but non null fraction of errors. Such a situation is common in the applications and it may depend on: (1) a neural network that does not have enough degrees of freedom; (2) an inaccurate training algorithm; (3) noise on the examples.
_____
ICNN97 Learning & Memory Session: LM2D Paper Number: 48 Oral
D-Entropy minimization: Integration of Mutual Information Maximization and Minimization
Ryotaro Kamimura
Keywords: D-entropy information maximization Renyi entropy Shannon entropy
Abstract:
In this paper, we propose a D-entropy minimization method, aiming to unify information maximization and minimization methods. The D-entropy is defined by difference between R\'{e}nyi entropy and Shannon entropy. The D-entropy minimization corresponds to mutual information maximization and at the same time minimization. Thus, the method can be used to interpret explicitly internal representations and to improve generalization. The D-entropy minimization was applied to two problems: six alphabet character recognition and the inference of well-formedness of artificial strings. Experimental results confirmed that by minimizing the D-entropy a small number of principal hidden units could be detected and generalization performance could be improved.
_____
ICNN97 Learning & Memory Session: LM2E Paper Number: 438 Oral
Projection pursuit and the solvability condition applied to constructive learning
Fernando J. Von Zuben and Marcio L. de Andrade Netto
Keywords: projection pursuit solvability condition constructive learning
Abstract:
Keywords: constructive learning, nonlinear optimization, function approximation
Single hidden layer neural networks with supervised learning have been successfully applied to approximate unknown functions defined in compact functional spaces. The most advanced results also give rates of convergence, stipulating how many hidden neurons with a given activation function should be used to achieve a specific order of approximation. However, independently of the activation function employed, these connectionist models for function approximation suffer from a severe limitation: all hidden neurons use the same activation function. If each activation function of a hidden neuron is optimally defined for every approximation problem, then better rates of convergence will be achieved. This is exactly the purpose of constructive learning using projection pursuit techniques. Since the training process operates the hidden neurons individually, a pertinent activation function employing automatic smoothing splines can be iteratively developed for each neuron as a function of the learning set. Different from other papers, we apply projection pursuit in association with the optimization of the solvability condition, giving rise to a more efficient and accurate computational learning algorithm.
_____
ICNN97 Learning & Memory Session: LM2F Paper Number: 569 Oral
The generalization capabilities of ARTMAP
Gregory L. Heileman, Michael Georgiopoulos, Michael J. Healy and Stephen J. Verzi
Keywords: generalization capability ARTMAP number of training examples
Abstract:
Bounds on the number of training examples needed to guarantee a certain level of generalization performance in the ARTMAP architecture are derived. Conditions are derived under which ARTMAP can achieve a specific level of performance assuming any unknown, but fixed, probability distribution on the training data.
_____
ICNN97 Learning & Memory Session: LM3A Paper Number: 167 Oral
Automatic learning rate optimization by higher-order derivatives
Xiao-Hu Yu and Li-Qun Xu
Keywords: learning rate optimization higher-order derivatives backpropagation learning
Abstract:
Automatic optimization of learning rate is a central issue to improving the efficiency and applicability of backpropagation learning. In this paper, techniques have been investigated which explore the first four derivatives of the learning rate of backpropagation error surface. The derivatives are derived from an extended feedforward propagation procedure and can be calculated in an iterative manner. The near-optimal dynamic learning rate is obtained with only a moderate increase in computational complexity at each iteration, scaling like the plain backpropagation algorithm (BPA), but the proposed method achieves rapid convergence and very significant gains in running time savings to at least an order of magnitude as compared with the BPA.
_____
ICNN97 Learning & Memory Session: LM3B Paper Number: 406 Oral
Improved back propagation training algorithm using conic section functions
Tulay Yildirim and John S. Marsland
Keywords: back propagation conic section functions Radial basis funtion
Abstract:
A new training algorithm composed of a propagation rule which contains both MLP and RBF parts to improve the performance of back propagation is proposed. The network using this propagation rule is known as a Conic Section Function Network (CSFN). This network allows to convert the open decision boundaries in an MLP to closed ones in an RBF, or vice versa. It reduces the number of centres needed for an RBF and the hidden nodes for an MLP. It is important since this work is aimed at designing a VLSI hardware neural network. Furthermore, it converges to a determined error goal at lower training epochs than an MLP. The performance of an MLP trained back propagation and also fast back propagation using adapted learning rates, an RBFN, and the proposed algorithm is compared using Iris plant database. The results show that the introduced algorithm is much better than the others in most cases, in terms of not only training epochs but also the number of hidden units and centres.
_____
ICNN97 Learning & Memory Session: LM3C Paper Number: 96 Oral
Comparing parameterless learning rate adaptation methods
E. Fiesler and M. Moreira
Keywords: parematerless learning rate backpropagation adaptive learning rate
Abstract:
Since the popularization of the backpropagation learning rule for training multilayer neural networks, many improvements and extensions of it have been proposed. Adaptive learning rate techniques are certainly among the most well-known of such improvements, promising a significant increase in learning speed and, in case no new tunable parameters are introduced, eliminating the trial-and-error process of finding a suitable learning rate. Hence, in order to compare the most promising of these, five methods without tunable parameters have been selected. Both the online and batch versions of standard backpropagation are also integrated into the study as points of reference. However, in order to compare the convergence speed of different learning rules, a better complexity measure is needed than the commonly used `number of training iterations'. Hence, a refined complexity measure is introduced here and used in the comparison of the seven chosen methods.
_____
ICNN97 Learning & Memory Session: LM3D Paper Number: 339 Oral
Optimal stopped training via algebraic on-line estimation of the expected test-set error
Joachim Utans
Keywords: optimal stopped training algebraic on-line estimation expected test-set error
Abstract:
_____
ICNN97 Learning & Memory Session: LM3E Paper Number: 97 Oral
Weight evolution algorithm with dynamic offset range
S. C. Ng, S. H. Leung and A. Luk
Keywords: multi-layered neural network back propagation weight evolution
Abstract:
The main problems for gradient-descent algorithms such as back-propagation are its slow convergence rate and the possibility of being trapped in local minima.
In this paper, a weight evolution algorithm with dynamic offset range is proposed to remedy the above problems. The idea of weight evolution is to evolve the network weights in a controlled manner during the learning phase of back-propagation so as to jump to the regions of smaller mean squared error whenever the back-propagation stops at a local minimum. If the algorithm is consistently being trapped in a local minimum, the offset range for weight evolution will be incremented to allow larger weight space to be searched.
When the local minimum is bypassed, the offset range will be reset to the initial value. It can be proved that this method can always escape local minima and guarantee convergence to the global solution. Simulation results show that the weight evolution algorithm with dynamic offset range gives a faster convergence rate and global search capability.
_____
ICNN97 Learning & Memory Session: LM3F Paper Number: 400 Oral
Incorporating state space constraints into a neural network
Daryl H. Graf
Keywords: state space constraints continuous neural network manifold learning
Abstract:
In this paper we investigate the problem of constraining the dynamic trajectories of a continuous time neural network to a differentiable manifold in the network's state space. This problem occurs in diverse application areas where the network states can be assigned a measure of quality or cost. We derive conditions which, if satisfied, guarantee that the network dynamics will not deviate from the desired manifold.
In addition, we illustrate the approach by showing how to incorporate a mechanism for learning linear manifold constraints into a recurrent back propagation network. The network can perform associative learning in conjunction with the learning of manifold constraints.
_____
ICNN97 Learning & Memory Session: LM4A Paper Number: 551 Oral
A structural learning algorithm for multi-layered neural networks
Manabu Kotani, Akihiro Kajiki and Kenzo Akazawa
Keywords: structural learning multi-layered neural network pruning algorithm
Abstract:
We propose a new structural learning algorithm for organizing the structure of the multi-layered neural networks. The proposed pruning algorithm consists of two already known algorithms, the structural learning algorithm with forgetting and the optimal brain damage algorithm using the second derivatives of the assessment. After the network is slimmed by the structural learning algorithm with forgetting, unimportant weights are pruned from the network using the second derivatives.
The simulations are performed for the Boolean function and the acoustic diagnosis of compressors. The results show that the proposed algorithm is effective for eliminating the unimportant weights.
_____
ICNN97 Learning & Memory Session: LM4B Paper Number: 618 Oral
An improved expand-and-truncate learning
Atsushi Yamamoto and Toshimichi Saito
Keywords: expand-and-truncate learning binary-to-binary mapping number of hidden units
Abstract:
This paper proposes a novel learning algorithm that can realize any binary-to-binary mapping by using three-layer binary neural networks. The algorithm includes an improved expand-and-truncate learning routine that can reduce the number of the hidden neurons by conventional methods. Also, the output layer parameters can be given by simple analytic formulae.
_____
ICNN97 Learning & Memory Session: LM4C Paper Number: 336 Oral
A vector quantisation reduction method for the probabilistic neural network
Anthony Zaknich
Keywords: vector quantisation probabilistic neural network regression
Abstract:
This paper introduces a vector quantisation method to reduce the Probabilistic Neural Network classifier size. It has been derived from the Modified Probabilistic Neural Network which was developed as a general regression technique but can also be used for classification. It is a very practical and easy to implement method requiring a very low level of computation. The method is described and demonstrated using 4 different sets of classification data.
_____
ICNN97 Learning & Memory Session: LM5A Paper Number: 636 Oral
Robust adaptive identification of dynamic systems by neural networks
James Ting-Ho Lo
Keywords: dynamic systems adaptive identification on-line weight adjustment
Abstract:
_____
ICNN97 Learning & Memory Session: LM5B Paper Number: 521 Oral
Fibre bundles and receptive neural fields
S. Puechmorel
Keywords: fiber bundles receptive neural fields membership function
Abstract:
_____
ICNN97 Learning & Memory Session: LM5C Paper Number: 105 Oral
Fast binary cellular neural networks
Iztok Fajfar
Keywords: cellular neural network analog transient parameter optimization
Abstract:
Cellular neural networks (CNNs), with their most important generalization, CNN Universal Machine, are well suited for many real time (fast) image processing applications. However, little has been said about optimizing their processing speed so far. In the paper we show that for a class of monotonic binary CNNs one can speed up the analog transient by introducing simple non-linear cell interconnections. Employing additional parameter optimization we acquire the speed gain of up to two orders of magnitude.
_____
ICNN97 Learning & Memory Session: LM5D Paper Number: 316 Oral
Associative memory of weakly connected oscillators
Frank C. Hoppensteadt and Eugene M. Izhikevich
Keywords: multiple andronov-hopf bifurcations weakly connected neural networks Cohen-Grossberg convergence limit cycle attractors
Abstract:
It is a well-known fact that oscillatory networks can operate as Hopfield-like neural networks, the only difference being that their attractors are limit cycles: one for each memorized pattern. The neuron activities are synchronized on the limit cycles, and neurons oscillate with fixed phase differences (time delays). We prove that this property is a natural attribute of general weakly connected neural networks, and it is relatively independent of the equations that describe the network activity. In particular, we prove an analogue of the Cohen-Grossberg convergence theorem for oscillatory neural networks.
_____
ICNN97 Learning & Memory Session: LM5E Paper Number: 372 Oral
Input-to-state (ISS) analysis for dynamic neural networks
Edgar N. Sanchez and Jose P. Perez
Keywords: neural network dynamics intelligent control stability ISS analysis
Abstract:
This paper presents the ISS analysis for dynamic neural networks. We determine, using a Lyapunov function, conditions to guarantee ISS; they also guarantee globally asymptotically stability.
_____
ICNN97 Learning & Memory Session: LMP2 Paper Number: 110 Poster
Simultaneous information maximization and minimization
Ryotaro Kamimura
Keywords: information maximization information minimization internal representation
Abstract:
In this paper, we propose a method to unify information maximization and minimization methods so far developed independently to control internal representations. Information maximization methods have been applied to the interpretation of internal representations. On the other hand, information minimization methods have been used to improve generalization performance. Thus, if it is possible to maximize and minimize information, interpretation and generalization performance can simultaneously be improved. To maximize and simultaneously minimize information, we propose networks with two hidden layers. In one layer, information is forced to be maximum, while in another layer, information is decreased as much as possible. The method with two layers was applied to the simple XOR problem. Experimental results confirmed that information can be maximized and simultaneously minimized. In addition, simpler internal representations could be obtained.
_____
ICNN97 Learning & Memory Session: LMP2 Paper Number: 535 Poster
Hybrid learning algorithm with low input-to-output mapping sensitivity for iterated time-series prediction
So-Young Jeong, Minho Lee and Soo-Young Lee
Keywords: hybrid learning input-to-output mapping sensitivity iterated time-series prediction
Abstract:
A hybrid backpropagation\Hebbian learning rule had been developed to enforce low input-to-output mapping sensitivities for feedforward neural networks. This additional functionality is coded as additional weak constraints into the cost function. For numerical efficiency and easy interpretations we specifically designed the additional cost terms with the first order derivatives at hidden-layer neural activation. The additional descent term follows the Hebbian learning rule, and this new algorithms incorporate two popular learning algorithms, i.e.,the backpropagation and Hebbian learning rules. In this paper we provide theoretical justifications for the hybrid learning algorithm, and demonstrate its better performance for iterated time-series prediction problems.
_____
ICNN97 Learning & Memory Session: LMP2 Paper Number: 23 Poster
On viewing the transform performed by a hidden layer in a feedforward ANN as a complex mobius mapping
Adriana Dumitras and Vasile Lazarescu
Keywords: hidden layer feedforward network mobius mapping Hinton diagram
Abstract:
Keywords: hidden layer, feedforward neural networ, Mobius mapping, Hinton diagram
The current approach is based on our previous results [1] on studying the outcome of the learning process in a feedforward artificial network (FANN), using projections in the complex plane. Possible links between the conformal mappings and the transform performed by a hidden layer in the FANN are investigated. We show that the latter may be viewed as a Mobius transform. The results are checked experimentally and followed by conclusions and topics of future work.
_____
ICNN97 Learning & Memory Session: LMP2 Paper Number: 21 Poster
An upper bound on the node complexity of depth-2 multilayer perceptrons
Masahiko Arai
Keywords: node complexity multilayer perceptron hidden units
Abstract:
_____
ICNN97 Learning & Memory Session: LMP2 Paper Number: 548 Poster
Improved sufficient convergence condition for the discrete-time cellular neural networks
Sungjun Park and Soo-Ik Chae
Keywords: convergence condition discrete-time cellular neural network positive semi-definite constraint
Abstract:
In this paper, we derive an improved sufficient convergence condition for the discrete-time cellular neural networks (DTCNN) using the positive semidefinite (PSD) constraint and the boundary condition of DTCNN. The experimental results confirm that the derived condition offers a wider convergence range than the convergence condition of Fruehauf et al. The new condition does not depend on the type of the nonlinear output function of the DTCNN.
_____
ICNN97 Learning & Memory Session: LMP2 Paper Number: 455 Poster
Beyond weights adaptation: a new neuron model with trainable activation function and its supervised learning
Youshou Wu, Mingsheng Zhao and Xiaoqing Ding
Keywords: weights adaptation trainable activation functions supervised learning
Abstract:
This paper proposes a new kind of neuron model, which has Trainable Activation Function (TAF) in addition to only trainable weights in the conventional M-P model. The final neuron activation function can be derived by training a primitive neuron activation function. BP like learning algorithm has been presented for MFNN constructed by neurons of TAF model.
Two simulation examples are given to show the network capacity and performance advantages of the new MFNN in comparison with that of conventional sigmoid MFNN.