IJCNN 2005 Montreal Tutorial Contents Home Page

Bioinformatics and Machine Learning: the Prediction of Protein Structures on a Genomic Scale

This tutorial focuses on the application of machine learning methods, and in particular of neural networks, to problems in protein structure prediction. The tutorial first addresses the general problem of how to apply neural networks techniques to variable-size structured data, such a sequences, trees, and graphs in bioinformatics and other applications. The tutorial covers the design of feedforward and recursive neural network architectures for the prediction of protein structural features, such as secondary structures, relative solvent accessibility, disulphide bridges, and contact maps. Simulations and state-of-the-art results are presented and discussed together with more general issues, such as tradeoffs and integration with other methods, and applications to other related problems.

Keywords: neural networks, recurssive neural networks, bioinformatics, sequence analysis, protein, protein structure prediction, structural proteomic

Vita: Pierre Baldi is a Professor in the School of Information and Computer Sciences and the Department of Biological Chemistry at the University of California, Irvine and the Director of the Institute for Genomics and Bioinformatics at UCI. Born and raised in Europe, he received his PhD from the California Institute of Technology in 1986. From 1986 to 1988 he was a postdoctoral fellow at the University of California, San Diego. From 1988 to 1995 he held faculty and member of the technical staff positions at the California Institute of Technology and at the Jet Propulsion Laboratory. He was CEO of a startup company from 1995 to 1999 and joined UCI in 1999. He is the recipient of a 1993 Lew Allen Award at JPL and a Laurel Wilkening Faculty Innovation Award at UCI. Dr. Baldi has written over 100 research articles and four books:

• Modeling the Internet and the Web--Probabilistic Methods and Algorithms, Wiley, (2003);
• DNA Microarrays and Gene Regulation--From Experiments to Data Analysis and Modeling, Cambridge University Press, (2002);
• The Shattered Self--The End of Evolution, MIT Press, (2001);
• Bioinformatics: the Machine Learning Approach, MIT Press, Second Edition (2001).

His research focuses in AI, machine learning, computational biology, and biological and chemical informatics.
(Back to Top of Abstracts)

Most existing learning algorithms have been developed under standard formulations of the learning problem, such as classification or regression. However, many real-life applications motivate approaches that go beyond traditional learning methods. These new directions include new types of inference different from standard inductive inference (i.e., transduction), as well as new formulations of the learning problem. This tutorial will review standard learning formulations using the framework of Vapnik-Chervonenkis (VC) learning theory, and then discuss several possible ‘non-standard’ formulations. The tutorial consists of four parts:

(1) Motivation for new learning formulations. We provide historical, philosophical and application-driven motivations for developing new learning formulations and new types of inference (as opposed to research on new learning algorithms) under general conceptual framework of Vapnik’s Statistical Learning Theory. Then we present standard inductive learning formulations, and critically review the assumptions underlying such formulations. Relaxing some of these assumptions leads to several ‘non-standard’ formulations as discussed below.

(2) Non-inductive types of inference. These include non-inductive types of inference such as transduction and selection learning formulations. We describe these formulations, and several recent applications studies comparing inductive vs. transductive learning.

(3) Application-driven formulations. We outline general framework for mapping application requirements onto a learning problem formulation, and then provide examples of such new application-driven formulations. Such examples range from financial engineering (trading) to identity fraud detection.

(4) Multiple model estimation. All standard inductive learning formulations assume that all available (or training) data can be described by a single statistical model. For example, in classification setting the goal is to estimate a (single) decision boundary. Likewise, under regression formulation, the goal is to estimate a single target function from finite and noisy data samples. However, there are many applications which naturally lead to learning formulations where the goal is to estimate several models from available (finite) data. For example, in computer vision, a sequence of video frames may contain several moving objects (with overlapping trajectories), so the problem of multiple motion estimation can be viewed under multiple model estimation framework. I will describe a new learning formulation for such settings, and then introduce SVM-based learning algorithms for this new formulation.

Finally, I will discuss methodological issues arising in using learning methods for various real-life applications, such as:

- clear separation between the learning problem formulation and the solution approach (e.g, learning algorithm);

- importance of mapping application requirements onto an appropriate problem formulation, rather than inventing/ reinventing new learning algorithms;

INTENDED AUDIENCE: researchers and practitioners interested in understanding new learning formulations motivated by practical application requirements, rather than standard formulations motivated by mathematical modeling considerations.

OTHER CONSIDERATIONS: the proposed tutorial is based on two plenary talks (on the same subject of New Learning Problem Formulations) given by the instructor at ICANN-01 in Vienna, Austria and ANNIE-04 in St Louis, Missouri.

Keywords: Inductive principle, inductive inference, neuroimaging, new learning formulations, Popper's falsifiability, regularization, statistical learning, support vector machines, transduction, Vapnik-Chervonenkis theory

Vita: Vladimir Cherkassky is Professor of Electrical and Computer Engineering at the University of Minnesota. He received Ph.D. in Electrical Engineering from University of Texas at Austin in 1985. His current research is on methods for predictive learning from data, and he has written a monograph Learning From Data published by Wiley in 1998. Prof. Cherkassky has served on the Governing Board of INNS. He has served on editorial boards of IEEE Transactions on Neural Networks, the Neural Networks Journal, the Natural Computing Journal and the Neural Processing Letters. He served on the program committee of major international conferences on Artificial Neural Networks. He was Director of NATO Advanced Study Institute (ASI) From Statistics to Neural Networks: Theory and Pattern Recognition Applications held in France, in 1993. He has presented numerous tutorials on neural network and statistical methods for learning from data.
(Back to Top of Abstracts) (Back to Top of Authors)

Evolutionary Robotics is the artificial evolution of control systems for autonomous robots. Most of the past and ongoing research in Evolutionary Robotics is concerned with evolution of neural controllers for which available learning algorithms are not suitable. The tutorial will put special emphasis on how Evolutionary Robotics can be used to study open questions in evolutionary biology, neurophysiology, and cognitive science. A short introduction to artificial evolution will allow participants to fully understand and replicate most of the experiments by using freely available software.

An highlight of the topics: Evolution of simple navigation skills. The principle of parsimony: Evolution of complex behaviours using simple reactive mechanisms. The role of fitness functions. Evolution of vision-based navigation. The role of the environment in the evolutionary process and the emergence of spatial maps. Co-evolution of active vision and feature sensitivity for image recognition, robot navigation, and car driving. Interactions between evolution and learning. Evolution of learning rules and analysis of evolved neural controllers. Competitive co-evolution in biology and robotics. The case of predator-prey robots. Introducing learning in co-evolutionary competitive systems. Evolution of spiking control networks and other exotic neural controllers.

All topics will be illustrated with several examples of robots with wheels, legs, wings.

Keywords: mobile robots, evolutionary systems, autonomous systems, time-dependent networks, control systems

Vita: DARIO FLOREANO is Professor of Intelligent Systems at the Swiss Federal Institute of Technology in Lausanne (EPFL) where he is director of the Institute of Systems Engineering. His research activities include embodied and embedded neural networks, artificial evolution, robotics, bio-mimetic electronics, self-organizing systems, and artificial life. He held senior research positions at the National Research Council in Roma, at the University of Stirling, at the Swiss Federal Institute of Technology in Lausanne, and at Sony Computer Science Laboratory in Tokyo. Dario published more than 100 peer-reviewed papers, authored 2 books, and edited 3 other books. His book with S. Nolfi, Evolutionary Robotics, was reprinted by MIT Press three times over the last four years. He co-organized 6 international conferences and joined the program committee of more than 50 other conferences. He is on the editorial board of the journals Neural Networks, Genetic Programming and Evolvable Machines, Adaptive Behavior, Artificial Life, Connection Science, IEEE Transactions on Evolutionary Computation, and Autonomous Robots. He is co-founder and member of the Board of Directors of the International Society for Artificial Life, Inc. and member of the Board of Governors of the International Society for Neural Networks.(Back to Top of Abstracts) (Back to Top of Authors)

Evolving Connectionist Systems: Biological Principles, Models and Applications

Prof. Nik Kasabov (nkasabov@aut.ac.nz)
Knowledge Engineering and Discovery Research Institute (KEDRI), http://www.kedri.info, Auckland University of Technology, New Zealand

Evolving connectionist systems (ECOS) is a general paradigm of neural systems that evolve their structure, their functionality and their internal knowledge representation through continuous learning from data and interaction with the environment. The learning process can be: on-line, off-line, incremental, supervised, unsupervised, active, sleep/dream, etc. ECOS are inspired by the evolving processes in the brain. These processes are self-organized and are governed by both external stimuli (data) and internal parameters - genes. The tutorial consists of three parts, each presented in 35 minutes with a break of 15 min. after the first two parts. Part one presents biological principles of evolving brains, that include: synaptic changes through learning, neuron creation, brain development, the role of genes for brain functions and malfunctions, gene interaction networks, the role of evolution.
Part two presents several ECOS models, that include both simple ECOS and computational neurogenetic models (CNGM) explained and illustrated through simulations and examples. Simple ECOS models are presented in [N.Kasabov, Evolving connectionist systems: Methods and Applications in Bioinformatics, Brain study and intelligent machines, Springer, 2002, www.springer.de)]. They are adaptive neural systems that learn on-line and facilitate rule extraction. CNGM are connectionist models that integrate neural networks and gene interaction networks. Interaction of genes in every neuron affects the dynamics of this neuron and may influence the whole neural network performance which changes as a function of gene expression. Through optimization of the gene interaction networks, the initial gene/protein expression values and the CNGM parameters, particular target states of the CNGM can be achieved. This is illustrated by means of a simple neurogenetic model of a spiking neural network (SNN). The behaviour of the SNN is evaluated by means of the local field potential, thus making it possible to attempt modelling the role of genes in different brain states, where EEG data is available to test the model.
Part three presents applications of ECOS models in bioinformatics, brain study, adaptive robot control, finance and business prediction, medical decision support, evolvable hardware, evolvable software, adaptive integrated speech, image and fingerprint recognition.
The tutorial targets computer scientists, neuroscientists, biologists, engineers and graduate students.

Keywords: Computational Intelligence, Neuroinformatics, Bioinformatics, Knowledge-based neural networks, Evolving connectionist systems, Gene regulatory networks, Computational neurogenetic modeling.

Vita: Professor Nik Kasabov is the Founding Director and the Chief Scientist of the Knowledge Engineering and Discovery Research Institute KEDRI, Auckland (http://www.kedri.info/). He obtained a MSc and a PhD from the Technical University of Sofia. At present he holds a Chair of Knowledge Engineering at the School of Computer and Information Sciences at Auckland University of Technology. He is a Fellow of the Royal Society of New Zealand, Fellow of the New Zealand Computer Society, and a Senior Member of IEEE. Kasabov is the chair of the Adaptive Systems Task Force of the Neural Network Technical Committee of the IEEE Computational Intelligence Society. He is a member of the Board of Governors of the INNS and a board member of the Asia Pacific Neural Network Assembly (APNNA). His main research interests are in the areas of intelligent information systems, soft computing, neuro-computing, bioinformatics, brain study, speech and image processing, data mining and knowledge discovery, where he has published 10 books, 380 refereed papers, 25 patents and other work. He has an extensive academic experience, holding positions at the Technical University of Sofia- Bulgaria, University of Essex - UK, University of Otago -New Zealand. Kasabov is on the editorial boards of 8 international journals and has been on the PC of 50 international conferences in the last 5 years. He chaired the series of ANNES conferences (1993-2001). More information of Prof. Kasabov can be found on the Web site: http://www.kedri.info.
(Back to Top of Abstracts) (Back to Top of Authors)

Feedback control systems occur throughout nature. Included is the biological cell, which uses feedback mechanisms to pump ions across its membrane to sustain its equilibrium, or homeostasis. Volterra showed that the population balance of predator and prey species of fish in a closed pond is due to a feedback mechanism. Darwin showed that feedback over long time periods is responsible for natural selection. Man-made feedback control systems make possible the reliable performance of all modern complex systems including aircraft, spacecraft, vehicle motor systems, submarine stabilization, and robotic industrial machines. Though Thomas Newcomen invented the steam engine in 1712, the Industrial Revolution started in earnest only after J. Watt invented the feedback control systems for the steam engine in 1786.

In this tutorial workshop, we focus on Neural Networks for Feedback Control of Dynamical Systems, showing how NN can be tailored in topology for highly effective use in control systems design, and tailored in tuning of memory to provide stability and performance results that significantly extend what had been possible using standard control theory approaches. The issue in feedback control theory is to provide: (1) rigorous mathematical stability proofs, without which results will not be accepted by the Control Systems Community, and (2) performance and robustness guarantees, without which results will not be accepted by industry.

We will provide a background on dynamical systems properties, feedback control issues, and a roundup of NN properties that prove very useful for feedback control purposes. Relations will be drawn with fuzzy logic systems, which are nonlinear approximators based on the properties of human linguistic systems. Some feedback control topologies that use NN will be presented, including NN designs for feedback linearization, backstepping, force control, and singular perturbations. Techniques for stability and performance proofs in feedback control will be outlined. It is shown that NN designs significantly improve upon standard control theory techniques and allow improved robustness and performance for systems with nonlinearities, unknown dynamics and disturbances, and friction. Applications are to robotics, industrial systems, aircraft control, and vehicle motion platforms.

Most naturally occurring feedback systems are based on optimality or minimality principles to conserve energy, fuel, or effort. Therefore, during the latter portion of the tutorial we will show how NN can be used for optimal control systems using designs based on the Hamilton-Jacobi equations. Both H-2 and H-infinity robust design will be treated. Connections will be drawn with Approximate Dynamic Programming techniques.

Keywords: Feedback control using neural networks, Dynamical neural network control, Nonlinear adaptive control systems

Vita: Dr. F.L. Lewis is a University Distinguished Scholar Professor and the Moncrief-O'Donnell Endowed Chair at The University of Texas at Arlington. His current interests include intelligent control, neural and fuzzy systems, MEMS control, wireless sensor networks, nonlinear systems, robotics, condition-based maintenance, and manufacturing process control. He is the author/co-author of 4 U.S. patents, 150 journal papers, 240 refereed conference papers, nine books including Optimal Control, Optimal Estimation, Applied Optimal Control and Estimation, Aircraft Control and Simulation, Control of Robot Manipulators, Neural Network Control, High-Level Feedback Control with Neural Networks and the IEEE reprint volume Robot Control. He is a Fellow of the IEEE, a Fulbright Awardee, a member of the New York Academy of Sciences, and a registered Professional Engineer in the State of Texas. He has served as Visiting Professor at Democritus University in Greece, Hong Kong University of Science and Technology, Chinese University of Hong Kong, National University of Singapore. He is an elected Guest Consulting Professor at both Shanghai Jiao Tong University and South China University of Technology. He is the recipient of an NSF Research Initiation Grant and the American Society of Engineering Education F.E. Terman Award, and was selected as Engineer of the Year in 1994 by the Ft. Worth IEEE Section. He was appointed to the NAE Committee on Space Station in 1995 and to the IEEE Control Systems Society Board of Governors in 1996. In 1998 he was selected as an IEEE Control Systems Society Distinguished Lecturer. He is a Founding Member of the Board of Governors of the Mediterranean Control Association.
(Back to Top of Abstracts) (Back to Top of Authors)

Many of the problems in learning can be posed as an optimization one on underlying manifolds. While Euclidean space is commonly used due to its simplicity, intrinsic constraints of problems often lead to manifolds that are nonlinear in nature. For example, common to component analysis techniques (such as PCA, ICA, FDA, and so on), one often requires the components to be orthonormal and this requirement leads to Stiefel manifold. By exploiting the geometric structure of the manifolds, one can often design and derive efficient and effective algorithms.

The main goal of this tutorial is to give a systematic coverage of nonlinear manifolds in pattern recognition and image analysis areas. A novel aspect of this tutorial is that we will introduce the tools from differential geometry that are necessary to understand existing techniques and more importantly to build new ones. Another important feature of this tutorial is that we will use live demonstrations to show the effectiveness of such techniques for pattern recognition. For example, by using optimal component analysis, we can often improve recognition performance significantly. It is widely recognized that neural network performance often depends critically on the features that are used. Compared to other areas, systematic study on feature extraction and selection is very limited. These nonlinear manifold techniques provide solutions to some of these problems by optimization in the feature space, which is often nonlinear in nature. We believe this tutorial on nonlinear manifolds is very timely and should be very useful for reseachers in pattern recognition, computer vision, machine learning, and other related areas.

This tutorial will consist of four main parts (see http://www.cavis.fsu.edu/manifold-tutorials/). In the first part, we will define nonlinear manifolds, and give examples to motivate the importance of nonlinear manifolds especially in the pattern recognition and image analysis areas. In the second part, we will cover the mathematical tools from differential geometry for optimization on nonlinear manifolds, including differentiable manifolds, examples of manifolds, tangent vectors, tangent spaces, examples of tangent vectors and tangent spaces, differential geometry of manifolds, geodesics, numerical techniques for calculating geodesics, and optimization on nonlinear manifolds. In the third part, we will extend usual techniques from statistics onto nonlinear manifolds, including probability distributions on manifolds, intrinsic means and covariances, sampling and estimation on manifolds, and learning and hypothesis testing on manifolds. In the last part, we will give two concrete examples that are important for many pattern recognition and image analysis applications: optimal component analysis, and shape inference and clustering on shape manifolds. We will give live demonstrations using techniques presented in this tutorial.

Keywords: Nonlinear manifolds, optimal component analysis, pattern recognition, feature extraction, statistics on nonlinear manifolds, image analysis, Grassman manifold, and Stiefel manifold.

Vita: Dr. Xiuwen Liu is a computer scientist with research interests in computer vision, biologically plausible principles for visual modeling, and image analysis. He has developed and implemented distinctive computational texture models that provide quantitative explanations for texture perception as well as successful applications in texture synthesis, image segmentation, object recognition, and terrain classification. His work with Dr. Srivastava is the first to computationally derive sparse filters that are optimal for recognition. His expertise in efficient computing has helped translate many theoretical ideas into efficient practical algorithms. He is a co-director of the Center for Applied Vision and Imaging Sciences (CAVIS) at the Florida State University.

Dr. Anuj Srivastava has expertise in the areas of statistical image analysis, signal processing, and statistical inferences on nonlinear manifolds. He has worked on applications of automated target recognition (ATR), derived performance bounds for statistical ATR, derived new probability models for natural images, derived nonlinear filtering algorithms on Lie groups and their quotient spaces, and is currently working on algorithms for statistical shape analysis. He directs the Center for Applied Vision and Imaging Sciences (CAVIS) at the Florida State University.

Dr. Washington Mio is a geometric topologist with expertise in the areas of topology and geometry of manifolds, who is currently working on problems in computer vision and pattern recognition. Dr. Mio’s present investigations include the algorithmic study of the differential geometry of various spaces of curves, energy functionals on these spaces, and applications to shape and image analysis. His previous research includes the study of the geometric topology of manifolds and generalized manifolds, group actions on these spaces, knot theory, and submanifold and map transversality. Dr. Mio is a co-director of CAVIS.
(Back to Top of Abstracts) (Back to Top of Authors)

Integrating Language and Cognition: New Results in Computational Intelligence

During the last decades computational intelligence methods have being developed taking advantage of the known mechanisms of the mind and brain. Much is known; still the knowledge is far from complete. Among the big unknown are relationships between cognition and language: do we think with words? Or are words just labels used at the end of the thinking process? The talk reviews the known mechanisms of cognition and suggests a computational mechanism that the mind might use for integrating cognition and language. Mathematics is related to results of other sciences studying the mind and brain. Mathematical descriptions of conscious and unconscious, imagination, symbols, concepts and emotions, including aesthetic emotion will be briefly discussed. The talk will review recent results in origin and evolution of language. Applications to recognition, fusion, and Internet search engines based on language understanding will be discussed.

The course focuses on the current understanding of the fundamental principles of working of the mind in the areas of cognition and language, their computational implementations, and practical applications. The course discusses the mind mechanisms, including concepts, emotions, instincts, behavior, language, cognition, understanding, thinking, intuitions, conscious and unconscious, abilities for formation of symbols and aesthetic feelings. Computational techniques are given for these mechanisms and abilities. The goals of the course are: First, to provide a basic mathematical understanding of the working of the mind. Second, to demonstrate practical applications of these mechanisms for pattern recognition, tracking, fusion, search engines, and for integrated systems combining sensor signals and communication data. Third, to outline future research directions. Historical and current difficulties in developing intelligent systems (IS) and applications will be discussed along with how the mind and new computational techniques overcome these difficulties. By the end of the course, students will be familiar with several general applications addressed by IS, especially, smart search engines, computational difficulties encountered over fifty years, and basic novel approaches to overcoming these difficulties.

Targeted for: Individuals interested in the development and application of intelligent systems and intelligent signal processing, especially in the area of smart search engines based on language understanding. This course assumes a basic understanding of theory of probability.

Keywords: mind, brain, concepts, emotions, instincts, cognition, language, recognition, tracking, fusion, search engines

Vita: Dr. Leonid Perlovsky is Principal Research Physicist and Technical Advisor at the Air Force Research Laboratory. Previously, from 1985 to 1999, he served as Chief Scientist at Nichols Research, a $0.5 B high-tech organization, leading the corporate research in information science, intelligent systems, neural networks, optimization, sensor fusion, and algorithm development. In the past he served as professor at Novosibirsk University and New York University. He participated as a principal in commercial startups developing tools for text understanding, search engines, biotechnology, and financial predictions. He published about 50 papers in refereed scientific journals and about 180 papers in conferences, delivered invited keynote plenary talks and authored a book "Neural Networks and Intellect: model-based concepts", Oxford University Press, 2001. Dr. Perlovsky serves on IEEE Computational Intelligence Technical Committee, Computational Intelligence Society Multimedia Tutorial Committee, Chair IEEE Boston Computational Intelligence Chapter, IEEE World Congress on Computation Intelligence Operational Committee, Program Chair for IEEE International Conference on Computational Intelligence Measurement, General Chair for IEEE KIMAS conference, and as Editor-in-Chief for an Elsevier journal “Physics of Life Reviews”.
(Back to Top of Abstracts) (Back to Top of Authors)

The objectives of the proposed tutorial are to provide artificial neural networks notions and to show how this intelligent computational tool is related to neuroscience. Why current artificial neural network models are so distant from biological ones? Existing connectionist mathematical models are compared with biological neurons and networks and the main differences are stressed. Desired characteristics in biologically plausible artificial neural networks and their corresponding elements in cerebral cortex are discussed. Neuroscience basic concepts about neuron physiology, its connections and the physical chemistry of central nervous system are approached. Several biological features are being taken into account in order to achieve new models that restore the artificial neural systems first concerns. Connectionist models based on neuroscience are about to be considered the next generation of artificial neural networks, inasmuch as nowadays models are far from biology, mainly for mathematical simplicity reasons. The tutorial encompasses the biological neuron, chemical and electrical synapses, the roles of transmitters and receptors, and the neuromodulators of the synaptic function. How the brain became a model to such machine learning technique? Conventional and biologically plausible connectionist algorithms are presented. Why backpropagation, the most used supervised connectionist algorithm, is considered biologically implausible? Some connectionist architectures are taken into account: feedforward, recurrent, and bi-directional. The latter structure is considered more physiologically realistic. A number of applications are presented. They show that the alleged biologically plausible artificial neural networks are even more computationally efficient than conventional ones.

Keywords: Artificial neural networks, Biologically plausible connectionist architectures, Biologically plausible connectionist algorithms, Computational neuroscience.

Vita: João Luís Garcia Rosa is Titular Professor of Pontifical Catholic University of Campinas (PUC-Campinas), São Paulo, Brazil. He got his doctorate in Computational Linguistics from Linguistics Department, Language Studies Institute, State University of Campinas (Unicamp), Sao Paulo, Brazil, in 1999, and his master's degree in Electrical Engineering (Artificial Intelligence), from Department of Computer Engineering and Industrial Automation, School of Electrical and Computer Engineering, State University of Campinas in 1993. He was graduated in Electrical Engineering (Automation and Electronics) from the latter department in 1983. His research interests are: Artificial Neural Networks, Natural Language Processing, Computational Psycholinguistics, and Biologically Plausible Connectionist Models. His academic experience: taught postgraduate level courses for masters level on computing systems: Intelligent Systems, Artificial Neural Networks, and Natural Language Processing disciplines, and taught graduate level courses for Computer Engineering and System Analysis: Programming Languages, Artificial Intelligence, and Formal Languages and Automata disciplines, at Pontifical Catholic University of Campinas (PUC-Campinas), Campinas, Sao Paulo, Brazil. Since 1998, he contributes to the field of Biologically Plausible Artificial Neural Networks with published papers and supervision of undergraduate and graduate students.
(Back to Top of Abstracts) (Back to Top of Authors)

The recent technological advances are producing huge data sets in almost all fields of scientific research, from astronomy to genetics. Although each research field often requires ad-hoc, fine tuned, procedures to properly exploit all the available information inherently present in the data, there is an urgent need for a new generation of general computational theories and tools capable to boost most human activities of data analysis. Traditional data analysis methods, in fact, are inadequate to cope with such exponential growth in the data volume and especially in the data complexity (ten or hundreds of dimensions of the parameter space). Among the data mining methodologies, visualization plays a key role in developing good models for data especially when the quantity of data is large. For a scientist, i.e. the expert in a specific domain, is essential the need for a visual environment that facilitates exploring high-dimensional data dependent on many parameters.

Data visualization is an important means of extracting useful information from large quantities of raw data. The human eye and brain together make a formidable pattern detection tool, but for them to work the data must be represented in a low-dimensional space, usually two or three dimensions. Even quite simple relationship can seem very obscure when the data is presented in tabular form, but are often very easy to see by visual inspection.

Many algorithms for data visualization have been proposed by both neural computing and statistics communities, most of which are based on a projection of the data onto a two or three dimensional visualization space. This tutorial embraces a number of these visualization techniques both linear and nonlinear:

• Principal Component Analysis (PCA), a classical linear projection method for mapping the data to a lower dimensional space. PCA does this with a linear transformation that preserves as much data variance as possible.

• Probabilistic PCA. The disadvantage of PCA is that it does not define a generative model and indeed there is no principled interpretation of its error function. However classical PCA can be made into a density model by using a latent variable approach, derived from factor analysis, i.e. Probabilistic Principal Component Analysis (PPCA). PPCA is a Gaussian mixture model in which each Gaussian has a covariance matrix which is diagonal and has a single variance parameter to describe all the variance in each component.

• Mixture of PPCA. Because PCA only defines a linear projection of data, it is a rather limited technique. One way around this is to use a collection of local linear models. The attraction of this is that each model is simpler to understand and usually easier to fit. PCA, PPCA and mixture of PPCA are appropriate when the data is linear or approximately piece-wise linear. An alternative approach is to use global nonlinear methods:

• Self Organizing Maps (SOM), a neural network algorithm based on a competitive learning which summarizes a set of data vectors in a high-dimensional space by a set of reference vectors organized on a lower dimensional sheet (usually two dimensional). SOM has been used for a wide variety of applications thanks to its simplicity and for its several plotting options. Although SOM provides easy of computation and powerful visualizations it, indeed, does not define any density model and suffers of other drawbacks which can be overcame employing nonlinear latent variable models;

• Generative Topographic Mapping (GTM), a nonlinear latent variable model in which the density is a Gaussian mixture model. The centre of each Gaussian is constrained to lie on a smooth manifold of low dimension (usually 1 or 2) but all Gaussians have a common spherical covariance;

• Probabilistic Principal Surfaces (PPS), a generalization of GTM with oriented covariance. PPS allows a spherical manifold in 3D space which is better suitable to characterize high-dimensional data which usually is sparse and located at the periphery (curse of dimensionality). While such algorithms can usefully display the structure of simple and more complex data sets, they often prove inadequate in the face of data sets which are too complex. A single two or three dimensional projection, even if it is nonlinear, may be insufficient to capture all the interesting aspects of the data set. For example, the projection which best separates two clusters may not be the best for revealing internal structure within one of the clusters. This motivates the consideration of a hierarchical model involving multiple two or three dimensional visualization spaces. The goal is that the top-level projection should display the entire data set, perhaps revealing the presence of clusters, while lower-level projections display internal structure within individual clusters, such as the presence of subclusters, which might not be apparent in higher-level projections. Therefore, the tutorial reviewes hierarchical linear (based on mixture of PPCA) and nonlinear (based on GTM) latent variable models and concludes by illustrating a new proposed hierarchical model based on Probabilistic Principal Surfaces.

Vita: Antonino Staiano is a post-doctoral research fellow at the Department of Mathematics and Computer Science, University of Salerno, Italy. He got his PhD in Computer Science at University of Salerno in 2003, discussing a thesis on unsupervised neural network models for the extraction of scientific information from astronomical databases. Since October 2000 he has been working within the AstroNeural collaboration project (a joint project together with the Department of Physical Science, University Federico II of Naples) to the development of neural -based tools for data visualization and clustering for astronomical and genetic data mining. He has worked at European Southern Observatory (ESO) - Space Telescope European Coordinating Facility and is currently involved in the European Virtual Observatory. Fields of interest and research: Data Mining, Data Visualization, Statistical Pattern Recognition, Bioinformatics, Machine Learning, Ensemble of learning systems, Evolutionary Algorithms.

Roberto Tagliaferri is Associate Professor in Computer Science and Neural Networks at the University of Salerno. His research covers the area of neural nets: neural dynamics, fuzzy neural nets, applications to signal and image processing with astronomical and geological data, industrial and medical computer aided diagnosis. Co-chairman of special sessions at the AMSE-ISIS’97, at IJCNN’99, IJCNN 2001, IJCNN 2003, WILF 2003, IJCNN 2004 and co-editor of a special issue of Neural Networks. He had a tutorial on Learning with multiple machines: ECOC models vs Bayesian Framework at IJCNN 2003. He is the author or the co-author of more than 80 publications in the area of neural networks. Since 1995, co-editor of the Proceedings of the Italian Workshops on Neural Nets (WIRN). Secretary of SIREN (Società Italiana Reti Neuroniche) and vice-president of the Italian SIG of the INNS. Member of the Director Council of the IIASS (International Institute for Advanced Scientific Studies) "E. R. Caianiello". Senior member of the IEEE, member of INFN, INFM and AIIA.
(Back to Top of Abstracts) (Back to Top of Authors)

The use of kernel based learning techniques has become increasingly popular, largely stimulated by the work on support vector machines as originally introduced within the context of statistical learning theory. Standard support vector machines for classification and regression lead to solving convex optimization problems. The problem formulation and solution is characterized by a primal and dual problem where multilayer neural network interpretations can be given in both worlds. In contrast with many classical models, support vector machines are able to learn and generalize in huge dimensional input spaces.

The method makes use of a high dimensional feature map (which should not be explicitly known) in relation to a Mercer kernel (the so-called kernel trick). Notions as large margin classification and regularization techniques play an important role at this point. The method can also be viewed as a functional optimization problem in reproducing kernel Hilbert spaces (RKHS).

The kernel trick has been further applied to a wider variety of problems such in kernel Fisher discriminant analysis (KFDA), kernel principal component analysis (KPCA), kernel canonical correlation analysis (KCCA), kernel partial least squares (KPLS), kernel clustering and others. Optimization formulations with primal and dual problem has been given with least-squares support vector machines (LS-SVM), thereby extending support vector machine methodology to a wider range of problems for regression, classification, supervised and unsupervised learning, recurrent networks and control.

One often has the choice to either solve the primal or the dual problem depending on the dimensionality of the feature map (or its approximation) and select the most suitable representation for the given problem at hand (e.g. high dimensional input space or large data sets). Kernels have also been customized towards specific application areas such as in textmining, bioinformatics or in relation to graphical models. Novel techniques of hierarchical kernel machines even allow to find the model together with the tuning parameters by solving a convex problem. At the same time one can achieve in this way e.g. sparse representations, stability of learning machines and input selection.

In this tutorial we outline the main concepts of support vector machines and kernel based learning, show interesting research directions and illustrate the methods with many successful real-life examples in different areas such as e.g. time-series prediction, nonlinear modelling and microarray data analysis.

Keywords: Support vector machines, kernel based learning, convex optimization, primal and dual problem, supervised and unsupervised learning

Vita: Johan A.K. Suykens is an associate professor with K. U. Leuven Belgium. His research interests are mainly in the areas of the theory and application of neural networks and nonlinear systems. He is author of the books "Artificial Neural Networks for Modelling and Control of Non-linear Systems" (Kluwer Academic Publishers) and "Least Squares Support Vector Machines" (World Scientific) and editor of the books "Nonlinear Modeling: Advanced Black-Box Techniques" (Kluwer Academic Publishers) and "Advances in Learning Theory: Methods, Models and Applications" (IOS Press). In 1998 he organized an International Workshop on Nonlinear Modelling with Time-series Prediction Competition. He is a Senior IEEE member and has served as associate editor for the IEEE Transactions on Circuits and Systems - Part I (1997-1999) and Part II (since 2004) and since 1998 he is serving as associate editor for the IEEE Transactions on Neural Networks. He received an IEEE Signal Processing Society 1999 Best Paper Senior Award and several Best Paper Awards at International Conferences. He is a recipient of the International Neural Networks Society INNS 2000 Young Investigator Award for significant contributions in the field of neural networks. He has served as Director and Organizer of a NATO Advanced Study Institute on Learning Theory and Practice (Leuven 2002) and as a program co-chair for the International Joint Conference on Neural Networks IJCNN 2004.
(Back to Top of Abstracts) (Back to Top of Authors)

The tutorial presents unsupervised learning artificial neural networks (ANN), which yields unbiased computational intelligence (CI), information acquisition, feature extraction and knowledge discovery. Their applications in bioinformatics, brain study, unmanned robot control, finance and business are briefly presented.

The necessary and sufficient conditions are that (1) all CPU Brain is always on all the time and is kept at an isothermal equilibrium temperature 37oC and (2) each CPU Brain is equipped with power of 5 sensors pairs e.g. two eyes, two ears, etc. altogether 10 dimensional vector time series for input data space. The overwhelming incoming excitations must not be replenished, a natural choice is letting fluctuating disagreements so-called noise decay away, and only reinforce coincident signals so-called features. This biomimetics is our minimum free energy principle of unsupervised feature extraction & fusion methodology. The tutorial proves these conditions (1) (2) constituted the basis of unsupervised learning ANN and real world applications in remote sensing and early tumor detection are given for computer scientists, neuroscientists, biologist, engineers and graduate students.

Keywords: Unsupervised Learning, Computational Intelligence, Knowledge-based neural networks, Computational Neurogenetic modeling.

Vita: Harold Szu received in 1971 Ph. D. in Physics/Statistical Mechanics from Prof. George Uhlenbeck at the Rockefeller University, New York, N.Y. and worked at Naval Research Lab Wash DC in three Divisions, and became the Information Science Group Leader of Naval Surface Warfare Center at White Oak (1990-1996) and at Dahlgren VA(1997-now).

He has been a Fellow of SPIE (since 1995) for neural nets, Fellow of OSA (1996) for adaptive wavelets, Fellow of IEEE (1997) for bi-sensor fusion, Fellow of Am. Inst. Med. & Bio. Eng. 2004 for medical image diagnoses. Dr. Szu is a Champaign of brain-style computing for decades, a founder, former president, and a current governor of International Neural Network Society (INNS), He has contributed to the establishment of 30 SIG/Chapter worldwide. Notably, he has scientifically contributed to the unsupervised redundancy reduction of sensory pair fusion by postulating the thermodynamic free energy for learning. He received INNS D. Gabor Award, in 1997, “for outstanding contribution to neural network applications in information sciences and pioneer implementations of fast simulated annealing search," and Italy Academy the Eduardo R. Caianiello Award, 1999, for “elucidating and implementing a chaotic neural net as a dynamic realization for fuzzy logic membership function." Dr. Szu is a foreign academician of Russian Academy of Nonlinear Sciences in 1999 for the homeostasis unsupervised learning based on constant Cybernetic Temperature.
(Back to Top of Abstracts) (Back to Top of Authors)

Evangelia Micheli-Tzanakou, PhD
Department of Biomedical Engineering, RUTGERS UNIV.

One of the major problems a researcher faces is what is learned from data obtained by various methods and different techniques. This tutorial will discuss and compare topics such as: Statistical Advances and Challenges, as well as Feature Extraction in Computational Intelligence methodologies.

Often a simple model describes the data well, simply because the S/N ratio is too small for detection of more complex structures-which for example is the case with medical data involving human subjects. One has a lot of variability both in intra- and inter-sets of data.

Some important Simple Tools that have been used for a long time are: Linear Regression, Discriminant analysis, Principal Component Analysis etc. In all of these, the size of the data set matters. Huge data sets create memory problems. The question is how do we handle different data types and how do we handle them? What if the data are correlated? What if we have complex data structures?

In this tutorial you will learn more about Computational Intelligence, how to get to know your data and how to do Feature Extraction.

Some examples of "features" will be given and different feature extraction methods will be discussed.

Vita: Dr. Tzanakou is Professor II and director of the Computational Intelligence Laboratories (2000-present); Chair of Biomedical Engineering Dept at Rutgers University (1990-2000);established the BME Undergraduate curriculum in 1999. Has an active research group and has published over 250 scientific papers. Author/co-author of 2 books. Has graduated 37 MS and PhDs.

Fellowships: IEEE Fellow (1992);Founding Fellow, American Institute for Medical & Biological Eng. (AIMBE), 1993; Member of Sigma Xi and Eta Kappa Nu; Listed in American Men and Women in Science; Honorary Member of the British Brain Research Association; Honorary Member of the European Brain and Behavior Society.

Awards: NJ Women of Achievement Award, 1995; Featured in the “Notable Twentieth Century Scientists, 1994; Achievement Award Recipient, Society of Women Engineers, 1992; Outstanding IEEE Branch Advisor/Counselor Award, 1985 ; cited as a Pioneer in the IEEE Web page (http://www.ieee.org/innovators).

Editorships: Editorial Board, IEEE Trans. On Nano-BioScience (2002-present); Editorial Board Biomedical Engineering On Line (2001-present ); Associate Editor, IEEE Transactions on Neural Networks (2000-present); Book Series Editor in Biomedical Engineering, Plenum/Klewer (1999-present); Book Series Editor, Biomedical Systems, IES Book series, CRC Press (1997-present); Editorial Board, IEEE Trans. Biomedical Information Tech.(1997-2001); Editorial Board, International J. On Advanced Computational Intelligence (1997-1999); Associate Editor, IEEE Transactions on Neural Networks (1989-1992).
(Back to Top of Abstracts) (Back to Top of Authors)

Neural Networks That Actually Work In Prediction and Decision/Control: Common Misconceptions Versus Real-World Success

Neural nets are used in a growing variety of real-world applications, in a growing number of areas (including the modeling of natural intelligence). But this has been a mixed blessing, as various subcultures have not kept up with each other. For example, at http://www.eas.asu.edu/~nsfadp/ applications of adaptive dynamic programming (aka advanced reinforcement learning) are reported which beat the previous best performance in electric power grid control, aircraft control, in large-scale logistics, etc. -- but best performance requires integrating concepts from many subcultures. Yet in some subcultures, people are still using "mappings from sensory to motor coordinates" (direct adaptive control), whose limits were clear years ago. Some are using indirect methods inspired by control theory with well-known stability theorems -- theorems whose conditions are rarely satisfied, and methods which are easily improved upon. Similarly, in prediction, some subcultures falsely believe that recurrent networks are hard to train and of marginal benefit, even as people in industry use them routinely, and report performance better than extended Kalman filters and equal (at lower cost) to elaborate particle filter methods. The foundations of learning also bear on the practical choices.

This tutorial will try to offer a kind of practical roadmap of these issues, and suggest how it points towards future functionality in tasks as large-scale as what brains can handle.

Vita: Paul J. Werbos is best known as the original inventor of backpropagation, as part of his Harvard PhD thesis, which was reprinted in full in his book the Roots of Backpropagation, Wiley 1994, along with his classic 1990 tutorial on backpropagation through time for Proc IEEE. Also, his 1967 article in Cybernetica first proposed the idea of approximating dynamic programming as a way to improve reinforcement learning, elaborated in many later papers. He was one of the three original two-year presidents of the International Neural Network Society, and winner of the IEEE Pioneer Award. He is Program Director for Control, Networks and Computational Intelligence at NSF, which actively seeks more proposals in this area. He has also been active in many cross-cutting funding initiatives; for example, he serves on the Working Group for energy storage and distribution of the interagency Climate Change Technology Program, and coordinated the NASA-NSF-EPRI solicitation on space solar power (NSF 02-098). He is also on the Planning Committee of the Millennium Project of the United Nations University (http://millennium-project.org), and has published a few papers on quantum foundations and technology (see arXiv.org, physics and nonlinear systems). He also has two degrees in economics from Harvard and the London School of Economics.

Regarding the workings of the human mind, memory and pattern recognition seem to be intertwined. You generally do not have one without the other. Taking inspiration from life experience, a new form of computer memory has been devised. It has been used successfully in diverse applications such as visual aircraft identification, aircraft navigation, and human facial recognition. Other uses are being explored. The basic idea will have many new areas of application.

Vita: Bernard Widrow received the S.B., S.M., and Sc.D. degrees in Electrical Engineering from the Massachusetts Institute of Technology in 1951, 1953, and 1956, respectively. He joined the MIT faculty and taught there from 1956 to 1959. In 1959, he joined the faculty of Stanford University, where he is currently Professor of Electrical Engineering.

He began research on adaptive filters, learning processes, and artificial neural models in 1957. Together with M.E. Hoff, Jr., his first doctoral student at Stanford, he invented with LMS algorithm on in the Autumn of 1959. Today, this is the world's most widely used learning algorithm. He has continued working on adaptive signal processing, adaptive controls, and neural networks since that time.

Dr. Widrow is a Life Fellow of the IEEE and a Fellow of AAAS. He received the IEEE Centennial Medal in 1984, the IEEE Alexander Graham Bell Medal in 1986, the IEEE Signal Processing Society Medal in 1986, the IEEE Neural Networks Pioneer Medal in 1991, the IEEE Millennium Medal in 2000, and the Benjamin Franklin Medal for Engineering from the Franklin Institute of Philadelphia in 2001. He was inducted into the National Academy of Engineering in 1995 and into the Silicon Valley Engineering Council Hall of Fame in 1999.

(Click on tutorial title to access the abstract and the author's vita.
Dario Floreano ”Evolutionary Robotics" Paul J. Werbos "Neural Networks That Actually Work In Prediction and Decision/Control: Common Misconceptions Versus Real-World Success" Frank Lewis "Neural Networks for Dynamic Systems Feedback Control" Bernard Widrow "Cognitive Memory" (Back to Top)	E. Tzanakou "Feature extraction in Computational Intelligence" Harold Szu "Unsupervised Learning" X. Liu, A. Srivastava & W. Mio "Nonlinear Manifolds in Pattern Recognition and Image Analysis" (Back to Top)	Pierre Baldi "Bioinfomatics and Machine Learning: the Prediction of Protein Structures on a Genomic Scale" Leonid Perlovsky "Integrating Language and Cognition: New Results in Computational Intelligence" Vladimir Cherkassky "New Formulations for Predictive Learning" Johan A. K. Suykens "Support Vector Machines and Kernel Based Learning" (Back to Top)	Nik Kasabov "Evolving Connectionist Systems: Biological Principles, Models and Applications A. Staiano & R. Tagliaferri "Data Visualization of High Dimensional Scientific Data" Joao Luis Garcia Rosa "Biologically Plausible Artificial Neural Networks" (Back to Top)
Abstracts and Vitas	(Back to Top of Authors)
Bioinformatics and Machine Learning: the Prediction of Protein Structures on a Genomic Scale Pierre Baldi University of California, Irvine This tutorial focuses on the application of machine learning methods, and in particular of neural networks, to problems in protein structure prediction. The tutorial first addresses the general problem of how to apply neural networks techniques to variable-size structured data, such a sequences, trees, and graphs in bioinformatics and other applications. The tutorial covers the design of feedforward and recursive neural network architectures for the prediction of protein structural features, such as secondary structures, relative solvent accessibility, disulphide bridges, and contact maps. Simulations and state-of-the-art results are presented and discussed together with more general issues, such as tradeoffs and integration with other methods, and applications to other related problems. Keywords: neural networks, recurssive neural networks, bioinformatics, sequence analysis, protein, protein structure prediction, structural proteomic Vita: Pierre Baldi is a Professor in the School of Information and Computer Sciences and the Department of Biological Chemistry at the University of California, Irvine and the Director of the Institute for Genomics and Bioinformatics at UCI. Born and raised in Europe, he received his PhD from the California Institute of Technology in 1986. From 1986 to 1988 he was a postdoctoral fellow at the University of California, San Diego. From 1988 to 1995 he held faculty and member of the technical staff positions at the California Institute of Technology and at the Jet Propulsion Laboratory. He was CEO of a startup company from 1995 to 1999 and joined UCI in 1999. He is the recipient of a 1993 Lew Allen Award at JPL and a Laurel Wilkening Faculty Innovation Award at UCI. Dr. Baldi has written over 100 research articles and four books: • Modeling the Internet and the Web--Probabilistic Methods and Algorithms, Wiley, (2003); • DNA Microarrays and Gene Regulation--From Experiments to Data Analysis and Modeling, Cambridge University Press, (2002); • The Shattered Self--The End of Evolution, MIT Press, (2001); • Bioinformatics: the Machine Learning Approach, MIT Press, Second Edition (2001). His research focuses in AI, machine learning, computational biology, and biological and chemical informatics. (Back to Top of Abstracts) New Formulations for Predictive Learning Vladimir Cherkassky University of Minnesota Most existing learning algorithms have been developed under standard formulations of the learning problem, such as classification or regression. However, many real-life applications motivate approaches that go beyond traditional learning methods. These new directions include new types of inference different from standard inductive inference (i.e., transduction), as well as new formulations of the learning problem. This tutorial will review standard learning formulations using the framework of Vapnik-Chervonenkis (VC) learning theory, and then discuss several possible ‘non-standard’ formulations. The tutorial consists of four parts: (1) Motivation for new learning formulations. We provide historical, philosophical and application-driven motivations for developing new learning formulations and new types of inference (as opposed to research on new learning algorithms) under general conceptual framework of Vapnik’s Statistical Learning Theory. Then we present standard inductive learning formulations, and critically review the assumptions underlying such formulations. Relaxing some of these assumptions leads to several ‘non-standard’ formulations as discussed below. (2) Non-inductive types of inference. These include non-inductive types of inference such as transduction and selection learning formulations. We describe these formulations, and several recent applications studies comparing inductive vs. transductive learning. (3) Application-driven formulations. We outline general framework for mapping application requirements onto a learning problem formulation, and then provide examples of such new application-driven formulations. Such examples range from financial engineering (trading) to identity fraud detection. (4) Multiple model estimation. All standard inductive learning formulations assume that all available (or training) data can be described by a single statistical model. For example, in classification setting the goal is to estimate a (single) decision boundary. Likewise, under regression formulation, the goal is to estimate a single target function from finite and noisy data samples. However, there are many applications which naturally lead to learning formulations where the goal is to estimate several models from available (finite) data. For example, in computer vision, a sequence of video frames may contain several moving objects (with overlapping trajectories), so the problem of multiple motion estimation can be viewed under multiple model estimation framework. I will describe a new learning formulation for such settings, and then introduce SVM-based learning algorithms for this new formulation. Finally, I will discuss methodological issues arising in using learning methods for various real-life applications, such as: - clear separation between the learning problem formulation and the solution approach (e.g, learning algorithm); - importance of mapping application requirements onto an appropriate problem formulation, rather than inventing/ reinventing new learning algorithms; - empirical comparisons of learning algorithms. INTENDED AUDIENCE: researchers and practitioners interested in understanding new learning formulations motivated by practical application requirements, rather than standard formulations motivated by mathematical modeling considerations. OTHER CONSIDERATIONS: the proposed tutorial is based on two plenary talks (on the same subject of New Learning Problem Formulations) given by the instructor at ICANN-01 in Vienna, Austria and ANNIE-04 in St Louis, Missouri. Keywords: Inductive principle, inductive inference, neuroimaging, new learning formulations, Popper's falsifiability, regularization, statistical learning, support vector machines, transduction, Vapnik-Chervonenkis theory Vita: Vladimir Cherkassky is Professor of Electrical and Computer Engineering at the University of Minnesota. He received Ph.D. in Electrical Engineering from University of Texas at Austin in 1985. His current research is on methods for predictive learning from data, and he has written a monograph Learning From Data published by Wiley in 1998. Prof. Cherkassky has served on the Governing Board of INNS. He has served on editorial boards of IEEE Transactions on Neural Networks, the Neural Networks Journal, the Natural Computing Journal and the Neural Processing Letters. He served on the program committee of major international conferences on Artificial Neural Networks. He was Director of NATO Advanced Study Institute (ASI) From Statistics to Neural Networks: Theory and Pattern Recognition Applications held in France, in 1993. He has presented numerous tutorials on neural network and statistical methods for learning from data. (Back to Top of Abstracts) (Back to Top of Authors) Evolutionary Robotics Dario Floreano EPFL, Switzerland Evolutionary Robotics is the artificial evolution of control systems for autonomous robots. Most of the past and ongoing research in Evolutionary Robotics is concerned with evolution of neural controllers for which available learning algorithms are not suitable. The tutorial will put special emphasis on how Evolutionary Robotics can be used to study open questions in evolutionary biology, neurophysiology, and cognitive science. A short introduction to artificial evolution will allow participants to fully understand and replicate most of the experiments by using freely available software. An highlight of the topics: Evolution of simple navigation skills. The principle of parsimony: Evolution of complex behaviours using simple reactive mechanisms. The role of fitness functions. Evolution of vision-based navigation. The role of the environment in the evolutionary process and the emergence of spatial maps. Co-evolution of active vision and feature sensitivity for image recognition, robot navigation, and car driving. Interactions between evolution and learning. Evolution of learning rules and analysis of evolved neural controllers. Competitive co-evolution in biology and robotics. The case of predator-prey robots. Introducing learning in co-evolutionary competitive systems. Evolution of spiking control networks and other exotic neural controllers. All topics will be illustrated with several examples of robots with wheels, legs, wings. Keywords: mobile robots, evolutionary systems, autonomous systems, time-dependent networks, control systems Vita: DARIO FLOREANO is Professor of Intelligent Systems at the Swiss Federal Institute of Technology in Lausanne (EPFL) where he is director of the Institute of Systems Engineering. His research activities include embodied and embedded neural networks, artificial evolution, robotics, bio-mimetic electronics, self-organizing systems, and artificial life. He held senior research positions at the National Research Council in Roma, at the University of Stirling, at the Swiss Federal Institute of Technology in Lausanne, and at Sony Computer Science Laboratory in Tokyo. Dario published more than 100 peer-reviewed papers, authored 2 books, and edited 3 other books. His book with S. Nolfi, Evolutionary Robotics, was reprinted by MIT Press three times over the last four years. He co-organized 6 international conferences and joined the program committee of more than 50 other conferences. He is on the editorial board of the journals Neural Networks, Genetic Programming and Evolvable Machines, Adaptive Behavior, Artificial Life, Connection Science, IEEE Transactions on Evolutionary Computation, and Autonomous Robots. He is co-founder and member of the Board of Directors of the International Society for Artificial Life, Inc. and member of the Board of Governors of the International Society for Neural Networks.(Back to Top of Abstracts) (Back to Top of Authors) Evolving Connectionist Systems: Biological Principles, Models and Applications Prof. Nik Kasabov (nkasabov@aut.ac.nz) Knowledge Engineering and Discovery Research Institute (KEDRI), http://www.kedri.info, Auckland University of Technology, New Zealand Evolving connectionist systems (ECOS) is a general paradigm of neural systems that evolve their structure, their functionality and their internal knowledge representation through continuous learning from data and interaction with the environment. The learning process can be: on-line, off-line, incremental, supervised, unsupervised, active, sleep/dream, etc. ECOS are inspired by the evolving processes in the brain. These processes are self-organized and are governed by both external stimuli (data) and internal parameters - genes. The tutorial consists of three parts, each presented in 35 minutes with a break of 15 min. after the first two parts. Part one presents biological principles of evolving brains, that include: synaptic changes through learning, neuron creation, brain development, the role of genes for brain functions and malfunctions, gene interaction networks, the role of evolution. Part two presents several ECOS models, that include both simple ECOS and computational neurogenetic models (CNGM) explained and illustrated through simulations and examples. Simple ECOS models are presented in [N.Kasabov, Evolving connectionist systems: Methods and Applications in Bioinformatics, Brain study and intelligent machines, Springer, 2002, www.springer.de)]. They are adaptive neural systems that learn on-line and facilitate rule extraction. CNGM are connectionist models that integrate neural networks and gene interaction networks. Interaction of genes in every neuron affects the dynamics of this neuron and may influence the whole neural network performance which changes as a function of gene expression. Through optimization of the gene interaction networks, the initial gene/protein expression values and the CNGM parameters, particular target states of the CNGM can be achieved. This is illustrated by means of a simple neurogenetic model of a spiking neural network (SNN). The behaviour of the SNN is evaluated by means of the local field potential, thus making it possible to attempt modelling the role of genes in different brain states, where EEG data is available to test the model. Part three presents applications of ECOS models in bioinformatics, brain study, adaptive robot control, finance and business prediction, medical decision support, evolvable hardware, evolvable software, adaptive integrated speech, image and fingerprint recognition. The tutorial targets computer scientists, neuroscientists, biologists, engineers and graduate students. Keywords: Computational Intelligence, Neuroinformatics, Bioinformatics, Knowledge-based neural networks, Evolving connectionist systems, Gene regulatory networks, Computational neurogenetic modeling. Vita: Professor Nik Kasabov is the Founding Director and the Chief Scientist of the Knowledge Engineering and Discovery Research Institute KEDRI, Auckland (http://www.kedri.info/). He obtained a MSc and a PhD from the Technical University of Sofia. At present he holds a Chair of Knowledge Engineering at the School of Computer and Information Sciences at Auckland University of Technology. He is a Fellow of the Royal Society of New Zealand, Fellow of the New Zealand Computer Society, and a Senior Member of IEEE. Kasabov is the chair of the Adaptive Systems Task Force of the Neural Network Technical Committee of the IEEE Computational Intelligence Society. He is a member of the Board of Governors of the INNS and a board member of the Asia Pacific Neural Network Assembly (APNNA). His main research interests are in the areas of intelligent information systems, soft computing, neuro-computing, bioinformatics, brain study, speech and image processing, data mining and knowledge discovery, where he has published 10 books, 380 refereed papers, 25 patents and other work. He has an extensive academic experience, holding positions at the Technical University of Sofia- Bulgaria, University of Essex - UK, University of Otago -New Zealand. Kasabov is on the editorial boards of 8 international journals and has been on the PC of 50 international conferences in the last 5 years. He chaired the series of ANNES conferences (1993-2001). More information of Prof. Kasabov can be found on the Web site: http://www.kedri.info. (Back to Top of Abstracts) (Back to Top of Authors) Neural Networks for Dynamic Systems Feedback Control F. L. Lewis University of Texas at Arlington Feedback control systems occur throughout nature. Included is the biological cell, which uses feedback mechanisms to pump ions across its membrane to sustain its equilibrium, or homeostasis. Volterra showed that the population balance of predator and prey species of fish in a closed pond is due to a feedback mechanism. Darwin showed that feedback over long time periods is responsible for natural selection. Man-made feedback control systems make possible the reliable performance of all modern complex systems including aircraft, spacecraft, vehicle motor systems, submarine stabilization, and robotic industrial machines. Though Thomas Newcomen invented the steam engine in 1712, the Industrial Revolution started in earnest only after J. Watt invented the feedback control systems for the steam engine in 1786. In this tutorial workshop, we focus on Neural Networks for Feedback Control of Dynamical Systems, showing how NN can be tailored in topology for highly effective use in control systems design, and tailored in tuning of memory to provide stability and performance results that significantly extend what had been possible using standard control theory approaches. The issue in feedback control theory is to provide: (1) rigorous mathematical stability proofs, without which results will not be accepted by the Control Systems Community, and (2) performance and robustness guarantees, without which results will not be accepted by industry. We will provide a background on dynamical systems properties, feedback control issues, and a roundup of NN properties that prove very useful for feedback control purposes. Relations will be drawn with fuzzy logic systems, which are nonlinear approximators based on the properties of human linguistic systems. Some feedback control topologies that use NN will be presented, including NN designs for feedback linearization, backstepping, force control, and singular perturbations. Techniques for stability and performance proofs in feedback control will be outlined. It is shown that NN designs significantly improve upon standard control theory techniques and allow improved robustness and performance for systems with nonlinearities, unknown dynamics and disturbances, and friction. Applications are to robotics, industrial systems, aircraft control, and vehicle motion platforms. Most naturally occurring feedback systems are based on optimality or minimality principles to conserve energy, fuel, or effort. Therefore, during the latter portion of the tutorial we will show how NN can be used for optimal control systems using designs based on the Hamilton-Jacobi equations. Both H-2 and H-infinity robust design will be treated. Connections will be drawn with Approximate Dynamic Programming techniques. Keywords: Feedback control using neural networks, Dynamical neural network control, Nonlinear adaptive control systems Vita: Dr. F.L. Lewis is a University Distinguished Scholar Professor and the Moncrief-O'Donnell Endowed Chair at The University of Texas at Arlington. His current interests include intelligent control, neural and fuzzy systems, MEMS control, wireless sensor networks, nonlinear systems, robotics, condition-based maintenance, and manufacturing process control. He is the author/co-author of 4 U.S. patents, 150 journal papers, 240 refereed conference papers, nine books including Optimal Control, Optimal Estimation, Applied Optimal Control and Estimation, Aircraft Control and Simulation, Control of Robot Manipulators, Neural Network Control, High-Level Feedback Control with Neural Networks and the IEEE reprint volume Robot Control. He is a Fellow of the IEEE, a Fulbright Awardee, a member of the New York Academy of Sciences, and a registered Professional Engineer in the State of Texas. He has served as Visiting Professor at Democritus University in Greece, Hong Kong University of Science and Technology, Chinese University of Hong Kong, National University of Singapore. He is an elected Guest Consulting Professor at both Shanghai Jiao Tong University and South China University of Technology. He is the recipient of an NSF Research Initiation Grant and the American Society of Engineering Education F.E. Terman Award, and was selected as Engineer of the Year in 1994 by the Ft. Worth IEEE Section. He was appointed to the NAE Committee on Space Station in 1995 and to the IEEE Control Systems Society Board of Governors in 1996. In 1998 he was selected as an IEEE Control Systems Society Distinguished Lecturer. He is a Founding Member of the Board of Governors of the Mediterranean Control Association. (Back to Top of Abstracts) (Back to Top of Authors) .Nonlinear Manifolds in Patten Recognition and Image Analysis Xiuwen Liu, Anuj Srivastava and Washington Mio Dept. of Computer Science, Dept. of Statistics and Dept. of Mathematics, Florida State University Web site: http://www.cavis.fsu.edu/manifold-tutorials/ (A preliminary version of the presentation is available at http://cavis.fsu.edu/manifold-tutorials/manifolds-ijcnn2005.pdf). Many of the problems in learning can be posed as an optimization one on underlying manifolds. While Euclidean space is commonly used due to its simplicity, intrinsic constraints of problems often lead to manifolds that are nonlinear in nature. For example, common to component analysis techniques (such as PCA, ICA, FDA, and so on), one often requires the components to be orthonormal and this requirement leads to Stiefel manifold. By exploiting the geometric structure of the manifolds, one can often design and derive efficient and effective algorithms. The main goal of this tutorial is to give a systematic coverage of nonlinear manifolds in pattern recognition and image analysis areas. A novel aspect of this tutorial is that we will introduce the tools from differential geometry that are necessary to understand existing techniques and more importantly to build new ones. Another important feature of this tutorial is that we will use live demonstrations to show the effectiveness of such techniques for pattern recognition. For example, by using optimal component analysis, we can often improve recognition performance significantly. It is widely recognized that neural network performance often depends critically on the features that are used. Compared to other areas, systematic study on feature extraction and selection is very limited. These nonlinear manifold techniques provide solutions to some of these problems by optimization in the feature space, which is often nonlinear in nature. We believe this tutorial on nonlinear manifolds is very timely and should be very useful for reseachers in pattern recognition, computer vision, machine learning, and other related areas. This tutorial will consist of four main parts (see http://www.cavis.fsu.edu/manifold-tutorials/). In the first part, we will define nonlinear manifolds, and give examples to motivate the importance of nonlinear manifolds especially in the pattern recognition and image analysis areas. In the second part, we will cover the mathematical tools from differential geometry for optimization on nonlinear manifolds, including differentiable manifolds, examples of manifolds, tangent vectors, tangent spaces, examples of tangent vectors and tangent spaces, differential geometry of manifolds, geodesics, numerical techniques for calculating geodesics, and optimization on nonlinear manifolds. In the third part, we will extend usual techniques from statistics onto nonlinear manifolds, including probability distributions on manifolds, intrinsic means and covariances, sampling and estimation on manifolds, and learning and hypothesis testing on manifolds. In the last part, we will give two concrete examples that are important for many pattern recognition and image analysis applications: optimal component analysis, and shape inference and clustering on shape manifolds. We will give live demonstrations using techniques presented in this tutorial. Keywords: Nonlinear manifolds, optimal component analysis, pattern recognition, feature extraction, statistics on nonlinear manifolds, image analysis, Grassman manifold, and Stiefel manifold. Vita: Dr. Xiuwen Liu is a computer scientist with research interests in computer vision, biologically plausible principles for visual modeling, and image analysis. He has developed and implemented distinctive computational texture models that provide quantitative explanations for texture perception as well as successful applications in texture synthesis, image segmentation, object recognition, and terrain classification. His work with Dr. Srivastava is the first to computationally derive sparse filters that are optimal for recognition. His expertise in efficient computing has helped translate many theoretical ideas into efficient practical algorithms. He is a co-director of the Center for Applied Vision and Imaging Sciences (CAVIS) at the Florida State University. Dr. Anuj Srivastava has expertise in the areas of statistical image analysis, signal processing, and statistical inferences on nonlinear manifolds. He has worked on applications of automated target recognition (ATR), derived performance bounds for statistical ATR, derived new probability models for natural images, derived nonlinear filtering algorithms on Lie groups and their quotient spaces, and is currently working on algorithms for statistical shape analysis. He directs the Center for Applied Vision and Imaging Sciences (CAVIS) at the Florida State University. Dr. Washington Mio is a geometric topologist with expertise in the areas of topology and geometry of manifolds, who is currently working on problems in computer vision and pattern recognition. Dr. Mio’s present investigations include the algorithmic study of the differential geometry of various spaces of curves, energy functionals on these spaces, and applications to shape and image analysis. His previous research includes the study of the geometric topology of manifolds and generalized manifolds, group actions on these spaces, knot theory, and submanifold and map transversality. Dr. Mio is a co-director of CAVIS. (Back to Top of Abstracts) (Back to Top of Authors) Integrating Language and Cognition: New Results in Computational Intelligence Leonid Perlovsky, Ph.D. Hanscom AFB, MA During the last decades computational intelligence methods have being developed taking advantage of the known mechanisms of the mind and brain. Much is known; still the knowledge is far from complete. Among the big unknown are relationships between cognition and language: do we think with words? Or are words just labels used at the end of the thinking process? The talk reviews the known mechanisms of cognition and suggests a computational mechanism that the mind might use for integrating cognition and language. Mathematics is related to results of other sciences studying the mind and brain. Mathematical descriptions of conscious and unconscious, imagination, symbols, concepts and emotions, including aesthetic emotion will be briefly discussed. The talk will review recent results in origin and evolution of language. Applications to recognition, fusion, and Internet search engines based on language understanding will be discussed. The course focuses on the current understanding of the fundamental principles of working of the mind in the areas of cognition and language, their computational implementations, and practical applications. The course discusses the mind mechanisms, including concepts, emotions, instincts, behavior, language, cognition, understanding, thinking, intuitions, conscious and unconscious, abilities for formation of symbols and aesthetic feelings. Computational techniques are given for these mechanisms and abilities. The goals of the course are: First, to provide a basic mathematical understanding of the working of the mind. Second, to demonstrate practical applications of these mechanisms for pattern recognition, tracking, fusion, search engines, and for integrated systems combining sensor signals and communication data. Third, to outline future research directions. Historical and current difficulties in developing intelligent systems (IS) and applications will be discussed along with how the mind and new computational techniques overcome these difficulties. By the end of the course, students will be familiar with several general applications addressed by IS, especially, smart search engines, computational difficulties encountered over fifty years, and basic novel approaches to overcoming these difficulties. Targeted for: Individuals interested in the development and application of intelligent systems and intelligent signal processing, especially in the area of smart search engines based on language understanding. This course assumes a basic understanding of theory of probability. Keywords: mind, brain, concepts, emotions, instincts, cognition, language, recognition, tracking, fusion, search engines Vita: Dr. Leonid Perlovsky is Principal Research Physicist and Technical Advisor at the Air Force Research Laboratory. Previously, from 1985 to 1999, he served as Chief Scientist at Nichols Research, a $0.5 B high-tech organization, leading the corporate research in information science, intelligent systems, neural networks, optimization, sensor fusion, and algorithm development. In the past he served as professor at Novosibirsk University and New York University. He participated as a principal in commercial startups developing tools for text understanding, search engines, biotechnology, and financial predictions. He published about 50 papers in refereed scientific journals and about 180 papers in conferences, delivered invited keynote plenary talks and authored a book "Neural Networks and Intellect: model-based concepts", Oxford University Press, 2001. Dr. Perlovsky serves on IEEE Computational Intelligence Technical Committee, Computational Intelligence Society Multimedia Tutorial Committee, Chair IEEE Boston Computational Intelligence Chapter, IEEE World Congress on Computation Intelligence Operational Committee, Program Chair for IEEE International Conference on Computational Intelligence Measurement, General Chair for IEEE KIMAS conference, and as Editor-in-Chief for an Elsevier journal “Physics of Life Reviews”. (Back to Top of Abstracts) (Back to Top of Authors) Biologically Plausible Neural Networks Joao Luıs Garcia Rosa PUC-Campinas, Brazil The objectives of the proposed tutorial are to provide artificial neural networks notions and to show how this intelligent computational tool is related to neuroscience. Why current artificial neural network models are so distant from biological ones? Existing connectionist mathematical models are compared with biological neurons and networks and the main differences are stressed. Desired characteristics in biologically plausible artificial neural networks and their corresponding elements in cerebral cortex are discussed. Neuroscience basic concepts about neuron physiology, its connections and the physical chemistry of central nervous system are approached. Several biological features are being taken into account in order to achieve new models that restore the artificial neural systems first concerns. Connectionist models based on neuroscience are about to be considered the next generation of artificial neural networks, inasmuch as nowadays models are far from biology, mainly for mathematical simplicity reasons. The tutorial encompasses the biological neuron, chemical and electrical synapses, the roles of transmitters and receptors, and the neuromodulators of the synaptic function. How the brain became a model to such machine learning technique? Conventional and biologically plausible connectionist algorithms are presented. Why backpropagation, the most used supervised connectionist algorithm, is considered biologically implausible? Some connectionist architectures are taken into account: feedforward, recurrent, and bi-directional. The latter structure is considered more physiologically realistic. A number of applications are presented. They show that the alleged biologically plausible artificial neural networks are even more computationally efficient than conventional ones. Keywords: Artificial neural networks, Biologically plausible connectionist architectures, Biologically plausible connectionist algorithms, Computational neuroscience. Vita: João Luís Garcia Rosa is Titular Professor of Pontifical Catholic University of Campinas (PUC-Campinas), São Paulo, Brazil. He got his doctorate in Computational Linguistics from Linguistics Department, Language Studies Institute, State University of Campinas (Unicamp), Sao Paulo, Brazil, in 1999, and his master's degree in Electrical Engineering (Artificial Intelligence), from Department of Computer Engineering and Industrial Automation, School of Electrical and Computer Engineering, State University of Campinas in 1993. He was graduated in Electrical Engineering (Automation and Electronics) from the latter department in 1983. His research interests are: Artificial Neural Networks, Natural Language Processing, Computational Psycholinguistics, and Biologically Plausible Connectionist Models. His academic experience: taught postgraduate level courses for masters level on computing systems: Intelligent Systems, Artificial Neural Networks, and Natural Language Processing disciplines, and taught graduate level courses for Computer Engineering and System Analysis: Programming Languages, Artificial Intelligence, and Formal Languages and Automata disciplines, at Pontifical Catholic University of Campinas (PUC-Campinas), Campinas, Sao Paulo, Brazil. Since 1998, he contributes to the field of Biologically Plausible Artificial Neural Networks with published papers and supervision of undergraduate and graduate students. (Back to Top of Abstracts) (Back to Top of Authors) Data Visualization of High Dimensional Scientific Data Antonino Staiano and Roberto Tagliaferri Università di Salerno, Italia The recent technological advances are producing huge data sets in almost all fields of scientific research, from astronomy to genetics. Although each research field often requires ad-hoc, fine tuned, procedures to properly exploit all the available information inherently present in the data, there is an urgent need for a new generation of general computational theories and tools capable to boost most human activities of data analysis. Traditional data analysis methods, in fact, are inadequate to cope with such exponential growth in the data volume and especially in the data complexity (ten or hundreds of dimensions of the parameter space). Among the data mining methodologies, visualization plays a key role in developing good models for data especially when the quantity of data is large. For a scientist, i.e. the expert in a specific domain, is essential the need for a visual environment that facilitates exploring high-dimensional data dependent on many parameters. Data visualization is an important means of extracting useful information from large quantities of raw data. The human eye and brain together make a formidable pattern detection tool, but for them to work the data must be represented in a low-dimensional space, usually two or three dimensions. Even quite simple relationship can seem very obscure when the data is presented in tabular form, but are often very easy to see by visual inspection. Many algorithms for data visualization have been proposed by both neural computing and statistics communities, most of which are based on a projection of the data onto a two or three dimensional visualization space. This tutorial embraces a number of these visualization techniques both linear and nonlinear: • Principal Component Analysis (PCA), a classical linear projection method for mapping the data to a lower dimensional space. PCA does this with a linear transformation that preserves as much data variance as possible. • Probabilistic PCA. The disadvantage of PCA is that it does not define a generative model and indeed there is no principled interpretation of its error function. However classical PCA can be made into a density model by using a latent variable approach, derived from factor analysis, i.e. Probabilistic Principal Component Analysis (PPCA). PPCA is a Gaussian mixture model in which each Gaussian has a covariance matrix which is diagonal and has a single variance parameter to describe all the variance in each component. • Mixture of PPCA. Because PCA only defines a linear projection of data, it is a rather limited technique. One way around this is to use a collection of local linear models. The attraction of this is that each model is simpler to understand and usually easier to fit. PCA, PPCA and mixture of PPCA are appropriate when the data is linear or approximately piece-wise linear. An alternative approach is to use global nonlinear methods: • Self Organizing Maps (SOM), a neural network algorithm based on a competitive learning which summarizes a set of data vectors in a high-dimensional space by a set of reference vectors organized on a lower dimensional sheet (usually two dimensional). SOM has been used for a wide variety of applications thanks to its simplicity and for its several plotting options. Although SOM provides easy of computation and powerful visualizations it, indeed, does not define any density model and suffers of other drawbacks which can be overcame employing nonlinear latent variable models; • Generative Topographic Mapping (GTM), a nonlinear latent variable model in which the density is a Gaussian mixture model. The centre of each Gaussian is constrained to lie on a smooth manifold of low dimension (usually 1 or 2) but all Gaussians have a common spherical covariance; • Probabilistic Principal Surfaces (PPS), a generalization of GTM with oriented covariance. PPS allows a spherical manifold in 3D space which is better suitable to characterize high-dimensional data which usually is sparse and located at the periphery (curse of dimensionality). While such algorithms can usefully display the structure of simple and more complex data sets, they often prove inadequate in the face of data sets which are too complex. A single two or three dimensional projection, even if it is nonlinear, may be insufficient to capture all the interesting aspects of the data set. For example, the projection which best separates two clusters may not be the best for revealing internal structure within one of the clusters. This motivates the consideration of a hierarchical model involving multiple two or three dimensional visualization spaces. The goal is that the top-level projection should display the entire data set, perhaps revealing the presence of clusters, while lower-level projections display internal structure within individual clusters, such as the presence of subclusters, which might not be apparent in higher-level projections. Therefore, the tutorial reviewes hierarchical linear (based on mixture of PPCA) and nonlinear (based on GTM) latent variable models and concludes by illustrating a new proposed hierarchical model based on Probabilistic Principal Surfaces. Keywords: Informatics, Data mining Vita: Antonino Staiano is a post-doctoral research fellow at the Department of Mathematics and Computer Science, University of Salerno, Italy. He got his PhD in Computer Science at University of Salerno in 2003, discussing a thesis on unsupervised neural network models for the extraction of scientific information from astronomical databases. Since October 2000 he has been working within the AstroNeural collaboration project (a joint project together with the Department of Physical Science, University Federico II of Naples) to the development of neural -based tools for data visualization and clustering for astronomical and genetic data mining. He has worked at European Southern Observatory (ESO) - Space Telescope European Coordinating Facility and is currently involved in the European Virtual Observatory. Fields of interest and research: Data Mining, Data Visualization, Statistical Pattern Recognition, Bioinformatics, Machine Learning, Ensemble of learning systems, Evolutionary Algorithms. Roberto Tagliaferri is Associate Professor in Computer Science and Neural Networks at the University of Salerno. His research covers the area of neural nets: neural dynamics, fuzzy neural nets, applications to signal and image processing with astronomical and geological data, industrial and medical computer aided diagnosis. Co-chairman of special sessions at the AMSE-ISIS’97, at IJCNN’99, IJCNN 2001, IJCNN 2003, WILF 2003, IJCNN 2004 and co-editor of a special issue of Neural Networks. He had a tutorial on Learning with multiple machines: ECOC models vs Bayesian Framework at IJCNN 2003. He is the author or the co-author of more than 80 publications in the area of neural networks. Since 1995, co-editor of the Proceedings of the Italian Workshops on Neural Nets (WIRN). Secretary of SIREN (Società Italiana Reti Neuroniche) and vice-president of the Italian SIG of the INNS. Member of the Director Council of the IIASS (International Institute for Advanced Scientific Studies) "E. R. Caianiello". Senior member of the IEEE, member of INFN, INFM and AIIA. (Back to Top of Abstracts) (Back to Top of Authors) Support Vector Machines and Kernel Based Learning Johan Suykens The use of kernel based learning techniques has become increasingly popular, largely stimulated by the work on support vector machines as originally introduced within the context of statistical learning theory. Standard support vector machines for classification and regression lead to solving convex optimization problems. The problem formulation and solution is characterized by a primal and dual problem where multilayer neural network interpretations can be given in both worlds. In contrast with many classical models, support vector machines are able to learn and generalize in huge dimensional input spaces. The method makes use of a high dimensional feature map (which should not be explicitly known) in relation to a Mercer kernel (the so-called kernel trick). Notions as large margin classification and regularization techniques play an important role at this point. The method can also be viewed as a functional optimization problem in reproducing kernel Hilbert spaces (RKHS). The kernel trick has been further applied to a wider variety of problems such in kernel Fisher discriminant analysis (KFDA), kernel principal component analysis (KPCA), kernel canonical correlation analysis (KCCA), kernel partial least squares (KPLS), kernel clustering and others. Optimization formulations with primal and dual problem has been given with least-squares support vector machines (LS-SVM), thereby extending support vector machine methodology to a wider range of problems for regression, classification, supervised and unsupervised learning, recurrent networks and control. One often has the choice to either solve the primal or the dual problem depending on the dimensionality of the feature map (or its approximation) and select the most suitable representation for the given problem at hand (e.g. high dimensional input space or large data sets). Kernels have also been customized towards specific application areas such as in textmining, bioinformatics or in relation to graphical models. Novel techniques of hierarchical kernel machines even allow to find the model together with the tuning parameters by solving a convex problem. At the same time one can achieve in this way e.g. sparse representations, stability of learning machines and input selection. In this tutorial we outline the main concepts of support vector machines and kernel based learning, show interesting research directions and illustrate the methods with many successful real-life examples in different areas such as e.g. time-series prediction, nonlinear modelling and microarray data analysis. Keywords: Support vector machines, kernel based learning, convex optimization, primal and dual problem, supervised and unsupervised learning Links: http://www.esat.kuleuven.ac.be/sista/lssvmlab/ http://www.esat.kuleuven.ac.be/sista/members/suykens.html Vita: Johan A.K. Suykens is an associate professor with K. U. Leuven Belgium. His research interests are mainly in the areas of the theory and application of neural networks and nonlinear systems. He is author of the books "Artificial Neural Networks for Modelling and Control of Non-linear Systems" (Kluwer Academic Publishers) and "Least Squares Support Vector Machines" (World Scientific) and editor of the books "Nonlinear Modeling: Advanced Black-Box Techniques" (Kluwer Academic Publishers) and "Advances in Learning Theory: Methods, Models and Applications" (IOS Press). In 1998 he organized an International Workshop on Nonlinear Modelling with Time-series Prediction Competition. He is a Senior IEEE member and has served as associate editor for the IEEE Transactions on Circuits and Systems - Part I (1997-1999) and Part II (since 2004) and since 1998 he is serving as associate editor for the IEEE Transactions on Neural Networks. He received an IEEE Signal Processing Society 1999 Best Paper Senior Award and several Best Paper Awards at International Conferences. He is a recipient of the International Neural Networks Society INNS 2000 Young Investigator Award for significant contributions in the field of neural networks. He has served as Director and Organizer of a NATO Advanced Study Institute on Learning Theory and Practice (Leuven 2002) and as a program co-chair for the International Joint Conference on Neural Networks IJCNN 2004. (Back to Top of Abstracts) (Back to Top of Authors) Unsupervised Learning Real World Applications Prof. Harold Szu The tutorial presents unsupervised learning artificial neural networks (ANN), which yields unbiased computational intelligence (CI), information acquisition, feature extraction and knowledge discovery. Their applications in bioinformatics, brain study, unmanned robot control, finance and business are briefly presented. The necessary and sufficient conditions are that (1) all CPU Brain is always on all the time and is kept at an isothermal equilibrium temperature 37oC and (2) each CPU Brain is equipped with power of 5 sensors pairs e.g. two eyes, two ears, etc. altogether 10 dimensional vector time series for input data space. The overwhelming incoming excitations must not be replenished, a natural choice is letting fluctuating disagreements so-called noise decay away, and only reinforce coincident signals so-called features. This biomimetics is our minimum free energy principle of unsupervised feature extraction & fusion methodology. The tutorial proves these conditions (1) (2) constituted the basis of unsupervised learning ANN and real world applications in remote sensing and early tumor detection are given for computer scientists, neuroscientists, biologist, engineers and graduate students. Keywords: Unsupervised Learning, Computational Intelligence, Knowledge-based neural networks, Computational Neurogenetic modeling. Vita: Harold Szu received in 1971 Ph. D. in Physics/Statistical Mechanics from Prof. George Uhlenbeck at the Rockefeller University, New York, N.Y. and worked at Naval Research Lab Wash DC in three Divisions, and became the Information Science Group Leader of Naval Surface Warfare Center at White Oak (1990-1996) and at Dahlgren VA(1997-now). He has been a Fellow of SPIE (since 1995) for neural nets, Fellow of OSA (1996) for adaptive wavelets, Fellow of IEEE (1997) for bi-sensor fusion, Fellow of Am. Inst. Med. & Bio. Eng. 2004 for medical image diagnoses. Dr. Szu is a Champaign of brain-style computing for decades, a founder, former president, and a current governor of International Neural Network Society (INNS), He has contributed to the establishment of 30 SIG/Chapter worldwide. Notably, he has scientifically contributed to the unsupervised redundancy reduction of sensory pair fusion by postulating the thermodynamic free energy for learning. He received INNS D. Gabor Award, in 1997, “for outstanding contribution to neural network applications in information sciences and pioneer implementations of fast simulated annealing search," and Italy Academy the Eduardo R. Caianiello Award, 1999, for “elucidating and implementing a chaotic neural net as a dynamic realization for fuzzy logic membership function." Dr. Szu is a foreign academician of Russian Academy of Nonlinear Sciences in 1999 for the homeostasis unsupervised learning based on constant Cybernetic Temperature. (Back to Top of Abstracts) (Back to Top of Authors) Feature Extraction in Computational Intelligence Evangelia Micheli-Tzanakou, PhD Department of Biomedical Engineering, RUTGERS UNIV. One of the major problems a researcher faces is what is learned from data obtained by various methods and different techniques. This tutorial will discuss and compare topics such as: Statistical Advances and Challenges, as well as Feature Extraction in Computational Intelligence methodologies. Often a simple model describes the data well, simply because the S/N ratio is too small for detection of more complex structures-which for example is the case with medical data involving human subjects. One has a lot of variability both in intra- and inter-sets of data. Some important Simple Tools that have been used for a long time are: Linear Regression, Discriminant analysis, Principal Component Analysis etc. In all of these, the size of the data set matters. Huge data sets create memory problems. The question is how do we handle different data types and how do we handle them? What if the data are correlated? What if we have complex data structures? In this tutorial you will learn more about Computational Intelligence, how to get to know your data and how to do Feature Extraction. Some examples of "features" will be given and different feature extraction methods will be discussed. Keywords: Vita: Dr. Tzanakou is Professor II and director of the Computational Intelligence Laboratories (2000-present); Chair of Biomedical Engineering Dept at Rutgers University (1990-2000);established the BME Undergraduate curriculum in 1999. Has an active research group and has published over 250 scientific papers. Author/co-author of 2 books. Has graduated 37 MS and PhDs. Fellowships: IEEE Fellow (1992);Founding Fellow, American Institute for Medical & Biological Eng. (AIMBE), 1993; Member of Sigma Xi and Eta Kappa Nu; Listed in American Men and Women in Science; Honorary Member of the British Brain Research Association; Honorary Member of the European Brain and Behavior Society. Awards: NJ Women of Achievement Award, 1995; Featured in the “Notable Twentieth Century Scientists, 1994; Achievement Award Recipient, Society of Women Engineers, 1992; Outstanding IEEE Branch Advisor/Counselor Award, 1985 ; cited as a Pioneer in the IEEE Web page (http://www.ieee.org/innovators). Editorships: Editorial Board, IEEE Trans. On Nano-BioScience (2002-present); Editorial Board Biomedical Engineering On Line (2001-present ); Associate Editor, IEEE Transactions on Neural Networks (2000-present); Book Series Editor in Biomedical Engineering, Plenum/Klewer (1999-present); Book Series Editor, Biomedical Systems, IES Book series, CRC Press (1997-present); Editorial Board, IEEE Trans. Biomedical Information Tech.(1997-2001); Editorial Board, International J. On Advanced Computational Intelligence (1997-1999); Associate Editor, IEEE Transactions on Neural Networks (1989-1992). (Back to Top of Abstracts) (Back to Top of Authors) Neural Networks That Actually Work In Prediction and Decision/Control: Common Misconceptions Versus Real-World Success Paul J. Werbos NSF Neural nets are used in a growing variety of real-world applications, in a growing number of areas (including the modeling of natural intelligence). But this has been a mixed blessing, as various subcultures have not kept up with each other. For example, at http://www.eas.asu.edu/~nsfadp/ applications of adaptive dynamic programming (aka advanced reinforcement learning) are reported which beat the previous best performance in electric power grid control, aircraft control, in large-scale logistics, etc. -- but best performance requires integrating concepts from many subcultures. Yet in some subcultures, people are still using "mappings from sensory to motor coordinates" (direct adaptive control), whose limits were clear years ago. Some are using indirect methods inspired by control theory with well-known stability theorems -- theorems whose conditions are rarely satisfied, and methods which are easily improved upon. Similarly, in prediction, some subcultures falsely believe that recurrent networks are hard to train and of marginal benefit, even as people in industry use them routinely, and report performance better than extended Kalman filters and equal (at lower cost) to elaborate particle filter methods. The foundations of learning also bear on the practical choices. This tutorial will try to offer a kind of practical roadmap of these issues, and suggest how it points towards future functionality in tasks as large-scale as what brains can handle. Keywords: adaptive dynamic programming (advanced reinforcement learning) Vita: Paul J. Werbos is best known as the original inventor of backpropagation, as part of his Harvard PhD thesis, which was reprinted in full in his book the Roots of Backpropagation, Wiley 1994, along with his classic 1990 tutorial on backpropagation through time for Proc IEEE. Also, his 1967 article in Cybernetica first proposed the idea of approximating dynamic programming as a way to improve reinforcement learning, elaborated in many later papers. He was one of the three original two-year presidents of the International Neural Network Society, and winner of the IEEE Pioneer Award. He is Program Director for Control, Networks and Computational Intelligence at NSF, which actively seeks more proposals in this area. He has also been active in many cross-cutting funding initiatives; for example, he serves on the Working Group for energy storage and distribution of the interagency Climate Change Technology Program, and coordinated the NASA-NSF-EPRI solicitation on space solar power (NSF 02-098). He is also on the Planning Committee of the Millennium Project of the United Nations University (http://millennium-project.org), and has published a few papers on quantum foundations and technology (see arXiv.org, physics and nonlinear systems). He also has two degrees in economics from Harvard and the London School of Economics. He is on the governing boards of INNS and of the IEEE Industrial Electronics Society. (Back to Top of Abstracts) (Back to Top of Authors) Cognitive Memory Bernard Widrow Stanford University, USA Regarding the workings of the human mind, memory and pattern recognition seem to be intertwined. You generally do not have one without the other. Taking inspiration from life experience, a new form of computer memory has been devised. It has been used successfully in diverse applications such as visual aircraft identification, aircraft navigation, and human facial recognition. Other uses are being explored. The basic idea will have many new areas of application. Keywords: cognitive memory, pattern recognition, applications Vita: Bernard Widrow received the S.B., S.M., and Sc.D. degrees in Electrical Engineering from the Massachusetts Institute of Technology in 1951, 1953, and 1956, respectively. He joined the MIT faculty and taught there from 1956 to 1959. In 1959, he joined the faculty of Stanford University, where he is currently Professor of Electrical Engineering. He began research on adaptive filters, learning processes, and artificial neural models in 1957. Together with M.E. Hoff, Jr., his first doctoral student at Stanford, he invented with LMS algorithm on in the Autumn of 1959. Today, this is the world's most widely used learning algorithm. He has continued working on adaptive signal processing, adaptive controls, and neural networks since that time. Dr. Widrow is a Life Fellow of the IEEE and a Fellow of AAAS. He received the IEEE Centennial Medal in 1984, the IEEE Alexander Graham Bell Medal in 1986, the IEEE Signal Processing Society Medal in 1986, the IEEE Neural Networks Pioneer Medal in 1991, the IEEE Millennium Medal in 2000, and the Benjamin Franklin Medal for Engineering from the Franklin Institute of Philadelphia in 2001. He was inducted into the National Academy of Engineering in 1995 and into the Silicon Valley Engineering Council Hall of Fame in 1999. Dr. Widrow is a past president and currently a member of the Governing Board of the International Neural Network Society. He is a member of the AdCom of the IEEE Computational Intelligence Society. He is associate editor of several journals and is the author of over 100 technical papers and 18 patents. He is co-author of "Adaptive Signal Processing" and "Adaptive Inverse Control," both Prentice-Hall books. A new book, "Quantization Noise," is in preparation. (Back to Top of Abstracts) (Back to Top of Authors)



This page (http://www.mlpadgett.org/IJCNN05tutorials) was last modified on June 15, 2005. Contact: m.padgett(at)ieee.org