Neural Computation and Self-Organization

Teaching

A course about the intersection of computer science and biology: information processing in neurons, self-assembly and self organization, brain function, and neural networks.

  • decision theory, classifiers, learning
  • models of parallel computation, data-parallelism
  • psychophysics, neuroanatomy, neurophysiology
  • individual neurons, synapses, Hebbian learning, pattern generators, McCullogh-Pitts, spiking neural networks, boolean circuits
  • visual pathways, visual neurophysiology, what/where, stereo, motion, color
  • physics-based and computational models of motion, stereo, color perception
  • visual object recognition and learning: HMAX, spiking models, learning, slow features, deep learning, sparsity
  • attentional mechanisms, segmentation
  • recurrent neural networks, echo state networks
  • unsupervised learning: self-organizing maps, clustering, PCA, ICA
  • self-assembly, organic computing
  • multi-agent system
  • evolutionary algorithms

Materials

Questions

Lecture 1

  • What are major levels of description for digital systems?
  • What levels of neural organization correspond to transistors, gates, circuits, block diagrams, …?
  • Label the major parts of a neuron.
  • Describe how information is transmitted in a sensory neuron from the sensing element to the postsynaptic cell?
  • Describe briefly how a chemical synapse works.
  • What is the knee jerk reflex and how does the reflex arc work?
  • What is “synaptic integration” and what kinds are there?

Lecture 2

  • How are voltages measured inside neurons?
  • What is depolarization?
  • What is patch clamping and what does it measure?
  • What are the major ions involved in generating neural activity?
  • What are the approximate Nernst potentials of the major ions in neural activity?
  • What is the Nernst equation?
  • Describe the steps involved in action potential generation.
  • What is the Hodgkin-Huxley model?
  • Describe how differential equations are solved numerically using difference equation approximations.
  • What is “cable theory” and how does it relate to neurons?
  • What is the FitzHugh-Nagumo Model and what is its significance?
  • What is the “phase space” of a differential equation?
  • What does the phase space for the FitzHugh-Nagumo model look like?
  • How can we use k-means clustering to categorize different behaviors of a neuron model?

Lecture 3

  • What is the integrate-and-fire model?
  • What is the Izhikevich model?
  • What does the phase space for the Izhikevich model look like?
  • What is a McCulloch-Pitts neuron?
  • How is the McCulloch-Pitts neuron justified using a discretization of time in the Hodgkin-Huxley model?
  • How is the McCulloch-Pitts neuron justified using a rate coding model?
  • How can McCulloch-Pitts neurons be used to represent logic gates?
  • What is computaitonal universality?
  • What is a cellular automaton?
  • What is Rule 110?
  • What is a tag system?
  • What is a recurrent neural network?
  • How can Rule 110 be implemented as a recurrent neural network?

Lecture 4

  • Explain why a refractory period is important for achieving synchronization in a network of weakly coupled integrate-and-fire neurons.
  • State the equation for the van der Pol oscillator.
  • Draw the phase space and nullclines for the van der Pol oscillator.
  • Which neural model does the van der Pol oscillator relate to, and how?
  • What is entrainment?
  • Give a mathematical example of entrainment based on the van der Pol oscillator.
  • What phenomenon is the strongly forced van der Pol oscillator an example of?
  • What is a Lorenz system and what is it an example of?
  • Explain the meaning of “strong sensitivity to initial conditions”.
  • What is the difference between a limit cycle and a strange attractor?
  • Can the Lotka-Volterra equations produce chaos? How?
  • What real-world phenomena are described by Lotka-Volterra equations (give several examples)?
  • Give an example of a chaotic system based on a difference equation.
  • Give an example of a chaotic 1D cellular automaton.
  • Give an example of a class 4 1D cellular automaton.
  • Define the different classes of behavior in 1D cellular automata.
  • How does computational universality relate to chaotic behavior?
  • How does computational universality relate to the classes of 1D cellular automata?
  • What is Willshaw’s associative memory? State the equations and the training method.
  • What is a Hopfield network? State the equations and the training method.

Lecture 5

  • What is Bayes formula?
  • What is zero-one loss?
  • What is the Bayes error?
  • What is the optimal decision rule under zero-one loss?
  • What is the relationship between a linear threshold unit and similarity measures between vectors?
  • What are augmented vectors?
  • State the perceptron learning algorithm.
  • What is the perceptron criterion function?
  • How can a linear least square algorithm be used for classification?
  • What is the pseudo-inverse and how is it used for linear least square problems?
  • What is a linearly separable problem?
  • Does a perceptron learning algorithm always find a lowest error solution for a linearly separable problem?
  • Does a perceptron learning algorithm always find a lowest error solution for a problem that is not linearly separable?
  • What is logistic regression?
  • How are the parameters of logistic regression estimated with gradient descent?

Lecture 6

  • How are multilayer perceptrons defined?
  • Are the perceptrons in multilayer perceptrons the same as in single layer perceptrons?
  • What was the book “Perceptrons” by Minsky and Papert about?
  • What is the criterion function usually used for training MLPs?
  • What is the training algorithm usually used for training MLPs?
  • Derive the MLP learning algorithm.
  • Why is the MLP learning algorithm called “backpropagation”?
  • What is stochastic gradient descent?
  • What is the learning rate?
  • What is the number of hidden units?
  • How do you choose learning rates and the number of hidden units?
  • What is a test set?
  • What is a training curve?
  • Why would test set error be larger than training set error?
  • What is cross-validation?

Lecture 7

  • If you implement layers in an MLP library as classes, what are the three primary methods needed during training?
  • Describe a design for a modular MLP library.
  • Describe how deltas are passed around in an MLP library.
  • Assuming you have multiple perceptron layers being trained using backpropagation. In what order should the forward, backward, and update methods be invoked on the three layers?
  • How are classes encoded for training an MLP?
  • How does an MLP have to be trained so that its outputs approximate posterior probabilities?
  • Why is it important that the outputs of an MLP approximate posterior probabilities?
  • Describe the training of a single layer convolutional neural network with one output.
  • What is the relationship between “convolution” and a convolutional neural network?

Lecture 8

  • Describe the architecture of a multilayer convolutional neural network.
  • What is unsupervised learning?
  • What is the MNIST database?
  • How do you center a dataset?
  • What is the covariance matrix?
  • What does the covariance matrix of a centered dataset mean? How do you visualize it?
  • What is an eigenvector?
  • If you sample from a Gaussian density, what do the eigenvectors of the covariance matrix represent?
  • Explain what PCA accomplishes?
  • If you project onto the eigenvectors corresponding to the first n largest eigenvalues, what are you doing?
  • How can PCA be used to separate signal from noise?
  • How is PCA used to preprocess data for pattern recognition?
  • How does PCA speed up nearest neighbor classification?
  • What kind of matrix is the covariance matrix?
  • Describe a simple, iterative algorithm for finding the eigenvector corresponding to the largest eigenvalue of a covariance matrix.
  • Describe a simple, iterative algorithm for finding the top n eigenvectors of a symmetric positive definite matrix.

Lecture 9

  • Describe and explain the k-means algorithm.
  • Explain the mixture density estimation problem.
  • Describe the EM algorithm for mixture density estimation.
  • Explain “incremental” or “neural” analogs of the k-means algorithm.
  • Describe how to use the k-means algorithm for improving MNIST classification.
  • Describe and explain the Self-Organizing Map algorithm.

Lecture 10

  • What are simple cells, what are complex cells, where are they found? What is the difference?
  • What is a feature hierarchy?
  • What is the Neocognitron?
  • Be able to answer questions about the paper “Gradient-Based Learning Applied to Document Recognition” by Y. LeCun, L. Bottou, Y. Bengio, P. Haffner
  • Be able to answer questions about the paper “Learning Methods for Generic Object Recognition with Invariance to Poste and Lighting” by Yann LeCun et al.
  • Describe how convolutional neural networks are used for handwriting recognition.
  • Describe how convolutional neural networks are used for object recognition.
  • Describe how convolutional neural networks are trained for object recognition.
  • Be able to answer questions about the paper “Hierarchical models of object recognition in cortex” by Riesenhuber and Poggio
  • Describe the HMAX model of visual object recognition.
  • Describe experiments supporting analogies between the HMAX model and human visual object recognition.
  • Be able to answer questions about the paper “Comparing State-of-the-Art Visual Features on Invariant Object Recognition” by Pinto et al.
  • How did Pinto generate test cases for invariant object recognition?
  • What should object recognition be invariant to?
  • What does invariant object recognition mean?
  • Broadly summarize the benchmark results from Pinto’s paper.

Lecture 11

  • Describe how 3D model-based reocgnition works.
  • Explain the difficulties encountered in implementing 3D model-based recognition.
  • Explain component-based 3D object recognition.
  • What is a geon?
  • What is view-based recognition?
  • Summarize the paper “Psychophysical support for a two-dimensional view interpolation theory of object recognition.” and be able to answer questions about it.
  • What is a “2AFC experiment”?
  • What are the different predictions of 2D view interpolation and 3D model based recognition in visual object recognition experiments?
  • Summarize the results of the paper “Recognizing Depth-Rotated Objects” and be able to answer questions about it.
  • What is Biederman’s major criticism of Bülthoff’s approach and conclusions?
  • Give examples of viewpoint dependent parts changes and explain how they are important in experiments trying to prove a geon-based object recognition theory.
  • How did Biederman modify Bülthoff’s experiments by adding a distinctive geon? What results did he find?

Lecture 14

  • What is a time series?
  • Give examples of time series?
  • What is a stationary process?
  • What is the trend of a time series? How might you find it?
  • What is a random walk?
  • What is the normal density?
  • What is the Cauchy distribution? What unusual properties does it have?
  • What is the autocorrelation function? What does it tell you?
  • What is an autoregressive process?
  • What is a moving average process?
  • How do autoregressive and moving average processes relate to image/signal filtering?
  • Give some common sequence classification tasks.
  • Give some common language recognition tasks used for testing recurrent neural networks.
  • What is a recurrent neural network?
  • What is backpropagation through time?