Fully connected neuron network



Traditional NN

  • The weight matrix A is N by so that the network is “fully connected”.
  • All nodes on adjacent layers are fully connected with each other
  • Can be seen as with M “kernels” which has N dimensions each
  • Many parameters; suffer severe overfitting

Locally connected neural network

Local NN

  • Output is based only on the “receptive field” of size P, so weight matrix W is P by M (P<N)
  • kernels each with dimension P
  • Less parameters to train, less overfit

Convolutional Neural Network (shared-weight local)




  • Shared weight local connected neural network
  • Weight matrix W is P by 1 (since we can write P as a Matrix as well)
  • Even fewer parameters to train
  • Can simultaneously train many maps to extract more features