3 Matching Annotations
 Nov 2021

distill.pub distill.pub

The following figure presents a simple functional diagram of the neural network we will use throughout the article. The neural network is a sequence of linear (both convolutional A convolution calculates weighted sums of regions in the input. In neural networks, the learnable weights in convolutional layers are referred to as the kernel. For example Image credit to https://towardsdatascience.com/gentlediveintomathbehindconvolutionalneuralnetworks79a07dd44cf9. See also Convolution arithmetic. and fullyconnected A fullyconnected layer computes output neurons as weighted sum of input neurons. In matrix form, it is a matrix that linearly transforms the input vector into the output vector. ), maxpooling, and ReLU First introduced by Nair and Hinton, ReLU calculates f(x)=max(0,x)f(x)=max(0,x)f(x)=max(0,x) for each entry in a vector input. Graphically, it is a hinge at the origin: Image credit to https://pytorch.org/docs/stable/nn.html#relu layers, culminating in a softmax Softmax function calculates S(yi)=eyiΣj=1NeyjS(y_i)=\frac{e^{y_i}}{\Sigma_{j=1}^{N} e^{y_j}}S(yi)=Σj=1Neyjeyi for each entry (yiy_iyi) in a vector input (yyy). For example, Image credit to https://ljvmiranda921.github.io/notebook/2017/08/13/softmaxandthenegativeloglikelihood/ layer.
This is a great visualization of MNIST hidden layers.
Tags
Annotators
URL

 Oct 2021

cloud.google.com cloud.google.com

Even with this very primitive single neuron, you can achieve 90% accuracy when recognizing a handwritten text image1. To recognize all the digits from 0 to 9, you would need just ten neurons to recognize them with 92% accuracy.
And here is a Google Colab notebook that demonstrates that

 Sep 2021

colah.github.io colah.github.io

One popular theory among machine learning researchers is the manifold hypothesis: MNIST is a low dimensional manifold, sweeping and curving through its highdimensional embedding space. Another hypothesis, more associated with topological data analysis, is that data like MNIST consists of blobs with tentaclelike protrusions sticking out into the surrounding space.
