Analyses of Deep Learning (STATS 385)

Convolutional (Conv) layer

Accepts as input:

feature vector of size $W_1 \times H_1 \times D_1$
filters of size $F \times F \times D_1 \times D_2$
biases of length $D_2$
stride $S$
amount of zero padding $P$

Outputs another feature vector of size $W_2 \times H_2 \times D_2$ , where

The d-th channel in the output feature vector is obtained by performing a valid convolution with stride $S$ of the d-th filter and the padded input.
source

Stride

The amount by which a filter shifts spatially when convolving it with a feature vector.
source
image source

Dilation

A filter is dilated by a factor $Q$ by inserting in every one of its channels independently $Q-1$ zeros between the filter elements.
source
image source

Fully connected (FC) layer

In practice, FC layers are implemented using a convolutional layer. To see how this might be possible, note that when an input feature vector of size $H \times W \times D_1$ is convolved with a filter bank of size $H \times W \times D_1 \times D_2$ , it results in an output feature vector of size $1 \times 1 \times D_2$ . Since the convolution is valid and the filter can not move spatially, the operation is equivalent to a fully connected one. More over, when this feature vector of size 1x1xD_2 is convolved with another filter bank of size $1 \times 1 \times D_2 \times D_3$ , the result is of size $1 \times 1 \times D_3$ . In this case, again, the convolution is done over a single spatial location and therefore equivalent to a fully connected layer.
source
image source

Linear classifier

This is implemented in practice by employing a fully connected layer of size $H \times W \times D \times C$ , where $C$ is the number of classes. Each one of the filters of size $H \times W \times D$ corresponds to a certain class and there are $C$ classifiers, one for each class.
image source

back

Analyses of Deep Learning (STATS 385)

Stanford University, Fall 2019