CNN ARCHITECTURES: SQUEEZENET

This is part of the CNN Architectures series by Dimitris Katsios. Find all CNN Architectures online:

SqueezeNet

We will use the tensorflow.keras Functional API to build SqueezeNet from the original paper: “SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size” by Forrest N. Iandola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, Kurt Keutzer.


In the paper we can read:

[i] “[…] we implement our expand layer with two separate convolution layers: a layer with 1×1 filters, and a layer with 3×3 filters. Then, we concatenate the outputs of these layers together in the channel dimension.”

We will also make use of the following Table [ii]:

as well the following Diagrams [iii] and [iv]


Network architecture

Based on [ii] the network

  • starts with a Convolution-MaxPool block
  • continues with a series of Fire blocks separated by MaxPool layers
  • finishes with Convolution and Average Pool layers.

Notice that there is no Fully Connected layer in the model which means that the network can process different image sizes.

Fire block

The Fire block is depicted at [iii] and consists of:

  1. a 1×1 Convolution layer that outputs the squeezed tensor
  2. a 1×1 Convolution layer and a 3×3 Convolution layer applied on the squeeze tensor and the ouputs of which are then concatenated as described in [i]

Workflow

We will:

  1. import the neccesary layers
  2. write a helper function for the Fire block ([iii])
  3. write the stem of the model
  4. use the helper function to write the main part of the model
  5. write the last part of the model

1. Imports

Code:

from tensorflow.keras.layers import Input, Conv2D, Concatenate, \
     MaxPool2D, GlobalAvgPool2D, Activation

2. Fire block

Next, we will write the Fire block function

This function will:

  • take as inputs:
    • a tensor (x)
    • the filters of the 1st 1×1 Convolution layer (squeeze_filters)
    • the filters of the 2nd 1×1 Convolution and the 3×3 Convolution layers (expand_filters)
  • run:
    • apply a 1×1 conv operation on x to get squeezed tensor
    • apply a 1×1 conv and a 3×3 conv operation on squeezed
    • Concatenate these two tensors
  • return the concatenated tensor

Code:

def fire_block(x, squeeze_filters, expand_filters):
    squeezed = Conv2D(filters=squeeze_filters,
                      kernel_size=1,
                      activation='relu')(x)
    expanded_1x1 = Conv2D(filters=expand_filters,
                        kernel_size=1,
                        activation='relu')(squeezed)
    expanded_3x3 = Conv2D(filters=expand_filters,
                        kernel_size=3,
                        padding='same',
                        activation='relu')(squeezed)

    output = Concatenate()([expanded_1x1, expanded_3x3])
    return output

3. Model stem

Based on [ii]:

layer name/typeoutput sizefilter size / stride
input image224x224x3
conv1111x111x967×7/2 (x96)
maxpool155x55x963×3/2

the model starts with:

  1. a Convolution layer with 96 filters and kernel size 7×7 applied on a 224x224x3 input image
  2. a MaxPool layer with pool size 3×3 and stride 2

Code:

input = Input([224, 224, 3])

x = Conv2D(96, 7, strides=2, padding='same', activation='relu')(input)
x = MaxPool2D(3, strides=2, padding='same')(x)

4. Main part

Based on [ii]:

layer name/typefilter size / strides1x1(#1×1 squeeze)e1x1(#1×1 expand)e3x3(#3×3 expand)
fire2166464
fire3166464
fire432128128
maxpool43×3/2
fire532128128
fire648192192
fire748192192
fire864256256
maxpool83×3/2
fire964256256

the model continues with:

  1. Fire block (fire2) with 16 squeeze and 64 expand filters
  2. Fire block (fire3) with 16 squeeze and 64 expand filters
  3. Fire block (fire4) with 32 squeeze and 128 expand filters
  4. a MaxPool layer (maxpool4) with pool size 3×3 and stride 2
  5. Fire block (fire5) with 32 squeeze and 128 expand filters
  6. Fire block (fire6) with 48 squeeze and 192 expand filters
  7. Fire block (fire7) with 48 squeeze and 192 expand filters
  8. Fire block (fire8) with 64 squeeze and 256 expand filters
  9. a MaxPool layer (maxpool8) with pool size 3×3 and stride 2
  10. Fire block (fire9) with 64 squeeze and 256 expand filters

Code:

x = fire_block(x, squeeze_filters=16, expand_filters=64)
x = fire_block(x, squeeze_filters=16, expand_filters=64)
x = fire_block(x, squeeze_filters=32, expand_filters=128)
x = MaxPool2D(pool_size=3, strides=2, padding='same')(x)

x = fire_block(x, squeeze_filters=32, expand_filters=128)
x = fire_block(x, squeeze_filters=48, expand_filters=192)
x = fire_block(x, squeeze_filters=48, expand_filters=192)
x = fire_block(x, squeeze_filters=64, expand_filters=256)
x = MaxPool2D(pool_size=3, strides=2, padding='same')(x)

x = fire_block(x, squeeze_filters=64, expand_filters=256)

5. Last part

Based on [ii]:

layer name/typefilter size / stride
conv101×1/1 (x1000)
avgpool1013×13/1

the model ends with:

  1. a Convolution layer with 1000 filters and kernel size 1×1
  2. a Average Pool layer with stride 1 which based on [iv] is Global
  3. Softmax activation applied on the output number ([iv])

Code:

x = Conv2D(filters=1000, kernel_size=1)(x)
x = GlobalAvgPool2D()(x)

output = Activation('softmax')(x)

from tensorflow.keras import Model
model = Model(input, output)

Check number of parameters

We can also check the total number of parameters of the model by calling count_params() on each result element of model.trainable_weights.

According to [ii] (col: #parameter before pruning) there are 1,248,424 (total) parameters at SqueezeNet model.

Code:

>>> import numpy as np
>>> import tensorflow.keras.backend as K
>>> int(np.sum([K.count_params(p) for p in model.trainable_weights]))
1248424

Final code

Code:

from tensorflow.keras.layers import Input, Conv2D, Concatenate, \
     MaxPool2D, GlobalAvgPool2D, Activation


def fire_block(x, squeeze_filters, expand_filters):
    squeezed = Conv2D(filters=squeeze_filters,
                      kernel_size=1,
                      activation='relu')(x)
    expanded_1x1 = Conv2D(filters=expand_filters,
                        kernel_size=1,
                        activation='relu')(squeezed)
    expanded_3x3 = Conv2D(filters=expand_filters,
                        kernel_size=3,
                        padding='same',
                        activation='relu')(squeezed)

    output = Concatenate()([expanded_1x1, expanded_3x3])
    return output


input = Input([224, 224, 3])

x = Conv2D(96, 7, strides=2, padding='same', activation='relu')(input)
x = MaxPool2D(3, strides=2, padding='same')(x)


x = fire_block(x, squeeze_filters=16, expand_filters=64)
x = fire_block(x, squeeze_filters=16, expand_filters=64)
x = fire_block(x, squeeze_filters=32, expand_filters=128)
x = MaxPool2D(pool_size=3, strides=2, padding='same')(x)

x = fire_block(x, squeeze_filters=32, expand_filters=128)
x = fire_block(x, squeeze_filters=48, expand_filters=192)
x = fire_block(x, squeeze_filters=48, expand_filters=192)
x = fire_block(x, squeeze_filters=64, expand_filters=256)
x = MaxPool2D(pool_size=3, strides=2, padding='same')(x)

x = fire_block(x, squeeze_filters=64, expand_filters=256)


x = Conv2D(filters=1000, kernel_size=1)(x)
x = GlobalAvgPool2D()(x)

output = Activation('softmax')(x)

from tensorflow.keras import Model
model = Model(input, output)

Model diagram

Leave a Reply