In this exercise we will be looking to implement a custom ResNet architecture on the CIFAR 10 dataset using PyTorch. In addition we will be using the imgae augmentations from the Albumentations library and using one cycle learning rate after find the maximum learning rate using the LRFinder module.
We will try to build a custom Resnet Architecture that can achieve atleast 90% accuracy on the CIFAR10 dataset. We will try to achieve this in 24 epochs. This experimentation is similar to David Page's exercise. We will also use the LRFinder module to find the maximum learning rate and use the one cycle learning rate policy for the model. We will try to achieve the maximum learning rate by the
The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. The 10 classes in CIFAR-10 are:
- Airplane
- Automobile
- Bird
- Cat
- Deer
- Dog
- Frog
- Horse
- Ship
- Truck
CIFAR-10 is a widely used benchmark dataset in the field of computer vision and deep learning. It consists of 60,000 color images, each of size 32x32 pixels, belonging to 10 different classes. The dataset is divided into 50,000 training images and 10,000 testing images.
The images in CIFAR-10 are relatively low-resolution compared to some other datasets, making it a challenging task for machine learning models to accurately classify the images. The dataset is commonly used for tasks such as image classification, object detection, and image segmentation.
The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training batches contain exactly 5000 images from each class.
Let us look at some samples of the CIFAR 10 dataset.
Looking at different samples of each class
The colour and RGB representation of the same image can be seen below
Albumentations is a computer vision tool that boosts the performance of deep convolutional neural networks. The library is widely used in industry, deep learning research, machine learning competitions, and open source projects.
The Albumentations library is an open-source Python library used for image augmentation in machine learning and computer vision tasks. Image augmentation is a technique that involves applying a variety of transformations to images to create new training samples, thus expanding the diversity of the training dataset. This process helps improve the generalization and robustness of machine learning models by exposing them to a wider range of image variations.
Albumentations provides a wide range of image augmentation techniques, including geometric transformations (such as scaling, rotation, and cropping), color manipulations (such as brightness adjustment, contrast enhancement, and saturation changes), noise injection, and more. It is designed to be fast, flexible, and easy to use.
One of the key features of Albumentations is its integration with popular deep learning frameworks such as PyTorch and TensorFlow. It provides a simple and efficient API that allows users to easily incorporate image augmentation into their data preprocessing pipeline, seamlessly integrating it with their training process.
Albumentations supports a diverse set of image data types, including numpy arrays, PIL images, OpenCV images, and others. It also allows for custom augmentation pipelines and provides a rich set of options to control the augmentation parameters and their probabilities.
Overall, Albumentations is a powerful library that simplifies the process of applying image augmentation techniques, helping researchers and practitioners improve the performance and reliability of their computer vision models.
-
Normalization: This transformation normalizes the image by subtracting the mean values (0.4914, 0.4822, 0.4465) and dividing by the standard deviation values (0.247, 0.243, 0.261) for each color channel (RGB). Normalization helps to standardize the pixel values and ensure that they have a similar range. -
PadIfNeeded: Resizes the image to the desired size while maintaining the aspect ratio, and if the image is smaller than the specified size, it pads the image with zeros or any other specified value. In this demo, the padding is set to 4. -
RandomCrop: Used to randomly crop an image and optionally its corresponding annotations or masks. It is a common transformation used for data augmentation in computer vision tasks. -
HorizontalFlip: This transformation horizontally flips the image with a probability of 50%. It creates a mirror image of the original image, which helps invariance to left-right orientation. -
Cutout: Used to randomly remove rectangular regions from an image. This technique is often employed as a form of data augmentation to enhance the model's robustness and generalization.The Cutout transform helps introduce regularization and prevents the model from relying on specific local patterns or details in the training data. It encourages the model to focus on more relevant features and improves its ability to generalize to unseen examples. In this demo the cutout is set to (8,8). -
ToTensorV2: This transformation converts the image from a numpy array to a PyTorch tensor. It also adjusts the dimensions and channel ordering to match PyTorch's convention (C x H x W).
We can see the train data loader having images with transformations from the albumentations library applied below -
These transformations collectively create a diverse set of augmented images for the training data, allowing the model to learn from different variations and improve its generalization capability.
The LRfinder module, short for Learning Rate Finder, is a tool commonly used in deep learning frameworks to help determine an optimal learning rate for training neural networks. It aids in selecting a learning rate that leads to fast and stable convergence during the training process.
The LRfinder module works by gradually increasing the learning rate over a defined range and observing the corresponding loss or accuracy values. It then plots the learning rate against the loss or accuracy values to visualize the behavior and identify an appropriate learning rate.
Here's a general outline of how the LRfinder module typically works:
-
Initialize Model: Set up the neural network model architecture.
-
Define LR Range: Specify a range of learning rates to explore. It usually spans several orders of magnitude, from a very small value to a relatively large value.
-
Train with Varying Learning Rates: Iterate through the specified learning rate range and train the model for a fixed number of iterations or epochs using each learning rate. During training, record the loss or accuracy values.
-
Plot Learning Rate vs. Loss/Accuracy: Visualize the learning rate values on the x-axis and the corresponding loss or accuracy values on the y-axis. This plot helps identify the learning rate range where the loss decreases or the accuracy increases most rapidly.
-
Choose Learning Rate: Based on the plot, select a learning rate that represents the steepest descent of the loss or the highest increase in accuracy before any instability or divergence occurs. This learning rate is typically a value slightly before the point where the loss starts to increase or accuracy starts to plateau.
The LRfinder module is a useful tool for automatically exploring and finding a suitable learning rate without extensive manual tuning. It helps strike a balance between a learning rate that is too small, leading to slow convergence, and a learning rate that is too large, causing unstable training or divergence.
The specific implementation and availability of the LRfinder module can vary depending on the deep learning framework or library being used. Some frameworks provide built-in LRfinder modules, while others may require custom implementation or the use of external libraries specifically designed for learning rate finding.
In the notebook we have used the following parameters for the LR finder:
We found the maximum LR as
The One Cycle Learning Rate (LR) policy is a learning rate scheduling strategy that aims to improve the training process by varying the learning rate over the course of training. It was introduced by Leslie N. Smith in the paper "Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates."
The key idea behind the One Cycle LR policy is to start with a relatively high learning rate, gradually increase it to a maximum value, and then gradually decrease it again towards the end of training. This approach has shown to accelerate the training process, improve model convergence, and achieve better generalization.
The basic steps of the One Cycle LR policy are as follows:
-
Define LR Range: Determine the lower and upper bounds of the learning rate. The lower bound is typically set to a small value, while the upper bound is set to a higher value.
-
Set LR Schedule: Define the LR schedule that varies the learning rate over time. It consists of three phases: increasing LR, decreasing LR, and optionally a final fine-tuning phase.
-
Phase 1: Increasing LR: Start with the minimum learning rate and gradually increase it to the maximum learning rate. This phase is typically set to cover around 30-50% of the total training iterations.
-
Phase 2: Decreasing LR: Gradually decrease the learning rate from the maximum value to the initial minimum value. This phase covers the remaining iterations after Phase 1.
-
Phase 3: Fine-tuning (optional): Optionally, a final phase can be added to further fine-tune the model using a lower learning rate. This phase is typically applied for a smaller number of iterations.
-
-
Apply LR Schedule: During each training iteration, set the learning rate according to the LR schedule defined in Step 2.
The specific implementation of the One Cycle LR policy may vary depending on the deep learning framework or library being used. Most deep learning frameworks provide built-in functionality or libraries that facilitate the implementation of the One Cycle LR policy.
It's worth noting that the One Cycle LR policy is just one of many learning rate scheduling strategies available, and its effectiveness can depend on factors such as the dataset, model architecture, and specific training scenario. It's often recommended to experiment and tune the LR range and schedule parameters to find the best settings for a particular task.
In the notebook, the one cycle LR policy is implemented as
# Define the number of epochs and the max epoch for max learning rate
total_epochs = 24
max_lr_epoch = 5
# Define the learning rate scheduler with max LR from LRFinder
# Max LR is achieved by 5th epoch
# Annealing is set to false by three_phase=False
lr_scheduler = OneCycleLR( optimizer, max_lr=max_LR, steps_per_epoch=len(train_loader), epochs=total_epochs,pct_start=max_lr_epoch/total_epochs, div_factor=100, three_phase=False, final_div_factor=100,anneal_strategy='linear')In the demo notebook we can see that the max LR was achieved at the 5th epoch.
EPOCH: 5
Batch_id=97: 100%|██████████| 98/98 [00:19<00:00, 4.91it/s]
Train Average Loss: 0.7527
Train Accuracy: 73.83%
Maximum Learning Rate: 0.0002475746158006558
Test Average loss: 0.7346
Test Accuracy: 74.54%
Researchers observed that it makes sense to affirm that “the deeper the better” when it comes to convolutional neural networks. This makes sense, since the models should be more capable (their flexibility to adapt to any space increase because they have a bigger parameter space to explore). However, it has been noticed that after some depth, the performance degrades.
One of the problems ResNets solve is the famous known vanishing gradient. This is because when the network is too deep, the gradients from where the loss function is calculated easily shrink to zero after several applications of the chain rule. This result on the weights never updating its values and therefore, no learning is being performed.
Even after resolving the issue of vanishing/exploding gradients, it was observed that training accuracy dropped when the count of layers was increased. This can be seen in the image below.
It is observed that the network having a higher count (56-layer) of layers are resulting in higher training error in contrast to the network having a much lower count (20-layer) of layers thus resulting in higher test errors! Image Credits to the authors of original ResNet paper.
One might assume that this could be the result of overfitting. However, that is not the case here as deeper networks show higher training error not testing errors. Overfitting tends to occur when training errors are significantly lower than test errors.
This is called the degradation problem. With the network depth increasing the accuracy saturates(the networks learns everything before reaching the final layer) and then begins to degrade rapidly if more layers are introduced.
Since neural networks are good function approximators, they should be able to easily solve the identify function, where the output of a function becomes the input itself.
Following the same logic, if we bypass the input to the first layer of the model to be the output of the last layer of the model, the network should be able to predict whatever function it was learning before with the input added to it.
The intuition is that learning f(x) = 0 has to be easy for the network.
Kaiming He, Xiangyu Zhang, Shaoqin Ren, Jian Sun of the Microsoft Research team presented a residual learning framework (ResNets) to help ease the training of the networks that are substantially deeper than before by eliminating the degradation problem. They have proved with evidence that ResNets are easier to optimize and can have high accuracy at considerable depths.
As we have seen previously that latter layers in deeper networks are unable to learn the identity function that is required to carry the result to the output. In residual networks instead of hoping that the layers fit the desired mapping, we let these layers fit a residual mapping.
Initially, the desired mapping is H(x). We let the networks, however, to fit the residual mapping F(x) = H(x)-x, as the network found it easier to optimize the residual mapping rather than the original mapping.
This method of bypassing the data from one layer to another is called as shortcut connections or skip connections. This approach allows the data to flow easily between the layers without hampering the learning ability of the deep learning model. The advantage of adding this type of skip connection is that if any layer hurts the performance of the model, it will be skipped.
The intuition behind the skip connection is that it is easier for the network to learn to convert the value of f(x) to zero so that it behaves like an identity function rather than learning to behave like an identity function altogether on its own by trying to find the right set of values that would give you the result.
ResNet uses two major building blocks to construct the entire network.
-
The Identity Block - The identity block consists of a sequence of convolutional layers with the same number of filters, followed by batch normalization and ReLU activation functions. The key characteristic of the identity block is that it incorporates a shortcut connection that bypasses the convolutional layers. The shortcut connection allows the gradient to flow directly through the block without passing through non-linear activation functions, which helps in preserving the gradient during backpropagation.
-
The Conv Block - Also known as a residual block, is another key building block used to construct deep neural networks. It is designed to facilitate the learning process in very deep networks by addressing the vanishing gradient problem. A convolutional block consists of a series of convolutional layers, batch normalization, and non-linear activation functions, along with a shortcut connection. The main difference between a convolutional block and an identity block is the inclusion of a convolutional layer in the former. The convolutional layer in the block allows the network to learn more complex feature representations.
These components help achieve higher optimization and accuracy for the deep learning models. The results accurately show the effect of using ResNet over plain layers in the graph below.
As seen ResNet performs better than plain neural network models.
The model consists of 5 convolutional blocks. The output block consists of a fully connected layer.
# PREPARATION BLOCK
self.prepblock = nn.Sequential(nn.Conv2d(in_channels=3, out_channels=64, kernel_size=(3, 3), padding=1, dilation=1, stride=1, bias=False),
nn.ReLU(),nn.BatchNorm2d(64))
# output_size = 32, RF=3
# CONVOLUTION BLOCK 1
self.convblock1_l1 = nn.Sequential(nn.Conv2d(in_channels=64, out_channels=128, kernel_size=(3, 3), padding=1, dilation=1, stride=1, bias=False),
# output_size = 32, RF=5
nn.MaxPool2d(2, 2),nn.ReLU(),nn.BatchNorm2d(128))
# output_size = 16, RF=6
self.convblock1_r1 = nn.Sequential(nn.Conv2d(in_channels=128, out_channels=128, kernel_size=(3, 3), padding=1, dilation=1, stride=1, bias=False),nn.ReLU(),nn.BatchNorm2d(128),
# output_size = 16, RF=10
nn.Conv2d(in_channels=128, out_channels=128, kernel_size=(3, 3), padding=1, dilation=1, stride=1, bias=False),nn.ReLU(),nn.BatchNorm2d(128))
# output_size = 16, RF=14
# CONVOLUTION BLOCK 2
self.convblock2_l1 = nn.Sequential(nn.Conv2d(in_channels=128, out_channels=256, kernel_size=(3, 3), padding=1, dilation=1, stride=1, bias=False),
# output_size = 16, RF=18
nn.MaxPool2d(2, 2),nn.ReLU(),nn.BatchNorm2d(256))
# output_size = 8, RF=20
# CONVOLUTION BLOCK 3
self.convblock3_l1 = nn.Sequential(nn.Conv2d(in_channels=256, out_channels=512, kernel_size=(3, 3), padding=1, dilation=1, stride=1, bias=False),
# output_size = 8, RF=28
nn.MaxPool2d(2, 2),
nn.ReLU(),nn.BatchNorm2d(512))
# output_size = 4, RF=32
self.convblock3_r2 = nn.Sequential(nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3, 3), padding=1, dilation=1, stride=1, bias=False),nn.ReLU(),nn.BatchNorm2d(512),
# output_size = 4, RF=48
nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3, 3), padding=1, dilation=1, stride=1, bias=False),nn.ReLU(),nn.BatchNorm2d(512))
# output_size = 4, RF=64
# CONVOLUTION BLOCK 4
self.convblock4_mp = nn.Sequential(nn.MaxPool2d(4))
# output_size = 1, RF = 88
# OUTPUT BLOCK - Fully Connected layer
self.output_block = nn.Sequential(nn.Linear(in_features=512, out_features=10, bias=False))
# output_size = 1, RF = 88
def forward(self, x):
# Preparation Block
x1 = self.prepblock(x)
# Convolution Block 1
x2 = self.convblock1_l1(x1)
x3 = self.convblock1_r1(x2)
x4 = x2 + x3
# Convolution Block 2
x5 = self.convblock2_l1(x4)
# Convolution Block 3
x6 = self.convblock3_l1(x5)
x7 = self.convblock3_r2(x6)
x8 = x7 + x6
# Convolution Block 4
x9 = self.convblock4_mp(x8)
# Output Block
x9 = x9.view(x9.size(0), -1)
x10 = self.output_block(x9)
return F.log_softmax(x10, dim=1)The model architecture consists of several convolutional blocks with varying numbers of layers. The architecture and how it relates to ResNet concepts can be described as:
-
Preparation Block: This block consists of a single convolutional layer followed by ReLU activation and batch normalization. It serves as the initial processing of the input image. -
Convolution Block 1: This block starts with a convolutional layer, followed by a max-pooling layer. Then, another set of convolutional layers is applied. The output of the first convolutional layer is added element-wise to the output of the second set of convolutional layers. This addition introduces a residual connection-like concept. -
Convolution Block 2: This block follows a similar structure to Block 1. It includes a convolutional layer, a max-pooling layer, and no residual connection. -
Convolution Block 3: This block is similar to Block 2 but includes an additional set of convolutional layers. Again, there is no explicit residual connection. -
Convolution Block 4: This block consists of a max-pooling layer only. -
Output Block: The output block includes a fully connected layer that maps the flattened features to the output classes. It uses a log-softmax activation function to produce the final class probabilities.
The calculations for Receptive field and output channel size are shown below:
The final model can be visualized as:
The model summary shows us that we are only using about 173k parameters for this model.
----------------------------------------------------------------
Layer (type) Output Shape Param #
================================================================
Conv2d-1 [-1, 64, 32, 32] 1,728
ReLU-2 [-1, 64, 32, 32] 0
BatchNorm2d-3 [-1, 64, 32, 32] 128
Conv2d-4 [-1, 128, 32, 32] 73,728
MaxPool2d-5 [-1, 128, 16, 16] 0
ReLU-6 [-1, 128, 16, 16] 0
BatchNorm2d-7 [-1, 128, 16, 16] 256
Conv2d-8 [-1, 128, 16, 16] 147,456
ReLU-9 [-1, 128, 16, 16] 0
BatchNorm2d-10 [-1, 128, 16, 16] 256
Conv2d-11 [-1, 128, 16, 16] 147,456
ReLU-12 [-1, 128, 16, 16] 0
BatchNorm2d-13 [-1, 128, 16, 16] 256
Conv2d-14 [-1, 256, 16, 16] 294,912
MaxPool2d-15 [-1, 256, 8, 8] 0
ReLU-16 [-1, 256, 8, 8] 0
BatchNorm2d-17 [-1, 256, 8, 8] 512
Conv2d-18 [-1, 512, 8, 8] 1,179,648
MaxPool2d-19 [-1, 512, 4, 4] 0
ReLU-20 [-1, 512, 4, 4] 0
BatchNorm2d-21 [-1, 512, 4, 4] 1,024
Conv2d-22 [-1, 512, 4, 4] 2,359,296
ReLU-23 [-1, 512, 4, 4] 0
BatchNorm2d-24 [-1, 512, 4, 4] 1,024
Conv2d-25 [-1, 512, 4, 4] 2,359,296
ReLU-26 [-1, 512, 4, 4] 0
BatchNorm2d-27 [-1, 512, 4, 4] 1,024
MaxPool2d-28 [-1, 512, 1, 1] 0
Linear-29 [-1, 10] 5,120
================================================================
Total params: 6,573,120
Trainable params: 6,573,120
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.01
Forward/backward pass size (MB): 6.44
Params size (MB): 25.07
Estimated Total Size (MB): 31.53
----------------------------------------------------------------
From the plots below we can see that we have achieved accuracy greater than 90% from about
EPOCH: 20
Batch_id=97: 100%|██████████| 98/98 [00:19<00:00, 4.91it/s]
Train Average Loss: 0.2473
Train Accuracy: 91.82%
Maximum Learning Rate: 5.2035511983067946e-05
Test Average loss: 0.3112
Test Accuracy: 89.12%
EPOCH: 21
Batch_id=97: 100%|██████████| 98/98 [00:19<00:00, 4.94it/s]
Train Average Loss: 0.2125
Train Accuracy: 92.91%
Maximum Learning Rate: 3.89995717285621e-05
Test Average loss: 0.2745
Test Accuracy: 90.98%
EPOCH: 22
Batch_id=97: 100%|██████████| 98/98 [00:19<00:00, 4.93it/s]
Train Average Loss: 0.1768
Train Accuracy: 94.22%
Maximum Learning Rate: 2.5963631474056227e-05
Test Average loss: 0.2578
Test Accuracy: 91.47%
EPOCH: 23
Batch_id=97: 100%|██████████| 98/98 [00:20<00:00, 4.82it/s]
Train Average Loss: 0.1339
Train Accuracy: 95.75%
Maximum Learning Rate: 1.2927691219550408e-05
Test Average loss: 0.2337
Test Accuracy: 92.31%
EPOCH: 24
Batch_id=97: 100%|██████████| 98/98 [00:20<00:00, 4.81it/s]
Train Average Loss: 0.0974
Train Accuracy: 97.09%
Maximum Learning Rate: -1.0824903495543778e-07
Test Average loss: 0.2182
Test Accuracy: 92.86%
Now we shall look at the misclassified images that are present after applying the model on the dataset.
In conclusion, David C Page's CIFAR-10 Fast model is a custom ResNet-inspired architecture designed for image classification on the CIFAR-10 dataset. The model incorporates multiple convolutional blocks with increasing complexity to capture hierarchical features and facilitate learning.
While the CIFAR-10 Fast model shares some similarities with the original ResNet architecture, it deviates from the exact structure and skip connection principles. Instead, it employs convolutional blocks with residual connections-like additions in certain stages.
The model follows a sequential flow, beginning with a preparation block for initial image processing. It then proceeds through multiple convolutional blocks, with some blocks incorporating max-pooling layers. The output block consists of a max-pooling layer followed by a fully connected layer for class prediction.
It's important to note that the model summary and architectural details were provided based on the code snippet shared. However, the CIFAR-10 Fast model may undergo further modifications or variations beyond the provided code.
The LRfinder module provides a systematic approach to finding an optimal learning rate for training neural networks. By gradually increasing the learning rate and observing the corresponding loss or accuracy values, the LRfinder module helps identify the learning rate range that leads to faster convergence and better generalization. This automated process eliminates the need for manual trial-and-error tuning of the learning rate and saves considerable time and effort.
Once an appropriate learning rate range is identified using the LRfinder module, the One Cycle LR policy can be applied to further enhance the training process. The One Cycle LR policy dynamically adjusts the learning rate during training, starting with a relatively high learning rate, gradually increasing it to a maximum value, and then gradually decreasing it. This policy has been shown to accelerate convergence, improve model stability, and achieve better generalization.
By combining the LRfinder module and One Cycle LR policy, deep learning practitioners can efficiently discover an optimal learning rate and leverage it during the training process. This approach helps overcome the challenges of manual learning rate tuning and enhances the training dynamics, leading to improved model performance and faster convergence.
However, it is important to note that the effectiveness of these techniques may vary depending on the specific dataset, model architecture, and training scenario. It is recommended to experiment and fine-tune the LR range and schedule parameters to find the optimal settings for a given task.