Activation Function Changes

Base Model:

class Challenge(nn.Module):
    def __init__(self, block):
        super(Challenge, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, 5, (2,2), 2)
        self.conv2 = nn.Conv2d(16, 64, 5, (2,2), 2)
        self.conv3 = nn.Conv2d(64, 32, 5, (2,2), 2)
        self.fc1 = nn.Linear(512, 64)
        self.fc2 = nn.Linear(64, 32)
        self.fc3 = nn.Linear(32, 5)

    def forward(self, x):
        x = F.relu_(self.conv1(x))
        x = F.relu_(self.conv2(x))
        x = F.relu_(self.conv3(x))
        x = torch.flatten(x, start_dim=1)
        x = F.relu_(self.fc1(x))
        x = F.relu_(self.fc2(x))
        z = self.fc3(x)
        return z

changed all activation function to elu:

changed all activation function to tanh:

changed all activation function to leakey_relu:

→ Seems like leaky_ReLU is best at increasing accuracy. While tanh does converge faster, it isn’t good at increasing accuracy.

Additional Fully Connected Layers with Bottle neck design