Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures

Major winning Convolutional Neural Networks (CNNs), such as AlexNet, <PRE_TAG>VGGNet</POST_TAG>, ResNet, GoogleNet, include tens to hundreds of millions of parameters, which impose considerable computation and memory overhead. This limits their practical use for training, optimization and memory efficiency. On the contrary, light-weight <PRE_TAG>architectures</POST_TAG>, being proposed to address this issue, mainly suffer from low accuracy. These inefficiencies mostly stem from following an ad hoc procedure. We propose a simple architecture, called SimpleNet, based on a set of designing principles, with which we empirically show, a well-crafted yet simple and reasonably deep architecture can perform on par with deeper and more complex architectures. SimpleNet provides a good tradeoff between the computation/memory efficiency and the accuracy. Our simple 13-layer architecture outperforms most of the deeper and complex architectures to date such as <PRE_TAG>VGGNet</POST_TAG>, ResNet, and GoogleNet on several well-known benchmarks while having 2 to 25 times fewer number of parameters and operations. This makes it very handy for embedded systems or systems with computational and memory limitations. We achieved state-of-the-art result on CIFAR10 outperforming several heavier architectures, near state of the art on MNIST and competitive results on <PRE_TAG>CIFAR100</POST_TAG> and SVHN. We also outperformed the much larger and deeper architectures such as <PRE_TAG>VGGNet</POST_TAG> and popular variants of <PRE_TAG>ResNets</POST_TAG> among others on the ImageNet dataset. Models are made available at: https://github.com/Coderx7/SimpleNet