This article states the fully automatic tomato pest image classification system using modified deep learning algorithm. It contains the training of the pest image classification system and the testing the proposed pest image classification system. The training system contains data augmentation along with the modified Visual Geometry Group (VGG) – ConvolutionalNeural Networks (CNN) algorithm. The purpose of the training system is to generate the training values from the good and bad pest tomato images. After generating the training values from the good and bad case tomato pest images, the testing system is activated with the generated training values. In this case, the test tomato pest image is directly fed into the proposed VGG-CNN algorithm along with the generated training values from the training system to produce the pest image results. This system has been checked and validated by the two tomato plant imaging datasets.