One of the most watched and a played sport is cricket, especially in South Asian countries. In cricket, umpire has the power for making significant decisions about events in the field. With the growing increase of the utilization of technology in sports, this paper presents the umpire detection and classification by proposing an optimization algorithm. The overall procedure of the proposed approach involves three steps, like segmentation, feature extraction, and the classification. At first, the video frames are extracted from input cricket video, and the segmentation is performed based on Viola-Jones algorithm. Once the segmentation is done, the feature extraction is carried out using Histogram of Oriented Gradients (HOG), and Fuzzy Local Gradient Patterns (Fuzzy LGP). Finally, the extracted features are given to the classification step. Here, the classification is done using the proposed Bird Swarm Optimization-based stacked auto encoder deep learning classifier (BSO-Stacked Autoencoders), that categories into umpire or others. The performance of the umpire detection and classification based on BSO-Stacked Autoencoders is evaluated based on sensitivity, specificity, and accuracy. The proposed BSO-Stacked Autoencoder method achieves the maximal accuracy of 96.562%, the maximal sensitivity of 91.884%, and the maximal specificity of 99%, that indicates its superiority.