Breast cancer is a deadly disease that affects the lives of millions of women throughout the world. Over time, the number of cases of breast cancer has increased. Preventing this disease is difficult and remains unidentified, but the survival percentage can be improved if diagnosed early. The progress of computer-assisted diagnosis (CAD) of breast cancer has seen a lot of improvements thanks to advances in deep learning. With the notable advancement of deep neural networks, diagnostic capabilities are nearing a human expert's. In this paper, we used EfficientNet to classify mammograms. This model is introduced with the new concept of model scaling called compound scaling. Compound scaling is the strategy which scales the model by adding more layers to extend the receptive field along with more channels to catch the detailed features of larger input. We also compare the performance of various variants of EfficientNet over CBIS-DDSM mammogram datasets. We used the optimum fine-tuning procedure to represent the importance of transfer learning (TL) during training.