The outcome of endoscopic tasks can be significantly affected by the presence of specular reflections. Although numerous methods have been proposed for specular reflection detection and suppression from endoscopic images, they are inefficient, usually require tedious empirical parameters selection, and perform poorly when handling large specular regions. To this end, we propose a robust and efficient deep learning framework termed EasySpec for identifying and suppressing specular reflections from endoscopic images. Our proposed EasySpec consists of two stages: a detection stage and a suppression stage. The former stage is performed using a lightweight UNet-variant dubbed Scaled-UNet, which is trained exploiting a novel hybrid strategy that integrates the advantages of both transfer learning and weakly supervised techniques. The latter stage is achieved utilizing the concept of deep image inpainting. Specifically, a new and fast end-to-end approach for inpainting multi-size specular reflection regions in endoscopic images is developed. Our proposed approach termed GatedResUNet, takes advantage of the gated convolution to accurately differentiate specular pixels from non-specular pixels, and the U-Net architecture to learn multi-level semantic representative features, which helps reconstruct more realistic and semantically plausible images. Extensive qualitative and quantitative experiments on several endoscopic datasets acquired from different MIS scenes reveal that our proposed EasySpec outperforms state-of-the-art endoscopic specular reflection suppression approaches. Specifically, EasySpec generalizes well on various MIS scenes, and can properly reconstruct blood vessels and fine details in a reasonable processing time, which validates its practical significance.