Deep learning constitutes a recent, modern technique for image processing and data analysis, with promising results and large potential. This allows larger learning capabilities and thus higher performance and precision. As deep learning has been successfully applied in various domains, it has recently entered also the domain of agriculture. This paper proposes and in this experimentally demonstrates fault tolerant optical penetration-based silkworm gender identification. The key idea lies in the exploitation of the inherent dual wavelength of white and red light illumination. In particular, the image of the posterior area of the silkworm pupa created under white light is not only transformed into an optical region-of-interest but also is used for pinpointing the female silkworm pupa, thus speeding up the identification time twice. For the male and unidentified female silkworm pupae, their images are later on analyzed under red light illumination, implying fault-tolerant operation of the system.