⚠ Official Notice: www.ijisrt.com is the official website of the International Journal of Innovative Science and Research Technology (IJISRT) Journal for research paper submission and publication. Please beware of fake or duplicate websites using the IJISRT name.



Chinese Lip-Reading Based on Residual Network and Attention Mechanism


Authors : Chuan Qin; ZiLong Liu

Volume/Issue : Volume 11 - 2026, Issue 11 - November


Google Scholar : https://tinyurl.com/mr46c6tb

DOI : https://doi.org/10.38124/ijisrt/26nov599

Note : A published paper may take 4-5 working days from the publication date to appear in PlumX Metrics, Semantic Scholar, and ResearchGate.


Abstract : To address the common issues in existing lip-reading recognition systems, such as insufficient feature extraction and visual ambiguities, a Chinese sentence-level lip-reading recognition system based on residual networks and attention mechanisms (Spatio-Temporal Cascade Network, STCNet) is proposed. This system can first translate continuous videos containing speakers into pinyin sequences sentence by sentence, and then convert them into continuous Chinese sentences. Firstly, the residual network in the front-end feature extraction module can achieve bidirectional transmission and fusion of multi-level features, enhancing the extraction capability of long-term temporal features. In the back-end pinyin classification module, a Temporal Convolutional Network (TCN) is used to extract long-term dependency information, and an attention mechanism is introduced to weight discriminative features at different time intervals to reduce visual ambiguities

Keywords : Lip-Reading; Feature Extraction; Residual Network; Attention Mechanism; Deep Neural Network.

References :

  1. Y. Q. Ren, G. D. Tian, X. D. Zhou, et al., "Lip-reading algorithm in face recognition systems for high security," Application Research of Computers 34(4), 1221-1225+1230 (2017).
  2. J. H. Lei, "Experimental study on lip-reading Chinese character pronunciation recognition in hearing-impaired students," Dissertation, East China Normal University, Shanghai (2006).
  3. Y. M. Wang, K. Chen, Aihaitilamu Saabdusalam, "End-to-end audio-visual speech recognition based on SDBN and BLSTM attention fusion," Telecommunications Science 35(12), 79-89 (2019).
  4. J. S. Chung, A. Zisserman, "Lip reading in profile," in Proc. 28th British Machine Vision Conference, London, UK, ChungZ17 (2017).
  5. S. Yang, Y. Zhang, D. Feng, et al., "LRW1000: A naturally-distributed large-scale benchmark for lip reading in the wild," in 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), 1-8, IEEE (2019).
  6. T. Afouras, J. S. Chung, A. Senior, et al., "Deep audio-visual speech recognition," IEEE Trans. Pattern Anal. Mach. Intell. 44(12), 8717-8727 (2018).
  7. X. Zhang, H. Gong, X. Dai, et al., "Understanding pictograph with facial features: End-to-end sentence-level lip reading of Chinese," in Proc. AAAI Conf. Artif. Intell. 33(01), 9211-9218 (2019).
  8. Y. Yao, Z. X. Hu, T. Deng, et al., "A lip reading method based on adaptive pooling attention transformer," Chinese Journal of Intelligent Science and Technology 7(2), 211-220 (2025).
  9. K. Xu, D. W. Li, N. Cassimatis, et al., "LCANet: End-to-end lipreading with cascaded attention-CTC," in Proc. 2018 13th IEEE Int. Conf. Autom. Face Gesture Recognit. (FG 2018), 548-555, IEEE Press, Piscataway (2018).
  10. T. Stafylakis, G. Tzimiropoulos, "Combining residual networks with LSTMs for lipreading," arXiv preprint (2017) [doi:arXiv:1703.04105].
  11. T. Zhang, L. He, X. D. Li, et al., "Efficient end-to-end sentence-level lip reading with temporal convolutional networks," Appl. Sci. 11(15), 6975-6982 (2021).
  12. P. C. Ma, B. Martinez, S. Petridis, et al., "Towards practical lipreading with distilled and efficient models," in Proc. ICASSP 2021 - 2021 IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), 7608-7612, IEEE Press, Piscataway (2021).
  13. S. N. Xie, R. Girshick, P. Dollár, et al., "Aggregated residual transformations for deep neural networks," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 5987-5995, IEEE Computer Society Press, Los Alamitos (2017).
  14. I. C. Duta, L. Liu, F. Zhu, et al., "Improved residual networks for image and video recognition," in Proc. 25th Int. Conf. Pattern Recognit., 9415-9422, IEEE Computer Society Press, Los Alamitos (2021).
  15. J. L. Ma, Z. W. Guo, Z. P. Ma, et al., "Multi-scale gated spatio-temporal enhancement for lip recognition," Journal of Computer-Aided Design & Computer Graphics 37(7), 1229-1238 (2025).
  16. Y. M. Assael, B. Shillingford, S. Whiteson, et al., "LipNet: End-to-end sentence-level lipreading," arXiv preprint (2016) [doi:arXiv:1611.01599].
  17. J. S. Chung, A. Senior, O. Vinyals, et al., "Lip reading sentences in the wild," in Proc. 2017 IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 3444-3453, IEEE Press, Piscataway (2017).
  18. B. X. Wang, "Design and development of a lip-reading recognition system based on multi-feature fusion inference," Dissertation, Beijing University of Posts and Telecommunications, Beijing (2025) [doi:10.26969/d.cnki.gbydu.2025.002166].
  19. W. L. Wu, R. K. Li, S. Y. Xu, et al., "End-to-end lip-reading algorithm based on adaptive sequence frame length," Life Science Instruments 21(4), 35-39 (2023).
  20. Y. Y. Wang, P. Wang, K. C. Wu, "Research on lip reading based on self-attention sequence model," Chinese Journal of Electron Devices 44(3), 624-627 (2021).
  21. C. Wang, "Research on lip reading methods based on deep learning," Dissertation, Shenyang University of Technology, Shenyang (2025) [doi:10.27322/d.cnki.gsgyu.2025.000966].
  22. M. Miao, "Research on multimodal Chinese lip reading methods," Dissertation, Inner Mongolia University, Hohhot (2025) [doi:10.27224/d.cnki.gnmdu.2025.001040].
  23. Y. W. Zhang, "Design and implementation of a lip-reading algorithm based on multi-scale feature fusion," Dissertation, Beijing University of Posts and Telecommunications, Beijing (2025) [doi:10.26969/d.cnki.gbydu.2025.001578].
  24. W. J. Xu, K. Li, "Sentence-level lip reading based on optical flow and attention mechanism," Chinese Journal of Electron Devices 46(5), 1339-1348 (2023).
  25. J. He, X. Wang, "Research progress on Chinese lip-reading corpus," Journal of Information Engineering University 26(6), 690-696 (2025).
  26. C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, et al., "300 faces in-the-wild challenge: The first facial landmark localization challenge," in Proc. IEEE Int. Conf. Comput. Vis. Workshops (2013).
  27. A. Cutler, Prosody and the word boundary problem, Psychology Press, Germany (2014).
  28. P. P. Liu, "Research on Chinese sentence-level lip reading based on Transformer," Dissertation, North China Electric Power University (Beijing), Beijing (2023) [doi:10.27140/d.cnki.ghbbu.2023.001292].
  29. P. P. Liu, J. P. Jia, "Chinese sentence-level lip reading algorithm based on temporal convolutional network," Application Research of Computers 40(9), 2596-2602 (2023) [doi:10.19734/j.issn.1001-3695.2023.02.0051].
  30. Y. Zhao, R. Xu, M. Song, "A cascade sequence-to-sequence model for Chinese Mandarin lip reading," in Proc. ACM Multimedia Asia, 1-6, ACM, New York (2019).
  31. S. He, J. B. Yuan, Y. Y. Lu, "Research on lip-reading method based on visual characteristics of Chinese pronunciation," Computer Engineering and Applications 58(4), 157-162 (2022).

To address the common issues in existing lip-reading recognition systems, such as insufficient feature extraction and visual ambiguities, a Chinese sentence-level lip-reading recognition system based on residual networks and attention mechanisms (Spatio-Temporal Cascade Network, STCNet) is proposed. This system can first translate continuous videos containing speakers into pinyin sequences sentence by sentence, and then convert them into continuous Chinese sentences. Firstly, the residual network in the front-end feature extraction module can achieve bidirectional transmission and fusion of multi-level features, enhancing the extraction capability of long-term temporal features. In the back-end pinyin classification module, a Temporal Convolutional Network (TCN) is used to extract long-term dependency information, and an attention mechanism is introduced to weight discriminative features at different time intervals to reduce visual ambiguities

Keywords : Lip-Reading; Feature Extraction; Residual Network; Attention Mechanism; Deep Neural Network.

Paper Submission Last Date
30 - November - 2026

SUBMIT YOUR PAPER CALL FOR PAPERS
Video Explanation for Published paper

Never miss an update from Papermashup

Get notified about the latest tutorials and downloads.

Subscribe by Email

Get alerts directly into your inbox after each post and stay updated.
Subscribe
OR

Subscribe by RSS

Add our RSS to your feedreader to get regular updates from us.
Subscribe