⚠ Official Notice: www.ijisrt.com is the official website of the International Journal of Innovative Science and Research Technology (IJISRT) Journal for research paper submission and publication. Please beware of fake or duplicate websites using the IJISRT name.



Real-Time French Speech Translation into Four Congolese National Languages Using Whisper and Language-Specific LoRA Adapters for NLLB-200


Authors : Jean-Christophe B. Wizamo; Selain K. Kasereka

Volume/Issue : Volume 11 - 2026, Issue 11 - November


Google Scholar : https://tinyurl.com/mw9phzz

DOI : https://doi.org/10.38124/ijisrt/26nov245

Note : A published paper may take 4-5 working days from the publication date to appear in PlumX Metrics, Semantic Scholar, and ResearchGate.


Abstract : French is the sole official language of the Democratic Republic of the Congo, yet much of the population communicates daily in Lingala, Swahili, Tshiluba or Kikongo. This paper presents a cascaded system that transcribes spoken French with Whisper and translates it with NLLB-200-distilled-600M, specialized by one Low-Rank Adaptation (LoRA) adapter per target language on a shared frozen base model.

Keywords : Low-Resource Machine Translation; Speech Translation; Whisper; NLLB-200; LoRA; Lingala; Swahili; Tshiluba; Kikongo; Edge AI.

References :

  1. République Démocratique du Congo, Constitution de la République Démocratique du Congo. Kinshasa, Democratic Republic of the Congo: Journal Officiel de la RDC, Feb. 18, 2006, art. 1.
  2. R. E. Mandiya et al., “A transformer-based method for bidirectional French–Lingala machine translation in speech and text,” Appl. Sci., vol. 16, no. 7, art. no. 3399, 2026, doi: 10.3390/app16073399.
  3. W. Nekoto et al., “Participatory research for low-resourced machine translation: A case study in African languages,” in Findings of the Association for Computational Linguistics: EMNLP 2020, Online, 2020, pp. 2144–2160.
  4. A. Radford et al., “Robust speech recognition via large-scale weak supervision,” in Proc. 40th Int. Conf. Machine Learning (ICML), Honolulu, HI, USA, 2023, vol. 202, pp. 28492–28518.
  5. NLLB Team et al., “No language left behind: Scaling human-centered machine translation,” arXiv:2207.04672, 2022.
  6. Ž. Agić and I. Vulić, “JW300: A wide-coverage parallel corpus for low-resource languages,” in Proc. 57th Annu. Meeting of the Association for Computational Linguistics (ACL), Florence, Italy, 2019, pp. 3204–3210.
  7. Y. Mao, C. You, J. Zhang, K. Huang, and K. B. Letaief, “A survey on mobile edge computing: The communication perspective,” IEEE Commun. Surveys Tuts., vol. 19, no. 4, pp. 2322–2358, 2017.
  8. J. Kreutzer et al., “Quality at a glance: An audit of web-crawled multilingual datasets,” Trans. Assoc. Comput. Linguistics, vol. 10, pp. 50–72, 2022.
  9. E. J. Hu et al., “LoRA: Low-rank adaptation of large language models,” in Proc. 10th Int. Conf. Learning Representations (ICLR), Virtual, 2022.
  10. B. Jacob et al., “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 2018, pp. 2704–2713.
  11. A. Gholami et al., “A survey of quantization methods for efficient neural network inference,” arXiv:2103.13630, 2021.
  12. A. Bérard, O. Pietquin, C. Servan, and L. Besacier, “Listen and translate: A proof of concept for end-to-end speech-to-text translation,” in Proc. NIPS Workshop on End-to-End Learning for Speech and Audio Processing, Barcelona, Spain, 2016.
  13. R. J. Weiss, J. Chorowski, N. Jaitly, Y. Wu, and Z. Chen, “Sequence-to-sequence models can directly translate foreign speech,” in Proc. Interspeech, Stockholm, Sweden, 2017, pp. 2625–2629.
  14. M. Sperber and M. Paulik, “Speech translation and the end-to-end promise: Taking stock of where we are,” in Proc. 58th Annu. Meeting of the Association for Computational Linguistics (ACL), Online, 2020, pp. 7409–7421.
  15. Seamless Communication et al., “SeamlessM4T: Massively multilingual and multimodal machine translation,” arXiv:2308.11596, 2023.
  16. V. Pratap et al., “Scaling speech technology to 1,000+ languages,” arXiv:2305.13516, 2023.
  17. M. Ma et al., “STACL: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework,” in Proc. 57th Annu. Meeting of the Association for Computational Linguistics (ACL), Florence, Italy, 2019, pp. 3025–3036.
  18. D. Macháček, R. Dabre, and O. Bojar, “Turning Whisper into real-time transcription system,” in Proc. 13th Int. Joint Conf. Natural Language Processing and 3rd Conf. Asia-Pacific Chapter of the ACL: System Demonstrations, Bali, Indonesia, 2023.
  19. I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Advances in Neural Information Processing Systems, vol. 27, 2014.
  20. D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in Proc. 3rd Int. Conf. Learning Representations (ICLR), San Diego, CA, USA, 2015.
  21. A. Vaswani et al., “Attention is all you need,” in Advances in Neural Information Processing Systems, vol. 30, 2017, pp. 5998–6008.
  22. M. Johnson et al., “Google’s multilingual neural machine translation system: Enabling zero-shot translation,” Trans. Assoc. Comput. Linguistics, vol. 5, pp. 339–351, 2017.
  23. N. Arivazhagan et al., “Massively multilingual neural machine translation in the wild: Findings and challenges,” arXiv:1907.05019, 2019.
  24. A. Fan et al., “Beyond English-centric multilingual machine translation,” J. Mach. Learn. Res., vol. 22, pp. 1–48, 2021.
  25. M. Artetxe and H. Schwenk, “Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond,” Trans. Assoc. Comput. Linguistics, vol. 7, pp. 597–610, 2019.
  26. N. Goyal et al., “The Flores-101 evaluation benchmark for low-resource and multilingual machine translation,” Trans. Assoc. Comput. Linguistics, vol. 10, pp. 522–538, 2022.
  27. D. I. Adelani et al., “A few thousand translations go a long way! Leveraging pre-trained models for African news translation,” in Proc. 2022 Conf. North American Chapter of the ACL: Human Language Technologies (NAACL-HLT), Seattle, WA, USA, 2022, pp. 3053–3070.
  28. N. Houlsby et al., “Parameter-efficient transfer learning for NLP,” in Proc. 36th Int. Conf. Machine Learning (ICML), Long Beach, CA, USA, 2019, vol. 97, pp. 2790–2799.
  29. A. Bapna and O. Firat, “Simple, scalable adaptation for neural machine translation,” in Proc. 2019 Conf. Empirical Methods in Natural Language Processing and 9th Int. Joint Conf. Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 2019, pp. 1538–1548.
  30. M. Nagel et al., “A white paper on neural network quantization,” arXiv:2106.08295, 2021.
  31. O. Zafrir, G. Boudoukh, P. Izsak, and M. Wasserblat, “Q8BERT: Quantized 8Bit BERT,” in Proc. 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing (EMC2-NeurIPS), Vancouver, BC, Canada, 2019, pp. 36–39.
  32. Y. Bondarenko, M. Nagel, and T. Blankevoort, “Understanding and overcoming the challenges of efficient Transformer quantization,” in Proc. 2021 Conf. Empirical Methods in Natural Language Processing (EMNLP), Punta Cana, Dominican Republic, 2021, pp. 7947–7969.
  33. G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv:1503.02531, 2015.
  34. S. Gandhi, P. von Platen, and A. M. Rush, “Distil-Whisper: Robust knowledge distillation via large-scale pseudo labelling,” arXiv:2311.00430, 2023.
  35. ONNX Runtime Developers, “Quantize ONNX models,” ONNX Runtime documentation. [Online]. Available: https://onnxruntime.ai/docs/performance/model-optimizations/quantization.html (accessed Oct. 21, 2026).
  36. M. T. Mukungu, A. M. Mbayandjambe, T. Tashev, K. Kyamakya, and S. K. Kasereka, “BabaSpeech: A deep learning-based translation of sign language into Lingala text and speech for deaf-mute inclusivity,” in Proc. Int. Joint Conf. Artificial Intelligence, Singapore: Springer, 2025, pp. 91–105.
  37. A. M. Mbayandjambe, S. K. Kasereka, K. Kyamakya, and V. T. Ho, “Enhancing printed Lingala script recognition using deep learning techniques,” Procedia Comput. Sci., vol. 257, pp. 111–118, 2025.
  38. Silero Team, “Silero VAD: Pre-trained enterprise-grade voice activity detector,” GitHub repository, 2021. [Online]. Available: https://github.com/snakers4/silero-vad
  39. J. Tiedemann, “The Tatoeba translation challenge – Realistic data sets for low resource and multilingual MT,” in Proc. 5th Conf. Machine Translation (WMT), Online, 2020, pp. 1174–1182.
  40. A. Anastasopoulos et al., “TICO-19: The translation initiative for COvid-19,” in Proc. 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Online, 2020.
  41. J. Tiedemann, “Parallel data, tools and interfaces in OPUS,” in Proc. 8th Int. Conf. Language Resources and Evaluation (LREC), Istanbul, Turkey, 2012, pp. 2214–2218.
  42. T. Kudo and J. Richardson, “SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,” in Proc. 2018 Conf. Empirical Methods in Natural Language Processing: System Demonstrations, Brussels, Belgium, 2018, pp. 66–71.
  43. R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in Proc. 54th Annu. Meeting of the Association for Computational Linguistics (ACL), Berlin, Germany, 2016, pp. 1715–1725.
  44. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. 7th Int. Conf. Learning Representations (ICLR), New Orleans, LA, USA, 2019.
  45. S. Migacz, “8-bit inference with TensorRT,” presented at the GPU Technology Conf., San Jose, CA, USA, May 2017.
  46. A. Abid et al., “Gradio: Hassle-free sharing and testing of ML models in the wild,” arXiv:1906.02569, 2019.
  47. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: A method for automatic evaluation of machine translation,” in Proc. 40th Annu. Meeting of the Association for Computational Linguistics (ACL), Philadelphia, PA, USA, 2002, pp. 311–318.
  48. M. Popović, “chrF: Character n-gram F-score for automatic MT evaluation,” in Proc. 10th Workshop on Statistical Machine Translation (WMT), Lisbon, Portugal, 2015, pp. 392–395.
  49. M. Post, “A call for clarity in reporting BLEU scores,” in Proc. 3rd Conf. Machine Translation: Research Papers (WMT), Brussels, Belgium, 2018, pp. 186–191.
  50. T. Kocmi et al., “To ship or not to ship: An extensive evaluation of automatic metrics for machine translation,” in Proc. 6th Conf. Machine Translation (WMT), Online, 2021, pp. 478–494.
  51. T. Wolf et al., “Transformers: State-of-the-art natural language processing,” in Proc. 2020 Conf. Empirical Methods in Natural Language Processing: System Demonstrations, Online, 2020, pp. 38–45.
  52. S. Mangrulkar et al., “PEFT: State-of-the-art parameter-efficient fine-tuning methods,” GitHub repository, 2022. [Online]. Available: https://github.com/huggingface/peft.

French is the sole official language of the Democratic Republic of the Congo, yet much of the population communicates daily in Lingala, Swahili, Tshiluba or Kikongo. This paper presents a cascaded system that transcribes spoken French with Whisper and translates it with NLLB-200-distilled-600M, specialized by one Low-Rank Adaptation (LoRA) adapter per target language on a shared frozen base model.

Keywords : Low-Resource Machine Translation; Speech Translation; Whisper; NLLB-200; LoRA; Lingala; Swahili; Tshiluba; Kikongo; Edge AI.

Paper Submission Last Date
30 - November - 2026

SUBMIT YOUR PAPER CALL FOR PAPERS
Video Explanation for Published paper

Never miss an update from Papermashup

Get notified about the latest tutorials and downloads.

Subscribe by Email

Get alerts directly into your inbox after each post and stay updated.
Subscribe
OR

Subscribe by RSS

Add our RSS to your feedreader to get regular updates from us.
Subscribe