Authors : Anushka Athulathmudali

Volume/Issue : Volume 10 - 2025, Issue 12 - December

Google Scholar : https://tinyurl.com/422jj5mn

Scribd : https://tinyurl.com/bddeftf9

DOI : https://doi.org/10.38124/ijisrt/25dec441

Note : A published paper may take 4-5 working days from the publication date to appear in PlumX Metrics, Semantic Scholar, and ResearchGate.

Note : Google Scholar may take 30 to 40 days to display the article.

Abstract : Sri Lanka’s pluralistic legal system is difficult for many citizens to navigate, leading to missed deadlines and reduced access to justice. This research investigates whether Retrieval-Augmented Generation (RAG) with Large Language Models (LLMs) can provide accurate, cost-efficient civil-law guidance tailored to Sri Lanka. A curated dataset of expert-validated legal scenarios was developed and integrated into a modular RAG pipeline. Two LLM backends, GPT- 3.5-Turbo and Mistral-7B-v0.1 were evaluated under identical conditions for legal accuracy, latency, and cost. Results show that GPT-3.5-Turbo achieved the best overall performance with 92.5% accuracy, 4.17s latency, lowest cost per correct response, making it suitable for responsive public-facing legal-information services. Mistral-7B-v0.1 demonstrated competitive accuracy of 82.5% with full data-sovereignty benefits, but higher latency limits interactive deployment, better aligning it with institutional environments prioritizing privacy and local infrastructure control. The study provides a practical framework for responsible legal-AI adoption in Sri Lanka and contributes a reusable RAG architecture and localized dataset. These findings suggest that AI-enabled legal guidance can support access-to-justice goals when deployed with appropriate safeguards and governance.

Keywords : Retrieval-Augmented Generation (RAG), Large Language Models (LLMs), Legal Corpus, Procedural Accuracy Evaluation, Latency and Cost Profiling, GPT-3.5-Turbo; Mistral-7B, Legal Hallucination Mitigation, Access to Justice in Sri Lanka.

References :

1. “Legal framework in Sri Lanka,” aplawjapan, Jun. 2024 [Accessed: 20 January 2025]
2. “The Constitution of the Democratic Socialist Republic of Sri Lanka,”. [Online]. Available: https://parliament.lk/files/pdf/constitution.pdf
3. Jayasinghe, I. Dilshani, and I. Weerasekara, “Provide Equal Access to Justice and Enhance Legal Awareness Through the Use of ICT,” ResearchGate, Nov. 26, 2021. https://www.researchgate.net/publication/369358154_Provide_Equal_Access_to_Justice_and_Enhance_Legal_Awareness_Through_the_Use_of_ICT [Accessed: 08 June 2025]
4. W. X. Zhao et al., “A survey of large language models,” arXiv.org, Mar. 31, 2023. https://arxiv.org/abs/2303.18223 [Accessed: 19 February 2025]
5. P. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP tasks,” arXiv.org, May 22, 2020. https://arxiv.org/abs/2005.11401 [Accessed: 08 June 2025]
6. Dasanayake, C.G. (2024) ‘Evaluating the Use of Artificial Intelligence for an Effective Justice System in Sri Lanka’. Available: https://ir.kdu.ac.lk/bitstream/handle/345/7608/LAWJ%20Vol4%20Iss2_2.pdf [Accessed: 07 June 2025]
7. S. Liu, Z. Li, R. Ma, H. Zhao, and M. Du, “ContractEval: Benchmarking LLMs for Clause-Level legal risk identification in commercial contracts,” arXiv.org, Aug. 05, 2025. https://arxiv.org/abs/2508.03080 [Accessed: 02 July 2025]
8. J. Wang et al., “Legal evalutions and challenges of large language models,” arXiv.org, Nov. 15, 2024. https://arxiv.org/abs/2411.10137 [Accessed: 17 August 2025]
9. S. A. Thorat and V. Jadhav, “A review on implementation issues of rule-based chatbot systems,” SSRN Electronic Journal, Jan. 2020, doi: 10.2139/ssrn.3567047 [Accessed: 08 June 2025]
10. S. K. Ojha, A. Kumar, T. Bhole, S. Naaz, and CSE Department Sharda University Greater Noida, “Rule-based A.I. chatbot,” INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH IN TECHNOLOGY, vol. 11, no. 5, pp. 639–640, 2024, [Online]. Available: https://ijirt.org/publishedpaper/IJIRT168361_PAPER.pdf [Accessed: 13 June 2025]
11. J. Williamson, “The rise of AI in legal practice: Opportunities, challenges, & ethical considerations,” Mar. 21, 2025. https://ctlj.colorado.edu/?p=1297 [Accessed: 17 June 2025]
12. O. Isaac and N. Johnson, “The Use of Chatbots in Providing Free Legal Guidance: Benefits and Limitations,” Researchgate.net. [Online]. Available: https://www.researchgate.net/publication/388179232_The_Use_of_Chatbots_in_Providing_Free_Legal_Guidance_Benefits_and_Limitations. [Accessed: 22 June 2025].
13. J. Cui et al., “ChatLaw: Open-Source Legal Large Language Model with Integrated External Knowledge Bases,” Jun. 2023, doi: 10.48550/arxiv.2306.16092. [Accessed: 13 June 2025]
14. J. Cui et al., “Chatlaw: A Multi-Agent Collaborative Legal Assistant with Knowledge Graph Enhanced Mixture-of-Experts Large Language Model,” arXiv.org, Jun. 28, 2023. https://arxiv.org/abs/2306.16092 [Accessed: 13 June 2025]
15. L. Bull, “Legal advice chatbot to go head-to-head with living lawyers,” Irish Legal News, Aug. 03, 2017. [Online]. Available: https://www.irishlegal.com/articles/legal-advice-chatbot-to-go-head-to-head-with-living-lawyers [Accessed: 07 June 2025]
16. V. Magesh, F. Surani, M. Dahl, M. Suzgun, C. D. Manning, and D. E. Ho, “Hallucination-Free? Assessing the reliability of leading AI legal research tools,” arXiv.org, May 30, 2024. https://arxiv.org/abs/2405.20362 [Accessed: 07 June 2025]
17. Surani, D. Ho, “AI on Trial: Legal Models Hallucinate in 1 out of 6 (or More) Benchmarking Queries | Stanford HAI.” https://hai.stanford.edu/news/ai-trial-legal-models-hallucinate-1-out-6-or-more-benchmarking-queries [Accessed: 04 September 2025]
18. Vassilopoulos, S. Titah, “Can AI chatbots improve access to legal information? A case study on the inhibitors of AI chatbots’ implementation in the context of JuridiQC.” [Online]. Available: https://biblos.hec.ca/biblio/memoires/vassilopoulos_athena_m2022.pdf [Accessed: 07 June 2025]
19. Caldarini, S. Jaf, and K. McGarry, “A literature survey of recent advances in chatbots,” Information, vol. 13, no. 1, p. 41, Jan. 2022, doi: 10.3390/info13010041. [Accessed: 06 June 2025]
20. D. Necz, “Rules over words: Regulation of chatbots in the legal market and ethical considerations,” Hungarian Journal of Legal Studies, vol. 64, no. 3, pp. 472–485, Jun. 2024, doi: 10.1556/2052.2023.00472. [Accessed: 26 June 2025]
21. J. Xue, Y.-C. Wang, C. Wei, X. Liu, J. Woo, and C. -c. J. Kuo, “Bias and Fairness in Chatbots: An Overview,” arXiv.org, Sep. 16, 2023. https://arxiv.org/abs/2309.08836 [Accessed: 19 April 2025]
22. Palfreyman, “Minimizing legal risks of AI-Powered chatbots,” Harris Beach Murtha, Mar. 11, 2025. https://www.harrisbeachmurtha.com/insights/minimizing-legal-risks-of-ai-powered-chatbots/ [Accessed: 19 April 2025]
23. V. Karpukhin et al., “Dense passage retrieval for Open-Domain question answering,” arXiv.org, Apr. 10, 2020. https://arxiv.org/abs/2004.04906 [Accessed: 08 June 2025]
24. Izacard and É. Grave, “Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering,” ACL Anthology, Apr. 2021, doi: 10.18653/v1/2021.eacl-main.74. [Accessed: 22 June 2025]
25. N. Pipitone and G. H. Alami, “LegalBench-RAG: a benchmark for Retrieval-Augmented Generation in the legal domain,” arXiv.org, Aug. 19, 2024. https://arxiv.org/abs/2408.10343 [Accessed: 13 August 2025]
26. Yu, A. Gan, K. Zhang, S. Tong, Q. Liu, and Z. Liu, “Evaluation of Retrieval-Augmented Generation: A survey,” arXiv.org, May 13, 2024. https://arxiv.org/abs/2405.07437v1 [Accessed: 16 June 2025]
27. F. Petroniet al., “KILT: a Benchmark for Knowledge Intensive Language Tasks,” Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, [Online]. Available: https://aclanthology.org/2021.naacl-main.200.pdf [Accessed: 08 June 2025]
28. P. Ahadian and Q. Guan, “A survey on hallucination in large language and foundation models,” Preprints.org, May 2025, doi: 10.20944/preprints202504.1236.v2. [Accessed: 17 April 2025]
29. Rau 1 et al., “BERGEN: A benchmarking library for Retrieval-Augmented Generation,” Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 7640–7663, Nov. 2024, [Online]. Available: https://aclanthology.org/2024.findings-emnlp.449.pdf [Accessed: 03 July 2025]
30. Z. Fei et al., “LawBench: Benchmarking legal knowledge of large language models,” arXiv.org, Sep. 28, 2023. https://arxiv.org/abs/2309.16289 [Accessed: 03 July 2025]
31. Laurent, “LLM Benchmarks in Life Sciences: Comprehensive Overview,” IntuitionLabs, Nov. 07, 2025. https://intuitionlabs.ai/articles/large-language-model-benchmarks-life-sciences-overview
32. M. S. Y. Alassan, J. L. Espejel, M. Bouhandi, W. Dahhane, and E. H. Ettifouri, “Comparison of Open-Source and Proprietary LLMs for Machine Reading Comprehension: A Practical Analysis for Industrial applications,” arXiv.org, Jun. 19, 2024. https://arxiv.org/abs/2406.13713v2 [Accessed: 13 August 2025]
33. “LawNet - American Institute for Sri Lankan Studies,” American Institute for Sri Lankan Studies. https://www.aisls.org/lawnet/“Search judgments and orders – court of appeal.” https://courtofappeal.lk/?page_id=1191 [Accessed: 03 July 2025]
34. “Search judgments and orders – court of appeal.” https://courtofappeal.lk/?page_id=1191 [Accessed: 03 July 2025]
35. “Department of Government Printing.” https://documents.gov.lk/
36. “Home,” Srilanka Law. https://www.srilankalaw.lk/
37. “Sri Lankan Legal Materials,” Commonlii. https://www.commonlii.org/lk/
38. “VLex | Sri Lanka.” https://vlex.com/coverage/sri-lanka
39. WorldJusticeProject, “Paths followed by people in Sri Lanka to deal with their everyday justice problems, summarizing the incidence of legal problems, respondents’ legal capability, access to sources of help, problem status, assessment of the resolution process, and problem impact.,” survey, 2017. [Online]. Available: https://worldjusticeproject.org/sites/default/files/documents/Access-to-Justice-2019-SriLanka.pdf [Accessed: 03 July 2025]
40. W. N. S. Perera, A. M. Perera, S. Hulathduwa, and P. Paranitharan, “Artificial intelligence-driven digitization of legal system in Sri Lanka - A challenging approach,” Sri Lanka Journal of Forensic Medicine Science & Law, vol. 16, no. 1, pp. 51–57, Jun. 2025, doi: 10.4038/sljfmsl.v16i1.8040. [Accessed: 22 June 2025]
41. Östling et al., “The Cambridge Law Corpus: a dataset for Legal AI research,” The Cambridge Law Corpus, 2023, doi: 10.17863/CAM.100221. [Accessed: 17 June 2025]
42. N. Guha et al., “LegalBench: a collaboratively built benchmark for measuring legal reasoning in large language models,” arXiv.org, Aug. 20, 2023. https://arxiv.org/abs/2308.11462 [Accessed: 07 June 2025]
43. S. H. Wang et al., “ACORD: an Expert-Annotated Retrieval Dataset for Legal Contract Drafting,” Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, vol. 1, Jan. 2025, doi: 10.18653/v1/2025.acl-long.1206. [Accessed: 07 June 2025]
44. “Chroma is the open-source search and retrieval database for AI applications,” Chroma. https://www.trychroma.com/
45. Y. Gao et al., “Retrieval-Augmented Generation for Large Language Models: A survey,” arXiv.org, Dec. 18, 2023. https://arxiv.org/abs/2312.10997 [Accessed: 08 July 2025]
46. Q. Jiang et al., “Mistral 7B,” arXiv.org, Oct. 10, 2023. https://arxiv.org/abs/2310.06825 [Accessed: 10 August 2025]
47. Bronsdon, “RAG vs Traditional LLMs: Key Differences,” Galileo AI, Nov. 19, 2024. https://galileo.ai/blog/comparing-rag-and-traditional-llms-which-suits-your-project [Accessed: 09 April 2025]
48. “Pricing,” OpenAI. https://platform.openai.com/docs/pricing
49. “Colab paid services pricing.” https://colab.research.google.com/signup
50. M. H. Erol, B. El, M. Suzgun, M. Yuksekgonul, and J. Zou, “Cost-of-Pass: an economic framework for evaluating language models,” arXiv.org, Apr. 17, 2025. https://arxiv.org/abs/2504.13359 [Accessed: 20 July 2025]
51. “LLM Comparison: key concepts & best practices | NExLa,” Nexla, Sep. 15, 2025. https://nexla.com/ai-readiness/llm-comparison [Accessed: 23 Sep 2025]
52. Publications Office of the European Union, “Ethics guidelines for trustworthy AI.,” Publications Office of the EU, 2019. https://op.europa.eu/en/publication-detail/-/publication/d3988569-0434-11ea-8c1f-01aa75ed71a1 [Accessed: 06 April 2025]