Design and Development of a Hybrid NLP and Rule-based QA Assistant for Indonesian User Stories

Cheria Sevani Apiani, Ichsan Ibrahim

Abstract


This study aims to address challenges in Agile-based software testing, where manually creating test cases from user stories is often time-consuming and produces inconsistent quality. Although Natural Language Processing (NLP) techniques and rule-based systems have been proposed, each approach has limitations in handling linguistic ambiguity and variations in sentence structure, particularly in the Indonesian language context. This research proposes a hybrid Quality Assurance (QA) assistant that integrates an IndoBERT-based Named Entity Recognition (NER) model with a deterministic rule-based system. The NER model is used to extract functional elements, including actors, actions, objects, conditions, and expected outcomes, while the rule-based system maps these elements into structured test case templates. Qualitative evaluation conducted by QA practitioners showed that the hybrid approach achieved an average score of 4.67 on a 5-point Likert scale, outperforming both the NLP-only approach (3.87) and the rule-only approach (4.60). The proposed system was proven to improve testing efficiency by more than 99% while generating test cases that are more complete, readable, and traceable. These findings confirm that integrating the flexibility of NLP with the consistency of rule-based systems is highly effective for automating Quality Assurance processes in the Indonesian local context.

Keywords


hybrid approach; indonesian user stories; natural language processing; rule-based systems; test case generation

Full Text:

PDF

References


A. Mustafa et al., “Automated Test Case Generation from Requirements: A Systematic Literature Review,” Computers, Materials and Continua, Vol. 67, No. 2, pp. 1819–1833, Jan. 2021, DOI: 10.32604/CMC.2021.014391.

V. Garousi, S. Bauer, and M. Felderer, “NLP-Assisted Software Testing: A Systematic Mapping of the literature,” Inf. Softw. Technol., Vol. 126, p. 106321, Oct. 2020, DOI: 10.1016/J.INFSOF.2020.106321.

H. Imhmed, K. Ahmed, Y. Salem, and H. Zulzalil, “Leveraging Latent Natural Language Processing Techniques for User Story Management in Agile Software Development,” Journal of Pure & Applied Sciences, Vol. 22, No. 2, pp. 5–9, Oct. 2023, DOI: 10.51984/jopas.v22i2.2599.

J. W. Lim et al., “Test Case Information Extraction from Requirements Specifications using NLP-based Unified Boilerplate Approach,” Journal of Systems and Software, Vol. 211, p. 112005, May 2024, doi: 10.1016/J.JSS.2024.112005.

A. Chinnnaswamy, B. Sabarish, and R. Menan, “User Story based Automated Test Case Generation using NLP,” in Computational Intelligence in Data Science, Chennai, India, May 2024, pp. 156–166. DOI: 10.1007/978-3-031-69982-5_12.

T. Rahman and Y. Zhu, “Automated User Story Generation with Test Case Specification using Large Language Model,” Apr. 2024, [Online]. Available: http://arxiv.org/abs/2404.01558

N. Medeshetty, A. N. Ghazi, S. Alawadi, and F. Alkhabbas, “From Requirements to Test Cases: An NLP-based Approach for High-Performance ECU Test Case Automation,” 2025, DOI: https://doi.org/10.48550/arXiv.2505.00547.

M. Boukhlif, M. Hanine, N. Kharmoum, A. Ruigomez Noriega, D. Garcia Obeso, and I. Ashraf, “Natural Language Processing-based Software Testing: A Systematic Literature Review,” IEEE Access, Vol. 12, pp. 79383–79400, 2024, DOI: 10.1109/ACCESS.2024.3407753.

L. Zhao et al., “Natural Language Processing for Requirements Engineering,” ACM Comput. Surv., Vol. 54, No. 3, Jan. 2022, DOI: 10.1145/3444689.

J. Navarro and R. Ibarra, “Automatic Test Case Generation using Natural Language Processing: A systematic mapping study,” Inf. Softw. Technol., Vol. 189, Jan. 2026, DOI: 10.1016/j.infsof.2025.107929.

J. Fischbach et al., “Automatic Creation of Acceptance Tests by Extracting Conditionals from Requirements: NLP approach and case study,” Journal of Systems and Software, Vol. 197, p. 111549, Mar. 2023, DOI: 10.1016/J.JSS.2022.111549.

R. Gröpler, V. Sudhi, E. J. Calleja García, and A. Bergmann, “NLP-based Requirements Formalization for Automatic Test Case Generation,” in 29th International Workshop on Concurrency, Specification and Programming (CS&P’21), Magdeburg, Germany: CEUR Workshop Proceedings, 2021. [Online]. Available: http://ceur-ws.org

A. Fatima, A. Haider, and S. Reza, “Automated Test Case Generation From Natural Language Requirements using Natural Language Processing,” Journal of Computing and Biomedical Informatics, Vol. 9, No. 2, Aug. 2025, DOI: 10.56979/902/2025.

A. Prabu, S. L. A, S. B A, and A. K. C, “User Story-based Automatic Test Case Classification and Prioritization using Natural Language Processing-based Deep Learning,” IEEE Potentials, Vol. 43, No. 5, pp. 20–28, 2024, DOI: 10.1109/MPOT.2023.3342366.

N. Gupta, V. Yadav, and M. Singh, “Decision Tree based Test Case Generation using Story Board and Natural Language Processing,” in Advances in Computing and Data Sciences, Cham: Springer Nature Switzerland, 2023, pp. 581–591.

S. El Farisi and F. A. Marva, “Evaluasi Teknik Prompting pada Large Language Model untuk Otomatisasi Penyusunan Skenario Unit Testing Smart Contract,” Jurnal Teknologi Informasi dan Multimedia, Vol. 8, No. 1, pp. 99–108, Jan. 2026, DOI: 10.35746/jtim.v8i1.912.

B. Hardika et al., “Pengujian Blackbox Testing Website Garuda Farm menggunakan Teknik Equivalence Partitioning,” Jurnal Kridatama Sains dan Teknologi, Vol. 06, No. 02, pp. 747–751, 2024.

F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, “IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP,” Nov. 2020, [Online]. Available: http://arxiv.org/abs/2011.00677

W. Widyawan, B. P. Utomo, and M. N. Rizala, “A Novel Fusion of Machine Learning Methods for Enhancing Named Entity Recognition in Indonesian Language Text,” Jurnal Sistem Informasi Bisnis, Vol. 14, No. 4, pp. 311–320, Oct. 2024, DOI: 10.21456/vol14iss4pp311-320




DOI: https://doi.org/10.32520/stmsi.v15i5.6383

Article Metrics

Abstract view : 30 times
PDF - 5 times

Refbacks

  • There are currently no refbacks.


Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.