Confidence-Gated Semantic Retrieval for Indonesian Customer-Service Intent Classification
DOI:
https://doi.org/10.55123/jomlai.v5i3.9547Kata Kunci:
Abstention Mechanism, Confidence Calibration, Indonesian Language, Intent Classification, Sentence EmbeddingAbstrak
Customer-service chatbots deployed in low-resource languages must decide not only which intent a message expresses but also when to decline answering. This study evaluated a confidence-gated retrieval architecture for Indonesian intent classification in which a deterministic keyword layer, a multilingual sentence encoder, and an explicit abstention rule were combined. A corpus of twenty-two intents containing three thousand nine hundred sixty training patterns and one thousand fifty-six held-out test patterns was constructed, covering commerce enquiries, complaint handling, and instrumentation engineering questions. Two retrieval backends were compared under identical decision logic: a lexical backend that fused word and character term weighting with a naive Bayes component, and a semantic backend built on multilingual sentence embeddings. The semantic backend reached an accuracy of 82.67 percent and a macro-averaged F-measure of 83.04 percent, exceeding the lexical backend by 11.74 and 12.31 percentage points, at a mean response latency of 30.66 milliseconds on a single processor. Raising the confidence threshold increased selective accuracy from 71.06 to 82.82 percent while coverage fell to 61.17 percent, which quantified the price of caution. Of 100 balanced out-of-scope probes, 93 were rejected (93.00 percent). The deterministic layer resolved just over half of the queries and carried the higher accuracy of the two decision paths.
Referensi
[1] J. Weizenbaum, “ELIZA—a computer program for the study of natural language communication between man and machine,” Communications of the ACM, vol. 9, no. 1, pp. 36–45, January 1966.
[2] A. Vaswani et al., “Attention is all you need,” in Proc. 31st International Conference on Neural Information Processing Systems, December 2017, pp. 5998–6008.
[3] J. Devlin, M. W. Chang, K. Lee and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics, June 2019, Vol. 1, pp. 4171–4186.
[4] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using Siamese BERT-networks,” in Proc. 2019 Conference on Empirical Methods in Natural Language Processing, November 2019, pp. 3982–3992.
[5] N. Reimers and I. Gurevych, “Making monolingual sentence embeddings multilingual using knowledge distillation,” in Proc. 2020 Conference on Empirical Methods in Natural Language Processing, November 2020, pp. 4512–4525.
[6] A. Conneau et al., “Unsupervised cross-lingual representation learning at scale,” in Proc. 58th Annual Meeting of the Association for Computational Linguistics, July 2020, pp. 8440–8451.
[7] B. Wilie et al., “IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding,” in Proc. 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, December 2020, pp. 843–857.
[8] F. Koto, A. Rahimi, J. H. Lau and T. Baldwin, “IndoLEM and IndoBERT: A benchmark dataset and pre-trained language model for Indonesian NLP,” in Proc. 28th International Conference on Computational Linguistics, December 2020, pp. 757–770.
[9] S. Larson et al., “An evaluation dataset for intent classification and out-of-scope prediction,” in Proc. 2019 Conference on Empirical Methods in Natural Language Processing, November 2019, pp. 1311–1316.
[10] I. Casanueva, T. Temčinas, D. Gerz, M. Henderson and I. Vulić, “Efficient intent detection with dual sentence encoders,” in Proc. 2nd Workshop on Natural Language Processing for Conversational AI, July 2020, pp. 38–45.
[11] J. G. Zhang et al., “Discriminative nearest neighbor few-shot intent detection by transferring natural language inference,” in Proc. 2020 Conference on Empirical Methods in Natural Language Processing, November 2020, pp. 5064–5082.
[12] C. K. Chow, “On optimum recognition error and reject tradeoff,” IEEE Transactions on Information Theory, vol. 16, no. 1, pp. 41–46, January 1970.
[13] R. El-Yaniv and Y. Wiener, “On the foundations of noise-free selective classification,” Journal of Machine Learning Research, vol. 11, pp. 1605–1641, May 2010.
[14] D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” in Proc. 5th International Conference on Learning Representations, April 2017.
[15] C. Guo, G. Pleiss, Y. Sun and K. Q. Weinberger, “On calibration of modern neural networks,” in Proc. 34th International Conference on Machine Learning, August 2017, vol. 70, pp. 1321–1330.
[16] Z. Ji et al., “Survey of hallucination in natural language generation,” ACM Computing Surveys, vol. 55, no. 12, pp. 1–38, December 2023.
[17] P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive NLP tasks,” in Proc. 34th International Conference on Neural Information Processing Systems, December 2020, pp. 9459–9474.
[18] G. Salton and C. Buckley, “Term-weighting approaches in automatic text retrieval,” Information Processing and Management, vol. 24, no. 5, pp. 513–523, 1988.
[19] A. Joulin, E. Grave, P. Bojanowski and T. Mikolov, “Bag of tricks for efficient text classification,” in Proc. 15th Conference of the European Chapter of the Association for Computational Linguistics, April 2017, vol. 2, pp. 427–431.
[20] P. Bojanowski, E. Grave, A. Joulin and T. Mikolov, “Enriching word vectors with subword information,” Transactions of the Association for Computational Linguistics, vol. 5, pp. 135–146, June 2017.
[21] K. Cho et al., “Learning phrase representations using RNN encoder–decoder for statistical machine translation,” in Proc. 2014 Conference on Empirical Methods in Natural Language Processing, October 2014, pp. 1724–1734.
[22] R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine Learning, vol. 8, no. 3, pp. 229–256, May 1992.
[23] A. Paszke et al., “PyTorch: An imperative style, high-performance deep learning library,” in Proc. 33rd International Conference on Neural Information Processing Systems, December 2019, pp. 8024–8035.
[24] V. Sanh, L. Debut, J. Chaumond and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” in Proc. 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing, December 2019.
[25] W. Wang, F. Wei, L. Dong, H. Bao, N. Yang and M. Zhou, “MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers,” in Proc. 34th International Conference on Neural Information Processing Systems, December 2020, pp. 5776–5788.
[26] N. Reimers and I. Gurevych, “paraphrase-multilingual-MiniLM-L12-v2,” Hugging Face Model Hub, 2021. [Online]. Available: https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 [Accessed: 17 September 2026].
Unduhan
Diterbitkan
Cara Mengutip
Terbitan
Bagian
Lisensi
Hak Cipta (c) 2026 Marco Julius Andreas Bakkara, Mashelina Mashelina
Artikel ini berlisensiCreative Commons Attribution-ShareAlike 4.0 International License.
Copyright (c) 2022 The authors. Published by Yayasan Literasi Indonesia 
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License
The author(s) whose article is published in the JOMLAI journal attain the copyright for their article and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. By submitting the manuscript to JOMLAI, the author(s) agree with this policy. No special document approval is required.
The author(s) guarantee that:their article is original, written by the mentioned author(s),
- has never been published before,
- does not contain statements that violate the law, and
- does not violate the rights of others, is subject to copyright held exclusively by the author(s), and is free from the rights of third parties, and that the necessary written permission to quote from other sources has been obtained by the author(s).
The author(s) retain all rights to the published work, such as (but not limited to) the following rights:
- Copyright and other proprietary rights related to the article, such as patents,
- The right to use the substance of the article in its own future works, including lectures and books,
- The right to reproduce the article for its own purposes,
- The right to archive all versions of the article in any repository, and
- The right to enter into separate additional contractual arrangements for the non-exclusive distribution of published versions of the article (for example, posting them to institutional repositories or publishing them in a book), acknowledging its initial publication in this journal (JOMLAI: Journal of Machine Learning and Artificial Intelligence).
Suppose the article was prepared jointly by more than one author. Each author submitting the manuscript warrants that all co-authors have given their permission to agree to copyright and license notices (agreements) on their behalf and notify co-authors of the terms of this policy. JOMLAI will not be held responsible for anything that may arise because of the writer's internal dispute. JOMLAI will only communicate with correspondence authors.
Authors should also understand that their articles (and any additional files, including data sets, and analysis/computation data) will become publicly available once published. The license of published articles (and additional data) will be governed by a Creative Commons Attribution-ShareAlike 4.0 International License. JOMLAI allows users to copy, distribute, display and perform work under license. Users need to attribute the author(s) and JOMLAI to distribute works in journals and other publication media. Unless otherwise stated, the author(s) is a public entity as soon as the article is published
