Academic Chatbot for Campus Information Services Using Retrieval-Augmented Generation

Abstract

University service centers handle many repetitive queries about academic schedules, registration, and policies stored in internal documents. Manual lookup is inefficient, and answers given by staff can be inconsistent. Rule-based chatbots only handle limited question patterns, while large language models are hard to update and may produce unsupported answers (hallucinations). This research designs an academic chatbot that combines document retrieval with answer generation so that each answer remains traceable to its source. The system extracts text from campus documents, segments it, encodes it using a multilingual embedding model, and stores it in a vector index for context retrieval. A response is generated through an instruction template that confines the output to the retrieved information and includes page references. Evaluation followed a mixed-method design: a quantitative layer measured retrieval quality (Precision@5, Recall@5) and generation quality using the four RAGAS sub-metrics (faithfulness, answer_relevancy, context_precision, context_recall) on a 100-question test set, while a qualitative layer applied thematic analysis to open-ended user comments. Statistical testing used McNemar's test for accuracy and a paired bootstrap (10,000 resamples) for retrieval metrics; 95% confidence intervals are reported. Results: the proposed RAG system achieved 84% answer accuracy (95% CI 76–90%), Precision@5 = 0.80 and Recall@5 = 0.72, with a System Usability Scale (SUS) score of 78 and a Net Promoter Score (NPS) of +32 from 30 participants. Differences in accuracy versus the lexical and LLM-only baselines were statistically significant (McNemar p < 0.05). The system offers a replicable instantiation of RAG for transparent, citation-backed campus information services in Indonesian.

GS Cited Analysis