Can small language models handle context-summarized multi-turn customer-service QA? A synthetic data-driven comparative evaluation

Abstract

Customer-service question answering (QA) systems increasingly rely on conversational language understanding. While Large Language Models (LLMs) achieve strong performance, their high computational cost and deployment constraints limit practical use in resource-constrained environments. Small Language Models (SLMs) provide a more efficient alternative, yet their effectiveness for multi-turn customer-service QA remains underexplored, particularly in scenarios requiring dialogue continuity and contextual understanding. In this study, we evaluate whether instruction-tuned SLMs, fine-tuned using parameter-efficient finetuning, can effectively handle context-summarized multi-turn customer-service QA while preserving contextual consistency, response quality and task relevance under computational constraints. We further investigate instruction-tuned SLMs for context-summarized multi-turn customer-service QA using a history summarization strategy to preserve essential conversational state and introduce a conversation stage-based qualitative analysis to evaluate model behavior across different phases of customer-service interactions. The main contributions of this work include the application of parameter-efficient fine-tuning to adapt SLMs for context-summarized multi-turn customer-service QA, a synthetic data construction pipeline for generating a context-summarized multi-turn QA dataset, and a structured evaluation framework combining quantitative metrics with human and LLM-as-a-judge assessments for customer-service QA evaluation. Nine instruction-tuned SLMs are evaluated against three commercial LLMs using lexical and semantic similarity metrics alongside qualitative assessments, including human evaluation and LLM-as-a-judge methods. Results show notable variation across SLMs, with some models demonstrating near-LLM performance, while others struggle to maintain dialogue continuity and contextual alignment. These findings highlight both the potential and current limitations of low-parameter language models for real-world customer-service QA systems.

1 Introduction

Customer service interactions are a critical component of modern business operations, directly influencing customer satisfaction, organizational reputation and operational efficiency. Customers across sectors such as banking, telecommunications, and e-commerce frequently contact service providers to resolve issues, seek information, or request account modifications (Bird, 2021). These interactions typically involve multiple exchanges between clients and agents, incorporate domain-specific terminology and require contextual continuity across dialogue turns. Manual handling of such conversations imposes substantial operational costs related to agent recruitment, training and supervision, motivating growing interest in automation technologies.

Early customer service automation relied on rule-based systems and statistical machine learning models such as Support Vector Machines and Hidden Markov Models. Although effective for basic intent detection, these approaches struggled with linguistic variability and long-range dependencies in multi-turn dialogue (interactions consisting of multiple conversational exchanges between a client and an agent) (Wang et al., 2017). Transformer architectures advanced the field by enabling contextual representations through self-attention, supporting more coherent conversations (Vaswani et al., 2017). Building on this, LLMs showed strong ability in understanding context, reasoning over queries and generating fluent customer service responses (Wulf and Meierhofer, 2024; Xiaoliang et al., 2024). However, their large size leads to high computational cost, latency and dependence on cloud APIs, intensifying privacy and data governance concerns since customer interactions often contain sensitive or personally identifiable information. Such data sharing raises legal and ethical issues in regulated domains requiring strict compliance (Ilse and Blackwood, 2024; Kamisetty and Nagamangalam, 2025). These factors limit deployment in resource-constrained or on-premise settings.

SLMs, typically defined as models with less than ten billion parameters, have emerged as efficient alternatives (Belcak et al., 2025; Meconi et al., 2025; Xu et al., 2025). Recent SLM families such as SmolLM, Qwen, Phi and LLaMA demonstrate strong instruction-following (ability to generate responses based on explicit user instructions) and reasoning capabilities while remaining deployable on standard hardware (Allal et al., 2025). Parameter-efficient fine-tuning methods further enable adaptation to specialized domains with reduced computational and memory requirements (Hu et al., 2022; Dettmers et al., 2023; Zhang et al., 2023; Li and Liang, 2021), suggesting that SLMs offer a practical balance between performance and efficiency for customer-service automation.

Despite growing interest, the effectiveness of SLMs for customer-service QA remains underexplored, particularly in multi-turn client-agent interactions requiring dialogue continuity and contextual understanding across turns. Existing research has largely focused on single-turn QA settings, where conversational history is not modeled or leveraged (Wang et al., 2017; Sanjani et al., 2025). No work has systematically evaluated recently introduced instruction-tuned SLMs (SLMs fine-tuned to follow task-specific instructions) under multi-turn customer-service settings (Xiaoliang et al., 2024; Lijaya et al., 2025). Evaluation practices are often inconsistent, relying either on automatic metrics such as ROUGE (Recall-Oriented Understudy for Gisting Evaluation) and BERTScore (a semantic similarity metric based on contextual embeddings) (Lin, 2004; Zhang et al., 2020) or on qualitative approaches such as human assessment and LLM-as-a-judge methods (using a LLM to evaluate generated responses) in isolation (Liu et al., 2023; Park et al., 2024). Additionally, the lack of publicly available English benchmark datasets for multi-turn customer-service conversations limits experimental comparability, as existing datasets such as TelBench and TeleEval CS are restricted to Chinese and Korean languages (Lee et al., 2024; Li et al., 2024).

To address these limitations, this study systematically evaluates fine-tuned, instruction-tuned SLMs for context-summarized multi-turn customer-service QA. A synthetic data construction pipeline is introduced to mitigate the limited availability of publicly accessible context-summarized multi-turn customer-service QA data. The pipeline transforms single-turn QA instances into structured multi-turn interactions, applies context summarization to refine dialogue history and performs LLM-based response refinement prior to fine-tuning. In addition, a conversation stage-based segmentation is employed to categorize interactions into early, mid and late stages, enabling stage-wise qualitative analysis of model behavior across different phases of customer-service conversations. Performance is assessed using a comprehensive evaluation framework combining lexical and semantic similarity metrics with LLM-as-a-judge and human assessment. All models are evaluated under identical experimental conditions in context-summarized multi-turn customer-service settings, enabling a fair comparison between SLMs and LLMs.

Our main contributions are as follows:

• A systematic evaluation of fine-tuned, instruction-tuned SLMs for context-summarized multi-turn customer-service QA.

• A synthetic data construction and fine-tuning pipeline that integrates multi-turn context summarization with LLM-based response refinement to produce privacy-preserving training data for SLMs.

• A comparative evaluation of fine-tuned SLMs against state-of-the-art LLMs for context-summarized multi-turn customer-service QA using automatic metrics, human evaluation and conversation stage-based analysis.

This paper is structured as follows: Section 2 reviews related work and Section 3 details the proposed methodology and dataset construction. Section 4 presents the experimental setup, while Section 5 discusses the obtained results. Section 6 concludes the paper and outlines future research directions.

2 Related work

2.1 Usage of NLP in customer service QA

Early customer-service QA systems relied on retrieval and embedding-based methods. For example, Wang et al. (2017) proposed a hybrid system combining BM25 keyword search, a probabilistic ranking algorithm based on term frequency and document relevance with Word2Vec and Doc2Vec embeddings, enhanced by a k-nearest neighbor classifier for intent awareness and answer re-ranking. While effective in structured settings, these approaches lacked advanced contextual understanding and struggled to maintain dialogue continuity across multiple exchanges. Subsequent research introduced progressively more advanced paradigms, evolving from retrieval-based systems to transformer models, LLMs and more recently SLMs. This overall progression of customer-service QA research is illustrated in Figure 1.

Figure 1

The introduction of the transformer architecture made a significant advance in conversational modeling (Vaswani et al., 2017). Pre-trained sequence-to-sequence models trained on large-scale datasets, such as customer support tweets, were adapted to domain-specific chatbots and deployed on social robots like Temi and Pepper (Bird, 2021). Encoder-decoder models such as T5 (Raffel et al., 2020) and Flan-T5 (Chung et al., 2024) both introduced by Google further improved robustness to diverse queries through fine-tuning. More recent work showed that retrieval-augmented generation approaches using models such as LLaMA (by Meta), Gemma (by Google), and Mistral (by Mistral AI) achieved higher accuracy on reworded questions but incurred slower inference and increased system complexity (Sanjani et al., 2025).

As LLMs became dominant, research increasingly focused on efficient adaptation strategies. Ilse and Blackwood (2024) compared full fine-tuning, LoRA-based parameter-efficient tuning (Low-Rank Adaptation) and domain-adaptive pre-training on models such as GPT-4, Gemini and LLaMA-2 (Meta). Domain-adaptive pre-training has obtained the strongest performance, while LoRA enabled faster and more resource-efficient adaptation, underscoring the importance of parameter-efficient fine-tuning for real-time customer-service deployment.

Applications of LLMs in customer-service QA reveal both strengths and limitations. A Swiss telecom study showed that GPT-4 could draft email responses but struggled with multi-step reasoning and hallucinations (Wulf and Meierhofer, 2024). A hybrid system based on ChatGLM2-6B with LangChain, LoRA fine-tuning and reinforcement learning via Proximal Policy Optimization achieved a 74.8% user acceptance rate, outperforming GPT-4 and baseline models (Xiaoliang et al., 2024). Emotion-aware QA systems further improved response relevance and positivity (Albuquerque et al., 2024). Despite these advances, LLM-based systems continue to face challenges related to scalability, latency, and deployment cost.

Reliability has been addressed through validation pipelines such as CHOPS (CHat with custOmer Profile in existing System), which employ classifier-executor-verifier frameworks (Shi et al., 2024) and collaborative generation methods such as Reconcile (a collaborative multi-LLM evaluation framework) and SCRABLE (Self-Correcting Response Approach Based on LLM Evaluation) that use multi-model voting and self-improving loops (Watanabe et al., 2024; Azov et al., 2024). While these approaches enhance reliability, they remain dependent on large models. A further limitation is the limited availability of publicly accessible customer-service multi-turn QA datasets due to privacy concerns. TelBench introduced the TelTask and TelInstruct datasets (Lee et al., 2024) and TeleEval CS expanded this effort with 90,000 instruction-tuning examples (Li et al., 2024). However, both benchmarks are restricted to Chinese and Korean and are not explicitly designed for multi-turn customer-service QA. As a result, reproducible evaluation of English multi-turn customer-service QA remains underexplored.

2.2 Rise of small language models

The growing interest in SLMs is driven by the limitations of LLMs, including high computational cost, latency and reliance on cloud-based deployment. These challenges are particularly critical in privacy-sensitive and resource-constrained environments. As a result, there is increasing demand for efficient models that can deliver strong performance while remaining cost-effective and deployable on standard hardware. Several SLM families have been introduced in recent years, including SmolLM (Allal et al., 2025), Qwen (Qwen et al., 2025; Yang et al., 2024), Gemma (Team et al., 2025, 2024), Phi (Abouelenin et al., 2025; Abdin et al., 2024), and LLaMA (Grattafiori et al., 2024; Touvron et al., 2023). While differing in size, these models demonstrate strong capabilities in multilingual processing, long-context handling and reasoning with improved efficiency.

Domain-specific adaptation of SLMs has gained increasing attention. In healthcare, models such as BioGPT, PMC-LLaMA, RadPhi2, and CancerGPT have been applied to clinical QA tasks (Garg et al., 2025). In finance, FinGPT and Instruct-FinGPT have shown strong alignment with domain-specific data (Li et al., 2023). Customer service, however, remains comparatively underexplored.

Although no specific work has focused on recently introduced instruction-tuned SLMs for multi-turn customer-service QA, some studies have explored customer-service applications using medium-sized SLMs. LoRA-adapted LLaMA-3.1-8B models improved QA accuracy in telecommunications (Lovtsov and Skvortsova, 2025), while ChatGLM2-6B achieved high intent accuracy in the electric power sector (Cui et al., 2025). Studies in banking and restaurant domains reported strong performance using Gemma, Mistral, Falcon and LLaMA-based models (Lijaya et al., 2025; Albuquerque et al., 2024), with real-world prototypes further demonstrating feasibility (Chen et al., 2024). Nevertheless, systematic evaluation of instruction-tuned SLMs for customer-service tasks involving multi-turn interactions and dialogue continuity remains unexplored.

2.3 Evaluation methods used in customer service QA

Evaluation of customer service QA systems mainly relies on automatic metrics and qualitative assessments. Automatic methods include lexical overlap metrics such as BLEU (Bilingual Evaluation Understudy, a precision-based metric measuring n-gram overlap between generated and reference text) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation, a metric measuring the overlap of words and sequences between generated and reference text) and semantic similarity metrics like BERTScore (a metric that computes token-level semantic similarity between generated and reference text using contextual embeddings from pre-trained encoder-only transformer models) and BARTScore (a metric that uses a pre-trained BART encoder-decoder model to score generated text based on the likelihood of generating the reference text from the candidate response) (Papineni et al., 2001; Lin, 2004; Zhang et al., 2020; Yuan et al., 2021). While efficient, these often capture surface-level similarity rather than dialogue coherence. Qualitative evaluations focus on human-centered qualities like correctness, clarity and empathy. Human assessment remains the gold standard, while LLM-as-a-judge frameworks such as G-Eval and PairEval provide scalable alternatives (Liu et al., 2023; Park et al., 2024; Gu et al., 2026). However, most studies focus on overall conversation evaluations and have not focused on stage-based evaluations to assess SLMs' abilities across different conversational stages.

Overall, prior work shows that current quantitative and qualitative evaluation approaches do not fully cover all aspects of customer-service QA, particularly in multi-turn settings, due to a lack of benchmarks designed to evaluate dialogue continuity and contextual understanding across conversational turns. These gaps motivate the synthetic dataset construction, fine-tuning pipeline and evaluation framework presented in the following section.

3 Methodology

This section outlines the methodological framework adopted to evaluate instruction-tuned SLMs for context-summarized multi-turn customer-service QA. We first describe the synthetic data construction pipeline designed to address privacy constraints and the lack of publicly available multi-turn customer-service datasets. We then detail the process of multi-turn conversation construction, context summarization and response refinement used to generate high-quality training data. Finally, we present the model selection criteria, fine-tuning configuration and inference setup employed to ensure a controlled and fair comparison between SLMs and LLMs under identical experimental conditions (code for experiments is available at https://github.com/Lakshan2023/Small_language_models_for_multi_turn_context_summarized_conversations).

3.1 Dataset construction

To address the limited availability of publicly accessible context-summarized multi-turn customer-service data, we constructed a synthetic QA dataset through a controlled pipeline of multi-turn conversation construction, context summarization, response refinement and moderation filtering, designed to preserve essential conversational context while maintaining privacy constraints. The overall dataset construction and processing workflow is illustrated in Figure 2.

Figure 2

3.1.1 Initial data source

We utilized the Customer Service Banking Conversation Corpus from Hugging Face's TalkMap repository as our foundational dataset (Talkmap, 2024) (source dataset is available at https://huggingface.co/datasets/talkmap/banking-conversation-corpus). While this corpus contained proper conversation sequences, it consisted only of single-turn QA pairs without multi-turn dialogue structure. The initial dataset consisted with 301,822 unique synthetic conversations with 2,880,214 agent messages and 2,651,898 client messages, averaging 18.33 messages per conversation.

3.1.2 Preprocessing and filtering

We applied initial filtering to retain only conversations containing between 5 and 100 turns to ensure realistic conversational depth while excluding extremely short or anomalously long interactions. Very short interactions (less than 5 turns) were excluded as they typically lack sufficient contextual development for evaluating multi-turn reasoning and dialogue continuity. In addition, some conversations in the corpus ended without completing the full interaction flow, which contributed to a number of short conversations falling below this threshold and influenced the selection of this limit. Sample excluded conversations with less than 5 turns are illustrated in Table 1. Conversations exceeding 100 turns were removed because such extreme lengths are uncommon in real-world customer-service scenarios and are often associated with repetitive exchanges. These long dialogues substantially increase context length, introduce redundancy and reduce the reliability of history summarization and downstream evaluation. The selection of the 5 and 100 turn thresholds was determined through an initial qualitative evaluation of the dataset, followed by consultation with a customer-service domain expert, ensuring that the retained conversations reflect realistic interaction patterns. This filtering resulted in approximately 200,000 conversations used for subsequent processing. Regex-based noise removal was applied to individual conversational turns to eliminate formatting artifacts and non-textual elements.

3.1.3 Multi-turn conversation construction

Since the original dataset consisted of isolated single turns that were already in proper sequence, we aggregated all turns belonging to the same conversation to construct multi-turn dialogue instances. De-duplication was subsequently applied to remove redundant conversations, where conversations that appeared more than once in the corpus were discarded to avoid training bias. To create structured training instances, we constructed client-agent pairs with conversational history by randomly partitioning each conversation into early (20%), middle (70%), and late (10%) segments. This conversation splitting strategy ensured balanced coverage of different conversation stages while prioritizing middle turns, which typically contain the most substantive exchanges. As customer-service conversations generally progress with the main interaction occurring in the middle stage, a larger portion of training samples is taken from this stage to improve context understanding during training. The early stage captures the initial problem, while the late stage represents resolution and closing, ensuring learning of the full conversation flow.

3.1.4 Context summarization

SLMs often struggle to maintain context understanding in multi-turn conversational histories. To address this, we apply a history summarization strategy that summarizes prior conversational turns into concise representations while preserving essential information. A specialized prompt instructs the model to generate summaries containing: (1) the client's primary issue or request and its current status, (2) explicit identification of client and agent names when mentioned, (3) verification steps completed or pending, (4) exact names, account identifiers, dates, amounts and actions taken or agreed upon, (5) commitments, deadlines and scheduled follow-ups, and (6) the current conversation status. Context summarization was applied to all conversations using the GPT-4o-mini model with a maximum output length of 250 tokens and a temperature parameter of 0.3. This low temperature was chosen to keep the summaries factually accurate and consistent, since context summarization requires capturing key details from prior dialogue precisely rather than generating creative or varied outputs. The context summarization prompt used to generate the summarized multi-turn conversation histories is provided in Supplementary File 1. In addition, the context summarization process is illustrated in Figure 3.

Figure 3

3.1.5 Response refinement

To improve the qualitative aspects of training data, agent answers in the constructed multi-turn QA dataset were refined using the GPT-4.1 model with a temperature parameter of 0.4. A slightly higher temperature than that used for context summarization was chosen to allow some natural variation in phrasing, while remaining low enough to preserve factual consistency with the original agent answer. The refinement process considered the instruction, client-agent conversation summary, client question, and original agent answer. This step improved several qualitative dimensions: naturalness and human-like speaking patterns, appropriate response length according to question complexity, clarity and precision, contextual understanding and coherence with conversational history and removal of noise in the original responses. Since the primary objective of this work is to assess the quality of responses generated by SLMs, ensuring high-quality reference answers in the training data was essential. Following refinement, additional regex-based filtering was applied to remove remaining noise or formatting inconsistencies. The prompt used for response refinement is provided in Supplementary File 1. In addition, Figure 4 illustrates an example instance of the response refinement process. Subsequently, we also used OpenAI's Moderation API to flag and filter potentially offensive content from the refined agent answers, ensuring the final dataset adhered to appropriate content standards.

Figure 4

3.1.6 Structured instance formation and dataset splitting

Following response refinement, the dataset was organized into structured instances, each comprising summarized history turns, the current client question, the corresponding agent answer and a task-specific instruction prompt. While the prompt structure remained consistent across all instances, the company or financial institution name was varied to maintain diversity, with context summaries and agent answers refined accordingly to remain contextually aligned. An example of a context-summarized multi-turn customer-service QA instance is shown in Figure 5.

Figure 5

The dataset was split into training (70%), validation (10%), and test (20%) sets, with detailed statistics reported in Appendix Table 9. All splits exhibit similar turn-count distributions, ensuring consistent evaluation conditions across experiments. Token counts were computed using GPT-4 tokenization. The constructed synthetic context-summarized multi-turn QA dataset is available at https://huggingface.co/datasets/Lakshan2003/customer-support-client-agent-conversations.

3.2 Model selection and training

3.2.1 Selected models

We evaluate a total of nine fine tuned SLMs spanning multiple parameter ranges. Five models fall within the three to four billion parameter range, namely Qwen 3-4B Instruct, Phi-4 Mini, LLaMA-3.2-3B Instruct, Gemma 3-4B Instruct, and SmolLM3-3B. SmolLM3-3B includes enhanced reasoning capabilities; however, explicit reasoning was disabled by not using thinking tags during both training and inference. Gemma 3-4B Instruct is a multimodal model; for this study, fine tuning of the vision components was disabled to focus exclusively on text based conversational understanding. To support comparison across model scales, we additionally include Qwen-3-1.7B Instruct and LLaMA-3.2-1B Instruct from the one to two billion parameter range, as well as Qwen-3-8B Instruct and LLaMA 3.1-8B Instruct from the eight billion parameter range. As Qwen-3-8B Instruct also supports explicit reasoning, reasoning was similarly disabled to ensure consistent evaluation conditions across all fine tuned SLMs.

3.2.2 Training configuration

All SLMs included in this study were fine-tuned using Quantized Low-Rank Adaptation (QLoRA) as a parameter-efficient fine-tuning method (Dettmers et al., 2023). QLoRA combines 4-bit quantization with Low-Rank Adaptation, significantly reducing memory requirements while maintaining model performance. Training was conducted using the Unsloth and Hugging Face frameworks.

The models were configured with a maximum sequence length of 512 tokens to accommodate the instruction, summarized conversational history, client question and expected agent response. Prior to adding LoRA adapters, models were quantized to 4-bit precision. The LoRA configuration employed a rank of 16, alpha value of 32 and dropout rate of 0.1, following the recommended configuration provided by Unsloth. A rank of 16 was selected as it provides a good balance between efficiency and model capacity. The alpha was set to twice the rank value, following standard practice for stable training, and a dropout rate of 0.1 was applied to reduce overfitting during fine-tuning. LoRA adapters were applied to all attention and feed-forward projection layers within the Transformer architecture.

Training was performed for 3 epochs using the AdamW 8-bit optimizer with a learning rate of 2 × 10⁻⁵, weight decay of 0.01 and a warmup ratio of 0.05. A cosine learning rate scheduler was employed to gradually reduce the learning rate over training. All models were trained on an NVIDIA RTX A100 40GB GPU, with training time ranging from 5 to 14 hours per model.

3.3 Model inference

Inference was conducted on the test split containing 36,669 examples. For all models, we set a maximum generation length of 128 tokens to encourage concise responses suitable for customer service interactions.

3.3.1 Small language models

Inference parameters for each SLM were configured according to recommendations provided by the original model publishers to ensure stable and representative performance. For all SLMs, we set the maximum generation length to 128 tokens and enabled sampling during inference. For SmolLM3-3B, we used a temperature of 0.6 with nucleus sampling set to 0.95 and a top-k value of 50. Qwen-3 models, including Qwen-3-4B, Qwen-3-1.7B and Qwen-3-8B, were configured with a temperature of 0.7, nucleus sampling of 0.8, a top-k value of 20 and a minimum probability threshold of 0. Phi-4-Mini employed a temperature of 0.7 with nucleus sampling set to 0.9 and a top-k value of 50. LLaMA-3.2-3B-Instruct, as well as LLaMA-3.2-1B-Instruct and LLaMA-3.1-8B-Instruct, were evaluated using identical decoding parameters, with a temperature of 0.7, nucleus sampling of 0.9 and a top-k value of 50. Gemma-3-4B-Instruct was configured with a temperature of 0.6, nucleus sampling of 0.95, a top-k value of 64 and a repetition penalty of 1.15 to reduce redundant phrasing.

3.3.2 Large language models

To benchmark the performance of the fine-tuned SLMs, we additionally evaluate three commercial LLMs, namely GPT-4.1, Gemini-2.5-Flash, and Virtuoso-Large. All proprietary LLMs were evaluated under identical input prompts and context conditions to ensure fair comparison with the fine-tuned SLMs. GPT-4.1, Virtuoso-Large and Gemini-2.5-Flash were configured with the same decoding parameters (temperature = 0.7, top-p = 0.9). Virtuoso-Large is accessed via the Arcee AI platform and is based on a Qwen-2.5-72B model architecture. As Gemini-2.5-Flash is a reasoning-oriented model, its thinking budget was explicitly set to zero to disable explicit reasoning during inference. For consistency, all LLMs were evaluated using the same maximum generation length of 128 tokens. All LLM inferences were conducted via their respective API endpoints using the same test set and evaluation settings as the fine-tuned SLMs.

4 Experimental setup and evaluation

We evaluate the performance of instruction-tuned SLMs for context-summarized multi-turn customer-service QA using a combination of quantitative and qualitative evaluation methods. The evaluation framework is designed to assess both surface-level alignment with reference answers and higher-level conversational quality, including contextual continuity, tone and task completion. All models are evaluated under identical experimental conditions using the same test set and input format.

4.1 Quantitative evaluation

Quantitative evaluation is conducted on the full test split of 36,669 examples, focusing on lexical and semantic similarity between generated agent responses and refined reference answers. Although automatic metrics cannot fully capture conversational quality, they provide a reproducible and scalable measure of response alignment.

Lexical similarity is evaluated using ROUGE-L (measures longest common subsequence overlap between generated and reference text) and METEOR (Metric for Evaluation of Translation with Explicit ORdering). ROUGE-L (Lin, 2004) measures the longest common subsequence between the generated response and reference answer, capturing structural overlap, while METEOR (Banerjee and Lavie, 2005) accounts for exact matches, stemming and synonym matches, enabling flexible lexical comparison. Semantic similarity is assessed using BERTScore (F1), BARTScore and cosine similarity between sentence embeddings. BERTScore computes token-level semantic alignment using contextual embeddings (Zhang et al., 2020), BARTScore evaluates the likelihood of generating the reference text from the model output using a pre-trained BART model and cosine similarity captures sentence-level semantic closeness using all-mpnet-base-v2 embeddings. All quantitative metrics are computed using the Hugging Face evaluate library. The official implementation is used for BERTScore, BARTScore is computed with the bart-large model (Yuan et al., 2021) and cosine similarity is calculated over normalized sentence embeddings. Higher values indicate better performance for all metrics except BARTScore, where values closer to zero indicate stronger alignment. Results are reported in Table 2.

4.2 Conversation stage segmentation

Before qualitative evaluation, test instances are grouped into three conversation stages: Early, Mid, and Late. This stage-based segmentation reflects the natural progression of customer-service interactions, where early-stage turns focus on issue identification, mid-stage turns contain the core interaction and information exchange and late-stage turns emphasize resolution and closure. Conversation stage assignment is performed using the GPT-4.1-mini model with a temperature of 0 to ensure deterministic and reproducible outputs. This segmentation enables controlled sampling, balanced stage-wise coverage and targeted analysis of model behavior under varying contextual demands. For both human and LLM-as-a-judge evaluation, samples are selected using a fixed ratio of 10% early-stage, 80% mid-stage, and 10% late-stage instances, placing greater emphasis on mid-stage interactions that require stronger contextual reasoning and dialogue continuity. Based on this segmentation, we conduct a stage-based qualitative analysis that evaluates model performance at the Early, Mid and Late stages of customer-service conversations and apply this analysis consistently across both evaluation settings.

4.3 LLM-as-a-judge evaluation

To assess conversational quality at scale, we employ an LLM-as-a-judge evaluation framework based on the G-Eval methodology (Liu et al., 2023). A specialized prompt is used to score generated responses across four qualitative dimensions: Human-Likeness, Continuity and Context Understanding, Tone and Clarity and Task Appropriateness. Each dimension is scored independently on a 1-5 Likert scale, where higher scores indicate better performance. Claude Sonnet 4.5 was selected as the judge model because it was not among the models being evaluated, which helps avoid any bias a model might have toward its own outputs. A temperature of 0 was used to minimize output variability, ensuring that the judge produces consistent and reproducible scores and judgments across all evaluated models. For each response, the judge is provided with the summarized conversation history, the client question, a reference agent response supplied only as guidance and the model-generated response. LLM-as-a-judge evaluation is performed on 6,000 randomly sampled test instances per model. Scores are averaged across all evaluated instances to obtain overall performance metrics, which are reported in Table 3.

4.4 Human evaluation

Human evaluation was conducted to provide a gold-standard assessment of conversational quality. Due to limited evaluation resources, this analysis was restricted to SLMs in the 3-4B parameter range and the selected commercial LLMs. Three human evaluators independently assessed model-generated responses using the same qualitative dimensions as the LLM-as-a-judge evaluation. Evaluators were provided with the summarized conversation history, the client question and the model-generated response and model identities were hidden to avoid bias. Scores were assigned on a 1-5 Likert scale and averaged across evaluators. This setup enabled both overall analysis of conversational performance and allowed direct comparison with the LLM-as-a-judge results. Aggregated overall human evaluation results are reported in Table 4.

4.5 Pairwise evaluation

Pairwise evaluation is conducted between selected high-performing SLMs and commercial LLMs. For each input, two responses (A and B) are compared directly and the judge selects the better response overall (Park et al., 2024; Liu et al., 2024). The evaluation uses 1,000 test instances with responses generated by the selected models and Claude Haiku 4.5 as the judge, with the temperature set to 0. To mitigate positional bias, each instance is evaluated twice by swapping the A and B ordering. If the same model is preferred in both orderings, the outcome is recorded as a win for that model. If the preferred model differs across orderings, the instance is treated as a tie. The judge applies the same four qualitative criteria used in the Likert-scale evaluations and outputs a single winner per comparison. This evaluation provides a direct preference-based comparison between LLMs and SLMs that complements score-based assessments by highlighting relative response quality under identical prompts. Results are reported as win and tie percentages for each model pair and summarized in Table 5.

4.6 Conversational stage-based evaluation

Most existing customer-service QA research reports qualitative results at the overall conversation level. However, the performance of fine-tuned models can vary across different conversation stages (Early, Mid, and Late). To capture these variations, we apply the same qualitative evaluation approach across stage-wise grouped conversations, following the segmentation procedure described in Section 4.2.

4.6.1 Conversational stage-based LLM-as-a-judge evaluation

The stage-wise LLM-as-a-judge evaluation is conducted using Claude Sonnet 4.5 as the judge on the same set of 6,000 responses used in the overall LLM-as-a-judge evaluation, segmented into 600 Early-stage, 4,800 Mid-stage and 600 Late-stage responses, following the segmentation ratio described in Section 4.2. Scores are assigned independently across four qualitative dimensions using a 1-5 Likert scale, following the same criteria described in Section 4.3. Table 6 reports the LLM-as-a-judge-based stage-wise results separately for each conversation phase, enabling a direct comparison of model behavior across Early, Mid, and Late interactions. This stage-based analysis enables identification of models that perform consistently across stages and models that show performance variations across different conversation phases.

4.6.2 Conversational stage-based human evaluation

The stage-wise human evaluation is conducted on the same set of 500 responses per model used in the overall human evaluation, segmented into 50 Early-stage, 400 Mid-stage, and 50 Late-stage instances, following the same assessment criteria and scoring procedure described in Section 4.4. However, due to limitations in human evaluation resources, this experiment is also conducted only on commercial LLMs and SLMs in the 3-4B parameter range. Scores are reported separately for each conversation stage and qualitative dimensions. Stage-wise human evaluation results are reported in Table 7, enabling direct cross-validation of stage-wise patterns observed in the LLM-as-a-judge evaluation and providing a gold-standard assessment of model behavior across different conversation phases.

4.6.3 Conversational stage-based pairwise evaluation

The stage-wise pairwise evaluation is conducted on the same 1,000 randomly selected test instances used in Section 4.5, with responses evaluated using Claude Haiku 4.5 as the judge (temperature set to 0). These instances are segmented into Early, Mid, and Late stages following the procedure described in Section 4.2, with a distribution of 80.4% Mid-stage, 9.9% Early-stage, and 9.7% Late-stage samples. Win and tie percentages are computed separately for each conversation stage to directly compare SLMs with commercial LLMs. Table 8 reports the stage-wise results, enabling comparison of model performance across different conversation phases.

5 Discussion on obtained results

The evaluation framework applied in this study provides comprehensive insight into the performance of fine-tuned instruction-tuned SLMs for context-summarized multi-turn customer-service QA. Although the majority of fine-tuned SLMs obtain higher scores than commercial LLMs on quantitative metrics (Table 2), this primarily reflects closer alignment with the reference response distribution used for evaluation rather than overall conversational quality. Commercial LLMs achieve higher scores in LLM-as-a-judge (Table 3) and human evaluation (Table 4), demonstrating that lexical and semantic similarity metrics alone are insufficient for evaluating context-summarized multi-turn customer-service QA.

Quantitative evaluation shows that the strongest fine-tuned SLMs achieve consistently competitive performance across lexical and semantic metrics (Table 2). Qwen-3-4B-Instruct attains the highest scores in ROUGE-L (0.3959), BARTScore (–2.2311), and BERTScore F1 (0.9137), while LLaMA-3.1-8B-Instruct records the highest METEOR score (0.4569) and cosine similarity (0.7051). LLaMA-3.2-3B-Instruct (ROUGE-L: 0.3842, BERTScore F1: 0.9121) and Phi-4-Mini (ROUGE-L: 0.3747, BERTScore F1: 0.9107) also perform competitively, indicating strong lexical overlap and semantic alignment. Within the less than 2B parameter range, Qwen-3-1.7B-Instruct (ROUGE-L: 0.3697, BERTScore F1: 0.9096) outperforms LLaMA-3.2-1B-Instruct (ROUGE-L: 0.2332, BERTScore F1: 0.8821). In addition, SmolLM3-3B-Instruct (ROUGE-L: 0.2393, cosine similarity: 0.5428) and Gemma-3-4B-Instruct (ROUGE-L: 0.2024, cosine similarity: 0.5134) record the weakest quantitative performance. Notably, the strongest fine-tuned SLMs outperform all three commercial LLMs across most automatic metrics, where GPT-4.1 (ROUGE-L: 0.3038, cosine similarity: 0.6749), Gemini-2.5-Flash (ROUGE-L: 0.2771, cosine similarity: 0.6234), and Virtuoso-Large (ROUGE-L: 0.3161, cosine similarity: 0.6676) obtain lower scores, suggesting that domain-specific fine-tuning of SLMs improves response alignment for customer-service interactions.

In LLM-as-a-judge evaluation, GPT-4.1 achieves the strongest overall performance (4.146), while several fine-tuned SLMs show competitive performance, mainly in Human-Likeness and Tone and Clarity. LLaMA-3.1-8B-Instruct (3.794) outperforms Gemini-2.5-Flash (3.769), with strong Human-Likeness (4.115) and Tone and Clarity (4.149) (Table 3). Qwen-3-8B-Instruct (3.743), LLaMA-3.2-3B-Instruct (3.718), Qwen-3-4B-Instruct (3.679), and Phi-4-Mini (3.619) also achieve solid overall performance. However, Continuity and Context Understanding and Task Appropriateness remain lower across SLMs, indicating limitations in maintaining dialogue coherence and task completion. Within the 3-4B parameter range, LLaMA-3.2-3B-Instruct, Qwen-3-4B-Instruct and Phi-4-Mini show stronger performance, while SmolLM3-3B (2.210) and Gemma-3-4B-Instruct (2.145) record the weakest results, especially in Continuity and Context Understanding. Within the less than 2B parameter range, Qwen-3-1.7B-Instruct (3.478) performs better than LLaMA-3.2-1B-Instruct (2.759), although both remain below 3-4B parameter SLMs. Stage-based analysis shows clear performance variation across conversation phases (Table 6). In Early-stage interactions, LLaMA-3.1-8B-Instruct (3.819), LLaMA-3.2-3B-Instruct (3.800), Qwen-3-4B-Instruct (3.764), and Phi-4-Mini (3.700) achieve the strongest SLM results. In Mid-stage interactions, most SLMs show lower performance in Continuity and Task Appropriateness, indicating difficulty in maintaining coherent responses. However, LLaMA-3.1-8B-Instruct (3.737) and Qwen-3-8B-Instruct (3.712) remain competitive with Gemini-2.5-Flash (3.752). In Late-stage interactions, performance improves, where leading SLMs achieve scores above 4.1, with LLaMA-3.1-8B-Instruct (4.229) surpassing Gemini-2.5-Flash (3.818).

Human evaluation shows similar trends (Table 4). Among the evaluated 3-4B parameter models, LLaMA-3.2-3B-Instruct achieves the highest SLM score (4.146), with strong performance in Human-Likeness (4.250), Continuity and Context Understanding (4.325) and Tone and Clarity (4.286). Qwen-3-4B-Instruct (4.069) and Phi-4-Mini (4.059) also perform at a similar level, though Task Appropriateness remains lower (3.553-3.721), indicating that accurate task resolution is still the main limitation despite strong fluency and contextual alignment. Compared to commercial LLMs, LLaMA-3.2-3B-Instruct (4.146) performs close to Gemini-2.5-Flash (4.191), with similar Human-Likeness and Tone and Clarity, while GPT-4.1 (4.627) and Virtuoso-Large (4.529) maintain higher scores in Continuity and Task Appropriateness. SmolLM3-3B (2.711) and Gemma-3-4B-Instruct (2.686) obtain lower scores across all qualitative dimensions, consistent with LLM-as-a-judge results, showing that not all SLM architectures within the 3-4B range are equally suited for this task. Stage-based human evaluation (Table 7) shows consistent variation across conversation phases. In Early-stage interactions, scores range from 3.882 to 4.005, with LLaMA-3.2-3B-Instruct (4.005) and Qwen-3-4B-Instruct (3.915) achieving the highest SLM results. Mid-stage performance remains stable (4.019-4.104) and competitive with Gemini-2.5-Flash (4.177), although GPT-4.1 (4.632) and Virtuoso-Large (4.517) remain higher. Late-stage performance is the strongest, where LLaMA-3.2-3B-Instruct (4.620), Phi-4-Mini (4.548), and Qwen-3-4B-Instruct (4.532) exceed Gemini-2.5-Flash (4.278), while GPT-4.1 (4.755) and Virtuoso-Large (4.753) remain above all SLMs.

Pairwise evaluation (Table 5) shows SLM strength mainly against Gemini-2.5-Flash, where Qwen-3-8B-Instruct achieves the highest SLM win rate (55.8%), followed by LLaMA-3.1-8B-Instruct (52.9%) and LLaMA-3.2-3B-Instruct (49.7%). GPT-4.1 maintains clear dominance, with LLM win rates ranging from 54.2% to 79.0%, while the strongest SLM (Qwen-3-8B-Instruct) reaches 23.8% wins. Against Virtuoso-Large, results are more balanced, with Qwen-3-8B-Instruct achieving 31.9% wins, but LLM win rates (46.7%–73.4%) remain higher overall. Stage-wise pairwise results (Table 8) indicate that SLM competitiveness is most evident against Gemini-2.5-Flash, particularly in the mid stage, where Qwen-3-8B-Instruct achieves 60.45% wins and LLaMA-3.1-8B-Instruct 55.97%. Against GPT-4.1 and Virtuoso-Large, LLM win rates remain higher in early and mid stages across most models. Late-stage performance increases for several SLMs, with Phi-4-Mini (52.58%), LLaMA-3.2-3B-Instruct (47.42%), and Qwen-3-4B-Instruct (45.36%) recording higher win rates against Gemini-2.5-Flash, while LLaMA-3.1-8B-Instruct achieves a 54.64% combined win and tie rate against GPT-4.1.

Across all evaluation methods (Tables 2–5), clear performance differences are observed across models in the less than 2B, 3-4B, and 8B parameter ranges. Within the less than 2B range, both models show limited competitiveness, with Qwen-3-1.7B-Instruct consistently outperforming LLaMA-3.2-1B-Instruct in LLM-as-a-judge (3.478 vs. 2.759) and pairwise evaluation. Within the 3-4B range, two distinct performance groups are observed: LLaMA-3.2-3B-Instruct, Qwen-3-4B-Instruct and Phi-4-Mini demonstrate strong performance with competitive pairwise win rates against Gemini-2.5-Flash (49.7%, 45.0%, 43.9%), while SmolLM3-3B and Gemma-3-4B-Instruct record substantially lower scores across all methods. In the 8B range, LLaMA-3.1-8B-Instruct and Qwen-3-8B-Instruct both show strong performance, with Qwen-3-8B-Instruct achieving the highest overall SLM pairwise win rate (55.8% against Gemini-2.5-Flash). These results indicate that higher parameter capacity generally leads to stronger performance, although model architecture within the same parameter range also plays an important role.

SLM performance shows a consistent stage-wise pattern across all evaluation methods (Tables 6–8), with moderate performance in the Early stage, lowest in the Mid stage and strongest in the Late stage. In the Early stage, top SLMs achieve competitive results, including LLaMA-3.1-8B-Instruct (3.819) in LLM-as-a-judge, LLaMA-3.2-3B-Instruct (4.005) in human evaluation and pairwise win rates reaching up to 44.44% against Gemini-2.5-Flash. Mid-stage interactions are the most challenging, with the largest gaps in Continuity and Context Understanding and Task Appropriateness, although Qwen-3-8B-Instruct achieves a 60.45% pairwise win rate against Gemini-2.5-Flash at this stage. In the Late stage, fine-tuned SLMs achieve their strongest results across all methods, where LLaMA-3.1-8B-Instruct surpasses Gemini-2.5-Flash in LLM-as-a-judge (4.229 vs. 3.818), and LLaMA-3.2-3B-Instruct (4.620), Phi-4-Mini (4.548) and Qwen-3-4B-Instruct (4.532) exceed Gemini-2.5-Flash (4.278) in human evaluation, while Phi-4-Mini achieves a 52.58% pairwise win rate, indicating stronger performance in resolution-focused interactions.

Results also show that model architecture impacts adaptability in customer-service QA settings, as this experiment is conducted under instruction-tuned settings. Although thinking budgets are set to enforce instruction-tuned behavior, Gemini-2.5-Flash and Qwen-3-8B-Instruct, which are reasoning-oriented models, show mixed performance when explicit reasoning is disabled through the removal of thinking budgets and tags. Under this setup, they perform lower than several instruction-focused SLMs. Qwen-3-8B-Instruct also underperforms compared to Qwen-3-4B-Instruct in some stages despite having more parameters (Table 6), suggesting that reasoning-oriented architectures may require careful adaptation for instruction-tuned settings. In addition, Gemma-3-4B-Instruct, which is based on a multimodal architecture, also records lower performance across qualitative dimensions even after disabling multimodal components (Tables 3, 4), indicating that such architectures may be less suitable for this task under instruction-tuned settings. Overall, fine-tuned SLMs, including 3-4B models such as Qwen-3-4B-Instruct, LLaMA-3.2-3B-Instruct and Phi-4-Mini, as well as 8B models such as LLaMA-3.1-8B-Instruct and Qwen-3-8B-Instruct, demonstrate strong performance for context-summarized multi-turn customer-service QA. However, SmolLM3-3B-Instruct and Gemma-3-4B-Instruct show consistent limitations across qualitative dimensions, indicating that not all SLM architectures are equally suitable for this task despite similar parameter sizes.

6 Conclusion and future work

This study provides a comprehensive evaluation of finetuned instruction-tuned SLMs for multi-turn context-summarized customer-service QA. In addition to the evaluation framework, we introduced a context-summarized synthetic multi-turn customer-service QA dataset designed to address privacy constraints and the lack of publicly available multi-turn conversational resources. Using automatic metrics, LLM-as-a-judge evaluation, human assessment, pairwise comparison and the conversation stage-based evaluation framework proposed in this work, the experiments examine how well SLMs maintain dialogue continuity, contextual understanding and response appropriateness across different phases of customer-service interactions. Results show that leading fine-tuned SLMs in the 3-8B parameter range, specifically Qwen-3-4B-Instruct, LLaMA-3.2-3B-Instruct, Phi-4-Mini, LLaMA-3.1-8B-Instruct and Qwen-3-8B-Instruct, have gained near-LLM performance in this task, particularly in human-likeness, tone and clarity and late-stage resolution. The proposed stage-based analysis demonstrates that SLM performance varies across Early, Mid and Late conversation phases, with mid-stage interactions presenting the greatest challenge due to higher contextual demands, and late-stage interactions producing the strongest results across all evaluation methods. Nevertheless, Continuity and Context Understanding and Task Appropriateness remain the primary limitations across all fine-tuned SLMs, with commercial LLMs such as GPT-4.1 and Virtuoso-Large continuing to achieve stronger results in these dimensions. SmolLM3-3B and Gemma-3-4B-Instruct show consistent limitations across all qualitative dimensions, indicating that model architecture plays a critical role in task suitability beyond parameter size alone.

These findings suggest that effective multi-turn customer-service systems do not necessarily require very large models, even though slight deviations remain between the best-performing SLMs and commercial LLMs. With context summarization and instruction tuning, SLMs offer a strong balance between performance and efficiency. The dataset contribution further supports reproducible research by providing structured multi-turn conversational instances suitable for evaluating dialogue continuity under privacy-aware conditions. This has practical and societal benefits, as smaller models reduce computational cost and energy usage, enabling wider access to customer-service automation while supporting privacy-conscious deployment. Overall, the study positions SLMs as a practical and scalable solution for multi-turn context-summarized customer-service QA, encouraging broader adoption of efficient conversational AI beyond high-resource settings.

Future work can extend evaluation to sectors such as healthcare, telecommunications and e-commerce to assess cross-domain robustness and the generalization capability of fine-tuned SLMs across different customer-service settings. Evaluation can also be expanded using real-world customer-service conversations, enabling assessment under more realistic interaction patterns and operational constraints. In addition, the dataset and evaluation can also be extended to multilingual settings, including code-mixed conversations, to assess the effectiveness of fine-tuned SLMs for multilingual customer-service interactions. Preference optimization methods, including Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning from AI Feedback (RLAIF), can be explored to further improve conversational quality and better align SLMs with human expectations, particularly for dialogue continuity, tone control and task appropriateness (Wang et al., 2024). Future work can also investigate end-to-end customer-service systems where both multi-turn context summarization and response generation are performed using SLMs, supporting on-premise deployment and strengthening privacy protection within customer-service QA systems. Benchmarking medium-scale models is also valuable, as results indicate that performance within SLMs generally increases with parameter count, and further analysis can be conducted to examine the trade-offs between efficiency and conversational performance within small language models.

Limitations

Since this study is conducted primarily on a banking-domain corpus, which may limit generalization to other customer-service domains. Although the synthetic dataset approach supports privacy preservation and avoids exposure of sensitive customer data, it may not capture the full variability of real-world interactions. These factors may influence the generalizability of the results beyond the evaluated setting. In addition, due to limited resources, human evaluation was conducted only on selected 3-4B SLMs and three commercial LLMs rather than across all evaluated models.

References

• 1

  AbdinM.AnejaJ.BehlH.BubeckS.EldanR.GunasekarS.et al. (2024). Phi-4 technical report. arXiv preprint arXiv:2412.08905.

  • Google Scholar

• 2

  AboueleninA.AshfaqA.AtkinsonA.AwadallaH.BachN.BaoJ.et al. (2025). Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743.

  • Google Scholar

• 3

  AlbuquerqueM.BarbosaL.MoreiraJ.SilvaA.daMelo, T. (2024). “Fine-tuning open-source large language models for automated response to customer feedback,” in Anais Do XII Symposium on Knowledge Discovery, Mining and Learning (KDMiLe 2024), 65–72. doi: 10.5753/kdmile.2024.244556

  • CrossRef
  • Google Scholar

• 4

  AllalL. B.LozhkovA.BakouchE.BlázquezG. M.PenedoG.TunstallL.et al. (2025). SmolLM2: when smol goes big–data-centric training of a small language model. arXiv preprint arXiv:2502.02737.

  • Google Scholar

• 5

  AzovG.PelcT.Fledel AlonA.KamhiG. (2024). “Self-improving customer review response generation based on LLMs,” in Proceedings of the Seventh Workshop on E-Commerce and NLP @ LREC-COLING 2024, 40–57. doi: 10.63317/2przze2uzjqk

  • CrossRef
  • Google Scholar

• 6

  BanerjeeS.LavieA. (2005). “METEOR: an automatic metric for MT evaluation with improved correlation with human judgments,” in Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, eds. J. Goldstein, A. Lavie, C.-Y. Lin, and C. Voss (Ann Arbor, MI: Association for Computational Linguistics), 65–72.

  • Google Scholar

• 7

  BelcakP.HeinrichG.DiaoS.FuY.DongX.MuralidharanS.et al. (2025). Small language models are the future of agentic AI. arXiv preprint arXiv:2506.02153.

  • Google Scholar

• 8

  BirdJ. J. (2021). Improving customer service chatbots with attention-based transfer learning. arXiv preprint arXiv:2111.14621.

  • Google Scholar

• 9

  ChenJ.TungomC. E.ZhongG. (2024). “LLM intelligent customer service in property management using a RAG approach,” in 2024 4th International Conference on Artificial Intelligence, Robotics, and Communication (ICAIRC), 852–860. doi: 10.1109/ICAIRC64177.2024.10900207

  • CrossRef
  • Google Scholar

• 10

  ChungH. W.HouL.LongpreS.ZophB.TayY.FedusW.et al. (2024). Scaling instruction-finetuned language models. J. Mach. Learn. Res. 25, 1–53. doi: 10.48550/arXiv.2210.11416

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 11

  CuiY.ZhaoF.JiaoY.SuY.ZhouY.LiuJ. (2025). “Research on fine-tuning and optimization techniques of language models in the field of electric power customer service,” in 2025 2nd International Conference on Smart Grid and Artificial Intelligence (SGAI), 1622–1625. doi: 10.1109/SGAI64825.2025.11009601

  • CrossRef
  • Google Scholar

• 12

  DettmersT.HoltzmanA.PagnoniA.ZettlemoyerL. (2023). “QLoRA: efficient finetuning of quantized LLMs,” in Advances in Neural Information Processing Systems, 10088–10115. doi: 10.52202/075280-0441

  • CrossRef
  • Google Scholar

• 13

  GargM.RazaS.RayanaS.LiuX.SohnS. (2025). The rise of small language models in healthcare: a comprehensive survey. arXiv preprint arXiv:2504.17119.

  • Google Scholar

• 14

  GrattafioriA.DubeyA.JauhriA.PandeyA.KadianA.Al-DahleA.et al. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

  • Google Scholar

• 15

  GuJ.JiangX.ShiZ.TanH.ZhaiX.XuC.et al. (2026). A survey on LLM-as-a-judge. Innovation2026:101253. doi: 10.1016/j.xinn.2025.101253

  • CrossRef
  • Google Scholar

• 16

  HuE. J.ShenY.WallisP.Allen-ZhuZ.LiY.WangS.et al. (2022). Lora: low-rank adaptation of large language models. ICLR1:3. doi: 10.48550/arXiv.2106.09685

  • CrossRef
  • Google Scholar

• 17

  IlseB.BlackwoodF. (2024). Comparative analysis of finetuning strategies and automated evaluation metrics for large language models in customer service chatbots. Res. Sq. doi: 10.21203/rs.3.rs-4895456/v1

  • CrossRef
  • Google Scholar

• 18

  KamisettyR.NagamangalamR. (2025). Transforming banking with LLMs: enhancing customer experience, fraud detection, and decision-making through AI. Int. Res. J. Innov. Eng. Technol. 9, 172–180. doi: 10.47001/IRJIET/2025.902026

  • CrossRef
  • Google Scholar

• 19

  LeeS.AryaD.ChoS.-M.HanG.HongS.JangW.et al. (2024). “TelBench: a benchmark for evaluating telco-specific large language models,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, 609–626. doi: 10.18653/v1/2024.emnlp-industry.45

  • CrossRef
  • Google Scholar

• 20

  LiF.WangY.XuY.WangS.LiangJ.ChenZ.et al. (2024). Performance evaluations of large language models for customer service. Int. J. Mach. Learn. Cyber. 16, 2997–3017. doi: 10.1007/s13042-024-02432-9

  • CrossRef
  • Google Scholar

• 21

  LiX. L.LiangP. (2021). “Prefix-tuning: optimizing continuous prompts for generation,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 4582–4597. doi: 10.18653/v1/2021.acl-long.353

  • CrossRef
  • Google Scholar

• 22

  LiY.WangS.DingH.ChenH. (2023). “Large language models in finance: a survey,” in 4th ACM International Conference on AI in Finance, 374–382. doi: 10.1145/3604237.3626869

  • CrossRef
  • Google Scholar

• 23

  LijayaH.HoP.SantosoH. (2025). Comparative analysis of RAG-based open-source LLMs for Indonesian banking customer service optimization using simulated data. J. Sisfokom14, 330–341. doi: 10.32736/sisfokom.v14i3.2383

  • CrossRef
  • Google Scholar

• 24

  LinC.-Y. (2004). “ROUGE: a package for automatic evaluation of summaries,” in Text Summarization Branches Out (Barcelona, Spain: Association for Computational Linguistics), 74–81.

  • Google Scholar

• 25

  LiuY.IterD.XuY.WangS.XuR.ZhuC. (2023). “G-Eval: NLG evaluation using GPT-4 with better human alignment,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2511–2522. doi: 10.18653/v1/2023.emnlp-main.153

  • CrossRef
  • Google Scholar

• 26

  LiuY.ZhouH.GuoZ.ShareghiE.VuliI.KorhonenA.et al. (2024). Aligning with human judgement: the role of pairwise preference in large language model evaluators. arXiv preprint arXiv:2403.16950.

  • Google Scholar

• 27

  LovtsovV. A.SkvortsovaM. A. (2025). “Automated mobile operator customer service using large language models combined with RAG system,” in 2025 7th International Youth Conference on Radio Electronics, Electrical and Power Engineering (REEPE), 1–6. doi: 10.1109/REEPE63962.2025.10971107

  • CrossRef
  • Google Scholar

• 28

  MeconiD.StirpeS.MartelliF.LavalleL.NavigliR. (2025). “Do large language models understand word senses?,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 33885–33904. doi: 10.18653/v1/2025.emnlp-main.1720

  • CrossRef
  • Google Scholar

• 29

  PapineniK.RoukosS.WardT.ZhuW.-J. (2001). “BLEU,” in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL –02, 311. doi: 10.3115/1073083.1073135

  • CrossRef
  • Google Scholar

• 30

  ParkC.ChoiM.LeeD.ChooJ. (2024). Paireval: open-domain dialogue evaluation with pairwise comparison. arXiv preprint arXiv:2404.01015.

  • Google Scholar

• 31

  QwenA. Y.YangB.ZhangB.HuiB.ZhengB.YuB.et al. (2025). Qwen2.5 technical report. arXiv. Available online at: https://arxiv.org/abs/2412.15115

  • Google Scholar

• 32

  RaffelC.ShazeerN.RobertsA.LeeK.NarangS.MatenaM.et al. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21, 1–67. doi: 10.48550/arXiv.1910.10683

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 33

  SanjaniL. A.SarnoR.SungkonoK. R.HaryonoA. T.SeptiyantoA. F.SunaryonoD. (2025). “Performance analysis of LLM models with RAG and fine-tuning T5 for chatbot optimization in call centers,” in 2025 International Conference on Computer Sciences, Engineering, and Technology Innovation (ICoCSETI), 152–157. doi: 10.1109/ICoCSETI63724.2025.11018908

  • CrossRef
  • Google Scholar

• 34

  ShiJ.LiJ.MaQ.YangZ.MaH.LiL. (2024). Chops: chat with customer profile systems for customer service with llms. arXiv preprint arXiv:2404.01343.

  • Google Scholar

• 35

  Talkmap (2024). Customer service banking conversation corpus. Available online at: https://huggingface.co/datasets/talkmap/banking-conversation-corpus (accessed January 17, 2026).

  • Google Scholar

• 36

  TeamG.KamathA.FerretJ.PathakS.VieillardN.MerhejR.et al. (2025). Gemma 3 technical report.

  • Google Scholar

• 37

  TeamG.RiviereM.PathakS.SessaP. G.HardinC.BhupatirajuS.et al. (2024). Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. doi: 10.48550/arXiv.2408.00118

  • CrossRef
  • Google Scholar

• 38

  TouvronH.MartinL.StoneK.AlbertP.AlmahairiA.BabaeiY.et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

  • Google Scholar

• 39

  VaswaniA.ShazeerN.ParmarN.UszkoreitJ.JonesL.GomezA. N.et al. (2017). “Attention is all you need,” in Advances in Neural Information Processing Systems, 30.

  • Google Scholar

• 40

  WangJ.-Y.KuoM.-F.HanJ.-C.ShihC.-C.ChenC.-H.LeeP.-C.et al. (2017). “A telecom-domain online customer service assistant based on question answering with word embedding and intent classification,” in Proceedings of the IJCNLP 2017, System Demonstrations, eds. S.-B. Park, and T. Supnithi (Tapei, Taiwan: Association for Computational Linguistics), 17–20.

  • Google Scholar

• 41

  WangZ.BiB.PentyalaS. K.RamnathK.ChaudhuriS.MehrotraS.et al. (2024). A comprehensive survey of LLM alignment techniques: Rlhf, rlaif, ppo, dpo and more. arXiv preprint arXiv:2407.16216.

  • Google Scholar

• 42

  WatanabeS.LeowC. S.HoshinoJ.UtsuroT.NishizakiH. (2024). “Assessment and improvement of customer service speech with multiple large language models,” in 2024 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 1–6. doi: 10.1109/APSIPAASC63619.2025.10849072

  • CrossRef
  • Google Scholar

• 43

  WulfJ.MeierhoferJ. (2024). Exploring the potential of large language models for automation in technical customer service. arXiv preprint arXiv:2405.09161.

  • Google Scholar

• 44

  XiaoliangM.RuQiangZ.YingL.CongjianD.DequanD. (2024). Design of a large language model for improving customer service in telecom operators. Electron. Lett. 60:e13218. doi: 10.1049/ell2.13218

  • CrossRef
  • Google Scholar

• 45

  XuB.ChenY.WenZ.LiuW.HeB. (2025). “Evaluating small language models for news summarization: implications and factors influencing performance,” in Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 4909–4922. doi: 10.18653/v1/2025.naacl-long.253

  • CrossRef
  • Google Scholar

• 46

  YangA.YangB.HuiB.ZhengB.YuB.ZhouC.et al. (2024). Qwen2 technical report. arXiv. doi: 10.48550/arXiv.2407.10671

  • CrossRef
  • Google Scholar

• 47

  YuanW.NeubigG.LiuP. (2021). “Bartscore: evaluating generated text as text generation,” in Advances in Neural Information Processing Systems, 27263–27277.

  • Google Scholar

• 48

  ZhangQ.ChenM.BukharinA.KarampatziakisN.HeP.ChengY.et al. (2023). Adalora: adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512.

  • Google Scholar

• 49

  ZhangT.KishoreV.WuF.WeinbergerK. Q.ArtziY. (2020). Bertscore: evaluating text generation with bert. arXiv preprint arXiv:1904.09675. Available online at: https://arxiv.org/abs/1904.09675

  • Google Scholar