Swedish Medical LLM Benchmark (SMLB): Development and Evaluation of a Framework for Assessing Large Language Models in the Swedish Medical Domain

Birger Moell^1*Jonas Beskow¹Fabian Farenstam²

• ¹Division of Speech Music and Hearing, KTH Royal Institute of Technology, Stockholm, Sweden
• ²ETH Zürich, Zurich, Zürich, Switzerland

We present the Swedish Medical LLM Benchmark (SMLB), an evaluation framework for assessing Large Language Models (LLMs) in the Swedish medical domain. The SMLB addresses the lack of language-specific, clinically relevant benchmarks by incorporating four datasets: translated PubMedQA questions, Swedish Medical Exams, Emergency Medicine scenarios, and General Medicine cases. Our evaluation of 18 state-of-the-art LLMs reveals GPT-4-turbo, Claude-3.5 (October 2023), and the o3 model as top performers, demonstrating strong alignment between medical reasoning and general language understanding capabilities. Hybrid systems incorporating retrieval-augmented generation (RAG) improved accuracy for clinical knowledge questions, highlighting promising directions for safe implementation. The SMLB provides not only an evaluation tool but also reveals fundamental insights about LLM capabilities and limitations in Swedish healthcare applications, including significant performance variations between models.By open-sourcing the benchmark, we enable transparent assessment of medical LLMs while promoting responsible development through community-driven refinement. This work emphasizes the critical need for rigorous evaluation frameworks as LLMs become increasingly integrated into clinical workflows, particularly for non-English medical contexts where linguistic and cultural specificity are paramount.