Hallucination in Turkish–Arabic LLM Translation Outputs: A Text-Type Based Translation Studies Analysis
Sezer Yılmaz
Abstract. Hallucination, which has emerged as a significant research problem in machine translation literature, refers to a phenomenon in which large language models (LLMs), despite producing fluent and seemingly coherent translation outputs, may distort source-text information, generate information not found in the source, or produce outputs with a weak semantic connection to it. This study examines hallucination types that may arise in turkish – arabic machine translation from a translation studies perspective. Considering the morphological and syntactic characteristics of Turkish and Arabic, this language pair offers a productive and distinctive field of inquiry for discussing hallucination in translation. The study analyzes three sample texts representing literary, legal, and medical text types through outputs generated by ChatGPT, Grok, and Gemini. The translation outputs were qualitatively examined in terms of fidelity to the source text, semantic integrity, terminological accuracy, source-external additions, task-irrelevant explanations, and repetition. The qualitative findings were interpreted in relation to BERTScore results calculated in Google Colaboratory. BERTScore was treated as a complementary indicator of semantic proximity between the machine translation outputs and the human reference translation, while the final identification and classification of hallucination and deviation types were based on qualitative comparison with the source text. The analyses show that a high semantic similarity score does not always mean that the translation is free from hallucination. In some outputs, even when the score is high, task-irrelevant explanations, interpretive expansions, terminological deviations, or repetitions can still be found. For this reason, automatic metrics should not be used alone when evaluating hallucinated translation outputs. In this study, hallucination is examined together with source-text fidelity, automatic evaluation metrics, and text type.
Keywords: hallucination, machine translation, large language models, turkish–arabic translation, BERTScore, translation studies