Workshop — BLP — 2025

Benchmarking Large Language Models on Bangla Dialect Translation and Dialectal Sentiment Analysis

Md Mahir Jawad, Rafid Ahmed, Ishita Sur Apan, Tasnimul Hossain Tomal, Fabiha Haider, Mir Sazzat Hossain, Md Farhad Alam Bhuiyan

Best Long Paper Award

In Proceedings of the Second Workshop on Bangla Language Processing (BLP-2025), pages 322–337, Mumbai, India. Association for Computational Linguistics.

Abstract

We present a novel Bangla Dialect Dataset comprising 600 annotated instances across four major dialects: Chattogram, Barishal, Sylhet, and Noakhali. The dataset was constructed from YouTube comments spanning diverse domains to capture authentic dialectal variations in informal online communication. Each instance includes the original dialectical text, its standard Bangla translation, and sentiment labels (Positive and Negative). We benchmark several state-of-the-art large language models on dialect-to-standard translation and sentiment analysis tasks using zero-shot and few-shot prompting strategies. Our experiments reveal that transliteration significantly improves translation quality for closed-source models, with GPT-4o-mini achieving the highest BLEU score of 0.343 in zero-shot with transliteration. For sentiment analysis, GPT-4o-mini demonstrates perfect precision, recall, and F1 scores (1.000) in few-shot settings. This dataset addresses the critical gap in resources for low-resource Bangla dialects and provides a foundation for developing dialect-aware NLP systems.

Cite

@inproceedings{jawad-etal-2025-benchmarking,
    title = "Benchmarking Large Language Models on {B}angla Dialect Translation and Dialectal Sentiment Analysis",
    author = "Jawad, Md Mahir  and
      Ahmed, Rafid  and
      Apan, Ishita Sur  and
      Tomal, Tasnimul Hossain  and
      Haider, Fabiha  and
      Hossain, Mir Sazzat  and
      Bhuiyan, Md Farhad Alam",
    editor = "Alam, Firoj  and
      Kar, Sudipta  and
      Chowdhury, Shammur Absar  and
      Hassan, Naeemul  and
      Prince, Enamul Hoque  and
      Tasnim, Mohiuddin  and
      Rony, Md Rashad Al Hasan  and
      Rahman, Md Tahmid Rahman",
    booktitle = "Proceedings of the Second Workshop on Bangla Language Processing (BLP-2025)",
    month = dec,
    year = "2025",
    address = "Mumbai, India",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.banglalp-1.26/",
    doi = "10.18653/v1/2025.banglalp-1.26",
    pages = "322--337",
    ISBN = "979-8-89176-314-2",
    abstract = "We present a novel Bangla Dialect Dataset comprising 600 annotated instances across four major dialects: Chattogram, Barishal, Sylhet, and Noakhali. The dataset was constructed from YouTube comments spanning diverse domains to capture authentic dialectal variations in informal online communication. Each instance includes the original dialectical text, its standard Bangla translation, and sentiment labels (Positive and Negative). We benchmark several state-of-the-art large language models on dialect-to-standard translation and sentiment analysis tasks using zero-shot and few-shot prompting strategies. Our experiments reveal that transliteration significantly improves translation quality for closed-source models, with GPT-4o-mini achieving the highest BLEU score of 0.343 in zero-shot with transliteration. For sentiment analysis, GPT-4o-mini demonstrates perfect precision, recall, and F1 scores (1.000) in few-shot settings. This dataset addresses the critical gap in resources for low-resource Bangla dialects and provides a foundation for developing dialect-aware NLP systems."
}