Keywords
neural networks
transformers
multilingual models
quality evaluation
BLEU
METEOR
POS-BLEU
How to Cite
Abstract
we conducted a comprehensive analysis of the current challenges and recent advancements in machine translation (MT), which plays a vital role in enabling multilingual communication in today’s globalized world. We focused on the application of advanced technologies – neural networks and transformer architectures – to improve translation accuracy. We evaluated leading MT systems (DeepL, Yandex Translate, and Google Translate) using standard quality metrics: BLEU, METEOR, and POS-BLEU. Our evaluation emphasized lexical matching, semantic fidelity, and syntactic structure. We developed a Python-based tool that automatically assessed translation quality using these metrics. The program identified weaknesses in translations and generated detailed reports to guide optimization. Our results showed that DeepL and Yandex achieved higher BLEU scores, indicating better performance in lexical and syntactic accuracy. However, all three systems exhibited issues with syntax, vocabulary choices, and article usage, highlighting areas that require further development.
We emphasize the importance of multi-dimensional evaluation – combining automated metrics with expert assessment. We also examined MT performance in specialized domains, where high translation precision is critical. Our findings underscore the need for continued research to further improve MT systems and adapt them to diverse linguistic and cultural contexts.
References
Воронович В. В. Машинный перевод: конспект лекций по специальности «Компьютерная лингвистика, Компьютерное обучение языкам». Минск: Белорусский государственный университет; 2013. 39 с.
Net.lang: на пути к многоязычному киберпространству: пер. с англ. / ред. перевода Е. И. Кузьмин, А. В. Паршакова. М.: Межрегиональный центр библиотечного сотрудничества; 2014. 464 с.
Jawahar G., Sagot B., Seddah D. What Does BERT Learn about the Structure of Language? ACL 201957th Annual Meeting of the Association for Computational Linguistics. 2019.
Devlin J. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805. 2018.
Rogers A., Kovaleva O., Rumshisky A. A Primer in BERTology: What We Know about How BERT Works. Transactions of the Association for Computational Linguistics. 2021;8:842–866.
Shukla A. et al. An Evaluation of Google Translate for Sanskrit to English Translation via Sentiment and Semantic Analysis. Natural Language Processing Journal. 2023;4:100025.
Caswell I. 110 New Languages are Coming to Google Translate. Google Translate Blog. 2024.
Fan A. et al. Beyond English-Centric Multilingual Machine Translation. Journal of Machine Learning Research. 2021;22:1–48.
Koehn P., Knowles R. Six Challenges for Neural Machine Translation. arXiv:1706.03872. 2017.
Maruf S., Saleh F., Haffari G. A Survey on Document-Level Neural Machine Translation: Methods and Evaluation. ACM Computing Surveys (CSUR). 2021;54(2):1–36.
Stahlberg F. Neural Machine Translation: A Review and Survey. arXiv:1912.02047. 2019.
Fitria T. N. Performance of Google Translate, Microsoft Translator, and DeepL Translator: Error Analysis of Translation Result. Al-Lisan: Jurnal Bahasa. 2023;8(2):115–138.
Varela-Salinas M. J. et al. Google Translate and DeepL: Breaking Taboos in Translator Training. Observational Study and Analysis. Ibérica. 2023;45:243–266.
Borisov A., Galinskaya I. Y. Yandex School of Data Analysis Russian-English Machine Translation System for WMT14. Proceedings of the Ninth Workshop on Statistical Machine Translation. 2014:66– 70.
Cambedda G., Di Nunzio G. M., Nosilia V. A Study on Automatic Machine Translation Tools: A Comparative Error Analysis Between DeepL and Yandex for Russian-Italian Medical Translation. Umanistica Digitale. 2021;10:139–163.
Kulikov V., Kulikova V., Yerkebulan G. Google/Yandex Translation Detection in the Patterns Identifying System of Multilingual Texts. International Journal of Computers. 2021;20(1):72–77.
Kamaluddin M. I. et al. Accuracy Analysis of DeepL: Breakthroughs in Machine Translation Technology. Journal of English Education Forum (JEEF). 2024;4(2):122–126.
Linlin L. Artificial Intelligence Translator DeepL Translation Quality Control. Procedia Computer Science. 2024;247:710–717.
Sriram A. et al. Cold Fusion: Training seq2seq Models Together with Language Models. arXiv:1708.06426. 2017.
Egonmwan E., Chali Y. Transformer and seq2seq Model for Paraphrase Generation. Proceedings of the 3rd Workshop on Neural Generation and Translation. 2019:249–255.
Li Z. et al. Seq2seq Dependency Parsing. Proceedings of the 27th International Conference on Computational Linguistics. 2018:3203–3214.
Sutskever I. Sequence to Sequence Learning with Neural Networks. arXiv:1409.3215. 2014.
Ni J. et al. Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models. arXiv:2108.08877. 2021.
Chen G. et al. Towards Making the Most of Cross-Lingual Transfer for Zero-Shot Neural Machine Translation. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022;1:142–157.
Rothman D. Transformers for Natural Language Processing: Build, Train, and Fine-Tune Deep Neural Network Architectures for NLP with Python, Hugging Face, and OpenAI’s GPT-3, ChatGPT, and GPT-4. Birmingham: Packt Publishing Ltd; 2022.
Maltas S. I. et al. Efficient Finetuning Strategies for Multilingual Neural Machine Translation. Universitat Politecnica de Catalunya; 2024.`
Bengesi S. et al. Advancements in Generative AI: A Comprehensive Review of GANs, GPT, Autoencoders, Diffusion Model, and Transformers. IEEE Access. 2024;PP(99):1–1.
Achiam J. et al. GPT-4 Technical Report. arXiv:2303.08774. 2023.
Stap D., Araabi A. ChatGPT is Not a Good Indigenous Translator. Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP). 2023:163–167.
Agarwal A., Lavie A. Meteor, M-BLEU and M-TER: Evaluation Metrics for High-Correlation with Human Rankings of Machine Translation Output. Proceedings of the Third Workshop on Statistical Machine Translation. 2008:115–118.
Babych B., Hartley T. Extending the BLEU MT Evaluation Method with Frequency Weightings. Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04). 2004:621–628.
Chen B., Cherry C. A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU. Proceedings of the Ninth Workshop on Statistical Machine Translation. 2014:362–367.
Papineni K. et al. BLEU: a Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 2002:311–318.
Perez D., Alfonseca E. Application of the BLEU Algorithm for Recognising Textual Entailments.´ Proceedings of the First Challenge Workshop Recognising Textual Entailment. 2005:9–12.
Banerjee S., Lavie A. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. 2005:65–72.
Denkowski M., Lavie A. Extending the METEOR Machine Translation Evaluation Metric to the Phrase Level. Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. 2010:250–253.
Denkowski M., Lavie A. Meteor Universal: Language-Specific Translation Evaluation for Any Target Language. Proceedings of the Ninth Workshop on Statistical Machine Translation. 2014:376–380.
Бесшапошников Н. О., Дьяченко М. С., Леонов А. Г., Матюшин М. А., Орловский А. Е. Использование машинного обучения и нейронных сетей для автоматической верификации заданий в текстовом и графическом представлении и помощи преподавателю. Успехи кибернетики.2020;1(2):35–41. DOI: 10.51790/2712-9942-2020-1-2-4.
Wang F. Methods for Evaluating the Translation Quality of Artificial Intelligence Translator DEEPL Based on Multi-Translation Parallel Corpus. 2024 First International Conference on Software, Systems and Information Technology (SSITCON). 2024:1–5.
Мыльников А. Г., Климов А. Е., Курбанниёзов Т. Ш., Буйместру Н. В., Черняева А. А., Гусарова Т. А. Экстренная тотальная дуоденопанкреатэктомия по поводу профузного желудочнокишечного кровотечения, вызванного метастазами рака почки в поджелудочную железу. Вестник Российского университета дружбы народов. Серия: Медицина. 2023;27(2):246–253. DOI: 10.22363/2313-0245-2023-27-2-246-253.