Moses (software)

Moses (software)
Name	Moses
Developer	Center for Applied Language Technology (University of Edinburgh), RWTH Aachen University, DFKI
Latest release	2014.06?
Programming language	C++, Perl, Python
Operating system	Unix-like, Linux
License	LGPL

Moses (software) Moses is an open-source statistical machine translation toolkit originally created to enable researchers and developers to build, train, and deploy phrase-based and hierarchical translation models for language pairs such as English–French, German–English, and Chinese–English. The project integrates components for corpus preparation, alignment, phrase extraction, language modeling, tuning, and decoding, and has been used alongside toolkits and institutions like GIZA++, SRILM, KenLM, Europarl, and the ACL community.

Overview

Moses provides a pipeline for building statistical and phrase-based translation systems that interoperate with aligners and toolkits such as GIZA++, MGIZA, SRILM, KenLM, and datasets from Europarl, United Nations Parallel Corpus, WMT shared tasks. The toolkit supports translation models including phrase-based, hierarchical phrase-based, and factored models used in projects by groups at University of Edinburgh, RWTH Aachen University, and DFKI and evaluated in workshops organized by ACL, EMNLP, and NAACL.

History and Development

Development began in the mid-2000s as part of academic efforts at University of Edinburgh and expanded through collaborations with RWTH Aachen University and DFKI, with contributions from researchers who published in venues like ACL, EMNLP, and NAACL. Moses rose to prominence during the era of statistical machine translation exemplified by systems evaluated in WMT campaigns and compared to earlier approaches such as IBM models described by researchers at IBM Research and alignment tool development like GIZA++. Over time, Moses incorporated techniques from hierarchical models influenced by work from Philipp Koehn and others, and its community responded to paradigm shifts driven by neural methods presented in papers from Google Research and Facebook AI Research.

Architecture and Components

Moses' architecture comprises modules for preprocessing, alignment, phrase extraction, language modeling, tuning, and decoding, interoperating with external tools such as GIZA++, MGIZA, IRSTLM, SRILM, KenLM, and evaluation scripts used by WMT and ACL shared tasks. The toolkit includes a decoder implemented in C++ linked with scripts in Perl and Python, and supports factored models influenced by research from JHU and University of Edinburgh groups. Components are organized to interact with corpora like Europarl, ParaCrawl, and parallel datasets distributed by ELRA and LDC.

Features and Capabilities

Moses implements phrase-based and hierarchical models, factored translation, cube pruning decoding, and minimum error rate training as introduced in papers from Microsoft Research and Cambridge University. It supports integration with language models built by SRILM and KenLM, tuning algorithms such as MERT and MIRA used in systems from Johns Hopkins University and University of Maryland, and evaluation metrics popularized by work from Kishore Papineni et al. The toolkit also provides utilities for truecasing, tokenization, and recasing compatible with corpora from Europarl, United Nations Parallel Corpus, and datasets used in WMT.

Training and Decoding Workflow

Training in Moses typically follows a pipeline: corpus cleaning using scripts similar to those adopted by WMT, word alignment with GIZA++ or MGIZA, phrase extraction following formulations in papers by researchers at IBM Research and Philipp Koehn, language model training with SRILM or KenLM, tuning via MERT or MIRA as used in ACL evaluations, and decoding with the Moses decoder employing cube pruning and beam search techniques referenced in literature from Microsoft Research and Carnegie Mellon University. Decoding supports feature weights, reordering models influenced by research at RWTH Aachen University, and constraints for handling out-of-vocabulary items noted in studies from Johns Hopkins University.

Evaluation and Performance

Moses systems have been benchmarked in shared tasks like WMT and reported in proceedings of ACL and EMNLP, often evaluated with BLEU scores introduced by researchers at IBM Research and complemented by metrics discussed at WMT workshops. Performance depends on corpora such as Europarl, ParaCrawl, and United Nations Parallel Corpus, model choices (phrase-based vs hierarchical), language modeling via KenLM or SRILM, and tuning strategies from MIRA or MERT. Comparative studies in journals and conferences contrasted Moses with systems from Google Translate, Microsoft Translator, and neural models from Google Brain and Facebook AI Research.

Usage and Applications

Moses has been applied in academic research at institutions like University of Edinburgh, RWTH Aachen University, DFKI, and Johns Hopkins University and in industry prototypes by teams at Microsoft Research and IBM Research. Use cases include building bilingual corpora for European Commission projects, prototyping translation services for NGOs, preprocessing pipelines for corpora curated by ELRA and LDC, and serving as a baseline in WMT shared tasks and ACL evaluations. The toolkit has also been embedded in workflows that later migrated to neural toolkits from TensorFlow and PyTorch research groups.

Licensing and Community Support

Moses is distributed under the LGPL license and has attracted a community of contributors from universities and labs including University of Edinburgh, RWTH Aachen University, DFKI, and researchers who published in ACL and EMNLP venues. Community support has historically been coordinated via mailing lists, workshops at WMT and ACL events, and repositories hosting code and scripts used by participants in shared tasks organized by WMT and datasets distributed by ELRA and LDC.

Category:Machine translation