Methods to Translate Languages with MarianMT and Hugging Face Transformers

August 29, 2024

Picture by Creator | Canva

Language translation has turn out to be a necessary device in our more and more globalized world. Whether or not you are a developer, researcher, or traveler, you’ll all the time discover the necessity to talk with folks from totally different cultures. Therefore, the power to translate textual content shortly and precisely will be very useful for you. One highly effective useful resource for reaching that is the MarianMT mannequin, part of the Hugging Face Transformers library.

On this information, we’ll stroll you thru the method of utilizing MarianMT to translate textual content between a number of languages, making it accessible even for these with minimal technical background.

What’s MarianMT?

MarianMT is a machine translation framework based mostly on the Transformer structure, which is well known for its effectiveness in pure language processing duties. Developed utilizing the Marian C++ library, the MarianMT fashions have an enormous benefit of being quick. Hugging Face has included MarianMT into their Transformers library, making it simpler to entry and use by Python.

Step-by-Step Information to Use MarianMT

1. Set up

To start, you must set up the mandatory libraries. Guarantee you’ve Python put in in your system, then run the next command to put in the Hugging Face Transformers library:

You’ll additionally want the torch library for dealing with the mannequin’s computations:

2. Selecting a Mannequin

MarianMT fashions are pre-trained on numerous language pairs. The fashions observe a naming conference of Helsinki-NLP/opus-mt-{src}-{tgt} in hugging face, the place {src} and {tgt} are the supply and goal language codes, respectively. For instance, should you search Helsinki-NLP/opus-mt-en-fr in hugging face, the corresponding mannequin would translate from English to French.

3. Loading the Mannequin and Tokenizer

Let’s say you resolve to translate English to a selected language, i.e., French. You then would want to load the fitting mannequin and its corresponding tokenizer. Right here’s the way you load the mannequin and tokenizer:

from transformers import MarianMTModel, MarianTokenizer

# Specify the mannequin identify
model_name = "Helsinki-NLP/opus-mt-en-fr"

# Load the tokenizer and mannequin
tokenizer = MarianTokenizer.from_pretrained(model_name)
mannequin = MarianMTModel.from_pretrained(model_name)

4. Translating Textual content

Now that you’ve got your mannequin and tokenizer prepared, you may translate textual content in simply 4 easy steps! Right here’s a fundamental instance.To start with, you’ll specify the supply textual content in a variable that you just wish to translate.

# Outline the supply textual content
src_text = ["this is a sentence in English that we want to translate to French"]

Since transformers (or any machine studying mannequin) doesn’t perceive textual content, we wish to convert the supply textual content into numeric kind. For that, we might tokenize our textual content. For an intensive understanding of how one can do tokenization, you may seek advice from my Tokenization article.

# Tokenize the supply textual content
inputs = tokenizer(src_text, return_tensors="pt", padding=True)

Then we’ll go the tokenized sentence to the mannequin and it’ll output some numbers.

# Generate the interpretation
translated = mannequin.generate(**inputs)

Discover that mannequin outputs tokens, and never textual content immediately. We must decode these tokens again to textual content so people can perceive the translated output of the mannequin.

# Decode the translated textual content
tgt_text = [tokenizer.decode(t, skip_special_tokens=True) for t in translated]
print(tgt_text)

Within the above code, the output would be the translated textual content in French:

c'est une phrase en anglais que nous voulons traduire en français

5. Translating to A number of Languages

If you wish to translate English textual content into a number of languages, you should use multilingual fashions. For instance, the mannequin Helsinki-NLP/opus-mt-en-ROMANCE can translate english to a number of Romance languages (French, Portuguese, Spanish, and many others.). Specify the goal language by prepending the supply textual content with the goal language code:

src_text = [
    ">>fr<< this is a sentence in English that we want to translate to French",
    ">>pt<< This should go to Portuguese",
    ">>es<< And this to Spanish",
]

# Specify the multilingual mannequin
model_name = "Helsinki-NLP/opus-mt-en-ROMANCE"
tokenizer = MarianTokenizer.from_pretrained(model_name)
mannequin = MarianMTModel.from_pretrained(model_name)

# Tokenize the supply textual content
inputs = tokenizer(src_text, return_tensors="pt", padding=True)

# Generate the translations
translated = mannequin.generate(**inputs)

# Decode the translated textual content
tgt_text = [tokenizer.decode(t, skip_special_tokens=True) for t in translated]
print(tgt_text)

Output would appear to be this:

["c'est une phrase en anglais que nous voulons traduire en français",
 'Isto deve ir para o português.',
 'Y esto al español']

With this setup, you may simply translate your English textual content into French, Portuguese, and Spanish. There are some teams of languages apart from ROMANCE languages as effectively. Here’s a listing of them:

GROUP_MEMBERS = {
 'ZH': ['cmn', 'cn', 'yue', 'ze_zh', 'zh_cn', 'zh_CN', 'zh_HK', 'zh_tw', 'zh_TW', 'zh_yue', 'zhs', 'zht', 'zh'],
 'ROMANCE': ['fr', 'fr_BE', 'fr_CA', 'fr_FR', 'wa', 'frp', 'oc', 'ca', 'rm', 'lld', 'fur', 'lij', 'lmo', 'es', 'es_AR', 'es_CL', 'es_CO', 'es_CR', 'es_DO', 'es_EC', 'es_ES', 'es_GT', 'es_HN', 'es_MX', 'es_NI', 'es_PA', 'es_PE', 'es_PR', 'es_SV', 'es_UY', 'es_VE', 'pt', 'pt_br', 'pt_BR', 'pt_PT', 'gl', 'lad', 'an', 'mwl', 'it', 'it_IT', 'co', 'nap', 'scn', 'vec', 'sc', 'ro', 'la'],
 'NORTH_EU': ['de', 'nl', 'fy', 'af', 'da', 'fo', 'is', 'no', 'nb', 'nn', 'sv'],
 'SCANDINAVIA': ['da', 'fo', 'is', 'no', 'nb', 'nn', 'sv'],
 'SAMI': ['se', 'sma', 'smj', 'smn', 'sms'],
 'NORWAY': ['nb_NO', 'nb', 'nn_NO', 'nn', 'nog', 'no_nb', 'no'],
 'CELTIC': ['ga', 'cy', 'br', 'gd', 'kw', 'gv']
}

Wrapping Up

Utilizing MarianMT fashions with the Hugging Face Transformers library supplies a strong and versatile solution to carry out language translations. Whether or not you’re translating textual content for private use, analysis, or integrating translation capabilities into your functions, MarianMT affords a dependable and easy-to-use resolution. With the steps outlined on this information, you will get began with translating languages effectively and successfully.

Kanwal Mehreen Kanwal is a machine studying engineer and a technical author with a profound ardour for information science and the intersection of AI with medication. She co-authored the e-book “Maximizing Productiveness with ChatGPT”. As a Google Era Scholar 2022 for APAC, she champions variety and tutorial excellence. She’s additionally acknowledged as a Teradata Variety in Tech Scholar, Mitacs Globalink Analysis Scholar, and Harvard WeCode Scholar. Kanwal is an ardent advocate for change, having based FEMCodes to empower girls in STEM fields.

Buy now