Buy low-resource language data for AI training

What is low-resource language data?

Low-resource language data is text in a language with too little digital material to train a strong model, often with English translations attached. The label is relative. Meta counts 150 low-resource languages among the 200 its No Language Left Behind model translates.

This page covers buying that data. Our multilingual text data page covers widely spoken languages and translation memories in general.

Why is low-resource language data hard to find?

Low-resource language data is hard to find because the open web covers few languages well. Google wrote in November 2022, launching its 1,000 Languages Initiative, that only a few of more than 7,000 spoken languages are well represented online. Meta states that reliable translation data is not available for many languages.

English alone makes up about 42.6% of pages in Common Crawl's CC-MAIN-2026-43 crawl. A corpus scraped from the web for a small language can be thin, repetitive or machine translated.

Who sells low-resource language data?

Low-resource language data is sold by research distributors and training-data companies, and some is commissioned from native speakers.

Coverage claims are self-reported, so ask each seller for word counts per language and for who wrote the text. See the Linguistic Data Consortium, Appen and LXT profiles.

How much does low-resource language data cost?

Low-resource language data has no public per-word rate from the sellers we checked. LDC shows fees for its LORELEI packs, including the Wolof pack of more than 225,000 words translated into English, only to logged-in users, and Appen's catalog adds datasets to a quote list rather than showing prices.

For commissioned text, ask for a quote that separates translation, native review and annotation, so you can compare sellers line by line. Our guide to AI training data pricing models explains per-unit and flat-fee deals.

Translated text carries two sets of rights: those in the source text and those in the translation. 17 USC 101 lists a translation as a derivative work, so the source owner's rights survive, and it names translations among the nine kinds of commissioned work that can be a work made for hire when both sides sign a written agreement.

Without that agreement or a written assignment, a freelance translator keeps the copyright. Ask for the contract behind each translator's work and for the license covering the source texts. Personal information in correspondence, chat or social posts still falls under the CCPA and other state privacy laws, whatever the language.

How should you handle Indigenous and community language data?

Indigenous and community language data needs the consent of the community, not just of the person who wrote it down. The CARE Principles for Indigenous Data Governance, published by the Global Indigenous Data Alliance, set collective benefit, authority to control, responsibility and ethics as the tests for using Indigenous data.

For Native American and Alaska Native languages, ask whether the tribal government or language program concerned approved the license. Tribes are sovereign nations, and some set their own rules for research and data.

What should you check in a low-resource sample?

A low-resource sample should be checked by native speakers as well as by automatic tools, in every language the seller lists.

• Native speaker review of a random sample in every language
• Language identification run across the full set to catch mislabeled text
• Share of machine-translated text disclosed and labeled
• Script and orthography documented where a language uses more than one
• Domain spread across news, everyday writing and specialist text shown in the manifest

Our guide on how to evaluate a dataset sample covers the tests. Meta's FLORES-200, a human-translated benchmark covering 200 languages, is one way to measure gains after training.

If you hold UK or EU data

If text comes from the UK or EU, UK GDPR or EU GDPR applies to any personal data in it. Providers of general-purpose AI models in the EU must also publish a summary of training content under the EU AI Act, which our EU AI Act guide explains.

How do you source vetted low-resource data?

You source vetted low-resource data by naming the languages, dialects, scripts and word counts you need. Every seller we shortlist passes our eight-point vetting standard. Book a free consultation or submit a data requirement.