Buy multilingual text data for AI training

What is multilingual text data?

Multilingual text data is human-written text in languages other than English, or text aligned across two or more languages so each passage can be matched to its translation. Sources include publishers, translation agencies, broadcasters, public bodies and companies that operate in several languages.

Why do AI teams buy multilingual text?

AI teams buy multilingual text because public web data is dominated by a few languages. In Common Crawl's CC-MAIN-2026-43 crawl, English accounts for about 42.6% of pages, followed by Russian at 6.3%, German at 5.8%, Japanese at 4.9% and French at 4.8%.

Many languages have far less. Meta's No Language Left Behind project, which translates between 200 languages including 150 low-resource ones, states that reliable translation data is not available for many languages.

What makes multilingual data useful?

Multilingual data is most useful when it is written by native speakers, in a language or dialect that public data covers poorly, across domains such as law, health and customer service. Ask how much was machine translated, since a model trained on machine translation learns that system's output rather than native usage.

Parallel text, where each sentence is matched to its translation, is the raw material for translation models. Translation memories held by agencies can supply it, but each one sits under the rights of the client who owned the source and the translator who wrote the target.

US copyright law spells this out. 17 USC 101 lists a translation as a derivative work, and under 17 USC 103(b) copyright in a derivative work extends only to the material its author contributed, leaving the source owner's rights unchanged.

Multilingual text is priced per million words or tokens for catalog sales, or as a flat fee for a company archive. No buyer publishes a per-language rate card. Getphyllo's reported market ranges put commercial licenses at $25,000 to $500,000 or more and research licenses at $5,000 to $15,000, and research licenses rarely permit production training.

Our guide to AI training data pricing models explains per-unit and flat-fee structures, and the glossary defines a data license.

Who sells multilingual text?

Multilingual text is sold by publishers, news agencies, translation and localization companies, and businesses that serve customers in several languages, such as call centers and ecommerce firms. Support and sales records in a local language show everyday usage that published text rarely contains.

How should multilingual text be delivered?

Multilingual text should be delivered as UTF-8 JSONL or plain text, with an ISO 639 language code on every record and dialect or region labels where they matter. Parallel corpora should keep alignment explicit, in TMX or tab-separated files, with document IDs so each segment can be traced to its source.

Ask for word or token counts per language in the data manifest, and for the share of each language that came from each source. A language drawn mostly from one source reflects that source's style and subject matter.

What should you check before buying multilingual data?

Before buying multilingual data, have native speakers review a random sample for each language and run a language identification tool across the set to confirm the labels.

Check for boilerplate too. Menus, legal notices and repeated templates add little to training in any language, and they can inflate word counts in a corpus built from websites.

Confirm who owns translations. Translation memories can contain client documents, and an agency may hold only a license to use them for that client's work. Personal information in correspondence must be removed or de-identified to the standard of the US state privacy laws that apply, such as the CCPA. If you hold UK or EU correspondence, UK GDPR or EU GDPR also applies unless the data is anonymized.

Publishers can read about selling publisher and education data, and buyers can learn how to evaluate a dataset sample. To source vetted multilingual corpora, book a free consultation.