How we measure Anonimatum’s AI in 10 languages: corpus, recall and precision

An anonymization tool is judged by what it misses. That is why every time Anonimatum’s AI changes, we test it against a corpus of 6,123 documents in 10 languages (Spanish, Catalan, Galician, Basque, English, French, German, Italian, Portuguese and Romanian) and compare what it detects with what it should detect. This article explains what is in that corpus, how we measure, and today’s figures, including the ones we are not happy with yet.

What is in the test corpus

The corpus mixes two kinds of documents. The real ones (4,805) are, for the most part, publications by public bodies released under re-use licences: official gazettes and journals, court judgments and orders, decisions by data protection and freedom-of-information authorities, minutes, agreements and EU legislation. The synthetic ones (1,318) are generated with made-up data to cover what rarely appears in real documents: payslips, invoices, contracts, medical reports, forms, emails, spreadsheets and Word or OpenDocument files with complex layouts. They include invoices in every shape: text PDFs, scans, phone photos, Word and Excel files, email attachments and XML e-invoices.

1,272 documents are scans: before looking for personal data they go through OCR with the model for the document’s language. They include skewed or rotated pages, stamps, stains, faxes, phone photos and handwriting, because that is where anonymization fails in practice (see OCR risks when redacting scanned documents).

How we measure: recall and precision

• Recall: of all the personal data in a document, how much the AI detects. A recall of 0.95 means 5 out of 100 slip through. It is the metric that matters most for anonymization: what is not detected gets published.
• Precision: of everything the AI marks, how much really is personal data. Low precision does not leak data, but it means removing extra marks during review.
• Structured identifiers (national ID numbers, tax numbers, IBANs…): all of them must be detected in documents with a text layer; in scans we tolerate OCR misreading a digit.

Hand-reviewed versus automatic annotation

To measure recall you need to know which personal data each document contains. For synthetic documents we know, because we made them up. For real ones there are two levels:

That is why we give two recall figures for real documents: hand-reviewed, which best reflects a complete document, and all real documents (hand-reviewed plus automatic annotation), which is harsher because it concentrates names exactly where mistakes are easiest. Precision is measured on synthetic and hand-reviewed documents, which have everything marked.

Results by language

Figures from AI test run #18, October 21, 2026. Precision is between 0.966 and 0.993 in every language.

On hand-reviewed real documents, 9 of the 10 languages reach 0.95 or more; the lowest value is 0.921 (Basque). Across all real documents, the lowest values are 0.855 (Romanian), 0.870 (Portuguese) and 0.899 (Italian). Among synthetic documents, which gather the hardest cases in the corpus (handwritten scans, faxes, stains, rotated pages), the lowest value is 0.963 (Romanian).

What changed in October 2026

Before this month’s improvements the figures were considerably worse. For the Iberian languages we compare recall on hand-reviewed real documents; for the rest, recall on all real documents, which is where the problem was.

• Basque (0.597 → 0.921): addresses with the street type after the street name («kalea», «hiribidea», «solairua»), signatures in capitals, honorifics such as «andrea» and «jauna», and legal forms such as «S. Koop.» or «Fundazioa».
• French, German, Italian, Portuguese, English and Romanian (Romanian from 0.412 to 0.855): honorifics before names, court compositions, lists of names, the Italian codice fiscale, addresses and each country’s legal forms.
• No regressions: every improvement is tested against the full corpus, so a gain in one language cannot cost another.

What is left to improve

What slips through today falls into two cases, and both show up mostly in the all-real figure:

• Names misread by OCR. In a scan, a name can come out glued to the next word, hyphenated at the end of a line or with swapped letters. If OCR does not read it as a name, the AI does not recognise it.
• Stand-alone names without context. An uncommon surname in a list, with no honorific, role or verb around it, is hard to tell apart from any other word. This is what separates the two figures most in the languages with the lowest recall across all real documents.
• In Basque, split surnames carrying their case ending and foreign names on top of that. Its provisional minimum of 0.92 can only go up, until it reaches the 0.95 of the other languages.

In the meantime, the practical answer is the usual one: automatic detection proposes and a person reviews before applying, especially with scans. Zone-based visual anonymization covers whatever OCR cannot read.

It runs on every AI change

These tests are not a snapshot for the blog. The full corpus runs through continuous integration on every change to the AI, before it reaches production: text is extracted (with OCR for scans), personal data is detected and the result is compared with what is expected. Each language has a minimum recall and precision that can only go up; a change that falls below it is not merged. The figures in this article come from those reports and we will update them when they change.