Ukrainian generation produces Russianisms / calques even with explicit 'avoid Russianisms' instruction — offering verified corpus + eval set

Summary

When generating Ukrainian content, Claude (verified just now on Claude 4.7 Opus) produces Russianisms, calques, and Surzhyk forms even when the prompt explicitly says "avoid Russianisms." The pattern is consistent and survives heavy prompt engineering. I've built infrastructure to detect and correct this for a public open-source Ukrainian-curriculum project, and I'd like to offer the corpus + methodology + eval set to Anthropic.

This is not strictly a Claude Code (CLI) bug — it's a model-level training-data imbalance. Filing it here because I don't see a better channel for "we have data and an eval set that would help."

Live reproducer (Claude 4.7 Opus, 2026-05-14)

Prompt:

You are writing for a Ukrainian-language email at A2 level. Translate the following English text into idiomatic standard Ukrainian. Avoid Russianisms. "Hello! I am writing to let you know that I will be taking part in the meeting next week. I want to receive the documents in advance so I can prepare. Generally, I think the following points are the most important: budget, timeline, and the next steps. Please send everything by any moment that works for you. Thanks!" Output ONLY the Ukrainian translation — no commentary.

Claude 4.7 Opus output:

Привіт! Пишу, щоб повідомити, що я буду приймати участь у нараді на наступному тижні. Хочу отримати документи заздалегідь, щоб мати змогу підготуватися. Взагалі, я вважаю, що слідуючі пункти найбільш важливі: бюджет, графік і подальші кроки. Будь ласка, надішліть усе в любий момент, який вам зручний. Дякую!

Russianisms in that output (3 in ~70 words, despite explicit "avoid Russianisms"):

| Bad output | Standard Ukrainian | Source of error |
|---|---|---|
| приймати участь | брати участь | calque of RU принимать участие |
| слідуючі (пункти) | наступні (пункти) | RU следующие |
| (у) любий момент | (у) будь-який момент | RU любой (UK любий = "dear / beloved", not "any") |

Two of the three were caught by our deterministic 50-pattern Russianism detector (scripts/audit/checks/russicism_detection.py in the project linked below). The third (слідуючі) slipped past because our pattern matches the masculine singular слідуючий only — proving a parallel point: the rate of Russianisms is higher than even our gates catch.

I encourage Anthropic to re-run this exact prompt against the model and confirm independently.

Other concrete bug-form classes

The detector and our build pipeline have catalogued these as recurring failures across model versions:

| Bad output (Russianism / calque) | Standard Ukrainian | Source of error |
|---|---|---|
| приймати участь | брати участь | calque of RU принимать участие |
| самий кращий | найкращий | RU superlative самый лучший |
| получати | отримувати | direct RU получать |
| відноситися (in "concerns" sense) | стосуватися / ставитися | RU относиться |
| слідуючий / слідуючі | наступний / наступні | RU следующий |
| кушати | їсти | RU кушать |
| на то, що | на те, що | RU на то, что |
| любий момент (= "any moment") | будь-який момент | RU любой |
| вообще | взагалі | direct RU borrowing |

Semantic false friends (valid Ukrainian words emitted with their Russian meaning):

| Word | Claude often emits as… | Actual UK meaning | Correct UK |
|---|---|---|---|
| лук | "onion" (RU sense) | bow (weapon) | цибуля |
| неділя | "week" (RU sense) | Sunday | тиждень |
| город | "city" (RU sense) | garden / vegetable patch | місто |
| луна | "moon" (RU sense) | echo | місяць |
| річ | "speech" (RU sense) | thing / item | промова |

Subtler issues that don't match the static detector but appear constantly: купляти (non-standard for купувати), у скільки? (Surzhyk for о котрій годині?), ходити за продуктами (calque of RU ходить за продуктами; native UK is ходити по продукти).

Why this happens (best guess)

Russian is over-represented in Slavic web corpora by roughly an order of magnitude. When the model needs a "Slavic-shaped" continuation in Ukrainian context, the closest training-data neighbor is often Russian, and morphology or lexicon leaks across. The model can describe the difference correctly when asked (it'll explain Antonenko-Davydovych accurately) but produces the wrong form under generation load. Classic recognition-vs-production gap.

Why this matters

Ukrainian is being actively decolonized linguistically — replacing Russianisms with authentic forms is part of the cultural and educational project. An LLM that produces Russianized Ukrainian as L2 teaching content actively teaches the wrong forms to learners. This isn't polish; it's the difference between "Claude helps Ukrainian" and "Claude introduces durable L2 errors at scale."

What we built around it (evidence the prompt-only fix doesn't fully work)

Project: <https://github.com/learn-ukrainian/learn-ukrainian.github.io>

Approach: assume pre-training contamination, verify everything against authentic Ukrainian sources at generation time. The existence of every layer below is evidence that prompt-level "avoid Russianisms" wasn't enough.

• VESUM — 409K lemmas / 6.7M morphological forms (the authority for "does this form exist in standard Ukrainian")
• ЕСУМ — Ukrainian etymological dictionary, all 6 volumes, 29,171 entries
• Грінченко 1907–1909 — 67,275 pre-Soviet attestations (heritage defense: proves a "suspicious" word is genuinely Ukrainian, not borrowed)
• Антоненко-Давидович «Як ми говоримо» — full book ingested; the canonical Ukrainian authority on Russianism→Ukrainian corrections
• СУМ-11 with 7,152 Sovietized entries flagged (~5.6% of the dictionary carries late-Soviet ideological framing; we route around them)
• Custom Russian-shadow detector using pymorphy3 morphological heuristics
• 50+ deterministic Russianism + semantic-false-friend patterns (scripts/audit/checks/russicism_detection.py) that gate every generated module
• Adjudication hierarchy: VESUM → Правопис 2019 → Горох → Антоненко-Давидович → Грінченко / ЕСУМ
• Adversarial cross-agent review (one model writes, a different model reviews — never self-review)

Even with all of this, the deterministic gate still flags Russianisms in fresh model output, as the live reproducer above shows. The gates work; the underlying generation behaviour is the bug.

What I'm offering Anthropic

1. Curated Ukrainian training-data corpus — textbooks, literary works, dictionaries, Wikipedia, full source provenance. ~600K dictionary rows + 25K textbook chunks + 137K literary chunks + the lexical infrastructure above. Most of it is open-license; the project itself is permanently non-commercial open-source.
2. A Russianism / Surzhyk / calque eval set — concrete bad-form → correct-form pairs, machine-readable. The tables above are a starter; we have a few hundred patterns + semantic false friends already coded as deterministic checks. Drop-in for any language-fidelity eval suite.
3. The adjudication methodology — which authority decides which class of question, in what order. Useful for RLHF labeling guidelines.
4. The Sovietization-flagged dictionary list — which entries in СУМ-11 carry late-Soviet ideological framing, with risk scores and triggering keywords. Useful for filtering training data that looks Ukrainian but carries imperial framing.

If there's a better channel than this issue tracker for an offer like this, please point me to it. Happy to package the eval set as a HuggingFace dataset or send under whatever data-sharing arrangement Anthropic prefers.

Suggested action items

• [ ] Add a Ukrainian-specific eval to whatever language-fidelity suite the training team uses. Russianism rate against the prompts above is the cleanest signal.
• [ ] Acknowledge the Russian-shadow effect in model-card "limitations" for Ukrainian (and likely Belarusian, where the same dynamic applies).
• [ ] If interested in the corpus, tell me what format / license you need.

Environment

• Platform: darwin
• Terminal: iTerm.app
• Claude Code version: 2.1.141
• Model used for live reproducer: Claude 4.7 Opus (Claude Code session)
• Feedback ID: 38b01980-5e62-40e7-9154-3063aefa48fc
• Project repo: <https://github.com/learn-ukrainian/learn-ukrainian.github.io>