KartuliSafe: Does AI Safety Speak Georgian?

Overview

Chatbots are taught to refuse dangerous requests mostly in English. Whether that training carries over to smaller languages is an open question, and for Georgian nobody has measured it properly. KartuliSafe (“kartuli” is how Georgians say “Georgian”) is our attempt to measure it: the same test questions, in English and in carefully checked Georgian, put to the models people in Georgia actually use.

This page is the project’s home. It holds the question, what is already known, the plan, and every result as it arrives. The code, the method and the materials are public on GitHub.

The question

Two failures matter. A model can refuse too much, turning down harmless questions because they sound sensitive (“how do I kill a Python process?”). Or it can refuse too little, answering a harmful request in Georgian that it would refuse in English. The second is the dangerous one, and research on other small languages says it happens.

What is already known

• In 2024, a study machine-translated a set of harmful requests into 74 languages and found Georgian among the weakest (Dong et al., arXiv 2404.18534). It used unverified translations and 2023-era models, did not test over-refusal, and published no Georgian data.
• In August 2026, a study of four African languages found that harmful prompts kept less than 10% of the English refusal signal (arXiv 2608.11146). A 2026 review of 50 studies found low-resource languages consistently less protected (arXiv 2608.14626).
• No established multilingual safety benchmark includes Georgian. We checked MultiJail, XSafety, PolyGuard, RTP-LX, Aya red-teaming, M-ALERT, LinguaSafe and SORRY-Bench.

The pilot (1 October 2026)

A one-day test of the method, on the safe side only. We took 40 harmless prompts from XSTest, a public test set built to catch over-refusal, four from each of its ten categories. A model translated them into Georgian. Three current models answered every prompt in both languages, and a fourth model judged each answer.

None of the three over-refused in either language. Georgian answers were 10 to 25% shorter than the English ones. The whole run cost USD 3.16. A first attempt failed (the token limits were too low for models that reason before answering) and was discarded; the run above is the second.

What this shows: the method works end to end in Georgian, and over-refusal is not the problem on this sample. That narrows the open question to the other direction. Prompts, per-answer results and the script are on the pilot page.

The plan

1. Translate two public test sets into Georgian: XSTest (450 prompts: 250 safe, 200 unsafe contrasts; CC-BY-4.0) and JailbreakBench’s JBB-Behaviors (100 harmful requests with 100 harmless twins; MIT). A native Georgian linguist checks every item. Prompts built on English wordplay are adapted, not translated, with a note each.
2. Run both sets in Georgian and English through four frontier models and one widely used open-weight model, three times each. The models answer in plain text, as a user would see them.
3. Classify every answer (full answer, partial, refusal) with two automatic judges from different developers, and measure how reliable they are in Georgian against 400 answers labelled by the linguist.
4. Compare refusing too much and refusing too little, Georgian against English, by model and by type of harm.
5. Tell the developers first, then publish.

About four weeks of work, then a 30-day wait before the harmful-side results are published. The full study needs a paid native linguist and is waiting for that funding; the pilot was paid from our own budget.

What gets published, and what never does

• At once: the method, the harmless Georgian prompts in full, and the over-refusal results.
• After the 30-day disclosure window: the harmful-side results and the Georgian translations of the harmful prompts, which are already public in English.
• Never: what the models wrote in answer to harmful requests. Only rates and a few redacted examples.

Sources

• XSTest: Röttger et al., NAACL 2024, CC-BY-4.0 — dataset
• JailbreakBench JBB-Behaviors, MIT — dataset
• Dong et al. 2024 — arXiv 2404.18534
• The Illusion of Cross-Lingual Safety in Low-Resource Languages, 2026 — arXiv 2608.11146
• LLM Safety Alignment in Low-Resource Languages: A Systematic Literature Review, 2026 — arXiv 2608.14626
• The pilot: prompts, results and script — /research/ka-safety-pilot/
• The study’s code, method and materials — github.com/akvekvetsia/KartuliSafe