Research: Georgian Language Benchmark, 12 October 2026
Georgian Language Benchmark: Google's and DeepSeek's Models Added
Giorgi Kobaidze's new Georgian Language Benchmark compared Anthropic and OpenAI. We ran his questions, prompts and scoring unchanged on three models it did not include. Gemini 3.1 Pro scores 0.89, one hundredth behind the leader, GPT-6 Astra; Gemini 3.8 Flash scores 0.88 for $0.40; DeepSeek V4 Pro scores 0.67.
What this is
Kobaidze’s benchmark tests how well AI models produce Georgian: write numbers out in words, build verb forms, fill gaps in an article, correct planted grammar errors, and translate idiom-heavy Georgian into English. He ran it on Kaggle’s free model credits, which limited him to eight models from Anthropic and OpenAI. His article names the next step himself: “Google, xAI and the open models, once the budget allows”. Because his code and data are open, we could take that step for three of them: Gemini 3.1 Pro, Gemini 3.8 Flash and DeepSeek V4 Pro, with his questions, prompts and scoring unchanged.
What we found
- Gemini 3.1 Pro scores 0.89, one hundredth behind the leader of his table, GPT-6 Astra (0.90), at the same “low” thinking level and for less ($1.23 against $1.73).
- Gemini 3.8 Flash scores 0.88 for $0.40, above every model in his table except GPT-6 Astra. It cannot switch thinking off, so it is not fully comparable with the light pair it sits next to (see below).
- DeepSeek V4 Pro, an open-weights model, scores 0.67 for $0.13, between Claude Sonnet 4.6 (0.62) and GPT-5.4 (0.71).
- Numbers stay the hardest task for everyone. The two new Gemini models reach 0.75 and 0.76 there; GPT-6 Astra 0.83.
- Translating Georgian into English is solved (0.96–1.00 for every model). Writing Georgian is where models differ.
The table
Score per task as on his dashboard (share correct; per gap for cloze; judge score for translation; partial credit for correction). The average is the plain mean of the five tasks. Cost is what each full run cost.
| Level | Model | Average | Numbers | Verbs | Translation | Cloze | Correction | Cost |
|---|---|---|---|---|---|---|---|---|
| Light | Claude Haiku 4.5 | 0.48 | 0.30 | 0.34 | 0.96 | 0.54 | 0.25 | $0.10 |
| Light | GPT-5.4 mini | 0.60 | 0.41 | 0.42 | 1.00 | 0.52 | 0.67 | $0.05 |
| Light | Gemini 3.8 Flash (ours, minimal thinking) | 0.88 | 0.76 | 0.82 | 1.00 | 0.94 | 0.87 | $0.40 |
| Decent | Claude Sonnet 4.6 | 0.62 | 0.39 | 0.52 | 1.00 | 0.76 | 0.44 | $0.29 |
| Decent | GPT-5.4 | 0.71 | 0.46 | 0.54 | 1.00 | 0.88 | 0.66 | $0.15 |
| Top | Claude Opus 4.8 | 0.75 | 0.54 | 0.68 | 1.00 | 0.84 | 0.68 | $0.58 |
| Top | GPT-5.5 | 0.74 | 0.51 | 0.60 | 0.98 | 0.88 | 0.74 | $0.30 |
| Newest, low thinking | Claude Opus 5 | 0.82 | 0.65 | 0.84 | 1.00 | 0.86 | 0.77 | $1.22 |
| Newest, low thinking | GPT-6 Astra | 0.90 | 0.83 | 0.90 | 0.98 | 0.96 | 0.82 | $1.73 |
| Newest, low thinking | Gemini 3.1 Pro (ours) | 0.89 | 0.75 | 0.94 | 0.98 | 0.94 | 0.86 | $1.23 |
| Open weights | DeepSeek V4 Pro (ours) | 0.67 | 0.47 | 0.54 | 1.00 | 0.72 | 0.63 | $0.13 |
Rows marked ours were run by openclaw.ge via OpenRouter on 12 October 2026, with the same items, prompts and scoring. The other eight rows are from his runs, selected by his own rule.
Does our setup match his?
Every time he pushed a new version of the benchmark, Kaggle ran it automatically on Gemini 3.7 Flash, so he has scores for that model too (his article mentions “around 0.89”). We ran it again our way as a check. Average: 0.89 ours, 0.88 his. Per task, ours/his: numbers 0.77/0.77, verbs 0.90/0.86, translation 1.00/1.00, cloze 0.94/0.90, correction 0.84/0.88. He reports re-runs of the same model varying by about ±0.03 per task, and our differences are at most 0.01 beyond that.
How we ran it
- His files, unchanged. His task files ran through a small adapter that sends the calls to OpenRouter instead of Kaggle’s model proxy: his prompts, his JSON answer format, his scoring code. Translations were judged by his judge, Gemini 3 Flash.
- His settings. Output limit 32,000 tokens; temperature left at the provider’s default; one run per model.
- Thinking. His rule is no thinking, except for the newest pair at “low”. DeepSeek V4 Pro ran with thinking off (the provider still produced some thinking on 4 of 239 calls). Gemini 3.1 Pro cannot go below “low”, the same exception he made for GPT-6 Astra. The Gemini Flash models cannot switch thinking off at all and ran at the lowest level, “minimal”, about 370–390 thinking tokens per call; that is a small advantage over the no-thinking models in his light, decent and top levels.
- Cost: $2.15 for the three new models and the check together.
Read with care
- The numbers set was trimmed against a Google model. His archive notes say the numbers questions were cut to the rows that Gemini 3 Flash got at least one answer wrong. His README says it plainly: the sets are hard by design, and the ranking matters more than the absolute scores. On that task the selection works against Google’s models, so the Gemini numbers scores lean low.
- The translation judge is a model, Gemini 3 Flash, the same judge for every model. The translation scores are 0.96–1.00 for everyone, so it hardly moves the ranking.
- Small sets. Some tasks have only 12–17 items, so differences of a few hundredths are not significant.
Data
Our raw outputs, one JSON line per model call (prompt, reply, thinking level sent, tokens, cost) and per scored item, plus the combined table: openclaw-runs.zip (100 KB; README inside) and combined.csv. Released under CC BY 4.0, like his data.
Credit
The benchmark, the questions, the prompts and the scoring are all Giorgi Kobaidze’s. If you use these numbers, cite his benchmark first; our rows are a small addition to his work.