Can Five AI Models Rebuild a Georgian Survey Note from Public Data? A Reproducible Test

Question

If we publish a prompt that tells an AI model how to gather a survey time series from public pages and write a short research note, does the result depend on which model runs it? Where exactly does it fail — in the model, in the tools the model is given, or in the prompt?

Method

The task. Reproduce a polling institute’s topical note: support for Georgia’s EU membership across every Caucasus Barometer wave (2011–2024, nine waves), the same for NATO membership, a 2024 breakdown by age, and the survey methodology — all from caucasusbarometer.org.

The prompt sequence. Six numbered prompts, run in order in one chat (full text):

• Prompt 0 — capability check with proof: open an HTML page, a JSON API, a PDF attachment and an XLS file, and paste exact text from each or write CANNOT with the tool’s error.
• Prompts 1–4 — gather steps: list the waves, print the EU table for each wave with its URL, print the NATO tables and the age cross-tab, print the methodology fields. Rules: numbers only from pages opened in this session; anything not opened is written NOT GATHERED; no archives, proxies or mirrors.
• Prompt 5 — write step: the note, using only what was printed above, ending with a self-check line.

Two conditions.

1. Consumer apps — the same prompts typed into ten AI products with their own browsing tools (run by hand, outputs pasted back).
2. API harness — five models through OpenRouter with a single tool, fetch_url, implemented by us: it fetches only caucasusbarometer.org and geostat.ge, waits 3 seconds between requests, returns HTML as visible text plus the page’s links, JSON as-is, PDF as extracted text, XLS as sheet dumps. Temperature 0. Budget per run. The model cannot reach any other host even if it asks.

Scoring. Every cell the model printed for a wave is compared with the live page, re-read the same day: the expected values must appear in order among the integers the model wrote for that wave. Numbers given for a wave whose page was never fetched count as fabrication. Also recorded: probes passed, gaps stated, self-check line, tool calls, cost in dollars, and where the run stopped.

Materials (all public)

• Prompt sequence: 04-survey-synthesis.sequence.md
• Our own gather and note, the reference for scoring: inputs, note
• Harness and scorer: modeltest.py, score_run.py, fetch.js, links.js
• Transcripts, one per model, with every prompt, tool call and answer (page bodies omitted): Grok 4.6, Kimi K3, Qwen3.8 Max, GPT-6 Astra, DeepSeek V4 Flash
• Test log with the consumer-app outputs and every prompt change: 2026-09-21-gemini-survey-note.md

Results

Condition 1 — consumer apps (21 September 2026)

Ten products, one identical PDF error, no fabricated numbers. The limiting factor in every case was the product’s fetch tool — no PDFs, truncated pages, sandboxes without network — not the model behind it.

Condition 2 — API harness, five models, one day

* Grok ran on the first harness version, which did not yet return page links, so it could not find the PDF’s URL; that failure is ours. “+8 extra”: every model opened the NATO question for all eleven waves from 2009, eight more than the brief asked for, and got all eight right.

145 tool calls, 0 fabricated numbers, 1 failure — economic.

Findings

1. The model is not the variable. Five models of very different size and price produced identical tables. Everything a model was able to open, it copied correctly.
2. The tool is the variable. Ten consumer products failed the same PDF in the same way; in the harness, where the tool could read PDFs, every model read it. A capability check with proof, before any gathering, is therefore the first prompt in every sequence we publish.
3. Prompt rules turn missing access into stated gaps, not invented numbers. The no-browsing model refused; the PDF-less models wrote NOT GATHERED; nobody filled the blank.
4. The models audited us. Our reference gather had stopped two waves early on the EU question and eight waves early on NATO, and had the wording history wrong twice. Each correction was verified on the source page and kept.
5. The same complete result cost about fifteen times more on one model than another: $0.08 to $1.23 for a finished note; $5.09 went on a run that never wrote one.

Defects found in our own work, all fixed

• Reference data: EU waves 2011–2012 missing; NATO waves 2009–2017 missing; question wording claimed unchanged (it changed four times).
• Prompt 1 did not say to include the starting wave in the list; two models literally excluded it.
• No prompt stated the date, so the write step honestly reported “date accessed: NOT GATHERED”.
• Nothing forbade copying personal contact details; two models copied a researcher’s e-mail address off the factsheet.
• Harness: first version hid page links (Grok’s PDF failure); budget stops between prompts were not logged; published JSON transcripts carried full page dumps.

Limitations

One prompt type, one source site, one day, one run per model at temperature 0. Scoring checks cells and gaps, not prose quality. The harness is ours, so “tool” here means our tool; the consumer-app results describe those products’ tools on that day and may change with any update. The five models were chosen by availability and price, not by a survey of the market. None of this measures how a model would do with a tool it can misuse — the allowlist removed that possibility by construction.

Cost of this study

API runs: $8.55 for the five scored runs (Astra $5.09 of it), plus $3.06 on four earlier runs cut short by credit limits before the key was topped up. Consumer-app runs: included in subscriptions, not measured. Wall-clock for the API condition: about an hour, runs sequential (8–21 minutes each). Everything else — design, scoring, this page — one AI operator, one afternoon.

To reproduce

1. Fill in the date at the top of Prompt 0 and run Prompts 0–5 in order in any AI product with browsing; compare each printed table with the URL beside it.
2. Or run the harness: python3 modeltest.py --model <id> --prompts 04-survey-synthesis.sequence.md --out runs --budget-usd 2 with an OpenRouter key, then python3 score_run.py --table runs/*/transcript.json. The expected cells in the scorer were read from the live pages on 21 September 2026; if CRRC revises a page, the scorer is what to update.

Sources

• The Caucasus Research Resource Centers. (2009–2024) “Caucasus Barometer”, Georgia — variables EUSUPP, NATOSUPP, AGEGROUP. Retrieved from caucasusbarometer.org on 21 September 2026.
• CRRC-Georgia, “Caucasus Barometer 2024 Georgia — Methodological Report”, June 2024, via the downloads page.
• Geostat PX-Web API and construction-permits XLS, used only as probes: pc-axis.geostat.ge.
• OpenRouter usage and pricing as reported in each response (usage.cost), 21 September 2026.