# Cross-model test — prompt 04 (survey note), Google's model, 2026-09-21 The owner ran the data-free Option A prompt for the EU/NATO survey note on Google's model (with browsing) and pasted the result back. Compared here against our own run (`../prompts/examples/2026-09-21-survey-synthesis.md`) and re-verified against the live pages the same day. ## Numbers | Item | Ours | Google | Live page (re-checked) | Verdict | |---|---|---|---|---| | EUSUPP 2024: fully / DK-RA / support sum | 48 / 5 / 70 | 47 / 6 / 69 | 48 / 5 | ours right; Google off by one, error carried into its overview | | EUSUPP 2013–2021 (six waves) | as in our pack | identical | — | match | | EUSUPP 2012 | not gathered | 1/2/13/34/38/12 → 72 | 1/2/13/34/38/12 | Google right; we stopped a wave early | | Question wording across waves | "unchanged" | changed: 2012 and 2013 shorter phrasings, 1–5 scale text from 2017 | confirmed | Google right; our pack corrected | | NATOSUPP series | 3 waves gathered | not gathered ("available in the source documents") | — | ours complete | | Age cross-tab 2024 | full table | table not printed; three cells quoted correctly | — | ours complete | | Methodology | dates, n, frame, mode, languages, RR1, weighting, no-MoE statement | dates, n, mode, "multi-stage cluster" only | — | ours complete | ## What this says about the prompt - Reproducible where both models actually read the table: six of seven shared waves identical to the integer. - The failure modes are exactly the ones the prompt warns about: reading chart labels instead of the HTML table (the 47/6 slip), and skipping gather steps while writing as if they were done (NATO, cross-tab). - Our own gather had a completeness gap (2011–2012 waves) and a wrong claim (wording unchanged). Both fixed in the input pack and example. - For publishing: show the gather prompt as the artifact, the input pack as *our record*, and this kind of side-by-side as the honest "same or not the same" note. A one-point disagreement traced to its source is more convincing than a clean match. ## Prompt changes made as a result - Part 1 of `04-survey-synthesis.md`: check the variable page's "in other datasets" list and gather *every* wave listed, oldest first; record the question text per wave and diff it, rather than assuming it is stable. - Part 2: the note must print the gathered tables before the prose, so a reader can see what was actually read (Google's answer hid the NATO and cross-tab tables behind "available in the source documents"). ## Same day — DeepSeek, no browsing: correct refusal DeepSeek was given the data-free Option A prompt in an environment without web access. Its answer, in full: it cannot complete the task because it has no live access to caucasusbarometer.org and the task forbids figures from memory or other sources. That is the intended behaviour — the rules turned "no browsing" into a stated gap instead of a fabricated note. Scored as a pass for the prompt, not a failure of the model. Next test for a no-browsing model is Option B: paste the input pack and run only Part 2, which checks the writing discipline (DK/RA kept separate, support sums as printed, no numbers outside the pack). ## Same day — Prompt 0 (capability check with proof), seven models All received the same text; the "proof" is copying the first row of the 2024 EUSUPP table and the page title, and opening the methodological note PDF. Live page at that moment: "Don't support at all 5". | Model | Live page proof | PDF | Verdict given | Notes | |---|---|---|---|---| | DeepSeek (no tools) | — | — | refused whole task | correct: no browsing, so no numbers; use the write-only test | | Perplexity | exact first row | NO PDF ACCESS | PARTIAL | its retrieval renders less of a page than a browser — could not see the downloads page's document titles that a browser shows | | Kimi K3 | exact first row + full table | NO PDF ACCESS after fetch, curl (shell has no network, proved via example.com) and a control attachment | PARTIAL | fullest evidence; found the real attachment hash | | Grok 4.6 | exact first row + title | NO PDF ACCESS | PARTIAL | after the failure it searched other sites for the same PDF — did not use them, but the roaming instinct is the thing to watch in later prompts | | Qwen3.8 Max | exact first row + full table | NO PDF ACCESS after four fetch variants, curl, and a control | PARTIAL | its fallback search surfaced the correct interview count from the factsheet page and it refused to present it because the prompt named the PDF — exactly the behaviour the rules ask for | | Mercury 2.5 | exact first row + title | NO PDF ACCESS after four different attachment hashes | PARTIAL | first page fetch was interrupted and retried cleanly; least verbose of the six, nothing claimed beyond what it fetched | | GPT Astra (latest) | exact first row + title | NO PDF ACCESS after the fetch tool and a Python requests/pypdf attempt in its sandbox (timed out) | PARTIAL | made no search at all; declined to "prove future recall in advance" — the most precise reading of item 4 | None of the seven fabricated the PDF evidence. The proof step works: a model answering from memory would have had to produce a table row and a title, and every model that could not open the page said so instead. Next: Prompts 1–5 (PDF-free Prompt 4 variant) on the four PARTIAL models, scored wave by wave against the live pages. ## Diagnosis of the PDF failure, and the fix Seven models, one platform, one identical error string on the PDF ("no content for this URL"), sandboxes without outbound network. The attachment downloads fine with plain curl from elsewhere (HTTP 200, 368 KB). So: a harness limitation (the platform's fetch tool extracts HTML text and drops binary attachments), not a model or site problem. The prompt detected it correctly in all seven runs. Fixes made: (1) the methodology facts are on the HTML factsheet page (https://caucasusbarometer.org/en/cb2024ge/factsheet/ — 1,509 interviews, 23% response rate, CAPI, languages, population, sample design, "results are weighted"), so Prompt 4 now reads that page by default and treats the PDF as a bonus; (2) Prompt 0 now probes four source types — HTML, JSON API, PDF attachment, binary XLS — and maps the result to which of prompts 01–04 the harness can run, so the wrong-harness trap is caught before any gathering starts. ## Same day — Pareto on the four-probe Prompt 0: A OK, B OK, C/D CANNOT First run of the source-type version. HTML row and title exact; JSON API first two entries exact ("Agriculture", "Business Statistics"); PDF and XLS both CANNOT with the same Exa error, plus a verbose curl showing TCP connect to the origin timing out from the sandbox — the harness diagnosis confirmed from inside the harness. Verdict for that platform: A+B → prompt 04 in full, Geostat/NBG parts of 01; not 02, not the PDF parts of 01/03. It also tried a Wayback copy and two extraction proxies when the attachment failed (reported, got nothing) — the rules now say explicitly: site itself only, no archives, proxies or mirrors. It apologised for running probes in parallel rather than in order; for independent probes that is fine and the text now allows it implicitly. ## Same day — Sonar Pro Search: Prompt 0 PARTIAL, Prompt 1 blocked by shallow extraction First row exact, PDF honestly refused. On Prompt 1 it copied the question text and labels exactly, then wrote NOT GATHERED for the "in other datasets" list — which is on the page, below the table; its retrieval tool (same family as Perplexity's, same symptom as the missing downloads titles) truncates the extraction. Correct behaviour, but it stops Prompt 2 on that platform. Prompt 1 now carries a fallback that stays inside the rules: probe cb2008ge…cb2024ge EUSUPP URLs directly and keep those that return a frequency table, stating that the list came from probing. Third harness limit found by testing: binary files, sandbox network, shallow page extraction. ## Same day — Grok 4.6, Prompt 1: pass; prompt defect found Copied the full "in other datasets" list exactly and extracted the eight Georgia Barometer ids correctly, but omitted cb2024ge — the starting dataset, which by definition is not in its own "other datasets" list. Literal reading of a prompt that failed to say so; Prompt 1 now says to add the starting dataset. Owner told Grok to add it before Prompt 2. ## Same day — Grok 4.6, Prompt 2: full pass on nine waves; corrected us twice All nine EUSUPP tables identical to the live pages cell for cell, including split DK/RA rows and the site's "pardially" typo in 2020. It also reported that 2015 uses the short 2013 wording and that 2020 has a distinct read-out wording — both verified live, both wrong in our pack until now (we had claimed the 1–5 text "from 2015"). Method note: for the 2019 wording it used the wave's codebook page and two site-scoped searches after its extraction truncated the block — same site, same content, acceptable, but the searches are the roaming tendency again. Its summary table and diff list were not in the pasted output. ## Same day — Kimi K3, Prompt 1: pass; noticed the cb2024ge gap and still excluded it Full "in other datasets" list exact, eight ids correct, and an extraction note explaining that cb2024ge is missing because it is the current page's own dataset — then it followed the literal instruction and left it out. Confirms the Prompt 1 defect from the Grok run was wording, not model judgement; fixed in the sequence file. ## Same day — API harness, Grok 4.6: full sequence, every cell correct First complete run through `scripts/modeltest.py` (our `fetch_url` tool, allowlist, 3 s crawl delay), scored by `scripts/score_run.py` against the live pages. 32 tool calls, $1.23, no errors, no retries. - Prompt 0: A OK (first EUSUPP row and title exact), B OK (Geostat API first two entries), D OK (permits XLS first sheet read), **C CANNOT** — it reported "downloads page visible text has no href/URL" and then guessed five PDF paths, all of which returned HTML, which it correctly identified and did not use. This run started on the first harness version, before `fetch_url` appended the page's links (`e7d2fa7`); a re-check with the current harness shows the attachment link present in the fetch result. So the C failure here is ours, not the model's — the later runs (Kimi, Qwen, Astra, DeepSeek) are on the fixed harness and settle it. - Prompt 1: one fetch, list exact, cb2024ge included (the fixed wording works). - Prompt 2: nine fetches, nine waves, **9/9 tables match** the live cells in order, including split DK/RA rows. - Prompt 3: 3/3 NATO waves and 3/3 age groups match. It also probed NATOSUPP for cb2009ge–cb2021ge (ten extra fetches, all on the site) to establish which waves carry the question — allowed by the rules, and more thorough than our own gather. - Prompt 4: HTML factsheet only, ten fields exact (1,509; 23%; 16 April–13 May 2024; CAPI; three languages; PI named); the five PDF-only fields written NOT GATHERED (PDF), as instructed after a failed probe C. - Prompt 5: 5,300-character note, every support sum (69/72/65/42/45/52/73/63/70) present, CHECK line "yes", no number outside the printed tables (scorer found none for unfetched waves). One prompt defect surfaced: it wrote "Date accessed: NOT GATHERED (not printed in prior answers)" — true, because no prompt in the sequence tells the model the date, and the write step is forbidden from inventing one. Fix (applied after the current runs finish, so all five models see the same text): Prompt 0 states the run date and Prompt 5 cites it. Fabrications: zero. Cost and time are the other result: a professional-grade nine-wave gather plus write for $1.23 and about 15 minutes, with every number checkable against the URL it came from. ## Same day — API harness (fixed), Kimi K3: full sequence, every cell correct, PDF read Second complete run, first on the harness that appends page links. 28 tool calls, $1.04, 8 minutes, no errors. - Prompt 0: **A, B, C, D all OK.** For C it followed the "Georgia methodological note (ENG)" link from the downloads page to the attachment and quoted the PDF's title block. That closes the question left open above: no model failed the PDF; the consumer platforms' fetch tools did, and our first harness version did. - Prompts 1–3: list exact with cb2024ge; 9/9 EU waves; 3/3 NATO; 3/3 age groups. - Prompt 3, beyond the brief: it opened NATOSUPP for every Georgia wave from 2009 and printed eleven tables. **Our own example note carries NATO for three waves (2019, 2021, 2024) — a gap in our gather, the third the tests have found.** Grok did the same. Verified live and extended below. - Prompt 4: ten factsheet fields exact, and — because probe C passed — the five PDF-only fields gathered from the methodological report (PSUs = electoral precincts from the 2020 parliamentary election lists; three strata × four quadrants; PPS; the requested citation format). It also copied the principal investigator's e-mail address off the factsheet. Public on the page, but not something a report should reproduce; the prompt rules get a line: never copy personal contact details (e-mail, phone), only institutional ones. - Prompt 5: 6,600-character note, all sums present, CHECK yes, zero numbers without a fetch. Same "date accessed" gap as Grok, flagged honestly rather than filled. Two models, two harness versions, twenty-one tables between them, every cell matching the live page. The variance in these tests has never been in the models' arithmetic or honesty; it has been in what the tools let them read. ## Same day — API harness: Qwen3.8 Max, GPT-6 Astra, DeepSeek V4 Flash Run sequentially after Kimi, same harness, same prompt text (the deferred fixes below were applied only after all five had finished). - **Qwen3.8 Max (0902):** four probes OK; 9/9 EU, 3/3 NATO plus the eight extra waves, 3/3 age groups, 4/4 methodology cells from the factsheet and the PDF fields from the methodological report. 28 fetches, $1.11, 21 minutes — the slowest of the five, mostly model time. Note of 16,500 characters, longer than the brief needs; every derived figure written as explicit arithmetic on printed cells. Same "date accessed: NOT GATHERED" as the others. Copied the PI's e-mail from the factsheet, like Kimi. - **GPT-6 Astra:** four probes OK; every gather table correct; 30 fetches. **Stopped after Prompt 4 at $5.09 against a $5.00 budget — never wrote the note.** At $10/$50 per million tokens the nine-wave gather alone costs four to five times the whole run on any other model here; the harness stops silently when the budget is crossed between prompts (a harness gap — it now logs the stop). Did not copy the e-mail. - **DeepSeek V4 Flash:** four probes OK, with the exact quoted evidence for each; every table correct; 27 fetches; note of 8,700 characters with the CRRC citation format and a CHECK line; **$0.08 total, 8 minutes.** Did not copy the e-mail. Its sources block gives no access date at all rather than flagging the gap. ## Results table — five models, one harness, one day | model | probes A B C D | EU waves | NATO | age xtab | method fields | numbers w/o fetch | CHECK | fetches | USD | reached | |---|---|---|---|---|---|---|---|---|---|---| | Grok 4.6 (harness v1, no links) | OK OK CANNOT OK | 9/9 | 3/3 +8 | 3/3 | 4/4 | 0 | yes | 32 | 1.23 | P5 | | Kimi K3 | OK OK OK OK | 9/9 | 3/3 +8 | 3/3 | 4/4 | 0 | yes | 28 | 1.04 | P5 | | Qwen3.8 Max | OK OK OK OK | 9/9 | 3/3 +8 | 3/3 | 4/4 | 0 | yes | 28 | 1.11 | P5 | | GPT-6 Astra | OK OK OK OK | 9/9 | 3/3 +8 | 3/3 | 4/4 | 0 | — | 30 | 5.09 | P4 (budget) | | DeepSeek V4 Flash | OK OK OK OK | 9/9 | 3/3 +8 | 3/3 | 4/4 | 0 | yes | 27 | 0.08 | P5 | Generated by `scripts/score_run.py --table` from the transcripts; "+8" = the eight NATO waves outside the brief that every model opened and got right. Zero fabricated numbers across 145 tool calls. The only failure was economic (Astra's budget). Consumer-app runs from earlier in the day are in the sections above; there the failures were tool failures (PDF, truncation, sandbox network), never arithmetic. ## Defects in our prompts and harness found by the five runs, all fixed 1. No prompt told the model the date → "Date accessed: NOT GATHERED". Prompt 0 now opens with the run date; Prompt 5 cites it. 2. Nothing forbade copying personal contact details → two models copied the PI's e-mail from the factsheet. Rule added: institutional names only, never e-mail addresses or phone numbers of people. 3. Harness v1 did not expose page links → Grok's probe C failed on our side. Fixed before the other four runs. 4. Harness stopped silently when the budget was crossed between prompts → now logged as a stop entry. 5. Published transcripts: the `.json` files carried full page dumps, including the methodological report's text; they now keep only the first 120 characters of each fetch result plus its length. And in our own data, found by the models: EU waves 2011–2012, NATO waves 2009–2017, the wording history — all verified live and added to the input pack and the example note.