Five rules for AI over official statistics

Statistics offices across the continent are buying generative AI this year, often in a hurry. They tend to judge it by how fluent it sounds, and check whether its numbers are right later. When a model gives a confident, wrong district population, the office that published it pays for the mistake. Official statistics only matter because the public trusts the figure.

So before the Bureau releases Sanyu to the public, we wrote down the conditions we had to meet first. There are five, and each one came from a problem we hit while building it. They're short enough to check.

1. Every figure comes from the statistical system

The model chooses what to look up, and the database supplies the value. A number reaches the user only if a structured query returned it. That's the whole architecture in one sentence. The language model never writes a figure itself. It calls a typed function, and Sanyu copies the verified result into the answer word for word.

2. Every figure carries its source

Each value travels with its source table and survey, so a reader can trace it to the printed report. Charts draw from the same verified values as the text around them, so a picture can't disagree with its caption.

3. Sanyu says no outside its data

When a question goes beyond the published statistics, the right answer says so plainly. Keeping quiet about a limit counts as a bug. This one was harder to build than it sounds, because a fluent model would much rather answer.

4. We publish the test results before launch

A named benchmark with known answers runs on every build, along with regression tests that check exact published figures. Ours has 45 tests, including golden facts that must match the printed census tables to the digit. Anyone can see the results.

5. A statistician signs

A named statistician reviews any text the model writes before it goes out. The machine drafts, and the office stays the author.

Sanyu already meets all five.

None of these rules depends on a particular vendor or model. Any statistics office could adopt them this year, and we'd like other offices to argue with them and tell us what we missed. The details of how Sanyu meets them are in Sanyu's write-up.