A week in Rome with SDMX

Ivan Atwiine and I spent last week in Rome at the 10th SDMX Global Conference, which the Bank of Italy hosted. It ran for three days of plenaries and two days of training. The theme was "Smarter data for better insights", and the call for papers asked for exactly our kind of work: AI in statistical pipelines, and dissemination tools that people actually use.

We went to talk about the pipeline. Our paper, "From fragmentation to intelligence", describes how UBOS pulls data straight from source systems into one platform and checks it on the way in, before it reaches the national dashboard. Rule-based checks catch the obvious problems, and machine learning helps spot odd values in what communities submit. It's the plumbing behind everything else the Bureau now does with data.

What I took home

SDMX works as a join key. I had been treating it as a way to publish. But once two datasets share the same concepts and code lists, a question that spans both becomes cheap to answer. A chatbot over official statistics needs exactly that. Most of the work of grounding a language model is agreeing on what an indicator means, and SDMX writes that agreement down.

Validation rules can be data too. The sessions on SDMX 3.0 and VTL, the validation and transformation language, treated rules as something offices share with each other. We write ours in Python scripts. Some of them should probably live next to the data instead.

Everyone struggles with scattered data. Central banks and statistical offices told the same story at different sizes. The exciting demos all rested on the boring, careful part, which is people agreeing on structure. I found that reassuring, because it's the part we can control.

The two training days made the tooling concrete. Reading the standard is one thing. Building a dataflow next to someone who has done it a hundred times, and watching where they slow down, teaches you more.

What changed back home

When we got back, we started a prototype that turns tables from the demographic and health survey and the household survey into SDMX-ready structures. We define structures by table shape, so one definition covers many tables. A crosswalk of indicators tells us quickly which survey holds a given figure. It's a first step toward chatbot answers that draw on more than one survey.

As for Rome, it was warm, and the coffee was as good as everyone says.