Skip to content

Data freshness

Every dataset is re-checked against its source on data.gov.ro once a night. This page explains what that means in practice, how to tell when a dataset last changed, and why "checked" and "changed" are different things.

How the refresh works

A nightly pass walks the whole catalogue and asks each source whether it has changed. Most nights, for most datasets, the answer is no — Romanian public data is largely published annually or quarterly, so a re-check is cheap and a re-import is rare.

Change detection runs cheapest-first, and stops at the first rung that settles the question:

  1. Conditional request. We send the ETag and Last-Modified recorded on the last successful fetch. A 304 Not Modified ends the run immediately — no file is transferred.
  2. Content hash. If the source does answer with a file, its SHA-256 is compared against the last one. Byte-identical content ends the run too, before anything is parsed.
  3. Full pipeline. Only when the bytes genuinely differ does the dataset go through extract → standardize → validate → load, producing a new version.

So a dataset that hasn't been republished costs one HTTP request per night and produces no new version. Nothing you query changes underneath you.

Checked vs changed

Two timestamps matter, and they usually differ:

  • Last checked — the last time we asked the source whether it had new data. Under normal operation this is within the last 24 hours for every dataset.
  • Last changed — the last time the source actually published something different, and a new version was loaded. For an annual dataset this may be months ago, and that is correct, not stale.

GET /v1/datasets/{id} reports the dataset's latest ingestion, and every list response carries a Last-Modified header reflecting the most recent successful import. A Last-Modified from eight months ago means the data is eight months old, not that the pipeline stopped running.

Guards on a refresh

An automated nightly import must not be able to damage good data. Two gates stand in front of the load:

  • Validation gates. A dataset can declare on_error: fail or a max_reject_ratio, so a source that starts emitting malformed rows aborts the run instead of loading rubbish. Bad rows are quarantined either way, with the reason recorded.
  • Row-collapse guard. A replace load truncates before loading, so a truncated or half-written upstream file could otherwise swap a full dataset for a stub. If the incoming row count falls more than 50% below the last successful run, the run is refused and the existing data is left exactly as it was. Datasets that legitimately shrink can raise or disable the threshold with quality.max_row_drop_ratio.

Both gates run before anything is written, so a refused run leaves the previously served version in place. A refused run is a loud failure, not a silent one.

Provenance

Every row carries an ingestion_id pointing at the run that produced it, and each run records the source URL, the fetch timestamp, the content checksum, and the pipeline version that processed it. The original bytes are archived, so any served value can be traced back to the exact file it came from.