Standardization rules

Every dataset in Canovix goes through the same normalization pipeline — one set of rules, applied uniformly, with no dataset-specific cleaning code. This is what makes the catalog predictable: what you learn querying one dataset holds for all of them.

Universal Silver rules

Applied to every raw file at ingest, before anything dataset-specific:

Column names — source headers become snake_case ASCII (Judeţ → judet), and a shared glossary maps common Romanian terms to canonical English names (judet → county, localitate → locality), so the same concept has the same name in every dataset.

Text hygiene — every text value: non-breaking/odd spaces normalized, internal whitespace runs collapsed to one space, leading/trailing whitespace stripped. Diacritics in values are preserved (data fidelity); accent-insensitive matching happens at query time.

Null tokens — cells that spell "no value" become real SQL NULLs: empty string, -, –, ., N/A, NA, n.a., null, nan, nedisponibil (case-insensitive).

Padding rows — rows where every cell is null (trailing Excel padding, visual separator rows) are dropped before validation.

Numbers — both thousands-separator conventions parse without configuration: 1,234,567.89 (US style), 1.234.567,89 (European style), and the bare decimal comma 3,14. Values that match neither become NULL rather than a wrong number.

The hard case is a lone group of three digits. 1.234 is either one thousand two hundred and thirty-four or one point two three four, and reading it wrongly is a thousand-fold error that looks perfectly valid — so the decision is made per column, from evidence, never per value:

• Some values settle it by themselves. Two or more groups (1.234.567) can only be grouping; a group closed by the other separator (1.234,56) settles both at once; a fraction that is not exactly three digits (3.14, 46.77012) can only be a decimal; and a leading zero (0.500) can never be grouping, because a grouped number never starts a group with zero.
• One such value anywhere in a column decides the convention for the whole column — real exports are internally consistent, which is what makes 1.234 readable at all.
• With no evidence either way, the declared type decides: an integer column reads it as grouping (a three-decimal value is not an integer), a decimal column keeps the digits as written rather than multiplying by a thousand on a guess.

Values read this way are counted per column and reported as ambiguous_separator in /v1/datasets/{id}/quality, so a coercion is something you can see rather than something you have to trust.

Dates — ISO 2023-05-01 plus the common Romanian formats 01.05.2023, 01/05/2023, and 01-05-2023 all parse to real DATE columns.

Types & quarantine — every column is cast to its declared type; unparseable values become NULL, and rows that then fail the dataset's validation rules are quarantined (stored with reasons), never served. A quality gate aborts the load if too much of a file is bad.

Provenance — every row references its ingestion run: source URL, checksum, pipeline version, import timestamp (surfaced at /v1/datasets/{id} → latest_ingestion).

Beyond these universal rules, a dataset's config may declare semantic transforms (mapping code values to labels, canonical entity resolution like manufacturer_canonical) — but never cleaning rules. Cleaning is the platform's job.

How datasets are onboarded

1. discover_candidates.py finds distinct, classifiable packages on data.gov.ro (collapsing monthly re-publications of the same dataset into one family).
2. scaffold_config.py downloads a resource, sniffs the format (delimiter, encoding, quoting, ragged rows, banner rows), infers columns + types, and writes a draft config.
3. bulk_onboard.py ingests each draft and keeps only what loads cleanly; everything else is dropped and recorded for manual review.

Datasets that need manual intervention

A large share of government Excel files have multi-row merged headers (title banners, merged cells, C0…C24 column codes) that no automatic parser can safely interpret. The pipeline deliberately refuses these rather than ingesting garbage.

They are queued, not forgotten: configs/_manual_queue.tsv records every refused package with its failure reason. The manual flow (a later milestone, ideally on the VPS) works the queue per file:

1. Open the sheet, identify the real header row → set format.header_row (and sheet).
2. Name the columns the header can't express (merged/coded headers) in the config's column map — the one genuinely human step.
3. Re-run the scaffolder with those hints, then ingest normally — everything downstream (universal rules, validation, serving, docs) is already generic.

No per-dataset code is ever written — even manual onboarding only produces a config.

Verifying it on your data

The rules above are claims; GET /v1/datasets/{id}/quality is the measurement. It reports, for the run whose rows you are currently querying:

• values_repaired — cells the text rules changed (odd spaces, collapsed runs, trimming).
• null_tokens_converted — cells spelling "no value" (-, n/a, nedisponibil, …) turned into real NULLs.
• padding_rows_dropped — all-null rows removed before validation.
• rows_rejected — rows quarantined by this dataset's validation rules.
• per column — nulls, completeness_pct, and cast_failed (values present in the source that could not be parsed into the declared type).
• ingestion — the run these figures describe: version, checksum, source URL and fetch time, so any number can be traced to an archived source file.

curl -s https://canovix.ro/v1/datasets/vehicle_registry/quality \
  -H "Authorization: Bearer $CANOVIX_KEY"

Completeness is floored, not rounded: a column with a single missing value in 380,000 rows reports 99.99%, never 100%. Only an exactly-complete column reads 100.

A dataset last imported before this measurement existed returns "measured": false with empty counts, rather than zeros that would read as "nothing needed fixing".