How to Build a Geospatial Data Pipeline That Survives Its Second Run

A geospatial pipeline that runs once is a script. A pipeline is what you have when the same process runs next month against changed source data, and you can tell immediately whether the result is trustworthy.

Separate the three layers

Keep ingestion, transformation and validation strictly apart. When they are mixed, a change in a source format means rewriting analysis logic, and a validation failure gives you no idea which stage caused it.

1. Ingestion: fetch and store raw source data, unmodified, with a timestamp. Never transform at this stage.
2. Transformation: parse, standardise, reproject, join. Deterministic, and re-runnable from the stored raw data.
3. Validation: assert the result meets stated criteria. Runs on every execution, not only when something looks wrong.

Be explicit about coordinate systems, everywhere

The most common failure in spatial pipelines is a silent projection mismatch. Nothing errors. Points simply land in the wrong place, or a distance comes out in degrees, and the result looks plausible enough to ship.

-- Constrain the CRS at the table level so bad data cannot be inserted
ALTER TABLE poi
  ADD CONSTRAINT poi_geom_srid CHECK (ST_SRID(geom) = 4326);

-- And measure in a projected system, never in degrees
SELECT id,
       ST_Distance(
         ST_Transform(geom, 32633),
         ST_Transform(:target, 32633)
       ) AS metres
FROM poi;

Write validation as assertions, not as eyeballing

Every run should assert a set of conditions and fail loudly when they are not met. The point is to find out from the pipeline rather than from the person who consumes the output three weeks later.

Make every stage idempotent

Running a stage twice must produce the same result as running it once. That means upserts rather than blind inserts, deterministic identifiers rather than sequence numbers, and no state carried between runs except explicitly. Without idempotency, a partial failure leaves data in a condition nobody can reason about.

Version the outputs, not just the code

• Tag each run with a version and a timestamp, and keep previous outputs.
• Record the vintage of every input layer, including the ones that rarely change.
• Store the parameters used, so a result can be reproduced exactly rather than approximately.
• Produce a change log per run: what was added, removed and modified.

Reproducibility is not an academic nicety here. When two analyses disagree, the version record is what lets you find out why in an hour instead of abandoning both.

A stack that does not fight you