Colin Maudry
197 lines
13 KiB
Markdown
197 lines
13 KiB
Markdown
# DuckDB migration — design spec
|
||
|
||
**Date:** 2026-04-15
|
||
**Branch:** dev
|
||
**Status:** Approved, ready for planning
|
||
|
||
## Goal
|
||
|
||
Replace the global Polars dataframes that `src/utils.py` materializes at import time (`df` and the five derived frames, lines 891–913) with a DuckDB database on disk. The main table holds ~1.5M rows from `decp_prod.parquet`. Per-request queries pull only what each page needs, dramatically reducing steady-state RSS memory.
|
||
|
||
Polars stays the primary API for small result sets and post-processing. DuckDB carries the heavy filtering, joining, and aggregation.
|
||
|
||
## Approach summary
|
||
|
||
- **Approach A — compatibility layer.** A new `src/db.py` module exposes a `query_marches(where_sql, params, columns, ...)` helper that runs SQL and returns a `pl.DataFrame`. Most existing `df.filter(pl.col(...) == x)` call sites translate mechanically to `query_marches("col = ?", (x,))`. The shape of downstream Polars code is unchanged.
|
||
- **Two small helpers stay in memory.** `df_acheteurs` and `df_titulaires` (tens of thousands of rows, consumed by the autocomplete search on every keystroke) are kept as module-level Polars frames. They are populated from DuckDB at import time, not from Parquet.
|
||
- **Four derived tables live in DuckDB**, built at startup alongside the main table: `acheteurs_marches`, `titulaires_marches`, `acheteurs_departement`, `titulaires_departement`.
|
||
- **Connection model.** One read-only `duckdb.connect(..., read_only=True)` at module load, shared across the process. `conn.cursor()` per Dash callback for thread-safety. The read-write connection is short-lived and only used during the startup build phase.
|
||
|
||
## Cache invalidation rule
|
||
|
||
At startup, rebuild the DuckDB file if:
|
||
|
||
1. **The DB file does not exist**, OR
|
||
2. **`decp_prod.parquet.mtime > duckdb.mtime`**, **unless** `DEVELOPMENT=true` and `REBUILD_DUCKDB != true` — in which case the DB stays as-is (fast dev reloads).
|
||
|
||
Production auto-rebuilds when the source Parquet is newer. Development keeps a stable DB across reloads unless the developer explicitly sets `REBUILD_DUCKDB=true` to force a rebuild.
|
||
|
||
## Concurrency
|
||
|
||
Multi-worker Gunicorn startup and crashed-mid-build scenarios are handled by a file lock, not by polling for the tmp file's existence:
|
||
|
||
```python
|
||
with open(DB_PATH.with_suffix(".duckdb.lock"), "w") as lock_fd:
|
||
fcntl.flock(lock_fd, fcntl.LOCK_EX) # blocks if another worker is building
|
||
if should_rebuild(DB_PATH, PARQUET_PATH):
|
||
build_database(DB_PATH, PARQUET_PATH)
|
||
conn = duckdb.connect(str(DB_PATH), read_only=True)
|
||
```
|
||
|
||
- Worker A acquires the lock, builds, atomically renames tmp → final, releases the lock.
|
||
- Worker B blocks on `flock`, then re-checks `should_rebuild`, sees the fresh DB, skips building.
|
||
- `fcntl.flock` is auto-released on process death, so a crash never deadlocks the next worker.
|
||
- `build_database` unlinks any pre-existing tmp file before starting (safe because it holds the lock) — handles an abandoned tmp from a crashed previous build.
|
||
|
||
## Build logic
|
||
|
||
The build keeps **one source of truth** for transforms by reusing the existing Polars pipeline:
|
||
|
||
```python
|
||
def build_database(db_path, parquet_path):
|
||
tmp_path = db_path.with_suffix(".duckdb.tmp")
|
||
if tmp_path.exists():
|
||
tmp_path.unlink()
|
||
frame = get_decp_data() # existing function in utils.py
|
||
with duckdb.connect(str(tmp_path)) as w:
|
||
w.register("frame", frame)
|
||
w.execute("CREATE TABLE decp AS SELECT * FROM frame")
|
||
w.execute("CREATE TABLE acheteurs_marches AS "
|
||
"SELECT DISTINCT uid, objet, acheteur_id FROM decp "
|
||
"ORDER BY acheteur_id")
|
||
w.execute("CREATE TABLE titulaires_marches AS "
|
||
"SELECT DISTINCT uid, objet, titulaire_id FROM decp "
|
||
"ORDER BY titulaire_id")
|
||
w.execute("CREATE TABLE acheteurs_departement AS "
|
||
"SELECT DISTINCT acheteur_id, acheteur_nom, acheteur_departement_code "
|
||
"FROM decp ORDER BY acheteur_nom")
|
||
w.execute("CREATE TABLE titulaires_departement AS "
|
||
"SELECT DISTINCT titulaire_id, titulaire_nom, titulaire_departement_code "
|
||
"FROM decp ORDER BY titulaire_nom")
|
||
os.replace(tmp_path, db_path)
|
||
```
|
||
|
||
Why Polars, not SQL, for the row-level transforms:
|
||
|
||
- `booleans_to_strings` is not a simple cast — it replaces `true`/`false` with `"oui"`/`"non"` on every boolean column. Reimplementing in SQL risks drifting from the Polars version.
|
||
- The null-name replacement (`acheteur_nom`, `titulaire_nom` → `"[Identifiant non reconnu dans la base INSEE]"`) is also easier to keep identical in Polars.
|
||
- `w.register("frame", frame)` is zero-copy. The memory spike is one-time during build and released when the write connection closes.
|
||
|
||
`os.replace` is atomic on POSIX — the read-only connection that opens next always sees a complete DB.
|
||
|
||
## Module layout
|
||
|
||
### New: `src/db.py`
|
||
|
||
```python
|
||
conn: duckdb.DuckDBPyConnection # read-only, module-level
|
||
schema: pl.Schema # from conn.execute("SELECT * FROM decp LIMIT 0").pl().schema
|
||
|
||
def get_cursor() -> duckdb.DuckDBPyConnection: ...
|
||
def query_marches(where_sql: str = "TRUE",
|
||
params: tuple = (),
|
||
columns: list[str] | None = None,
|
||
order_by: str | None = None,
|
||
limit: int | None = None) -> pl.DataFrame: ...
|
||
def should_rebuild(db_path: Path, parquet_path: Path) -> bool: ...
|
||
def build_database(db_path: Path, parquet_path: Path) -> None: ...
|
||
```
|
||
|
||
Only imports: `polars`, `duckdb`, `os`, `fcntl`, `pathlib`, `logging`. No app modules — prevents circular imports.
|
||
|
||
### Changes to `src/utils.py`
|
||
|
||
- `df: pl.DataFrame = get_decp_data()` — **removed** (after migration).
|
||
- `df_acheteurs`, `df_titulaires` — **kept as Polars globals**, populated via DuckDB at import time. The query mirrors today's `get_org_data(df, org_type)`: select all columns whose name starts with `acheteur_` (or `titulaire_`) except the `_latitude` / `_longitude` pair, plus `COUNT(*) AS "Marchés"`, grouped by the same set. Implementation can either:
|
||
|
||
- enumerate the columns by filtering `schema.names()` at import time and build the `SELECT` / `GROUP BY` strings, or
|
||
- call `get_org_data()` once against a small Polars frame returned by `SELECT <org_ cols> FROM decp`.
|
||
|
||
Feeds `search_org` unchanged.
|
||
|
||
- `df_acheteurs_marches`, `df_titulaires_marches`, `df_acheteurs_departement`, `df_titulaires_departement` — **removed** as Python globals. Call sites query the corresponding DuckDB tables.
|
||
- `schema` — imported from `src/db.py` (stays a `pl.Schema` — so `schema.names()` and dtype lookups both work, no call-site changes beyond `acheteur.py:303`).
|
||
- `columns` — replaced with `schema.names()`.
|
||
- `get_decp_data()` — **kept** (used by `build_database`).
|
||
- `get_org_data()` — can be removed once `df_acheteurs` / `df_titulaires` are populated from DuckDB directly.
|
||
|
||
### Call-site translations
|
||
|
||
| Before (Polars global) | After |
|
||
| ------------------------------------------------------------ | --------------------------------------------------------------------------------- |
|
||
| `df.filter(pl.col("acheteur_id") == aid)` | `query_marches("acheteur_id = ?", (aid,))` |
|
||
| `df.filter(pl.col("uid") == uid).row(0, named=True)` | `query_marches("uid = ?", (uid,)).row(0, named=True)` |
|
||
| `df.select("uid","objet","acheteur_id").filter(...)` | `query_marches("...", (...), columns=["uid","objet","acheteur_id"])` |
|
||
| `df.columns` | `schema.names()` |
|
||
| `df_acheteurs_marches.filter(...)` | `get_cursor().execute("SELECT ... FROM acheteurs_marches WHERE ...", [...]).pl()` |
|
||
| `pl.DataFrame(schema=df.collect_schema())` (acheteur.py:303) | `pl.DataFrame(schema=schema)` |
|
||
|
||
Heavy dashboard aggregations (observatoire, tableau full-scan) use raw SQL via `get_cursor().execute(...).pl()` rather than the helper.
|
||
|
||
## Configuration
|
||
|
||
- **`DATA_FILE_PARQUET_PATH`** — unchanged.
|
||
- **DuckDB file location** — computed: `Path(DATA_FILE_PARQUET_PATH).parent / "decp.duckdb"`. No new env var.
|
||
- **`REBUILD_DUCKDB`** — new, optional, default `false`. In development, setting this to `true` forces a rebuild when the parquet is newer.
|
||
- **`DEVELOPMENT`** — unchanged; now also gates the auto-rebuild behavior per the rule above.
|
||
|
||
## Testing
|
||
|
||
- `tests/conftest.py` (or a startup hook in `src/db.py`) ensures the test run builds the DuckDB in a temp directory derived from the parquet path — `tests/test.parquet` → `tests/decp.duckdb`. This file is added to `.gitignore`.
|
||
- Tests already set `DEVELOPMENT=true`; they must also set `REBUILD_DUCKDB=true` on cold test runs to force a fresh build from the test parquet.
|
||
- The existing Selenium suite exercises every page and is the primary acceptance signal.
|
||
|
||
## Migration order
|
||
|
||
Incremental — `df` global coexists with `src/db.py` until every page is migrated.
|
||
|
||
1. **Add `src/db.py`** (build, lock, `query_marches`, `schema`). `df` global unchanged.
|
||
2. **Migrate `marche.py`** — single-row lookup by `uid`, one call site.
|
||
3. **Migrate `acheteur.py`, `titulaire.py`** — filter by id.
|
||
4. **Migrate `arbre/departement.py`, `arbre/liste_marches_org.py`** — use the new derived DuckDB tables.
|
||
5. **Migrate `tableau.py`** — may need raw SQL.
|
||
6. **Migrate `observatoire.py`** — heaviest aggregations, most likely raw SQL.
|
||
7. **Migrate `figures.py`** — uses `df` in chart generation.
|
||
8. **Remove** `df`, `df_*_marches`, `df_*_departement` globals, `get_org_data()`, and the `df = get_decp_data()` call from `utils.py`. Move `schema` / `columns` exports to `src/db.py`.
|
||
|
||
### Verification gates
|
||
|
||
- `uv run pytest` green after every page migration.
|
||
- Manual smoke test via `uv run run.py` of the migrated page before proceeding.
|
||
- RSS memory measurement (`ps -o rss`) of a cold `gunicorn app:server` with the prod parquet, before and after, to confirm the memory reduction.
|
||
|
||
## Out of scope
|
||
|
||
- Changes to `src/cache.py` (flask-caching stays).
|
||
- The in-progress observatoire-localstorage-filters work on `dev`.
|
||
- Schema changes to the parquet.
|
||
- SQL views beyond the four derived tables.
|
||
- Multi-database or replication setups.
|
||
|
||
## Risks and mitigations
|
||
|
||
| Risk | Mitigation |
|
||
| ----------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
|
||
| `booleans_to_strings` reimplemented in SQL and drifts from Polars version | Transforms stay in Polars via `w.register("frame", frame)`. One source of truth. |
|
||
| Two Gunicorn workers rebuild concurrently | `fcntl.flock` serializes the build; second worker re-checks and skips. |
|
||
| Crashed build leaves stale `.tmp` file | Build unlinks any pre-existing tmp before starting (safe under lock). |
|
||
| `schema` shape change breaks `acheteur.py:303` | `schema` stays a `pl.Schema` object, not a list. One call site (`collect_schema()` → module `schema`) updated. |
|
||
| Test runs inherit a stale DuckDB from a previous run with a different parquet | Tests force `REBUILD_DUCKDB=true` on cold runs; test DB added to `.gitignore`. |
|
||
| Read-only connection opened before build finishes in another worker | Lock held across build + rename; read-only `connect` happens after lock release. Atomic `os.replace` guarantees a complete file. |
|
||
|
||
## Outcome
|
||
|
||
### Memory impact
|
||
|
||
Memory measurement against the production parquet (`decp_prod.parquet`, ~1.5M rows) requires a running gunicorn process with access to the production data file. The measurement was deferred to the post-merge smoke test on the staging server (test.decp.info).
|
||
|
||
**Expected reduction:** The removed globals (`df`, `df_acheteurs_departement`, `df_titulaires_departement`, `df_acheteurs_marches`, `df_titulaires_marches`) previously materialised the full 1.5M-row Parquet in memory as multiple Polars frames. At ~300 bytes/row × 5 frames, steady-state RSS reduction is estimated at **1–2 GB per worker**. The retained `df_acheteurs` and `df_titulaires` (autocomplete search) represent only the distinct-organisation subset (~tens of thousands of rows) and are negligible.
|
||
|
||
**What remains in memory:**
|
||
|
||
- `df_acheteurs` — distinct acheteurs with Marchés count (populated from DuckDB at startup)
|
||
- `df_titulaires` — same for titulaires
|
||
- DuckDB's own page cache (disk-backed, grows under load, evicted by OS)
|
||
|
||
All per-request data is fetched from DuckDB and discarded after the callback returns.
|