From 71f21b733f0343346f9f378168d3996d4966a56b Mon Sep 17 00:00:00 2001 From: Colin Maudry Date: Thu, 16 Apr 2026 11:12:50 +0200 Subject: [PATCH] =?UTF-8?q?Mesure=20de=20l'impact=20m=C3=A9moire=20post-mi?= =?UTF-8?q?gration=20DuckDB=20(#71)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../specs/2026-04-15-duckdb-migration-design.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/superpowers/specs/2026-04-15-duckdb-migration-design.md b/docs/superpowers/specs/2026-04-15-duckdb-migration-design.md index daaee49..87c749d 100644 --- a/docs/superpowers/specs/2026-04-15-duckdb-migration-design.md +++ b/docs/superpowers/specs/2026-04-15-duckdb-migration-design.md @@ -178,3 +178,19 @@ Incremental — `df` global coexists with `src/db.py` until every page is migrat | `schema` shape change breaks `acheteur.py:303` | `schema` stays a `pl.Schema` object, not a list. One call site (`collect_schema()` → module `schema`) updated. | | Test runs inherit a stale DuckDB from a previous run with a different parquet | Tests force `REBUILD_DUCKDB=true` on cold runs; test DB added to `.gitignore`. | | Read-only connection opened before build finishes in another worker | Lock held across build + rename; read-only `connect` happens after lock release. Atomic `os.replace` guarantees a complete file. | + +## Outcome + +### Memory impact + +Memory measurement against the production parquet (`decp_prod.parquet`, ~1.5M rows) requires a running gunicorn process with access to the production data file. The measurement was deferred to the post-merge smoke test on the staging server (test.decp.info). + +**Expected reduction:** The removed globals (`df`, `df_acheteurs_departement`, `df_titulaires_departement`, `df_acheteurs_marches`, `df_titulaires_marches`) previously materialised the full 1.5M-row Parquet in memory as multiple Polars frames. At ~300 bytes/row × 5 frames, steady-state RSS reduction is estimated at **1–2 GB per worker**. The retained `df_acheteurs` and `df_titulaires` (autocomplete search) represent only the distinct-organisation subset (~tens of thousands of rows) and are negligible. + +**What remains in memory:** + +- `df_acheteurs` — distinct acheteurs with Marchés count (populated from DuckDB at startup) +- `df_titulaires` — same for titulaires +- DuckDB's own page cache (disk-backed, grows under load, evicted by OS) + +All per-request data is fetched from DuckDB and discarded after the callback returns.