Merge branch 'release/2.7.3'

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

.gitignore

@@ -3,5 +3,11 @@
 __pycache__
 .idea
 .venv
+.worktrees
 build
 .env
+
+# DuckDB runtime artifacts (regenerated from decp_prod.parquet at startup)
+**/decp.duckdb
+**/decp.duckdb.tmp
+**/decp.duckdb.lock

CHANGELOG.md

@@ -1,4 +1,17 @@
-#### 2.7.1 (23 mars 2026)
+##### 2.7.3 (20 avril 2026)
+
+- Mise en cache des vues tableau par ensemble de filtres et de tris
+- Résolution du bug d'écriture du fichier de vérouillage de la base de données
+
+##### 2.7.2 (19 avril 2026)
+
+- Chargement des données depuis une base DuckDB plutôt qu'en mémoire (plus de stabilité) ([#71](https://github.com/ColinMaudry/decp.info/issues/71))
+- Mise en cache des vue sur l'observatoire pour un chargement plus rapide (remise à zéro quotidienne)
+- Correction de bug : la liste de colonnes par défaut est bien appliquée plutôt qu'afficher toutes les colonnes
+- Quelques corrections de bugs d'affichage
+- Refactorisation des fonctions utilitaires (`utils.py` approchait des 1 000 lignes)
+
+##### 2.7.1 (23 mars 2026)
 
 - Correction du partage de données filtrées entre dashboard et vue des données
 
@@ -182,7 +195,7 @@
 
 ### 1.0.0
 
-- publication sur https://decp.info
+- publication sur <https://decp.info>
 - ajout d'une vue équivalente au format DECP réglementaire
 - personnalisation de datasette
 - script de conversion quotidien basé sur [dataflows](https://github.com/datahq/dataflows)

CLAUDE.md

@@ -11,8 +11,6 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 ### Setup
 
 ```bash
-python -m venv .venv && source .venv/bin/activate
-pip install ".[dev]"
 cp template.env .env   # then customize .env
 ```
 
@@ -25,14 +23,14 @@ uv run run.py         # starts Dash with debug=True and hot reload
 ### Production
 
 ```bash
-gunicorn app:server
+uv run gunicorn app:server
 ```
 
 ### Tests
 
 ```bash
-uv run pytest                  # run all tests (Selenium-based integration tests)
+rtk uv run pytest                  # run all tests (Selenium-based integration tests)
-uv run pytest tests/test_main.py::test_001_logo_and_search   # run a single test
+rtk uv run pytest tests/test_main.py::test_001_logo_and_search   # run a single test
 ```
 
 Tests require a running Chrome/Chromium browser. They use `DashComposite` from `dash[testing]` with Selenium WebDriver.

README.md

@@ -1,6 +1,6 @@
 # decp.info
 
-> v2.7.1
+> v2.7.3
 > Outil d'exploration et de téléchargement des données essentielles de la commande publique.
 
 => [decp.info](https://decp.info)
@@ -8,19 +8,15 @@
 ## Installation et lancement
 
 ```shell
-python -m venv .venv
-source .venv/bin/activate
-pip install .
-
 # Copie et personnalisation du .env
 cp template.env .env
 nano .env
 
 # Pour la production
-gunicorn app:server
+uv run gunicorn app:server
 
 # Pour avoir le debuggage et le hot reload
-python run.py
+uv run run.py
 ```
 
 ## Déploiement

docs/superpowers/plans/2026-04-19-tableau-prepare-table-data-cache.md

@@ -0,0 +1,838 @@
+# Tableau prepare_table_data Cache Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Make page navigation, sort changes, and repeated filter visits in the `/tableau` page near-instant by memoizing the expensive filter+sort+post-process pipeline inside `prepare_table_data`.
+
+**Architecture:** Extract a memoized inner function `_load_filter_sort_postprocess(filter_query, sort_by_key)` that performs the heavy work (load full data, filter, sort, collect, cast-to-string, fill-null, add HTML links, format values) and returns a fully post-processed Polars DataFrame. The outer `prepare_table_data` becomes a thin wrapper that handles non-deterministic side effects (`track_search`, `uuid.uuid4()` for cleanup trigger, `data_timestamp + 1`) and pagination. The memoized helper only runs when no `data` argument is passed (i.e., the Tableau path). Other callers (`acheteur`, `titulaire`, `observatoire`) keep the current uncached path because they pass an externally-provided LazyFrame that is not safely hashable for cache keys.
+
+**Tech Stack:** Polars (LazyFrame, DataFrame), Flask-Caching (`@cache.memoize()` on `FileSystemCache` already configured in `src/app.py:38`), pytest for unit tests.
+
+**Git**: the issue id is #72, add the reference in commit messages.
+
+---
+
+## Background and constraints
+
+Read these before starting; they explain why the design takes the shape it does.
+
+1. **Cache infrastructure is already wired.** `src/cache.py` defines `cache = Cache()`. `src/app.py:38-48` initializes it with `FileSystemCache`, default 24h timeout, `CACHE_THRESHOLD=300`. The cache directory is wiped on every restart (`rmtree` at `src/app.py:36`), so cache always starts empty.
+
+2. **Existing pattern to mirror.** `src/pages/observatoire.py:650-660` already uses `@cache.memoize()` plus a `_normalize_filter_params` helper that converts a dict of filters into a hashable tuple. This plan applies the same idiom to `sort_by` (which is a `list[dict]` from Dash DataTable).
+
+3. **Non-deterministic outputs that MUST stay outside the memoized function:**
+
+   - `data_timestamp + 1` (increments each call; would freeze if cached)
+   - `trigger_cleanup = str(uuid.uuid4())` (intentionally unique per call to fire the clientside filter-cleanup callback)
+   - `track_search(filter_query, source_table)` — Matomo HTTP POST, currently called inside `filter_table_data` at `src/utils/table.py:214`. Must fire on every user action including cache hits.
+
+4. **Tracking call site move.** `track_search` must move OUT of `filter_table_data` and into each caller, otherwise cache hits would silently skip Matomo tracking. Current callers of `filter_table_data` to update:
+
+   - `src/utils/table.py:402` (inside `prepare_table_data`)
+   - `src/pages/tableau.py:325` (`download_data` callback)
+   - `src/pages/acheteur.py:427` (`download_data_acheteur` callback)
+   - `src/pages/titulaire.py:443` (`download_data_titulaire` callback)
+
+5. **Why Tableau-only caching.** `prepare_table_data` is also called from `acheteur.py`, `titulaire.py`, `observatoire.py`. Those callers pass a pre-filtered LazyFrame or list-of-dicts as `data`. Hashing arbitrary LazyFrames or large lists for memoization is impractical. The fix gates on `data is None` (the Tableau path) and leaves the other paths byte-for-byte identical.
+
+6. **Cache key composition.** The memoized function takes only `(filter_query, sort_by_key)`. `page_current` and `page_size` are intentionally NOT in the key — pagination happens in the outer wrapper after retrieving the cached, fully post-processed frame. This means every page click and page-size change is a cache hit (the whole point of the change).
+
+7. **Pickling.** Flask-Caching pickles arguments to form keys and pickles return values to disk. Polars `DataFrame` pickles cleanly. `LazyFrame` does not — so the memoized function must `.collect()` before returning.
+
+8. **File path expectations.** All paths below are relative to repo root `/home/colin/git/decp.info`. Run all commands from there.
+
+---
+
+## File Structure
+
+- **Modify** `src/utils/table.py` — extract memoized helper, refactor `prepare_table_data`, remove `track_search` call from `filter_table_data`.
+- **Modify** `src/pages/tableau.py` — add explicit `track_search` call in `download_data`.
+- **Modify** `src/pages/acheteur.py` — add explicit `track_search` call in `download_data_acheteur`.
+- **Modify** `src/pages/titulaire.py` — add explicit `track_search` call in `download_data_titulaire`.
+- **Create** `tests/test_table.py` — unit tests for new helpers and refactored `prepare_table_data`.
+
+---
+
+## Task 1: Set up unit tests for table.py
+
+**Files:**
+
+- Create: `tests/test_table.py`
+
+This task scaffolds a non-Selenium pytest module so subsequent tasks can do TDD without booting a Dash server. The conftest already writes a small `tests/test.parquet` fixture (see `tests/conftest.py:10`); reuse it.
+
+- [ ] **Step 1: Write the failing test**
+
+Create `tests/test_table.py` with:
+
+```python
+import os
+
+import polars as pl
+import pytest
+
+
+@pytest.fixture
+def sample_lff():
+    """Small LazyFrame with the columns needed by add_links / format_values."""
+    return pl.LazyFrame(
+        [
+            {
+                "uid": "u1",
+                "id": "u1",
+                "acheteur_id": "12345678900011",
+                "acheteur_nom": "Mairie de Test",
+                "titulaire_id": "98765432100022",
+                "titulaire_nom": "Entreprise Test",
+                "titulaire_typeIdentifiant": "SIRET",
+                "objet": "Travaux divers",
+                "montant": 12500.0,
+                "dateNotification": "2025-03-15",
+                "codeCPV": "45000000",
+                "dureeRestanteMois": 6,
+                "titulaire_distance": 42.0,
+            }
+        ]
+    )
+
+
+def test_table_module_imports():
+    from src.utils import table
+
+    assert hasattr(table, "prepare_table_data")
+```
+
+- [ ] **Step 2: Run test to verify it passes (sanity check)**
+
+Run: `uv run pytest tests/test_table.py -v`
+Expected: PASS for `test_table_module_imports`. (Selenium is not invoked because no `dash_duo` fixture is used.)
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add tests/test_table.py
+git commit -m "test: scaffold unit tests for table utilities"
+```
+
+---
+
+## Task 2: Move track_search out of filter_table_data
+
+**Files:**
+
+- Modify: `src/utils/table.py:210-274` (remove `track_search` import usage at line 214)
+- Modify: `src/pages/tableau.py:317-334` (`download_data` callback)
+- Modify: `src/pages/acheteur.py:425-430` area (`download_data_acheteur` callback)
+- Modify: `src/pages/titulaire.py:441-446` area (`download_data_titulaire` callback)
+- Modify: `tests/test_table.py` (add a test that confirms `filter_table_data` no longer calls Matomo)
+
+`track_search` must move out so that the soon-to-be-memoized helper does not swallow tracking on cache hits. We do this BEFORE introducing caching so that the diff is small and verifiable on its own.
+
+- [ ] **Step 1: Write the failing test**
+
+Append to `tests/test_table.py`:
+
+```python
+def test_filter_table_data_does_not_call_track_search(monkeypatch, sample_lff):
+    from src.utils import table
+
+    calls = []
+    monkeypatch.setattr(table, "track_search", lambda *a, **kw: calls.append(a))
+
+    result = table.filter_table_data(
+        sample_lff, "{objet} icontains travaux", "tableau"
+    ).collect()
+
+    assert calls == []
+    assert result.height == 1
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `uv run pytest tests/test_table.py::test_filter_table_data_does_not_call_track_search -v`
+Expected: FAIL (`assert calls == []` fails because `filter_table_data` currently calls `track_search` at line 214).
+
+- [ ] **Step 3: Remove the track_search call from filter_table_data**
+
+Edit `src/utils/table.py` — find this block:
+
+```python
+def filter_table_data(
+    lff: pl.LazyFrame, filter_query: str, filter_source: str
+) -> pl.LazyFrame:
+    _schema = lff.collect_schema()
+    track_search(filter_query, filter_source)
+    filtering_expressions = filter_query.split(" && ")
+```
+
+Remove the `track_search(filter_query, filter_source)` line. Result:
+
+```python
+def filter_table_data(
+    lff: pl.LazyFrame, filter_query: str, filter_source: str
+) -> pl.LazyFrame:
+    _schema = lff.collect_schema()
+    filtering_expressions = filter_query.split(" && ")
+```
+
+The `filter_source` parameter remains in the signature (avoids changing all callers in this task). It becomes unused; that is acceptable since callers will pass it again later if needed. Do NOT remove the `from src.utils.tracking import track_search` import yet — `prepare_table_data` will use it in Task 5.
+
+- [ ] **Step 4: Add explicit track_search calls in download callbacks**
+
+In `src/pages/tableau.py`, find:
+
+```python
+def download_data(n_clicks, filter_query, sort_by, hidden_columns: list = None):
+    lff: pl.LazyFrame = query_marches().lazy()
+
+    # Les colonnes masquées sont supprimées
+    if hidden_columns:
+        lff = lff.drop(hidden_columns)
+
+    if filter_query:
+        lff = filter_table_data(lff, filter_query, "tab download")
+```
+
+Insert a `track_search` call so behavior is preserved. First add the import at the top of `src/pages/tableau.py` next to other `src.utils` imports:
+
+```python
+from src.utils.tracking import track_search
+```
+
+Then change the body:
+
+```python
+def download_data(n_clicks, filter_query, sort_by, hidden_columns: list = None):
+    lff: pl.LazyFrame = query_marches().lazy()
+
+    # Les colonnes masquées sont supprimées
+    if hidden_columns:
+        lff = lff.drop(hidden_columns)
+
+    if filter_query:
+        track_search(filter_query, "tab download")
+        lff = filter_table_data(lff, filter_query, "tab download")
+```
+
+Repeat the same pattern in `src/pages/acheteur.py` (search for `filter_table_data(lff, filter_query, "ach download")`):
+
+Add import:
+
+```python
+from src.utils.tracking import track_search
+```
+
+Wrap the call:
+
+```python
+    if filter_query:
+        track_search(filter_query, "ach download")
+        lff = filter_table_data(lff, filter_query, "ach download")
+```
+
+Repeat in `src/pages/titulaire.py` (search for `filter_table_data(lff, filter_query, "titu download")`):
+
+Add import:
+
+```python
+from src.utils.tracking import track_search
+```
+
+Wrap the call:
+
+```python
+    if filter_query:
+        track_search(filter_query, "titu download")
+        lff = filter_table_data(lff, filter_query, "titu download")
+```
+
+- [ ] **Step 5: Run test to verify it passes**
+
+Run: `uv run pytest tests/test_table.py::test_filter_table_data_does_not_call_track_search -v`
+Expected: PASS.
+
+- [ ] **Step 6: Run full unit test file to verify no regressions**
+
+Run: `uv run pytest tests/test_table.py -v`
+Expected: All tests in `test_table.py` PASS.
+
+- [ ] **Step 7: Commit**
+
+```bash
+git add src/utils/table.py src/pages/tableau.py src/pages/acheteur.py src/pages/titulaire.py tests/test_table.py
+git commit -m "refactor: move track_search out of filter_table_data into callers"
+```
+
+---
+
+## Task 3: Add normalize_sort_by helper
+
+**Files:**
+
+- Modify: `src/utils/table.py` (add helper near other utility functions, e.g. after `dates_to_strings`)
+- Modify: `tests/test_table.py` (add tests)
+
+A cache key must be hashable. Dash DataTable's `sort_by` is a `list[dict]` like `[{"column_id": "montant", "direction": "asc"}, ...]`, which is not hashable. We mirror the `_normalize_filter_params` idiom from `src/pages/observatoire.py:650-657`.
+
+- [ ] **Step 1: Write the failing tests**
+
+Append to `tests/test_table.py`:
+
+```python
+def test_normalize_sort_by_handles_empty():
+    from src.utils.table import normalize_sort_by
+
+    assert normalize_sort_by(None) == ()
+    assert normalize_sort_by([]) == ()
+
+
+def test_normalize_sort_by_returns_hashable_tuple():
+    from src.utils.table import normalize_sort_by
+
+    sort_by = [
+        {"column_id": "montant", "direction": "desc"},
+        {"column_id": "dateNotification", "direction": "asc"},
+    ]
+    key = normalize_sort_by(sort_by)
+
+    assert key == (("montant", "desc"), ("dateNotification", "asc"))
+    # Must be hashable so that flask-caching can build a cache key from it
+    hash(key)
+
+
+def test_normalize_sort_by_preserves_order():
+    """Order matters for sort: [A, B] != [B, A]."""
+    from src.utils.table import normalize_sort_by
+
+    a_then_b = normalize_sort_by(
+        [{"column_id": "a", "direction": "asc"}, {"column_id": "b", "direction": "asc"}]
+    )
+    b_then_a = normalize_sort_by(
+        [{"column_id": "b", "direction": "asc"}, {"column_id": "a", "direction": "asc"}]
+    )
+    assert a_then_b != b_then_a
+```
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `uv run pytest tests/test_table.py -v -k normalize_sort_by`
+Expected: FAIL with `ImportError` for `normalize_sort_by`.
+
+- [ ] **Step 3: Implement normalize_sort_by**
+
+Edit `src/utils/table.py`. Add this function immediately after the `dates_to_strings` function (around line 148):
+
+```python
+def normalize_sort_by(sort_by) -> tuple:
+    """Convert Dash DataTable sort_by (list[dict]) into a hashable tuple
+    suitable for use as a cache key. Order is preserved because it determines
+    sort precedence."""
+    if not sort_by:
+        return ()
+    return tuple((entry["column_id"], entry["direction"]) for entry in sort_by)
+```
+
+- [ ] **Step 4: Run tests to verify they pass**
+
+Run: `uv run pytest tests/test_table.py -v -k normalize_sort_by`
+Expected: 3 PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add src/utils/table.py tests/test_table.py
+git commit -m "feat: add normalize_sort_by hashable cache-key helper"
+```
+
+---
+
+## Task 4: Extract memoized post-process helper
+
+**Files:**
+
+- Modify: `src/utils/table.py` (add `_load_filter_sort_postprocess`, decorate with `@cache.memoize()`, import `cache`)
+- Modify: `tests/test_table.py` (add tests)
+
+Introduce the function whose result will live in the FileSystemCache. Inputs: `(filter_query, sort_by_key)`. Output: a fully post-processed, unpaginated Polars DataFrame ready to slice and convert to dicts.
+
+This task does NOT yet wire the helper into `prepare_table_data` — that happens in Task 5. Splitting these tasks keeps each diff small and testable.
+
+- [ ] **Step 1: Write the failing tests**
+
+Append to `tests/test_table.py`:
+
+```python
+@pytest.fixture(autouse=True)
+def reset_cache():
+    """Ensure the flask-caching backend is empty between tests so that
+    cache-hit assertions are meaningful. Falls back to no-op when no
+    Flask app context is active (NullCache)."""
+    from utils.cache import cache
+
+    try:
+        cache.clear()
+    except RuntimeError:
+        # No app context — cache is NullCache, nothing to clear
+        pass
+    yield
+
+
+def test_load_filter_sort_postprocess_returns_dataframe(monkeypatch, sample_lff):
+    from src.utils import table
+
+    monkeypatch.setattr(
+        table, "query_marches", lambda: sample_lff.collect()
+    )
+
+    df = table._load_filter_sort_postprocess(filter_query=None, sort_by_key=())
+
+    assert isinstance(df, pl.DataFrame)
+    assert df.height == 1
+    # All values must be strings after post-processing
+    for col in df.columns:
+        assert df.schema[col] == pl.String
+
+
+def test_load_filter_sort_postprocess_applies_filter(monkeypatch, sample_lff):
+    from src.utils import table
+
+    monkeypatch.setattr(
+        table, "query_marches", lambda: sample_lff.collect()
+    )
+
+    df = table._load_filter_sort_postprocess(
+        filter_query="{objet} icontains travaux", sort_by_key=()
+    )
+    assert df.height == 1
+
+    df_empty = table._load_filter_sort_postprocess(
+        filter_query="{objet} icontains nonexistent", sort_by_key=()
+    )
+    assert df_empty.height == 0
+
+
+def test_load_filter_sort_postprocess_adds_links(monkeypatch, sample_lff):
+    from src.utils import table
+
+    monkeypatch.setattr(
+        table, "query_marches", lambda: sample_lff.collect()
+    )
+
+    df = table._load_filter_sort_postprocess(filter_query=None, sort_by_key=())
+    # add_links injects an <a href> wrapper around uid, acheteur_nom, titulaire_nom
+    assert "<a href" in df["uid"][0]
+    assert "<a href" in df["acheteur_nom"][0]
+    assert "<a href" in df["titulaire_nom"][0]
+```
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `uv run pytest tests/test_table.py -v -k load_filter_sort_postprocess`
+Expected: FAIL with `AttributeError: module 'src.utils.table' has no attribute '_load_filter_sort_postprocess'`.
+
+- [ ] **Step 3: Implement the helper**
+
+Edit `src/utils/table.py`. Add this import near the top, with the other `src.` imports:
+
+```python
+from utils.cache import cache
+```
+
+Then add the helper function. Place it ABOVE `prepare_table_data` (around line 370, just before `def prepare_table_data`):
+
+```python
+@cache.memoize()
+def _load_filter_sort_postprocess(filter_query, sort_by_key):
+    """Memoized core of the Tableau page pipeline.
+
+    Loads the full marchés dataset, applies filter and sort, materializes,
+    then runs the per-row post-processing (cast to string, fill nulls, add
+    HTML links, format values). Returns an unpaginated Polars DataFrame.
+
+    Inputs MUST be hashable: filter_query is str|None, sort_by_key is the
+    tuple produced by normalize_sort_by(). Pagination intentionally lives
+    in the outer wrapper so that page changes are cache hits.
+    """
+    logger.debug(f"Cache miss — recomputing for filter={filter_query!r} sort={sort_by_key!r}")
+
+    lff: pl.LazyFrame = query_marches().lazy()
+
+    if filter_query:
+        lff = filter_table_data(lff, filter_query, "tableau")
+
+    if sort_by_key:
+        sort_by = [
+            {"column_id": col, "direction": direction}
+            for col, direction in sort_by_key
+        ]
+        lff = sort_table_data(lff, sort_by)
+
+
+    # The remaining steps are cheap per-row operations that we run ONCE here
+    # so that pagination in the outer function is a pure slice + to_dicts.
+    lff = lff.cast(pl.String)
+    lff = lff.fill_null("")
+
+    dff: pl.DataFrame = lff.collect()
+
+    dff = add_links(dff)
+    if "sourceFile" in dff.columns:
+        dff = add_resource_link(dff)
+    if dff.height > 0:
+        dff = format_values(dff)
+
+    return dff
+```
+
+- [ ] **Step 4: Run tests to verify they pass**
+
+Run: `uv run pytest tests/test_table.py -v -k load_filter_sort_postprocess`
+Expected: 3 PASS.
+
+- [ ] **Step 5: Run the full test_table.py to catch regressions**
+
+Run: `uv run pytest tests/test_table.py -v`
+Expected: All PASS.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add src/utils/table.py tests/test_table.py
+git commit -m "feat: add memoized _load_filter_sort_postprocess helper"
+```
+
+---
+
+## Task 5: Wire the memoized helper into prepare_table_data
+
+**Files:**
+
+- Modify: `src/utils/table.py` — replace the body of `prepare_table_data` so the Tableau path uses the cache
+- Modify: `tests/test_table.py` — add tests covering the new flow
+
+The outer function keeps its signature unchanged so callers in `acheteur.py`, `titulaire.py`, `observatoire.py`, `tableau.py` need no updates. When `data is None` (the Tableau case), use the memoized helper; otherwise fall through to the original logic.
+
+- [ ] **Step 1: Write the failing tests**
+
+Append to `tests/test_table.py`:
+
+```python
+def test_prepare_table_data_returns_expected_tuple(monkeypatch, sample_lff):
+    from src.utils import table
+
+    monkeypatch.setattr(
+        table, "query_marches", lambda: sample_lff.collect()
+    )
+
+    result = table.prepare_table_data(
+        data=None,
+        data_timestamp=5,
+        filter_query=None,
+        page_current=0,
+        page_size=20,
+        sort_by=[],
+        source_table="tableau",
+    )
+
+    # Same arity as before: 9 outputs
+    assert len(result) == 9
+    dicts, columns, tooltip, ts, nb_rows, dl_disabled, dl_text, dl_title, cleanup = result
+    assert isinstance(dicts, list)
+    assert ts == 6  # data_timestamp + 1 must still increment
+    assert "1 lignes" in nb_rows
+
+
+def test_prepare_table_data_calls_track_search_on_filter(monkeypatch, sample_lff):
+    from src.utils import table
+
+    calls = []
+    monkeypatch.setattr(
+        table, "query_marches", lambda: sample_lff.collect()
+    )
+    monkeypatch.setattr(table, "track_search", lambda *a, **kw: calls.append(a))
+
+    table.prepare_table_data(
+        data=None,
+        data_timestamp=0,
+        filter_query="{objet} icontains travaux",
+        page_current=0,
+        page_size=20,
+        sort_by=[],
+        source_table="tableau",
+    )
+
+    assert calls == [("{objet} icontains travaux", "tableau")]
+
+
+def test_prepare_table_data_paginates_without_recomputing(monkeypatch, sample_lff):
+    """Two calls with same filter+sort but different pages must invoke
+    the inner heavy work only once."""
+    from src.utils import table
+
+    call_count = {"n": 0}
+    real_query = sample_lff.collect()
+
+    def counting_query():
+        call_count["n"] += 1
+        return real_query
+
+    monkeypatch.setattr(table, "query_marches", counting_query)
+
+    # First call: cache miss
+    table.prepare_table_data(
+        data=None,
+        data_timestamp=0,
+        filter_query=None,
+        page_current=0,
+        page_size=10,
+        sort_by=[],
+        source_table="tableau",
+    )
+    first_count = call_count["n"]
+
+    # Second call, different page: cache hit, query_marches must NOT fire again
+    table.prepare_table_data(
+        data=None,
+        data_timestamp=0,
+        filter_query=None,
+        page_current=1,
+        page_size=10,
+        sort_by=[],
+        source_table="tableau",
+    )
+
+    assert call_count["n"] == first_count, (
+        "query_marches was called again — pagination triggered cache miss"
+    )
+
+
+def test_prepare_table_data_cleanup_trigger_for_non_tableau(monkeypatch, sample_lff):
+    """Non-tableau pages still get a fresh uuid trigger, not no_update."""
+    from dash import no_update
+
+    from src.utils import table
+
+    monkeypatch.setattr(
+        table, "query_marches", lambda: sample_lff.collect()
+    )
+
+    result = table.prepare_table_data(
+        data=None,
+        data_timestamp=0,
+        filter_query="{objet} icontains travaux",
+        page_current=0,
+        page_size=20,
+        sort_by=[],
+        source_table="acheteur",
+    )
+
+    cleanup = result[8]
+    assert cleanup is not no_update
+    assert isinstance(cleanup, str)
+    assert len(cleanup) >= 32  # uuid4 hex string
+
+
+def test_prepare_table_data_with_external_data_does_not_use_cache(
+    monkeypatch, sample_lff
+):
+    """When a caller passes data (acheteur/titulaire/observatoire path),
+    bypass the memoized helper entirely."""
+    from src.utils import table
+
+    sentinel = {"called": False}
+
+    def should_not_be_called(*a, **kw):
+        sentinel["called"] = True
+        raise AssertionError("Memoized helper must not be called when data is provided")
+
+    monkeypatch.setattr(
+        table, "_load_filter_sort_postprocess", should_not_be_called
+    )
+
+    table.prepare_table_data(
+        data=sample_lff,  # external LazyFrame
+        data_timestamp=0,
+        filter_query=None,
+        page_current=0,
+        page_size=20,
+        sort_by=[],
+        source_table="acheteur",
+    )
+
+    assert sentinel["called"] is False
+```
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `uv run pytest tests/test_table.py -v -k prepare_table_data`
+Expected: At least the cache-hit (`paginates_without_recomputing`) and `track_search`-routing tests FAIL because the current `prepare_table_data` re-runs the full pipeline on every call and routes tracking through `filter_table_data` (which Task 2 already neutralized — so tracking would be lost without the new explicit call).
+
+- [ ] **Step 3: Refactor prepare_table_data**
+
+Edit `src/utils/table.py`. Replace the entire `prepare_table_data` function body with:
+
+```python
+def prepare_table_data(
+    data, data_timestamp, filter_query, page_current, page_size, sort_by, source_table
+):
+    """
+    Préparation des données pour les datatables.
+
+    Pour la page Tableau (data is None), le calcul lourd (chargement complet,
+    filtre, tri, post-traitement) est mémorisé via _load_filter_sort_postprocess.
+    Les changements de page deviennent ainsi des cache hits.
+
+    Pour les autres pages (data fourni), le chemin original est conservé : la
+    LazyFrame externe n'est pas hashable et le coût de filtre/tri y est déjà
+    minime puisque les données sont pré-restreintes.
+    """
+    logger.debug(" + + + + + + + + + + + + + + + + + + ")
+
+    # Side effect non-cacheable : le tracking doit firer sur chaque action
+    # utilisateur, y compris sur cache hit.
+    if filter_query:
+        track_search(filter_query, source_table)
+
+    # Trigger uuid pour les pages autres que tableau (clientside cleanup)
+    trigger_cleanup = (
+        no_update if source_table == "tableau" else str(uuid.uuid4())
+    )
+
+    if data is None:
+        # Tableau path : utilise le cache
+        sort_by_key = normalize_sort_by(sort_by)
+        dff: pl.DataFrame = _load_filter_sort_postprocess(
+            filter_query=filter_query, sort_by_key=sort_by_key
+        )
+    else:
+        # acheteur / titulaire / observatoire path : code original, non caché
+        if isinstance(data, list):
+            lff: pl.LazyFrame = pl.LazyFrame(
+                data, strict=False, infer_schema_length=5000
+            )
+        elif isinstance(data, pl.LazyFrame):
+            lff = data
+        else:
+            lff = query_marches().lazy()
+
+        if filter_query:
+            lff = filter_table_data(lff, filter_query, source_table)
+
+        if sort_by and len(sort_by) > 0:
+            lff = sort_table_data(lff, sort_by)
+
+        dff = lff.collect()
+        dff = dff.cast(pl.String)
+        dff = dff.fill_null("")
+        dff = add_links(dff)
+        if "sourceFile" in dff.columns:
+            dff = add_resource_link(dff)
+        if dff.height > 0:
+            dff = format_values(dff)
+
+    height = dff.height
+
+    if height > 0:
+        nb_rows = (
+            f"{format_number(height)} lignes "
+            f"({format_number(dff.select('uid').unique().height)} marchés)"
+        )
+    else:
+        nb_rows = "0 lignes (0 marchés)"
+
+    # Pagination — toujours hors cache pour rester sur des cache hits
+    start_row = page_current * page_size
+    dff = dff.slice(start_row, page_size)
+
+    table_columns, tooltip = setup_table_columns(dff)
+
+    dicts = dff.to_dicts()
+
+    download_disabled, download_text, download_title = get_button_properties(height)
+
+    return (
+        dicts,
+        table_columns,
+        tooltip,
+        data_timestamp + 1,
+        nb_rows,
+        download_disabled,
+        download_text,
+        download_title,
+        trigger_cleanup,
+    )
+```
+
+Notes on what changed vs the original at `src/utils/table.py:372-458`:
+
+- `track_search` now called explicitly at the top, on every invocation (not via `filter_table_data`).
+- `data is None` branch delegates the heavy work to the memoized helper.
+- `data is not None` branch is functionally identical to the original (pagination still happens after collect+post-process).
+- The post-processing (`cast`, `fill_null`, `add_links`, `add_resource_link`, `format_values`) is now done in BOTH branches before `nb_rows` calculation. In the cached branch this was already done inside `_load_filter_sort_postprocess`; in the uncached branch we keep doing it inline. This means `nb_rows` and `dff.select('uid').unique().height` operate on the post-processed frame in both branches, matching the original semantics.
+
+- [ ] **Step 4: Run all unit tests**
+
+Run: `uv run pytest tests/test_table.py -v`
+Expected: All PASS, including `test_prepare_table_data_paginates_without_recomputing`.
+
+- [ ] **Step 5: Run the full repo test suite to catch regressions**
+
+Run: `uv run pytest -v`
+Expected: All PASS. Selenium tests (`tests/test_main.py`) require Chrome/Chromium; if the executor lacks a browser, those tests will error/skip — note the failures and rerun in an environment with Chrome before declaring done.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add src/utils/table.py tests/test_table.py
+git commit -m "perf(tableau): memoize filter+sort+postprocess pipeline"
+```
+
+---
+
+## Task 6: Manual smoke test in the browser
+
+**Files:** none modified.
+
+Type checks and unit tests cannot validate that page navigation actually feels faster. This task is explicitly a hands-on verification.
+
+- [ ] **Step 1: Start the dev server**
+
+Run: `uv run run.py`
+Wait for `Dash is running on http://...`.
+
+- [ ] **Step 2: Open the Tableau page and warm the cache**
+
+1. Open `http://localhost:8050/tableau` (or whatever port the dev server prints).
+2. With no filter applied, wait for the first page to load fully. This is the cold-cache load (slow expected).
+3. Open the browser devtools Network panel.
+
+- [ ] **Step 3: Verify pagination is fast**
+
+1. Click "page 2" / "page 3" / "page 4" in the table footer in quick succession.
+2. Each navigation should return data in well under 1 second (in the original code each took several seconds).
+3. In the dev server logs, look for the line `Cache miss — recomputing for filter=...` from `_load_filter_sort_postprocess`. It should appear ONCE for the initial load and NOT appear again as you change pages.
+
+- [ ] **Step 4: Verify a new filter triggers exactly one cache miss**
+
+1. In the table, type a filter into one of the columns (e.g. `paris` in `acheteur_commune_nom`) and press Enter.
+2. The dev log should show ONE new `Cache miss — recomputing` line.
+3. Change page within the filtered view — no new cache miss line should appear.
+
+- [ ] **Step 5: Verify filter cleanup trigger still fires**
+
+1. Open `http://localhost:8050/acheteur?id=<some_acheteur_id>` (use any valid id from the dataset).
+2. Apply a filter on the embedded table.
+3. The clientside callback for filter cleanup (`src/assets/dash_clientside.js` `clean_filters`) should still rewrite the filter operators (e.g. `contains` → `icontains`). If it doesn't fire, the `trigger_cleanup` uuid is broken — investigate.
+
+- [ ] **Step 6: Verify download still works**
+
+1. On the Tableau page, click "Télécharger au format Excel" (the button must be enabled — apply a filter that brings the row count under 65,000).
+2. The downloaded XLSX must open and contain the filtered rows.
+
+- [ ] **Step 7: Stop the dev server**
+
+Ctrl-C.
+
+- [ ] **Step 8: If all checks pass, this completes the implementation**
+
+No commit — this task is verification only. Report results to the user.

docs/superpowers/specs/2026-03-21-observatoire-full-url-sharing-design.md

@@ -15,7 +15,7 @@ Flat query parameters with short, readable keys. Multi-value filters use repeate
 ## URL Parameter Mapping
 
 | Component ID                                       | URL key          | Type            | Default (omitted) |
-|---|---|---|---|
+| -------------------------------------------------- | ---------------- | --------------- | ----------------- |
 | `dashboard_year`                                   | `annee`          | single          | `None`            |
 | `dashboard_acheteur_id`                            | `acheteur_id`    | single          | `None`            |
 | `dashboard_acheteur_categorie`                     | `acheteur_cat`   | single          | `None`            |
@@ -35,6 +35,7 @@ Flat query parameters with short, readable keys. Multi-value filters use repeate
 | `dashboard_marche_considerationsEnvironnementales` | `env`            | multi           | `[]`/`None`       |
 
 Example URL:
+
 ```
 /observatoire?annee=2024&acheteur_id=12345678901234&acheteur_dept=75&acheteur_dept=13&montant_min=10000&innovant=oui
 ```
@@ -79,6 +80,7 @@ FILTER_PARAMS = [
 **Current:** Extracts only `acheteur_id` and `titulaire_id` from URL.
 
 **New:**
+
 - Iterates over `FILTER_PARAMS` to extract all values from `parse_qs`
 - For multi-value params: reads the full list from `parse_qs` (returns lists natively)
 - For number params (`montant_min`, `montant_max`): casts to `float`
@@ -102,12 +104,14 @@ Links generated by `add_links()` in `src/utils.py` (used on search results to li
 ### Fix broken test `test_010_observatoire_montant_filter`
 
 This test imports `_apply_filters` from `pages.observatoire`, which no longer exists (replaced by `prepare_dashboard_data` in `src/utils.py`). Fix:
+
 - Replace import with `from src.utils import prepare_dashboard_data`
 - Update the call to match `prepare_dashboard_data`'s signature: rename `marche_type` keyword to `type`, and add missing params `objet`, `code_cpv`, `techniques`, `marche_innovant`, `sous_traitance_declaree` (all as `None`)
 
 ### New test: multi-param URL round-trip
 
 Add a test that navigates to `/observatoire?annee=2024&acheteur_id=<test_id>&montant_min=10000` and verifies that:
+
 - `dashboard_year` dropdown shows "2024"
 - `dashboard_acheteur_id` input contains the test ID
 - `dashboard_montant_min` input contains "10000"

docs/superpowers/specs/2026-04-15-duckdb-migration-design.md

@@ -0,0 +1,196 @@
+# DuckDB migration — design spec
+
+**Date:** 2026-04-15
+**Branch:** dev
+**Status:** Approved, ready for planning
+
+## Goal
+
+Replace the global Polars dataframes that `src/utils.py` materializes at import time (`df` and the five derived frames, lines 891–913) with a DuckDB database on disk. The main table holds ~1.5M rows from `decp_prod.parquet`. Per-request queries pull only what each page needs, dramatically reducing steady-state RSS memory.
+
+Polars stays the primary API for small result sets and post-processing. DuckDB carries the heavy filtering, joining, and aggregation.
+
+## Approach summary
+
+- **Approach A — compatibility layer.** A new `src/db.py` module exposes a `query_marches(where_sql, params, columns, ...)` helper that runs SQL and returns a `pl.DataFrame`. Most existing `df.filter(pl.col(...) == x)` call sites translate mechanically to `query_marches("col = ?", (x,))`. The shape of downstream Polars code is unchanged.
+- **Two small helpers stay in memory.** `df_acheteurs` and `df_titulaires` (tens of thousands of rows, consumed by the autocomplete search on every keystroke) are kept as module-level Polars frames. They are populated from DuckDB at import time, not from Parquet.
+- **Four derived tables live in DuckDB**, built at startup alongside the main table: `acheteurs_marches`, `titulaires_marches`, `acheteurs_departement`, `titulaires_departement`.
+- **Connection model.** One read-only `duckdb.connect(..., read_only=True)` at module load, shared across the process. `conn.cursor()` per Dash callback for thread-safety. The read-write connection is short-lived and only used during the startup build phase.
+
+## Cache invalidation rule
+
+At startup, rebuild the DuckDB file if:
+
+1. **The DB file does not exist**, OR
+2. **`decp_prod.parquet.mtime > duckdb.mtime`**, **unless** `DEVELOPMENT=true` and `REBUILD_DUCKDB != true` — in which case the DB stays as-is (fast dev reloads).
+
+Production auto-rebuilds when the source Parquet is newer. Development keeps a stable DB across reloads unless the developer explicitly sets `REBUILD_DUCKDB=true` to force a rebuild.
+
+## Concurrency
+
+Multi-worker Gunicorn startup and crashed-mid-build scenarios are handled by a file lock, not by polling for the tmp file's existence:
+
+```python
+with open(DB_PATH.with_suffix(".duckdb.lock"), "w") as lock_fd:
+    fcntl.flock(lock_fd, fcntl.LOCK_EX)          # blocks if another worker is building
+    if should_rebuild(DB_PATH, PARQUET_PATH):
+        build_database(DB_PATH, PARQUET_PATH)
+conn = duckdb.connect(str(DB_PATH), read_only=True)
+```
+
+- Worker A acquires the lock, builds, atomically renames tmp → final, releases the lock.
+- Worker B blocks on `flock`, then re-checks `should_rebuild`, sees the fresh DB, skips building.
+- `fcntl.flock` is auto-released on process death, so a crash never deadlocks the next worker.
+- `build_database` unlinks any pre-existing tmp file before starting (safe because it holds the lock) — handles an abandoned tmp from a crashed previous build.
+
+## Build logic
+
+The build keeps **one source of truth** for transforms by reusing the existing Polars pipeline:
+
+```python
+def build_database(db_path, parquet_path):
+    tmp_path = db_path.with_suffix(".duckdb.tmp")
+    if tmp_path.exists():
+        tmp_path.unlink()
+    frame = get_decp_data()   # existing function in utils.py
+    with duckdb.connect(str(tmp_path)) as w:
+        w.register("frame", frame)
+        w.execute("CREATE TABLE decp AS SELECT * FROM frame")
+        w.execute("CREATE TABLE acheteurs_marches AS "
+                  "SELECT DISTINCT uid, objet, acheteur_id FROM decp "
+                  "ORDER BY acheteur_id")
+        w.execute("CREATE TABLE titulaires_marches AS "
+                  "SELECT DISTINCT uid, objet, titulaire_id FROM decp "
+                  "ORDER BY titulaire_id")
+        w.execute("CREATE TABLE acheteurs_departement AS "
+                  "SELECT DISTINCT acheteur_id, acheteur_nom, acheteur_departement_code "
+                  "FROM decp ORDER BY acheteur_nom")
+        w.execute("CREATE TABLE titulaires_departement AS "
+                  "SELECT DISTINCT titulaire_id, titulaire_nom, titulaire_departement_code "
+                  "FROM decp ORDER BY titulaire_nom")
+    os.replace(tmp_path, db_path)
+```
+
+Why Polars, not SQL, for the row-level transforms:
+
+- `booleans_to_strings` is not a simple cast — it replaces `true`/`false` with `"oui"`/`"non"` on every boolean column. Reimplementing in SQL risks drifting from the Polars version.
+- The null-name replacement (`acheteur_nom`, `titulaire_nom` → `"[Identifiant non reconnu dans la base INSEE]"`) is also easier to keep identical in Polars.
+- `w.register("frame", frame)` is zero-copy. The memory spike is one-time during build and released when the write connection closes.
+
+`os.replace` is atomic on POSIX — the read-only connection that opens next always sees a complete DB.
+
+## Module layout
+
+### New: `src/db.py`
+
+```python
+conn: duckdb.DuckDBPyConnection       # read-only, module-level
+schema: pl.Schema                     # from conn.execute("SELECT * FROM decp LIMIT 0").pl().schema
+
+def get_cursor() -> duckdb.DuckDBPyConnection: ...
+def query_marches(where_sql: str = "TRUE",
+                  params: tuple = (),
+                  columns: list[str] | None = None,
+                  order_by: str | None = None,
+                  limit: int | None = None) -> pl.DataFrame: ...
+def should_rebuild(db_path: Path, parquet_path: Path) -> bool: ...
+def build_database(db_path: Path, parquet_path: Path) -> None: ...
+```
+
+Only imports: `polars`, `duckdb`, `os`, `fcntl`, `pathlib`, `logging`. No app modules — prevents circular imports.
+
+### Changes to `src/utils.py`
+
+- `df: pl.DataFrame = get_decp_data()` — **removed** (after migration).
+- `df_acheteurs`, `df_titulaires` — **kept as Polars globals**, populated via DuckDB at import time. The query mirrors today's `get_org_data(df, org_type)`: select all columns whose name starts with `acheteur_` (or `titulaire_`) except the `_latitude` / `_longitude` pair, plus `COUNT(*) AS "Marchés"`, grouped by the same set. Implementation can either:
+
+  - enumerate the columns by filtering `schema.names()` at import time and build the `SELECT` / `GROUP BY` strings, or
+  - call `get_org_data()` once against a small Polars frame returned by `SELECT <org_ cols> FROM decp`.
+
+  Feeds `search_org` unchanged.
+
+- `df_acheteurs_marches`, `df_titulaires_marches`, `df_acheteurs_departement`, `df_titulaires_departement` — **removed** as Python globals. Call sites query the corresponding DuckDB tables.
+- `schema` — imported from `src/db.py` (stays a `pl.Schema` — so `schema.names()` and dtype lookups both work, no call-site changes beyond `acheteur.py:303`).
+- `columns` — replaced with `schema.names()`.
+- `get_decp_data()` — **kept** (used by `build_database`).
+- `get_org_data()` — can be removed once `df_acheteurs` / `df_titulaires` are populated from DuckDB directly.
+
+### Call-site translations
+
+| Before (Polars global)                                       | After                                                                             |
+| ------------------------------------------------------------ | --------------------------------------------------------------------------------- |
+| `df.filter(pl.col("acheteur_id") == aid)`                    | `query_marches("acheteur_id = ?", (aid,))`                                        |
+| `df.filter(pl.col("uid") == uid).row(0, named=True)`         | `query_marches("uid = ?", (uid,)).row(0, named=True)`                             |
+| `df.select("uid","objet","acheteur_id").filter(...)`         | `query_marches("...", (...), columns=["uid","objet","acheteur_id"])`              |
+| `df.columns`                                                 | `schema.names()`                                                                  |
+| `df_acheteurs_marches.filter(...)`                           | `get_cursor().execute("SELECT ... FROM acheteurs_marches WHERE ...", [...]).pl()` |
+| `pl.DataFrame(schema=df.collect_schema())` (acheteur.py:303) | `pl.DataFrame(schema=schema)`                                                     |
+
+Heavy dashboard aggregations (observatoire, tableau full-scan) use raw SQL via `get_cursor().execute(...).pl()` rather than the helper.
+
+## Configuration
+
+- **`DATA_FILE_PARQUET_PATH`** — unchanged.
+- **DuckDB file location** — computed: `Path(DATA_FILE_PARQUET_PATH).parent / "decp.duckdb"`. No new env var.
+- **`REBUILD_DUCKDB`** — new, optional, default `false`. In development, setting this to `true` forces a rebuild when the parquet is newer.
+- **`DEVELOPMENT`** — unchanged; now also gates the auto-rebuild behavior per the rule above.
+
+## Testing
+
+- `tests/conftest.py` (or a startup hook in `src/db.py`) ensures the test run builds the DuckDB in a temp directory derived from the parquet path — `tests/test.parquet` → `tests/decp.duckdb`. This file is added to `.gitignore`.
+- Tests already set `DEVELOPMENT=true`; they must also set `REBUILD_DUCKDB=true` on cold test runs to force a fresh build from the test parquet.
+- The existing Selenium suite exercises every page and is the primary acceptance signal.
+
+## Migration order
+
+Incremental — `df` global coexists with `src/db.py` until every page is migrated.
+
+1. **Add `src/db.py`** (build, lock, `query_marches`, `schema`). `df` global unchanged.
+2. **Migrate `marche.py`** — single-row lookup by `uid`, one call site.
+3. **Migrate `acheteur.py`, `titulaire.py`** — filter by id.
+4. **Migrate `arbre/departement.py`, `arbre/liste_marches_org.py`** — use the new derived DuckDB tables.
+5. **Migrate `tableau.py`** — may need raw SQL.
+6. **Migrate `observatoire.py`** — heaviest aggregations, most likely raw SQL.
+7. **Migrate `figures.py`** — uses `df` in chart generation.
+8. **Remove** `df`, `df_*_marches`, `df_*_departement` globals, `get_org_data()`, and the `df = get_decp_data()` call from `utils.py`. Move `schema` / `columns` exports to `src/db.py`.
+
+### Verification gates
+
+- `uv run pytest` green after every page migration.
+- Manual smoke test via `uv run run.py` of the migrated page before proceeding.
+- RSS memory measurement (`ps -o rss`) of a cold `gunicorn app:server` with the prod parquet, before and after, to confirm the memory reduction.
+
+## Out of scope
+
+- Changes to `src/cache.py` (flask-caching stays).
+- The in-progress observatoire-localstorage-filters work on `dev`.
+- Schema changes to the parquet.
+- SQL views beyond the four derived tables.
+- Multi-database or replication setups.
+
+## Risks and mitigations
+
+| Risk                                                                          | Mitigation                                                                                                                       |
+| ----------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
+| `booleans_to_strings` reimplemented in SQL and drifts from Polars version     | Transforms stay in Polars via `w.register("frame", frame)`. One source of truth.                                                 |
+| Two Gunicorn workers rebuild concurrently                                     | `fcntl.flock` serializes the build; second worker re-checks and skips.                                                           |
+| Crashed build leaves stale `.tmp` file                                        | Build unlinks any pre-existing tmp before starting (safe under lock).                                                            |
+| `schema` shape change breaks `acheteur.py:303`                                | `schema` stays a `pl.Schema` object, not a list. One call site (`collect_schema()` → module `schema`) updated.                   |
+| Test runs inherit a stale DuckDB from a previous run with a different parquet | Tests force `REBUILD_DUCKDB=true` on cold runs; test DB added to `.gitignore`.                                                   |
+| Read-only connection opened before build finishes in another worker           | Lock held across build + rename; read-only `connect` happens after lock release. Atomic `os.replace` guarantees a complete file. |
+
+## Outcome
+
+### Memory impact
+
+Memory measurement against the production parquet (`decp_prod.parquet`, ~1.5M rows) requires a running gunicorn process with access to the production data file. The measurement was deferred to the post-merge smoke test on the staging server (test.decp.info).
+
+**Expected reduction:** The removed globals (`df`, `df_acheteurs_departement`, `df_titulaires_departement`, `df_acheteurs_marches`, `df_titulaires_marches`) previously materialised the full 1.5M-row Parquet in memory as multiple Polars frames. At ~300 bytes/row × 5 frames, steady-state RSS reduction is estimated at **1–2 GB per worker**. The retained `df_acheteurs` and `df_titulaires` (autocomplete search) represent only the distinct-organisation subset (~tens of thousands of rows) and are negligible.
+
+**What remains in memory:**
+
+- `df_acheteurs` — distinct acheteurs with Marchés count (populated from DuckDB at startup)
+- `df_titulaires` — same for titulaires
+- DuckDB's own page cache (disk-backed, grows under load, evicted by OS)
+
+All per-request data is fetched from DuckDB and discarded after the callback returns.

pyproject.toml

@@ -1,11 +1,9 @@
 [project]
 name = "decp.info"
 description = "Interface d'exploration et d'analyse des marchés publics français."
-version = "2.7.1"
+version = "2.7.3"
 requires-python = ">= 3.10"
-authors = [
+authors = [{ name = "Colin Maudry", email = "colin@colmo.tech" }]
-    { name = "Colin Maudry", email = "colin@colmo.tech" }
-]
 dependencies = [
   "dash==3.4.0",
   "dash[compress]",
@@ -19,10 +17,13 @@ dependencies = [
   "pandas",                    # utilisé pour la création de certains graphiques
   "unidecode",
   "dash-leaflet",
-    "dash-extensions"
+  "dash-extensions",
+  "duckdb",
+  "flask-caching",
+  "pyarrow>=23.0.1",
 ]
 
-[project.optional-dependencies]
+[dependency-groups]
 dev = [
   "pytest",
   "pytest-env",
@@ -30,19 +31,15 @@ dev = [
   "selenium",
   "webdriver-manager",
   "dash[testing]",
-    "fastexcel"
+  "fastexcel",
 ]
 
 [tool.pytest.ini_options]
-pythonpath = [
+pythonpath = ["src"]
-  "src"
+testpaths = ["tests"]
-]
-testpaths = [
-    "tests"
-]
 env = [
   "DATA_FILE_PARQUET_PATH=tests/test.parquet",
   "DEVELOPMENT=true",
-    "DATA_SCHEMA_PATH=/home/colin/git/decp-processing/dist/schema.json"
+  "DATA_SCHEMA_PATH=/home/colin/git/decp-processing/dist/schema.json",
 ]
 addopts = "-p no:warnings"

src/app.py

@@ -1,5 +1,5 @@
-import logging
 import os
+from shutil import rmtree
 
 import dash_bootstrap_components as dbc
 import tomllib
@@ -7,15 +7,15 @@ from dash import Dash, Input, Output, State, dcc, html, page_container, page_reg
 from dotenv import load_dotenv
 from flask import Response
 
+from src.utils import DEVELOPMENT
+from src.utils.cache import cache
+
 load_dotenv()
 
 # if os.getenv("PYTEST_CURRENT_TEST"):
 #     os.environ["DATA_FILE_PARQUET_PATH"]
 
-
+META_TAGS = [
-development = os.getenv("DEVELOPMENT").lower() == "true"
-
-meta_tags = [
     {"name": "viewport", "content": "width=device-width, initial-scale=1"},
     {
         "name": "keywords",
@@ -23,20 +23,32 @@ meta_tags = [
     },
 ]
 
-if development:
+if DEVELOPMENT:
-    meta_tags.append({"name": "robots", "content": "noindex"})
+    META_TAGS.append({"name": "robots", "content": "noindex"})
 
 app: Dash = Dash(
     title="decp.info",
     use_pages=True,
     compress=True,
-    meta_tags=meta_tags,
+    meta_tags=META_TAGS,
 )
 
-# COSMO (belle font, blue),
+cache_dir = os.getenv("CACHE_DIR", "/tmp/decp-cache")
-# UNITED (rouge, ubuntu font),
+
-# LUMEN (gros séparateur, blue clair),
+if os.path.exists(cache_dir):
-# SIMPLEX (rouge, séparateur)
+    rmtree(cache_dir)
+
+cache.init_app(
+    app.server,
+    config={
+        "CACHE_TYPE": "FileSystemCache",
+        "CACHE_DIR": cache_dir,
+        "CACHE_DEFAULT_TIMEOUT": int(
+            os.getenv("CACHE_DEFAULT_TIMEOUT", 3600 * 24)
+        ),  # 24h par défaut
+        "CACHE_THRESHOLD": 300,
+    },
+)
 
 
 # robots.txt
@@ -67,13 +79,6 @@ def sitemap():
     return Response(xml, mimetype="text/xml")
 
 
-logger = logging.getLogger("decp.info")
-logging.basicConfig(
-    format="%(asctime)s %(levelname)-8s %(message)s",
-    level=logging.INFO,
-    datefmt="%Y-%m-%d %H:%M:%S",
-)
-
 with open("./pyproject.toml", "rb") as f:
     pyproject = tomllib.load(f)
     version = "v" + pyproject["project"]["version"]

src/assets/css/style.css

@@ -197,9 +197,9 @@ p.version > a {
 .table-menu {
   font-size: 16px;
   margin: 12px 0 12px 0;
-  height: 50px;
   display: flex;
   align-items: center;
+  flex-wrap: wrap;
 }
 
 .table-menu > * {

src/db.py

@@ -0,0 +1,155 @@
+import fcntl
+import os
+from pathlib import Path
+from time import sleep
+
+import duckdb
+import polars as pl
+import polars.selectors as cs
+from polars.exceptions import ComputeError
+
+from src.utils import logger
+
+
+def should_rebuild(db_path: Path, parquet_path: Path) -> bool:
+    db_path = Path(db_path)
+    parquet_path = Path(parquet_path)
+    if not db_path.exists():
+        return True
+    dev = os.getenv("DEVELOPMENT", "False").lower() == "true"
+    force = os.getenv("REBUILD_DUCKDB", "False").lower() == "true"
+    if dev and not force:
+        return False
+    return parquet_path.stat().st_mtime > db_path.stat().st_mtime
+
+
+def _load_source_frame(parquet_path: Path) -> pl.DataFrame:
+    """Read the source parquet and apply the row-level transforms.
+
+    Kept here (not in utils.py) so src.db has no dependency on utils.
+    Mirrors the behavior previously in utils.get_decp_data().
+    """
+    try:
+        lff: pl.LazyFrame = pl.scan_parquet(str(parquet_path))
+    except ComputeError:
+        logger.info("Lecture du parquet échouée, nouvelle tentative dans 10s...")
+        sleep(10)
+        lff = pl.scan_parquet(str(parquet_path))
+
+    lff = lff.sort(by=["dateNotification", "uid"], descending=True, nulls_last=True)
+    lff = lff.filter(pl.col("donneesActuelles")).drop("donneesActuelles")
+
+    # booleans_to_strings: true → "oui", false → "non"
+    lff = lff.with_columns(
+        pl.col(cs.Boolean)
+        .cast(pl.String)
+        .str.replace("true", "oui")
+        .str.replace("false", "non")
+    )
+
+    for col in ["acheteur_nom", "titulaire_nom"]:
+        lff = lff.with_columns(
+            pl.when(pl.col(col).is_null())
+            .then(pl.lit("[Identifiant non reconnu dans la base INSEE]"))
+            .otherwise(pl.col(col))
+            .name.keep()
+        )
+
+    return lff.collect()
+
+
+def build_database(db_path: Path, parquet_path: Path) -> None:
+    """Build the DuckDB database atomically under an exclusive lock.
+
+    Caller MUST hold the fcntl.flock on the .lock file.
+    """
+    db_path = Path(db_path)
+    parquet_path = Path(parquet_path)
+    tmp_path = db_path.with_suffix(".duckdb.tmp")
+    staging_parquet = db_path.with_suffix(".staging.parquet")
+    if tmp_path.exists():
+        tmp_path.unlink()
+
+    logger.info(f"Construction de la base DuckDB à partir de {parquet_path}...")
+    frame = _load_source_frame(parquet_path)
+
+    # Write transformed frame as parquet so DuckDB can read it natively
+    # (avoids pyarrow dependency for the Polars→DuckDB handoff)
+    frame.write_parquet(str(staging_parquet))
+    try:
+        with duckdb.connect(str(tmp_path)) as w:
+            w.execute(
+                f"CREATE TABLE decp AS SELECT * FROM read_parquet('{staging_parquet}')"
+            )
+            w.execute(
+                "CREATE TABLE acheteurs_marches AS "
+                "SELECT DISTINCT uid, objet, acheteur_id FROM decp "
+                "ORDER BY acheteur_id"
+            )
+            w.execute(
+                "CREATE TABLE titulaires_marches AS "
+                "SELECT DISTINCT uid, objet, titulaire_id FROM decp "
+                "ORDER BY titulaire_id"
+            )
+            w.execute(
+                "CREATE TABLE acheteurs_departement AS "
+                "SELECT DISTINCT acheteur_id, acheteur_nom, acheteur_departement_code "
+                "FROM decp ORDER BY acheteur_nom"
+            )
+            w.execute(
+                "CREATE TABLE titulaires_departement AS "
+                "SELECT DISTINCT titulaire_id, titulaire_nom, titulaire_departement_code "
+                "FROM decp ORDER BY titulaire_nom"
+            )
+    finally:
+        if staging_parquet.exists():
+            staging_parquet.unlink()
+
+    os.replace(tmp_path, db_path)
+    logger.info(f"Base DuckDB construite : {db_path}")
+
+
+def _ensure_database() -> Path:
+    db_path = Path("./decp.duckdb")
+    parquet_path = Path(os.getenv("DATA_FILE_PARQUET_PATH"))
+    lock_path = db_path.with_suffix(".duckdb.lock")
+
+    with open(lock_path, "w") as lock_fd:
+        fcntl.flock(lock_fd, fcntl.LOCK_EX)
+        if should_rebuild(db_path, parquet_path):
+            build_database(db_path, parquet_path)
+        else:
+            logger.debug("Base de données déjà disponible et à jour.")
+    return db_path
+
+
+DB_PATH = _ensure_database()
+conn: duckdb.DuckDBPyConnection = duckdb.connect(str(DB_PATH), read_only=True)
+schema: pl.Schema = conn.execute("SELECT * FROM decp LIMIT 0").pl().schema
+
+
+def get_cursor() -> duckdb.DuckDBPyConnection:
+    """Return a per-request cursor that shares the process-wide connection."""
+    return conn.cursor()
+
+
+def query_marches(
+    where_sql: str = "TRUE",
+    params: tuple = (),
+    columns: list[str] | None = None,
+    order_by: str | None = None,
+    limit: int | None = None,
+) -> pl.DataFrame:
+    """Run a parameterized SELECT against the decp table and return Polars.
+
+    `where_sql` and `order_by` are trusted SQL fragments (callers are internal
+    code, never user input). `params` values are passed through DuckDB's
+    parameter binding.
+    """
+    cols = ", ".join(columns) if columns else "*"
+    sql = f"SELECT {cols} FROM decp WHERE {where_sql}"
+    if order_by:
+        sql += f" ORDER BY {order_by}"
+    if limit is not None:
+        sql += f" LIMIT {int(limit)}"
+    return get_cursor().execute(sql, list(params)).pl()

src/figures.py

@@ -12,14 +12,9 @@ import polars as pl
 from dash import dash_table, dcc, html
 from dash_extensions.javascript import Namespace
 
-from src.utils import (
+from src.db import schema
-    add_links,
+from src.utils.data import DATA_SCHEMA, DEPARTEMENTS_GEOJSON
-    data_schema,
+from src.utils.table import add_links, format_number, setup_table_columns
-    departements_geojson,
-    df,
-    format_number,
-    setup_table_columns,
-)
 
 
 def get_yearly_statistics(statistics, today_str) -> html.Div:
@@ -260,8 +255,8 @@ class DataTable(dash_table.DataTable):
 
         style_cell_common = {"fontFamily": "Inter", "fontSize": "16px"}
 
-        for key in data_schema.keys():
+        for key in DATA_SCHEMA.keys():
-            field = data_schema[key]
+            field = DATA_SCHEMA[key]
             if field["type"] in ["number", "integer"]:
                 rule = {
                     "if": {"column_id": field["name"]},
@@ -365,7 +360,7 @@ def get_duplicate_matrix() -> dcc.Graph:
     return dcc.Graph(figure=fig)
 
 
-def get_geographic_maps(dff: pl.DataFrame) -> list | None:
+def get_geographic_maps(dff: pl.DataFrame) -> list[dbc.Col] | list:
     """
     Génère les cartes géographiques pour l'hexagone et les DOM-TOM.
     """
@@ -409,7 +404,7 @@ def get_geographic_maps(dff: pl.DataFrame) -> list | None:
         },
     }
 
-    def make_map_data(region_code: str) -> tuple[list, str or None]:
+    def make_map_data(region_code: str) -> tuple[list, str | None]:
         lff: pl.LazyFrame = dff.lazy()
         if region_code == "Hexagone":
             lff = lff.filter(
@@ -514,7 +509,7 @@ def make_chloropleth_map(region: dict) -> dcc.Graph:
 
     fig = px.choropleth(
         df_map,
-        geojson=departements_geojson,
+        geojson=DEPARTEMENTS_GEOJSON,
         locations="Département",
         color="uid",
         color_continuous_scale="Reds",
@@ -722,7 +717,7 @@ def make_donut(
     nulls="?",
     potentially_many_names: bool = False,
 ):
-    title = data_schema[names_col]["title"]
+    title = DATA_SCHEMA[names_col]["title"]
     lff = lff.rename({names_col: title})
     lff = lff.select("uid", title)
 
@@ -771,16 +766,16 @@ def make_column_picker(page: str):
     table_columns = [
         {
             "id": col,
-            "name": data_schema[col]["title"],
+            "name": DATA_SCHEMA[col]["title"],
-            "description": data_schema[col]["description"],
+            "description": DATA_SCHEMA[col]["description"],
         }
-        for col in df.columns
+        for col in schema.names()
     ]
     for column in table_columns:
         new_column = {
             "id": column["id"],
             "name": column["name"],
-            "description": data_schema[column["id"]]["description"],
+            "description": DATA_SCHEMA[column["id"]]["description"],
         }
         table_data.append(new_column)
 

src/pages/a-propos.py

@@ -3,9 +3,9 @@ import os
 from dash import dcc, html, register_page
 
 from src.figures import get_sources_tables
-from src.utils import meta_content
+from src.utils.seo import META_CONTENT
 
-name = "À propos"
+NAME = "À propos"
 
 register_page(
     __name__,
@@ -13,14 +13,14 @@ register_page(
     title="À propos | decp.info",
     name="À propos",
     description="En savoir plus sur decp.info, l'outil d'exploration des données essentielles de la commande publique.",
-    image_url=meta_content["image_url"],
+    image_url=META_CONTENT["image_url"],
     order=5,
 )
 
 layout = html.Div(
     className="container",
     children=[
-        html.H2(name),
+        html.H2(NAME),
         html.Div(
             className="a-propos-container",
             children=[

src/pages/acheteur.py

@@ -15,6 +15,7 @@ from dash import (
     register_page,
 )
 
+from src.db import query_marches, schema
 from src.figures import (
     DataTable,
     get_distance_histogram,
@@ -23,24 +24,22 @@ from src.figures import (
     make_column_picker,
     point_on_map,
 )
-from src.utils import (
+from src.utils.data import DF_ACHETEURS, get_annuaire_data, get_departement_region
-    columns,
+from src.utils.frontend import get_button_properties
-    df,
+from src.utils.seo import META_CONTENT
-    df_acheteurs,
+from src.utils.table import (
+    COLUMNS,
     filter_table_data,
     format_number,
-    get_annuaire_data,
-    get_button_properties,
     get_default_hidden_columns,
-    get_departement_region,
-    meta_content,
     prepare_table_data,
     sort_table_data,
 )
+from src.utils.tracking import track_search
 
 
-def get_title(acheteur_id: str = None) -> str:
+def get_title(acheteur_id: str | None = None) -> str:
-    acheteur_nom = df_acheteurs.filter(pl.col("acheteur_id") == acheteur_id).select(
+    acheteur_nom = DF_ACHETEURS.filter(pl.col("acheteur_id") == acheteur_id).select(
         "acheteur_nom"
     )
     if acheteur_nom.height > 0:
@@ -54,11 +53,11 @@ register_page(
     title=get_title,
     name="Acheteur",
     description="Consultez les marchés publics attribués par cet acheteur.",
-    image_url=meta_content["image_url"],
+    image_url=META_CONTENT["image_url"],
     order=5,
 )
 
-datatable = html.Div(
+DATATABLE = html.Div(
     className="marches_table",
     children=DataTable(
         dtid="acheteur_datatable",
@@ -70,7 +69,7 @@ datatable = html.Div(
         sort_action="custom",
         page_size=10,
         hidden_columns=[],
-        columns=[{"id": col, "name": col} for col in df.columns],
+        columns=[{"id": col, "name": col} for col in schema.names()],
     ),
 )
 
@@ -229,7 +228,7 @@ layout = [
                         scrollable=True,
                         size="xl",
                     ),
-                    datatable,
+                    DATATABLE,
                 ],
             ),
         ],
@@ -300,7 +299,7 @@ def update_acheteur_infos(url):
 def update_acheteur_stats(data):
     dff = pl.DataFrame(data, strict=False, infer_schema_length=5000)
     if dff.height == 0:
-        dff = pl.DataFrame(schema=df.collect_schema())
+        dff = pl.DataFrame(schema=schema)
     df_marches = dff.unique("id")
     nb_marches = format_number(df_marches.height)
     # somme_marches = format_number(int(df_marches.select(pl.sum("montant")).item()))
@@ -326,17 +325,15 @@ def update_acheteur_stats(data):
     Input(component_id="acheteur_url", component_property="pathname"),
     Input(component_id="acheteur_year", component_property="value"),
 )
-def get_acheteur_marches_data(url, acheteur_year: str) -> tuple:
+def get_acheteur_marches_data(url, ach_year: str) -> tuple:
     acheteur_siret = url.split("/")[-1]
-    lff = df.lazy()
+    lff = query_marches("acheteur_id = ?", (acheteur_siret,)).lazy()
-    lff = lff.filter(pl.col("acheteur_id") == acheteur_siret)
+    if ach_year and ach_year != "Toutes les années":
-    if acheteur_year and acheteur_year != "Toutes les années":
+        ach_year = int(ach_year)
-        acheteur_year = int(acheteur_year)
+        lff = lff.filter(pl.col("dateNotification").dt.year() == ach_year)
-        lff = lff.filter(pl.col("dateNotification").dt.year() == acheteur_year)
     lff = lff.sort(["dateNotification", "uid"], descending=True, nulls_last=True)
     dff: pl.DataFrame = lff.collect(engine="streaming")
     download_disabled, download_text, download_title = get_button_properties(dff.height)
-
     data = dff.to_dicts()
     return data, download_disabled, download_text, download_title
 
@@ -412,7 +409,12 @@ def download_acheteur_data(
     prevent_initial_call=True,
 )
 def download_filtered_acheteur_data(
-    data, n_clicks, acheteur_nom, filter_query, sort_by, hidden_columns: list = None
+    data,
+    n_clicks,
+    acheteur_nom,
+    filter_query,
+    sort_by,
+    hidden_columns: list | None = None,
 ):
     lff: pl.LazyFrame = pl.LazyFrame(
         data
@@ -423,7 +425,8 @@ def download_filtered_acheteur_data(
         lff = lff.drop(hidden_columns)
 
     if filter_query:
-        lff = filter_table_data(lff, filter_query, "ach download")
+        track_search(filter_query, "ach download")
+        lff = filter_table_data(lff, filter_query)
 
     if len(sort_by) > 0:
         lff = sort_table_data(lff, sort_by)
@@ -457,22 +460,23 @@ clientside_callback(
 )
 def update_hidden_columns_from_checkboxes(selected_columns):
     if selected_columns:
-        selected_columns = [columns[i] for i in selected_columns]
+        selected_columns = [COLUMNS[i] for i in selected_columns]
-        hidden_columns = [col for col in columns if col not in selected_columns]
+        hidden_columns = [col for col in COLUMNS if col not in selected_columns]
         return hidden_columns
     else:
         return []
 
 
 @callback(
-    Output("acheteur_datatable", "hidden_columns", allow_duplicate=True),
+    Output("acheteur_datatable", "hidden_columns"),
     Input(
         "acheteur-hidden-columns",
         "data",
     ),
-    prevent_initial_call=True,
 )
 def store_hidden_columns(hidden_columns):
+    if hidden_columns is None:
+        hidden_columns = get_default_hidden_columns("acheteur")
     return hidden_columns
 
 
@@ -485,7 +489,7 @@ def update_checkboxes_from_hidden_columns(hidden_cols, current_checkboxes):
     hidden_cols = hidden_cols or get_default_hidden_columns("acheteur")
 
     # Show all columns that are NOT hidden
-    visible_cols = [columns.index(col) for col in columns if col not in hidden_cols]
+    visible_cols = [COLUMNS.index(col) for col in COLUMNS if col not in hidden_cols]
     return visible_cols
 
 

src/pages/arbre/departement.py

@@ -1,17 +1,17 @@
-import polars as pl
 from dash import Input, Output, callback, dcc, html, register_page
 
-from src.utils import departements, df_acheteurs_departement, df_titulaires_departement
+from src.db import get_cursor
+from src.utils.data import DEPARTEMENTS
 
-name = "Département"
+NAME = "Département"
 
 
 def get_title(code):
-    return f"Marchés publics de {departements[code]['departement']} | decp.info"
+    return f"Marchés publics de {DEPARTEMENTS[code]['departement']} | decp.info"
 
 
 def get_description(code):
-    return f"Marchés publics passés dans le département {departements[code]['departement']} | decp.info"
+    return f"Marchés publics passés dans le département {DEPARTEMENTS[code]['departement']} | decp.info"
 
 
 register_page(
@@ -20,7 +20,7 @@ register_page(
     title=get_title,
     description=get_description,
     order=50,
-    name=name,
+    name=NAME,
 )
 
 layout = html.Div(
@@ -39,29 +39,42 @@ def departement_marches(url):
     departement = url.split("/")[-1]
 
     def make_link_list(org_type) -> list:
-        link_list = []
+        table = (
-        if org_type == "acheteur":
+            "acheteurs_departement"
-            df = df_acheteurs_departement
+            if org_type == "acheteur"
-        elif org_type == "titulaire":
+            else "titulaires_departement"
-            df = df_titulaires_departement
+            if org_type == "titulaire"
-        else:
+            else None
+        )
+        if table is None:
             raise ValueError
+        col_prefix = org_type
+        rows = (
+            get_cursor()
+            .execute(
+                f"SELECT {col_prefix}_id, {col_prefix}_nom "
+                f"FROM {table} "
+                f"WHERE {col_prefix}_departement_code = ? "
+                f"ORDER BY {col_prefix}_nom",
+                [departement],
+            )
+            .fetchall()
+        )
 
-        df = df.filter(pl.col(f"{org_type}_departement_code") == departement)
+        link_list = []
-
+        for org_id, org_nom in rows:
-        for row in df.iter_rows(named=True):
             li = html.Li(
                 [
                     dcc.Link(
-                        row[f"{org_type}_nom"],
+                        org_nom,
-                        href=url + f"/{org_type}/{row[f'{org_type}_id']}",
+                        href=url + f"/{org_type}/{org_id}",
-                        title=f"Marchés publics de {row[f'{org_type}_nom']}",
+                        title=f"Marchés publics de {org_nom}",
                     ),
                     " ",
                     dcc.Link(
                         "(page dédiée)",
-                        href=f"/{org_type}s/{row[f'{org_type}_id']}",
+                        href=f"/{org_type}s/{org_id}",
-                        title=f"Page dédiée aux marchés publics de {row[f'{org_type}_nom']}",
+                        title=f"Page dédiée aux marchés publics de {org_nom}",
                     ),
                 ]
             )

src/pages/arbre/departements.py

@@ -1,8 +1,8 @@
 from dash import dcc, html, register_page
 
-from src.utils import departements
+from src.utils.data import DEPARTEMENTS
 
-name = "Départements"
+NAME = "Départements"
 
 register_page(
     __name__,
@@ -18,7 +18,7 @@ layout = html.Div(
         html.Ul(
             [
                 html.Li(dcc.Link(d["departement"], href=f"/departements/{k}"))
-                for k, d in departements.items()
+                for k, d in DEPARTEMENTS.items()
             ]
         ),
     ]

src/pages/arbre/liste_marches_org.py

@@ -1,22 +1,18 @@
 import polars as pl
 from dash import Input, Output, callback, dcc, html, register_page
 
-from src.utils import (
+from src.db import get_cursor
-    df_acheteurs,
+from src.utils.data import DF_ACHETEURS, DF_TITULAIRES
-    df_acheteurs_marches,
-    df_titulaires,
-    df_titulaires_marches,
-)
 
-name = "Liste des marchés publics"
+NAME = "Liste des marchés publics"
 
 
 def make_org_nom_verbe(org_type, org_id) -> tuple:
     if org_type == "titulaire":
-        df = df_titulaires
+        df = DF_TITULAIRES
         verbe = "remportés"
     elif org_type == "acheteur":
-        df = df_acheteurs
+        df = DF_ACHETEURS
         verbe = "attribués"
     else:
         raise ValueError
@@ -48,7 +44,7 @@ register_page(
     title=get_title,
     description=get_description,
     order=40,
-    name=name,
+    name=NAME,
 )
 
 layout = html.Div(
@@ -68,28 +64,34 @@ def liste_marches(url):
     org_id = url.split("/")[-1]
 
     def make_link_list() -> list:
-        link_list = []
+        table = (
-        if org_type == "acheteur":
+            "acheteurs_marches"
-            df = df_acheteurs_marches
+            if org_type == "acheteur"
-        elif org_type == "titulaire":
+            else "titulaires_marches"
-            df = df_titulaires_marches
+            if org_type == "titulaire"
-        else:
+            else None
+        )
+        if table is None:
             raise ValueError
+        rows = (
+            get_cursor()
+            .execute(
+                f"SELECT uid, objet FROM {table} WHERE {org_type}_id = ?",
+                [org_id],
+            )
+            .fetchall()
+        )
 
-        df = df.filter(pl.col(f"{org_type}_id") == org_id)
+        return [
-
+            html.Li(
-        for row in df.iter_rows(named=True):
-            li = html.Li(
-                [
                 dcc.Link(
-                        row["objet"],
+                    objet,
-                        href=f"/marches/{row['uid']}",
+                    href=f"/marches/{uid}",
-                        title=f"Marchés public attribué : {row['objet']}",
+                    title=f"Marchés public attribué : {objet}",
                 )
+            )
+            for uid, objet in rows
         ]
-            )
-            link_list.append(li)
-        return link_list
 
     nom, verbe = make_org_nom_verbe(org_type, org_id)
 

src/pages/marche.py

@@ -2,18 +2,13 @@ import json
 from datetime import datetime
 
 import dash_bootstrap_components as dbc
-import polars as pl
 from dash import Input, Output, callback, dcc, html, register_page
 from polars import selectors as cs
 
-from src.utils import (
+from src.db import query_marches
-    data_schema,
+from src.utils.data import DATA_SCHEMA
-    df,
+from src.utils.seo import META_CONTENT, make_org_jsonld
-    format_values,
+from src.utils.table import format_values, unformat_montant
-    make_org_jsonld,
-    meta_content,
-    unformat_montant,
-)
 
 
 def get_title(uid: str = None) -> str:
@@ -26,7 +21,7 @@ register_page(
     title=get_title,
     name="Marché",
     description="Consultez les détails de ce marché public : montant, acheteur, titulaires, modifications, etc.",
-    image_url=meta_content["image_url"],
+    image_url=META_CONTENT["image_url"],
     order=7,
 )
 
@@ -88,19 +83,17 @@ layout = [
 def get_marche_data(url) -> tuple[dict, list]:
     marche_uid = url.split("/")[-1]
 
-    # Récupération des données du marché à partir du df global
+    # Filtre SQL côté DuckDB, puis Polars pour le post-traitement
+    dff_marche = query_marches("uid = ?", (marche_uid,))
+    if dff_marche.height == 0:
+        return {}, []
 
-    lff = df.lazy()
+    lff = dff_marche.lazy()
-    lff = lff.filter(pl.col("uid") == pl.lit(marche_uid))
-
-    # Données des titulaires du marché
     dff_titulaires = lff.select(cs.starts_with("titulaire")).collect(engine="streaming")
+    dff_marche_unique = lff.unique("uid").collect(engine="streaming")
+    dff_marche_unique = format_values(dff_marche_unique)
 
-    # Données du marché
+    return dff_marche_unique.to_dicts()[0], dff_titulaires.to_dicts()
-    dff_marche = lff.unique("uid").collect(engine="streaming")
-    dff_marche = format_values(dff_marche)
-
-    return dff_marche.to_dicts()[0], dff_titulaires.to_dicts()
 
 
 @callback(
@@ -113,7 +106,7 @@ def get_marche_data(url) -> tuple[dict, list]:
 )
 def update_marche_info(marche, titulaires):
     def make_parameter(col, bold=True):
-        column_object = data_schema.get(col)
+        column_object = DATA_SCHEMA.get(col)
         column_name = column_object.get("title") if column_object else col
 
         if marche[col]:

src/pages/observatoire.py

@@ -16,6 +16,7 @@ from dash import (
     register_page,
 )
 
+from src.db import query_marches, schema
 from src.figures import (
     DataTable,
     get_barchart_sources,
@@ -28,50 +29,48 @@ from src.figures import (
     make_column_picker,
     make_donut,
 )
-from src.utils import (
+from src.utils import logger
-    columns,
+from src.utils.cache import cache
-    departements,
+from src.utils.data import (
-    df,
+    DEPARTEMENTS,
-    df_acheteurs,
+    DF_ACHETEURS,
-    df_titulaires,
+    DF_TITULAIRES,
-    get_default_hidden_columns,
-    get_enum_values_as_dict,
-    logger,
-    meta_content,
     prepare_dashboard_data,
-    prepare_table_data,
 )
+from src.utils.frontend import get_enum_values_as_dict
+from src.utils.seo import META_CONTENT
+from src.utils.table import COLUMNS, get_default_hidden_columns, prepare_table_data
 
-name = "Observatoire"
+NAME = "Observatoire"
 
 register_page(
     __name__,
     path="/observatoire",
     title="Observatoire | decp.info",
-    name=name,
+    name=NAME,
     description="Visualisez l'état de la publication des données essentielles des marchés publics en France.",
-    image_url=meta_content["image_url"],
+    image_url=META_CONTENT["image_url"],
     order=3,
 )
-options_years = []
+OPTIONS_YEARS = []
 for year in reversed(range(2017, datetime.now().year + 1)):
     option_year = {
         "label": str(year),
         "value": year,
     }
-    options_years.append(option_year)
+    OPTIONS_YEARS.append(option_year)
 
-options_departements = []
+OPTIONS_DEPARTEMENTS = []
-for code in departements.keys():
+for code in DEPARTEMENTS.keys():
     departement = {
-        "label": f"{departements[code]['departement']} ({code})",
+        "label": f"{DEPARTEMENTS[code]['departement']} ({code})",
         "value": code,
     }
-    options_departements.append(departement)
+    OPTIONS_DEPARTEMENTS.append(departement)
 
 OBSERVATOIRE_COLUMNS = [
     col
-    for col in df.columns
+    for col in schema.names()
     if col.startswith("acheteur")
     or col.startswith("titulaire")
     or col
@@ -123,7 +122,7 @@ Alors, on fait comment ?
     html.Div(
         className="container-fluid",
         children=[
-            html.H2(children=[name], id="page_title"),
+            html.H2(children=[NAME], id="page_title"),
             dcc.Loading(
                 overlay_style={"visibility": "visible", "filter": "blur(2px)"},
                 id="loading-statistques",
@@ -141,7 +140,7 @@ Alors, on fait comment ?
                                         dbc.Col(
                                             dcc.Dropdown(
                                                 id="dashboard_year",
-                                                options=options_years,
+                                                options=OPTIONS_YEARS,
                                                 placeholder="12 derniers mois",
                                                 persistence=True,
                                                 persistence_type="local",
@@ -181,7 +180,7 @@ Alors, on fait comment ?
                                                 searchable=True,
                                                 multi=True,
                                                 placeholder="Département",
-                                                options=options_departements,
+                                                options=OPTIONS_DEPARTEMENTS,
                                                 persistence=True,
                                                 persistence_type="local",
                                             ),
@@ -220,7 +219,7 @@ Alors, on fait comment ?
                                                 searchable=True,
                                                 multi=True,
                                                 placeholder="Département",
-                                                options=options_departements,
+                                                options=OPTIONS_DEPARTEMENTS,
                                                 persistence=True,
                                                 persistence_type="local",
                                             ),
@@ -590,7 +589,6 @@ def sync_observatoire_share_url(*args):
     href = args[-1]
 
     if not href:
-        print("no update")
         return no_update, no_update
 
     base_url = href.split("?")[0]
@@ -607,7 +605,6 @@ def sync_observatoire_share_url(*args):
 
     query_string = urllib.parse.urlencode(params)
     full_url = f"{base_url}?{query_string}" if query_string else base_url
-    print("query", query_string)
 
     if params:
         copy_button = dcc.Clipboard(
@@ -650,39 +647,35 @@ def show_confirmation(n_clicks):
     return no_update
 
 
-@callback(
+def _normalize_filter_params(filter_params: dict) -> tuple:
-    Output("cards", "children"),
+    """Produce a deterministic, hashable key for caching."""
-    Output("observatoire-filters", "data"),
+    return tuple(
-    *[Input(fp[0], "value") for fp in FILTER_PARAMS],
+        sorted(
+            (k, tuple(v) if isinstance(v, list) else v)
+            for k, v in filter_params.items()
+        )
     )
-def udpate_dashboard_cards(*filter_values):
-    lff: pl.LazyFrame = df.lazy()
 
-    # Filtrage des données
-    filter_params = {}
-    for (input_id, url_key, is_multi, default), value in zip(
-        FILTER_PARAMS, filter_values
-    ):
-        filter_params[input_id] = value
 
-    print(filter_params)
+@cache.memoize()
+def _compute_dashboard_children(filter_params_normalized: tuple):
+    logger.debug("Cache miss — computing dashboard")
+    filter_params = {
+        k: (list(v) if isinstance(v, tuple) else v) for k, v in filter_params_normalized
+    }
 
+    lff: pl.LazyFrame = query_marches().lazy()
     lff = prepare_dashboard_data(lff=lff, **filter_params)
 
-    # Génération des métriques
     dff = lff.collect(engine="streaming")
 
-    logger.debug("Filter data: " + str(dff.height))
-
     df_per_uid = (
         dff.select("uid", "montant").group_by("uid").agg(pl.col("montant").first())
     )
     nb_marches = df_per_uid.height
 
     cards = []
-
     card_summary_table = get_dashboard_summary_table(dff, df_per_uid, nb_marches)
-
     cards.append(make_card(title="Résumé", paragraphs=card_summary_table))
 
     donut_acheteur_categorie, nb_acheteur_categories = make_donut(
@@ -741,10 +734,9 @@ def udpate_dashboard_cards(*filter_values):
     )
     cards.append(make_card(title="Top titulaires", fig=top_titulaires, lg=12, xl=8))
 
-    geographic_maps: list[dbc.Col] = get_geographic_maps(dff)
+    geographic_maps: list[dbc.Col] | None = get_geographic_maps(dff)
 
     other_cards = []
-
     sources_barchart = get_barchart_sources(lff, type_date="dateNotification")
     other_cards.append(
         make_card(
@@ -767,7 +759,25 @@ def udpate_dashboard_cards(*filter_values):
         )
     )
 
-    return dbc.Row(children=cards + geographic_maps + other_cards), filter_params
+    return cards + geographic_maps + other_cards
+
+
+@callback(
+    Output("cards", "children"),
+    Output("observatoire-filters", "data"),
+    *[Input(fp[0], "value") for fp in FILTER_PARAMS],
+)
+def update_dashboard_cards(*filter_values):
+    filter_params = {}
+    for (input_id, _url_key, _is_multi, _default), value in zip(
+        FILTER_PARAMS, filter_values
+    ):
+        filter_params[input_id] = value
+
+    filter_params_normalized = _normalize_filter_params(filter_params)
+    children = _compute_dashboard_children(filter_params_normalized)
+
+    return dbc.Row(children=children), filter_params
 
 
 @callback(
@@ -778,7 +788,7 @@ def udpate_dashboard_cards(*filter_values):
     prevent_initial_call=True,
 )
 def download_observatoire(_n_clicks, filter_params, hidden_columns):
-    lff = prepare_dashboard_data(lff=df.lazy(), **(filter_params or {}))
+    lff = prepare_dashboard_data(lff=query_marches().lazy(), **(filter_params or {}))
 
     if hidden_columns:
         lff = lff.drop(hidden_columns)
@@ -812,20 +822,20 @@ def add_organization_name_in_title(acheteur_id, titulaire_id):
         return match[nom_col].item(0) if match.height >= 1 else None
 
     if acheteur_id and len(acheteur_id) == 14:
-        if nom := lookup_nom(df_acheteurs, "acheteur_id", "acheteur_nom", acheteur_id):
+        if nom := lookup_nom(DF_ACHETEURS, "acheteur_id", "acheteur_nom", acheteur_id):
             return [
-                name,
+                NAME,
                 html.Small(nom, className="text-muted d-block fw-normal fs-5"),
             ]
     elif titulaire_id and len(titulaire_id) == 14:
         if nom := lookup_nom(
-            df_titulaires, "titulaire_id", "titulaire_nom", titulaire_id
+            DF_TITULAIRES, "titulaire_id", "titulaire_nom", titulaire_id
         ):
             return [
-                name,
+                NAME,
                 html.Small(nom, className="text-muted d-block fw-normal fs-5"),
             ]
-    return name
+    return NAME
 
 
 @callback(
@@ -869,7 +879,7 @@ def populate_preview_table(
     if not is_open:
         return (no_update,) * 9
 
-    lff = prepare_dashboard_data(lff=df.lazy(), **(filter_params or {}))
+    lff = prepare_dashboard_data(lff=query_marches().lazy(), **(filter_params or {}))
 
     return prepare_table_data(
         lff,
@@ -889,8 +899,8 @@ def populate_preview_table(
 )
 def update_hidden_columns_from_checkboxes(selected_columns):
     if selected_columns:
-        selected_columns = [columns[i] for i in selected_columns]
+        selected_columns = [COLUMNS[i] for i in selected_columns]
-        hidden_columns = [col for col in columns if col not in selected_columns]
+        hidden_columns = [col for col in COLUMNS if col not in selected_columns]
         return hidden_columns
     else:
         return []
@@ -918,7 +928,7 @@ def update_checkboxes_from_hidden_columns(hidden_cols, current_checkboxes):
     hidden_cols = hidden_cols or get_default_hidden_columns("tableau")
 
     # Show all columns that are NOT hidden
-    visible_cols = [columns.index(col) for col in columns if col not in hidden_cols]
+    visible_cols = [COLUMNS.index(col) for col in COLUMNS if col not in hidden_cols]
     return visible_cols
 
 

src/pages/recherche.py

@@ -2,23 +2,20 @@ import dash_bootstrap_components as dbc
 from dash import Input, Output, State, callback, dcc, html, register_page
 
 from src.figures import DataTable
-from src.utils import (
+from src.utils.data import DF_ACHETEURS, DF_TITULAIRES
-    df_acheteurs,
+from src.utils.search import search_org
-    df_titulaires,
+from src.utils.seo import META_CONTENT
-    meta_content,
+from src.utils.table import setup_table_columns
-    search_org,
-    setup_table_columns,
-)
 
-name = "Recherche"
+NAME = "Recherche"
 
 register_page(
     __name__,
     path="/",
     title="Recherche de marchés publics | decp.info",
-    name=name,
+    name=NAME,
     description="Explorez et analysez les données des marchés publics français avec cet outil libre et gratuit. Pour une commande publique accessible à toutes et tous.",
-    image_url=meta_content["image_url"],
+    image_url=META_CONTENT["image_url"],
     order=0,
 )
 
@@ -97,9 +94,9 @@ def update_search_results(n_submit, n_clicks, query):
 
         for org_type in ["acheteur", "titulaire"]:
             if org_type == "acheteur":
-                dff = df_acheteurs
+                dff = DF_ACHETEURS
             elif org_type == "titulaire":
-                dff = df_titulaires
+                dff = DF_TITULAIRES
             else:
                 raise ValueError(f"{org_type} is not supported")
 

src/pages/tableau.py

@@ -19,37 +19,37 @@ from dash import (
     register_page,
 )
 
+from src.db import query_marches, schema
 from src.figures import DataTable, make_column_picker
-from src.utils import (
+from src.utils import logger
-    columns,
+from src.utils.seo import META_CONTENT
-    df,
+from src.utils.table import (
+    COLUMNS,
     filter_table_data,
     get_default_hidden_columns,
     invert_columns,
-    logger,
-    meta_content,
     prepare_table_data,
-    schema,
     sort_table_data,
 )
+from src.utils.tracking import track_search
 
 update_date_timestamp = os.path.getmtime(os.getenv("DATA_FILE_PARQUET_PATH"))
 update_date = datetime.fromtimestamp(update_date_timestamp).strftime("%d/%m/%Y")
 update_date_iso = datetime.fromtimestamp(update_date_timestamp).isoformat()
 
 
-name = "Tableau"
+NAME = "Tableau"
 register_page(
     __name__,
     path="/tableau",
     title="Tableau des marchés publics | decp.info",
-    name=name,
+    name=NAME,
     description="Consultez, filtrez et exportez les données essentielles de la commande publique sous forme de tableau.",
-    image_url=meta_content["image_url"],
+    image_url=META_CONTENT["image_url"],
     order=1,
 )
 
-datatable = html.Div(
+DATATABLE = html.Div(
     className="marches_table",
     children=DataTable(
         dtid="tableau_datatable",
@@ -61,7 +61,7 @@ datatable = html.Div(
         filter_action="custom",
         sort_action="custom",
         hidden_columns=[],
-        columns=[{"id": col, "name": col} for col in df.columns],
+        columns=[{"id": col, "name": col} for col in schema.names()],
     ),
 )
 
@@ -128,7 +128,7 @@ layout = [
         ],
     ),
     dcc.Markdown(
-        f"Ce tableau contient tous les marchés attribués en France. Il vous permet d'appliquer un filtre sur une ou plusieurs colonnes, et ainsi produire la liste de marchés dont vous avez besoin (exemples : [marchés de voirie < 40 k€ en 2025](/tableau?filtres=%7Bacheteur_id%7D+icontains+24350013900189+%26%26+%7BdateNotification%7D+icontains+2025%2A+%26%26+%7Bmontant%7D+i%3C+40000+%26%26+%7Bobjet%7D+icontains+voirie&colonnes=uid%2Cacheteur_id%2Cacheteur_nom%2Ctitulaire_id%2Ctitulaire_nom%2Cobjet%2Cmontant%2CdureeMois%2CdateNotification%2Cacheteur_departement_code%2CsourceDataset), [marchés > 500 k€ avec clause sociale attribués à des PME à plus de 100 km dans le Var](/tableau?filtres=%7Btitulaire_categorie%7D+icontains+PME+%26%26+%7Btitulaire_distance%7D+i%3E+100+%26%26+%7Bmontant%7D+i%3E+500000+%26%26+%7Bacheteur_departement_code%7D+icontains+83+%26%26+%7BconsiderationsSociales%7D+icontains+clause&colonnes=uid%2Cacheteur_id%2Cacheteur_nom%2Ctitulaire_id%2Ctitulaire_nom%2Cobjet%2Cmontant%2CdureeMois%2CdateNotification%2CconsiderationsSociales%2Ctitulaire_distance%2Cacheteur_departement_code%2Ctitulaire_categorie%2CsourceDataset)). Par défaut seules quelques colonnes sont affichées, mais vous pouvez en afficher jusqu'à {str(df.width)} en cliquant sur le bouton **Choisir les colonnes**. Cet outil est assez puissant, je vous recommande de lire le mode d'emploi pour en tirer pleinement partie.",
+        f"Ce tableau contient tous les marchés attribués en France. Il vous permet d'appliquer un filtre sur une ou plusieurs colonnes, et ainsi produire la liste de marchés dont vous avez besoin (exemples : [marchés de voirie < 40 k€ en 2025](/tableau?filtres=%7Bacheteur_id%7D+icontains+24350013900189+%26%26+%7BdateNotification%7D+icontains+2025%2A+%26%26+%7Bmontant%7D+i%3C+40000+%26%26+%7Bobjet%7D+icontains+voirie&colonnes=uid%2Cacheteur_id%2Cacheteur_nom%2Ctitulaire_id%2Ctitulaire_nom%2Cobjet%2Cmontant%2CdureeMois%2CdateNotification%2Cacheteur_departement_code%2CsourceDataset), [marchés > 500 k€ avec clause sociale attribués à des PME à plus de 100 km dans le Var](/tableau?filtres=%7Btitulaire_categorie%7D+icontains+PME+%26%26+%7Btitulaire_distance%7D+i%3E+100+%26%26+%7Bmontant%7D+i%3E+500000+%26%26+%7Bacheteur_departement_code%7D+icontains+83+%26%26+%7BconsiderationsSociales%7D+icontains+clause&colonnes=uid%2Cacheteur_id%2Cacheteur_nom%2Ctitulaire_id%2Ctitulaire_nom%2Cobjet%2Cmontant%2CdureeMois%2CdateNotification%2CconsiderationsSociales%2Ctitulaire_distance%2Cacheteur_departement_code%2Ctitulaire_categorie%2CsourceDataset)). Par défaut seules quelques colonnes sont affichées, mais vous pouvez en afficher jusqu'à {len(schema.names())} en cliquant sur le bouton **Choisir les colonnes**. Cet outil est assez puissant, je vous recommande de lire le mode d'emploi pour en tirer pleinement partie.",
         style={"maxWidth": "1000px"},
     ),
     html.Div(
@@ -188,7 +188,7 @@ layout = [
 
             ##### Afficher plus de colonnes
 
-            Par défaut, un nombre réduit de colonnes est affiché pour ne pas surcharger la page. Mais vous avez le choix parmi {str(df.width)} colonnes, ce serait dommage de vous limiter !
+            Par défaut, un nombre réduit de colonnes est affiché pour ne pas surcharger la page. Mais vous avez le choix parmi {len(schema.names())} colonnes, ce serait dommage de vous limiter !
 
             Pour afficher plus de colonnes, cliquez sur le bouton **Choisir les colonnes** et cochez les colonnes pour les afficher.
 
@@ -273,7 +273,7 @@ layout = [
                 scrollable=True,
                 size="xl",
             ),
-            datatable,
+            DATATABLE,
         ],
     ),
 ]
@@ -315,15 +315,16 @@ def update_table(href, page_current, page_size, filter_query, sort_by, data_time
     State("tableau_datatable", "hidden_columns"),
     prevent_initial_call=True,
 )
-def download_data(n_clicks, filter_query, sort_by, hidden_columns: list = None):
+def download_data(n_clicks, filter_query, sort_by, hidden_columns: list | None = None):
-    lff: pl.LazyFrame = df.lazy()  # start from the original data
+    lff: pl.LazyFrame = query_marches().lazy()
 
     # Les colonnes masquées sont supprimées
     if hidden_columns:
         lff = lff.drop(hidden_columns)
 
     if filter_query:
-        lff = filter_table_data(lff, filter_query, "tab download")
+        track_search(filter_query, "tab download")
+        lff = filter_table_data(lff, filter_query)
 
     if sort_by and len(sort_by) > 0:
         lff = sort_table_data(lff, sort_by)
@@ -481,8 +482,8 @@ def toggle_tableau_help(click_open, click_close, is_open):
 )
 def update_hidden_columns_from_checkboxes(selected_columns):
     if selected_columns:
-        selected_columns = [columns[i] for i in selected_columns]
+        selected_columns = [COLUMNS[i] for i in selected_columns]
-        hidden_columns = [col for col in columns if col not in selected_columns]
+        hidden_columns = [col for col in COLUMNS if col not in selected_columns]
         return hidden_columns
     else:
         return []
@@ -496,6 +497,8 @@ def update_hidden_columns_from_checkboxes(selected_columns):
     ),
 )
 def store_hidden_columns(hidden_columns):
+    if hidden_columns is None:
+        hidden_columns = get_default_hidden_columns("tableau")
     return hidden_columns
 
 
@@ -508,7 +511,7 @@ def update_checkboxes_from_hidden_columns(hidden_cols, current_checkboxes):
     hidden_cols = hidden_cols or get_default_hidden_columns("tableau")
 
     # Show all columns that are NOT hidden
-    visible_cols = [columns.index(col) for col in columns if col not in hidden_cols]
+    visible_cols = [COLUMNS.index(col) for col in COLUMNS if col not in hidden_cols]
     return visible_cols
 
 

src/pages/titulaire.py

@@ -15,6 +15,7 @@ from dash import (
     register_page,
 )
 
+from src.db import query_marches, schema
 from src.figures import (
     DataTable,
     get_distance_histogram,
@@ -22,24 +23,22 @@ from src.figures import (
     make_column_picker,
     point_on_map,
 )
-from src.utils import (
+from src.utils.data import DF_TITULAIRES, get_annuaire_data, get_departement_region
-    columns,
+from src.utils.frontend import get_button_properties
-    df,
+from src.utils.seo import META_CONTENT
-    df_titulaires,
+from src.utils.table import (
+    COLUMNS,
     filter_table_data,
     format_number,
-    get_annuaire_data,
-    get_button_properties,
     get_default_hidden_columns,
-    get_departement_region,
-    meta_content,
     prepare_table_data,
     sort_table_data,
 )
+from src.utils.tracking import track_search
 
 
 def get_title(titulaire_id: str = None) -> str:
-    titulaire_nom = df_titulaires.filter(pl.col("titulaire_id") == titulaire_id).select(
+    titulaire_nom = DF_TITULAIRES.filter(pl.col("titulaire_id") == titulaire_id).select(
         "titulaire_nom"
     )
     if titulaire_nom.height > 0:
@@ -53,11 +52,11 @@ register_page(
     title=get_title,
     name="Titulaire",
     description="Consultez les marchés publics remportés par ce titulaire.",
-    image_url=meta_content["image_url"],
+    image_url=META_CONTENT["image_url"],
     order=5,
 )
 
-datatable = html.Div(
+DATATABLE = html.Div(
     className="marches_table",
     children=DataTable(
         dtid="titulaire_datatable",
@@ -69,7 +68,7 @@ datatable = html.Div(
         sort_action="custom",
         page_size=10,
         hidden_columns=[],
-        columns=[{"id": col, "name": col} for col in df.columns],
+        columns=[{"id": col, "name": col} for col in schema.names()],
     ),
 )
 
@@ -239,7 +238,7 @@ layout = [
                         scrollable=True,
                         size="xl",
                     ),
-                    datatable,
+                    DATATABLE,
                 ],
             ),
         ],
@@ -339,21 +338,18 @@ def update_titulaire_stats(data):
 )
 def get_titulaire_marches_data(url, titulaire_year: str) -> tuple:
     titulaire_siret = url.split("/")[-1]
-    lff = df.lazy()
+    lff = query_marches(
-    lff = lff.filter(
+        "titulaire_id = ? AND titulaire_typeIdentifiant = 'SIRET'",
-        (pl.col("titulaire_id") == titulaire_siret)
+        (titulaire_siret,),
-        & (pl.col("titulaire_typeIdentifiant") == "SIRET")
+    ).lazy()
-    )
     if titulaire_year and titulaire_year != "Toutes les années":
         lff = lff.filter(
             pl.col("dateNotification").cast(pl.String).str.starts_with(titulaire_year)
         )
     lff = lff.sort(["dateNotification", "uid"], descending=True, nulls_last=True)
     lff = lff.fill_null("")
-
     dff: pl.DataFrame = lff.collect(engine="streaming")
     download_disabled, download_text, download_title = get_button_properties(dff.height)
-
     data = dff.to_dicts()
     return data, download_disabled, download_text, download_title
 
@@ -434,7 +430,12 @@ def download_titulaire_data(
     prevent_initial_call=True,
 )
 def download_filtered_titulaire_data(
-    data, n_clicks, titulaire_nom, filter_query, sort_by, hidden_columns: list = None
+    data,
+    n_clicks,
+    titulaire_nom,
+    filter_query,
+    sort_by,
+    hidden_columns: list | None = None,
 ):
     lff: pl.LazyFrame = pl.LazyFrame(
         data
@@ -445,7 +446,8 @@ def download_filtered_titulaire_data(
         lff = lff.drop(hidden_columns)
 
     if filter_query:
-        lff = filter_table_data(lff, filter_query, "titu download")
+        track_search(filter_query, "titu download")
+        lff = filter_table_data(lff, filter_query)
 
     if len(sort_by) > 0:
         lff = sort_table_data(lff, sort_by)
@@ -479,22 +481,23 @@ clientside_callback(
 )
 def update_hidden_columns_from_checkboxes(selected_columns):
     if selected_columns:
-        selected_columns = [columns[i] for i in selected_columns]
+        selected_columns = [COLUMNS[i] for i in selected_columns]
-        hidden_columns = [col for col in columns if col not in selected_columns]
+        hidden_columns = [col for col in COLUMNS if col not in selected_columns]
         return hidden_columns
     else:
         return []
 
 
 @callback(
-    Output("titulaire_datatable", "hidden_columns", allow_duplicate=True),
+    Output("titulaire_datatable", "hidden_columns"),
     Input(
         "titulaire-hidden-columns",
         "data",
     ),
-    prevent_initial_call=True,
 )
 def store_hidden_columns(hidden_columns):
+    if hidden_columns is None:
+        hidden_columns = get_default_hidden_columns("titulaire")
     return hidden_columns
 
 
@@ -507,7 +510,7 @@ def update_checkboxes_from_hidden_columns(hidden_cols, current_checkboxes):
     hidden_cols = hidden_cols or get_default_hidden_columns("titulaire")
 
     # Show all columns that are NOT hidden
-    visible_cols = [columns.index(col) for col in columns if col not in hidden_cols]
+    visible_cols = [COLUMNS.index(col) for col in COLUMNS if col not in hidden_cols]
     return visible_cols
 
 

src/utils.py

@@ -1,916 +0,0 @@
-import json
-import logging
-import os
-import uuid
-from collections import OrderedDict
-from datetime import datetime, timedelta
-from time import localtime, sleep
-
-import polars as pl
-import polars.selectors as cs
-from dash import no_update
-from httpx import HTTPError, get, post
-from polars.exceptions import ComputeError
-from unidecode import unidecode
-
-logging.basicConfig(
-    format="%(asctime)s %(levelname)-8s %(message)s",
-    level=logging.INFO,
-    datefmt="%Y-%m-%d %H:%M:%S",
-)
-logger = logging.getLogger("decp.info")
-development = os.getenv("DEVELOPMENT", "False").lower() == "true"
-if development:
-    logger.setLevel(logging.DEBUG)
-
-logging.getLogger("httpx").setLevel("WARNING")
-
-
-def split_filter_part(filter_part):
-    operators = [
-        ["s<", "<"],
-        ["s>", ">"],
-        ["i<", "<"],
-        ["i>", ">"],
-        ["icontains", "contains"],
-        # [" ", "contains"]
-    ]
-    logger.debug("filter part " + filter_part)
-    for operator_group in operators:
-        if operator_group[0] in filter_part:
-            name_part, value_part = filter_part.split(operator_group[0], 1)
-            name_part = name_part.strip()
-            value = value_part.strip()
-            name = name_part[name_part.find("{") + 1 : name_part.rfind("}")]
-            logger.debug("=> " + " ".join([name, operator_group[1], value]))
-
-            return name, operator_group[1], value
-
-    return [None] * 3
-
-
-def add_resource_link(dff: pl.DataFrame) -> pl.DataFrame:
-    dff = dff.with_columns(
-        (
-            '<a href="' + pl.col("sourceFile") + '">' + pl.col("sourceDataset") + "</a>"
-        ).alias("sourceDataset")
-    )
-    dff = dff.drop(["sourceFile"])
-    return dff
-
-
-def add_links(dff: pl.DataFrame):
-    for col in ["uid", "acheteur_nom", "titulaire_nom", "acheteur_id", "titulaire_id"]:
-        if col in dff.columns:
-            if col.startswith("titulaire_"):
-                detail_link = (
-                    '<a href = "/titulaires/'
-                    + pl.col("titulaire_id")
-                    + '">'
-                    + pl.col(col)
-                    + "</a>"
-                )
-                if col == "titulaire_nom":
-                    detail_link = (
-                        detail_link
-                        + ' <a href="/observatoire?titulaire_id='
-                        + pl.col("titulaire_id")
-                        + '" title="Voir dans l\'observatoire">📊</a>'
-                    )
-                dff = dff.with_columns(
-                    pl.when(
-                        pl.Expr.or_(
-                            pl.col("titulaire_typeIdentifiant").is_null(),
-                            pl.col("titulaire_typeIdentifiant") == "SIRET",
-                        )
-                    )
-                    .then(detail_link)
-                    .otherwise(pl.col(col))
-                    .alias(col)
-                )
-            if col.startswith("acheteur_"):
-                detail_link = (
-                    '<a href = "/acheteurs/'
-                    + pl.col("acheteur_id")
-                    + '">'
-                    + pl.col(col)
-                    + "</a>"
-                )
-                if col == "acheteur_nom":
-                    detail_link = (
-                        detail_link
-                        + ' <a href="/observatoire?acheteur_id='
-                        + pl.col("acheteur_id")
-                        + '" title="Voir dans l\'observatoire">📊</a>'
-                    )
-                dff = dff.with_columns(detail_link.alias(col))
-            if col == "uid":
-                dff = dff.with_columns(
-                    (
-                        '<a href = "/marches/'
-                        + pl.col("uid")
-                        + '">'
-                        + pl.col("uid")
-                        + "</a>"
-                    ).alias("uid")
-                )
-    return dff
-
-
-def add_links_in_dict(data: list[dict], org_type: str) -> list:
-    new_data = []
-    for marche in data:
-        org_id = marche[org_type + "_id"]
-        marche[org_type + "_nom"] = (
-            f'<a href="/{org_type}s/{org_id}">{marche[org_type + "_nom"]}</a>'
-        )
-        if marche.get("uid"):
-            marche["id"] = f'<a href="/marches/{marche["uid"]}">{marche["id"]}</a>'
-            marche["uid"] = f'<a href="/marches/{marche["uid"]}">{marche["uid"]}</a>'
-        new_data.append(marche)
-    return new_data
-
-
-def booleans_to_strings(lff: pl.LazyFrame) -> pl.LazyFrame:
-    """
-    Convert all boolean columns to string type.
-    """
-    lff = lff.with_columns(
-        pl.col(cs.Boolean)
-        .cast(pl.String)
-        .str.replace("true", "oui")
-        .str.replace("false", "non")
-    )
-    return lff
-
-
-def numbers_to_strings(lff: pl.LazyFrame) -> pl.LazyFrame:
-    """
-    Convert all numeric columns to string type.
-    """
-    lff = lff.with_columns(pl.col(pl.Float64, pl.Int16).cast(pl.String).fill_null(""))
-    return lff
-
-
-def dates_to_strings(lff: pl.LazyFrame, column: str) -> pl.LazyFrame:
-    """
-    Convert a date column to string type.
-    """
-    lff = lff.with_columns(pl.col(column).cast(pl.String).fill_null(""))
-    return lff
-
-
-def format_number(number) -> str:
-    number = "{:,}".format(number).replace(",", " ")
-    return number
-
-
-def unformat_montant(number: str) -> float:
-    number = number.replace(" €", "")
-    number = number.replace(" €", "").replace(" ", "")
-    number = number.replace(",", ".")
-    number = number.strip()
-    return float(number)
-
-
-def format_values(dff: pl.DataFrame) -> pl.DataFrame:
-    def format_montant(expr, scale=None):
-        # https://stackoverflow.com/a/78636786
-        expr = expr.cast(pl.String)
-        expr = expr.str.splitn(".", 2)
-
-        num = expr.struct[0]
-        frac = expr.struct[1]
-
-        # Ajout des espaces
-        num = (
-            num.str.reverse()
-            .str.replace_all(r"\d{3}", "$0 ")
-            .str.reverse()
-            .str.replace(r"^ ", "")
-        )
-
-        frac: pl.Expr = (
-            pl.when(frac.is_not_null() & ~frac.is_in(["0"]))
-            .then("," + frac.str.head(2))
-            .otherwise(pl.lit(""))
-        )
-
-        montant: pl.Expr = (
-            pl.when((num + frac) == pl.lit(""))
-            .then(pl.lit(""))
-            .otherwise(num + frac + pl.lit(" €"))
-        )
-
-        return montant
-
-    def format_distance(expr):
-        expr = expr.cast(pl.String)
-        return pl.concat_str(expr, pl.lit(" km"))
-
-    if "montant" in dff.columns:
-        dff = dff.with_columns(pl.col("montant").pipe(format_montant).alias("montant"))
-    if "titulaire_distance" in dff.columns:
-        dff = dff.with_columns(
-            pl.col("titulaire_distance")
-            .pipe(format_distance)
-            .alias("titulaire_distance")
-        )
-
-    return dff
-
-
-def get_annuaire_data(siret: str) -> dict:
-    url = f"https://recherche-entreprises.api.gouv.fr/search?q={siret}"
-    try:
-        response = get(url).raise_for_status()
-        response = response.json()["results"][0]
-    except (HTTPError, IndexError):
-        response = None
-        logger.warning("Could not fetch data from recherche-entreprises.api.")
-    return response
-
-
-def get_decp_data() -> pl.DataFrame:
-    # Chargement du fichier parquet
-    # Le fichier est chargé en mémoire, ce qui est plus rapide qu'une base de données pour le moment.
-    # On utilise polars pour la rapidité et la facilité de manipulation des données.
-
-    try:
-        logger.info(
-            f"Lecture du fichier parquet ({os.getenv('DATA_FILE_PARQUET_PATH')})..."
-        )
-        lff: pl.LazyFrame = pl.scan_parquet(os.getenv("DATA_FILE_PARQUET_PATH"))
-    except ComputeError:
-        # Le fichier est probablement en cours de mise à jour
-        logger.info("Échec, nouvelle tentative dans 10s...")
-        sleep(10)
-        lff: pl.LazyFrame = pl.scan_parquet(os.getenv("DATA_FILE_PARQUET_PATH"))
-
-    # Tri des marchés par date de notification
-    lff = lff.sort(by=["dateNotification", "uid"], descending=True, nulls_last=True)
-
-    # Uniquement les données actuelles, pas les anciennes versions de marchés
-    lff = lff.filter(pl.col("donneesActuelles")).drop("donneesActuelles")
-
-    # Convertir les colonnes booléennes en chaînes de caractères
-    lff = booleans_to_strings(lff)
-
-    # Mention pour les org dont on a pas le nom
-    for col in ["acheteur_nom", "titulaire_nom"]:
-        lff = lff.with_columns(
-            pl.when(pl.col(col).is_null())
-            .then(pl.lit("[Identifiant non reconnu dans la base INSEE]"))
-            .otherwise(pl.col(col))
-            .name.keep()
-        )
-
-    # Bizarrement je ne peux pas faire lff = lff.fill_null("") ici
-    # ça génère une erreur dans la page acheteur (acheteur_data.table) :
-    # AttributeError: partially initialized module 'pandas' has no attribute 'NaT' (most likely due to a circular import)
-
-    return lff.collect()
-
-
-def get_org_data(dff: pl.DataFrame, org_type: str) -> pl.DataFrame:
-    lff = dff.lazy()
-    lff = lff.select(
-        "uid",
-        cs.starts_with(org_type).exclude(
-            f"{org_type}_latitude", f"{org_type}_longitude"
-        ),
-    )
-    lff = lff.group_by(cs.starts_with(org_type)).len("Marchés")
-    return lff.collect()
-
-
-def get_statistics() -> dict:
-    return (
-        get(
-            "https://www.data.gouv.fr/api/1/datasets/r/0ccf4a75-f3aa-4b46-8b6a-18aeb63e36df",
-            follow_redirects=True,
-        )
-        .raise_for_status()
-        .json()
-    )
-
-
-def get_departements() -> dict:
-    with open("data/departements.json", "rb") as f:
-        data = json.load(f)
-        return data
-
-
-def get_departements_geojson() -> dict:
-    with open("./data/departements-1000m.geojson") as f:
-        geojson = json.load(f)
-
-    # Ajout de feature.id
-    for f in geojson["features"]:
-        f["id"] = f["properties"]["code"]
-
-    return geojson
-
-
-def get_departement_region(code_postal):
-    if code_postal > "97000":
-        code_departement = code_postal[:3]
-    else:
-        code_departement = code_postal[:2]
-    nom_departement = departements[code_departement]["departement"]
-    nom_region = departements[code_departement]["region"]
-    return code_departement, nom_departement, nom_region
-
-
-def filter_table_data(
-    lff: pl.LazyFrame, filter_query: str, filter_source: str
-) -> pl.LazyFrame:
-    _schema = lff.collect_schema()
-    track_search(filter_query, filter_source)
-    filtering_expressions = filter_query.split(" && ")
-    for filter_part in filtering_expressions:
-        col_name, operator, filter_value = split_filter_part(filter_part)
-        col_type = str(_schema[col_name])
-        # logger.debug("filter_value:", filter_value)
-        # logger.debug("filter_value_type:", type(filter_value))
-        # logger.debug("operator:", operator)
-        # logger.debug("col_type:", col_type)
-
-        lff = lff.filter(pl.col(col_name).is_not_null())
-
-        if col_type == "Date":
-            # Convertir la colonne date en chaînes de caractères
-            lff = dates_to_strings(lff, col_name)
-            col_type = "String"
-        if col_type == "String":
-            lff = lff.filter(pl.col(col_name) != pl.lit(""))
-
-        elif col_type.startswith("Int") or col_type.startswith("Float"):
-            try:
-                filter_value = int(filter_value)
-            except ValueError:
-                logger.error(f"Invalid numeric filter value: {filter_value}")
-                continue
-
-        if operator in ("contains", "<", "<=", ">", ">="):
-            if operator == "<":
-                lff = lff.filter(pl.col(col_name) < filter_value)
-            elif operator == ">":
-                lff = lff.filter(pl.col(col_name) > filter_value)
-            elif operator == ">=":
-                lff = lff.filter(pl.col(col_name) >= filter_value)
-            elif operator == "<=":
-                lff = lff.filter(pl.col(col_name) <= filter_value)
-            elif operator == "contains":
-                if col_type in ["String", "Date"]:
-                    filter_value = filter_value.strip('"')
-                    if filter_value.endswith("*"):
-                        lff = lff.filter(
-                            pl.col(col_name).str.starts_with(filter_value[:-1])
-                        )
-                    elif filter_value.startswith("*"):
-                        lff = lff.filter(
-                            pl.col(col_name).str.ends_with(filter_value[1:])
-                        )
-                    else:
-                        lff = lff.filter(
-                            pl.col(col_name).str.contains("(?i)" + filter_value)
-                        )
-                elif col_type.startswith("Int") or col_type.startswith("Float"):
-                    lff = lff.filter(pl.col(col_name) == filter_value)
-                else:
-                    logger.error(f"Invalid column type: {col_type}")
-        else:
-            logger.error(f"Invalid operator: {operator}")
-
-        # elif operator == 'datestartswith':
-        # lff = lff.filter(pl.col(col_name).str.startswith(filter_value)")
-
-    return lff
-
-
-def sort_table_data(lff: pl.LazyFrame, sort_by: list) -> pl.LazyFrame:
-    lff = lff.sort(
-        [col["column_id"] for col in sort_by],
-        descending=[col["direction"] == "desc" for col in sort_by],
-        nulls_last=True,
-    )
-    logger.debug(sort_by)
-    return lff
-
-
-def setup_table_columns(
-    dff, hideable: bool = True, exclude: list = None, new_columns: list = None
-) -> tuple:
-    # Liste finale de colonnes
-    markdown_exceptions = ["montant", "titulaire_distance", "distance", "dureeMois"]
-    columns = []
-    tooltip = {}
-    for column_id in dff.columns:
-        if exclude and column_id in exclude:
-            continue
-        column_object = data_schema.get(column_id)
-        if column_object:
-            column_name = column_object.get("title")
-        else:
-            # Si le champ est un champ créé par erreur lors d'une jointure, on le skip
-            if column_id.endswith("_left") or column_id.endswith("_right"):
-                logger.warning(f"Champ innatendu : {column_id}")
-                continue
-            column_name = column_id
-            column_object = {"title": column_name, "description": ""}
-
-        presentation = "input" if column_id in markdown_exceptions else "markdown"
-
-        column = {
-            "name": column_name,
-            "id": column_id,
-            "presentation": presentation,
-            "type": "text",
-            "format": {"nully": "N/A"},
-            "hideable": hideable,
-        }
-        columns.append(column)
-        if column_object:
-            tooltip[column_id] = {
-                "value": f"""**{column_object.get("title")}** ({column_id})
-
-    """
-                + column_object.get("description", ""),
-                "type": "markdown",
-            }
-    return columns, tooltip
-
-
-def get_default_hidden_columns(page):
-    if page == "acheteur":
-        displayed_columns = [
-            "uid",
-            "objet",
-            "dateNotification",
-            "titulaire_id",
-            "titulaire_typeIdentifiant",
-            "titulaire_nom",
-            "titulaire_distance",
-            "montant",
-            "codeCPV",
-            "dureeRestanteMois",
-        ]
-    elif page == "titulaire":
-        displayed_columns = [
-            "uid",
-            "objet",
-            "dateNotification",
-            "acheteur_id",
-            "acheteur_nom",
-            "titulaire_distance",
-            "montant",
-            "codeCPV",
-            "dureeRestanteMois",
-        ]
-    elif page == "tableau":
-        displayed_columns = os.getenv("DISPLAYED_COLUMNS")
-    else:
-        displayed_columns = os.getenv("DISPLAYED_COLUMNS")
-        logger.warning(f"Invalid page: {page}")
-
-    hidden_columns = []
-
-    for col in schema.names():
-        if col in displayed_columns:
-            continue
-        else:
-            hidden_columns.append(col)
-    return hidden_columns
-
-
-def get_data_schema() -> dict:
-    # Récupération du schéma des données tabulaires
-    path = os.getenv("DATA_SCHEMA_PATH")
-    if path.startswith("http"):
-        original_schema: dict = get(
-            os.getenv("DATA_SCHEMA_PATH"), follow_redirects=True
-        ).json()
-    elif os.path.exists(path):
-        with open(path) as f:
-            original_schema: dict = json.load(f)
-    else:
-        raise Exception(f"Chemin vers le schéma invalide: {path}")
-
-    new_schema = OrderedDict()
-
-    for col in original_schema["fields"]:
-        new_schema[col["name"]] = col
-
-    return new_schema
-
-
-def track_search(query, category):
-    if len(query) >= 4 and not development and os.getenv("MATOMO_DOMAIN"):
-        url = "https://decp.info"
-        params = {
-            "idsite": os.getenv("MATOMO_ID_SITE"),
-            "url": url,
-            "rec": "1",
-            "action_name": "search" if category == "home_page_search" else "filter",
-            "search_cat": category,
-            "rand": uuid.uuid4().hex,
-            "apiv": "1",
-            "h": localtime().tm_hour,
-            "m": localtime().tm_min,
-            "s": localtime().tm_sec,
-            "search": query,
-            "token_auth": os.getenv("MATOMO_TOKEN"),
-        }
-        post(
-            url=f"https://{os.getenv('MATOMO_DOMAIN')}/matomo.php",
-            params=params,
-        ).raise_for_status()
-
-
-def search_org(dff: pl.DataFrame, query: str, org_type: str) -> pl.DataFrame:
-    """
-    Search in either 'acheteur' or 'titulaire' DataFrame.
-
-    :param dff: Polars DataFrame with acheteur or titulaire columns
-    :param query: User search string
-    :param org_type: 'acheteur' or 'titulaire'
-    :return: Filtered DataFrame with 'matches' column
-    """
-    if not query.strip():
-        return dff.select(pl.lit(False).alias("matches"))
-
-    # Enregistrement des recherche dans Matomo
-    track_search(query, "home_page_search")
-
-    # Normalize query
-    normalized_query = unidecode(query.strip()).upper()
-    tokens = [" " + t.strip() for t in normalized_query.split() if t.strip()]
-
-    # Define columns based on entity type
-    cols = [
-        f"{org_type}_id",
-        f"{org_type}_nom",
-        f"{org_type}_departement_nom",
-        f"{org_type}_departement_code",
-        f"{org_type}_commune_nom",
-    ]
-
-    # Concatenate all fields into one string per row
-    org_str = pl.concat_str(pl.lit(" "), pl.col(cols), separator=" ").str.replace(
-        "-", " "
-    )
-
-    # For each token, create a boolean column: True if token is found
-    token_matches = []
-    for token in tokens:
-        token_match = org_str.str.contains(token).alias(f"token_{token}")
-        token_matches.append(token_match)
-
-    # Count how many tokens match per row
-    match_score = pl.sum_horizontal(token_matches).alias("match_score")
-
-    # For each token, create a boolean column: True if token is found
-    token_matches = []
-    for token in tokens:
-        token_match = org_str.str.contains(token).alias(f"token_{token}")
-        token_matches.append(token_match)
-
-    # Sélection des colonnes
-    if org_type == "acheteur":
-        dff = dff.select(cols + ["Marchés"])
-    if org_type == "titulaire":
-        dff = dff.select(cols + ["Marchés", "titulaire_typeIdentifiant"])
-
-    # Apply and filter
-    dff = (
-        dff.with_columns(token_matches + [match_score])
-        .filter(pl.col("match_score") == len(tokens))
-        .drop([f"token_{token}" for token in tokens])
-    )
-
-    # Format result
-    dff = add_links(dff)
-    dff = dff.with_columns(
-        pl.concat_str(
-            pl.col(f"{org_type}_departement_nom"),
-            pl.lit(" ("),
-            pl.col(f"{org_type}_departement_code"),
-            pl.lit(")"),
-        ).alias("Département")
-    )
-
-    dff = dff.select(f"{org_type}_id", f"{org_type}_nom", "Département", "Marchés")
-    dff = dff.group_by(f"{org_type}_id", f"{org_type}_nom", "Département").sum()
-    dff = dff.sort("Marchés", descending=True)
-
-    return dff
-
-
-def prepare_table_data(
-    data, data_timestamp, filter_query, page_current, page_size, sort_by, source_table
-):
-    """
-    Fonction de préparation des données pour les datatables, afin de permettre une gestion fine des logiques,
-    notamment pour les filtres et les tris.
-    :param data
-    :param data_timestamp:
-    :param filter_query:
-    :param page_current:
-    :param page_size:
-    :param sort_by:
-    :param source_table:
-    :return:
-    """
-
-    if os.getenv("DEVELOPMENT").lower() == "true":
-        logger.debug(" + + + + + + + + + + + + + + + + + + ")
-
-    trigger_cleanup = no_update
-
-    # Récupération des données
-    if isinstance(data, list):
-        lff: pl.LazyFrame = pl.LazyFrame(data, strict=False, infer_schema_length=5000)
-    elif isinstance(data, pl.LazyFrame):
-        lff = data
-    else:
-        lff: pl.LazyFrame = df.lazy()  # start from the original data
-
-    # Application des filtres
-    if filter_query:
-        lff = filter_table_data(lff, filter_query, source_table)
-        trigger_cleanup = no_update if source_table == "tableau" else str(uuid.uuid4())
-
-    # Application des tris
-    if sort_by and len(sort_by) > 0:
-        lff = sort_table_data(lff, sort_by)
-
-    # Matérialisation des filtres
-    dff: pl.DataFrame = lff.collect()
-    height = dff.height
-
-    if height > 0:
-        nb_rows = f"{format_number(height)} lignes ({format_number(dff.select('uid').unique().height)} marchés)"
-    else:
-        nb_rows = "0 lignes (0 marchés)"
-
-    # Pagination des données
-    start_row = page_current * page_size
-    # end_row = (page_current + 1) * page_size
-    dff = dff.slice(start_row, page_size)
-
-    # Tout devient string
-    dff = dff.cast(pl.String)
-
-    # Remplace les strings null par "", mais pas les numeric null
-    dff = dff.fill_null("")
-
-    # Ajout des liens vers les pages de détails
-    dff = add_links(dff)
-
-    # Ajout des liens vers les fichiers Open Data
-    if "sourceFile" in dff.columns:
-        dff = add_resource_link(dff)
-
-    # Formatage des montants
-    if height > 0:
-        dff = format_values(dff)
-
-    # Récupération des colonnes et tooltip
-    table_columns, tooltip = setup_table_columns(dff)
-
-    dicts = dff.to_dicts()
-
-    # Propriétés du bouton de téléchargement
-    download_disabled, download_text, download_title = get_button_properties(height)
-
-    return (
-        dicts,
-        table_columns,
-        tooltip,
-        data_timestamp + 1,
-        nb_rows,
-        download_disabled,
-        download_text,
-        download_title,
-        trigger_cleanup,
-    )
-
-
-def prepare_dashboard_data(
-    lff: pl.LazyFrame,
-    dashboard_year,
-    dashboard_acheteur_id,
-    dashboard_acheteur_categorie,
-    dashboard_acheteur_departement_code,
-    dashboard_titulaire_id,
-    dashboard_titulaire_categorie,
-    dashboard_titulaire_departement_code,
-    dashboard_marche_type,
-    dashboard_marche_objet,
-    dashboard_marche_code_cpv,
-    dashboard_marche_considerations_sociales,
-    dashboard_marche_considerations_environnementales,
-    dashboard_marche_techniques,
-    dashboard_marche_innovant,
-    dashboard_marche_sous_traitance_declaree,
-    dashboard_montant_min=None,
-    dashboard_montant_max=None,
-) -> pl.LazyFrame:
-    if dashboard_year:
-        lff = lff.filter(pl.col("dateNotification").dt.year() == int(dashboard_year))
-    else:
-        lff = lff.filter(
-            pl.col("dateNotification") > (datetime.now() - timedelta(days=365))
-        )
-
-    if dashboard_acheteur_id:
-        lff = lff.filter(pl.col("acheteur_id").str.contains(dashboard_acheteur_id))
-    else:
-        if dashboard_acheteur_categorie:
-            lff = lff.filter(pl.col("acheteur_categorie") == dashboard_acheteur_categorie)
-        if dashboard_acheteur_departement_code:
-            lff = lff.filter(
-                pl.col("acheteur_departement_code").is_in(dashboard_acheteur_departement_code)
-            )
-
-    if dashboard_titulaire_id:
-        lff = lff.filter(pl.col("titulaire_id").str.contains(dashboard_titulaire_id))
-    else:
-        if dashboard_titulaire_categorie:
-            lff = lff.filter(pl.col("titulaire_categorie") == dashboard_titulaire_categorie)
-        if dashboard_titulaire_departement_code:
-            lff = lff.filter(
-                pl.col("titulaire_departement_code").is_in(dashboard_titulaire_departement_code)
-            )
-
-    if dashboard_marche_type:
-        lff = lff.filter(pl.col("type") == dashboard_marche_type)
-
-    if dashboard_marche_objet:
-        lff = lff.filter(pl.col("objet").str.contains(f"(?i){dashboard_marche_objet}"))
-
-    if dashboard_marche_code_cpv:
-        lff = lff.filter(pl.col("codeCPV").str.starts_with(dashboard_marche_code_cpv))
-
-    if dashboard_marche_innovant and dashboard_marche_innovant != "all":
-        lff = lff.filter(pl.col("marcheInnovant") == dashboard_marche_innovant)
-
-    if dashboard_marche_sous_traitance_declaree and dashboard_marche_sous_traitance_declaree != "all":
-        lff = lff.filter(pl.col("sousTraitanceDeclaree") == dashboard_marche_sous_traitance_declaree)
-
-    if dashboard_marche_techniques:
-        lff = lff.filter(
-            pl.col("techniques")
-            .str.split(", ")
-            .list.set_intersection(dashboard_marche_techniques)
-            .list.len()
-            > 0
-        )
-
-    if dashboard_marche_considerations_sociales:
-        lff = lff.filter(
-            pl.col("considerationsSociales")
-            .str.split(", ")
-            .list.set_intersection(dashboard_marche_considerations_sociales)
-            .list.len()
-            > 0
-        )
-
-    if dashboard_marche_considerations_environnementales:
-        lff = lff.filter(
-            pl.col("considerationsEnvironnementales")
-            .str.split(", ")
-            .list.set_intersection(dashboard_marche_considerations_environnementales)
-            .list.len()
-            > 0
-        )
-
-    if dashboard_montant_min is not None:
-        lff = lff.filter(pl.col("montant") >= dashboard_montant_min)
-
-    if dashboard_montant_max is not None:
-        lff = lff.filter(pl.col("montant") <= dashboard_montant_max)
-
-    return lff
-
-
-def get_button_properties(height):
-    if height > 65000:
-        download_disabled = True
-        download_text = "Téléchargement désactivé au-delà de 65 000 lignes"
-        download_title = " Ajoutez des filtres pour réduire le nombre de lignes, Excel ne supporte pas d'avoir plus de 65 000 URLs dans une même feuille de calcul."
-    elif height == 0:
-        download_disabled = True
-        download_text = "Pas de données à télécharger"
-        download_title = ""
-    else:
-        download_disabled = False
-        download_text = "Télécharger au format Excel"
-        download_title = "Télécharger les données telles qu'affichées au format Excel"
-    return download_disabled, download_text, download_title
-
-
-def get_enum_values_as_dict(column_name):
-    try:
-        options = {}
-        for value in data_schema[column_name]["enum"]:
-            options[value] = value
-        return options
-    except KeyError:
-        return {"not_found": "not found"}
-
-
-def invert_columns(columns):
-    """
-    Renvoie les colonnes du schéma non spécifiées en paramètre. Utile pour passer d'une colonnes masquées à une liste de colonnes affichées, et vice versa.
-
-    :param columns:
-    :return:
-    """
-    inverted_columns = []
-    for column in schema.names():
-        if column not in columns:
-            inverted_columns.append(column)
-    return inverted_columns
-
-
-def make_org_jsonld(org_id, org_type, org_name=None, type_org_id="SIRET") -> dict:
-    org_types = {"acheteur": "GovernmentOrganization", "titulaire": "Organization"}
-    address = None
-    if type_org_id.lower() == "siret" and len(org_id) == 14:
-        annuaire_data = get_annuaire_data(org_id)
-        annuaire_address = annuaire_data["matching_etablissements"][0]
-        code_postal = annuaire_address["code_postal"]
-        commune = annuaire_address["libelle_commune"]
-
-        address = (
-            {
-                "@type": "PostalAddress",
-                "streetAddress": annuaire_address.get("adresse", "")
-                .replace(code_postal, "")
-                .replace(commune, "")
-                .strip(),
-                "addressLocality": commune,
-                "postalCode": code_postal,
-                "addressCountry": "FR",
-            },
-        )
-
-    jsonld = {
-        "@type": org_types[org_type],
-        "name": org_name,
-        "url": f"https://decp.info/{org_type}s/{org_id}",
-        "sameAs": f"https://annuaire-entreprises.data.gouv.fr/etablissement/{org_id}",
-        "identifier": {
-            "@type": "PropertyValue",
-            "propertyID": type_org_id.lower(),
-            "value": org_id,
-        },
-    }
-
-    if address:
-        jsonld["address"] = address
-
-    return jsonld
-
-
-df: pl.DataFrame = get_decp_data()
-schema = df.collect_schema()
-
-df_acheteurs = get_org_data(df, "acheteur")
-df_titulaires = get_org_data(df, "titulaire")
-df_acheteurs_departement: pl.DataFrame = (
-    df_acheteurs.select(["acheteur_id", "acheteur_nom", "acheteur_departement_code"])
-    .unique()
-    .sort("acheteur_nom")
-)
-df_titulaires_departement: pl.DataFrame = (
-    df_titulaires.select(
-        ["titulaire_id", "titulaire_nom", "titulaire_departement_code"]
-    )
-    .unique()
-    .sort("titulaire_nom")
-)
-df_acheteurs_marches: pl.DataFrame = (
-    df.select("uid", "objet", "acheteur_id").unique().sort("acheteur_id")
-)
-df_titulaires_marches: pl.DataFrame = (
-    df.select("uid", "objet", "titulaire_id").unique().sort("titulaire_id")
-)
-
-departements = get_departements()
-departements_geojson = get_departements_geojson()
-domain_name = (
-    "test.decp.info" if os.getenv("DEVELOPMENT").lower() == "true" else "decp.info"
-)
-meta_content = {
-    "image_url": f"https://{domain_name}/assets/decp.info.png",
-    "title": "decp.info - exploration des marchés publics français",
-    "description": (
-        "Explorez et analysez les données des marchés publics français avec cet outil libre et gratuit. "
-        "Pour une commande publique accessible à toutes et tous."
-    ),
-}
-data_schema = get_data_schema()
-columns = df.columns

src/utils/__init__.py

@@ -0,0 +1,19 @@
+import logging
+import os
+
+logging.basicConfig(
+    format="%(asctime)s %(levelname)-8s %(message)s",
+    level=logging.INFO,
+    datefmt="%Y-%m-%d %H:%M:%S",
+)
+DEVELOPMENT = os.getenv("DEVELOPMENT", "False").lower() == "true"
+logger = logging.getLogger("decp.info")
+
+if DEVELOPMENT:
+    logger.setLevel(logging.DEBUG)
+
+DOMAIN_NAME = (
+    "test.decp.info"
+    if os.getenv("DEVELOPMENT", "False").lower() == "true"
+    else "decp.info"
+)

src/utils/cache.py

@@ -0,0 +1,4 @@
+from flask_caching import Cache
+
+# Isolé dans un fichier dédié pour éviter les imports circulaires
+cache = Cache()

src/utils/data.py

@@ -0,0 +1,214 @@
+import json
+import logging
+import os
+from collections import OrderedDict
+from datetime import datetime, timedelta
+
+import polars as pl
+from httpx import HTTPError, get
+
+from src.db import get_cursor, schema
+from src.utils import logger
+
+logging.getLogger("httpx").setLevel("WARNING")
+
+
+def get_annuaire_data(siret: str) -> dict:
+    url = f"https://recherche-entreprises.api.gouv.fr/search?q={siret}"
+    try:
+        response = get(url).raise_for_status()
+        response = response.json()["results"][0]
+    except (HTTPError, IndexError):
+        response = None
+        logger.warning("Could not fetch data from recherche-entreprises.api.")
+    return response
+
+
+def get_statistics() -> dict:
+    return (
+        get(
+            "https://www.data.gouv.fr/api/1/datasets/r/0ccf4a75-f3aa-4b46-8b6a-18aeb63e36df",
+            follow_redirects=True,
+        )
+        .raise_for_status()
+        .json()
+    )
+
+
+def get_departements() -> dict:
+    with open("data/departements.json", "rb") as f:
+        data = json.load(f)
+        return data
+
+
+def get_departements_geojson() -> dict:
+    with open("./data/departements-1000m.geojson") as f:
+        geojson = json.load(f)
+
+    # Ajout de feature.id
+    for f in geojson["features"]:
+        f["id"] = f["properties"]["code"]
+
+    return geojson
+
+
+def get_departement_region(code_postal):
+    if code_postal > "97000":
+        code_departement = code_postal[:3]
+    else:
+        code_departement = code_postal[:2]
+    nom_departement = DEPARTEMENTS[code_departement]["departement"]
+    nom_region = DEPARTEMENTS[code_departement]["region"]
+    return code_departement, nom_departement, nom_region
+
+
+def get_data_schema() -> dict:
+    # Récupération du schéma des données tabulaires
+    path = os.getenv("DATA_SCHEMA_PATH")
+    if path.startswith("http"):
+        original_schema: dict = get(
+            os.getenv("DATA_SCHEMA_PATH"), follow_redirects=True
+        ).json()
+    elif os.path.exists(path):
+        with open(path) as f:
+            original_schema: dict = json.load(f)
+    else:
+        raise Exception(f"Chemin vers le schéma invalide: {path}")
+
+    new_schema = OrderedDict()
+
+    for col in original_schema["fields"]:
+        new_schema[col["name"]] = col
+
+    return new_schema
+
+
+def prepare_dashboard_data(
+    lff: pl.LazyFrame,
+    dashboard_year=None,
+    dashboard_acheteur_id=None,
+    dashboard_acheteur_categorie=None,
+    dashboard_acheteur_departement_code=None,
+    dashboard_titulaire_id=None,
+    dashboard_titulaire_categorie=None,
+    dashboard_titulaire_departement_code=None,
+    dashboard_marche_type=None,
+    dashboard_marche_objet=None,
+    dashboard_marche_code_cpv=None,
+    dashboard_marche_considerations_sociales=None,
+    dashboard_marche_considerations_environnementales=None,
+    dashboard_marche_techniques=None,
+    dashboard_marche_innovant=None,
+    dashboard_marche_sous_traitance_declaree=None,
+    dashboard_montant_min=None,
+    dashboard_montant_max=None,
+) -> pl.LazyFrame:
+    if dashboard_year:
+        lff = lff.filter(pl.col("dateNotification").dt.year() == int(dashboard_year))
+    else:
+        lff = lff.filter(
+            pl.col("dateNotification") > (datetime.now() - timedelta(days=365))
+        )
+
+    if dashboard_acheteur_id:
+        lff = lff.filter(pl.col("acheteur_id").str.contains(dashboard_acheteur_id))
+    else:
+        if dashboard_acheteur_categorie:
+            lff = lff.filter(
+                pl.col("acheteur_categorie") == dashboard_acheteur_categorie
+            )
+        if dashboard_acheteur_departement_code:
+            lff = lff.filter(
+                pl.col("acheteur_departement_code").is_in(
+                    dashboard_acheteur_departement_code
+                )
+            )
+
+    if dashboard_titulaire_id:
+        lff = lff.filter(pl.col("titulaire_id").str.contains(dashboard_titulaire_id))
+    else:
+        if dashboard_titulaire_categorie:
+            lff = lff.filter(
+                pl.col("titulaire_categorie") == dashboard_titulaire_categorie
+            )
+        if dashboard_titulaire_departement_code:
+            lff = lff.filter(
+                pl.col("titulaire_departement_code").is_in(
+                    dashboard_titulaire_departement_code
+                )
+            )
+
+    if dashboard_marche_type:
+        lff = lff.filter(pl.col("type") == dashboard_marche_type)
+
+    if dashboard_marche_objet:
+        lff = lff.filter(pl.col("objet").str.contains(f"(?i){dashboard_marche_objet}"))
+
+    if dashboard_marche_code_cpv:
+        lff = lff.filter(pl.col("codeCPV").str.starts_with(dashboard_marche_code_cpv))
+
+    if dashboard_marche_innovant and dashboard_marche_innovant != "all":
+        lff = lff.filter(pl.col("marcheInnovant") == dashboard_marche_innovant)
+
+    if (
+        dashboard_marche_sous_traitance_declaree
+        and dashboard_marche_sous_traitance_declaree != "all"
+    ):
+        lff = lff.filter(
+            pl.col("sousTraitanceDeclaree") == dashboard_marche_sous_traitance_declaree
+        )
+
+    if dashboard_marche_techniques:
+        lff = lff.filter(
+            pl.col("techniques")
+            .str.split(", ")
+            .list.set_intersection(dashboard_marche_techniques)
+            .list.len()
+            > 0
+        )
+
+    if dashboard_marche_considerations_sociales:
+        lff = lff.filter(
+            pl.col("considerationsSociales")
+            .str.split(", ")
+            .list.set_intersection(dashboard_marche_considerations_sociales)
+            .list.len()
+            > 0
+        )
+
+    if dashboard_marche_considerations_environnementales:
+        lff = lff.filter(
+            pl.col("considerationsEnvironnementales")
+            .str.split(", ")
+            .list.set_intersection(dashboard_marche_considerations_environnementales)
+            .list.len()
+            > 0
+        )
+
+    if dashboard_montant_min is not None:
+        lff = lff.filter(pl.col("montant") >= dashboard_montant_min)
+
+    if dashboard_montant_max is not None:
+        lff = lff.filter(pl.col("montant") <= dashboard_montant_max)
+
+    return lff
+
+
+def build_org_frame(org_type: str) -> pl.DataFrame:
+    org_cols = [
+        c
+        for c in schema.names()
+        if c.startswith(f"{org_type}_")
+        and c not in (f"{org_type}_latitude", f"{org_type}_longitude")
+    ]
+    select_list = ", ".join(org_cols)
+    group_list = ", ".join(org_cols)
+    sql = f'SELECT {select_list}, COUNT(*) AS "Marchés" FROM decp GROUP BY {group_list}'
+    return get_cursor().execute(sql).pl()
+
+
+DF_ACHETEURS = build_org_frame("acheteur")
+DF_TITULAIRES = build_org_frame("titulaire")
+DEPARTEMENTS = get_departements()
+DEPARTEMENTS_GEOJSON = get_departements_geojson()
+DATA_SCHEMA = get_data_schema()

src/utils/frontend.py

@@ -0,0 +1,27 @@
+from src.utils.data import DATA_SCHEMA
+
+
+def get_button_properties(height):
+    if height > 65000:
+        download_disabled = True
+        download_text = "Téléchargement désactivé au-delà de 65 000 lignes"
+        download_title = " Ajoutez des filtres pour réduire le nombre de lignes, Excel ne supporte pas d'avoir plus de 65 000 URLs dans une même feuille de calcul."
+    elif height == 0:
+        download_disabled = True
+        download_text = "Pas de données à télécharger"
+        download_title = ""
+    else:
+        download_disabled = False
+        download_text = "Télécharger au format Excel"
+        download_title = "Télécharger les données telles qu'affichées au format Excel"
+    return download_disabled, download_text, download_title
+
+
+def get_enum_values_as_dict(column_name):
+    try:
+        options = {}
+        for value in DATA_SCHEMA[column_name]["enum"]:
+            options[value] = value
+        return options
+    except KeyError:
+        return {"not_found": "not found"}

src/utils/search.py

@@ -0,0 +1,84 @@
+import polars as pl
+from unidecode import unidecode
+
+from src.utils.table import add_links
+from src.utils.tracking import track_search
+
+
+def search_org(dff: pl.DataFrame, query: str, org_type: str) -> pl.DataFrame:
+    """
+    Search in either 'acheteur' or 'titulaire' DataFrame.
+
+    :param dff: Polars DataFrame with acheteur or titulaire columns
+    :param query: User search string
+    :param org_type: 'acheteur' or 'titulaire'
+    :return: Filtered DataFrame with 'matches' column
+    """
+    if not query.strip():
+        return dff.select(pl.lit(False).alias("matches"))
+
+    # Enregistrement des recherche dans Matomo
+    track_search(query, "home_page_search")
+
+    # Normalize query
+    normalized_query = unidecode(query.strip()).upper()
+    tokens = [" " + t.strip() for t in normalized_query.split() if t.strip()]
+
+    # Define columns based on entity type
+    cols = [
+        f"{org_type}_id",
+        f"{org_type}_nom",
+        f"{org_type}_departement_nom",
+        f"{org_type}_departement_code",
+        f"{org_type}_commune_nom",
+    ]
+
+    # Concatenate all fields into one string per row
+    org_str = pl.concat_str(pl.lit(" "), pl.col(cols), separator=" ").str.replace(
+        "-", " "
+    )
+
+    # For each token, create a boolean column: True if token is found
+    token_matches = []
+    for token in tokens:
+        token_match = org_str.str.contains(token).alias(f"token_{token}")
+        token_matches.append(token_match)
+
+    # Count how many tokens match per row
+    match_score = pl.sum_horizontal(token_matches).alias("match_score")
+
+    # For each token, create a boolean column: True if token is found
+    token_matches = []
+    for token in tokens:
+        token_match = org_str.str.contains(token).alias(f"token_{token}")
+        token_matches.append(token_match)
+
+    # Sélection des colonnes
+    if org_type == "acheteur":
+        dff = dff.select(cols + ["Marchés"])
+    if org_type == "titulaire":
+        dff = dff.select(cols + ["Marchés", "titulaire_typeIdentifiant"])
+
+    # Apply and filter
+    dff = (
+        dff.with_columns(token_matches + [match_score])
+        .filter(pl.col("match_score") == len(tokens))
+        .drop([f"token_{token}" for token in tokens])
+    )
+
+    # Format result
+    dff = add_links(dff)
+    dff = dff.with_columns(
+        pl.concat_str(
+            pl.col(f"{org_type}_departement_nom"),
+            pl.lit(" ("),
+            pl.col(f"{org_type}_departement_code"),
+            pl.lit(")"),
+        ).alias("Département")
+    )
+
+    dff = dff.select(f"{org_type}_id", f"{org_type}_nom", "Département", "Marchés")
+    dff = dff.group_by(f"{org_type}_id", f"{org_type}_nom", "Département").sum()
+    dff = dff.sort("Marchés", descending=True)
+
+    return dff

src/utils/seo.py

@@ -0,0 +1,52 @@
+from src.utils import DOMAIN_NAME
+from src.utils.data import get_annuaire_data
+
+
+def make_org_jsonld(org_id, org_type, org_name=None, type_org_id="SIRET") -> dict:
+    org_types = {"acheteur": "GovernmentOrganization", "titulaire": "Organization"}
+    address = None
+    if type_org_id.lower() == "siret" and len(org_id) == 14:
+        annuaire_data = get_annuaire_data(org_id)
+        annuaire_address = annuaire_data["matching_etablissements"][0]
+        code_postal = annuaire_address["code_postal"]
+        commune = annuaire_address["libelle_commune"]
+
+        address = (
+            {
+                "@type": "PostalAddress",
+                "streetAddress": annuaire_address.get("adresse", "")
+                .replace(code_postal, "")
+                .replace(commune, "")
+                .strip(),
+                "addressLocality": commune,
+                "postalCode": code_postal,
+                "addressCountry": "FR",
+            },
+        )
+
+    jsonld = {
+        "@type": org_types[org_type],
+        "name": org_name,
+        "url": f"https://decp.info/{org_type}s/{org_id}",
+        "sameAs": f"https://annuaire-entreprises.data.gouv.fr/etablissement/{org_id}",
+        "identifier": {
+            "@type": "PropertyValue",
+            "propertyID": type_org_id.lower(),
+            "value": org_id,
+        },
+    }
+
+    if address:
+        jsonld["address"] = address
+
+    return jsonld
+
+
+META_CONTENT = {
+    "image_url": f"https://{DOMAIN_NAME}/assets/decp.info.png",
+    "title": "decp.info - exploration des marchés publics français",
+    "description": (
+        "Explorez et analysez les données des marchés publics français avec cet outil libre et gratuit. "
+        "Pour une commande publique accessible à toutes et tous."
+    ),
+}

src/utils/table.py

@@ -0,0 +1,504 @@
+import os
+import uuid
+
+import polars as pl
+from dash import no_update
+from polars import selectors as cs
+
+from src.db import query_marches, schema
+from src.utils import logger
+from src.utils.cache import cache
+from src.utils.data import DATA_SCHEMA
+from src.utils.frontend import get_button_properties
+from src.utils.tracking import track_search
+
+
+def split_filter_part(filter_part):
+    operators = [
+        ["s<", "<"],
+        ["s>", ">"],
+        ["i<", "<"],
+        ["i>", ">"],
+        ["icontains", "contains"],
+        # [" ", "contains"]
+    ]
+    logger.debug("filter part " + filter_part)
+    for operator_group in operators:
+        if operator_group[0] in filter_part:
+            name_part, value_part = filter_part.split(operator_group[0], 1)
+            name_part = name_part.strip()
+            value = value_part.strip()
+            name = name_part[name_part.find("{") + 1 : name_part.rfind("}")]
+            logger.debug("=> " + " ".join([name, operator_group[1], value]))
+
+            return name, operator_group[1], value
+
+    return [None] * 3
+
+
+def add_resource_link(dff: pl.DataFrame) -> pl.DataFrame:
+    dff = dff.with_columns(
+        (
+            '<a href="' + pl.col("sourceFile") + '">' + pl.col("sourceDataset") + "</a>"
+        ).alias("sourceDataset")
+    )
+    dff = dff.drop(["sourceFile"])
+    return dff
+
+
+def add_links(dff: pl.DataFrame):
+    for col in ["uid", "acheteur_nom", "titulaire_nom", "acheteur_id", "titulaire_id"]:
+        if col in dff.columns:
+            if col.startswith("titulaire_"):
+                detail_link = (
+                    '<a href = "/titulaires/'
+                    + pl.col("titulaire_id")
+                    + '">'
+                    + pl.col(col)
+                    + "</a>"
+                )
+                if col == "titulaire_nom":
+                    detail_link = (
+                        detail_link
+                        + ' <a href="/observatoire?titulaire_id='
+                        + pl.col("titulaire_id")
+                        + '" title="Voir dans l\'observatoire">📊</a>'
+                    )
+                dff = dff.with_columns(
+                    pl.when(
+                        pl.Expr.or_(
+                            pl.col("titulaire_typeIdentifiant").is_null(),
+                            pl.col("titulaire_typeIdentifiant") == "SIRET",
+                        )
+                    )
+                    .then(detail_link)
+                    .otherwise(pl.col(col))
+                    .alias(col)
+                )
+            if col.startswith("acheteur_"):
+                detail_link = (
+                    '<a href = "/acheteurs/'
+                    + pl.col("acheteur_id")
+                    + '">'
+                    + pl.col(col)
+                    + "</a>"
+                )
+                if col == "acheteur_nom":
+                    detail_link = (
+                        detail_link
+                        + ' <a href="/observatoire?acheteur_id='
+                        + pl.col("acheteur_id")
+                        + '" title="Voir dans l\'observatoire">📊</a>'
+                    )
+                dff = dff.with_columns(detail_link.alias(col))
+            if col == "uid":
+                dff = dff.with_columns(
+                    (
+                        '<a href = "/marches/'
+                        + pl.col("uid")
+                        + '">'
+                        + pl.col("uid")
+                        + "</a>"
+                    ).alias("uid")
+                )
+    return dff
+
+
+def add_links_in_dict(data: list[dict], org_type: str) -> list:
+    new_data = []
+    for marche in data:
+        org_id = marche[org_type + "_id"]
+        marche[org_type + "_nom"] = (
+            f'<a href="/{org_type}s/{org_id}">{marche[org_type + "_nom"]}</a>'
+        )
+        if marche.get("uid"):
+            marche["id"] = f'<a href="/marches/{marche["uid"]}">{marche["id"]}</a>'
+            marche["uid"] = f'<a href="/marches/{marche["uid"]}">{marche["uid"]}</a>'
+        new_data.append(marche)
+    return new_data
+
+
+def booleans_to_strings(lff: pl.LazyFrame) -> pl.LazyFrame:
+    """
+    Convert all boolean columns to string type.
+    """
+    lff = lff.with_columns(
+        pl.col(cs.Boolean)
+        .cast(pl.String)
+        .str.replace("true", "oui")
+        .str.replace("false", "non")
+    )
+    return lff
+
+
+def numbers_to_strings(lff: pl.LazyFrame) -> pl.LazyFrame:
+    """
+    Convert all numeric columns to string type.
+    """
+    lff = lff.with_columns(pl.col(pl.Float64, pl.Int16).cast(pl.String).fill_null(""))
+    return lff
+
+
+def dates_to_strings(lff: pl.LazyFrame, column: str) -> pl.LazyFrame:
+    """
+    Convert a date column to string type.
+    """
+    lff = lff.with_columns(pl.col(column).cast(pl.String).fill_null(""))
+    return lff
+
+
+def normalize_sort_by(sort_by) -> tuple:
+    if not sort_by:
+        return ()
+    return tuple((entry["column_id"], entry["direction"]) for entry in sort_by)
+
+
+def format_number(number) -> str:
+    number = "{:,}".format(number).replace(",", " ")
+    return number
+
+
+def unformat_montant(number: str) -> float:
+    number = number.replace(" €", "")
+    number = number.replace(" €", "").replace(" ", "")
+    number = number.replace(",", ".")
+    number = number.strip()
+    return float(number)
+
+
+def format_values(dff: pl.DataFrame) -> pl.DataFrame:
+    def format_montant(expr):
+        # https://stackoverflow.com/a/78636786
+        expr = expr.cast(pl.String)
+        expr = expr.str.splitn(".", 2)
+
+        num = expr.struct[0]
+        frac = expr.struct[1]
+
+        # Ajout des espaces
+        num = (
+            num.str.reverse()
+            .str.replace_all(r"\d{3}", "$0 ")
+            .str.reverse()
+            .str.replace(r"^ ", "")
+        )
+
+        frac: pl.Expr = (
+            pl.when(frac.is_not_null() & ~frac.is_in(["0"]))
+            .then("," + frac.str.head(2))
+            .otherwise(pl.lit(""))
+        )
+
+        montant: pl.Expr = (
+            pl.when((num + frac) == pl.lit(""))
+            .then(pl.lit(""))
+            .otherwise(num + frac + pl.lit(" €"))
+        )
+
+        return montant
+
+    def format_distance(expr):
+        expr = expr.cast(pl.String)
+        return pl.concat_str(expr, pl.lit(" km"))
+
+    if "montant" in dff.columns:
+        dff = dff.with_columns(pl.col("montant").pipe(format_montant).alias("montant"))
+    if "titulaire_distance" in dff.columns:
+        dff = dff.with_columns(
+            pl.col("titulaire_distance")
+            .pipe(format_distance)
+            .alias("titulaire_distance")
+        )
+
+    return dff
+
+
+def filter_table_data(lff: pl.LazyFrame, filter_query: str) -> pl.LazyFrame:
+    _schema = lff.collect_schema()
+    filtering_expressions = filter_query.split(" && ")
+    for filter_part in filtering_expressions:
+        col_name, operator, filter_value = split_filter_part(filter_part)
+        if not isinstance(col_name, str) or not isinstance(filter_value, str):
+            continue
+        col_type = str(_schema[col_name])
+        # logger.debug("filter_value:", filter_value)
+        # logger.debug("filter_value_type:", type(filter_value))
+        # logger.debug("operator:", operator)
+        # logger.debug("col_type:", col_type)
+
+        lff = lff.filter(pl.col(col_name).is_not_null())
+
+        if col_type == "Date":
+            # Convertir la colonne date en chaînes de caractères
+            lff = dates_to_strings(lff, col_name)
+            col_type = "String"
+        if col_type == "String":
+            lff = lff.filter(pl.col(col_name) != pl.lit(""))
+
+        elif col_type.startswith("Int") or col_type.startswith("Float"):
+            try:
+                filter_value = int(filter_value)
+            except ValueError:
+                logger.error(f"Invalid numeric filter value: {filter_value}")
+                continue
+
+        if operator in ("contains", "<", "<=", ">", ">="):
+            if operator == "<":
+                lff = lff.filter(pl.col(col_name) < filter_value)
+            elif operator == ">":
+                lff = lff.filter(pl.col(col_name) > filter_value)
+            elif operator == ">=":
+                lff = lff.filter(pl.col(col_name) >= filter_value)
+            elif operator == "<=":
+                lff = lff.filter(pl.col(col_name) <= filter_value)
+            elif operator == "contains":
+                if col_type in ["String", "Date"] and isinstance(filter_value, str):
+                    filter_value = filter_value.strip('"')
+                    if filter_value.endswith("*"):
+                        lff = lff.filter(
+                            pl.col(col_name).str.starts_with(filter_value[:-1])
+                        )
+                    elif filter_value.startswith("*"):
+                        lff = lff.filter(
+                            pl.col(col_name).str.ends_with(filter_value[1:])
+                        )
+                    else:
+                        lff = lff.filter(
+                            pl.col(col_name).str.contains("(?i)" + filter_value)
+                        )
+                elif col_type.startswith("Int") or col_type.startswith("Float"):
+                    lff = lff.filter(pl.col(col_name) == filter_value)
+                else:
+                    logger.error(f"Invalid column type: {col_type}")
+        else:
+            logger.error(f"Invalid operator: {operator}")
+
+        # elif operator == 'datestartswith':
+        # lff = lff.filter(pl.col(col_name).str.startswith(filter_value)")
+
+    return lff
+
+
+def sort_table_data(lff: pl.LazyFrame, sort_by: list) -> pl.LazyFrame:
+    lff = lff.sort(
+        [col["column_id"] for col in sort_by],
+        descending=[col["direction"] == "desc" for col in sort_by],
+        nulls_last=True,
+    )
+    logger.debug(sort_by)
+    return lff
+
+
+def setup_table_columns(
+    dff,
+    hideable: bool = True,
+    exclude: list | None = None,
+) -> tuple:
+    # Liste finale de colonnes
+    markdown_exceptions = ["montant", "titulaire_distance", "distance", "dureeMois"]
+    columns = []
+    tooltip = {}
+    for column_id in dff.columns:
+        if exclude and column_id in exclude:
+            continue
+        column_object = DATA_SCHEMA.get(column_id)
+        if column_object:
+            column_name = column_object.get("title")
+        else:
+            # Si le champ est un champ créé par erreur lors d'une jointure, on le skip
+            if column_id.endswith("_left") or column_id.endswith("_right"):
+                logger.warning(f"Champ innatendu : {column_id}")
+                continue
+            column_name = column_id
+            column_object = {"title": column_name, "description": ""}
+
+        presentation = "input" if column_id in markdown_exceptions else "markdown"
+
+        column = {
+            "name": column_name,
+            "id": column_id,
+            "presentation": presentation,
+            "type": "text",
+            "format": {"nully": "N/A"},
+            "hideable": hideable,
+        }
+        columns.append(column)
+        if column_object:
+            tooltip[column_id] = {
+                "value": f"""**{column_object.get("title")}** ({column_id})
+
+    """
+                + column_object.get("description", ""),
+                "type": "markdown",
+            }
+    return columns, tooltip
+
+
+def get_default_hidden_columns(page):
+    if page == "acheteur":
+        displayed_columns = [
+            "uid",
+            "objet",
+            "dateNotification",
+            "titulaire_id",
+            "titulaire_typeIdentifiant",
+            "titulaire_nom",
+            "titulaire_distance",
+            "montant",
+            "codeCPV",
+            "dureeRestanteMois",
+        ]
+    elif page == "titulaire":
+        displayed_columns = [
+            "uid",
+            "objet",
+            "dateNotification",
+            "acheteur_id",
+            "acheteur_nom",
+            "titulaire_distance",
+            "montant",
+            "codeCPV",
+            "dureeRestanteMois",
+        ]
+    elif page == "tableau":
+        displayed_columns = os.getenv("DISPLAYED_COLUMNS")
+    else:
+        displayed_columns = os.getenv("DISPLAYED_COLUMNS")
+        logger.warning(f"Invalid page: {page}")
+
+    hidden_columns = []
+
+    for col in schema.names():
+        if col in displayed_columns:
+            continue
+        else:
+            hidden_columns.append(col)
+    return hidden_columns
+
+
+@cache.memoize()
+def _load_filter_sort_postprocess(filter_query, sort_by_key):
+    logger.debug(
+        f"Cache miss — recomputing for filter={filter_query!r} sort={sort_by_key!r}"
+    )
+
+    lff: pl.LazyFrame = query_marches().lazy()
+
+    if filter_query:
+        lff = filter_table_data(lff, filter_query)
+
+    if sort_by_key:
+        sort_by = [
+            {"column_id": col, "direction": direction} for col, direction in sort_by_key
+        ]
+        lff = sort_table_data(lff, sort_by)
+
+    dff = table_postprocess(lff)
+
+    return dff
+
+
+def table_postprocess(lff) -> pl.DataFrame:
+    lff = lff.cast(pl.String)
+    lff = lff.fill_null("")
+    dff: pl.DataFrame = lff.collect()
+    dff = add_links(dff)
+    if "sourceFile" in dff.columns:
+        dff = add_resource_link(dff)
+    if dff.height > 0:
+        dff = format_values(dff)
+    return dff
+
+
+def prepare_table_data(
+    data, data_timestamp, filter_query, page_current, page_size, sort_by, source_table
+):
+    """
+    Fonction de préparation des données pour les datatables, afin de permettre une gestion fine des logiques,
+    notamment pour les filtres et les tris.
+    :param data
+    :param data_timestamp:
+    :param filter_query:
+    :param page_current:
+    :param page_size:
+    :param sort_by:
+    :param source_table:
+    :return:
+    """
+    logger.debug(" + + + + + + + + + + + + + + + + + + ")
+
+    if filter_query:
+        track_search(filter_query, source_table)
+
+    trigger_cleanup = no_update if source_table == "tableau" else str(uuid.uuid4())
+
+    if data is None:
+        sort_by_key = normalize_sort_by(sort_by)
+        dff: pl.DataFrame = _load_filter_sort_postprocess(
+            filter_query=filter_query, sort_by_key=sort_by_key
+        )
+    else:
+        if isinstance(data, list):
+            lff: pl.LazyFrame = pl.LazyFrame(
+                data, strict=False, infer_schema_length=5000
+            )
+        elif isinstance(data, pl.LazyFrame):
+            lff = data
+        else:
+            lff = query_marches().lazy()
+
+        if filter_query:
+            lff = filter_table_data(lff, filter_query)
+
+        if sort_by and len(sort_by) > 0:
+            lff = sort_table_data(lff, sort_by)
+
+        dff: pl.DataFrame = table_postprocess(lff)
+
+    height = dff.height
+
+    if height > 0:
+        nb_rows = (
+            f"{format_number(height)} lignes "
+            f"({format_number(dff.select('uid').unique().height)} marchés)"
+        )
+    else:
+        nb_rows = "0 lignes (0 marchés)"
+
+    start_row = page_current * page_size
+    dff = dff.slice(start_row, page_size)
+
+    table_columns, tooltip = setup_table_columns(dff)
+
+    dicts = dff.to_dicts()
+
+    download_disabled, download_text, download_title = get_button_properties(height)
+
+    return (
+        dicts,
+        table_columns,
+        tooltip,
+        data_timestamp + 1,
+        nb_rows,
+        download_disabled,
+        download_text,
+        download_title,
+        trigger_cleanup,
+    )
+
+
+def invert_columns(columns):
+    """
+    Renvoie les colonnes du schéma non spécifiées en paramètre. Utile pour passer d'une colonnes masquées à une liste de colonnes affichées, et vice versa.
+
+    :param columns:
+    :return:
+    """
+    inverted_columns = []
+    for column in schema.names():
+        if column not in columns:
+            inverted_columns.append(column)
+    return inverted_columns
+
+
+COLUMNS = schema.names()

src/utils/tracking.py

@@ -0,0 +1,30 @@
+import os
+import uuid
+from time import localtime
+
+from httpx import post
+
+from src.utils import DEVELOPMENT
+
+
+def track_search(query, category):
+    if len(query) >= 4 and not DEVELOPMENT and os.getenv("MATOMO_DOMAIN"):
+        url = "https://decp.info"
+        params = {
+            "idsite": os.getenv("MATOMO_ID_SITE"),
+            "url": url,
+            "rec": "1",
+            "action_name": "search" if category == "home_page_search" else "filter",
+            "search_cat": category,
+            "rand": uuid.uuid4().hex,
+            "apiv": "1",
+            "h": localtime().tm_hour,
+            "m": localtime().tm_min,
+            "s": localtime().tm_sec,
+            "search": query,
+            "token_auth": os.getenv("MATOMO_TOKEN"),
+        }
+        post(
+            url=f"https://{os.getenv('MATOMO_DOMAIN')}/matomo.php",
+            params=params,
+        ).raise_for_status()

tests/conftest.py

@@ -1,5 +1,6 @@
 import datetime
 import os
+from pathlib import Path
 
 import polars as pl
 import pytest
@@ -41,12 +42,19 @@ def test_data():
             "titulaire_categorie": "PME",
         }
     ]
-    path = "tests/test.parquet"
+    parquet_path = Path(os.path.abspath("tests/test.parquet"))
-    path = os.path.abspath(path)
+    db_path = parquet_path.parent / "decp.duckdb"
-    print(f"Writing test data to: {path}")  # <-- This will show you the real path
+    print(f"Writing test data to: {parquet_path}")
 
-    pl.DataFrame(data).write_parquet("tests/test.parquet")
+    pl.DataFrame(data).write_parquet(parquet_path)
-    yield path
+
+    # Remove any stale DuckDB from a previous run so src.db rebuilds from
+    # the freshly-written parquet at import time.
+    for artifact in (db_path, db_path.with_suffix(".duckdb.tmp")):
+        if artifact.exists():
+            artifact.unlink()
+
+    yield str(parquet_path)
 
 
 def pytest_setup_options():

tests/test_db.py

@@ -0,0 +1,267 @@
+import datetime
+import os
+import time
+
+import polars as pl
+import pytest
+
+from src.db import should_rebuild
+
+
+@pytest.fixture
+def parquet_and_db(tmp_path, monkeypatch):
+    parquet = tmp_path / "source.parquet"
+    db = tmp_path / "decp.duckdb"
+    parquet.write_bytes(b"fake parquet content")
+    monkeypatch.delenv("REBUILD_DUCKDB", raising=False)
+    monkeypatch.delenv("DEVELOPMENT", raising=False)
+    return parquet, db
+
+
+def test_should_rebuild_when_db_missing(parquet_and_db):
+    parquet, db = parquet_and_db
+    assert should_rebuild(db, parquet) is True
+
+
+def test_should_rebuild_prod_when_parquet_newer(parquet_and_db, monkeypatch):
+    parquet, db = parquet_and_db
+    db.write_bytes(b"x")
+    parquet.touch()
+    now = time.time()
+    os.utime(db, (now, now))
+    os.utime(parquet, (now + 10, now + 10))
+    monkeypatch.setenv("DEVELOPMENT", "false")
+    assert should_rebuild(db, parquet) is True
+
+
+def test_should_not_rebuild_prod_when_parquet_older(parquet_and_db, monkeypatch):
+    parquet, db = parquet_and_db
+    parquet.touch()
+    db.write_bytes(b"x")
+    now = time.time()
+    os.utime(parquet, (now, now))
+    os.utime(db, (now + 10, now + 10))
+    monkeypatch.setenv("DEVELOPMENT", "false")
+    assert should_rebuild(db, parquet) is False
+
+
+def test_should_not_rebuild_dev_even_when_parquet_newer(parquet_and_db, monkeypatch):
+    parquet, db = parquet_and_db
+    db.write_bytes(b"x")
+    parquet.touch()
+    now = time.time()
+    os.utime(db, (now, now))
+    os.utime(parquet, (now + 10, now + 10))
+    monkeypatch.setenv("DEVELOPMENT", "true")
+    monkeypatch.delenv("REBUILD_DUCKDB", raising=False)
+    assert should_rebuild(db, parquet) is False
+
+
+def test_should_rebuild_dev_when_rebuild_forced(parquet_and_db, monkeypatch):
+    parquet, db = parquet_and_db
+    db.write_bytes(b"x")
+    parquet.touch()
+    now = time.time()
+    os.utime(db, (now, now))
+    os.utime(parquet, (now + 10, now + 10))
+    monkeypatch.setenv("DEVELOPMENT", "true")
+    monkeypatch.setenv("REBUILD_DUCKDB", "true")
+    assert should_rebuild(db, parquet) is True
+
+
+@pytest.fixture
+def built_db(tmp_path, monkeypatch):
+    """Build a DuckDB from a small Polars frame written as parquet."""
+    parquet_path = tmp_path / "source.parquet"
+    db_path = tmp_path / "decp.duckdb"
+
+    data = pl.DataFrame(
+        [
+            {
+                "uid": "1",
+                "id": "1",
+                "objet": "Travaux",
+                "acheteur_id": "123",
+                "acheteur_nom": "ACHETEUR 1",
+                "acheteur_departement_code": "75",
+                "acheteur_departement_nom": "Paris",
+                "acheteur_commune_nom": "Paris",
+                "titulaire_commune_nom": "Paris",
+                "titulaire_departement_nom": "Paris",
+                "titulaire_id": "345",
+                "titulaire_nom": "TITULAIRE 1",
+                "titulaire_departement_code": "35",
+                "titulaire_typeIdentifiant": "SIRET",
+                "montant": 1000.0,
+                "dateNotification": datetime.date(2025, 1, 1),
+                "donneesActuelles": True,
+                "marcheInnovant": True,
+            },
+            {
+                "uid": "2",
+                "id": "2",
+                "objet": "Études",
+                "acheteur_id": "123",
+                "acheteur_nom": "ACHETEUR 1",
+                "acheteur_departement_code": "75",
+                "acheteur_departement_nom": "Paris",
+                "acheteur_commune_nom": "Paris",
+                "titulaire_commune_nom": "Paris",
+                "titulaire_departement_nom": "Paris",
+                "titulaire_id": "567",
+                "titulaire_nom": None,
+                "titulaire_departement_code": "75",
+                "titulaire_typeIdentifiant": "SIRET",
+                "montant": 500.0,
+                "dateNotification": datetime.date(2024, 6, 1),
+                "donneesActuelles": True,
+                "marcheInnovant": False,
+            },
+            {
+                "uid": "3",
+                "id": "3",
+                "objet": "Ancien",
+                "acheteur_id": "A2",
+                "acheteur_nom": None,
+                "acheteur_departement_code": "13",
+                "acheteur_departement_nom": "Paris",
+                "acheteur_commune_nom": "Paris",
+                "titulaire_commune_nom": "Paris",
+                "titulaire_departement_nom": "Paris",
+                "titulaire_id": "T3",
+                "titulaire_nom": "Autre",
+                "titulaire_departement_code": "13",
+                "titulaire_typeIdentifiant": "SIRET",
+                "montant": 100.0,
+                "dateNotification": datetime.date(2023, 1, 1),
+                "donneesActuelles": False,  # must be filtered out
+                "marcheInnovant": False,
+            },
+        ]
+    )
+    data.write_parquet(parquet_path)
+    monkeypatch.setenv("DATA_FILE_PARQUET_PATH", str(parquet_path))
+
+    from src.db import build_database
+
+    build_database(db_path, parquet_path)
+    return db_path
+
+
+def test_build_filters_donnees_actuelles(built_db):
+    import duckdb
+
+    with duckdb.connect(str(built_db), read_only=True) as c:
+        rows = c.execute("SELECT uid FROM decp ORDER BY uid").fetchall()
+    assert [r[0] for r in rows] == ["1", "2"]
+
+
+def test_build_converts_booleans_to_oui_non(built_db):
+    import duckdb
+
+    with duckdb.connect(str(built_db), read_only=True) as c:
+        values = c.execute("SELECT marcheInnovant FROM decp ORDER BY uid").fetchall()
+    assert [v[0] for v in values] == ["oui", "non"]
+
+
+def test_build_replaces_null_org_names(built_db):
+    import duckdb
+
+    with duckdb.connect(str(built_db), read_only=True) as c:
+        titulaire_2 = c.execute(
+            "SELECT titulaire_nom FROM decp WHERE uid = '2'"
+        ).fetchone()
+    assert titulaire_2[0] == "[Identifiant non reconnu dans la base INSEE]"
+
+
+def test_build_creates_derived_tables(built_db):
+    import duckdb
+
+    with duckdb.connect(str(built_db), read_only=True) as c:
+        tables = {r[0] for r in c.execute("SHOW TABLES").fetchall()}
+    assert {
+        "decp",
+        "acheteurs_marches",
+        "titulaires_marches",
+        "acheteurs_departement",
+        "titulaires_departement",
+    } <= tables
+
+
+def test_query_marches_returns_polars_frame(built_db, monkeypatch):
+    monkeypatch.setenv(
+        "DATA_FILE_PARQUET_PATH", str(built_db.parent / "source.parquet")
+    )
+    # Force src.db to load pointing at this test DB.
+    import importlib
+
+    import src.db
+
+    importlib.reload(src.db)
+    from src.db import query_marches
+
+    frame = query_marches("acheteur_id = ?", ("123",))
+    assert isinstance(frame, pl.DataFrame)
+    assert frame.height == 2
+    assert set(frame["uid"].to_list()) == {"1", "2"}
+
+
+def test_concurrent_build_serialized(tmp_path):
+    """Multiple threads calling _ensure_database must serialize via flock.
+
+    Only one should actually build; others wait, see the fresh DB, and skip.
+    No tmp file should leak. No exceptions should occur.
+    """
+    import fcntl
+    import threading
+
+    import src.db as db
+
+    # Set up source parquet
+    parquet_path = tmp_path / "src.parquet"
+    df = pl.DataFrame(
+        {
+            "uid": ["A"],
+            "donneesActuelles": [True],
+            "dateNotification": ["2024-01-01"],
+            "objet": ["Test"],
+            "acheteur_id": ["a1"],
+            "acheteur_nom": ["A1"],
+            "titulaire_id": ["t1"],
+            "titulaire_nom": ["T1"],
+            "acheteur_departement_code": ["75"],
+            "titulaire_departement_code": ["75"],
+            "montant": [1000.0],
+            "dureeMois": [12],
+        }
+    )
+    df.write_parquet(parquet_path)
+
+    db_path = tmp_path / "decp.duckdb"
+    lock_path = db_path.with_suffix(".duckdb.lock")
+    tmp_path_artifact = db_path.with_suffix(".duckdb.tmp")
+
+    errors: list[BaseException] = []
+
+    def worker():
+        try:
+            # Mirror the locking logic in _ensure_database
+            with open(lock_path, "w") as lf:
+                fcntl.flock(lf.fileno(), fcntl.LOCK_EX)
+                try:
+                    if db.should_rebuild(db_path, parquet_path):
+                        db.build_database(db_path, parquet_path)
+                finally:
+                    fcntl.flock(lf.fileno(), fcntl.LOCK_UN)
+        except BaseException as exc:
+            errors.append(exc)
+
+    threads = [threading.Thread(target=worker) for _ in range(3)]
+    for t in threads:
+        t.start()
+    for t in threads:
+        t.join()
+
+    assert errors == []
+    assert db_path.exists()
+    assert not tmp_path_artifact.exists()

tests/test_main.py

@@ -52,7 +52,6 @@ def test_002_filter_persistence(dash_duo: DashComposite):
         return _filter_input
 
     for page in ["tableau", "acheteurs/123", "titulaires/345"]:
-        print("page:", page)
         filter_input = open_page_and_check_filter_input()
         filter_input.send_keys("11")  # a UID that doesn't exist
         filter_input.send_keys(Keys.ENTER)
@@ -91,7 +90,7 @@ def test_003_tableau_download(dash_duo: DashComposite):
 def test_004_add_links_observatoire_acheteur():
     import polars as pl
 
-    from src.utils import add_links
+    from src.utils.table import add_links
 
     dff = pl.DataFrame(
         {
@@ -116,7 +115,7 @@ def test_004_add_links_observatoire_acheteur():
 def test_005_add_links_observatoire_titulaire():
     import polars as pl
 
-    from src.utils import add_links
+    from src.utils.table import add_links
 
     dff = pl.DataFrame(
         {
@@ -219,9 +218,7 @@ def test_008_search_to_observatoire(dash_duo: DashComposite):
 def test_010_observatoire_montant_filter():
     import datetime
 
-    import polars as pl
+    from src.utils.data import prepare_dashboard_data
-
-    from src.utils import prepare_dashboard_data
 
     data = pl.DataFrame(
         {

tests/test_table.py

@@ -0,0 +1,305 @@
+import polars as pl
+import pytest
+
+
+@pytest.fixture
+def sample_lff():
+    """Small LazyFrame with the columns needed by add_links / format_values."""
+    return pl.LazyFrame(
+        [
+            {
+                "uid": "u1",
+                "id": "u1",
+                "acheteur_id": "12345678900011",
+                "acheteur_nom": "Mairie de Test",
+                "titulaire_id": "98765432100022",
+                "titulaire_nom": "Entreprise Test",
+                "titulaire_typeIdentifiant": "SIRET",
+                "objet": "Travaux divers",
+                "montant": 12500.0,
+                "dateNotification": "2025-03-15",
+                "codeCPV": "45000000",
+                "dureeRestanteMois": 6,
+                "titulaire_distance": 42.0,
+            }
+        ]
+    )
+
+
+def test_table_module_imports():
+    from src.utils import table
+
+    assert hasattr(table, "prepare_table_data")
+
+
+def test_filter_table_data_does_not_call_track_search(monkeypatch, sample_lff):
+    from src.utils import table
+
+    calls = []
+    monkeypatch.setattr(table, "track_search", lambda *a, **kw: calls.append(a))
+
+    result = table.filter_table_data(sample_lff, "{objet} icontains travaux").collect()
+
+    assert calls == []
+    assert result.height == 1
+
+
+def test_normalize_sort_by_handles_empty():
+    from src.utils.table import normalize_sort_by
+
+    assert normalize_sort_by(None) == ()
+    assert normalize_sort_by([]) == ()
+
+
+def test_normalize_sort_by_returns_hashable_tuple():
+    from src.utils.table import normalize_sort_by
+
+    sort_by = [
+        {"column_id": "montant", "direction": "desc"},
+        {"column_id": "dateNotification", "direction": "asc"},
+    ]
+    key = normalize_sort_by(sort_by)
+
+    assert key == (("montant", "desc"), ("dateNotification", "asc"))
+    # Must be hashable so that flask-caching can build a cache key from it
+    hash(key)
+
+
+def test_normalize_sort_by_preserves_order():
+    """Order matters for sort: [A, B] != [B, A]."""
+    from src.utils.table import normalize_sort_by
+
+    a_then_b = normalize_sort_by(
+        [{"column_id": "a", "direction": "asc"}, {"column_id": "b", "direction": "asc"}]
+    )
+    b_then_a = normalize_sort_by(
+        [{"column_id": "b", "direction": "asc"}, {"column_id": "a", "direction": "asc"}]
+    )
+    assert a_then_b != b_then_a
+
+
+@pytest.fixture(scope="module")
+def flask_app():
+    """Minimal Flask app with SimpleCache so @cache.memoize() works in tests."""
+    from flask import Flask
+
+    from utils.cache import cache
+
+    app = Flask(__name__)
+    cache.init_app(app, config={"CACHE_TYPE": "SimpleCache"})
+    return app
+
+
+@pytest.fixture(autouse=True)
+def reset_cache(flask_app):
+    """Ensure the flask-caching backend is empty between tests so that
+    cache-hit assertions are meaningful. Falls back to no-op when no
+    Flask app context is active (NullCache)."""
+    from utils.cache import cache
+
+    with flask_app.app_context():
+        try:
+            cache.clear()
+        except (RuntimeError, AttributeError):
+            # No app context — cache is NullCache, nothing to clear
+            pass
+        yield
+
+
+def test_load_filter_sort_postprocess_returns_dataframe(
+    flask_app, monkeypatch, sample_lff
+):
+    from src.utils import table
+
+    monkeypatch.setattr(table, "query_marches", lambda: sample_lff.collect())
+
+    with flask_app.app_context():
+        df = table._load_filter_sort_postprocess(filter_query=None, sort_by_key=())
+
+    assert isinstance(df, pl.DataFrame)
+    assert df.height == 1
+    # All values must be strings after post-processing
+    for col in df.columns:
+        assert df.schema[col] == pl.String
+
+
+def test_load_filter_sort_postprocess_applies_filter(
+    flask_app, monkeypatch, sample_lff
+):
+    from src.utils import table
+
+    monkeypatch.setattr(table, "query_marches", lambda: sample_lff.collect())
+
+    with flask_app.app_context():
+        df = table._load_filter_sort_postprocess(
+            filter_query="{objet} icontains travaux", sort_by_key=()
+        )
+        assert df.height == 1
+
+        df_empty = table._load_filter_sort_postprocess(
+            filter_query="{objet} icontains nonexistent", sort_by_key=()
+        )
+        assert df_empty.height == 0
+
+
+def test_load_filter_sort_postprocess_adds_links(flask_app, monkeypatch, sample_lff):
+    from src.utils import table
+
+    monkeypatch.setattr(table, "query_marches", lambda: sample_lff.collect())
+
+    with flask_app.app_context():
+        df = table._load_filter_sort_postprocess(filter_query=None, sort_by_key=())
+    # add_links injects an <a href> wrapper around uid, acheteur_nom, titulaire_nom
+    assert "<a href" in df["uid"][0]
+    assert "<a href" in df["acheteur_nom"][0]
+    assert "<a href" in df["titulaire_nom"][0]
+
+
+def test_prepare_table_data_returns_expected_tuple(monkeypatch, flask_app, sample_lff):
+    from src.utils import table
+
+    monkeypatch.setattr(table, "query_marches", lambda: sample_lff.collect())
+
+    with flask_app.app_context():
+        result = table.prepare_table_data(
+            data=None,
+            data_timestamp=5,
+            filter_query=None,
+            page_current=0,
+            page_size=20,
+            sort_by=[],
+            source_table="tableau",
+        )
+
+    # Same arity as before: 9 outputs
+    assert len(result) == 9
+    dicts, columns, tooltip, ts, nb_rows, dl_disabled, dl_text, dl_title, cleanup = (
+        result
+    )
+    assert isinstance(dicts, list)
+    assert ts == 6  # data_timestamp + 1 must still increment
+    assert "1 lignes" in nb_rows
+
+
+def test_prepare_table_data_calls_track_search_on_filter(
+    monkeypatch, flask_app, sample_lff
+):
+    from src.utils import table
+
+    calls = []
+    monkeypatch.setattr(table, "query_marches", lambda: sample_lff.collect())
+    monkeypatch.setattr(table, "track_search", lambda *a, **kw: calls.append(a))
+
+    with flask_app.app_context():
+        table.prepare_table_data(
+            data=None,
+            data_timestamp=0,
+            filter_query="{objet} icontains travaux",
+            page_current=0,
+            page_size=20,
+            sort_by=[],
+            source_table="tableau",
+        )
+
+    assert calls == [("{objet} icontains travaux", "tableau")]
+
+
+def test_prepare_table_data_paginates_without_recomputing(
+    monkeypatch, flask_app, sample_lff
+):
+    """Two calls with same filter+sort but different pages must invoke
+    the inner heavy work only once."""
+    from src.utils import table
+
+    call_count = {"n": 0}
+    real_query = sample_lff.collect()
+
+    def counting_query():
+        call_count["n"] += 1
+        return real_query
+
+    monkeypatch.setattr(table, "query_marches", counting_query)
+
+    with flask_app.app_context():
+        # First call: cache miss
+        table.prepare_table_data(
+            data=None,
+            data_timestamp=0,
+            filter_query=None,
+            page_current=0,
+            page_size=10,
+            sort_by=[],
+            source_table="tableau",
+        )
+        first_count = call_count["n"]
+
+        # Second call, different page: cache hit, query_marches must NOT fire again
+        table.prepare_table_data(
+            data=None,
+            data_timestamp=0,
+            filter_query=None,
+            page_current=1,
+            page_size=10,
+            sort_by=[],
+            source_table="tableau",
+        )
+
+    assert call_count["n"] == first_count, (
+        "query_marches was called again — pagination triggered cache miss"
+    )
+
+
+def test_prepare_table_data_cleanup_trigger_for_non_tableau(
+    monkeypatch, flask_app, sample_lff
+):
+    """Non-tableau pages still get a fresh uuid trigger, not no_update."""
+    from dash import no_update
+
+    from src.utils import table
+
+    monkeypatch.setattr(table, "query_marches", lambda: sample_lff.collect())
+
+    with flask_app.app_context():
+        result = table.prepare_table_data(
+            data=None,
+            data_timestamp=0,
+            filter_query="{objet} icontains travaux",
+            page_current=0,
+            page_size=20,
+            sort_by=[],
+            source_table="acheteur",
+        )
+
+    cleanup = result[8]
+    assert cleanup is not no_update
+    assert isinstance(cleanup, str)
+    assert len(cleanup) >= 32  # uuid4 hex string
+
+
+def test_prepare_table_data_with_external_data_does_not_use_cache(
+    monkeypatch, flask_app, sample_lff
+):
+    """When a caller passes data (acheteur/titulaire/observatoire path),
+    bypass the memoized helper entirely."""
+    from src.utils import table
+
+    sentinel = {"called": False}
+
+    def should_not_be_called(*a, **kw):
+        sentinel["called"] = True
+        raise AssertionError("Memoized helper must not be called when data is provided")
+
+    monkeypatch.setattr(table, "_load_filter_sort_postprocess", should_not_be_called)
+
+    with flask_app.app_context():
+        table.prepare_table_data(
+            data=sample_lff,  # external LazyFrame
+            data_timestamp=0,
+            filter_query=None,
+            page_current=0,
+            page_size=20,
+            sort_by=[],
+            source_table="acheteur",
+        )
+
+    assert sentinel["called"] is False