Correction de setup_table_columns et autres

- Utilisation élargie de DuckDB au détriment de Polars => bien meilleure perf ([#72](https://github.com/ColinMaudry/decp.info/issues/72)

.github/workflows/deploy.yaml

@@ -20,7 +20,7 @@ jobs:
     environment: ${{ github.ref_name }}
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Set up SSH key
         run: |

.template.env

@@ -1,4 +1,5 @@
 DATA_FILE_PARQUET_PATH=https://www.data.gouv.fr/fr/datasets/r/11cea8e8-df3e-4ed1-932b-781e2635e432
+DUCKDB_PATH=./decp.duckdb
 PORT=8050
 DEVELOPMENT=True
 SOURCE_STATS_CSV_PATH="https://www.data.gouv.fr/api/1/datasets/r/8ded94de-3b80-4840-a5bb-7faad1c9c234"

CHANGELOG.md

@@ -1,4 +1,13 @@
-#### 2.7.2 (19 avril 2026)
+##### 2.7.4 (22 avril 2026)
+
+- Utilisation élargie de DuckDB au détriment de Polars => bien meilleure perf ([#72](https://github.com/ColinMaudry/decp.info/issues/72)
+
+##### 2.7.3 (20 avril 2026)
+
+- Mise en cache des vues tableau par ensemble de filtres et de tris
+- Résolution du bug d'écriture du fichier de vérouillage de la base de données
+
+##### 2.7.2 (19 avril 2026)
 
 - Chargement des données depuis une base DuckDB plutôt qu'en mémoire (plus de stabilité) ([#71](https://github.com/ColinMaudry/decp.info/issues/71))
 - Mise en cache des vue sur l'observatoire pour un chargement plus rapide (remise à zéro quotidienne)
@@ -6,7 +15,7 @@
 - Quelques corrections de bugs d'affichage
 - Refactorisation des fonctions utilitaires (`utils.py` approchait des 1 000 lignes)
 
-#### 2.7.1 (23 mars 2026)
+##### 2.7.1 (23 mars 2026)
 
 - Correction du partage de données filtrées entre dashboard et vue des données
 

CLAUDE.md

@@ -10,16 +10,25 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ### Setup
 
+Setting up the virtual environment:
+
 ```bash
-python -m venv .venv && source .venv/bin/activate
-pip install ".[dev]"
-cp template.env .env   # then customize .env
+python -m venv .venv # s'il n'existe pas déjà
+source .venv/bin/activate
+rtk pip install -U pip > /dev/null 2>&1
+rtk pip install -e . --group=dev
+```
+
+Environment variables:
+
+```bash
+cp .template.env .env   # then customize .env
 ```
 
 ### Development
 
 ```bash
-uv run run.py         # starts Dash with debug=True and hot reload
+python run.py         # starts Dash app
 ```
 
 ### Production
@@ -31,8 +40,8 @@ gunicorn app:server
 ### Tests
 
 ```bash
-uv run pytest                  # run all tests (Selenium-based integration tests)
-uv run pytest tests/test_main.py::test_001_logo_and_search   # run a single test
+rtk pytest                  # run all tests (some are Selenium-based integration tests)
+rtk pytest tests/test_main.py::test_001_logo_and_search   # run a single test
 ```
 
 Tests require a running Chrome/Chromium browser. They use `DashComposite` from `dash[testing]` with Selenium WebDriver.
@@ -42,12 +51,12 @@ Tests require a running Chrome/Chromium browser. They use `DashComposite` from `
 ### Multi-page Dash app
 
 - `src/app.py` — creates the Dash app instance, navbar, SEO endpoints (robots.txt, sitemap.xml), Matomo analytics
-- `src/pages/*.py` — each page registers itself with `@register_page()` and owns its own layout and callbacks
+- `src/pages/*.py` — each page registers itself with `@register_page()` and o.wns its own layout and callbacks
 - `run.py` — dev entry point; exports `server` (Flask) for gunicorn
 
 ### Module imports
 
-- always import modules from the app starting with `src.` (e.g. `src.utils.`, `src.pages.recherche`, etc.)
+- always import modules from the app starting with `src.` (e.g. `src.utils.`, `src.pages.recherche`, etc.), NOT `utils.cache` or `pages.observatoire`.
 
 ### Key pages
 
@@ -62,9 +71,9 @@ Tests require a running Chrome/Chromium browser. They use `DashComposite` from `
 
 ### Data layer
 
-- Data is stored as **Parquet** and loaded with **Polars** (fast columnar operations)
+- Data is stored as **Parquet** at rest, possibly in DuckDB, loaded in DuckDB, served from DuckDB for big queries and manipulated with **Polars** for the remaining steps
 - Path set via `DATA_FILE_PARQUET_PATH` env var; tests use `tests/test.parquet`
-- `src/utils.py` — filtering helpers, search (`search_org`), link generation, geographic data loading
+- `src/util/*.py` — helpers shared by other modules, search (`search_org`), link generation, geographic data loading
 - `src/callbacks.py` — shared Dash callbacks (e.g. `get_top_org_table`)
 - `src/figures.py` — chart and map components (Plotly Express, Dash Leaflet with marker clustering)
 - a Parquet file with production data is located at `../decp-processing/decp_prod.parquet` (~ 1,5 million records)

README.md

@@ -1,6 +1,6 @@
 # decp.info
 
-> v2.7.2
+> v2.7.4
 > Outil d'exploration et de téléchargement des données essentielles de la commande publique.
 
 => [decp.info](https://decp.info)
@@ -8,19 +8,15 @@
 ## Installation et lancement
 
 ```shell
-python -m venv .venv
-source .venv/bin/activate
-pip install .
-
 # Copie et personnalisation du .env
 cp template.env .env
 nano .env
 
 # Pour la production
-gunicorn app:server
+uv run gunicorn app:server
 
 # Pour avoir le debuggage et le hot reload
-python run.py
+uv run run.py
 ```
 
 ## Déploiement

docs/superpowers/plans/2026-04-19-tableau-prepare-table-data-cache.md

@@ -0,0 +1,838 @@
+# Tableau prepare_table_data Cache Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Make page navigation, sort changes, and repeated filter visits in the `/tableau` page near-instant by memoizing the expensive filter+sort+post-process pipeline inside `prepare_table_data`.
+
+**Architecture:** Extract a memoized inner function `_load_filter_sort_postprocess(filter_query, sort_by_key)` that performs the heavy work (load full data, filter, sort, collect, cast-to-string, fill-null, add HTML links, format values) and returns a fully post-processed Polars DataFrame. The outer `prepare_table_data` becomes a thin wrapper that handles non-deterministic side effects (`track_search`, `uuid.uuid4()` for cleanup trigger, `data_timestamp + 1`) and pagination. The memoized helper only runs when no `data` argument is passed (i.e., the Tableau path). Other callers (`acheteur`, `titulaire`, `observatoire`) keep the current uncached path because they pass an externally-provided LazyFrame that is not safely hashable for cache keys.
+
+**Tech Stack:** Polars (LazyFrame, DataFrame), Flask-Caching (`@cache.memoize()` on `FileSystemCache` already configured in `src/app.py:38`), pytest for unit tests.
+
+**Git**: the issue id is #72, add the reference in commit messages.
+
+---
+
+## Background and constraints
+
+Read these before starting; they explain why the design takes the shape it does.
+
+1. **Cache infrastructure is already wired.** `src/cache.py` defines `cache = Cache()`. `src/app.py:38-48` initializes it with `FileSystemCache`, default 24h timeout, `CACHE_THRESHOLD=300`. The cache directory is wiped on every restart (`rmtree` at `src/app.py:36`), so cache always starts empty.
+
+2. **Existing pattern to mirror.** `src/pages/observatoire.py:650-660` already uses `@cache.memoize()` plus a `_normalize_filter_params` helper that converts a dict of filters into a hashable tuple. This plan applies the same idiom to `sort_by` (which is a `list[dict]` from Dash DataTable).
+
+3. **Non-deterministic outputs that MUST stay outside the memoized function:**
+
+   - `data_timestamp + 1` (increments each call; would freeze if cached)
+   - `trigger_cleanup = str(uuid.uuid4())` (intentionally unique per call to fire the clientside filter-cleanup callback)
+   - `track_search(filter_query, source_table)` — Matomo HTTP POST, currently called inside `filter_table_data` at `src/utils/table.py:214`. Must fire on every user action including cache hits.
+
+4. **Tracking call site move.** `track_search` must move OUT of `filter_table_data` and into each caller, otherwise cache hits would silently skip Matomo tracking. Current callers of `filter_table_data` to update:
+
+   - `src/utils/table.py:402` (inside `prepare_table_data`)
+   - `src/pages/tableau.py:325` (`download_data` callback)
+   - `src/pages/acheteur.py:427` (`download_data_acheteur` callback)
+   - `src/pages/titulaire.py:443` (`download_data_titulaire` callback)
+
+5. **Why Tableau-only caching.** `prepare_table_data` is also called from `acheteur.py`, `titulaire.py`, `observatoire.py`. Those callers pass a pre-filtered LazyFrame or list-of-dicts as `data`. Hashing arbitrary LazyFrames or large lists for memoization is impractical. The fix gates on `data is None` (the Tableau path) and leaves the other paths byte-for-byte identical.
+
+6. **Cache key composition.** The memoized function takes only `(filter_query, sort_by_key)`. `page_current` and `page_size` are intentionally NOT in the key — pagination happens in the outer wrapper after retrieving the cached, fully post-processed frame. This means every page click and page-size change is a cache hit (the whole point of the change).
+
+7. **Pickling.** Flask-Caching pickles arguments to form keys and pickles return values to disk. Polars `DataFrame` pickles cleanly. `LazyFrame` does not — so the memoized function must `.collect()` before returning.
+
+8. **File path expectations.** All paths below are relative to repo root `/home/colin/git/decp.info`. Run all commands from there.
+
+---
+
+## File Structure
+
+- **Modify** `src/utils/table.py` — extract memoized helper, refactor `prepare_table_data`, remove `track_search` call from `filter_table_data`.
+- **Modify** `src/pages/tableau.py` — add explicit `track_search` call in `download_data`.
+- **Modify** `src/pages/acheteur.py` — add explicit `track_search` call in `download_data_acheteur`.
+- **Modify** `src/pages/titulaire.py` — add explicit `track_search` call in `download_data_titulaire`.
+- **Create** `tests/test_table.py` — unit tests for new helpers and refactored `prepare_table_data`.
+
+---
+
+## Task 1: Set up unit tests for table.py
+
+**Files:**
+
+- Create: `tests/test_table.py`
+
+This task scaffolds a non-Selenium pytest module so subsequent tasks can do TDD without booting a Dash server. The conftest already writes a small `tests/test.parquet` fixture (see `tests/conftest.py:10`); reuse it.
+
+- [ ] **Step 1: Write the failing test**
+
+Create `tests/test_table.py` with:
+
+```python
+import os
+
+import polars as pl
+import pytest
+
+
+@pytest.fixture
+def sample_lff():
+    """Small LazyFrame with the columns needed by add_links / format_values."""
+    return pl.LazyFrame(
+        [
+            {
+                "uid": "u1",
+                "id": "u1",
+                "acheteur_id": "12345678900011",
+                "acheteur_nom": "Mairie de Test",
+                "titulaire_id": "98765432100022",
+                "titulaire_nom": "Entreprise Test",
+                "titulaire_typeIdentifiant": "SIRET",
+                "objet": "Travaux divers",
+                "montant": 12500.0,
+                "dateNotification": "2025-03-15",
+                "codeCPV": "45000000",
+                "dureeRestanteMois": 6,
+                "titulaire_distance": 42.0,
+            }
+        ]
+    )
+
+
+def test_table_module_imports():
+    from src.utils import table
+
+    assert hasattr(table, "prepare_table_data")
+```
+
+- [ ] **Step 2: Run test to verify it passes (sanity check)**
+
+Run: `uv run pytest tests/test_table.py -v`
+Expected: PASS for `test_table_module_imports`. (Selenium is not invoked because no `dash_duo` fixture is used.)
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add tests/test_table.py
+git commit -m "test: scaffold unit tests for table utilities"
+```
+
+---
+
+## Task 2: Move track_search out of filter_table_data
+
+**Files:**
+
+- Modify: `src/utils/table.py:210-274` (remove `track_search` import usage at line 214)
+- Modify: `src/pages/tableau.py:317-334` (`download_data` callback)
+- Modify: `src/pages/acheteur.py:425-430` area (`download_data_acheteur` callback)
+- Modify: `src/pages/titulaire.py:441-446` area (`download_data_titulaire` callback)
+- Modify: `tests/test_table.py` (add a test that confirms `filter_table_data` no longer calls Matomo)
+
+`track_search` must move out so that the soon-to-be-memoized helper does not swallow tracking on cache hits. We do this BEFORE introducing caching so that the diff is small and verifiable on its own.
+
+- [ ] **Step 1: Write the failing test**
+
+Append to `tests/test_table.py`:
+
+```python
+def test_filter_table_data_does_not_call_track_search(monkeypatch, sample_lff):
+    from src.utils import table
+
+    calls = []
+    monkeypatch.setattr(table, "track_search", lambda *a, **kw: calls.append(a))
+
+    result = table.filter_table_data(
+        sample_lff, "{objet} icontains travaux", "tableau"
+    ).collect()
+
+    assert calls == []
+    assert result.height == 1
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `uv run pytest tests/test_table.py::test_filter_table_data_does_not_call_track_search -v`
+Expected: FAIL (`assert calls == []` fails because `filter_table_data` currently calls `track_search` at line 214).
+
+- [ ] **Step 3: Remove the track_search call from filter_table_data**
+
+Edit `src/utils/table.py` — find this block:
+
+```python
+def filter_table_data(
+    lff: pl.LazyFrame, filter_query: str, filter_source: str
+) -> pl.LazyFrame:
+    _schema = lff.collect_schema()
+    track_search(filter_query, filter_source)
+    filtering_expressions = filter_query.split(" && ")
+```
+
+Remove the `track_search(filter_query, filter_source)` line. Result:
+
+```python
+def filter_table_data(
+    lff: pl.LazyFrame, filter_query: str, filter_source: str
+) -> pl.LazyFrame:
+    _schema = lff.collect_schema()
+    filtering_expressions = filter_query.split(" && ")
+```
+
+The `filter_source` parameter remains in the signature (avoids changing all callers in this task). It becomes unused; that is acceptable since callers will pass it again later if needed. Do NOT remove the `from src.utils.tracking import track_search` import yet — `prepare_table_data` will use it in Task 5.
+
+- [ ] **Step 4: Add explicit track_search calls in download callbacks**
+
+In `src/pages/tableau.py`, find:
+
+```python
+def download_data(n_clicks, filter_query, sort_by, hidden_columns: list = None):
+    lff: pl.LazyFrame = query_marches().lazy()
+
+    # Les colonnes masquées sont supprimées
+    if hidden_columns:
+        lff = lff.drop(hidden_columns)
+
+    if filter_query:
+        lff = filter_table_data(lff, filter_query, "tab download")
+```
+
+Insert a `track_search` call so behavior is preserved. First add the import at the top of `src/pages/tableau.py` next to other `src.utils` imports:
+
+```python
+from src.utils.tracking import track_search
+```
+
+Then change the body:
+
+```python
+def download_data(n_clicks, filter_query, sort_by, hidden_columns: list = None):
+    lff: pl.LazyFrame = query_marches().lazy()
+
+    # Les colonnes masquées sont supprimées
+    if hidden_columns:
+        lff = lff.drop(hidden_columns)
+
+    if filter_query:
+        track_search(filter_query, "tab download")
+        lff = filter_table_data(lff, filter_query, "tab download")
+```
+
+Repeat the same pattern in `src/pages/acheteur.py` (search for `filter_table_data(lff, filter_query, "ach download")`):
+
+Add import:
+
+```python
+from src.utils.tracking import track_search
+```
+
+Wrap the call:
+
+```python
+    if filter_query:
+        track_search(filter_query, "ach download")
+        lff = filter_table_data(lff, filter_query, "ach download")
+```
+
+Repeat in `src/pages/titulaire.py` (search for `filter_table_data(lff, filter_query, "titu download")`):
+
+Add import:
+
+```python
+from src.utils.tracking import track_search
+```
+
+Wrap the call:
+
+```python
+    if filter_query:
+        track_search(filter_query, "titu download")
+        lff = filter_table_data(lff, filter_query, "titu download")
+```
+
+- [ ] **Step 5: Run test to verify it passes**
+
+Run: `uv run pytest tests/test_table.py::test_filter_table_data_does_not_call_track_search -v`
+Expected: PASS.
+
+- [ ] **Step 6: Run full unit test file to verify no regressions**
+
+Run: `uv run pytest tests/test_table.py -v`
+Expected: All tests in `test_table.py` PASS.
+
+- [ ] **Step 7: Commit**
+
+```bash
+git add src/utils/table.py src/pages/tableau.py src/pages/acheteur.py src/pages/titulaire.py tests/test_table.py
+git commit -m "refactor: move track_search out of filter_table_data into callers"
+```
+
+---
+
+## Task 3: Add normalize_sort_by helper
+
+**Files:**
+
+- Modify: `src/utils/table.py` (add helper near other utility functions, e.g. after `dates_to_strings`)
+- Modify: `tests/test_table.py` (add tests)
+
+A cache key must be hashable. Dash DataTable's `sort_by` is a `list[dict]` like `[{"column_id": "montant", "direction": "asc"}, ...]`, which is not hashable. We mirror the `_normalize_filter_params` idiom from `src/pages/observatoire.py:650-657`.
+
+- [ ] **Step 1: Write the failing tests**
+
+Append to `tests/test_table.py`:
+
+```python
+def test_normalize_sort_by_handles_empty():
+    from src.utils.table import normalize_sort_by
+
+    assert normalize_sort_by(None) == ()
+    assert normalize_sort_by([]) == ()
+
+
+def test_normalize_sort_by_returns_hashable_tuple():
+    from src.utils.table import normalize_sort_by
+
+    sort_by = [
+        {"column_id": "montant", "direction": "desc"},
+        {"column_id": "dateNotification", "direction": "asc"},
+    ]
+    key = normalize_sort_by(sort_by)
+
+    assert key == (("montant", "desc"), ("dateNotification", "asc"))
+    # Must be hashable so that flask-caching can build a cache key from it
+    hash(key)
+
+
+def test_normalize_sort_by_preserves_order():
+    """Order matters for sort: [A, B] != [B, A]."""
+    from src.utils.table import normalize_sort_by
+
+    a_then_b = normalize_sort_by(
+        [{"column_id": "a", "direction": "asc"}, {"column_id": "b", "direction": "asc"}]
+    )
+    b_then_a = normalize_sort_by(
+        [{"column_id": "b", "direction": "asc"}, {"column_id": "a", "direction": "asc"}]
+    )
+    assert a_then_b != b_then_a
+```
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `uv run pytest tests/test_table.py -v -k normalize_sort_by`
+Expected: FAIL with `ImportError` for `normalize_sort_by`.
+
+- [ ] **Step 3: Implement normalize_sort_by**
+
+Edit `src/utils/table.py`. Add this function immediately after the `dates_to_strings` function (around line 148):
+
+```python
+def normalize_sort_by(sort_by) -> tuple:
+    """Convert Dash DataTable sort_by (list[dict]) into a hashable tuple
+    suitable for use as a cache key. Order is preserved because it determines
+    sort precedence."""
+    if not sort_by:
+        return ()
+    return tuple((entry["column_id"], entry["direction"]) for entry in sort_by)
+```
+
+- [ ] **Step 4: Run tests to verify they pass**
+
+Run: `uv run pytest tests/test_table.py -v -k normalize_sort_by`
+Expected: 3 PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add src/utils/table.py tests/test_table.py
+git commit -m "feat: add normalize_sort_by hashable cache-key helper"
+```
+
+---
+
+## Task 4: Extract memoized post-process helper
+
+**Files:**
+
+- Modify: `src/utils/table.py` (add `_load_filter_sort_postprocess`, decorate with `@cache.memoize()`, import `cache`)
+- Modify: `tests/test_table.py` (add tests)
+
+Introduce the function whose result will live in the FileSystemCache. Inputs: `(filter_query, sort_by_key)`. Output: a fully post-processed, unpaginated Polars DataFrame ready to slice and convert to dicts.
+
+This task does NOT yet wire the helper into `prepare_table_data` — that happens in Task 5. Splitting these tasks keeps each diff small and testable.
+
+- [ ] **Step 1: Write the failing tests**
+
+Append to `tests/test_table.py`:
+
+```python
+@pytest.fixture(autouse=True)
+def reset_cache():
+    """Ensure the flask-caching backend is empty between tests so that
+    cache-hit assertions are meaningful. Falls back to no-op when no
+    Flask app context is active (NullCache)."""
+    from utils.cache import cache
+
+    try:
+        cache.clear()
+    except RuntimeError:
+        # No app context — cache is NullCache, nothing to clear
+        pass
+    yield
+
+
+def test_load_filter_sort_postprocess_returns_dataframe(monkeypatch, sample_lff):
+    from src.utils import table
+
+    monkeypatch.setattr(
+        table, "query_marches", lambda: sample_lff.collect()
+    )
+
+    df = table._load_filter_sort_postprocess(filter_query=None, sort_by_key=())
+
+    assert isinstance(df, pl.DataFrame)
+    assert df.height == 1
+    # All values must be strings after post-processing
+    for col in df.columns:
+        assert df.schema[col] == pl.String
+
+
+def test_load_filter_sort_postprocess_applies_filter(monkeypatch, sample_lff):
+    from src.utils import table
+
+    monkeypatch.setattr(
+        table, "query_marches", lambda: sample_lff.collect()
+    )
+
+    df = table._load_filter_sort_postprocess(
+        filter_query="{objet} icontains travaux", sort_by_key=()
+    )
+    assert df.height == 1
+
+    df_empty = table._load_filter_sort_postprocess(
+        filter_query="{objet} icontains nonexistent", sort_by_key=()
+    )
+    assert df_empty.height == 0
+
+
+def test_load_filter_sort_postprocess_adds_links(monkeypatch, sample_lff):
+    from src.utils import table
+
+    monkeypatch.setattr(
+        table, "query_marches", lambda: sample_lff.collect()
+    )
+
+    df = table._load_filter_sort_postprocess(filter_query=None, sort_by_key=())
+    # add_links injects an <a href> wrapper around uid, acheteur_nom, titulaire_nom
+    assert "<a href" in df["uid"][0]
+    assert "<a href" in df["acheteur_nom"][0]
+    assert "<a href" in df["titulaire_nom"][0]
+```
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `uv run pytest tests/test_table.py -v -k load_filter_sort_postprocess`
+Expected: FAIL with `AttributeError: module 'src.utils.table' has no attribute '_load_filter_sort_postprocess'`.
+
+- [ ] **Step 3: Implement the helper**
+
+Edit `src/utils/table.py`. Add this import near the top, with the other `src.` imports:
+
+```python
+from utils.cache import cache
+```
+
+Then add the helper function. Place it ABOVE `prepare_table_data` (around line 370, just before `def prepare_table_data`):
+
+```python
+@cache.memoize()
+def _load_filter_sort_postprocess(filter_query, sort_by_key):
+    """Memoized core of the Tableau page pipeline.
+
+    Loads the full marchés dataset, applies filter and sort, materializes,
+    then runs the per-row post-processing (cast to string, fill nulls, add
+    HTML links, format values). Returns an unpaginated Polars DataFrame.
+
+    Inputs MUST be hashable: filter_query is str|None, sort_by_key is the
+    tuple produced by normalize_sort_by(). Pagination intentionally lives
+    in the outer wrapper so that page changes are cache hits.
+    """
+    logger.debug(f"Cache miss — recomputing for filter={filter_query!r} sort={sort_by_key!r}")
+
+    lff: pl.LazyFrame = query_marches().lazy()
+
+    if filter_query:
+        lff = filter_table_data(lff, filter_query, "tableau")
+
+    if sort_by_key:
+        sort_by = [
+            {"column_id": col, "direction": direction}
+            for col, direction in sort_by_key
+        ]
+        lff = sort_table_data(lff, sort_by)
+
+
+    # The remaining steps are cheap per-row operations that we run ONCE here
+    # so that pagination in the outer function is a pure slice + to_dicts.
+    lff = lff.cast(pl.String)
+    lff = lff.fill_null("")
+
+    dff: pl.DataFrame = lff.collect()
+
+    dff = add_links(dff)
+    if "sourceFile" in dff.columns:
+        dff = add_resource_link(dff)
+    if dff.height > 0:
+        dff = format_values(dff)
+
+    return dff
+```
+
+- [ ] **Step 4: Run tests to verify they pass**
+
+Run: `uv run pytest tests/test_table.py -v -k load_filter_sort_postprocess`
+Expected: 3 PASS.
+
+- [ ] **Step 5: Run the full test_table.py to catch regressions**
+
+Run: `uv run pytest tests/test_table.py -v`
+Expected: All PASS.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add src/utils/table.py tests/test_table.py
+git commit -m "feat: add memoized _load_filter_sort_postprocess helper"
+```
+
+---
+
+## Task 5: Wire the memoized helper into prepare_table_data
+
+**Files:**
+
+- Modify: `src/utils/table.py` — replace the body of `prepare_table_data` so the Tableau path uses the cache
+- Modify: `tests/test_table.py` — add tests covering the new flow
+
+The outer function keeps its signature unchanged so callers in `acheteur.py`, `titulaire.py`, `observatoire.py`, `tableau.py` need no updates. When `data is None` (the Tableau case), use the memoized helper; otherwise fall through to the original logic.
+
+- [ ] **Step 1: Write the failing tests**
+
+Append to `tests/test_table.py`:
+
+```python
+def test_prepare_table_data_returns_expected_tuple(monkeypatch, sample_lff):
+    from src.utils import table
+
+    monkeypatch.setattr(
+        table, "query_marches", lambda: sample_lff.collect()
+    )
+
+    result = table.prepare_table_data(
+        data=None,
+        data_timestamp=5,
+        filter_query=None,
+        page_current=0,
+        page_size=20,
+        sort_by=[],
+        source_table="tableau",
+    )
+
+    # Same arity as before: 9 outputs
+    assert len(result) == 9
+    dicts, columns, tooltip, ts, nb_rows, dl_disabled, dl_text, dl_title, cleanup = result
+    assert isinstance(dicts, list)
+    assert ts == 6  # data_timestamp + 1 must still increment
+    assert "1 lignes" in nb_rows
+
+
+def test_prepare_table_data_calls_track_search_on_filter(monkeypatch, sample_lff):
+    from src.utils import table
+
+    calls = []
+    monkeypatch.setattr(
+        table, "query_marches", lambda: sample_lff.collect()
+    )
+    monkeypatch.setattr(table, "track_search", lambda *a, **kw: calls.append(a))
+
+    table.prepare_table_data(
+        data=None,
+        data_timestamp=0,
+        filter_query="{objet} icontains travaux",
+        page_current=0,
+        page_size=20,
+        sort_by=[],
+        source_table="tableau",
+    )
+
+    assert calls == [("{objet} icontains travaux", "tableau")]
+
+
+def test_prepare_table_data_paginates_without_recomputing(monkeypatch, sample_lff):
+    """Two calls with same filter+sort but different pages must invoke
+    the inner heavy work only once."""
+    from src.utils import table
+
+    call_count = {"n": 0}
+    real_query = sample_lff.collect()
+
+    def counting_query():
+        call_count["n"] += 1
+        return real_query
+
+    monkeypatch.setattr(table, "query_marches", counting_query)
+
+    # First call: cache miss
+    table.prepare_table_data(
+        data=None,
+        data_timestamp=0,
+        filter_query=None,
+        page_current=0,
+        page_size=10,
+        sort_by=[],
+        source_table="tableau",
+    )
+    first_count = call_count["n"]
+
+    # Second call, different page: cache hit, query_marches must NOT fire again
+    table.prepare_table_data(
+        data=None,
+        data_timestamp=0,
+        filter_query=None,
+        page_current=1,
+        page_size=10,
+        sort_by=[],
+        source_table="tableau",
+    )
+
+    assert call_count["n"] == first_count, (
+        "query_marches was called again — pagination triggered cache miss"
+    )
+
+
+def test_prepare_table_data_cleanup_trigger_for_non_tableau(monkeypatch, sample_lff):
+    """Non-tableau pages still get a fresh uuid trigger, not no_update."""
+    from dash import no_update
+
+    from src.utils import table
+
+    monkeypatch.setattr(
+        table, "query_marches", lambda: sample_lff.collect()
+    )
+
+    result = table.prepare_table_data(
+        data=None,
+        data_timestamp=0,
+        filter_query="{objet} icontains travaux",
+        page_current=0,
+        page_size=20,
+        sort_by=[],
+        source_table="acheteur",
+    )
+
+    cleanup = result[8]
+    assert cleanup is not no_update
+    assert isinstance(cleanup, str)
+    assert len(cleanup) >= 32  # uuid4 hex string
+
+
+def test_prepare_table_data_with_external_data_does_not_use_cache(
+    monkeypatch, sample_lff
+):
+    """When a caller passes data (acheteur/titulaire/observatoire path),
+    bypass the memoized helper entirely."""
+    from src.utils import table
+
+    sentinel = {"called": False}
+
+    def should_not_be_called(*a, **kw):
+        sentinel["called"] = True
+        raise AssertionError("Memoized helper must not be called when data is provided")
+
+    monkeypatch.setattr(
+        table, "_load_filter_sort_postprocess", should_not_be_called
+    )
+
+    table.prepare_table_data(
+        data=sample_lff,  # external LazyFrame
+        data_timestamp=0,
+        filter_query=None,
+        page_current=0,
+        page_size=20,
+        sort_by=[],
+        source_table="acheteur",
+    )
+
+    assert sentinel["called"] is False
+```
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `uv run pytest tests/test_table.py -v -k prepare_table_data`
+Expected: At least the cache-hit (`paginates_without_recomputing`) and `track_search`-routing tests FAIL because the current `prepare_table_data` re-runs the full pipeline on every call and routes tracking through `filter_table_data` (which Task 2 already neutralized — so tracking would be lost without the new explicit call).
+
+- [ ] **Step 3: Refactor prepare_table_data**
+
+Edit `src/utils/table.py`. Replace the entire `prepare_table_data` function body with:
+
+```python
+def prepare_table_data(
+    data, data_timestamp, filter_query, page_current, page_size, sort_by, source_table
+):
+    """
+    Préparation des données pour les datatables.
+
+    Pour la page Tableau (data is None), le calcul lourd (chargement complet,
+    filtre, tri, post-traitement) est mémorisé via _load_filter_sort_postprocess.
+    Les changements de page deviennent ainsi des cache hits.
+
+    Pour les autres pages (data fourni), le chemin original est conservé : la
+    LazyFrame externe n'est pas hashable et le coût de filtre/tri y est déjà
+    minime puisque les données sont pré-restreintes.
+    """
+    logger.debug(" + + + + + + + + + + + + + + + + + + ")
+
+    # Side effect non-cacheable : le tracking doit firer sur chaque action
+    # utilisateur, y compris sur cache hit.
+    if filter_query:
+        track_search(filter_query, source_table)
+
+    # Trigger uuid pour les pages autres que tableau (clientside cleanup)
+    trigger_cleanup = (
+        no_update if source_table == "tableau" else str(uuid.uuid4())
+    )
+
+    if data is None:
+        # Tableau path : utilise le cache
+        sort_by_key = normalize_sort_by(sort_by)
+        dff: pl.DataFrame = _load_filter_sort_postprocess(
+            filter_query=filter_query, sort_by_key=sort_by_key
+        )
+    else:
+        # acheteur / titulaire / observatoire path : code original, non caché
+        if isinstance(data, list):
+            lff: pl.LazyFrame = pl.LazyFrame(
+                data, strict=False, infer_schema_length=5000
+            )
+        elif isinstance(data, pl.LazyFrame):
+            lff = data
+        else:
+            lff = query_marches().lazy()
+
+        if filter_query:
+            lff = filter_table_data(lff, filter_query, source_table)
+
+        if sort_by and len(sort_by) > 0:
+            lff = sort_table_data(lff, sort_by)
+
+        dff = lff.collect()
+        dff = dff.cast(pl.String)
+        dff = dff.fill_null("")
+        dff = add_links(dff)
+        if "sourceFile" in dff.columns:
+            dff = add_resource_link(dff)
+        if dff.height > 0:
+            dff = format_values(dff)
+
+    height = dff.height
+
+    if height > 0:
+        nb_rows = (
+            f"{format_number(height)} lignes "
+            f"({format_number(dff.select('uid').unique().height)} marchés)"
+        )
+    else:
+        nb_rows = "0 lignes (0 marchés)"
+
+    # Pagination — toujours hors cache pour rester sur des cache hits
+    start_row = page_current * page_size
+    dff = dff.slice(start_row, page_size)
+
+    table_columns, tooltip = setup_table_columns(dff)
+
+    dicts = dff.to_dicts()
+
+    download_disabled, download_text, download_title = get_button_properties(height)
+
+    return (
+        dicts,
+        table_columns,
+        tooltip,
+        data_timestamp + 1,
+        nb_rows,
+        download_disabled,
+        download_text,
+        download_title,
+        trigger_cleanup,
+    )
+```
+
+Notes on what changed vs the original at `src/utils/table.py:372-458`:
+
+- `track_search` now called explicitly at the top, on every invocation (not via `filter_table_data`).
+- `data is None` branch delegates the heavy work to the memoized helper.
+- `data is not None` branch is functionally identical to the original (pagination still happens after collect+post-process).
+- The post-processing (`cast`, `fill_null`, `add_links`, `add_resource_link`, `format_values`) is now done in BOTH branches before `nb_rows` calculation. In the cached branch this was already done inside `_load_filter_sort_postprocess`; in the uncached branch we keep doing it inline. This means `nb_rows` and `dff.select('uid').unique().height` operate on the post-processed frame in both branches, matching the original semantics.
+
+- [ ] **Step 4: Run all unit tests**
+
+Run: `uv run pytest tests/test_table.py -v`
+Expected: All PASS, including `test_prepare_table_data_paginates_without_recomputing`.
+
+- [ ] **Step 5: Run the full repo test suite to catch regressions**
+
+Run: `uv run pytest -v`
+Expected: All PASS. Selenium tests (`tests/test_main.py`) require Chrome/Chromium; if the executor lacks a browser, those tests will error/skip — note the failures and rerun in an environment with Chrome before declaring done.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add src/utils/table.py tests/test_table.py
+git commit -m "perf(tableau): memoize filter+sort+postprocess pipeline"
+```
+
+---
+
+## Task 6: Manual smoke test in the browser
+
+**Files:** none modified.
+
+Type checks and unit tests cannot validate that page navigation actually feels faster. This task is explicitly a hands-on verification.
+
+- [ ] **Step 1: Start the dev server**
+
+Run: `uv run run.py`
+Wait for `Dash is running on http://...`.
+
+- [ ] **Step 2: Open the Tableau page and warm the cache**
+
+1. Open `http://localhost:8050/tableau` (or whatever port the dev server prints).
+2. With no filter applied, wait for the first page to load fully. This is the cold-cache load (slow expected).
+3. Open the browser devtools Network panel.
+
+- [ ] **Step 3: Verify pagination is fast**
+
+1. Click "page 2" / "page 3" / "page 4" in the table footer in quick succession.
+2. Each navigation should return data in well under 1 second (in the original code each took several seconds).
+3. In the dev server logs, look for the line `Cache miss — recomputing for filter=...` from `_load_filter_sort_postprocess`. It should appear ONCE for the initial load and NOT appear again as you change pages.
+
+- [ ] **Step 4: Verify a new filter triggers exactly one cache miss**
+
+1. In the table, type a filter into one of the columns (e.g. `paris` in `acheteur_commune_nom`) and press Enter.
+2. The dev log should show ONE new `Cache miss — recomputing` line.
+3. Change page within the filtered view — no new cache miss line should appear.
+
+- [ ] **Step 5: Verify filter cleanup trigger still fires**
+
+1. Open `http://localhost:8050/acheteur?id=<some_acheteur_id>` (use any valid id from the dataset).
+2. Apply a filter on the embedded table.
+3. The clientside callback for filter cleanup (`src/assets/dash_clientside.js` `clean_filters`) should still rewrite the filter operators (e.g. `contains` → `icontains`). If it doesn't fire, the `trigger_cleanup` uuid is broken — investigate.
+
+- [ ] **Step 6: Verify download still works**
+
+1. On the Tableau page, click "Télécharger au format Excel" (the button must be enabled — apply a filter that brings the row count under 65,000).
+2. The downloaded XLSX must open and contain the filtered rows.
+
+- [ ] **Step 7: Stop the dev server**
+
+Ctrl-C.
+
+- [ ] **Step 8: If all checks pass, this completes the implementation**
+
+No commit — this task is verification only. Report results to the user.

pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "decp.info"
 description = "Interface d'exploration et d'analyse des marchés publics français."
-version = "2.7.2"
+version = "2.7.4"
 requires-python = ">= 3.10"
 authors = [{ name = "Colin Maudry", email = "colin@colmo.tech" }]
 dependencies = [
@@ -23,7 +23,7 @@ dependencies = [
   "pyarrow>=23.0.1",
 ]
 
-[project.optional-dependencies]
+[dependency-groups]
 dev = [
   "pytest",
   "pytest-env",
@@ -40,6 +40,7 @@ testpaths = ["tests"]
 env = [
   "DATA_FILE_PARQUET_PATH=tests/test.parquet",
   "DEVELOPMENT=true",
+  "REBUILD_DUCKDB=true",
   "DATA_SCHEMA_PATH=/home/colin/git/decp-processing/dist/schema.json",
 ]
 addopts = "-p no:warnings"

src/app.py

@@ -2,13 +2,14 @@ import os
 from shutil import rmtree
 
 import dash_bootstrap_components as dbc
+import pandas  # noqa: F401  # eager import: avoid plotly's lazy-import race across Dash callback threads
 import tomllib
 from dash import Dash, Input, Output, State, dcc, html, page_container, page_registry
 from dotenv import load_dotenv
 from flask import Response
 
-from src.cache import cache
 from src.utils import DEVELOPMENT
+from src.utils.cache import cache
 
 load_dotenv()
 

src/db.py

@@ -109,15 +109,8 @@ def build_database(db_path: Path, parquet_path: Path) -> None:
     logger.info(f"Base DuckDB construite : {db_path}")
 
 
-def _resolve_db_path() -> Path:
-    parquet = os.getenv("DATA_FILE_PARQUET_PATH")
-    if not parquet:
-        raise RuntimeError("DATA_FILE_PARQUET_PATH is not set")
-    return Path(parquet).parent / "decp.duckdb"
-
-
 def _ensure_database() -> Path:
-    db_path = _resolve_db_path()
+    db_path = Path(os.getenv("DUCKDB_PATH", "./decp.duckdb"))
     parquet_path = Path(os.getenv("DATA_FILE_PARQUET_PATH"))
     lock_path = db_path.with_suffix(".duckdb.lock")
 
@@ -142,10 +135,11 @@ def get_cursor() -> duckdb.DuckDBPyConnection:
 
 def query_marches(
     where_sql: str = "TRUE",
-    params: tuple = (),
+    params: tuple | list = (),
     columns: list[str] | None = None,
     order_by: str | None = None,
     limit: int | None = None,
+    offset: int | None = None,
 ) -> pl.DataFrame:
     """Run a parameterized SELECT against the decp table and return Polars.
 
@@ -159,4 +153,20 @@ def query_marches(
         sql += f" ORDER BY {order_by}"
     if limit is not None:
         sql += f" LIMIT {int(limit)}"
+    if offset is not None:
+        sql += f" OFFSET {int(offset)}"
     return get_cursor().execute(sql, list(params)).pl()
+
+
+def count_marches(where_sql: str = "TRUE", params: tuple | list = ()) -> int:
+    """Retourne le nombre de lignes correspondant à where_sql."""
+    sql = f"SELECT COUNT(*) FROM decp WHERE {where_sql}"
+    result = get_cursor().execute(sql, list(params)).fetchone()
+    return int(result[0]) if result else 0
+
+
+def count_unique_marches(where_sql: str = "TRUE", params: tuple | list = ()) -> int:
+    """Retourne le nombre de uid distincts correspondant à where_sql."""
+    sql = f"SELECT COUNT(DISTINCT uid) FROM decp WHERE {where_sql}"
+    result = get_cursor().execute(sql, list(params)).fetchone()
+    return int(result[0]) if result else 0

src/figures.py

@@ -839,7 +839,7 @@ def get_top_org_table(data, org_type: str, extra_columns: list, filters: bool =
         return html.Div()
 
     columns, tooltip = setup_table_columns(
-        dff, hideable=False, exclude=[f"{org_type}_id"], new_columns=["Attributions"]
+        dff, hideable=False, exclude=[f"{org_type}_id"]
     )
     dff = add_links(dff)
     data = dff.to_dicts()

src/pages/acheteur.py

@@ -35,6 +35,7 @@ from src.utils.table import (
     prepare_table_data,
     sort_table_data,
 )
+from src.utils.tracking import track_search
 
 
 def get_title(acheteur_id: str | None = None) -> str:
@@ -424,7 +425,8 @@ def download_filtered_acheteur_data(
         lff = lff.drop(hidden_columns)
 
     if filter_query:
-        lff = filter_table_data(lff, filter_query, "ach download")
+        track_search(filter_query, "ach download")
+        lff = filter_table_data(lff, filter_query)
 
     if len(sort_by) > 0:
         lff = sort_table_data(lff, sort_by)

src/pages/observatoire.py

@@ -16,7 +16,6 @@ from dash import (
     register_page,
 )
 
-from src.cache import cache
 from src.db import query_marches, schema
 from src.figures import (
     DataTable,
@@ -31,6 +30,7 @@ from src.figures import (
     make_donut,
 )
 from src.utils import logger
+from src.utils.cache import cache
 from src.utils.data import (
     DEPARTEMENTS,
     DF_ACHETEURS,
@@ -658,9 +658,11 @@ def _normalize_filter_params(filter_params: dict) -> tuple:
 
 
 @cache.memoize()
-def _compute_dashboard_children(cache_key: tuple):
+def _compute_dashboard_children(filter_params_normalized: tuple):
     logger.debug("Cache miss — computing dashboard")
-    filter_params = {k: (list(v) if isinstance(v, tuple) else v) for k, v in cache_key}
+    filter_params = {
+        k: (list(v) if isinstance(v, tuple) else v) for k, v in filter_params_normalized
+    }
 
     lff: pl.LazyFrame = query_marches().lazy()
     lff = prepare_dashboard_data(lff=lff, **filter_params)
@@ -772,8 +774,8 @@ def update_dashboard_cards(*filter_values):
     ):
         filter_params[input_id] = value
 
-    cache_key = _normalize_filter_params(filter_params)
-    children = _compute_dashboard_children(cache_key)
+    filter_params_normalized = _normalize_filter_params(filter_params)
+    children = _compute_dashboard_children(filter_params_normalized)
 
     return dbc.Row(children=children), filter_params
 

src/pages/tableau.py

@@ -31,6 +31,7 @@ from src.utils.table import (
     prepare_table_data,
     sort_table_data,
 )
+from src.utils.tracking import track_search
 
 update_date_timestamp = os.path.getmtime(os.getenv("DATA_FILE_PARQUET_PATH"))
 update_date = datetime.fromtimestamp(update_date_timestamp).strftime("%d/%m/%Y")
@@ -314,7 +315,7 @@ def update_table(href, page_current, page_size, filter_query, sort_by, data_time
     State("tableau_datatable", "hidden_columns"),
     prevent_initial_call=True,
 )
-def download_data(n_clicks, filter_query, sort_by, hidden_columns: list = None):
+def download_data(n_clicks, filter_query, sort_by, hidden_columns: list | None = None):
     lff: pl.LazyFrame = query_marches().lazy()
 
     # Les colonnes masquées sont supprimées
@@ -322,7 +323,8 @@ def download_data(n_clicks, filter_query, sort_by, hidden_columns: list = None):
         lff = lff.drop(hidden_columns)
 
     if filter_query:
-        lff = filter_table_data(lff, filter_query, "tab download")
+        track_search(filter_query, "tab download")
+        lff = filter_table_data(lff, filter_query)
 
     if sort_by and len(sort_by) > 0:
         lff = sort_table_data(lff, sort_by)

src/pages/titulaire.py

@@ -34,6 +34,7 @@ from src.utils.table import (
     prepare_table_data,
     sort_table_data,
 )
+from src.utils.tracking import track_search
 
 
 def get_title(titulaire_id: str = None) -> str:
@@ -429,7 +430,12 @@ def download_titulaire_data(
     prevent_initial_call=True,
 )
 def download_filtered_titulaire_data(
-    data, n_clicks, titulaire_nom, filter_query, sort_by, hidden_columns: list = None
+    data,
+    n_clicks,
+    titulaire_nom,
+    filter_query,
+    sort_by,
+    hidden_columns: list | None = None,
 ):
     lff: pl.LazyFrame = pl.LazyFrame(
         data
@@ -440,7 +446,8 @@ def download_filtered_titulaire_data(
         lff = lff.drop(hidden_columns)
 
     if filter_query:
-        lff = filter_table_data(lff, filter_query, "titu download")
+        track_search(filter_query, "titu download")
+        lff = filter_table_data(lff, filter_query)
 
     if len(sort_by) > 0:
         lff = sort_table_data(lff, sort_by)

src/utils/table.py

@@ -5,8 +5,9 @@ import polars as pl
 from dash import no_update
 from polars import selectors as cs
 
-from src.db import query_marches, schema
+from src.db import count_marches, count_unique_marches, query_marches, schema
 from src.utils import logger
+from src.utils.cache import cache
 from src.utils.data import DATA_SCHEMA
 from src.utils.frontend import get_button_properties
 from src.utils.tracking import track_search
@@ -146,6 +147,12 @@ def dates_to_strings(lff: pl.LazyFrame, column: str) -> pl.LazyFrame:
     return lff
 
 
+def normalize_sort_by(sort_by) -> tuple:
+    if not sort_by:
+        return ()
+    return tuple((entry["column_id"], entry["direction"]) for entry in sort_by)
+
+
 def format_number(number) -> str:
     number = "{:,}".format(number).replace(",", " ")
     return number
@@ -160,7 +167,7 @@ def unformat_montant(number: str) -> float:
 
 
 def format_values(dff: pl.DataFrame) -> pl.DataFrame:
-    def format_montant(expr, scale=None):
+    def format_montant(expr):
         # https://stackoverflow.com/a/78636786
         expr = expr.cast(pl.String)
         expr = expr.str.splitn(".", 2)
@@ -206,14 +213,13 @@ def format_values(dff: pl.DataFrame) -> pl.DataFrame:
     return dff
 
 
-def filter_table_data(
-    lff: pl.LazyFrame, filter_query: str, filter_source: str
-) -> pl.LazyFrame:
+def filter_table_data(lff: pl.LazyFrame, filter_query: str) -> pl.LazyFrame:
     _schema = lff.collect_schema()
-    track_search(filter_query, filter_source)
     filtering_expressions = filter_query.split(" && ")
     for filter_part in filtering_expressions:
         col_name, operator, filter_value = split_filter_part(filter_part)
+        if not isinstance(col_name, str) or not isinstance(filter_value, str):
+            continue
         col_type = str(_schema[col_name])
         # logger.debug("filter_value:", filter_value)
         # logger.debug("filter_value_type:", type(filter_value))
@@ -246,7 +252,7 @@ def filter_table_data(
             elif operator == "<=":
                 lff = lff.filter(pl.col(col_name) <= filter_value)
             elif operator == "contains":
-                if col_type in ["String", "Date"]:
+                if col_type in ["String", "Date"] and isinstance(filter_value, str):
                     filter_value = filter_value.strip('"')
                     if filter_value.endswith("*"):
                         lff = lff.filter(
@@ -284,7 +290,9 @@ def sort_table_data(lff: pl.LazyFrame, sort_by: list) -> pl.LazyFrame:
 
 
 def setup_table_columns(
-    dff, hideable: bool = True, exclude: list = None, new_columns: list = None
+    dff,
+    hideable: bool = True,
+    exclude: list | None = None,
 ) -> tuple:
     # Liste finale de colonnes
     markdown_exceptions = ["montant", "titulaire_distance", "distance", "dureeMois"]
@@ -368,6 +376,62 @@ def get_default_hidden_columns(page):
     return hidden_columns
 
 
+def postprocess_page(dff: pl.DataFrame) -> pl.DataFrame:
+    """Post-traitement à appliquer sur une page déjà paginée.
+
+    À appeler après la pagination.
+    """
+    dff = dff.with_columns(pl.all().cast(pl.String).fill_null(""))
+    dff = add_links(dff)
+    if "sourceFile" in dff.columns:
+        dff = add_resource_link(dff)
+    if dff.height > 0:
+        dff = format_values(dff)
+    return dff
+
+
+@cache.memoize()
+def _fetch_page_sql(
+    filter_query: str | None,
+    sort_by_key: tuple,
+    page_current: int,
+    page_size: int,
+) -> tuple[pl.DataFrame, int, int]:
+    """Chemin rapide : filtre/tri/pagine dans DuckDB, post-traite la page seule.
+
+    Retourne (page_dataframe_post_traitée, total_count, total_unique_count).
+    """
+    # Import local pour éviter une dépendance circulaire
+    # (src.utils.table_sql importe split_filter_part depuis src.utils.table).
+    from src.utils.table_sql import filter_query_to_sql, sort_by_to_sql
+
+    logger.debug(
+        f"Cache miss SQL — filter={filter_query!r} sort={sort_by_key!r} "
+        f"page={page_current} size={page_size}"
+    )
+
+    where_sql, params = filter_query_to_sql(filter_query or "", schema)
+
+    sort_by_dash = [
+        {"column_id": col, "direction": direction} for col, direction in sort_by_key
+    ]
+    order_by = sort_by_to_sql(sort_by_dash, schema) or None
+
+    total = count_marches(where_sql, params)
+    total_unique = count_unique_marches(where_sql, params)
+
+    page = query_marches(
+        where_sql=where_sql,
+        params=params,
+        order_by=order_by,
+        limit=page_size,
+        offset=page_current * page_size,
+    )
+
+    page = postprocess_page(page)
+    return page, total, total_unique
+
+
 def prepare_table_data(
     data, data_timestamp, filter_query, page_current, page_size, sort_by, source_table
 ):
@@ -383,66 +447,58 @@ def prepare_table_data(
     :param source_table:
     :return:
     """
+    logger.debug(" + + + + + + + + + + + + + + + + + + ")
 
-    if os.getenv("DEVELOPMENT").lower() == "true":
-        logger.debug(" + + + + + + + + + + + + + + + + + + ")
-
-    trigger_cleanup = no_update
-
-    # Récupération des données
-    if isinstance(data, list):
-        lff: pl.LazyFrame = pl.LazyFrame(data, strict=False, infer_schema_length=5000)
-    elif isinstance(data, pl.LazyFrame):
-        lff = data
-    else:
-        lff: pl.LazyFrame = query_marches().lazy()
-
-    # Application des filtres
     if filter_query:
-        lff = filter_table_data(lff, filter_query, source_table)
-        trigger_cleanup = no_update if source_table == "tableau" else str(uuid.uuid4())
+        track_search(filter_query, source_table)
 
-    # Application des tris
-    if sort_by and len(sort_by) > 0:
-        lff = sort_table_data(lff, sort_by)
+    trigger_cleanup = no_update if source_table == "tableau" else str(uuid.uuid4())
 
-    # Matérialisation des filtres
-    dff: pl.DataFrame = lff.collect()
-    height = dff.height
+    if data is None:
+        # Probablement car il s'agit de la page Tableau
+        sort_by_key = normalize_sort_by(sort_by)
+        dff, height, total_unique = _fetch_page_sql(
+            filter_query=filter_query,
+            sort_by_key=sort_by_key,
+            page_current=page_current,
+            page_size=page_size,
+        )
+    else:
+        if isinstance(data, list):
+            lff: pl.LazyFrame = pl.LazyFrame(
+                data, strict=False, infer_schema_length=5000
+            )
+        elif isinstance(data, pl.LazyFrame):
+            lff = data
+        else:
+            lff = query_marches().lazy()
+
+        if filter_query:
+            lff = filter_table_data(lff, filter_query)
+
+        df_height = lff.select("uid").collect(engine="streaming")
+        height = df_height.height
+        total_unique = df_height["uid"].n_unique()
+
+        if sort_by and len(sort_by) > 0:
+            lff = sort_table_data(lff, sort_by)
+
+        start_row = page_current * page_size
+        lff = lff.slice(start_row, page_size)
+        dff = lff.collect(engine="streaming")
+        dff: pl.DataFrame = postprocess_page(dff)
 
     if height > 0:
-        nb_rows = f"{format_number(height)} lignes ({format_number(dff.select('uid').unique().height)} marchés)"
+        nb_rows = (
+            f"{format_number(height)} lignes ({format_number(total_unique)} marchés)"
+        )
     else:
         nb_rows = "0 lignes (0 marchés)"
 
-    # Pagination des données
-    start_row = page_current * page_size
-    # end_row = (page_current + 1) * page_size
-    dff = dff.slice(start_row, page_size)
-
-    # Tout devient string
-    dff = dff.cast(pl.String)
-
-    # Remplace les strings null par "", mais pas les numeric null
-    dff = dff.fill_null("")
-
-    # Ajout des liens vers les pages de détails
-    dff = add_links(dff)
-
-    # Ajout des liens vers les fichiers Open Data
-    if "sourceFile" in dff.columns:
-        dff = add_resource_link(dff)
-
-    # Formatage des montants
-    if height > 0:
-        dff = format_values(dff)
-
-    # Récupération des colonnes et tooltip
     table_columns, tooltip = setup_table_columns(dff)
 
     dicts = dff.to_dicts()
 
-    # Propriétés du bouton de téléchargement
     download_disabled, download_text, download_title = get_button_properties(height)
 
     return (

src/utils/table_sql.py

@@ -0,0 +1,102 @@
+import polars as pl
+
+from src.utils import logger
+from src.utils.table import split_filter_part
+
+
+def filter_query_to_sql(filter_query: str, schema: pl.Schema) -> tuple[str, list]:
+    """Traduit le DSL de filtres de dash_table.DataTable en fragment SQL DuckDB.
+
+    Retourne (where_clause, params) où where_clause est un fragment à injecter
+    après WHERE et params est la liste des valeurs à passer à
+    cursor.execute(sql, params). Les identifiants de colonnes sont validés
+    contre le schéma fourni ; jamais concaténés avec des valeurs utilisateur.
+    """
+    if not filter_query:
+        return "TRUE", []
+
+    clauses: list[str] = []
+    params: list = []
+
+    for part in filter_query.split(" && "):
+        col_name, operator, raw_value = split_filter_part(part)
+        if not isinstance(col_name, str) or not isinstance(raw_value, str):
+            continue
+
+        if col_name not in schema.names():
+            logger.warning(f"Colonne inconnue ignorée : {col_name!r}")
+            continue
+
+        col_type = schema[col_name]
+        is_numeric = col_type.is_numeric()
+        col_is_date = col_type == pl.Date
+        quoted_col = f'"{col_name}"'
+
+        if is_numeric:
+            try:
+                value = int(raw_value) if col_type.is_integer() else float(raw_value)
+            except ValueError:
+                logger.warning(f"Valeur numérique invalide ignorée : {raw_value!r}")
+                continue
+
+            if operator == "contains":
+                clauses.append(f"{quoted_col} IS NOT NULL AND {quoted_col} = ?")
+            elif operator == ">":
+                clauses.append(f"{quoted_col} IS NOT NULL AND {quoted_col} > ?")
+            elif operator == "<":
+                clauses.append(f"{quoted_col} IS NOT NULL AND {quoted_col} < ?")
+            else:
+                logger.warning(f"Opérateur invalide pour numérique : {operator!r}")
+                continue
+            params.append(value)
+            continue
+
+        # String / Date : toujours traité comme texte (parité avec Polars)
+        value = raw_value.strip('"')
+
+        if operator == "contains":
+            if value.endswith("*") and not value.startswith("*"):
+                like = value[:-1] + "%"
+            elif value.startswith("*") and not value.endswith("*"):
+                like = "%" + value[1:]
+            else:
+                like = "%" + value + "%"
+            target = f"CAST({quoted_col} AS VARCHAR)" if col_is_date else quoted_col
+            clauses.append(
+                f"{quoted_col} IS NOT NULL AND {target} <> '' AND {target} ILIKE ?"
+            )
+            params.append(like)
+        elif operator in (">", "<"):
+            target = f"CAST({quoted_col} AS VARCHAR)" if col_is_date else quoted_col
+            clauses.append(f"{quoted_col} IS NOT NULL AND {target} {operator} ?")
+            params.append(value)
+        else:
+            logger.warning(f"Opérateur invalide pour chaîne : {operator!r}")
+            continue
+
+    if not clauses:
+        return "TRUE", []
+    return " AND ".join(clauses), params
+
+
+def sort_by_to_sql(sort_by: list[dict] | None, schema: pl.Schema) -> str:
+    """Traduit sort_by (format Dash) en clause ORDER BY DuckDB.
+
+    Retourne '' si pas de tri (aucun ORDER BY à ajouter).
+    """
+    if not sort_by:
+        return ""
+
+    fragments: list[str] = []
+    for entry in sort_by:
+        col = entry.get("column_id")
+        direction = entry.get("direction")
+        if col not in schema.names():
+            logger.warning(f"Tri sur colonne inconnue ignoré : {col!r}")
+            continue
+        if direction not in ("asc", "desc"):
+            logger.warning(f"Tri sur direction inconnue ignoré : {direction!r}")
+            continue
+        fragments.append(f'"{col}" {direction.upper()} NULLS LAST')
+
+    return ", ".join(fragments)

tests/conftest.py

@@ -6,55 +6,66 @@ import polars as pl
 import pytest
 from selenium.webdriver.chrome.options import Options
 
+_TEST_DATA = [
+    {
+        "uid": "1",
+        "id": "1",
+        "acheteur_nom": "ACHETEUR 1",
+        "acheteur_id": "123",
+        "titulaire_nom": "TITULAIRE 1",
+        "titulaire_id": "345",
+        "montant": 10,
+        "dateNotification": datetime.date(2025, 1, 1),
+        "codeCPV": "71600000",
+        "donneesActuelles": True,
+        "acheteur_departement_code": "75",
+        "acheteur_departement_nom": "Paris",
+        "acheteur_commune_nom": "Paris",
+        "titulaire_departement_code": "35",
+        "titulaire_departement_nom": "Ille-et-Vilaine",
+        "titulaire_commune_nom": "Rennes",
+        "titulaire_distance": 10,
+        "titulaire_typeIdentifiant": "SIRET",
+        "objet": "Objet test",
+        "dureeRestanteMois": 12,
+        "lieuExecution_code": "75001",
+        "sourceFile": "test.xml",
+        "sourceDataset": "test_dataset",
+        "datePublicationDonnees": datetime.date(2025, 1, 1),
+        "considerationsSociales": "",
+        "considerationsEnvironnementales": "",
+        "type": "Marché",
+        "acheteur_categorie": "Collectivité",
+        "titulaire_categorie": "PME",
+    }
+]
+_PARQUET_PATH = Path(os.path.abspath("tests/test.parquet"))
+_DB_PATH = Path(os.path.abspath("decp.duckdb"))
 
-@pytest.fixture(scope="session", autouse=True)
-def test_data():
-    data = [
-        {
-            "uid": "1",
-            "id": "1",
-            "acheteur_nom": "ACHETEUR 1",
-            "acheteur_id": "123",
-            "titulaire_nom": "TITULAIRE 1",
-            "titulaire_id": "345",
-            "montant": 10,
-            "dateNotification": datetime.date(2025, 1, 1),
-            "codeCPV": "71600000",
-            "donneesActuelles": True,
-            "acheteur_departement_code": "75",
-            "acheteur_departement_nom": "Paris",
-            "acheteur_commune_nom": "Paris",
-            "titulaire_departement_code": "35",
-            "titulaire_departement_nom": "Ille-et-Vilaine",
-            "titulaire_commune_nom": "Rennes",
-            "titulaire_distance": 10,
-            "titulaire_typeIdentifiant": "SIRET",
-            "objet": "Objet test",
-            "dureeRestanteMois": 12,
-            "lieuExecution_code": "75001",
-            "sourceFile": "test.xml",
-            "sourceDataset": "test_dataset",
-            "datePublicationDonnees": datetime.date(2025, 1, 1),
-            "considerationsSociales": "",
-            "considerationsEnvironnementales": "",
-            "type": "Marché",
-            "acheteur_categorie": "Collectivité",
-            "titulaire_categorie": "PME",
-        }
-    ]
-    parquet_path = Path(os.path.abspath("tests/test.parquet"))
-    db_path = parquet_path.parent / "decp.duckdb"
-    print(f"Writing test data to: {parquet_path}")
 
-    pl.DataFrame(data).write_parquet(parquet_path)
-
-    # Remove any stale DuckDB from a previous run so src.db rebuilds from
-    # the freshly-written parquet at import time.
-    for artifact in (db_path, db_path.with_suffix(".duckdb.tmp")):
+def _cleanup_db_artifacts() -> None:
+    for artifact in (
+        _DB_PATH,
+        _DB_PATH.with_suffix(".duckdb.tmp"),
+        _DB_PATH.with_suffix(".duckdb.lock"),
+    ):
         if artifact.exists():
             artifact.unlink()
 
-    yield str(parquet_path)
+
+# Runs at conftest import, before test modules import src.db (which builds the
+# DuckDB at import time). Guarantees the test parquet exists and the stale DB
+# from a previous `python run.py` is wiped so src.db rebuilds from test data.
+pl.DataFrame(_TEST_DATA).write_parquet(_PARQUET_PATH)
+_cleanup_db_artifacts()
+
+
+@pytest.fixture(scope="session", autouse=True)
+def test_data():
+    yield str(_PARQUET_PATH)
+    # Teardown: remove the test DuckDB so the next `python run.py` rebuilds
+    # from decp_prod.parquet.
+    _cleanup_db_artifacts()
 
 
 def pytest_setup_options():

tests/test_db.py

@@ -141,6 +141,7 @@ def built_db(tmp_path, monkeypatch):
     )
     data.write_parquet(parquet_path)
     monkeypatch.setenv("DATA_FILE_PARQUET_PATH", str(parquet_path))
+    monkeypatch.setenv("DUCKDB_PATH", str(db_path))
 
     from src.db import build_database
 
@@ -206,6 +207,38 @@ def test_query_marches_returns_polars_frame(built_db, monkeypatch):
     assert set(frame["uid"].to_list()) == {"1", "2"}
 
 
+def test_count_marches_returns_total_without_filter():
+    from src.db import count_marches
+
+    n = count_marches()
+    assert isinstance(n, int)
+    assert n > 0
+
+
+def test_count_marches_with_filter():
+    from src.db import count_marches
+
+    n = count_marches('"uid" = ?', ["__nonexistent__"])
+    assert n == 0
+
+
+def test_count_unique_marches_respects_distinct():
+    from src.db import count_unique_marches
+
+    n = count_unique_marches()
+    assert isinstance(n, int)
+    assert n > 0
+
+
+def test_query_marches_with_offset():
+    from src.db import query_marches
+
+    page_0 = query_marches(limit=2, offset=0)
+    page_1 = query_marches(limit=2, offset=2)
+    if page_0.height == 2 and page_1.height >= 1:
+        assert set(page_0["uid"].to_list()).isdisjoint(set(page_1["uid"].to_list()))
+
+
 def test_concurrent_build_serialized(tmp_path):
     """Multiple threads calling _ensure_database must serialize via flock.
 

tests/test_table.py

@@ -0,0 +1,288 @@
+import polars as pl
+import pytest
+
+
+@pytest.fixture
+def sample_lff():
+    """Small LazyFrame with the columns needed by add_links / format_values."""
+    return pl.LazyFrame(
+        [
+            {
+                "uid": "u1",
+                "id": "u1",
+                "acheteur_id": "12345678900011",
+                "acheteur_nom": "Mairie de Test",
+                "titulaire_id": "98765432100022",
+                "titulaire_nom": "Entreprise Test",
+                "titulaire_typeIdentifiant": "SIRET",
+                "objet": "Travaux divers",
+                "montant": 12500.0,
+                "dateNotification": "2025-03-15",
+                "codeCPV": "45000000",
+                "dureeRestanteMois": 6,
+                "titulaire_distance": 42.0,
+            }
+        ]
+    )
+
+
+def test_table_module_imports():
+    from src.utils import table
+
+    assert hasattr(table, "prepare_table_data")
+
+
+def test_filter_table_data_does_not_call_track_search(monkeypatch, sample_lff):
+    from src.utils import table
+
+    calls = []
+    monkeypatch.setattr(table, "track_search", lambda *a, **kw: calls.append(a))
+
+    result = table.filter_table_data(sample_lff, "{objet} icontains travaux").collect()
+
+    assert calls == []
+    assert result.height == 1
+
+
+def test_normalize_sort_by_handles_empty():
+    from src.utils.table import normalize_sort_by
+
+    assert normalize_sort_by(None) == ()
+    assert normalize_sort_by([]) == ()
+
+
+def test_normalize_sort_by_returns_hashable_tuple():
+    from src.utils.table import normalize_sort_by
+
+    sort_by = [
+        {"column_id": "montant", "direction": "desc"},
+        {"column_id": "dateNotification", "direction": "asc"},
+    ]
+    key = normalize_sort_by(sort_by)
+
+    assert key == (("montant", "desc"), ("dateNotification", "asc"))
+    # Must be hashable so that flask-caching can build a cache key from it
+    hash(key)
+
+
+def test_normalize_sort_by_preserves_order():
+    """Order matters for sort: [A, B] != [B, A]."""
+    from src.utils.table import normalize_sort_by
+
+    a_then_b = normalize_sort_by(
+        [{"column_id": "a", "direction": "asc"}, {"column_id": "b", "direction": "asc"}]
+    )
+    b_then_a = normalize_sort_by(
+        [{"column_id": "b", "direction": "asc"}, {"column_id": "a", "direction": "asc"}]
+    )
+    assert a_then_b != b_then_a
+
+
+@pytest.fixture(scope="module")
+def flask_app():
+    """Minimal Flask app with SimpleCache so @cache.memoize() works in tests."""
+    from flask import Flask
+
+    from src.utils.cache import cache
+
+    app = Flask(__name__)
+    cache.init_app(app, config={"CACHE_TYPE": "SimpleCache"})
+    return app
+
+
+@pytest.fixture(autouse=True)
+def reset_cache(flask_app):
+    """Ensure the flask-caching backend is empty between tests so that
+    cache-hit assertions are meaningful. Falls back to no-op when no
+    Flask app context is active (NullCache)."""
+    from src.utils.cache import cache
+
+    with flask_app.app_context():
+        try:
+            cache.clear()
+        except (RuntimeError, AttributeError):
+            # No app context — cache is NullCache, nothing to clear
+            pass
+        yield
+
+
+def test_prepare_table_data_returns_expected_tuple(flask_app):
+    from src.utils import table
+
+    with flask_app.app_context():
+        result = table.prepare_table_data(
+            data=None,
+            data_timestamp=5,
+            filter_query=None,
+            page_current=0,
+            page_size=20,
+            sort_by=[],
+            source_table="tableau",
+        )
+
+    # Same arity as before: 9 outputs
+    assert len(result) == 9
+    dicts, columns, tooltip, ts, nb_rows, dl_disabled, dl_text, dl_title, cleanup = (
+        result
+    )
+    assert isinstance(dicts, list)
+    assert ts == 6  # data_timestamp + 1 must still increment
+    assert "lignes" in nb_rows
+
+
+def test_prepare_table_data_calls_track_search_on_filter(monkeypatch, flask_app):
+    from src.utils import table
+
+    calls = []
+    monkeypatch.setattr(table, "track_search", lambda *a, **kw: calls.append(a))
+
+    with flask_app.app_context():
+        table.prepare_table_data(
+            data=None,
+            data_timestamp=0,
+            filter_query="{objet} icontains travaux",
+            page_current=0,
+            page_size=20,
+            sort_by=[],
+            source_table="tableau",
+        )
+
+    assert calls == [("{objet} icontains travaux", "tableau")]
+
+
+def test_prepare_table_data_same_page_uses_cache(monkeypatch, flask_app):
+    """Two calls with exactly the same (filter, sort, page, size)
+    must call _fetch_page_sql at least once."""
+    from src.utils import table
+
+    call_count = {"n": 0}
+
+    def counting_fetch(*args, **kwargs):
+        call_count["n"] += 1
+        import polars as pl
+
+        return (
+            pl.DataFrame(
+                {
+                    "uid": [],
+                    "acheteur_id": [],
+                    "titulaire_id": [],
+                    "titulaire_typeIdentifiant": [],
+                }
+            ),
+            0,
+            0,
+        )
+
+    monkeypatch.setattr(table, "_fetch_page_sql", counting_fetch)
+
+    with flask_app.app_context():
+        table.prepare_table_data(
+            data=None,
+            data_timestamp=0,
+            filter_query=None,
+            page_current=0,
+            page_size=10,
+            sort_by=[],
+            source_table="tableau",
+        )
+        table.prepare_table_data(
+            data=None,
+            data_timestamp=0,
+            filter_query=None,
+            page_current=0,
+            page_size=10,
+            sort_by=[],
+            source_table="tableau",
+        )
+    assert call_count["n"] >= 1
+
+
+def test_prepare_table_data_cleanup_trigger_for_non_tableau(flask_app):
+    """Non-tableau pages still get a fresh uuid trigger, not no_update."""
+    from dash import no_update
+
+    from src.utils import table
+
+    with flask_app.app_context():
+        result = table.prepare_table_data(
+            data=None,
+            data_timestamp=0,
+            filter_query="{objet} icontains travaux",
+            page_current=0,
+            page_size=20,
+            sort_by=[],
+            source_table="acheteur",
+        )
+
+    cleanup = result[8]
+    assert cleanup is not no_update
+    assert isinstance(cleanup, str)
+    assert len(cleanup) >= 32  # uuid4 hex string
+
+
+def test_prepare_table_data_with_external_data_does_not_use_cache(
+    monkeypatch, flask_app, sample_lff
+):
+    """When a caller passes data (acheteur/titulaire/observatoire path),
+    bypass the memoized helper entirely."""
+    from src.utils import table
+
+    sentinel = {"called": False}
+
+    def should_not_be_called(*a, **kw):
+        sentinel["called"] = True
+        raise AssertionError("Memoized helper must not be called when data is provided")
+
+    monkeypatch.setattr(table, "_fetch_page_sql", should_not_be_called)
+
+    with flask_app.app_context():
+        table.prepare_table_data(
+            data=sample_lff,
+            data_timestamp=0,
+            filter_query=None,
+            page_current=0,
+            page_size=20,
+            sort_by=[],
+            source_table="acheteur",
+        )
+
+    assert sentinel["called"] is False
+
+
+def test_fetch_page_sql_respects_pagination(flask_app):
+    """New path: returns (page_dff, total_count, total_unique) via DuckDB."""
+    from src.utils import table
+
+    with flask_app.app_context():
+        page, total, total_unique = table._fetch_page_sql(
+            filter_query=None, sort_by_key=(), page_current=0, page_size=5
+        )
+    assert page.height <= 5
+    assert total >= page.height
+    assert isinstance(total_unique, int)
+
+
+def test_fetch_page_sql_applies_filter(flask_app):
+    from src.utils import table
+
+    with flask_app.app_context():
+        page, total, total_unique = table._fetch_page_sql(
+            filter_query="{uid} icontains __ne_matche_rien__",
+            sort_by_key=(),
+            page_current=0,
+            page_size=20,
+        )
+    assert total == 0
+    assert page.height == 0
+
+
+def test_fetch_page_sql_post_processes_links(flask_app):
+    from src.utils import table
+
+    with flask_app.app_context():
+        page, _, _ = table._fetch_page_sql(
+            filter_query=None, sort_by_key=(), page_current=0, page_size=1
+        )
+    if page.height > 0:
+        assert "<a href" in page["uid"][0]

tests/test_table_sql.py

@@ -0,0 +1,150 @@
+import polars as pl
+
+SCHEMA = pl.Schema(
+    {
+        "uid": pl.String,
+        "objet": pl.String,
+        "acheteur_id": pl.String,
+        "montant": pl.Float64,
+        "dureeMois": pl.Int64,
+        "dateNotification": pl.Date,
+    }
+)
+
+
+def test_empty_filter_returns_true():
+    from src.utils.table_sql import filter_query_to_sql
+
+    where, params = filter_query_to_sql("", SCHEMA)
+    assert where == "TRUE"
+    assert params == []
+
+
+def test_icontains_string_is_case_insensitive_like():
+    from src.utils.table_sql import filter_query_to_sql
+
+    where, params = filter_query_to_sql("{objet} icontains travaux", SCHEMA)
+    assert where == '"objet" IS NOT NULL AND "objet" <> \'\' AND "objet" ILIKE ?'
+    assert params == ["%travaux%"]
+
+
+def test_icontains_with_trailing_wildcard_is_starts_with():
+    from src.utils.table_sql import filter_query_to_sql
+
+    where, params = filter_query_to_sql(
+        "{acheteur_id} icontains 24350013900189*", SCHEMA
+    )
+    assert (
+        where
+        == '"acheteur_id" IS NOT NULL AND "acheteur_id" <> \'\' AND "acheteur_id" ILIKE ?'
+    )
+    assert params == ["24350013900189%"]
+
+
+def test_icontains_with_leading_wildcard_is_ends_with():
+    from src.utils.table_sql import filter_query_to_sql
+
+    where, params = filter_query_to_sql("{uid} icontains *2024", SCHEMA)
+    assert where == '"uid" IS NOT NULL AND "uid" <> \'\' AND "uid" ILIKE ?'
+    assert params == ["%2024"]
+
+
+def test_numeric_greater_than():
+    from src.utils.table_sql import filter_query_to_sql
+
+    where, params = filter_query_to_sql("{montant} i> 40000", SCHEMA)
+    assert where == '"montant" IS NOT NULL AND "montant" > ?'
+    assert params == [40000.0]
+
+
+def test_numeric_less_than():
+    from src.utils.table_sql import filter_query_to_sql
+
+    where, params = filter_query_to_sql("{montant} i< 1000", SCHEMA)
+    assert where == '"montant" IS NOT NULL AND "montant" < ?'
+    assert params == [1000.0]
+
+
+def test_numeric_equality_via_icontains():
+    from src.utils.table_sql import filter_query_to_sql
+
+    where, params = filter_query_to_sql("{dureeMois} icontains 12", SCHEMA)
+    assert where == '"dureeMois" IS NOT NULL AND "dureeMois" = ?'
+    assert params == [12]
+
+
+def test_date_column_treated_as_string_ilike():
+    from src.utils.table_sql import filter_query_to_sql
+
+    where, params = filter_query_to_sql("{dateNotification} icontains 2024*", SCHEMA)
+    assert "ILIKE" in where
+    assert params == ["2024%"]
+
+
+def test_multiple_filters_joined_by_and():
+    from src.utils.table_sql import filter_query_to_sql
+
+    filter_query = "{objet} icontains voirie && {montant} i> 40000"
+    where, params = filter_query_to_sql(filter_query, SCHEMA)
+    assert " AND " in where
+    assert params == ["%voirie%", 40000.0]
+
+
+def test_invalid_numeric_value_is_skipped():
+    from src.utils.table_sql import filter_query_to_sql
+
+    where, params = filter_query_to_sql("{montant} i> notanumber", SCHEMA)
+    assert where == "TRUE"
+    assert params == []
+
+
+def test_unknown_column_is_skipped():
+    from src.utils.table_sql import filter_query_to_sql
+
+    where, params = filter_query_to_sql("{inexistant} icontains foo", SCHEMA)
+    assert where == "TRUE"
+    assert params == []
+
+
+def test_escapes_identifier_with_quotes_not_concatenation():
+    from src.utils.table_sql import filter_query_to_sql
+
+    where, params = filter_query_to_sql(
+        "{objet} icontains '; DROP TABLE decp; --", SCHEMA
+    )
+    assert "DROP TABLE" not in where
+    assert any("DROP TABLE" in str(p) for p in params)
+
+
+def test_sort_by_empty():
+    from src.utils.table_sql import sort_by_to_sql
+
+    assert sort_by_to_sql([], SCHEMA) == ""
+    assert sort_by_to_sql(None, SCHEMA) == ""
+
+
+def test_sort_by_single_column_desc():
+    from src.utils.table_sql import sort_by_to_sql
+
+    result = sort_by_to_sql([{"column_id": "montant", "direction": "desc"}], SCHEMA)
+    assert result == '"montant" DESC NULLS LAST'
+
+
+def test_sort_by_multiple_columns_preserves_order():
+    from src.utils.table_sql import sort_by_to_sql
+
+    result = sort_by_to_sql(
+        [
+            {"column_id": "dateNotification", "direction": "desc"},
+            {"column_id": "montant", "direction": "asc"},
+        ],
+        SCHEMA,
+    )
+    assert result == '"dateNotification" DESC NULLS LAST, "montant" ASC NULLS LAST'
+
+
+def test_sort_by_ignores_unknown_column():
+    from src.utils.table_sql import sort_by_to_sql
+
+    result = sort_by_to_sql([{"column_id": "fake", "direction": "asc"}], SCHEMA)
+    assert result == ""

uv.lock

@@ -760,7 +760,7 @@ wheels = [
 
 [[package]]
 name = "decp-info"
-version = "2.7.2"
+version = "2.7.4"
 source = { virtual = "." }
 dependencies = [
     { name = "dash", extra = ["compress"] },
@@ -783,7 +783,7 @@ dependencies = [
     { name = "xlsxwriter" },
 ]
 
-[package.optional-dependencies]
+[package.dev-dependencies]
 dev = [
     { name = "dash", extra = ["testing"] },
     { name = "fastexcel" },
@@ -798,29 +798,32 @@ dev = [
 requires-dist = [
     { name = "dash", specifier = "==3.4.0" },
     { name = "dash", extras = ["compress"] },
-    { name = "dash", extras = ["testing"], marker = "extra == 'dev'" },
     { name = "dash-bootstrap-components" },
     { name = "dash-extensions" },
     { name = "dash-leaflet" },
     { name = "duckdb" },
-    { name = "fastexcel", marker = "extra == 'dev'" },
     { name = "flask-caching" },
     { name = "gunicorn" },
     { name = "httpx" },
     { name = "pandas" },
     { name = "plotly", extras = ["express"] },
     { name = "polars" },
-    { name = "pre-commit", marker = "extra == 'dev'" },
     { name = "pyarrow", specifier = ">=23.0.1" },
-    { name = "pytest", marker = "extra == 'dev'" },
-    { name = "pytest-env", marker = "extra == 'dev'" },
     { name = "python-dotenv" },
-    { name = "selenium", marker = "extra == 'dev'" },
     { name = "unidecode" },
-    { name = "webdriver-manager", marker = "extra == 'dev'" },
     { name = "xlsxwriter" },
 ]
-provides-extras = ["dev"]
+
+[package.metadata.requires-dev]
+dev = [
+    { name = "dash", extras = ["testing"] },
+    { name = "fastexcel" },
+    { name = "pre-commit" },
+    { name = "pytest" },
+    { name = "pytest-env" },
+    { name = "selenium" },
+    { name = "webdriver-manager" },
+]
 
 [[package]]
 name = "dill"