docs(observatoire): plan d'implémentation filtrage natif DuckDB (#72)

Plan en 10 tâches TDD : construction incrémentale de dashboard_filters_to_sql, réécriture de prepare_dashboard_data, adaptation des 3 appelants de observatoire.py, test d'intégration sur tests/test.parquet.
2026-04-22 23:21:47 +02:00
parent b996eb97cc
commit aaf54eef91
1 changed files with 951 additions and 0 deletions
@@ -0,0 +1,951 @@
 # Observatoire — filtrage natif DuckDB — Plan d'implémentation
 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
 **Goal:** Remplacer le filtrage Polars sur LazyFrame dans `prepare_dashboard_data` par un requêtage natif DuckDB, pour ne matérialiser que le sous-ensemble utile au lieu de l'intégralité de la table `decp` (~1,5 M lignes).
 **Architecture:** Nouveau helper pur `dashboard_filters_to_sql(**filter_params) -> (where_sql, params)` dans `src/utils/table_sql.py` (modèle de `filter_query_to_sql`). `prepare_dashboard_data` devient une fonction fine qui appelle `query_marches(where_sql, params)` et retourne une `pl.DataFrame`. Les 3 appelants dans `src/pages/observatoire.py` sont adaptés à la nouvelle signature.
 **Tech Stack:** Python 3.12, Polars, DuckDB, Dash, pytest.
 **Spec:** `docs/superpowers/specs/2026-04-22-observatoire-duckdb-filters-design.md`.
 ---
 ## File Structure
 **À créer :**
 - `tests/test_dashboard_filters_to_sql.py` — tests unitaires du nouveau helper SQL (cas vide + cas par filtre).
 - `tests/test_prepare_dashboard_data.py` — test d'intégration léger (appel DuckDB réel sur `tests/test.parquet`).
 **À modifier :**
 - `src/utils/table_sql.py` — ajouter `dashboard_filters_to_sql` + import `datetime`/`timedelta`.
 - `src/utils/data.py` — réécrire `prepare_dashboard_data` (signature et implémentation), ajouter `query_marches` aux imports `from src.db`.
 - `src/pages/observatoire.py` — adapter 3 sites d'appel (lignes ~668, ~791, ~882) ; retirer `query_marches` de l'import `from src.db` (plus utilisé).
 - `tests/test_main.py` — supprimer `test_010_observatoire_montant_filter` (migré en test unitaire du helper).
 ---
 ## Task 1: Tests unitaires — cas par défaut + filtre année
 **Files:**
 - Create: `tests/test_dashboard_filters_to_sql.py`
 - Modify: `src/utils/table_sql.py`
 - [ ] **Step 1: Write the failing tests**
 Create `tests/test_dashboard_filters_to_sql.py`:
 ```python
 from datetime import datetime, timedelta
 from src.utils.table_sql import dashboard_filters_to_sql
 def test_no_filters_uses_default_365_day_window():
    where_sql, params = dashboard_filters_to_sql()
    assert where_sql == '"dateNotification" > ?'
    assert len(params) == 1
    assert isinstance(params[0], datetime)
    expected = datetime.now() - timedelta(days=365)
    assert abs((params[0] - expected).total_seconds()) < 2
 def test_year_filter_overrides_default_window():
    where_sql, params = dashboard_filters_to_sql(dashboard_year="2025")
    assert where_sql == 'YEAR("dateNotification") = ?'
    assert params == [2025]
 ```
 - [ ] **Step 2: Run tests to verify they fail**
 Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
 Expected: FAIL with `ImportError: cannot import name 'dashboard_filters_to_sql'`.
 - [ ] **Step 3: Implement the helper**
 Add to the top of `src/utils/table_sql.py` (below existing imports):
 ```python
 from datetime import datetime, timedelta
 ```
 Append this function at the end of `src/utils/table_sql.py`:
 ```python
 def dashboard_filters_to_sql(
    dashboard_year=None,
    dashboard_acheteur_id=None,
    dashboard_acheteur_categorie=None,
    dashboard_acheteur_departement_code=None,
    dashboard_titulaire_id=None,
    dashboard_titulaire_categorie=None,
    dashboard_titulaire_departement_code=None,
    dashboard_marche_type=None,
    dashboard_marche_objet=None,
    dashboard_marche_code_cpv=None,
    dashboard_marche_considerations_sociales=None,
    dashboard_marche_considerations_environnementales=None,
    dashboard_marche_techniques=None,
    dashboard_marche_innovant=None,
    dashboard_marche_sous_traitance_declaree=None,
    dashboard_montant_min=None,
    dashboard_montant_max=None,
 ) -> tuple[str, list]:
    """Traduit les filtres du tableau de bord en (where_clause, params) DuckDB."""
    clauses: list[str] = []
    params: list = []
    if dashboard_year:
        clauses.append('YEAR("dateNotification") = ?')
        params.append(int(dashboard_year))
    else:
        clauses.append('"dateNotification" > ?')
        params.append(datetime.now() - timedelta(days=365))
    return " AND ".join(clauses), params
 ```
 - [ ] **Step 4: Run tests to verify they pass**
 Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
 Expected: PASS (2 tests).
 - [ ] **Step 5: Commit**
 ```bash
 rtk pre-commit run --files tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
 rtk git add tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
 rtk git commit -m "feat(observatoire): squelette de dashboard_filters_to_sql (#72)"
 ```
 ---
 ## Task 2: Filtres d'égalité simples (catégorie, type, innovant, sous-traitance)
 **Files:**
 - Modify: `tests/test_dashboard_filters_to_sql.py`
 - Modify: `src/utils/table_sql.py`
 - [ ] **Step 1: Add failing tests**
 Append to `tests/test_dashboard_filters_to_sql.py`:
 ```python
 def test_marche_type_equality():
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_marche_type="Marché",
    )
    assert where_sql == 'YEAR("dateNotification") = ? AND "type" = ?'
    assert params == [2025, "Marché"]
 def test_innovant_value_all_is_skipped():
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_marche_innovant="all",
    )
    assert where_sql == 'YEAR("dateNotification") = ?'
    assert params == [2025]
 def test_innovant_value_oui_adds_clause():
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_marche_innovant="oui",
    )
    assert where_sql == 'YEAR("dateNotification") = ? AND "marcheInnovant" = ?'
    assert params == [2025, "oui"]
 def test_sous_traitance_value_non_adds_clause():
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_marche_sous_traitance_declaree="non",
    )
    assert (
        where_sql
        == 'YEAR("dateNotification") = ? AND "sousTraitanceDeclaree" = ?'
    )
    assert params == [2025, "non"]
 ```
 - [ ] **Step 2: Run tests to verify they fail**
 Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
 Expected: 4 new tests FAIL (missing clauses).
 - [ ] **Step 3: Extend the helper**
 Insert the following block in `dashboard_filters_to_sql`, **after** the `if dashboard_year / else` block and **before** `return " AND ".join(clauses), params`:
 ```python
    if dashboard_marche_type:
        clauses.append('"type" = ?')
        params.append(dashboard_marche_type)
    if dashboard_marche_innovant and dashboard_marche_innovant != "all":
        clauses.append('"marcheInnovant" = ?')
        params.append(dashboard_marche_innovant)
    if (
        dashboard_marche_sous_traitance_declaree
        and dashboard_marche_sous_traitance_declaree != "all"
    ):
        clauses.append('"sousTraitanceDeclaree" = ?')
        params.append(dashboard_marche_sous_traitance_declaree)
 ```
 - [ ] **Step 4: Run tests to verify they pass**
 Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
 Expected: PASS (6 tests total).
 - [ ] **Step 5: Commit**
 ```bash
 rtk pre-commit run --files tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
 rtk git add tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
 rtk git commit -m "feat(observatoire): filtres d'égalité simples dans dashboard_filters_to_sql (#72)"
 ```
 ---
 ## Task 3: Filtres LIKE/ILIKE (ids, objet, cpv)
 **Files:**
 - Modify: `tests/test_dashboard_filters_to_sql.py`
 - Modify: `src/utils/table_sql.py`
 - [ ] **Step 1: Add failing tests**
 Append to `tests/test_dashboard_filters_to_sql.py`:
 ```python
 def test_acheteur_id_uses_like_wildcards():
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_acheteur_id="12345678900010",
    )
    assert where_sql == 'YEAR("dateNotification") = ? AND "acheteur_id" LIKE ?'
    assert params == [2025, "%12345678900010%"]
 def test_titulaire_id_uses_like_wildcards():
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_titulaire_id="999",
    )
    assert where_sql == 'YEAR("dateNotification") = ? AND "titulaire_id" LIKE ?'
    assert params == [2025, "%999%"]
 def test_marche_objet_uses_case_insensitive_ilike():
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_marche_objet="travaux",
    )
    assert where_sql == 'YEAR("dateNotification") = ? AND "objet" ILIKE ?'
    assert params == [2025, "%travaux%"]
 def test_code_cpv_uses_prefix_like():
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_marche_code_cpv="4521",
    )
    assert where_sql == 'YEAR("dateNotification") = ? AND "codeCPV" LIKE ?'
    assert params == [2025, "4521%"]
 ```
 - [ ] **Step 2: Run tests to verify they fail**
 Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
 Expected: 4 new tests FAIL.
 - [ ] **Step 3: Extend the helper**
 Insert the following block, **just after** the year/default block and **before** the `if dashboard_marche_type` block:
 ```python
    if dashboard_acheteur_id:
        clauses.append('"acheteur_id" LIKE ?')
        params.append(f"%{dashboard_acheteur_id}%")
    if dashboard_titulaire_id:
        clauses.append('"titulaire_id" LIKE ?')
        params.append(f"%{dashboard_titulaire_id}%")
 ```
 Insert in the "marché" block, **after** `dashboard_marche_type` and **before** `dashboard_marche_innovant`:
 ```python
    if dashboard_marche_objet:
        clauses.append('"objet" ILIKE ?')
        params.append(f"%{dashboard_marche_objet}%")
    if dashboard_marche_code_cpv:
        clauses.append('"codeCPV" LIKE ?')
        params.append(f"{dashboard_marche_code_cpv}%")
 ```
 - [ ] **Step 4: Run tests to verify they pass**
 Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
 Expected: PASS (10 tests total).
 - [ ] **Step 5: Commit**
 ```bash
 rtk pre-commit run --files tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
 rtk git add tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
 rtk git commit -m "feat(observatoire): filtres LIKE/ILIKE dans dashboard_filters_to_sql (#72)"
 ```
 ---
 ## Task 4: Filtre IN (départements) + skip conditionnel par ID
 **Files:**
 - Modify: `tests/test_dashboard_filters_to_sql.py`
 - Modify: `src/utils/table_sql.py`
 - [ ] **Step 1: Add failing tests**
 Append to `tests/test_dashboard_filters_to_sql.py`:
 ```python
 def test_acheteur_departement_multiple_uses_in_clause():
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_acheteur_departement_code=["75", "92", "93"],
    )
    assert where_sql == (
        'YEAR("dateNotification") = ? '
        'AND "acheteur_departement_code" IN (?, ?, ?)'
    )
    assert params == [2025, "75", "92", "93"]
 def test_acheteur_categorie_adds_clause():
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_acheteur_categorie="Commune",
    )
    assert where_sql == 'YEAR("dateNotification") = ? AND "acheteur_categorie" = ?'
    assert params == [2025, "Commune"]
 def test_titulaire_categorie_and_departement():
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_titulaire_categorie="PME",
        dashboard_titulaire_departement_code=["35"],
    )
    assert where_sql == (
        'YEAR("dateNotification") = ? '
        'AND "titulaire_categorie" = ? '
        'AND "titulaire_departement_code" IN (?)'
    )
    assert params == [2025, "PME", "35"]
 def test_acheteur_id_present_skips_categorie_and_departement():
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_acheteur_id="123",
        dashboard_acheteur_categorie="Commune",
        dashboard_acheteur_departement_code=["75"],
    )
    assert where_sql == 'YEAR("dateNotification") = ? AND "acheteur_id" LIKE ?'
    assert params == [2025, "%123%"]
 def test_titulaire_id_present_skips_categorie_and_departement():
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_titulaire_id="999",
        dashboard_titulaire_categorie="PME",
        dashboard_titulaire_departement_code=["35"],
    )
    assert where_sql == 'YEAR("dateNotification") = ? AND "titulaire_id" LIKE ?'
    assert params == [2025, "%999%"]
 ```
 - [ ] **Step 2: Run tests to verify they fail**
 Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
 Expected: 5 new tests FAIL.
 - [ ] **Step 3: Refactor the helper with conditional skip**
 Replace the two simple `if dashboard_acheteur_id` / `if dashboard_titulaire_id` blocks added in Task 3 with the nested form:
 ```python
    if dashboard_acheteur_id:
        clauses.append('"acheteur_id" LIKE ?')
        params.append(f"%{dashboard_acheteur_id}%")
    else:
        if dashboard_acheteur_categorie:
            clauses.append('"acheteur_categorie" = ?')
            params.append(dashboard_acheteur_categorie)
        if dashboard_acheteur_departement_code:
            placeholders = ", ".join(["?"] * len(dashboard_acheteur_departement_code))
            clauses.append(f'"acheteur_departement_code" IN ({placeholders})')
            params.extend(dashboard_acheteur_departement_code)
    if dashboard_titulaire_id:
        clauses.append('"titulaire_id" LIKE ?')
        params.append(f"%{dashboard_titulaire_id}%")
    else:
        if dashboard_titulaire_categorie:
            clauses.append('"titulaire_categorie" = ?')
            params.append(dashboard_titulaire_categorie)
        if dashboard_titulaire_departement_code:
            placeholders = ", ".join(
                ["?"] * len(dashboard_titulaire_departement_code)
            )
            clauses.append(f'"titulaire_departement_code" IN ({placeholders})')
            params.extend(dashboard_titulaire_departement_code)
 ```
 - [ ] **Step 4: Run tests to verify they pass**
 Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
 Expected: PASS (15 tests total).
 - [ ] **Step 5: Commit**
 ```bash
 rtk pre-commit run --files tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
 rtk git add tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
 rtk git commit -m "feat(observatoire): IN départements et skip conditionnel par ID (#72)"
 ```
 ---
 ## Task 5: Filtre liste (techniques, considérations sociales/environnementales)
 **Files:**
 - Modify: `tests/test_dashboard_filters_to_sql.py`
 - Modify: `src/utils/table_sql.py`
 - [ ] **Step 1: Add failing tests**
 Append to `tests/test_dashboard_filters_to_sql.py`:
 ```python
 def test_marche_techniques_uses_list_has_any():
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_marche_techniques=["Enchère", "Accord-cadre"],
    )
    assert where_sql == (
        'YEAR("dateNotification") = ? '
        "AND list_has_any(string_split(\"techniques\", ', '), ?::VARCHAR[])"
    )
    assert params == [2025, ["Enchère", "Accord-cadre"]]
 def test_considerations_sociales_uses_list_has_any():
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_marche_considerations_sociales=["Clause sociale"],
    )
    assert where_sql == (
        'YEAR("dateNotification") = ? '
        "AND list_has_any(string_split(\"considerationsSociales\", ', '), ?::VARCHAR[])"
    )
    assert params == [2025, ["Clause sociale"]]
 def test_considerations_environnementales_uses_list_has_any():
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_marche_considerations_environnementales=["Clause env."],
    )
    assert where_sql == (
        'YEAR("dateNotification") = ? '
        "AND list_has_any(string_split(\"considerationsEnvironnementales\", ', '), ?::VARCHAR[])"
    )
    assert params == [2025, ["Clause env."]]
 ```
 - [ ] **Step 2: Run tests to verify they fail**
 Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
 Expected: 3 new tests FAIL.
 - [ ] **Step 3: Extend the helper**
 Insert the following block in `dashboard_filters_to_sql`, **after** the `dashboard_marche_sous_traitance_declaree` block and **before** `return " AND ".join(clauses), params`:
 ```python
    if dashboard_marche_techniques:
        clauses.append(
            "list_has_any(string_split(\"techniques\", ', '), ?::VARCHAR[])"
        )
        params.append(list(dashboard_marche_techniques))
    if dashboard_marche_considerations_sociales:
        clauses.append(
            "list_has_any(string_split(\"considerationsSociales\", ', '), ?::VARCHAR[])"
        )
        params.append(list(dashboard_marche_considerations_sociales))
    if dashboard_marche_considerations_environnementales:
        clauses.append(
            "list_has_any(string_split(\"considerationsEnvironnementales\", ', '), ?::VARCHAR[])"
        )
        params.append(list(dashboard_marche_considerations_environnementales))
 ```
 - [ ] **Step 4: Run tests to verify they pass**
 Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
 Expected: PASS (18 tests total).
 - [ ] **Step 5: Commit**
 ```bash
 rtk pre-commit run --files tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
 rtk git add tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
 rtk git commit -m "feat(observatoire): filtres liste via list_has_any (#72)"
 ```
 ---
 ## Task 6: Filtres montant min/max (incluant 0)
 **Files:**
 - Modify: `tests/test_dashboard_filters_to_sql.py`
 - Modify: `src/utils/table_sql.py`
 - [ ] **Step 1: Add failing tests**
 Append to `tests/test_dashboard_filters_to_sql.py`:
 ```python
 def test_montant_min_only():
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_montant_min=1000,
    )
    assert where_sql == 'YEAR("dateNotification") = ? AND "montant" >= ?'
    assert params == [2025, 1000]
 def test_montant_max_only():
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_montant_max=500,
    )
    assert where_sql == 'YEAR("dateNotification") = ? AND "montant" <= ?'
    assert params == [2025, 500]
 def test_montant_zero_is_a_valid_lower_bound():
    # 0 est falsy mais reste un filtre valide (distinct de None)
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_montant_min=0,
    )
    assert where_sql == 'YEAR("dateNotification") = ? AND "montant" >= ?'
    assert params == [2025, 0]
 def test_montant_min_and_max_combined():
    where_sql, params = dashboard_filters_to_sql(
        dashboard_year="2025",
        dashboard_montant_min=100,
        dashboard_montant_max=1000,
    )
    assert where_sql == (
        'YEAR("dateNotification") = ? AND "montant" >= ? AND "montant" <= ?'
    )
    assert params == [2025, 100, 1000]
 ```
 - [ ] **Step 2: Run tests to verify they fail**
 Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
 Expected: 4 new tests FAIL.
 - [ ] **Step 3: Extend the helper**
 Insert at the very end of `dashboard_filters_to_sql`, **just before** `return " AND ".join(clauses), params`:
 ```python
    if dashboard_montant_min is not None:
        clauses.append('"montant" >= ?')
        params.append(dashboard_montant_min)
    if dashboard_montant_max is not None:
        clauses.append('"montant" <= ?')
        params.append(dashboard_montant_max)
 ```
 - [ ] **Step 4: Run tests to verify they pass**
 Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
 Expected: PASS (22 tests total).
 - [ ] **Step 5: Commit**
 ```bash
 rtk pre-commit run --files tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
 rtk git add tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
 rtk git commit -m "feat(observatoire): filtres montant min/max (#72)"
 ```
 ---
 ## Task 7: Réécriture de `prepare_dashboard_data`
 **Files:**
 - Modify: `src/utils/data.py`
 - Modify: `tests/test_main.py` (supprimer `test_010_observatoire_montant_filter`)
 - [ ] **Step 1: Remove the obsolete Polars-based test**
 Delete the function `test_010_observatoire_montant_filter` from `tests/test_main.py` (lines ~218-256). La couverture du filtre montant est déjà assurée par les tests unitaires `test_montant_*` de la Task 6.
 - [ ] **Step 2: Rewrite `prepare_dashboard_data`**
 Replace the entire `prepare_dashboard_data` function in `src/utils/data.py` (lines ~86-194) with:
 ```python
 def prepare_dashboard_data(**filter_params) -> pl.DataFrame:
    """Exécute la requête DuckDB filtrée pour le tableau de bord.
    Retourne une pl.DataFrame matérialisée uniquement pour le sous-ensemble
    correspondant aux filtres. Les appelants qui ont besoin d'une LazyFrame
    appellent `.lazy()` sur le résultat.
    """
    from src.utils.table_sql import dashboard_filters_to_sql
    where_sql, params = dashboard_filters_to_sql(**filter_params)
    return query_marches(where_sql=where_sql, params=params)
 ```
 Update the import at the top of `src/utils/data.py`:
 ```python
 from src.db import get_cursor, query_marches, schema
 ```
 Remove the now-unused import in `src/utils/data.py`:
 ```python
 from datetime import datetime, timedelta
 ```
 (Si `datetime` n'est plus référencé dans `data.py` hors de `prepare_dashboard_data`, sinon garder.)
 **Vérification rapide à effectuer avant de supprimer `datetime`/`timedelta`** :
 ```bash
 rtk grep -n "datetime\|timedelta" src/utils/data.py
 ```
 Si d'autres occurrences existent, conserver les imports.
 - [ ] **Step 3: Run the full test suite**
 Run: `rtk pytest tests/test_dashboard_filters_to_sql.py tests/test_main.py -v -k "not selenium and not dash_duo"`
 Ou, si filter n'est pas pratique :
 Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
 Expected: PASS (22 tests).
 - [ ] **Step 4: Commit**
 ```bash
 rtk pre-commit run --files src/utils/data.py tests/test_main.py
 rtk git add src/utils/data.py tests/test_main.py
 rtk git commit -m "refactor(observatoire): prepare_dashboard_data utilise DuckDB (#72)"
 ```
 ---
 ## Task 8: Adaptation des 3 appelants dans `observatoire.py`
 **Files:**
 - Modify: `src/pages/observatoire.py`
 - [ ] **Step 1: Update `_compute_dashboard_children`**
 Remplacer dans `src/pages/observatoire.py` (autour des lignes 660-670) :
 ```python
@cache.memoize()
 def _compute_dashboard_children(filter_params_normalized: tuple):
    logger.debug("Cache miss — computing dashboard")
    filter_params = {
        k: (list(v) if isinstance(v, tuple) else v) for k, v in filter_params_normalized
    }
    lff: pl.LazyFrame = query_marches().lazy()
    lff = prepare_dashboard_data(lff=lff, **filter_params)
    dff = lff.collect(engine="streaming")
 ```
 Par :
 ```python
@cache.memoize()
 def _compute_dashboard_children(filter_params_normalized: tuple):
    logger.debug("Cache miss — computing dashboard")
    filter_params = {
        k: (list(v) if isinstance(v, tuple) else v) for k, v in filter_params_normalized
    }
    dff = prepare_dashboard_data(**filter_params)
    lff = dff.lazy()
 ```
 Le reste de la fonction (à partir de `df_per_uid = ...`) est inchangé.
 - [ ] **Step 2: Update `download_observatoire`**
 Remplacer dans `src/pages/observatoire.py` (autour des lignes 789-800) :
 ```python
 def download_observatoire(_n_clicks, filter_params, hidden_columns):
    lff = prepare_dashboard_data(lff=query_marches().lazy(), **(filter_params or {}))
    if hidden_columns:
        lff = lff.drop(hidden_columns)
    def to_bytes(buffer):
        lff.collect(engine="streaming").write_excel(buffer, worksheet="DECP")
    date = datetime.now().strftime("%Y-%m-%d_%H:%M:%S")
    return dcc.send_bytes(to_bytes, filename=f"decp_observatoire_{date}.xlsx")
 ```
 Par :
 ```python
 def download_observatoire(_n_clicks, filter_params, hidden_columns):
    dff = prepare_dashboard_data(**(filter_params or {}))
    if hidden_columns:
        dff = dff.drop(hidden_columns)
    def to_bytes(buffer):
        dff.write_excel(buffer, worksheet="DECP")
    date = datetime.now().strftime("%Y-%m-%d_%H:%M:%S")
    return dcc.send_bytes(to_bytes, filename=f"decp_observatoire_{date}.xlsx")
 ```
 - [ ] **Step 3: Update `populate_preview_table`**
 Remplacer dans `src/pages/observatoire.py` (autour des lignes 879-892) :
 ```python
    if not is_open:
        return (no_update,) * 9
    lff = prepare_dashboard_data(lff=query_marches().lazy(), **(filter_params or {}))
    return prepare_table_data(
        lff,
        data_timestamp,
        filter_query,
        page_current,
        page_size,
        sort_by,
        "observatoire-preview",
    )
 ```
 Par :
 ```python
    if not is_open:
        return (no_update,) * 9
    dff = prepare_dashboard_data(**(filter_params or {}))
    return prepare_table_data(
        dff.lazy(),
        data_timestamp,
        filter_query,
        page_current,
        page_size,
        sort_by,
        "observatoire-preview",
    )
 ```
 - [ ] **Step 4: Remove unused `query_marches` import**
 Dans `src/pages/observatoire.py`, ligne ~19 :
 ```python
 from src.db import query_marches, schema
 ```
 Devient :
 ```python
 from src.db import schema
 ```
 Vérifier avant de committer :
 ```bash
 rtk grep -n "query_marches" src/pages/observatoire.py
 ```
 Expected: aucun résultat (ou uniquement des commentaires).
 - [ ] **Step 5: Smoke test**
 Démarrer l'app et naviguer sur `/observatoire`, vérifier à la main que :
 - Les cartes s'affichent.
 - Un filtre année se propage.
 - Un filtre acheteur par SIRET partiel fonctionne.
 - Un filtre département (multi-valeur) fonctionne.
 - Un filtre montant_min fonctionne.
 - Le bouton « Télécharger au format Excel » génère un fichier non vide.
 - Le bouton « Voir les données » ouvre l'offcanvas et peuple la table.
 Run: `python run.py`
 Expected: app démarre sans erreur ; les filtres se comportent comme avant.
 - [ ] **Step 6: Commit**
 ```bash
 rtk pre-commit run --files src/pages/observatoire.py
 rtk git add src/pages/observatoire.py
 rtk git commit -m "refactor(observatoire): appelants utilisent la nouvelle signature (#72)"
 ```
 ---
 ## Task 9: Test d'intégration — `prepare_dashboard_data` sur `tests/test.parquet`
 **Files:**
 - Create: `tests/test_prepare_dashboard_data.py`
 - [ ] **Step 1: Write the failing test**
 Le but : vérifier que la fonction s'exécute réellement contre DuckDB, retourne une `pl.DataFrame`, et applique bien les filtres simples. `conftest.py` construit `tests/test.parquet` avec un jeu de données d'une ligne : acheteur_id `123`, acheteur_departement_code `75`, dateNotification `2025-01-01`, montant `10`.
 Create `tests/test_prepare_dashboard_data.py`:
 ```python
 import polars as pl
 def test_returns_dataframe_with_year_filter():
    from src.utils.data import prepare_dashboard_data
    dff = prepare_dashboard_data(dashboard_year="2025")
    assert isinstance(dff, pl.DataFrame)
    assert dff.height == 1
 def test_year_mismatch_returns_empty():
    from src.utils.data import prepare_dashboard_data
    dff = prepare_dashboard_data(dashboard_year="2024")
    assert isinstance(dff, pl.DataFrame)
    assert dff.height == 0
 def test_acheteur_id_partial_match():
    from src.utils.data import prepare_dashboard_data
    dff = prepare_dashboard_data(
        dashboard_year="2025",
        dashboard_acheteur_id="12",
    )
    assert dff.height == 1
 def test_departement_in_clause():
    from src.utils.data import prepare_dashboard_data
    dff = prepare_dashboard_data(
        dashboard_year="2025",
        dashboard_acheteur_departement_code=["75", "92"],
    )
    assert dff.height == 1
 def test_montant_min_above_value_excludes_row():
    from src.utils.data import prepare_dashboard_data
    dff = prepare_dashboard_data(
        dashboard_year="2025",
        dashboard_montant_min=1000,
    )
    assert dff.height == 0
 ```
 - [ ] **Step 2: Run the test**
 Run: `rtk pytest tests/test_prepare_dashboard_data.py -v`
 Expected: PASS (5 tests).
 - [ ] **Step 3: Commit**
 ```bash
 rtk pre-commit run --files tests/test_prepare_dashboard_data.py
 rtk git add tests/test_prepare_dashboard_data.py
 rtk git commit -m "test(observatoire): intégration DuckDB pour prepare_dashboard_data (#72)"
 ```
 ---
 ## Task 10: Vérification finale
 **Files:** (aucune modification)
 - [ ] **Step 1: Run the full test suite**
 Run: `rtk pytest -v`
 Expected: tous les tests unitaires passent. Les tests Selenium peuvent échouer si Chrome n'est pas disponible — ce n'est pas bloquant s'ils étaient déjà rouges avant.
 - [ ] **Step 2: Check for leftover references**
 Run: `rtk grep -rn "prepare_dashboard_data(lff" src/ tests/`
 Expected: aucun résultat (plus d'appels avec l'ancienne signature).
 Run: `rtk grep -rn "query_marches().lazy()" src/`
 Expected: aucun résultat (ou uniquement dans `src/utils/table.py:prepare_table_data` pour le fallback).
 - [ ] **Step 3: Confirm `datetime`/`timedelta` in data.py if needed**
 Run: `rtk grep -n "datetime\|timedelta" src/utils/data.py`
 Si aucune occurrence hors imports, vérifier que les imports inutiles ont bien été retirés dans Task 7.
 - [ ] **Step 4: Manual timing sanity check (optionnel)**
 Si possible, comparer informellement le temps de `_compute_dashboard_children` sur un filtre sélectif (ex. un département) avant/après. Pas de benchmark formel attendu.
 - [ ] **Step 5: Push (manuel, à l'initiative de l'utilisateur)**
 Conformément aux consignes projet, ne jamais `git push`. Laisser l'utilisateur pousser la branche `feature/72_observatoire_duckdb_filters` et ouvrir la PR.