Compare commits

..

68 Commits

Author SHA1 Message Date
Colin Maudry 62eb4d98f0 Correction de l'ajout de CORS 2026-05-05 15:32:27 +02:00
Colin Maudry 1447a9fcaf Merge branch 'release/2.7.6' 2026-05-05 15:05:55 +02:00
Colin Maudry 9d7f33905f Bumped version 2.7.6 2026-05-05 15:05:36 +02:00
Colin Maudry 755b8c13ab Bug nom colonne, amélioration test #76 2026-05-05 14:56:59 +02:00
Colin Maudry f4b57dbe5c Gestion des filtres de dates comme du texte #76 2026-05-05 14:08:50 +02:00
Colin Maudry 0322c20513 Suppression des espaces pour l'affichage du nom de l'org #75 2026-04-29 18:41:45 +02:00
Colin Maudry edbdeaa370 Suppression des espaces dans les SIREN/SIRET entrés dans l'observatoire et le tableau #75 2026-04-29 18:25:22 +02:00
Colin Maudry 25746b4869 Amélioration du rendu de la carte org 2026-04-28 13:34:33 +02:00
Colin Maudry 1a5f049b1a Map fonctionne mais lf cassé 2026-04-28 11:23:02 +02:00
Colin Maudry bd6a4ff266 Tentative de rétablissement de la carte sur acheteur/titulaire 2026-04-28 10:40:50 +02:00
Colin Maudry 1839928e69 Réduction des petites erreurs 2026-04-24 13:36:15 +02:00
Colin Maudry a6049b3244 Bumped version checkout 2026-04-24 13:35:49 +02:00
Colin Maudry d8ee6e5b37 Màj du mode d'emploi de Tableau #42 2026-04-24 12:31:46 +02:00
Colin Maudry b437decf5f Merge branch 'main' into dev 2026-04-24 12:10:51 +02:00
Colin Maudry 10f24dec30 Petites corrections 2026-04-24 12:08:48 +02:00
Colin Maudry 18b5488051 Merge tag 'v2.7.5' into dev
- Amélioration des permormances de l'observatoire
- Possibilité dans observatoire (champ objet) et tableau (tous champs texte) de soit chercher des mots présents, soit une suite de mot précise (voir mode d'emploi dans Tableau)
- Ajout d'une animation pendant le chargement de la prévisualisation des données de l'observatoire
2026-04-24 11:51:18 +02:00
Colin Maudry 7d8f8a7c19 Merge branch 'release/2.7.5' 2026-04-24 11:50:47 +02:00
Colin Maudry 93777cce6d Changelog v2.7.5 2026-04-24 11:50:36 +02:00
Colin Maudry dadbb0aeff Possibilité de chercher soit des mots présents, soit une suite de mot précise #42 2026-04-24 11:40:02 +02:00
Colin Maudry f7b7954ed2 Merge branch 'feature/72_observatoire_duckdb_filters' into dev 2026-04-23 12:33:48 +02:00
Colin Maudry fc4d965b20 Spinner de chargemetn sur la préviusalisation des données 2026-04-23 12:33:30 +02:00
Colin Maudry a715140af0 test(observatoire): intégration DuckDB pour prepare_dashboard_data (#72) 2026-04-23 00:07:24 +02:00
Colin Maudry c45d4e0ea1 refactor(observatoire): appelants utilisent la nouvelle signature (#72)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 23:57:10 +02:00
Colin Maudry 0777153c82 refactor(observatoire): prepare_dashboard_data utilise DuckDB (#72) 2026-04-22 23:55:45 +02:00
Colin Maudry 6e670c97c9 feat(observatoire): filtres montant min/max (#72) 2026-04-22 23:53:43 +02:00
Colin Maudry 522c467702 feat(observatoire): filtres liste via list_has_any (#72) 2026-04-22 23:49:32 +02:00
Colin Maudry 74ae1fb008 feat(observatoire): IN départements et skip conditionnel par ID (#72) 2026-04-22 23:43:55 +02:00
Colin Maudry e3a0fba4df feat(observatoire): filtres LIKE/ILIKE dans dashboard_filters_to_sql (#72) 2026-04-22 23:42:43 +02:00
Colin Maudry a382370767 feat(observatoire): filtres d'égalité simples dans dashboard_filters_to_sql (#72) 2026-04-22 23:39:13 +02:00
Colin Maudry 653999693c feat(observatoire): squelette de dashboard_filters_to_sql (#72) 2026-04-22 23:36:59 +02:00
Colin Maudry aaf54eef91 docs(observatoire): plan d'implémentation filtrage natif DuckDB (#72)
Plan en 10 tâches TDD : construction incrémentale de dashboard_filters_to_sql,
réécriture de prepare_dashboard_data, adaptation des 3 appelants de
observatoire.py, test d'intégration sur tests/test.parquet.
2026-04-22 23:21:47 +02:00
Colin Maudry b996eb97cc docs(observatoire): spec du filtrage natif DuckDB (#72)
Décrit la refonte de prepare_dashboard_data pour pousser le filtrage au
niveau DuckDB via un nouveau helper dashboard_filters_to_sql, sur le
modèle de filter_query_to_sql / _fetch_page_sql.
2026-04-22 23:16:41 +02:00
Colin Maudry 3e89dacff9 Correction de setup_table_columns et autres 2026-04-22 21:16:51 +02:00
Colin Maudry eb8d7abe0d actions/checkout@v4 2026-04-22 20:55:22 +02:00
Colin Maudry a484984e40 Merge tag 'v2.7.4' into dev
- Utilisation élargie de DuckDB au détriment de Polars => bien meilleure perf ([#72](https://github.com/ColinMaudry/decp.info/issues/72)
2026-04-22 20:50:47 +02:00
Colin Maudry bea160aa00 Merge branch 'release/2.7.4' 2026-04-22 20:50:12 +02:00
Colin Maudry 8e603d2806 Bump version number 2026-04-22 20:49:57 +02:00
Colin Maudry 1d833bb800 Merge branch 'feature/72_duckdb_performance' into dev 2026-04-22 20:48:39 +02:00
Colin Maudry f81c897342 Changelog 2.7.4 2026-04-22 20:48:27 +02:00
Colin Maudry 3532a9c381 Path de duckdb configurable, correction des tests #72 2026-04-22 20:43:38 +02:00
Colin Maudry b4956c34d1 Utilisation d'une seule fonction postprocess #72 2026-04-22 18:30:36 +02:00
Colin Maudry 72d4881796 Simplifications du code #72 2026-04-22 17:17:54 +02:00
Colin Maudry 1e67d329d0 perf(tableau): pousser filtre/tri/pagination/comptage dans DuckDB
Remplace le chemin lent de prepare_table_data (chargement de toutes les
lignes depuis DuckDB puis filtrage/post-traitement Polars avant slice)
par _fetch_page_sql qui pousse filtre, tri, pagination et comptage dans
DuckDB via filter_query_to_sql / sort_by_to_sql, puis post-traite
uniquement la page de 20 lignes.

Supprime _load_filter_sort_postprocess (plus utilisé). Met à jour les
tests test_table.py en supprimant les tests associés et en ajoutant
des tests dédiés pour _fetch_page_sql. Corrige le fixture flask_app
pour utiliser src.utils.cache (même instance que le module) afin que
@cache.memoize() fonctionne.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-21 23:53:45 +02:00
Colin Maudry 4e25ff5c85 feat(table): ajouter postprocess_page pour post-traiter une page seule
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-21 23:42:52 +02:00
Colin Maudry cdb6a70f7a feat(db): ajouter count_marches, count_unique_marches et paramètre offset à query_marches
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-21 23:39:51 +02:00
Colin Maudry 74213d3844 fix(table_sql): gérer *foo* et utiliser isinstance pour les types Polars
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-21 23:37:03 +02:00
Colin Maudry 95c90e319e feat: ajouter traducteurs filter_query→SQL et sort_by→SQL
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-21 23:33:49 +02:00
Colin Maudry abc6390174 Améliorations CLAUDE.md pour plus utiliser rtk 2026-04-21 21:55:18 +02:00
Colin Maudry 6d22b7298a Merge tag 'v2.7.3' into dev
- Mise en cache des vues tableau par ensemble de filtres et de tris
- Résolution du bug d'écriture du fichier de vérouillage de la base de données
2026-04-20 11:55:55 +02:00
Colin Maudry e33e5da619 Merge branch 'release/2.7.3' 2026-04-20 11:42:17 +02:00
Colin Maudry ffeb708f1d Changelog 2.7.3 2026-04-20 11:42:04 +02:00
Colin Maudry aa445b6f01 Plan #72 2026-04-20 11:36:38 +02:00
Colin Maudry 330ed4f0cb Base de données et .lock à la racine de decp.info 2026-04-20 11:28:00 +02:00
Colin Maudry da5a99b3af Merge branch 'main' into dev 2026-04-20 10:49:31 +02:00
Colin Maudry 285ed37d79 Correction de l'import d'utils.cache 2026-04-20 00:06:44 +02:00
Colin Maudry e44fe452b2 Corrections de typage et d'appels à filter_table_data 2026-04-19 23:55:20 +02:00
Colin Maudry 7aef7acd34 Petits ajustements (cache => utils, noms de variables) 2026-04-19 23:49:02 +02:00
Colin Maudry 3ce6f224ae rtk, uv, pyproject 2026-04-19 23:39:33 +02:00
Colin Maudry c7c7c2c62c Factorisation des opération de postprocess des tables 2026-04-19 23:28:45 +02:00
Colin Maudry ad58c1152a Améliorations sur le typage 2026-04-19 23:23:07 +02:00
Colin Maudry 19449969d6 Mention de rtk dans CLAUDE.md 2026-04-19 23:22:48 +02:00
Colin Maudry b0d2aca4ff perf(tableau): memoize filter+sort+postprocess pipeline (#72)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-19 22:50:32 +02:00
Colin Maudry 0abbd982ea feat: add memoized _load_filter_sort_postprocess helper (#72)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-19 22:39:13 +02:00
Colin Maudry 18d07b5398 feat: add normalize_sort_by hashable cache-key helper (#72)
Add normalize_sort_by function to convert Dash DataTable's sort_by list
(unhashable) into a tuple representation (hashable) for use in cache keys.
Includes TDD-driven tests for empty inputs, hashability, and order preservation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-19 22:36:55 +02:00
Colin Maudry ff425108b0 fix: restore track_search import in table.py (#72) 2026-04-19 22:34:25 +02:00
Colin Maudry 0c7ca04f8e refactor: move track_search out of filter_table_data into callers (#72)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-19 22:32:28 +02:00
Colin Maudry 89904a5bad test: scaffold unit tests for table utilities (#72) 2026-04-19 22:30:28 +02:00
Colin Maudry 5ccfec35e9 Merge tag 'v2.7.2' into dev
- Chargement des données depuis une base DuckDB plutôt qu'en mémoire (plus de stabilité) ([#71](https://github.com/ColinMaudry/decp.info/issues/71))
- Mise en cache des vue sur l'observatoire pour un chargement plus rapide (remise à zéro quotidienne)
- Correction de bug : la liste de colonnes par défaut est bien appliquée plutôt qu'afficher toutes les colonnes
- Quelques corrections de bugs d'affichage
- Refactorisation des fonctions utilitaires (`utils.py` approchait des 1 000 lignes)
2026-04-19 15:33:58 +02:00
33 changed files with 4505 additions and 361 deletions
+1 -1
View File
@@ -20,7 +20,7 @@ jobs:
environment: ${{ github.ref_name }}
steps:
- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v5
- name: Set up SSH key
run: |
+1
View File
@@ -1,4 +1,5 @@
DATA_FILE_PARQUET_PATH=https://www.data.gouv.fr/fr/datasets/r/11cea8e8-df3e-4ed1-932b-781e2635e432
DUCKDB_PATH=./decp.duckdb
PORT=8050
DEVELOPMENT=True
SOURCE_STATS_CSV_PATH="https://www.data.gouv.fr/api/1/datasets/r/8ded94de-3b80-4840-a5bb-7faad1c9c234"
+23 -2
View File
@@ -1,4 +1,25 @@
#### 2.7.2 (19 avril 2026)
##### 2.7.6 (5 mai 2026)
- Correction du problème de filtre par date dans les tableaux
- Retour des cartes dans les pages acheteur et titulaire
- Possibilité de chercher un SIRET/SIREN avec des espaces dans les champs `SIRET acheteur` et `Identifiant titulaire`
##### 2.7.5 (24 avril 2026)
- Amélioration des permormances de l'observatoire
- Possibilité dans observatoire (champ objet) et tableau (tous champs texte) de soit chercher des mots présents, soit une suite de mot précise (voir mode d'emploi dans Tableau)
- Ajout d'une animation pendant le chargement de la prévisualisation des données de l'observatoire
##### 2.7.4 (22 avril 2026)
- Utilisation élargie de DuckDB au détriment de Polars => bien meilleure perf ([#72](https://github.com/ColinMaudry/decp.info/issues/72)
##### 2.7.3 (20 avril 2026)
- Mise en cache des vues tableau par ensemble de filtres et de tris
- Résolution du bug d'écriture du fichier de vérouillage de la base de données
##### 2.7.2 (19 avril 2026)
- Chargement des données depuis une base DuckDB plutôt qu'en mémoire (plus de stabilité) ([#71](https://github.com/ColinMaudry/decp.info/issues/71))
- Mise en cache des vue sur l'observatoire pour un chargement plus rapide (remise à zéro quotidienne)
@@ -6,7 +27,7 @@
- Quelques corrections de bugs d'affichage
- Refactorisation des fonctions utilitaires (`utils.py` approchait des 1 000 lignes)
#### 2.7.1 (23 mars 2026)
##### 2.7.1 (23 mars 2026)
- Correction du partage de données filtrées entre dashboard et vue des données
+19 -10
View File
@@ -10,16 +10,25 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
### Setup
Setting up the virtual environment:
```bash
python -m venv .venv && source .venv/bin/activate
pip install ".[dev]"
cp template.env .env # then customize .env
python -m venv .venv # s'il n'existe pas déjà
source .venv/bin/activate
rtk pip install -U pip > /dev/null 2>&1
rtk pip install -e . --group=dev
```
Environment variables:
```bash
cp .template.env .env # then customize .env
```
### Development
```bash
uv run run.py # starts Dash with debug=True and hot reload
python run.py # starts Dash app
```
### Production
@@ -31,8 +40,8 @@ gunicorn app:server
### Tests
```bash
uv run pytest # run all tests (Selenium-based integration tests)
uv run pytest tests/test_main.py::test_001_logo_and_search # run a single test
rtk pytest # run all tests (some are Selenium-based integration tests)
rtk pytest tests/test_main.py::test_001_logo_and_search # run a single test
```
Tests require a running Chrome/Chromium browser. They use `DashComposite` from `dash[testing]` with Selenium WebDriver.
@@ -42,12 +51,12 @@ Tests require a running Chrome/Chromium browser. They use `DashComposite` from `
### Multi-page Dash app
- `src/app.py` — creates the Dash app instance, navbar, SEO endpoints (robots.txt, sitemap.xml), Matomo analytics
- `src/pages/*.py` — each page registers itself with `@register_page()` and owns its own layout and callbacks
- `src/pages/*.py` — each page registers itself with `@register_page()` and o.wns its own layout and callbacks
- `run.py` — dev entry point; exports `server` (Flask) for gunicorn
### Module imports
- always import modules from the app starting with `src.` (e.g. `src.utils.`, `src.pages.recherche`, etc.)
- always import modules from the app starting with `src.` (e.g. `src.utils.`, `src.pages.recherche`, etc.), NOT `utils.cache` or `pages.observatoire`.
### Key pages
@@ -62,9 +71,9 @@ Tests require a running Chrome/Chromium browser. They use `DashComposite` from `
### Data layer
- Data is stored as **Parquet** and loaded with **Polars** (fast columnar operations)
- Data is stored as **Parquet** at rest, possibly in DuckDB, loaded in DuckDB, served from DuckDB for big queries and manipulated with **Polars** for the remaining steps
- Path set via `DATA_FILE_PARQUET_PATH` env var; tests use `tests/test.parquet`
- `src/utils.py`filtering helpers, search (`search_org`), link generation, geographic data loading
- `src/util/*.py`helpers shared by other modules, search (`search_org`), link generation, geographic data loading
- `src/callbacks.py` — shared Dash callbacks (e.g. `get_top_org_table`)
- `src/figures.py` — chart and map components (Plotly Express, Dash Leaflet with marker clustering)
- a Parquet file with production data is located at `../decp-processing/decp_prod.parquet` (~ 1,5 million records)
+2 -7
View File
@@ -1,6 +1,5 @@
# decp.info
> v2.7.2
> Outil d'exploration et de téléchargement des données essentielles de la commande publique.
=> [decp.info](https://decp.info)
@@ -8,19 +7,15 @@
## Installation et lancement
```shell
python -m venv .venv
source .venv/bin/activate
pip install .
# Copie et personnalisation du .env
cp template.env .env
nano .env
# Pour la production
gunicorn app:server
uv run gunicorn app:server
# Pour avoir le debuggage et le hot reload
python run.py
uv run run.py
```
## Déploiement
@@ -0,0 +1,838 @@
# Tableau prepare_table_data Cache Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Make page navigation, sort changes, and repeated filter visits in the `/tableau` page near-instant by memoizing the expensive filter+sort+post-process pipeline inside `prepare_table_data`.
**Architecture:** Extract a memoized inner function `_load_filter_sort_postprocess(filter_query, sort_by_key)` that performs the heavy work (load full data, filter, sort, collect, cast-to-string, fill-null, add HTML links, format values) and returns a fully post-processed Polars DataFrame. The outer `prepare_table_data` becomes a thin wrapper that handles non-deterministic side effects (`track_search`, `uuid.uuid4()` for cleanup trigger, `data_timestamp + 1`) and pagination. The memoized helper only runs when no `data` argument is passed (i.e., the Tableau path). Other callers (`acheteur`, `titulaire`, `observatoire`) keep the current uncached path because they pass an externally-provided LazyFrame that is not safely hashable for cache keys.
**Tech Stack:** Polars (LazyFrame, DataFrame), Flask-Caching (`@cache.memoize()` on `FileSystemCache` already configured in `src/app.py:38`), pytest for unit tests.
**Git**: the issue id is #72, add the reference in commit messages.
---
## Background and constraints
Read these before starting; they explain why the design takes the shape it does.
1. **Cache infrastructure is already wired.** `src/cache.py` defines `cache = Cache()`. `src/app.py:38-48` initializes it with `FileSystemCache`, default 24h timeout, `CACHE_THRESHOLD=300`. The cache directory is wiped on every restart (`rmtree` at `src/app.py:36`), so cache always starts empty.
2. **Existing pattern to mirror.** `src/pages/observatoire.py:650-660` already uses `@cache.memoize()` plus a `_normalize_filter_params` helper that converts a dict of filters into a hashable tuple. This plan applies the same idiom to `sort_by` (which is a `list[dict]` from Dash DataTable).
3. **Non-deterministic outputs that MUST stay outside the memoized function:**
- `data_timestamp + 1` (increments each call; would freeze if cached)
- `trigger_cleanup = str(uuid.uuid4())` (intentionally unique per call to fire the clientside filter-cleanup callback)
- `track_search(filter_query, source_table)` — Matomo HTTP POST, currently called inside `filter_table_data` at `src/utils/table.py:214`. Must fire on every user action including cache hits.
4. **Tracking call site move.** `track_search` must move OUT of `filter_table_data` and into each caller, otherwise cache hits would silently skip Matomo tracking. Current callers of `filter_table_data` to update:
- `src/utils/table.py:402` (inside `prepare_table_data`)
- `src/pages/tableau.py:325` (`download_data` callback)
- `src/pages/acheteur.py:427` (`download_data_acheteur` callback)
- `src/pages/titulaire.py:443` (`download_data_titulaire` callback)
5. **Why Tableau-only caching.** `prepare_table_data` is also called from `acheteur.py`, `titulaire.py`, `observatoire.py`. Those callers pass a pre-filtered LazyFrame or list-of-dicts as `data`. Hashing arbitrary LazyFrames or large lists for memoization is impractical. The fix gates on `data is None` (the Tableau path) and leaves the other paths byte-for-byte identical.
6. **Cache key composition.** The memoized function takes only `(filter_query, sort_by_key)`. `page_current` and `page_size` are intentionally NOT in the key — pagination happens in the outer wrapper after retrieving the cached, fully post-processed frame. This means every page click and page-size change is a cache hit (the whole point of the change).
7. **Pickling.** Flask-Caching pickles arguments to form keys and pickles return values to disk. Polars `DataFrame` pickles cleanly. `LazyFrame` does not — so the memoized function must `.collect()` before returning.
8. **File path expectations.** All paths below are relative to repo root `/home/colin/git/decp.info`. Run all commands from there.
---
## File Structure
- **Modify** `src/utils/table.py` — extract memoized helper, refactor `prepare_table_data`, remove `track_search` call from `filter_table_data`.
- **Modify** `src/pages/tableau.py` — add explicit `track_search` call in `download_data`.
- **Modify** `src/pages/acheteur.py` — add explicit `track_search` call in `download_data_acheteur`.
- **Modify** `src/pages/titulaire.py` — add explicit `track_search` call in `download_data_titulaire`.
- **Create** `tests/test_table.py` — unit tests for new helpers and refactored `prepare_table_data`.
---
## Task 1: Set up unit tests for table.py
**Files:**
- Create: `tests/test_table.py`
This task scaffolds a non-Selenium pytest module so subsequent tasks can do TDD without booting a Dash server. The conftest already writes a small `tests/test.parquet` fixture (see `tests/conftest.py:10`); reuse it.
- [ ] **Step 1: Write the failing test**
Create `tests/test_table.py` with:
```python
import os
import polars as pl
import pytest
@pytest.fixture
def sample_lff():
"""Small LazyFrame with the columns needed by add_links / format_values."""
return pl.LazyFrame(
[
{
"uid": "u1",
"id": "u1",
"acheteur_id": "12345678900011",
"acheteur_nom": "Mairie de Test",
"titulaire_id": "98765432100022",
"titulaire_nom": "Entreprise Test",
"titulaire_typeIdentifiant": "SIRET",
"objet": "Travaux divers",
"montant": 12500.0,
"dateNotification": "2025-03-15",
"codeCPV": "45000000",
"dureeRestanteMois": 6,
"titulaire_distance": 42.0,
}
]
)
def test_table_module_imports():
from src.utils import table
assert hasattr(table, "prepare_table_data")
```
- [ ] **Step 2: Run test to verify it passes (sanity check)**
Run: `uv run pytest tests/test_table.py -v`
Expected: PASS for `test_table_module_imports`. (Selenium is not invoked because no `dash_duo` fixture is used.)
- [ ] **Step 3: Commit**
```bash
git add tests/test_table.py
git commit -m "test: scaffold unit tests for table utilities"
```
---
## Task 2: Move track_search out of filter_table_data
**Files:**
- Modify: `src/utils/table.py:210-274` (remove `track_search` import usage at line 214)
- Modify: `src/pages/tableau.py:317-334` (`download_data` callback)
- Modify: `src/pages/acheteur.py:425-430` area (`download_data_acheteur` callback)
- Modify: `src/pages/titulaire.py:441-446` area (`download_data_titulaire` callback)
- Modify: `tests/test_table.py` (add a test that confirms `filter_table_data` no longer calls Matomo)
`track_search` must move out so that the soon-to-be-memoized helper does not swallow tracking on cache hits. We do this BEFORE introducing caching so that the diff is small and verifiable on its own.
- [ ] **Step 1: Write the failing test**
Append to `tests/test_table.py`:
```python
def test_filter_table_data_does_not_call_track_search(monkeypatch, sample_lff):
from src.utils import table
calls = []
monkeypatch.setattr(table, "track_search", lambda *a, **kw: calls.append(a))
result = table.filter_table_data(
sample_lff, "{objet} icontains travaux", "tableau"
).collect()
assert calls == []
assert result.height == 1
```
- [ ] **Step 2: Run test to verify it fails**
Run: `uv run pytest tests/test_table.py::test_filter_table_data_does_not_call_track_search -v`
Expected: FAIL (`assert calls == []` fails because `filter_table_data` currently calls `track_search` at line 214).
- [ ] **Step 3: Remove the track_search call from filter_table_data**
Edit `src/utils/table.py` — find this block:
```python
def filter_table_data(
lff: pl.LazyFrame, filter_query: str, filter_source: str
) -> pl.LazyFrame:
_schema = lff.collect_schema()
track_search(filter_query, filter_source)
filtering_expressions = filter_query.split(" && ")
```
Remove the `track_search(filter_query, filter_source)` line. Result:
```python
def filter_table_data(
lff: pl.LazyFrame, filter_query: str, filter_source: str
) -> pl.LazyFrame:
_schema = lff.collect_schema()
filtering_expressions = filter_query.split(" && ")
```
The `filter_source` parameter remains in the signature (avoids changing all callers in this task). It becomes unused; that is acceptable since callers will pass it again later if needed. Do NOT remove the `from src.utils.tracking import track_search` import yet — `prepare_table_data` will use it in Task 5.
- [ ] **Step 4: Add explicit track_search calls in download callbacks**
In `src/pages/tableau.py`, find:
```python
def download_data(n_clicks, filter_query, sort_by, hidden_columns: list = None):
lff: pl.LazyFrame = query_marches().lazy()
# Les colonnes masquées sont supprimées
if hidden_columns:
lff = lff.drop(hidden_columns)
if filter_query:
lff = filter_table_data(lff, filter_query, "tab download")
```
Insert a `track_search` call so behavior is preserved. First add the import at the top of `src/pages/tableau.py` next to other `src.utils` imports:
```python
from src.utils.tracking import track_search
```
Then change the body:
```python
def download_data(n_clicks, filter_query, sort_by, hidden_columns: list = None):
lff: pl.LazyFrame = query_marches().lazy()
# Les colonnes masquées sont supprimées
if hidden_columns:
lff = lff.drop(hidden_columns)
if filter_query:
track_search(filter_query, "tab download")
lff = filter_table_data(lff, filter_query, "tab download")
```
Repeat the same pattern in `src/pages/acheteur.py` (search for `filter_table_data(lff, filter_query, "ach download")`):
Add import:
```python
from src.utils.tracking import track_search
```
Wrap the call:
```python
if filter_query:
track_search(filter_query, "ach download")
lff = filter_table_data(lff, filter_query, "ach download")
```
Repeat in `src/pages/titulaire.py` (search for `filter_table_data(lff, filter_query, "titu download")`):
Add import:
```python
from src.utils.tracking import track_search
```
Wrap the call:
```python
if filter_query:
track_search(filter_query, "titu download")
lff = filter_table_data(lff, filter_query, "titu download")
```
- [ ] **Step 5: Run test to verify it passes**
Run: `uv run pytest tests/test_table.py::test_filter_table_data_does_not_call_track_search -v`
Expected: PASS.
- [ ] **Step 6: Run full unit test file to verify no regressions**
Run: `uv run pytest tests/test_table.py -v`
Expected: All tests in `test_table.py` PASS.
- [ ] **Step 7: Commit**
```bash
git add src/utils/table.py src/pages/tableau.py src/pages/acheteur.py src/pages/titulaire.py tests/test_table.py
git commit -m "refactor: move track_search out of filter_table_data into callers"
```
---
## Task 3: Add normalize_sort_by helper
**Files:**
- Modify: `src/utils/table.py` (add helper near other utility functions, e.g. after `dates_to_strings`)
- Modify: `tests/test_table.py` (add tests)
A cache key must be hashable. Dash DataTable's `sort_by` is a `list[dict]` like `[{"column_id": "montant", "direction": "asc"}, ...]`, which is not hashable. We mirror the `_normalize_filter_params` idiom from `src/pages/observatoire.py:650-657`.
- [ ] **Step 1: Write the failing tests**
Append to `tests/test_table.py`:
```python
def test_normalize_sort_by_handles_empty():
from src.utils.table import normalize_sort_by
assert normalize_sort_by(None) == ()
assert normalize_sort_by([]) == ()
def test_normalize_sort_by_returns_hashable_tuple():
from src.utils.table import normalize_sort_by
sort_by = [
{"column_id": "montant", "direction": "desc"},
{"column_id": "dateNotification", "direction": "asc"},
]
key = normalize_sort_by(sort_by)
assert key == (("montant", "desc"), ("dateNotification", "asc"))
# Must be hashable so that flask-caching can build a cache key from it
hash(key)
def test_normalize_sort_by_preserves_order():
"""Order matters for sort: [A, B] != [B, A]."""
from src.utils.table import normalize_sort_by
a_then_b = normalize_sort_by(
[{"column_id": "a", "direction": "asc"}, {"column_id": "b", "direction": "asc"}]
)
b_then_a = normalize_sort_by(
[{"column_id": "b", "direction": "asc"}, {"column_id": "a", "direction": "asc"}]
)
assert a_then_b != b_then_a
```
- [ ] **Step 2: Run tests to verify they fail**
Run: `uv run pytest tests/test_table.py -v -k normalize_sort_by`
Expected: FAIL with `ImportError` for `normalize_sort_by`.
- [ ] **Step 3: Implement normalize_sort_by**
Edit `src/utils/table.py`. Add this function immediately after the `dates_to_strings` function (around line 148):
```python
def normalize_sort_by(sort_by) -> tuple:
"""Convert Dash DataTable sort_by (list[dict]) into a hashable tuple
suitable for use as a cache key. Order is preserved because it determines
sort precedence."""
if not sort_by:
return ()
return tuple((entry["column_id"], entry["direction"]) for entry in sort_by)
```
- [ ] **Step 4: Run tests to verify they pass**
Run: `uv run pytest tests/test_table.py -v -k normalize_sort_by`
Expected: 3 PASS.
- [ ] **Step 5: Commit**
```bash
git add src/utils/table.py tests/test_table.py
git commit -m "feat: add normalize_sort_by hashable cache-key helper"
```
---
## Task 4: Extract memoized post-process helper
**Files:**
- Modify: `src/utils/table.py` (add `_load_filter_sort_postprocess`, decorate with `@cache.memoize()`, import `cache`)
- Modify: `tests/test_table.py` (add tests)
Introduce the function whose result will live in the FileSystemCache. Inputs: `(filter_query, sort_by_key)`. Output: a fully post-processed, unpaginated Polars DataFrame ready to slice and convert to dicts.
This task does NOT yet wire the helper into `prepare_table_data` — that happens in Task 5. Splitting these tasks keeps each diff small and testable.
- [ ] **Step 1: Write the failing tests**
Append to `tests/test_table.py`:
```python
@pytest.fixture(autouse=True)
def reset_cache():
"""Ensure the flask-caching backend is empty between tests so that
cache-hit assertions are meaningful. Falls back to no-op when no
Flask app context is active (NullCache)."""
from utils.cache import cache
try:
cache.clear()
except RuntimeError:
# No app context — cache is NullCache, nothing to clear
pass
yield
def test_load_filter_sort_postprocess_returns_dataframe(monkeypatch, sample_lff):
from src.utils import table
monkeypatch.setattr(
table, "query_marches", lambda: sample_lff.collect()
)
df = table._load_filter_sort_postprocess(filter_query=None, sort_by_key=())
assert isinstance(df, pl.DataFrame)
assert df.height == 1
# All values must be strings after post-processing
for col in df.columns:
assert df.schema[col] == pl.String
def test_load_filter_sort_postprocess_applies_filter(monkeypatch, sample_lff):
from src.utils import table
monkeypatch.setattr(
table, "query_marches", lambda: sample_lff.collect()
)
df = table._load_filter_sort_postprocess(
filter_query="{objet} icontains travaux", sort_by_key=()
)
assert df.height == 1
df_empty = table._load_filter_sort_postprocess(
filter_query="{objet} icontains nonexistent", sort_by_key=()
)
assert df_empty.height == 0
def test_load_filter_sort_postprocess_adds_links(monkeypatch, sample_lff):
from src.utils import table
monkeypatch.setattr(
table, "query_marches", lambda: sample_lff.collect()
)
df = table._load_filter_sort_postprocess(filter_query=None, sort_by_key=())
# add_links injects an <a href> wrapper around uid, acheteur_nom, titulaire_nom
assert "<a href" in df["uid"][0]
assert "<a href" in df["acheteur_nom"][0]
assert "<a href" in df["titulaire_nom"][0]
```
- [ ] **Step 2: Run tests to verify they fail**
Run: `uv run pytest tests/test_table.py -v -k load_filter_sort_postprocess`
Expected: FAIL with `AttributeError: module 'src.utils.table' has no attribute '_load_filter_sort_postprocess'`.
- [ ] **Step 3: Implement the helper**
Edit `src/utils/table.py`. Add this import near the top, with the other `src.` imports:
```python
from utils.cache import cache
```
Then add the helper function. Place it ABOVE `prepare_table_data` (around line 370, just before `def prepare_table_data`):
```python
@cache.memoize()
def _load_filter_sort_postprocess(filter_query, sort_by_key):
"""Memoized core of the Tableau page pipeline.
Loads the full marchés dataset, applies filter and sort, materializes,
then runs the per-row post-processing (cast to string, fill nulls, add
HTML links, format values). Returns an unpaginated Polars DataFrame.
Inputs MUST be hashable: filter_query is str|None, sort_by_key is the
tuple produced by normalize_sort_by(). Pagination intentionally lives
in the outer wrapper so that page changes are cache hits.
"""
logger.debug(f"Cache miss — recomputing for filter={filter_query!r} sort={sort_by_key!r}")
lff: pl.LazyFrame = query_marches().lazy()
if filter_query:
lff = filter_table_data(lff, filter_query, "tableau")
if sort_by_key:
sort_by = [
{"column_id": col, "direction": direction}
for col, direction in sort_by_key
]
lff = sort_table_data(lff, sort_by)
# The remaining steps are cheap per-row operations that we run ONCE here
# so that pagination in the outer function is a pure slice + to_dicts.
lff = lff.cast(pl.String)
lff = lff.fill_null("")
dff: pl.DataFrame = lff.collect()
dff = add_links(dff)
if "sourceFile" in dff.columns:
dff = add_resource_link(dff)
if dff.height > 0:
dff = format_values(dff)
return dff
```
- [ ] **Step 4: Run tests to verify they pass**
Run: `uv run pytest tests/test_table.py -v -k load_filter_sort_postprocess`
Expected: 3 PASS.
- [ ] **Step 5: Run the full test_table.py to catch regressions**
Run: `uv run pytest tests/test_table.py -v`
Expected: All PASS.
- [ ] **Step 6: Commit**
```bash
git add src/utils/table.py tests/test_table.py
git commit -m "feat: add memoized _load_filter_sort_postprocess helper"
```
---
## Task 5: Wire the memoized helper into prepare_table_data
**Files:**
- Modify: `src/utils/table.py` — replace the body of `prepare_table_data` so the Tableau path uses the cache
- Modify: `tests/test_table.py` — add tests covering the new flow
The outer function keeps its signature unchanged so callers in `acheteur.py`, `titulaire.py`, `observatoire.py`, `tableau.py` need no updates. When `data is None` (the Tableau case), use the memoized helper; otherwise fall through to the original logic.
- [ ] **Step 1: Write the failing tests**
Append to `tests/test_table.py`:
```python
def test_prepare_table_data_returns_expected_tuple(monkeypatch, sample_lff):
from src.utils import table
monkeypatch.setattr(
table, "query_marches", lambda: sample_lff.collect()
)
result = table.prepare_table_data(
data=None,
data_timestamp=5,
filter_query=None,
page_current=0,
page_size=20,
sort_by=[],
source_table="tableau",
)
# Same arity as before: 9 outputs
assert len(result) == 9
dicts, columns, tooltip, ts, nb_rows, dl_disabled, dl_text, dl_title, cleanup = result
assert isinstance(dicts, list)
assert ts == 6 # data_timestamp + 1 must still increment
assert "1 lignes" in nb_rows
def test_prepare_table_data_calls_track_search_on_filter(monkeypatch, sample_lff):
from src.utils import table
calls = []
monkeypatch.setattr(
table, "query_marches", lambda: sample_lff.collect()
)
monkeypatch.setattr(table, "track_search", lambda *a, **kw: calls.append(a))
table.prepare_table_data(
data=None,
data_timestamp=0,
filter_query="{objet} icontains travaux",
page_current=0,
page_size=20,
sort_by=[],
source_table="tableau",
)
assert calls == [("{objet} icontains travaux", "tableau")]
def test_prepare_table_data_paginates_without_recomputing(monkeypatch, sample_lff):
"""Two calls with same filter+sort but different pages must invoke
the inner heavy work only once."""
from src.utils import table
call_count = {"n": 0}
real_query = sample_lff.collect()
def counting_query():
call_count["n"] += 1
return real_query
monkeypatch.setattr(table, "query_marches", counting_query)
# First call: cache miss
table.prepare_table_data(
data=None,
data_timestamp=0,
filter_query=None,
page_current=0,
page_size=10,
sort_by=[],
source_table="tableau",
)
first_count = call_count["n"]
# Second call, different page: cache hit, query_marches must NOT fire again
table.prepare_table_data(
data=None,
data_timestamp=0,
filter_query=None,
page_current=1,
page_size=10,
sort_by=[],
source_table="tableau",
)
assert call_count["n"] == first_count, (
"query_marches was called again — pagination triggered cache miss"
)
def test_prepare_table_data_cleanup_trigger_for_non_tableau(monkeypatch, sample_lff):
"""Non-tableau pages still get a fresh uuid trigger, not no_update."""
from dash import no_update
from src.utils import table
monkeypatch.setattr(
table, "query_marches", lambda: sample_lff.collect()
)
result = table.prepare_table_data(
data=None,
data_timestamp=0,
filter_query="{objet} icontains travaux",
page_current=0,
page_size=20,
sort_by=[],
source_table="acheteur",
)
cleanup = result[8]
assert cleanup is not no_update
assert isinstance(cleanup, str)
assert len(cleanup) >= 32 # uuid4 hex string
def test_prepare_table_data_with_external_data_does_not_use_cache(
monkeypatch, sample_lff
):
"""When a caller passes data (acheteur/titulaire/observatoire path),
bypass the memoized helper entirely."""
from src.utils import table
sentinel = {"called": False}
def should_not_be_called(*a, **kw):
sentinel["called"] = True
raise AssertionError("Memoized helper must not be called when data is provided")
monkeypatch.setattr(
table, "_load_filter_sort_postprocess", should_not_be_called
)
table.prepare_table_data(
data=sample_lff, # external LazyFrame
data_timestamp=0,
filter_query=None,
page_current=0,
page_size=20,
sort_by=[],
source_table="acheteur",
)
assert sentinel["called"] is False
```
- [ ] **Step 2: Run tests to verify they fail**
Run: `uv run pytest tests/test_table.py -v -k prepare_table_data`
Expected: At least the cache-hit (`paginates_without_recomputing`) and `track_search`-routing tests FAIL because the current `prepare_table_data` re-runs the full pipeline on every call and routes tracking through `filter_table_data` (which Task 2 already neutralized — so tracking would be lost without the new explicit call).
- [ ] **Step 3: Refactor prepare_table_data**
Edit `src/utils/table.py`. Replace the entire `prepare_table_data` function body with:
```python
def prepare_table_data(
data, data_timestamp, filter_query, page_current, page_size, sort_by, source_table
):
"""
Préparation des données pour les datatables.
Pour la page Tableau (data is None), le calcul lourd (chargement complet,
filtre, tri, post-traitement) est mémorisé via _load_filter_sort_postprocess.
Les changements de page deviennent ainsi des cache hits.
Pour les autres pages (data fourni), le chemin original est conservé : la
LazyFrame externe n'est pas hashable et le coût de filtre/tri y est déjà
minime puisque les données sont pré-restreintes.
"""
logger.debug(" + + + + + + + + + + + + + + + + + + ")
# Side effect non-cacheable : le tracking doit firer sur chaque action
# utilisateur, y compris sur cache hit.
if filter_query:
track_search(filter_query, source_table)
# Trigger uuid pour les pages autres que tableau (clientside cleanup)
trigger_cleanup = (
no_update if source_table == "tableau" else str(uuid.uuid4())
)
if data is None:
# Tableau path : utilise le cache
sort_by_key = normalize_sort_by(sort_by)
dff: pl.DataFrame = _load_filter_sort_postprocess(
filter_query=filter_query, sort_by_key=sort_by_key
)
else:
# acheteur / titulaire / observatoire path : code original, non caché
if isinstance(data, list):
lff: pl.LazyFrame = pl.LazyFrame(
data, strict=False, infer_schema_length=5000
)
elif isinstance(data, pl.LazyFrame):
lff = data
else:
lff = query_marches().lazy()
if filter_query:
lff = filter_table_data(lff, filter_query, source_table)
if sort_by and len(sort_by) > 0:
lff = sort_table_data(lff, sort_by)
dff = lff.collect()
dff = dff.cast(pl.String)
dff = dff.fill_null("")
dff = add_links(dff)
if "sourceFile" in dff.columns:
dff = add_resource_link(dff)
if dff.height > 0:
dff = format_values(dff)
height = dff.height
if height > 0:
nb_rows = (
f"{format_number(height)} lignes "
f"({format_number(dff.select('uid').unique().height)} marchés)"
)
else:
nb_rows = "0 lignes (0 marchés)"
# Pagination — toujours hors cache pour rester sur des cache hits
start_row = page_current * page_size
dff = dff.slice(start_row, page_size)
table_columns, tooltip = setup_table_columns(dff)
dicts = dff.to_dicts()
download_disabled, download_text, download_title = get_button_properties(height)
return (
dicts,
table_columns,
tooltip,
data_timestamp + 1,
nb_rows,
download_disabled,
download_text,
download_title,
trigger_cleanup,
)
```
Notes on what changed vs the original at `src/utils/table.py:372-458`:
- `track_search` now called explicitly at the top, on every invocation (not via `filter_table_data`).
- `data is None` branch delegates the heavy work to the memoized helper.
- `data is not None` branch is functionally identical to the original (pagination still happens after collect+post-process).
- The post-processing (`cast`, `fill_null`, `add_links`, `add_resource_link`, `format_values`) is now done in BOTH branches before `nb_rows` calculation. In the cached branch this was already done inside `_load_filter_sort_postprocess`; in the uncached branch we keep doing it inline. This means `nb_rows` and `dff.select('uid').unique().height` operate on the post-processed frame in both branches, matching the original semantics.
- [ ] **Step 4: Run all unit tests**
Run: `uv run pytest tests/test_table.py -v`
Expected: All PASS, including `test_prepare_table_data_paginates_without_recomputing`.
- [ ] **Step 5: Run the full repo test suite to catch regressions**
Run: `uv run pytest -v`
Expected: All PASS. Selenium tests (`tests/test_main.py`) require Chrome/Chromium; if the executor lacks a browser, those tests will error/skip — note the failures and rerun in an environment with Chrome before declaring done.
- [ ] **Step 6: Commit**
```bash
git add src/utils/table.py tests/test_table.py
git commit -m "perf(tableau): memoize filter+sort+postprocess pipeline"
```
---
## Task 6: Manual smoke test in the browser
**Files:** none modified.
Type checks and unit tests cannot validate that page navigation actually feels faster. This task is explicitly a hands-on verification.
- [ ] **Step 1: Start the dev server**
Run: `uv run run.py`
Wait for `Dash is running on http://...`.
- [ ] **Step 2: Open the Tableau page and warm the cache**
1. Open `http://localhost:8050/tableau` (or whatever port the dev server prints).
2. With no filter applied, wait for the first page to load fully. This is the cold-cache load (slow expected).
3. Open the browser devtools Network panel.
- [ ] **Step 3: Verify pagination is fast**
1. Click "page 2" / "page 3" / "page 4" in the table footer in quick succession.
2. Each navigation should return data in well under 1 second (in the original code each took several seconds).
3. In the dev server logs, look for the line `Cache miss — recomputing for filter=...` from `_load_filter_sort_postprocess`. It should appear ONCE for the initial load and NOT appear again as you change pages.
- [ ] **Step 4: Verify a new filter triggers exactly one cache miss**
1. In the table, type a filter into one of the columns (e.g. `paris` in `acheteur_commune_nom`) and press Enter.
2. The dev log should show ONE new `Cache miss — recomputing` line.
3. Change page within the filtered view — no new cache miss line should appear.
- [ ] **Step 5: Verify filter cleanup trigger still fires**
1. Open `http://localhost:8050/acheteur?id=<some_acheteur_id>` (use any valid id from the dataset).
2. Apply a filter on the embedded table.
3. The clientside callback for filter cleanup (`src/assets/dash_clientside.js` `clean_filters`) should still rewrite the filter operators (e.g. `contains``icontains`). If it doesn't fire, the `trigger_cleanup` uuid is broken — investigate.
- [ ] **Step 6: Verify download still works**
1. On the Tableau page, click "Télécharger au format Excel" (the button must be enabled — apply a filter that brings the row count under 65,000).
2. The downloaded XLSX must open and contain the filtered rows.
- [ ] **Step 7: Stop the dev server**
Ctrl-C.
- [ ] **Step 8: If all checks pass, this completes the implementation**
No commit — this task is verification only. Report results to the user.
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,951 @@
# Observatoire — filtrage natif DuckDB — Plan d'implémentation
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Remplacer le filtrage Polars sur LazyFrame dans `prepare_dashboard_data` par un requêtage natif DuckDB, pour ne matérialiser que le sous-ensemble utile au lieu de l'intégralité de la table `decp` (~1,5 M lignes).
**Architecture:** Nouveau helper pur `dashboard_filters_to_sql(**filter_params) -> (where_sql, params)` dans `src/utils/table_sql.py` (modèle de `filter_query_to_sql`). `prepare_dashboard_data` devient une fonction fine qui appelle `query_marches(where_sql, params)` et retourne une `pl.DataFrame`. Les 3 appelants dans `src/pages/observatoire.py` sont adaptés à la nouvelle signature.
**Tech Stack:** Python 3.12, Polars, DuckDB, Dash, pytest.
**Spec:** `docs/superpowers/specs/2026-04-22-observatoire-duckdb-filters-design.md`.
---
## File Structure
**À créer :**
- `tests/test_dashboard_filters_to_sql.py` — tests unitaires du nouveau helper SQL (cas vide + cas par filtre).
- `tests/test_prepare_dashboard_data.py` — test d'intégration léger (appel DuckDB réel sur `tests/test.parquet`).
**À modifier :**
- `src/utils/table_sql.py` — ajouter `dashboard_filters_to_sql` + import `datetime`/`timedelta`.
- `src/utils/data.py` — réécrire `prepare_dashboard_data` (signature et implémentation), ajouter `query_marches` aux imports `from src.db`.
- `src/pages/observatoire.py` — adapter 3 sites d'appel (lignes ~668, ~791, ~882) ; retirer `query_marches` de l'import `from src.db` (plus utilisé).
- `tests/test_main.py` — supprimer `test_010_observatoire_montant_filter` (migré en test unitaire du helper).
---
## Task 1: Tests unitaires — cas par défaut + filtre année
**Files:**
- Create: `tests/test_dashboard_filters_to_sql.py`
- Modify: `src/utils/table_sql.py`
- [ ] **Step 1: Write the failing tests**
Create `tests/test_dashboard_filters_to_sql.py`:
```python
from datetime import datetime, timedelta
from src.utils.table_sql import dashboard_filters_to_sql
def test_no_filters_uses_default_365_day_window():
where_sql, params = dashboard_filters_to_sql()
assert where_sql == '"dateNotification" > ?'
assert len(params) == 1
assert isinstance(params[0], datetime)
expected = datetime.now() - timedelta(days=365)
assert abs((params[0] - expected).total_seconds()) < 2
def test_year_filter_overrides_default_window():
where_sql, params = dashboard_filters_to_sql(dashboard_year="2025")
assert where_sql == 'YEAR("dateNotification") = ?'
assert params == [2025]
```
- [ ] **Step 2: Run tests to verify they fail**
Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
Expected: FAIL with `ImportError: cannot import name 'dashboard_filters_to_sql'`.
- [ ] **Step 3: Implement the helper**
Add to the top of `src/utils/table_sql.py` (below existing imports):
```python
from datetime import datetime, timedelta
```
Append this function at the end of `src/utils/table_sql.py`:
```python
def dashboard_filters_to_sql(
dashboard_year=None,
dashboard_acheteur_id=None,
dashboard_acheteur_categorie=None,
dashboard_acheteur_departement_code=None,
dashboard_titulaire_id=None,
dashboard_titulaire_categorie=None,
dashboard_titulaire_departement_code=None,
dashboard_marche_type=None,
dashboard_marche_objet=None,
dashboard_marche_code_cpv=None,
dashboard_marche_considerations_sociales=None,
dashboard_marche_considerations_environnementales=None,
dashboard_marche_techniques=None,
dashboard_marche_innovant=None,
dashboard_marche_sous_traitance_declaree=None,
dashboard_montant_min=None,
dashboard_montant_max=None,
) -> tuple[str, list]:
"""Traduit les filtres du tableau de bord en (where_clause, params) DuckDB."""
clauses: list[str] = []
params: list = []
if dashboard_year:
clauses.append('YEAR("dateNotification") = ?')
params.append(int(dashboard_year))
else:
clauses.append('"dateNotification" > ?')
params.append(datetime.now() - timedelta(days=365))
return " AND ".join(clauses), params
```
- [ ] **Step 4: Run tests to verify they pass**
Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
Expected: PASS (2 tests).
- [ ] **Step 5: Commit**
```bash
rtk pre-commit run --files tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
rtk git add tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
rtk git commit -m "feat(observatoire): squelette de dashboard_filters_to_sql (#72)"
```
---
## Task 2: Filtres d'égalité simples (catégorie, type, innovant, sous-traitance)
**Files:**
- Modify: `tests/test_dashboard_filters_to_sql.py`
- Modify: `src/utils/table_sql.py`
- [ ] **Step 1: Add failing tests**
Append to `tests/test_dashboard_filters_to_sql.py`:
```python
def test_marche_type_equality():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_marche_type="Marché",
)
assert where_sql == 'YEAR("dateNotification") = ? AND "type" = ?'
assert params == [2025, "Marché"]
def test_innovant_value_all_is_skipped():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_marche_innovant="all",
)
assert where_sql == 'YEAR("dateNotification") = ?'
assert params == [2025]
def test_innovant_value_oui_adds_clause():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_marche_innovant="oui",
)
assert where_sql == 'YEAR("dateNotification") = ? AND "marcheInnovant" = ?'
assert params == [2025, "oui"]
def test_sous_traitance_value_non_adds_clause():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_marche_sous_traitance_declaree="non",
)
assert (
where_sql
== 'YEAR("dateNotification") = ? AND "sousTraitanceDeclaree" = ?'
)
assert params == [2025, "non"]
```
- [ ] **Step 2: Run tests to verify they fail**
Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
Expected: 4 new tests FAIL (missing clauses).
- [ ] **Step 3: Extend the helper**
Insert the following block in `dashboard_filters_to_sql`, **after** the `if dashboard_year / else` block and **before** `return " AND ".join(clauses), params`:
```python
if dashboard_marche_type:
clauses.append('"type" = ?')
params.append(dashboard_marche_type)
if dashboard_marche_innovant and dashboard_marche_innovant != "all":
clauses.append('"marcheInnovant" = ?')
params.append(dashboard_marche_innovant)
if (
dashboard_marche_sous_traitance_declaree
and dashboard_marche_sous_traitance_declaree != "all"
):
clauses.append('"sousTraitanceDeclaree" = ?')
params.append(dashboard_marche_sous_traitance_declaree)
```
- [ ] **Step 4: Run tests to verify they pass**
Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
Expected: PASS (6 tests total).
- [ ] **Step 5: Commit**
```bash
rtk pre-commit run --files tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
rtk git add tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
rtk git commit -m "feat(observatoire): filtres d'égalité simples dans dashboard_filters_to_sql (#72)"
```
---
## Task 3: Filtres LIKE/ILIKE (ids, objet, cpv)
**Files:**
- Modify: `tests/test_dashboard_filters_to_sql.py`
- Modify: `src/utils/table_sql.py`
- [ ] **Step 1: Add failing tests**
Append to `tests/test_dashboard_filters_to_sql.py`:
```python
def test_acheteur_id_uses_like_wildcards():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_acheteur_id="12345678900010",
)
assert where_sql == 'YEAR("dateNotification") = ? AND "acheteur_id" LIKE ?'
assert params == [2025, "%12345678900010%"]
def test_titulaire_id_uses_like_wildcards():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_titulaire_id="999",
)
assert where_sql == 'YEAR("dateNotification") = ? AND "titulaire_id" LIKE ?'
assert params == [2025, "%999%"]
def test_marche_objet_uses_case_insensitive_ilike():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_marche_objet="travaux",
)
assert where_sql == 'YEAR("dateNotification") = ? AND "objet" ILIKE ?'
assert params == [2025, "%travaux%"]
def test_code_cpv_uses_prefix_like():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_marche_code_cpv="4521",
)
assert where_sql == 'YEAR("dateNotification") = ? AND "codeCPV" LIKE ?'
assert params == [2025, "4521%"]
```
- [ ] **Step 2: Run tests to verify they fail**
Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
Expected: 4 new tests FAIL.
- [ ] **Step 3: Extend the helper**
Insert the following block, **just after** the year/default block and **before** the `if dashboard_marche_type` block:
```python
if dashboard_acheteur_id:
clauses.append('"acheteur_id" LIKE ?')
params.append(f"%{dashboard_acheteur_id}%")
if dashboard_titulaire_id:
clauses.append('"titulaire_id" LIKE ?')
params.append(f"%{dashboard_titulaire_id}%")
```
Insert in the "marché" block, **after** `dashboard_marche_type` and **before** `dashboard_marche_innovant`:
```python
if dashboard_marche_objet:
clauses.append('"objet" ILIKE ?')
params.append(f"%{dashboard_marche_objet}%")
if dashboard_marche_code_cpv:
clauses.append('"codeCPV" LIKE ?')
params.append(f"{dashboard_marche_code_cpv}%")
```
- [ ] **Step 4: Run tests to verify they pass**
Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
Expected: PASS (10 tests total).
- [ ] **Step 5: Commit**
```bash
rtk pre-commit run --files tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
rtk git add tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
rtk git commit -m "feat(observatoire): filtres LIKE/ILIKE dans dashboard_filters_to_sql (#72)"
```
---
## Task 4: Filtre IN (départements) + skip conditionnel par ID
**Files:**
- Modify: `tests/test_dashboard_filters_to_sql.py`
- Modify: `src/utils/table_sql.py`
- [ ] **Step 1: Add failing tests**
Append to `tests/test_dashboard_filters_to_sql.py`:
```python
def test_acheteur_departement_multiple_uses_in_clause():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_acheteur_departement_code=["75", "92", "93"],
)
assert where_sql == (
'YEAR("dateNotification") = ? '
'AND "acheteur_departement_code" IN (?, ?, ?)'
)
assert params == [2025, "75", "92", "93"]
def test_acheteur_categorie_adds_clause():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_acheteur_categorie="Commune",
)
assert where_sql == 'YEAR("dateNotification") = ? AND "acheteur_categorie" = ?'
assert params == [2025, "Commune"]
def test_titulaire_categorie_and_departement():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_titulaire_categorie="PME",
dashboard_titulaire_departement_code=["35"],
)
assert where_sql == (
'YEAR("dateNotification") = ? '
'AND "titulaire_categorie" = ? '
'AND "titulaire_departement_code" IN (?)'
)
assert params == [2025, "PME", "35"]
def test_acheteur_id_present_skips_categorie_and_departement():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_acheteur_id="123",
dashboard_acheteur_categorie="Commune",
dashboard_acheteur_departement_code=["75"],
)
assert where_sql == 'YEAR("dateNotification") = ? AND "acheteur_id" LIKE ?'
assert params == [2025, "%123%"]
def test_titulaire_id_present_skips_categorie_and_departement():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_titulaire_id="999",
dashboard_titulaire_categorie="PME",
dashboard_titulaire_departement_code=["35"],
)
assert where_sql == 'YEAR("dateNotification") = ? AND "titulaire_id" LIKE ?'
assert params == [2025, "%999%"]
```
- [ ] **Step 2: Run tests to verify they fail**
Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
Expected: 5 new tests FAIL.
- [ ] **Step 3: Refactor the helper with conditional skip**
Replace the two simple `if dashboard_acheteur_id` / `if dashboard_titulaire_id` blocks added in Task 3 with the nested form:
```python
if dashboard_acheteur_id:
clauses.append('"acheteur_id" LIKE ?')
params.append(f"%{dashboard_acheteur_id}%")
else:
if dashboard_acheteur_categorie:
clauses.append('"acheteur_categorie" = ?')
params.append(dashboard_acheteur_categorie)
if dashboard_acheteur_departement_code:
placeholders = ", ".join(["?"] * len(dashboard_acheteur_departement_code))
clauses.append(f'"acheteur_departement_code" IN ({placeholders})')
params.extend(dashboard_acheteur_departement_code)
if dashboard_titulaire_id:
clauses.append('"titulaire_id" LIKE ?')
params.append(f"%{dashboard_titulaire_id}%")
else:
if dashboard_titulaire_categorie:
clauses.append('"titulaire_categorie" = ?')
params.append(dashboard_titulaire_categorie)
if dashboard_titulaire_departement_code:
placeholders = ", ".join(
["?"] * len(dashboard_titulaire_departement_code)
)
clauses.append(f'"titulaire_departement_code" IN ({placeholders})')
params.extend(dashboard_titulaire_departement_code)
```
- [ ] **Step 4: Run tests to verify they pass**
Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
Expected: PASS (15 tests total).
- [ ] **Step 5: Commit**
```bash
rtk pre-commit run --files tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
rtk git add tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
rtk git commit -m "feat(observatoire): IN départements et skip conditionnel par ID (#72)"
```
---
## Task 5: Filtre liste (techniques, considérations sociales/environnementales)
**Files:**
- Modify: `tests/test_dashboard_filters_to_sql.py`
- Modify: `src/utils/table_sql.py`
- [ ] **Step 1: Add failing tests**
Append to `tests/test_dashboard_filters_to_sql.py`:
```python
def test_marche_techniques_uses_list_has_any():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_marche_techniques=["Enchère", "Accord-cadre"],
)
assert where_sql == (
'YEAR("dateNotification") = ? '
"AND list_has_any(string_split(\"techniques\", ', '), ?::VARCHAR[])"
)
assert params == [2025, ["Enchère", "Accord-cadre"]]
def test_considerations_sociales_uses_list_has_any():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_marche_considerations_sociales=["Clause sociale"],
)
assert where_sql == (
'YEAR("dateNotification") = ? '
"AND list_has_any(string_split(\"considerationsSociales\", ', '), ?::VARCHAR[])"
)
assert params == [2025, ["Clause sociale"]]
def test_considerations_environnementales_uses_list_has_any():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_marche_considerations_environnementales=["Clause env."],
)
assert where_sql == (
'YEAR("dateNotification") = ? '
"AND list_has_any(string_split(\"considerationsEnvironnementales\", ', '), ?::VARCHAR[])"
)
assert params == [2025, ["Clause env."]]
```
- [ ] **Step 2: Run tests to verify they fail**
Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
Expected: 3 new tests FAIL.
- [ ] **Step 3: Extend the helper**
Insert the following block in `dashboard_filters_to_sql`, **after** the `dashboard_marche_sous_traitance_declaree` block and **before** `return " AND ".join(clauses), params`:
```python
if dashboard_marche_techniques:
clauses.append(
"list_has_any(string_split(\"techniques\", ', '), ?::VARCHAR[])"
)
params.append(list(dashboard_marche_techniques))
if dashboard_marche_considerations_sociales:
clauses.append(
"list_has_any(string_split(\"considerationsSociales\", ', '), ?::VARCHAR[])"
)
params.append(list(dashboard_marche_considerations_sociales))
if dashboard_marche_considerations_environnementales:
clauses.append(
"list_has_any(string_split(\"considerationsEnvironnementales\", ', '), ?::VARCHAR[])"
)
params.append(list(dashboard_marche_considerations_environnementales))
```
- [ ] **Step 4: Run tests to verify they pass**
Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
Expected: PASS (18 tests total).
- [ ] **Step 5: Commit**
```bash
rtk pre-commit run --files tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
rtk git add tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
rtk git commit -m "feat(observatoire): filtres liste via list_has_any (#72)"
```
---
## Task 6: Filtres montant min/max (incluant 0)
**Files:**
- Modify: `tests/test_dashboard_filters_to_sql.py`
- Modify: `src/utils/table_sql.py`
- [ ] **Step 1: Add failing tests**
Append to `tests/test_dashboard_filters_to_sql.py`:
```python
def test_montant_min_only():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_montant_min=1000,
)
assert where_sql == 'YEAR("dateNotification") = ? AND "montant" >= ?'
assert params == [2025, 1000]
def test_montant_max_only():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_montant_max=500,
)
assert where_sql == 'YEAR("dateNotification") = ? AND "montant" <= ?'
assert params == [2025, 500]
def test_montant_zero_is_a_valid_lower_bound():
# 0 est falsy mais reste un filtre valide (distinct de None)
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_montant_min=0,
)
assert where_sql == 'YEAR("dateNotification") = ? AND "montant" >= ?'
assert params == [2025, 0]
def test_montant_min_and_max_combined():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_montant_min=100,
dashboard_montant_max=1000,
)
assert where_sql == (
'YEAR("dateNotification") = ? AND "montant" >= ? AND "montant" <= ?'
)
assert params == [2025, 100, 1000]
```
- [ ] **Step 2: Run tests to verify they fail**
Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
Expected: 4 new tests FAIL.
- [ ] **Step 3: Extend the helper**
Insert at the very end of `dashboard_filters_to_sql`, **just before** `return " AND ".join(clauses), params`:
```python
if dashboard_montant_min is not None:
clauses.append('"montant" >= ?')
params.append(dashboard_montant_min)
if dashboard_montant_max is not None:
clauses.append('"montant" <= ?')
params.append(dashboard_montant_max)
```
- [ ] **Step 4: Run tests to verify they pass**
Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
Expected: PASS (22 tests total).
- [ ] **Step 5: Commit**
```bash
rtk pre-commit run --files tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
rtk git add tests/test_dashboard_filters_to_sql.py src/utils/table_sql.py
rtk git commit -m "feat(observatoire): filtres montant min/max (#72)"
```
---
## Task 7: Réécriture de `prepare_dashboard_data`
**Files:**
- Modify: `src/utils/data.py`
- Modify: `tests/test_main.py` (supprimer `test_010_observatoire_montant_filter`)
- [ ] **Step 1: Remove the obsolete Polars-based test**
Delete the function `test_010_observatoire_montant_filter` from `tests/test_main.py` (lines ~218-256). La couverture du filtre montant est déjà assurée par les tests unitaires `test_montant_*` de la Task 6.
- [ ] **Step 2: Rewrite `prepare_dashboard_data`**
Replace the entire `prepare_dashboard_data` function in `src/utils/data.py` (lines ~86-194) with:
```python
def prepare_dashboard_data(**filter_params) -> pl.DataFrame:
"""Exécute la requête DuckDB filtrée pour le tableau de bord.
Retourne une pl.DataFrame matérialisée uniquement pour le sous-ensemble
correspondant aux filtres. Les appelants qui ont besoin d'une LazyFrame
appellent `.lazy()` sur le résultat.
"""
from src.utils.table_sql import dashboard_filters_to_sql
where_sql, params = dashboard_filters_to_sql(**filter_params)
return query_marches(where_sql=where_sql, params=params)
```
Update the import at the top of `src/utils/data.py`:
```python
from src.db import get_cursor, query_marches, schema
```
Remove the now-unused import in `src/utils/data.py`:
```python
from datetime import datetime, timedelta
```
(Si `datetime` n'est plus référencé dans `data.py` hors de `prepare_dashboard_data`, sinon garder.)
**Vérification rapide à effectuer avant de supprimer `datetime`/`timedelta`** :
```bash
rtk grep -n "datetime\|timedelta" src/utils/data.py
```
Si d'autres occurrences existent, conserver les imports.
- [ ] **Step 3: Run the full test suite**
Run: `rtk pytest tests/test_dashboard_filters_to_sql.py tests/test_main.py -v -k "not selenium and not dash_duo"`
Ou, si filter n'est pas pratique :
Run: `rtk pytest tests/test_dashboard_filters_to_sql.py -v`
Expected: PASS (22 tests).
- [ ] **Step 4: Commit**
```bash
rtk pre-commit run --files src/utils/data.py tests/test_main.py
rtk git add src/utils/data.py tests/test_main.py
rtk git commit -m "refactor(observatoire): prepare_dashboard_data utilise DuckDB (#72)"
```
---
## Task 8: Adaptation des 3 appelants dans `observatoire.py`
**Files:**
- Modify: `src/pages/observatoire.py`
- [ ] **Step 1: Update `_compute_dashboard_children`**
Remplacer dans `src/pages/observatoire.py` (autour des lignes 660-670) :
```python
@cache.memoize()
def _compute_dashboard_children(filter_params_normalized: tuple):
logger.debug("Cache miss — computing dashboard")
filter_params = {
k: (list(v) if isinstance(v, tuple) else v) for k, v in filter_params_normalized
}
lff: pl.LazyFrame = query_marches().lazy()
lff = prepare_dashboard_data(lff=lff, **filter_params)
dff = lff.collect(engine="streaming")
```
Par :
```python
@cache.memoize()
def _compute_dashboard_children(filter_params_normalized: tuple):
logger.debug("Cache miss — computing dashboard")
filter_params = {
k: (list(v) if isinstance(v, tuple) else v) for k, v in filter_params_normalized
}
dff = prepare_dashboard_data(**filter_params)
lff = dff.lazy()
```
Le reste de la fonction (à partir de `df_per_uid = ...`) est inchangé.
- [ ] **Step 2: Update `download_observatoire`**
Remplacer dans `src/pages/observatoire.py` (autour des lignes 789-800) :
```python
def download_observatoire(_n_clicks, filter_params, hidden_columns):
lff = prepare_dashboard_data(lff=query_marches().lazy(), **(filter_params or {}))
if hidden_columns:
lff = lff.drop(hidden_columns)
def to_bytes(buffer):
lff.collect(engine="streaming").write_excel(buffer, worksheet="DECP")
date = datetime.now().strftime("%Y-%m-%d_%H:%M:%S")
return dcc.send_bytes(to_bytes, filename=f"decp_observatoire_{date}.xlsx")
```
Par :
```python
def download_observatoire(_n_clicks, filter_params, hidden_columns):
dff = prepare_dashboard_data(**(filter_params or {}))
if hidden_columns:
dff = dff.drop(hidden_columns)
def to_bytes(buffer):
dff.write_excel(buffer, worksheet="DECP")
date = datetime.now().strftime("%Y-%m-%d_%H:%M:%S")
return dcc.send_bytes(to_bytes, filename=f"decp_observatoire_{date}.xlsx")
```
- [ ] **Step 3: Update `populate_preview_table`**
Remplacer dans `src/pages/observatoire.py` (autour des lignes 879-892) :
```python
if not is_open:
return (no_update,) * 9
lff = prepare_dashboard_data(lff=query_marches().lazy(), **(filter_params or {}))
return prepare_table_data(
lff,
data_timestamp,
filter_query,
page_current,
page_size,
sort_by,
"observatoire-preview",
)
```
Par :
```python
if not is_open:
return (no_update,) * 9
dff = prepare_dashboard_data(**(filter_params or {}))
return prepare_table_data(
dff.lazy(),
data_timestamp,
filter_query,
page_current,
page_size,
sort_by,
"observatoire-preview",
)
```
- [ ] **Step 4: Remove unused `query_marches` import**
Dans `src/pages/observatoire.py`, ligne ~19 :
```python
from src.db import query_marches, schema
```
Devient :
```python
from src.db import schema
```
Vérifier avant de committer :
```bash
rtk grep -n "query_marches" src/pages/observatoire.py
```
Expected: aucun résultat (ou uniquement des commentaires).
- [ ] **Step 5: Smoke test**
Démarrer l'app et naviguer sur `/observatoire`, vérifier à la main que :
- Les cartes s'affichent.
- Un filtre année se propage.
- Un filtre acheteur par SIRET partiel fonctionne.
- Un filtre département (multi-valeur) fonctionne.
- Un filtre montant_min fonctionne.
- Le bouton « Télécharger au format Excel » génère un fichier non vide.
- Le bouton « Voir les données » ouvre l'offcanvas et peuple la table.
Run: `python run.py`
Expected: app démarre sans erreur ; les filtres se comportent comme avant.
- [ ] **Step 6: Commit**
```bash
rtk pre-commit run --files src/pages/observatoire.py
rtk git add src/pages/observatoire.py
rtk git commit -m "refactor(observatoire): appelants utilisent la nouvelle signature (#72)"
```
---
## Task 9: Test d'intégration — `prepare_dashboard_data` sur `tests/test.parquet`
**Files:**
- Create: `tests/test_prepare_dashboard_data.py`
- [ ] **Step 1: Write the failing test**
Le but : vérifier que la fonction s'exécute réellement contre DuckDB, retourne une `pl.DataFrame`, et applique bien les filtres simples. `conftest.py` construit `tests/test.parquet` avec un jeu de données d'une ligne : acheteur_id `123`, acheteur_departement_code `75`, dateNotification `2025-01-01`, montant `10`.
Create `tests/test_prepare_dashboard_data.py`:
```python
import polars as pl
def test_returns_dataframe_with_year_filter():
from src.utils.data import prepare_dashboard_data
dff = prepare_dashboard_data(dashboard_year="2025")
assert isinstance(dff, pl.DataFrame)
assert dff.height == 1
def test_year_mismatch_returns_empty():
from src.utils.data import prepare_dashboard_data
dff = prepare_dashboard_data(dashboard_year="2024")
assert isinstance(dff, pl.DataFrame)
assert dff.height == 0
def test_acheteur_id_partial_match():
from src.utils.data import prepare_dashboard_data
dff = prepare_dashboard_data(
dashboard_year="2025",
dashboard_acheteur_id="12",
)
assert dff.height == 1
def test_departement_in_clause():
from src.utils.data import prepare_dashboard_data
dff = prepare_dashboard_data(
dashboard_year="2025",
dashboard_acheteur_departement_code=["75", "92"],
)
assert dff.height == 1
def test_montant_min_above_value_excludes_row():
from src.utils.data import prepare_dashboard_data
dff = prepare_dashboard_data(
dashboard_year="2025",
dashboard_montant_min=1000,
)
assert dff.height == 0
```
- [ ] **Step 2: Run the test**
Run: `rtk pytest tests/test_prepare_dashboard_data.py -v`
Expected: PASS (5 tests).
- [ ] **Step 3: Commit**
```bash
rtk pre-commit run --files tests/test_prepare_dashboard_data.py
rtk git add tests/test_prepare_dashboard_data.py
rtk git commit -m "test(observatoire): intégration DuckDB pour prepare_dashboard_data (#72)"
```
---
## Task 10: Vérification finale
**Files:** (aucune modification)
- [ ] **Step 1: Run the full test suite**
Run: `rtk pytest -v`
Expected: tous les tests unitaires passent. Les tests Selenium peuvent échouer si Chrome n'est pas disponible — ce n'est pas bloquant s'ils étaient déjà rouges avant.
- [ ] **Step 2: Check for leftover references**
Run: `rtk grep -rn "prepare_dashboard_data(lff" src/ tests/`
Expected: aucun résultat (plus d'appels avec l'ancienne signature).
Run: `rtk grep -rn "query_marches().lazy()" src/`
Expected: aucun résultat (ou uniquement dans `src/utils/table.py:prepare_table_data` pour le fallback).
- [ ] **Step 3: Confirm `datetime`/`timedelta` in data.py if needed**
Run: `rtk grep -n "datetime\|timedelta" src/utils/data.py`
Si aucune occurrence hors imports, vérifier que les imports inutiles ont bien été retirés dans Task 7.
- [ ] **Step 4: Manual timing sanity check (optionnel)**
Si possible, comparer informellement le temps de `_compute_dashboard_children` sur un filtre sélectif (ex. un département) avant/après. Pas de benchmark formel attendu.
- [ ] **Step 5: Push (manuel, à l'initiative de l'utilisateur)**
Conformément aux consignes projet, ne jamais `git push`. Laisser l'utilisateur pousser la branche `feature/72_observatoire_duckdb_filters` et ouvrir la PR.
@@ -0,0 +1,108 @@
# Plan: Ajouter des cartes de localisation aux pages acheteur et titulaire
## Date: 2026-04-28
## Statut: Approuvé
## Objectif: Ajouter des cartes interactives montrant la localisation des organisations sur les pages acheteur et titulaire
## Contexte
- Les pages acheteur et titulaire ont déjà des placeholders pour les cartes (`acheteur_map` et `titulaire_map`)
- La fonction `point_on_map()` existe déjà dans `src/figures.py` mais utilise un centrage fixe sur la France
- Les données de localisation proviennent de l'API Annuaire des Entreprises
- Les codes départementaux sont disponibles et plus fiables que les coordonnées pour la détection de région
## Exigences
### 1. Carte interactive
- **Localisation**: Colonne de droite dans la section d'informations sur l'organisation
- **Taille**: 400px de largeur × 300px de hauteur (fixe)
- **Contenu**: Carte centrée sur la France ou le département d'outre-mer approprié avec un point rouge à l'emplacement de l'organisation
- **Niveau de zoom**: Approprié pour montrer l'Hexagone ou le département d'outre-mer spécifique
- **Style**: Fond de carte clair avec point rouge visible
- **Interactivité**: Carte zoomable et déplaçable (pas de configuration statique)
### 2. Sources de données
- Utiliser les colonnes `acheteur_latitude` et `acheteur_longitude` pour les pages acheteur
- Utiliser les colonnes `titulaire_latitude` et `titulaire_longitude` pour les pages titulaire
- Utiliser les codes départementaux (`acheteur_departement_code`, `titulaire_departement_code`) pour la détection de région
- Solution de repli: Si les coordonnées ou codes départementaux sont manquants ou invalides, afficher une div vide
### 3. Détection de région
- **Départements métropolitains**: Codes à 2 caractères (ex: "75" pour Paris) → Carte Hexagone
- **Départements d'outre-mer**:
- "971" → Guadeloupe
- "972" → Martinique
- "973" → Guyane
- "974" → La Réunion
- "976" → Mayotte
- **Code département manquant**: Retourner une div vide (pas de détection basée sur les coordonnées)
### 4. Gestion des erreurs
- Coordonnées invalides → div vide
- Code département manquant → div vide
- Échec de l'API Annuaire → div vide (comportement existant)
- Format de code département invalide → div vide
## Implémentation
### Fichiers à modifier
#### 1. `src/figures.py` - Améliorer la fonction `point_on_map()`
**Ligne 178-209**: Remplacer la fonction existante par une version améliorée avec:
- Détection de région basée sur les codes départementaux
- Configuration de carte interactive (zoomable)
- Point plus grand (size=15)
- Commentaires en français
#### 2. `src/pages/acheteur.py` - Mettre à jour le callback
**Ligne 249-297**: Modifier `update_acheteur_infos()` pour:
- Extraire le code département du code postal
- Passer le code département à `point_on_map()`
- Ajouter des commentaires en français
#### 3. `src/pages/titulaire.py` - Mettre à jour le callback
**Ligne 259-297**: Modifier `update_titulaire_infos()` pour:
- Extraire le code département du code postal
- Passer le code département à `point_on_map()`
- Ajouter des commentaires en français
## Plan de Test
### Cas de test prioritaires
1. **Organisation métropolitaine**: Code département "75" (Paris) → Carte Hexagone
2. **Organisation à La Réunion**: Code département "974" → Carte centrée sur La Réunion
3. **Code département manquant**: Retourne une div vide
4. **Coordonnées invalides**: Retourne une div vide
5. **Interactivité**: Vérifier zoom et déplacement
### Critères d'acceptation
- [ ] Cartes fonctionnelles avec codes départementaux valides
- [ ] Div vide pour codes manquants/invalides
- [ ] Cartes correctement centrées et zoomées
- [ ] Interactivité (zoom et déplacement)
- [ ] Point de localisation visible (size=15)
## Approbation
Plan approuvé avec spécifications:
- Réutiliser et améliorer `point_on_map`
- Retourner div vide sans code département
- Point légèrement plus grand
- Cartes zoomables
- Utiliser codes départementaux pour détection de région
- Commentaires en français
@@ -0,0 +1,206 @@
# Observatoire — filtrage natif DuckDB
## Contexte
La page `/observatoire` construit ses cartes, ses téléchargements et sa prévisualisation
tabulaire à partir de la fonction `prepare_dashboard_data` (dans `src/utils/data.py`).
Aujourd'hui, cette fonction prend une `pl.LazyFrame` — typiquement obtenue par
`query_marches().lazy()` — et applique une série de filtres côté Polars.
`query_marches()` matérialise l'intégralité de la table `decp` (~1,5 M lignes) en
DataFrame Polars, même lorsqu'un utilisateur applique des filtres restrictifs. Les
filtres sont ensuite appliqués sur cet ensemble déjà matérialisé.
Le pattern utilisé par `_fetch_page_sql` (dans `src/utils/table.py`) montre comment
déléguer le filtrage à DuckDB :
1. Un traducteur (`filter_query_to_sql`, dans `src/utils/table_sql.py`) transforme le
DSL utilisateur en `(where_sql, params)`.
2. `query_marches(where_sql=..., params=...)` ne matérialise que le sous-ensemble utile.
Ce spec décrit comment appliquer ce même pattern aux filtres de l'observatoire.
## Objectifs
- Réduire la consommation mémoire et le temps de chaque callback de l'observatoire
en poussant le filtrage au niveau DuckDB.
- Conserver strictement la sémantique des filtres actuels (pas de régression
fonctionnelle).
- Garder une frontière claire : un helper pur `dashboard_filters_to_sql` qui ne
touche pas à la base, et une `prepare_dashboard_data` fine qui appelle DuckDB.
## Non-objectifs
- Pas de refonte de l'UI de filtres.
- Pas d'optimisation ou de cache supplémentaire autour de
`_compute_dashboard_children` (déjà `@cache.memoize()`).
- Pas de changement du comportement par défaut (365 derniers jours quand aucune
année n'est sélectionnée).
## Architecture
### Nouveau helper — `src/utils/table_sql.py`
```python
def dashboard_filters_to_sql(
dashboard_year=None,
dashboard_acheteur_id=None,
dashboard_acheteur_categorie=None,
dashboard_acheteur_departement_code=None,
dashboard_titulaire_id=None,
dashboard_titulaire_categorie=None,
dashboard_titulaire_departement_code=None,
dashboard_marche_type=None,
dashboard_marche_objet=None,
dashboard_marche_code_cpv=None,
dashboard_marche_considerations_sociales=None,
dashboard_marche_considerations_environnementales=None,
dashboard_marche_techniques=None,
dashboard_marche_innovant=None,
dashboard_marche_sous_traitance_declaree=None,
dashboard_montant_min=None,
dashboard_montant_max=None,
) -> tuple[str, list]:
"""Traduit les filtres du tableau de bord en (where_clause, params) DuckDB."""
```
Fonction pure, sans accès à la base. Même signature que `prepare_dashboard_data`
actuelle (hors `lff`). Retourne `("TRUE", [])` si aucun filtre n'est actif.
### Réécriture — `prepare_dashboard_data` (`src/utils/data.py`)
```python
def prepare_dashboard_data(**filter_params) -> pl.DataFrame:
where_sql, params = dashboard_filters_to_sql(**filter_params)
return query_marches(where_sql=where_sql, params=params)
```
- **Signature** : suppression du paramètre `lff`. Retour `pl.DataFrame` (et non plus
`pl.LazyFrame`).
- Les appelants qui ont besoin d'une LazyFrame appellent `.lazy()` sur le résultat.
### Appelants — `src/pages/observatoire.py`
Trois sites d'appel à adapter :
1. **`_compute_dashboard_children`** (ligne ~668) — on remplace
```python
lff: pl.LazyFrame = query_marches().lazy()
lff = prepare_dashboard_data(lff=lff, **filter_params)
dff = lff.collect(engine="streaming")
```
par
```python
dff = prepare_dashboard_data(**filter_params)
lff = dff.lazy()
```
Les appels existants à `make_donut`, `get_distance_histogram`, `get_top_org_table`,
`get_barchart_sources` continuent de recevoir `lff` ; `get_geographic_maps`
continue de recevoir `dff`. `df_per_uid` est calculé à partir de `dff`.
2. **`download_observatoire`** (ligne ~791) —
```python
dff = prepare_dashboard_data(**(filter_params or {}))
if hidden_columns:
dff = dff.drop(hidden_columns)
def to_bytes(buffer):
dff.write_excel(buffer, worksheet="DECP")
```
3. **`populate_preview_table`** (ligne ~882) —
```python
dff = prepare_dashboard_data(**(filter_params or {}))
return prepare_table_data(
dff.lazy(), # prepare_table_data accepte une LazyFrame
...
)
```
## Traduction des filtres
| Filtre | Actuel (Polars) | Cible (SQL DuckDB) |
| --------------------------------------------------------- | ---------------------------------------------------------- | -------------------------------------------------------------- |
| `dashboard_year` (présent) | `dt.year() == int(year)` | `YEAR("dateNotification") = ?` |
| `dashboard_year` (absent) — comportement par défaut | `> now - 365j` | `"dateNotification" > ?` (datetime calculé à l'appel) |
| `dashboard_acheteur_id` | `str.contains(val)` | `"acheteur_id" LIKE ?` avec `%val%` |
| `dashboard_acheteur_categorie` | `== val` (skip si acheteur_id présent) | `"acheteur_categorie" = ?` |
| `dashboard_acheteur_departement_code` | `is_in(list)` (skip si acheteur_id présent) | `"acheteur_departement_code" IN (?, ?, ...)` |
| `dashboard_titulaire_id` | idem acheteur | idem |
| `dashboard_titulaire_categorie` | idem | idem |
| `dashboard_titulaire_departement_code` | idem | idem |
| `dashboard_marche_type` | `== val` | `"type" = ?` |
| `dashboard_marche_objet` | `str.contains("(?i)val")` | `"objet" ILIKE ?` avec `%val%` |
| `dashboard_marche_code_cpv` | `str.starts_with(val)` | `"codeCPV" LIKE ?` avec `val%` |
| `dashboard_marche_techniques` | `str.split(", ").list.set_intersection(xs).list.len() > 0` | `list_has_any(string_split("techniques", ', '), ?::VARCHAR[])` |
| `dashboard_marche_considerations_sociales` | idem | idem sur `"considerationsSociales"` |
| `dashboard_marche_considerations_environnementales` | idem | idem sur `"considerationsEnvironnementales"` |
| `dashboard_marche_innovant` (`"oui"`/`"non"`, sinon skip) | `== val` | `"marcheInnovant" = ?` |
| `dashboard_marche_sous_traitance_declaree` | idem | `"sousTraitanceDeclaree" = ?` |
| `dashboard_montant_min` | `>= val` | `"montant" >= ?` |
| `dashboard_montant_max` | `<= val` | `"montant" <= ?` |
**Logique conditionnelle conservée** : si `dashboard_acheteur_id` est fourni, les filtres
`categorie` et `departement_code` acheteur sont ignorés (même chose pour titulaire).
**Traitement des valeurs spéciales** :
- `dashboard_marche_innovant` / `dashboard_marche_sous_traitance_declaree` : valeur
`"all"` ou falsy → aucun filtre ajouté.
- `dashboard_year` : converti en `int` avant injection.
- `dashboard_montant_min` / `_max` : `None` → aucun filtre (distinct de `0`, qui reste
un filtre valide via `>=` ou `<=`).
**Sécurité SQL** : toutes les valeurs utilisateurs passent par DuckDB en paramètres liés
(`?`). Seuls des noms de colonnes statiques (contrôlés par le code) sont injectés dans le
fragment SQL via `f"..."`. Pas de différence avec le pattern existant de
`filter_query_to_sql`.
## Tests
### Unitaires (nouveaux)
Nouveau fichier `tests/test_dashboard_filters_to_sql.py` :
- Cas vide → `("TRUE", [])`.
- Un seul filtre simple (année, type, etc.) → fragment SQL et params attendus.
- Filtre montant min/max (migration de l'actuel `test_010_observatoire_montant_filter`).
- Filtre liste (techniques, considerationsSociales) → usage de `list_has_any`.
- Filtre acheteur_id fourni → catégorie/département acheteur ignorés.
- Filtre `"all"` / `None` sur innovant/sous_traitance → aucun fragment ajouté.
- Comportement par défaut sans année → fragment `"dateNotification" > ?` avec un param
datetime à ~365 j dans le passé (tolérance de quelques secondes).
### Intégration (nouveau, léger)
Un test qui appelle `prepare_dashboard_data` contre `tests/test.parquet` avec un ou
deux filtres connus, vérifie le `height` et la bonne nature du retour (`pl.DataFrame`).
### Test Selenium existant
`test_009_observatoire_filter_persistence` et `test_008_observatoire_navigation_from_search`
ne touchent pas à la signature ; ils doivent continuer à passer.
## Risques et migration
- **Risque sémantique** : la fonction Polars `str.contains` utilisée pour les IDs est
un regex. Les utilisateurs attendent probablement un contains littéral sur un SIRET
(14 chiffres). Le passage à `LIKE '%val%'` est neutre si la valeur ne contient pas de
caractère spécial regex — ce qui est le cas pour des SIRET. **Hypothèse** acceptée :
le contenu `dashboard_acheteur_id`/`dashboard_titulaire_id` est alphanumérique.
- **Risque de drift du cache** : la date "365 derniers jours" n'est pas incluse dans
la clé de cache de `_compute_dashboard_children`. C'est un comportement pré-existant
; non traité par ce spec.
- **Import circulaire** : `src/utils/data.py` importe déjà depuis `src/db.py`.
`src/utils/table_sql.py` importe depuis `src/utils/table.py`. Pas de nouveau cycle.
## Succès
- Les 3 callbacks de l'observatoire restent fonctionnellement équivalents.
- Les tests unitaires et d'intégration passent.
- Une inspection manuelle confirme un temps d'exécution réduit sur un filtre
sélectif (par ex. un département + une année).
+4 -2
View File
@@ -1,7 +1,7 @@
[project]
name = "decp.info"
description = "Interface d'exploration et d'analyse des marchés publics français."
version = "2.7.2"
version = "2.7.6"
requires-python = ">= 3.10"
authors = [{ name = "Colin Maudry", email = "colin@colmo.tech" }]
dependencies = [
@@ -21,9 +21,10 @@ dependencies = [
"duckdb",
"flask-caching",
"pyarrow>=23.0.1",
"flask-cors>=6.0.2",
]
[project.optional-dependencies]
[dependency-groups]
dev = [
"pytest",
"pytest-env",
@@ -40,6 +41,7 @@ testpaths = ["tests"]
env = [
"DATA_FILE_PARQUET_PATH=tests/test.parquet",
"DEVELOPMENT=true",
"REBUILD_DUCKDB=true",
"DATA_SCHEMA_PATH=/home/colin/git/decp-processing/dist/schema.json",
]
addopts = "-p no:warnings"
+3
View File
@@ -1,7 +1,10 @@
from flask_cors import CORS
from src.app import app
# To use `gunicorn run:server` (prod)
server = app.server
CORS(server)
# To use `python run.py` (dev)
if __name__ == "__main__":
+2 -1
View File
@@ -2,13 +2,14 @@ import os
from shutil import rmtree
import dash_bootstrap_components as dbc
import pandas # noqa: F401 # eager import: avoid plotly's lazy-import race across Dash callback threads
import tomllib
from dash import Dash, Input, Output, State, dcc, html, page_container, page_registry
from dotenv import load_dotenv
from flask import Response
from src.cache import cache
from src.utils import DEVELOPMENT
from src.utils.cache import cache
load_dotenv()
+24 -9
View File
@@ -109,15 +109,8 @@ def build_database(db_path: Path, parquet_path: Path) -> None:
logger.info(f"Base DuckDB construite : {db_path}")
def _resolve_db_path() -> Path:
parquet = os.getenv("DATA_FILE_PARQUET_PATH")
if not parquet:
raise RuntimeError("DATA_FILE_PARQUET_PATH is not set")
return Path(parquet).parent / "decp.duckdb"
def _ensure_database() -> Path:
db_path = _resolve_db_path()
db_path = Path(os.getenv("DUCKDB_PATH", "./decp.duckdb"))
parquet_path = Path(os.getenv("DATA_FILE_PARQUET_PATH"))
lock_path = db_path.with_suffix(".duckdb.lock")
@@ -142,10 +135,11 @@ def get_cursor() -> duckdb.DuckDBPyConnection:
def query_marches(
where_sql: str = "TRUE",
params: tuple = (),
params: tuple | list = (),
columns: list[str] | None = None,
order_by: str | None = None,
limit: int | None = None,
offset: int | None = None,
) -> pl.DataFrame:
"""Run a parameterized SELECT against the decp table and return Polars.
@@ -159,4 +153,25 @@ def query_marches(
sql += f" ORDER BY {order_by}"
if limit is not None:
sql += f" LIMIT {int(limit)}"
if offset is not None:
sql += f" OFFSET {int(offset)}"
logger.debug("query_marches: " + sql.replace("?", "{}").format(*params))
return get_cursor().execute(sql, list(params)).pl()
def count_marches(where_sql: str = "TRUE", params: tuple | list = ()) -> int:
"""Retourne le nombre de lignes correspondant à where_sql."""
sql = f"SELECT COUNT(*) FROM decp WHERE {where_sql}"
logger.debug("count_marches: " + sql.replace("?", "{}").format(*params))
result = get_cursor().execute(sql, list(params)).fetchone()
return int(result[0]) if result else 0
def count_unique_marches(where_sql: str = "TRUE", params: tuple | list = ()) -> int:
"""Retourne le nombre de uid distincts correspondant à where_sql."""
sql = f"SELECT COUNT(DISTINCT uid) FROM decp WHERE {where_sql}"
logger.debug("count_unique_marches: " + sql.replace("?", "{}").format(*params))
result = get_cursor().execute(sql, list(params)).fetchone()
return int(result[0]) if result else 0
+66 -21
View File
@@ -11,8 +11,10 @@ import plotly.graph_objects as go
import polars as pl
from dash import dash_table, dcc, html
from dash_extensions.javascript import Namespace
from polars.exceptions import ColumnNotFoundError
from src.db import schema
from src.utils import logger
from src.utils.data import DATA_SCHEMA, DEPARTEMENTS_GEOJSON
from src.utils.table import add_links, format_number, setup_table_columns
@@ -173,39 +175,78 @@ def get_sources_tables(source_path) -> html.Div:
return html.Div(children=datatable)
def point_on_map(lat, lon):
def point_on_map(lat, lon, departement_code=None):
"""Fonction améliorée utilisant les codes départementaux pour la détection de région.
Args:
lat: Coordonnée de latitude
lon: Coordonnée de longitude
departement_code: Code du département (ex: '75', '971', etc.)
Returns:
html.Div contenant la carte, ou div vide si invalide
"""
# Validation des coordonnées
try:
lat = float(lat)
lon = float(lon)
except (TypeError, ValueError):
return html.Div() # Div vide pour les coordonnées invalides
# Create a scatter mapbox or choropleth map
# Vérification que les coordonnées sont valides
if not (-90 <= lat <= 90) or not (-180 <= lon <= 180):
return html.Div()
# Si aucun code département n'est fourni, retourner une div vide
if not departement_code:
return html.Div()
# Détermination de la région en utilisant le code département
# Logique identique à get_geographic_maps
if departement_code in ["971", "972", "973", "974", "976"]:
region_key = departement_code # Département d'outre-mer
elif len(departement_code) == 2: # Département métropolitain
region_key = "Hexagone"
else:
return html.Div() # Format de code département invalide
# Paramètres de carte par région (réutilisés de get_geographic_maps)
regions = {
"Hexagone": {"center": [46.6, 2.2], "zoom": 5},
"971": {"center": [16.23, -61.55], "zoom": 9}, # Guadeloupe
"972": {"center": [14.64, -61.02], "zoom": 10}, # Martinique
"973": {"center": [3.93, -53.12], "zoom": 7}, # Guyane
"974": {"center": [-21.11, 55.53], "zoom": 9}, # La Réunion
"976": {"center": [-12.82, 45.16], "zoom": 10}, # Mayotte
}
settings = regions.get(region_key, regions["Hexagone"])
# Création de la carte
fig = px.scatter_map(
lat=[lat], lon=[lon], height=300, width=400, color=[1], size=[1]
lat=[lat],
lon=[lon],
height=300,
# width=400,
color=[1],
zoom=settings["zoom"],
)
fig.update_coloraxes(showscale=False)
fig.update_traces(marker=dict(size=10))
# Set map style (you can use 'open-street-map', 'carto-positron', etc.)
# Configuration de la carte (interactive - zoomable)
fig.update_layout(
mapbox_style="light", # Light, clean background
map_style="light", # Fond de carte clair
margin={"r": 0, "t": 0, "l": 0, "b": 0},
mapbox_center={"lat": settings["center"][0], "lon": settings["center"][1]},
mapbox_zoom=settings["zoom"],
coloraxis_showscale=False,
)
# Optionally, center the map on France
fig.update_geos(
center=dict(lat=46.603354, lon=1.888334), # Center of France
lataxis_range=[41, 51.5], # Latitude range for France
lonaxis_range=[-5, 10], # Longitude range for France
return html.Div(
dcc.Graph(figure=fig, config={"displayModeBar": False}),
)
# But scatter_mapbox doesn't use geos, so better to control via zoom/center manually
# Let's reset and use proper centering in scatter_mapbox instead:
fig.update_layout(map_center={"lat": 46.6, "lon": 1.89}, map_zoom=4)
graph = dcc.Graph(id="map", figure=fig)
graph = html.Div(style={"width": "400px"})
return graph
class DataTable(dash_table.DataTable):
def __init__(
@@ -833,13 +874,17 @@ def get_top_org_table(data, org_type: str, extra_columns: list, filters: bool =
lff = lff.cast(pl.String)
lff = lff.fill_null("")
try:
dff: pl.DataFrame = lff.collect(engine="streaming")
except ColumnNotFoundError:
logger.warning(f"get_top_org_table: column not found. {lff.collect_schema()}")
return html.Div()
if dff.height == 0:
return html.Div()
columns, tooltip = setup_table_columns(
dff, hideable=False, exclude=[f"{org_type}_id"], new_columns=["Attributions"]
dff, hideable=False, exclude=[f"{org_type}_id"]
)
dff = add_links(dff)
data = dff.to_dicts()
+11 -2
View File
@@ -35,6 +35,7 @@ from src.utils.table import (
prepare_table_data,
sort_table_data,
)
from src.utils.tracking import track_search
def get_title(acheteur_id: str | None = None) -> str:
@@ -256,8 +257,15 @@ def update_acheteur_infos(url):
if data_etablissement:
data_etablissement = data_etablissement[0]
# Extraction du code département à partir du code postal
code_postal = data_etablissement.get("code_postal", "")
departement_code = code_postal[:2] if code_postal else None
# Création de la carte avec le code département pour un centrage approprié
acheteur_map = point_on_map(
data_etablissement["latitude"], data_etablissement["longitude"]
data_etablissement["latitude"],
data_etablissement["longitude"],
departement_code,
)
code_departement, nom_departement, nom_region = get_departement_region(
data_etablissement["code_postal"]
@@ -424,7 +432,8 @@ def download_filtered_acheteur_data(
lff = lff.drop(hidden_columns)
if filter_query:
lff = filter_table_data(lff, filter_query, "ach download")
track_search(filter_query, "ach download")
lff = filter_table_data(lff, filter_query)
if len(sort_by) > 0:
lff = sort_table_data(lff, sort_by)
+2 -2
View File
@@ -109,7 +109,7 @@ def update_marche_info(marche, titulaires):
column_object = DATA_SCHEMA.get(col)
column_name = column_object.get("title") if column_object else col
if marche[col]:
if col in marche:
if col == "acheteur_nom":
value = html.A(
href=f"/acheteurs/{marche['acheteur_id']}",
@@ -243,7 +243,7 @@ def get_marche_jsonld(marche, titulaires) -> str:
titulaire.get("titulaire_id"),
org_name=titulaire.get("titulaire_nom"),
org_type="titulaire",
type_org_id=titulaire.get("titulaire_typeIdentifiant"),
type_org_id=titulaire.get("titulaire_typeIdentifiant", "SIRET"),
),
"orderedItem": {
"@type": type_order,
+28 -16
View File
@@ -16,8 +16,7 @@ from dash import (
register_page,
)
from src.cache import cache
from src.db import query_marches, schema
from src.db import schema
from src.figures import (
DataTable,
get_barchart_sources,
@@ -31,6 +30,7 @@ from src.figures import (
make_donut,
)
from src.utils import logger
from src.utils.cache import cache
from src.utils.data import (
DEPARTEMENTS,
DF_ACHETEURS,
@@ -508,6 +508,11 @@ Alors, on fait comment ?
size="xl",
),
# DataTable
dcc.Loading(
overlay_style={"visibility": "visible", "filter": "blur(2px)"},
id="loading-statistques",
type="default",
children=[
html.Div(
className="marches_table",
children=DataTable(
@@ -517,8 +522,12 @@ Alors, on fait comment ?
sort_action="custom",
filter_action="custom",
hidden_columns=[],
columns=[{"id": col, "name": col} for col in OBSERVATOIRE_COLUMNS],
columns=[
{"id": col, "name": col} for col in OBSERVATOIRE_COLUMNS
],
),
)
],
),
],
),
@@ -658,14 +667,14 @@ def _normalize_filter_params(filter_params: dict) -> tuple:
@cache.memoize()
def _compute_dashboard_children(cache_key: tuple):
def _compute_dashboard_children(filter_params_normalized: tuple):
logger.debug("Cache miss — computing dashboard")
filter_params = {k: (list(v) if isinstance(v, tuple) else v) for k, v in cache_key}
filter_params = {
k: (list(v) if isinstance(v, tuple) else v) for k, v in filter_params_normalized
}
lff: pl.LazyFrame = query_marches().lazy()
lff = prepare_dashboard_data(lff=lff, **filter_params)
dff = lff.collect(engine="streaming")
dff = prepare_dashboard_data(**filter_params)
lff = dff.lazy()
df_per_uid = (
dff.select("uid", "montant").group_by("uid").agg(pl.col("montant").first())
@@ -772,8 +781,8 @@ def update_dashboard_cards(*filter_values):
):
filter_params[input_id] = value
cache_key = _normalize_filter_params(filter_params)
children = _compute_dashboard_children(cache_key)
filter_params_normalized = _normalize_filter_params(filter_params)
children = _compute_dashboard_children(filter_params_normalized)
return dbc.Row(children=children), filter_params
@@ -786,13 +795,13 @@ def update_dashboard_cards(*filter_values):
prevent_initial_call=True,
)
def download_observatoire(_n_clicks, filter_params, hidden_columns):
lff = prepare_dashboard_data(lff=query_marches().lazy(), **(filter_params or {}))
dff = prepare_dashboard_data(**(filter_params or {}))
if hidden_columns:
lff = lff.drop(hidden_columns)
dff = dff.drop(hidden_columns)
def to_bytes(buffer):
lff.collect(engine="streaming").write_excel(buffer, worksheet="DECP")
dff.write_excel(buffer, worksheet="DECP")
date = datetime.now().strftime("%Y-%m-%d_%H:%M:%S")
return dcc.send_bytes(to_bytes, filename=f"decp_observatoire_{date}.xlsx")
@@ -815,6 +824,9 @@ def toggle_montant_modal(n_triggers, _close):
prevent_initial_call=False,
)
def add_organization_name_in_title(acheteur_id, titulaire_id):
acheteur_id = acheteur_id.replace(" ", "") if acheteur_id else None
titulaire_id = titulaire_id.replace(" ", "") if titulaire_id else None
def lookup_nom(df_org, id_col, nom_col, org_id):
match = df_org.filter(pl.col(id_col) == org_id)
return match[nom_col].item(0) if match.height >= 1 else None
@@ -877,10 +889,10 @@ def populate_preview_table(
if not is_open:
return (no_update,) * 9
lff = prepare_dashboard_data(lff=query_marches().lazy(), **(filter_params or {}))
dff = prepare_dashboard_data(**(filter_params or {}))
return prepare_table_data(
lff,
dff.lazy(),
data_timestamp,
filter_query,
page_current,
+12 -9
View File
@@ -31,6 +31,7 @@ from src.utils.table import (
prepare_table_data,
sort_table_data,
)
from src.utils.tracking import track_search
update_date_timestamp = os.path.getmtime(os.getenv("DATA_FILE_PARQUET_PATH"))
update_date = datetime.fromtimestamp(update_date_timestamp).strftime("%d/%m/%Y")
@@ -162,18 +163,19 @@ layout = [
Vous pouvez appliquer un filtre pour chaque colonne en entrant du texte sous le nom de la colonne, puis en tapant sur `Entrée`.
- Champs textuels : la recherche retourne les valeurs qui contiennent le texte recherché et n'est pas sensible à la casse (majuscules/minuscules).
- Exemple : `rennes` retourne "RENNES METROPOLE".
- Champs textuels : la recherche retourne les valeurs qui contiennent le texte recherché, n'est pas sensible à la casse (majuscules/minuscules) et est sensbible à l'accentuation.
- `rennes` => le texte contient "rennes"
- `metro* *pole` => le texte contient un mot qui commence par "metro" et un mot qui finit par "pole"
- `metropole rennes` => le texte contient les mots "metropole" et "rennes", n'importe où dans le texte
- `metropole+rennes` => le texte contient "metropole rennes", collé et dans cet ordre
- `metropole+rennes travaux distri*` => le texte contient "metropole rennes", "travaux" et un mot qui commence par "distri"
- Les guillemets simples (apostrophe du 4) doivent être prédédées d'une barre oblique (AltGr + 8). Exemple : `services d\\\'assurances`
- Champs numériques (Durée en mois, Montant, ...) : vous pouvez...
- soit taper un nombre pour trouver les valeurs strictement égales. Exemple : `12` ne retourne que des 12
- soit le précéder de **>** ou **<** pour filtrer les valeurs supérieures ou inférieures. Exemple pour les offres reçues : `> 4` retourne les marchés ayant reçu plus de 4 offres.
- Champs date (Date de notification, ...) : vous pouvez également utiliser **>** ou **<**. Exemples :
- Champs date (Date de notification, ...) :
- `< 2024-01-31` pour "avant le 31 janvier 2024"
- `2024` pour "en 2024", `> 2022` pour "à partir de 2022".
- Pour les champs textuels et les champs dates :
- pour chercher du texte qui **commence par** votre texte, entrez `texte*`. C'est par exemple utile pour filtrer des acheteurs ou titulaires par numéro SIREN (`123456789*`) ou les marchés sur une année en particulier (`2024*`)
- pour chercher du texte qui **finit par** votre texte, entrez `*texte`
- `2024` pour "en 2024", `> 2022` pour "à partir de 2022"
Vous pouvez filtrer plusieurs colonnes à la fois.
@@ -314,7 +316,7 @@ def update_table(href, page_current, page_size, filter_query, sort_by, data_time
State("tableau_datatable", "hidden_columns"),
prevent_initial_call=True,
)
def download_data(n_clicks, filter_query, sort_by, hidden_columns: list = None):
def download_data(n_clicks, filter_query, sort_by, hidden_columns: list | None = None):
lff: pl.LazyFrame = query_marches().lazy()
# Les colonnes masquées sont supprimées
@@ -322,7 +324,8 @@ def download_data(n_clicks, filter_query, sort_by, hidden_columns: list = None):
lff = lff.drop(hidden_columns)
if filter_query:
lff = filter_table_data(lff, filter_query, "tab download")
track_search(filter_query, "tab download")
lff = filter_table_data(lff, filter_query)
if sort_by and len(sort_by) > 0:
lff = sort_table_data(lff, sort_by)
+17 -3
View File
@@ -34,6 +34,7 @@ from src.utils.table import (
prepare_table_data,
sort_table_data,
)
from src.utils.tracking import track_search
def get_title(titulaire_id: str = None) -> str:
@@ -262,8 +263,15 @@ def update_titulaire_infos(url):
if data_etablissement:
data_etablissement = data_etablissement[0]
# Extraction du code département à partir du code postal
code_postal = data_etablissement.get("code_postal", "")
departement_code = code_postal[:2] if code_postal else None
# Création de la carte avec le code département pour un centrage approprié
titulaire_map = point_on_map(
data_etablissement["latitude"], data_etablissement["longitude"]
data_etablissement["latitude"],
data_etablissement["longitude"],
departement_code,
)
code_departement, nom_departement, nom_region = get_departement_region(
data_etablissement["code_postal"]
@@ -429,7 +437,12 @@ def download_titulaire_data(
prevent_initial_call=True,
)
def download_filtered_titulaire_data(
data, n_clicks, titulaire_nom, filter_query, sort_by, hidden_columns: list = None
data,
n_clicks,
titulaire_nom,
filter_query,
sort_by,
hidden_columns: list | None = None,
):
lff: pl.LazyFrame = pl.LazyFrame(
data
@@ -440,7 +453,8 @@ def download_filtered_titulaire_data(
lff = lff.drop(hidden_columns)
if filter_query:
lff = filter_table_data(lff, filter_query, "titu download")
track_search(filter_query, "titu download")
lff = filter_table_data(lff, filter_query)
if len(sort_by) > 0:
lff = sort_table_data(lff, sort_by)
View File
+11 -110
View File
@@ -2,18 +2,17 @@ import json
import logging
import os
from collections import OrderedDict
from datetime import datetime, timedelta
import polars as pl
from httpx import HTTPError, get
from src.db import get_cursor, schema
from src.db import get_cursor, query_marches, schema
from src.utils import logger
logging.getLogger("httpx").setLevel("WARNING")
def get_annuaire_data(siret: str) -> dict:
def get_annuaire_data(siret: str) -> dict | None:
url = f"https://recherche-entreprises.api.gouv.fr/search?q={siret}"
try:
response = get(url).raise_for_status()
@@ -83,115 +82,17 @@ def get_data_schema() -> dict:
return new_schema
def prepare_dashboard_data(
lff: pl.LazyFrame,
dashboard_year=None,
dashboard_acheteur_id=None,
dashboard_acheteur_categorie=None,
dashboard_acheteur_departement_code=None,
dashboard_titulaire_id=None,
dashboard_titulaire_categorie=None,
dashboard_titulaire_departement_code=None,
dashboard_marche_type=None,
dashboard_marche_objet=None,
dashboard_marche_code_cpv=None,
dashboard_marche_considerations_sociales=None,
dashboard_marche_considerations_environnementales=None,
dashboard_marche_techniques=None,
dashboard_marche_innovant=None,
dashboard_marche_sous_traitance_declaree=None,
dashboard_montant_min=None,
dashboard_montant_max=None,
) -> pl.LazyFrame:
if dashboard_year:
lff = lff.filter(pl.col("dateNotification").dt.year() == int(dashboard_year))
else:
lff = lff.filter(
pl.col("dateNotification") > (datetime.now() - timedelta(days=365))
)
def prepare_dashboard_data(**filter_params) -> pl.DataFrame:
"""Exécute la requête DuckDB filtrée pour le tableau de bord.
if dashboard_acheteur_id:
lff = lff.filter(pl.col("acheteur_id").str.contains(dashboard_acheteur_id))
else:
if dashboard_acheteur_categorie:
lff = lff.filter(
pl.col("acheteur_categorie") == dashboard_acheteur_categorie
)
if dashboard_acheteur_departement_code:
lff = lff.filter(
pl.col("acheteur_departement_code").is_in(
dashboard_acheteur_departement_code
)
)
Retourne une pl.DataFrame matérialisée uniquement pour le sous-ensemble
correspondant aux filtres. Les appelants qui ont besoin d'une LazyFrame
appellent `.lazy()` sur le résultat.
"""
from src.utils.table_sql import dashboard_filters_to_sql
if dashboard_titulaire_id:
lff = lff.filter(pl.col("titulaire_id").str.contains(dashboard_titulaire_id))
else:
if dashboard_titulaire_categorie:
lff = lff.filter(
pl.col("titulaire_categorie") == dashboard_titulaire_categorie
)
if dashboard_titulaire_departement_code:
lff = lff.filter(
pl.col("titulaire_departement_code").is_in(
dashboard_titulaire_departement_code
)
)
if dashboard_marche_type:
lff = lff.filter(pl.col("type") == dashboard_marche_type)
if dashboard_marche_objet:
lff = lff.filter(pl.col("objet").str.contains(f"(?i){dashboard_marche_objet}"))
if dashboard_marche_code_cpv:
lff = lff.filter(pl.col("codeCPV").str.starts_with(dashboard_marche_code_cpv))
if dashboard_marche_innovant and dashboard_marche_innovant != "all":
lff = lff.filter(pl.col("marcheInnovant") == dashboard_marche_innovant)
if (
dashboard_marche_sous_traitance_declaree
and dashboard_marche_sous_traitance_declaree != "all"
):
lff = lff.filter(
pl.col("sousTraitanceDeclaree") == dashboard_marche_sous_traitance_declaree
)
if dashboard_marche_techniques:
lff = lff.filter(
pl.col("techniques")
.str.split(", ")
.list.set_intersection(dashboard_marche_techniques)
.list.len()
> 0
)
if dashboard_marche_considerations_sociales:
lff = lff.filter(
pl.col("considerationsSociales")
.str.split(", ")
.list.set_intersection(dashboard_marche_considerations_sociales)
.list.len()
> 0
)
if dashboard_marche_considerations_environnementales:
lff = lff.filter(
pl.col("considerationsEnvironnementales")
.str.split(", ")
.list.set_intersection(dashboard_marche_considerations_environnementales)
.list.len()
> 0
)
if dashboard_montant_min is not None:
lff = lff.filter(pl.col("montant") >= dashboard_montant_min)
if dashboard_montant_max is not None:
lff = lff.filter(pl.col("montant") <= dashboard_montant_max)
return lff
where_sql, params = dashboard_filters_to_sql(**filter_params)
return query_marches(where_sql=where_sql, params=params)
def build_org_frame(org_type: str) -> pl.DataFrame:
View File
+2
View File
@@ -7,6 +7,8 @@ def make_org_jsonld(org_id, org_type, org_name=None, type_org_id="SIRET") -> dic
address = None
if type_org_id.lower() == "siret" and len(org_id) == 14:
annuaire_data = get_annuaire_data(org_id)
if not annuaire_data:
return {}
annuaire_address = annuaire_data["matching_etablissements"][0]
code_postal = annuaire_address["code_postal"]
commune = annuaire_address["libelle_commune"]
+104 -46
View File
@@ -5,8 +5,9 @@ import polars as pl
from dash import no_update
from polars import selectors as cs
from src.db import query_marches, schema
from src.db import count_marches, count_unique_marches, query_marches, schema
from src.utils import logger
from src.utils.cache import cache
from src.utils.data import DATA_SCHEMA
from src.utils.frontend import get_button_properties
from src.utils.tracking import track_search
@@ -146,7 +147,15 @@ def dates_to_strings(lff: pl.LazyFrame, column: str) -> pl.LazyFrame:
return lff
def normalize_sort_by(sort_by) -> tuple:
if not sort_by:
return ()
return tuple((entry["column_id"], entry["direction"]) for entry in sort_by)
def format_number(number) -> str:
if not number:
return ""
number = "{:,}".format(number).replace(",", " ")
return number
@@ -160,7 +169,7 @@ def unformat_montant(number: str) -> float:
def format_values(dff: pl.DataFrame) -> pl.DataFrame:
def format_montant(expr, scale=None):
def format_montant(expr):
# https://stackoverflow.com/a/78636786
expr = expr.cast(pl.String)
expr = expr.str.splitn(".", 2)
@@ -206,14 +215,13 @@ def format_values(dff: pl.DataFrame) -> pl.DataFrame:
return dff
def filter_table_data(
lff: pl.LazyFrame, filter_query: str, filter_source: str
) -> pl.LazyFrame:
def filter_table_data(lff: pl.LazyFrame, filter_query: str) -> pl.LazyFrame:
_schema = lff.collect_schema()
track_search(filter_query, filter_source)
filtering_expressions = filter_query.split(" && ")
for filter_part in filtering_expressions:
col_name, operator, filter_value = split_filter_part(filter_part)
if not isinstance(col_name, str) or not isinstance(filter_value, str):
continue
col_type = str(_schema[col_name])
# logger.debug("filter_value:", filter_value)
# logger.debug("filter_value_type:", type(filter_value))
@@ -246,7 +254,7 @@ def filter_table_data(
elif operator == "<=":
lff = lff.filter(pl.col(col_name) <= filter_value)
elif operator == "contains":
if col_type in ["String", "Date"]:
if col_type in ["String", "Date"] and isinstance(filter_value, str):
filter_value = filter_value.strip('"')
if filter_value.endswith("*"):
lff = lff.filter(
@@ -284,7 +292,9 @@ def sort_table_data(lff: pl.LazyFrame, sort_by: list) -> pl.LazyFrame:
def setup_table_columns(
dff, hideable: bool = True, exclude: list = None, new_columns: list = None
dff,
hideable: bool = True,
exclude: list | None = None,
) -> tuple:
# Liste finale de colonnes
markdown_exceptions = ["montant", "titulaire_distance", "distance", "dureeMois"]
@@ -368,6 +378,62 @@ def get_default_hidden_columns(page):
return hidden_columns
def postprocess_page(dff: pl.DataFrame) -> pl.DataFrame:
"""Post-traitement à appliquer sur une page déjà paginée.
À appeler après la pagination.
"""
dff = dff.with_columns(pl.all().cast(pl.String).fill_null(""))
dff = add_links(dff)
if "sourceFile" in dff.columns:
dff = add_resource_link(dff)
if dff.height > 0:
dff = format_values(dff)
return dff
@cache.memoize()
def _fetch_page_sql(
filter_query: str | None,
sort_by_key: tuple,
page_current: int,
page_size: int,
) -> tuple[pl.DataFrame, int, int]:
"""Chemin rapide : filtre/tri/pagine dans DuckDB, post-traite la page seule.
Retourne (page_dataframe_post_traitée, total_count, total_unique_count).
"""
# Import local pour éviter une dépendance circulaire
# (src.utils.table_sql importe split_filter_part depuis src.utils.table).
from src.utils.table_sql import filter_query_to_sql, sort_by_to_sql
logger.debug(
f"Cache miss SQL — filter={filter_query!r} sort={sort_by_key!r} "
f"page={page_current} size={page_size}"
)
where_sql, params = filter_query_to_sql(filter_query or "", schema)
sort_by_dash = [
{"column_id": col, "direction": direction} for col, direction in sort_by_key
]
order_by = sort_by_to_sql(sort_by_dash, schema) or None
total = count_marches(where_sql, params)
total_unique = count_unique_marches(where_sql, params)
page = query_marches(
where_sql=where_sql,
params=params,
order_by=order_by,
limit=page_size,
offset=page_current * page_size,
)
page = postprocess_page(page)
return page, total, total_unique
def prepare_table_data(
data, data_timestamp, filter_query, page_current, page_size, sort_by, source_table
):
@@ -383,66 +449,58 @@ def prepare_table_data(
:param source_table:
:return:
"""
if os.getenv("DEVELOPMENT").lower() == "true":
logger.debug(" + + + + + + + + + + + + + + + + + + ")
trigger_cleanup = no_update
if filter_query:
track_search(filter_query, source_table)
# Récupération des données
trigger_cleanup = no_update if source_table == "tableau" else str(uuid.uuid4())
if data is None:
# Probablement car il s'agit de la page Tableau
sort_by_key = normalize_sort_by(sort_by)
dff, height, total_unique = _fetch_page_sql(
filter_query=filter_query,
sort_by_key=sort_by_key,
page_current=page_current,
page_size=page_size,
)
else:
if isinstance(data, list):
lff: pl.LazyFrame = pl.LazyFrame(data, strict=False, infer_schema_length=5000)
lff: pl.LazyFrame = pl.LazyFrame(
data, strict=False, infer_schema_length=5000
)
elif isinstance(data, pl.LazyFrame):
lff = data
else:
lff: pl.LazyFrame = query_marches().lazy()
lff = query_marches().lazy()
# Application des filtres
if filter_query:
lff = filter_table_data(lff, filter_query, source_table)
trigger_cleanup = no_update if source_table == "tableau" else str(uuid.uuid4())
lff = filter_table_data(lff, filter_query)
df_height = lff.select("uid").collect(engine="streaming")
height = df_height.height
total_unique = df_height["uid"].n_unique()
# Application des tris
if sort_by and len(sort_by) > 0:
lff = sort_table_data(lff, sort_by)
# Matérialisation des filtres
dff: pl.DataFrame = lff.collect()
height = dff.height
start_row = page_current * page_size
lff = lff.slice(start_row, page_size)
dff = lff.collect(engine="streaming")
dff: pl.DataFrame = postprocess_page(dff)
if height > 0:
nb_rows = f"{format_number(height)} lignes ({format_number(dff.select('uid').unique().height)} marchés)"
nb_rows = (
f"{format_number(height)} lignes ({format_number(total_unique)} marchés)"
)
else:
nb_rows = "0 lignes (0 marchés)"
# Pagination des données
start_row = page_current * page_size
# end_row = (page_current + 1) * page_size
dff = dff.slice(start_row, page_size)
# Tout devient string
dff = dff.cast(pl.String)
# Remplace les strings null par "", mais pas les numeric null
dff = dff.fill_null("")
# Ajout des liens vers les pages de détails
dff = add_links(dff)
# Ajout des liens vers les fichiers Open Data
if "sourceFile" in dff.columns:
dff = add_resource_link(dff)
# Formatage des montants
if height > 0:
dff = format_values(dff)
# Récupération des colonnes et tooltip
table_columns, tooltip = setup_table_columns(dff)
dicts = dff.to_dicts()
# Propriétés du bouton de téléchargement
download_disabled, download_text, download_title = get_button_properties(height)
return (
+241
View File
@@ -0,0 +1,241 @@
from datetime import datetime, timedelta
import polars as pl
from src.utils import logger
from src.utils.table import split_filter_part
def filter_query_to_sql(filter_query: str, schema: pl.Schema) -> tuple[str, list]:
"""Traduit le DSL de filtres de dash_table.DataTable en fragment SQL DuckDB.
Retourne (where_clause, params) where_clause est un fragment à injecter
après WHERE et params est la liste des valeurs à passer à
cursor.execute(sql, params). Les identifiants de colonnes sont validés
contre le schéma fourni ; jamais concaténés avec des valeurs utilisateur.
"""
if not filter_query:
return "TRUE", []
clauses: list[str] = []
params: list = []
for part in filter_query.split(" && "):
col_name, operator, raw_value = split_filter_part(part)
if not isinstance(col_name, str) or not isinstance(raw_value, str):
continue
if col_name not in schema.names():
logger.warning(f"Colonne inconnue ignorée : {col_name!r}")
continue
col_type = schema[col_name]
is_numeric = col_type.is_numeric()
col_is_date = col_type == pl.Date
quoted_col = f'"{col_name}"'
if is_numeric:
try:
value = int(raw_value) if col_type.is_integer() else float(raw_value)
except ValueError:
logger.warning(f"Valeur numérique invalide ignorée : {raw_value!r}")
continue
if operator == "contains":
clauses.append(f"{quoted_col} IS NOT NULL AND {quoted_col} = ?")
elif operator == ">":
clauses.append(f"{quoted_col} IS NOT NULL AND {quoted_col} > ?")
elif operator == "<":
clauses.append(f"{quoted_col} IS NOT NULL AND {quoted_col} < ?")
else:
logger.warning(f"Opérateur invalide pour numérique : {operator!r}")
continue
params.append(value)
continue
# String / Date : toujours traité comme texte (parité avec Polars)
value = raw_value.strip('"')
if operator == "contains":
if col_is_date:
target = f"CAST({quoted_col} AS VARCHAR)"
if col_name in ("acheteur_id", "titulaire_id"):
value = value.replace(" ", "")
where_clause, param_list = tokenize_text_filter(
col_name, value, col_is_date
)
clauses.append(where_clause)
params.extend(param_list)
logger.debug(params)
continue
elif operator in (">", "<"):
target = f"CAST({quoted_col} AS VARCHAR)" if col_is_date else quoted_col
clauses.append(f"{quoted_col} IS NOT NULL AND {target} {operator} ?")
params.append(value)
else:
logger.warning(f"Opérateur invalide pour chaîne : {operator!r}")
continue
if not clauses:
return "TRUE", []
return " AND ".join(clauses), params
def sort_by_to_sql(sort_by: list[dict] | None, schema: pl.Schema) -> str:
"""Traduit sort_by (format Dash) en clause ORDER BY DuckDB.
Retourne '' si pas de tri (aucun ORDER BY à ajouter).
"""
if not sort_by:
return ""
fragments: list[str] = []
for entry in sort_by:
col = entry.get("column_id")
direction = entry.get("direction")
if col not in schema.names():
logger.warning(f"Tri sur colonne inconnue ignoré : {col!r}")
continue
if direction not in ("asc", "desc"):
logger.warning(f"Tri sur direction inconnue ignoré : {direction!r}")
continue
fragments.append(f'"{col}" {direction.upper()} NULLS LAST')
return ", ".join(fragments)
def dashboard_filters_to_sql(
dashboard_year=None,
dashboard_acheteur_id=None,
dashboard_acheteur_categorie=None,
dashboard_acheteur_departement_code=None,
dashboard_titulaire_id=None,
dashboard_titulaire_categorie=None,
dashboard_titulaire_departement_code=None,
dashboard_marche_type=None,
dashboard_marche_objet=None,
dashboard_marche_code_cpv=None,
dashboard_marche_considerations_sociales=None,
dashboard_marche_considerations_environnementales=None,
dashboard_marche_techniques=None,
dashboard_marche_innovant=None,
dashboard_marche_sous_traitance_declaree=None,
dashboard_montant_min=None,
dashboard_montant_max=None,
) -> tuple[str, list]:
"""Traduit les filtres du tableau de bord en (where_clause, params) DuckDB."""
clauses: list[str] = []
params: list = []
if dashboard_year:
clauses.append('YEAR("dateNotification") = ?')
params.append(int(dashboard_year))
else:
clauses.append('"dateNotification" > ?')
params.append(datetime.now() - timedelta(days=365))
if dashboard_acheteur_id:
dashboard_acheteur_id = dashboard_acheteur_id.replace(" ", "")
clauses.append('"acheteur_id" LIKE ?')
params.append(f"%{dashboard_acheteur_id}%")
else:
if dashboard_acheteur_categorie:
clauses.append('"acheteur_categorie" = ?')
params.append(dashboard_acheteur_categorie)
if dashboard_acheteur_departement_code:
placeholders = ", ".join(["?"] * len(dashboard_acheteur_departement_code))
clauses.append(f'"acheteur_departement_code" IN ({placeholders})')
params.extend(dashboard_acheteur_departement_code)
if dashboard_titulaire_id:
dashboard_titulaire_id = dashboard_titulaire_id.replace(" ", "")
clauses.append('"titulaire_id" LIKE ?')
params.append(f"%{dashboard_titulaire_id}%")
else:
if dashboard_titulaire_categorie:
clauses.append('"titulaire_categorie" = ?')
params.append(dashboard_titulaire_categorie)
if dashboard_titulaire_departement_code:
placeholders = ", ".join(["?"] * len(dashboard_titulaire_departement_code))
clauses.append(f'"titulaire_departement_code" IN ({placeholders})')
params.extend(dashboard_titulaire_departement_code)
if dashboard_marche_type:
clauses.append('"type" = ?')
params.append(dashboard_marche_type)
if dashboard_marche_objet:
where_clause, param_list = tokenize_text_filter("objet", dashboard_marche_objet)
clauses.append(where_clause)
params.extend(param_list)
if dashboard_marche_code_cpv:
clauses.append('"codeCPV" LIKE ?')
params.append(f"{dashboard_marche_code_cpv}%")
if dashboard_marche_innovant and dashboard_marche_innovant != "all":
clauses.append('"marcheInnovant" = ?')
params.append(dashboard_marche_innovant)
if (
dashboard_marche_sous_traitance_declaree
and dashboard_marche_sous_traitance_declaree != "all"
):
clauses.append('"sousTraitanceDeclaree" = ?')
params.append(dashboard_marche_sous_traitance_declaree)
if dashboard_marche_techniques:
clauses.append("list_has_any(string_split(\"techniques\", ', '), ?::VARCHAR[])")
params.append(list(dashboard_marche_techniques))
if dashboard_marche_considerations_sociales:
clauses.append(
"list_has_any(string_split(\"considerationsSociales\", ', '), ?::VARCHAR[])"
)
params.append(list(dashboard_marche_considerations_sociales))
if dashboard_marche_considerations_environnementales:
clauses.append(
"list_has_any(string_split(\"considerationsEnvironnementales\", ', '), ?::VARCHAR[])"
)
params.append(list(dashboard_marche_considerations_environnementales))
if dashboard_montant_min is not None:
clauses.append('"montant" >= ?')
params.append(dashboard_montant_min)
if dashboard_montant_max is not None:
clauses.append('"montant" <= ?')
params.append(dashboard_montant_max)
return " AND ".join(clauses), params
def tokenize_text_filter(
column: str, text: str, col_is_date: bool = False
) -> tuple[str, list]:
terms = text.split()
# si col_is_date alors le deuxième doit être casté en VARCHAR
if col_is_date:
quoted_col = f'CAST("{column}" AS VARCHAR)'
else:
quoted_col = f'"{column}"'
conditions = [f'"{column}" IS NOT NULL', f"{quoted_col} <> ''"]
params = []
for term in terms:
conditions.append(f"{quoted_col} ILIKE ?")
if term.startswith("*") or term.endswith("*"):
params.append(term.replace("*", "%"))
elif "+" in term:
params.append(f"%{term.replace('+', ' ')}%")
else:
params.append(f"%{term}%")
where_clause = " AND ".join(conditions)
return where_clause, params
+23 -12
View File
@@ -6,10 +6,7 @@ import polars as pl
import pytest
from selenium.webdriver.chrome.options import Options
@pytest.fixture(scope="session", autouse=True)
def test_data():
data = [
_TEST_DATA = [
{
"uid": "1",
"id": "1",
@@ -42,19 +39,33 @@ def test_data():
"titulaire_categorie": "PME",
}
]
parquet_path = Path(os.path.abspath("tests/test.parquet"))
db_path = parquet_path.parent / "decp.duckdb"
print(f"Writing test data to: {parquet_path}")
_PARQUET_PATH = Path(os.path.abspath("tests/test.parquet"))
_DB_PATH = Path(os.path.abspath("decp.duckdb"))
pl.DataFrame(data).write_parquet(parquet_path)
# Remove any stale DuckDB from a previous run so src.db rebuilds from
# the freshly-written parquet at import time.
for artifact in (db_path, db_path.with_suffix(".duckdb.tmp")):
def _cleanup_db_artifacts() -> None:
for artifact in (
_DB_PATH,
_DB_PATH.with_suffix(".duckdb.tmp"),
_DB_PATH.with_suffix(".duckdb.lock"),
):
if artifact.exists():
artifact.unlink()
yield str(parquet_path)
# Runs at conftest import, before test modules import src.db (which builds the
# DuckDB at import time). Guarantees the test parquet exists and the stale DB
# from a previous `python run.py` is wiped so src.db rebuilds from test data.
pl.DataFrame(_TEST_DATA).write_parquet(_PARQUET_PATH)
_cleanup_db_artifacts()
@pytest.fixture(scope="session", autouse=True)
def test_data():
yield str(_PARQUET_PATH)
# Teardown: remove the test DuckDB so the next `python run.py` rebuilds
# from decp_prod.parquet.
_cleanup_db_artifacts()
def pytest_setup_options():
+225
View File
@@ -0,0 +1,225 @@
from datetime import datetime, timedelta
from src.utils.table_sql import dashboard_filters_to_sql
def test_no_filters_uses_default_365_day_window():
where_sql, params = dashboard_filters_to_sql()
assert where_sql == '"dateNotification" > ?'
assert len(params) == 1
assert isinstance(params[0], datetime)
expected = datetime.now() - timedelta(days=365)
assert abs((params[0] - expected).total_seconds()) < 2
def test_year_filter_overrides_default_window():
where_sql, params = dashboard_filters_to_sql(dashboard_year="2025")
assert where_sql == 'YEAR("dateNotification") = ?'
assert params == [2025]
def test_marche_type_equality():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_marche_type="Marché",
)
assert where_sql == 'YEAR("dateNotification") = ? AND "type" = ?'
assert params == [2025, "Marché"]
def test_innovant_value_all_is_skipped():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_marche_innovant="all",
)
assert where_sql == 'YEAR("dateNotification") = ?'
assert params == [2025]
def test_innovant_value_oui_adds_clause():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_marche_innovant="oui",
)
assert where_sql == 'YEAR("dateNotification") = ? AND "marcheInnovant" = ?'
assert params == [2025, "oui"]
def test_sous_traitance_value_non_adds_clause():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_marche_sous_traitance_declaree="non",
)
assert where_sql == 'YEAR("dateNotification") = ? AND "sousTraitanceDeclaree" = ?'
assert params == [2025, "non"]
def test_acheteur_id_uses_like_wildcards():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_acheteur_id="12345678900010",
)
assert where_sql == 'YEAR("dateNotification") = ? AND "acheteur_id" LIKE ?'
assert params == [2025, "%12345678900010%"]
def test_titulaire_id_uses_like_wildcards():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_titulaire_id="999",
)
assert where_sql == 'YEAR("dateNotification") = ? AND "titulaire_id" LIKE ?'
assert params == [2025, "%999%"]
def test_marche_objet_uses_case_insensitive_ilike():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_marche_objet="travaux",
)
assert (
where_sql
== 'YEAR("dateNotification") = ? AND "objet" IS NOT NULL AND "objet" <> \'\' AND "objet" ILIKE ?'
)
assert params == [2025, "%travaux%"]
def test_code_cpv_uses_prefix_like():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_marche_code_cpv="4521",
)
assert where_sql == 'YEAR("dateNotification") = ? AND "codeCPV" LIKE ?'
assert params == [2025, "4521%"]
def test_acheteur_departement_multiple_uses_in_clause():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_acheteur_departement_code=["75", "92", "93"],
)
assert where_sql == (
'YEAR("dateNotification") = ? AND "acheteur_departement_code" IN (?, ?, ?)'
)
assert params == [2025, "75", "92", "93"]
def test_acheteur_categorie_adds_clause():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_acheteur_categorie="Commune",
)
assert where_sql == 'YEAR("dateNotification") = ? AND "acheteur_categorie" = ?'
assert params == [2025, "Commune"]
def test_titulaire_categorie_and_departement():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_titulaire_categorie="PME",
dashboard_titulaire_departement_code=["35"],
)
assert where_sql == (
'YEAR("dateNotification") = ? '
'AND "titulaire_categorie" = ? '
'AND "titulaire_departement_code" IN (?)'
)
assert params == [2025, "PME", "35"]
def test_acheteur_id_present_skips_categorie_and_departement():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_acheteur_id="123",
dashboard_acheteur_categorie="Commune",
dashboard_acheteur_departement_code=["75"],
)
assert where_sql == 'YEAR("dateNotification") = ? AND "acheteur_id" LIKE ?'
assert params == [2025, "%123%"]
def test_titulaire_id_present_skips_categorie_and_departement():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_titulaire_id="999",
dashboard_titulaire_categorie="PME",
dashboard_titulaire_departement_code=["35"],
)
assert where_sql == 'YEAR("dateNotification") = ? AND "titulaire_id" LIKE ?'
assert params == [2025, "%999%"]
def test_marche_techniques_uses_list_has_any():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_marche_techniques=["Enchère", "Accord-cadre"],
)
assert where_sql == (
'YEAR("dateNotification") = ? '
"AND list_has_any(string_split(\"techniques\", ', '), ?::VARCHAR[])"
)
assert params == [2025, ["Enchère", "Accord-cadre"]]
def test_considerations_sociales_uses_list_has_any():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_marche_considerations_sociales=["Clause sociale"],
)
assert where_sql == (
'YEAR("dateNotification") = ? '
"AND list_has_any(string_split(\"considerationsSociales\", ', '), ?::VARCHAR[])"
)
assert params == [2025, ["Clause sociale"]]
def test_considerations_environnementales_uses_list_has_any():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_marche_considerations_environnementales=["Clause env."],
)
assert where_sql == (
'YEAR("dateNotification") = ? '
"AND list_has_any(string_split(\"considerationsEnvironnementales\", ', '), ?::VARCHAR[])"
)
assert params == [2025, ["Clause env."]]
def test_montant_min_only():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_montant_min=1000,
)
assert where_sql == 'YEAR("dateNotification") = ? AND "montant" >= ?'
assert params == [2025, 1000]
def test_montant_max_only():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_montant_max=500,
)
assert where_sql == 'YEAR("dateNotification") = ? AND "montant" <= ?'
assert params == [2025, 500]
def test_montant_zero_is_a_valid_lower_bound():
# 0 est falsy mais reste un filtre valide (distinct de None)
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_montant_min=0,
)
assert where_sql == 'YEAR("dateNotification") = ? AND "montant" >= ?'
assert params == [2025, 0]
def test_montant_min_and_max_combined():
where_sql, params = dashboard_filters_to_sql(
dashboard_year="2025",
dashboard_montant_min=100,
dashboard_montant_max=1000,
)
assert where_sql == (
'YEAR("dateNotification") = ? AND "montant" >= ? AND "montant" <= ?'
)
assert params == [2025, 100, 1000]
+33
View File
@@ -141,6 +141,7 @@ def built_db(tmp_path, monkeypatch):
)
data.write_parquet(parquet_path)
monkeypatch.setenv("DATA_FILE_PARQUET_PATH", str(parquet_path))
monkeypatch.setenv("DUCKDB_PATH", str(db_path))
from src.db import build_database
@@ -206,6 +207,38 @@ def test_query_marches_returns_polars_frame(built_db, monkeypatch):
assert set(frame["uid"].to_list()) == {"1", "2"}
def test_count_marches_returns_total_without_filter():
from src.db import count_marches
n = count_marches()
assert isinstance(n, int)
assert n > 0
def test_count_marches_with_filter():
from src.db import count_marches
n = count_marches('"uid" = ?', ["__nonexistent__"])
assert n == 0
def test_count_unique_marches_respects_distinct():
from src.db import count_unique_marches
n = count_unique_marches()
assert isinstance(n, int)
assert n > 0
def test_query_marches_with_offset():
from src.db import query_marches
page_0 = query_marches(limit=2, offset=0)
page_1 = query_marches(limit=2, offset=2)
if page_0.height == 2 and page_1.height >= 1:
assert set(page_0["uid"].to_list()).isdisjoint(set(page_1["uid"].to_list()))
def test_concurrent_build_serialized(tmp_path):
"""Multiple threads calling _ensure_database must serialize via flock.
+22 -44
View File
@@ -215,47 +215,6 @@ def test_008_search_to_observatoire(dash_duo: DashComposite):
)
def test_010_observatoire_montant_filter():
import datetime
from src.utils.data import prepare_dashboard_data
data = pl.DataFrame(
{
"uid": ["1", "2", "3"],
"montant": [100.0, 500.0, 1000.0],
"dateNotification": [datetime.date(2025, 1, 1)] * 3,
}
)
def apply(min_val=None, max_val=None):
return prepare_dashboard_data(
data.lazy(),
dashboard_year="2025",
dashboard_acheteur_id=None,
dashboard_acheteur_categorie=None,
dashboard_acheteur_departement_code=None,
dashboard_titulaire_id=None,
dashboard_titulaire_categorie=None,
dashboard_titulaire_departement_code=None,
dashboard_marche_type=None,
dashboard_marche_objet=None,
dashboard_marche_code_cpv=None,
dashboard_marche_considerations_sociales=None,
dashboard_marche_considerations_environnementales=None,
dashboard_marche_techniques=None,
dashboard_marche_innovant=None,
dashboard_marche_sous_traitance_declaree=None,
dashboard_montant_min=min_val,
dashboard_montant_max=max_val,
).collect()
assert apply().height == 3
assert apply(min_val=400).height == 2 # 500, 1000
assert apply(max_val=500).height == 2 # 100, 500
assert apply(min_val=200, max_val=600).height == 1 # 500 only
def test_009_observatoire_filter_persistence(dash_duo: DashComposite):
import time
@@ -333,7 +292,7 @@ def test_011_observatoire_multi_param_url(dash_duo: DashComposite):
)
def test_get_distance_histogram_returns_graph():
def test_012_get_distance_histogram_returns_graph():
import polars as pl
from dash import dcc
@@ -344,7 +303,7 @@ def test_get_distance_histogram_returns_graph():
assert isinstance(result, dcc.Graph)
def test_get_distance_histogram_handles_nulls():
def test_013_get_distance_histogram_handles_nulls():
import polars as pl
from dash import dcc
@@ -355,7 +314,7 @@ def test_get_distance_histogram_handles_nulls():
assert isinstance(result, dcc.Graph)
def test_get_distance_histogram_all_nulls():
def test_014_get_distance_histogram_all_nulls():
import polars as pl
from dash import dcc
@@ -363,4 +322,23 @@ def test_get_distance_histogram_all_nulls():
lff = pl.LazyFrame({"titulaire_distance": pl.Series([], dtype=pl.Int64)})
result = get_distance_histogram(lff)
assert isinstance(result, dcc.Graph)
def test_015_tableau_filter_date(dash_duo: DashComposite):
from src.app import app
dash_duo.start_server(app)
dash_duo.wait_for_text_to_equal(".logo > h1", "decp.info", timeout=4)
for page in ["tableau", "acheteurs/123", "titulaires/345"]:
dash_duo.wait_for_page(f"{dash_duo.server_url}/{page}")
filter_input = '.marches_table th[data-dash-column="dateNotification"] input'
filter_cell_result = '.marches_table td[data-dash-column="dateNotification"] p'
dash_duo.wait_for_element(filter_input, timeout=2)
_filter_input: WebElement = dash_duo.find_element(filter_input)
_filter_input.send_keys("3333") # a dateNotification that doesn't exist
_filter_input.send_keys(Keys.ENTER)
_filter_result: list[WebElement] = dash_duo.find_elements(filter_cell_result)
assert len(_filter_result) == 0, f"Page : {page}"
+288
View File
@@ -0,0 +1,288 @@
import polars as pl
import pytest
@pytest.fixture
def sample_lff():
"""Small LazyFrame with the columns needed by add_links / format_values."""
return pl.LazyFrame(
[
{
"uid": "u1",
"id": "u1",
"acheteur_id": "12345678900011",
"acheteur_nom": "Mairie de Test",
"titulaire_id": "98765432100022",
"titulaire_nom": "Entreprise Test",
"titulaire_typeIdentifiant": "SIRET",
"objet": "Travaux divers",
"montant": 12500.0,
"dateNotification": "2025-03-15",
"codeCPV": "45000000",
"dureeRestanteMois": 6,
"titulaire_distance": 42.0,
}
]
)
def test_table_module_imports():
from src.utils import table
assert hasattr(table, "prepare_table_data")
def test_filter_table_data_does_not_call_track_search(monkeypatch, sample_lff):
from src.utils import table
calls = []
monkeypatch.setattr(table, "track_search", lambda *a, **kw: calls.append(a))
result = table.filter_table_data(sample_lff, "{objet} icontains travaux").collect()
assert calls == []
assert result.height == 1
def test_normalize_sort_by_handles_empty():
from src.utils.table import normalize_sort_by
assert normalize_sort_by(None) == ()
assert normalize_sort_by([]) == ()
def test_normalize_sort_by_returns_hashable_tuple():
from src.utils.table import normalize_sort_by
sort_by = [
{"column_id": "montant", "direction": "desc"},
{"column_id": "dateNotification", "direction": "asc"},
]
key = normalize_sort_by(sort_by)
assert key == (("montant", "desc"), ("dateNotification", "asc"))
# Must be hashable so that flask-caching can build a cache key from it
hash(key)
def test_normalize_sort_by_preserves_order():
"""Order matters for sort: [A, B] != [B, A]."""
from src.utils.table import normalize_sort_by
a_then_b = normalize_sort_by(
[{"column_id": "a", "direction": "asc"}, {"column_id": "b", "direction": "asc"}]
)
b_then_a = normalize_sort_by(
[{"column_id": "b", "direction": "asc"}, {"column_id": "a", "direction": "asc"}]
)
assert a_then_b != b_then_a
@pytest.fixture(scope="module")
def flask_app():
"""Minimal Flask app with SimpleCache so @cache.memoize() works in tests."""
from flask import Flask
from src.utils.cache import cache
app = Flask(__name__)
cache.init_app(app, config={"CACHE_TYPE": "SimpleCache"})
return app
@pytest.fixture(autouse=True)
def reset_cache(flask_app):
"""Ensure the flask-caching backend is empty between tests so that
cache-hit assertions are meaningful. Falls back to no-op when no
Flask app context is active (NullCache)."""
from src.utils.cache import cache
with flask_app.app_context():
try:
cache.clear()
except (RuntimeError, AttributeError):
# No app context — cache is NullCache, nothing to clear
pass
yield
def test_prepare_table_data_returns_expected_tuple(flask_app):
from src.utils import table
with flask_app.app_context():
result = table.prepare_table_data(
data=None,
data_timestamp=5,
filter_query=None,
page_current=0,
page_size=20,
sort_by=[],
source_table="tableau",
)
# Same arity as before: 9 outputs
assert len(result) == 9
dicts, columns, tooltip, ts, nb_rows, dl_disabled, dl_text, dl_title, cleanup = (
result
)
assert isinstance(dicts, list)
assert ts == 6 # data_timestamp + 1 must still increment
assert "lignes" in nb_rows
def test_prepare_table_data_calls_track_search_on_filter(monkeypatch, flask_app):
from src.utils import table
calls = []
monkeypatch.setattr(table, "track_search", lambda *a, **kw: calls.append(a))
with flask_app.app_context():
table.prepare_table_data(
data=None,
data_timestamp=0,
filter_query="{objet} icontains travaux",
page_current=0,
page_size=20,
sort_by=[],
source_table="tableau",
)
assert calls == [("{objet} icontains travaux", "tableau")]
def test_prepare_table_data_same_page_uses_cache(monkeypatch, flask_app):
"""Two calls with exactly the same (filter, sort, page, size)
must call _fetch_page_sql at least once."""
from src.utils import table
call_count = {"n": 0}
def counting_fetch(*args, **kwargs):
call_count["n"] += 1
import polars as pl
return (
pl.DataFrame(
{
"uid": [],
"acheteur_id": [],
"titulaire_id": [],
"titulaire_typeIdentifiant": [],
}
),
0,
0,
)
monkeypatch.setattr(table, "_fetch_page_sql", counting_fetch)
with flask_app.app_context():
table.prepare_table_data(
data=None,
data_timestamp=0,
filter_query=None,
page_current=0,
page_size=10,
sort_by=[],
source_table="tableau",
)
table.prepare_table_data(
data=None,
data_timestamp=0,
filter_query=None,
page_current=0,
page_size=10,
sort_by=[],
source_table="tableau",
)
assert call_count["n"] >= 1
def test_prepare_table_data_cleanup_trigger_for_non_tableau(flask_app):
"""Non-tableau pages still get a fresh uuid trigger, not no_update."""
from dash import no_update
from src.utils import table
with flask_app.app_context():
result = table.prepare_table_data(
data=None,
data_timestamp=0,
filter_query="{objet} icontains travaux",
page_current=0,
page_size=20,
sort_by=[],
source_table="acheteur",
)
cleanup = result[8]
assert cleanup is not no_update
assert isinstance(cleanup, str)
assert len(cleanup) >= 32 # uuid4 hex string
def test_prepare_table_data_with_external_data_does_not_use_cache(
monkeypatch, flask_app, sample_lff
):
"""When a caller passes data (acheteur/titulaire/observatoire path),
bypass the memoized helper entirely."""
from src.utils import table
sentinel = {"called": False}
def should_not_be_called(*a, **kw):
sentinel["called"] = True
raise AssertionError("Memoized helper must not be called when data is provided")
monkeypatch.setattr(table, "_fetch_page_sql", should_not_be_called)
with flask_app.app_context():
table.prepare_table_data(
data=sample_lff,
data_timestamp=0,
filter_query=None,
page_current=0,
page_size=20,
sort_by=[],
source_table="acheteur",
)
assert sentinel["called"] is False
def test_fetch_page_sql_respects_pagination(flask_app):
"""New path: returns (page_dff, total_count, total_unique) via DuckDB."""
from src.utils import table
with flask_app.app_context():
page, total, total_unique = table._fetch_page_sql(
filter_query=None, sort_by_key=(), page_current=0, page_size=5
)
assert page.height <= 5
assert total >= page.height
assert isinstance(total_unique, int)
def test_fetch_page_sql_applies_filter(flask_app):
from src.utils import table
with flask_app.app_context():
page, total, total_unique = table._fetch_page_sql(
filter_query="{uid} icontains __ne_matche_rien__",
sort_by_key=(),
page_current=0,
page_size=20,
)
assert total == 0
assert page.height == 0
def test_fetch_page_sql_post_processes_links(flask_app):
from src.utils import table
with flask_app.app_context():
page, _, _ = table._fetch_page_sql(
filter_query=None, sort_by_key=(), page_current=0, page_size=1
)
if page.height > 0:
assert "<a href" in page["uid"][0]
+140
View File
@@ -0,0 +1,140 @@
import polars as pl
SCHEMA = pl.Schema(
{
"uid": pl.String,
"objet": pl.String,
"acheteur_id": pl.String,
"montant": pl.Float64,
"dureeMois": pl.Int64,
"dateNotification": pl.Date,
}
)
def test_empty_filter_returns_true():
from src.utils.table_sql import filter_query_to_sql
where, params = filter_query_to_sql("", SCHEMA)
assert where == "TRUE"
assert params == []
def test_icontains_string_is_case_insensitive_like():
from src.utils.table_sql import filter_query_to_sql
where, params = filter_query_to_sql("{objet} icontains travaux", SCHEMA)
assert where == '"objet" IS NOT NULL AND "objet" <> \'\' AND "objet" ILIKE ?'
assert params == ["%travaux%"]
def test_icontains_with_trailing_wildcard_is_starts_with():
from src.utils.table_sql import filter_query_to_sql
where, params = filter_query_to_sql(
"{acheteur_id} icontains 24350013900189*", SCHEMA
)
assert (
where
== '"acheteur_id" IS NOT NULL AND "acheteur_id" <> \'\' AND "acheteur_id" ILIKE ?'
)
assert params == ["24350013900189%"]
def test_icontains_with_leading_wildcard_is_ends_with():
from src.utils.table_sql import filter_query_to_sql
where, params = filter_query_to_sql("{uid} icontains *2024", SCHEMA)
assert where == '"uid" IS NOT NULL AND "uid" <> \'\' AND "uid" ILIKE ?'
assert params == ["%2024"]
def test_numeric_greater_than():
from src.utils.table_sql import filter_query_to_sql
where, params = filter_query_to_sql("{montant} i> 40000", SCHEMA)
assert where == '"montant" IS NOT NULL AND "montant" > ?'
assert params == [40000.0]
def test_numeric_less_than():
from src.utils.table_sql import filter_query_to_sql
where, params = filter_query_to_sql("{montant} i< 1000", SCHEMA)
assert where == '"montant" IS NOT NULL AND "montant" < ?'
assert params == [1000.0]
def test_numeric_equality_via_icontains():
from src.utils.table_sql import filter_query_to_sql
where, params = filter_query_to_sql("{dureeMois} icontains 12", SCHEMA)
assert where == '"dureeMois" IS NOT NULL AND "dureeMois" = ?'
assert params == [12]
def test_date_column_treated_as_string_ilike():
from src.utils.table_sql import filter_query_to_sql
where, params = filter_query_to_sql("{dateNotification} icontains 2024*", SCHEMA)
assert "ILIKE" in where
assert params == ["2024%"]
def test_multiple_filters_joined_by_and():
from src.utils.table_sql import filter_query_to_sql
filter_query = "{objet} icontains voirie && {montant} i> 40000"
where, params = filter_query_to_sql(filter_query, SCHEMA)
assert " AND " in where
assert params == ["%voirie%", 40000.0]
def test_invalid_numeric_value_is_skipped():
from src.utils.table_sql import filter_query_to_sql
where, params = filter_query_to_sql("{montant} i> notanumber", SCHEMA)
assert where == "TRUE"
assert params == []
def test_unknown_column_is_skipped():
from src.utils.table_sql import filter_query_to_sql
where, params = filter_query_to_sql("{inexistant} icontains foo", SCHEMA)
assert where == "TRUE"
assert params == []
def test_sort_by_empty():
from src.utils.table_sql import sort_by_to_sql
assert sort_by_to_sql([], SCHEMA) == ""
assert sort_by_to_sql(None, SCHEMA) == ""
def test_sort_by_single_column_desc():
from src.utils.table_sql import sort_by_to_sql
result = sort_by_to_sql([{"column_id": "montant", "direction": "desc"}], SCHEMA)
assert result == '"montant" DESC NULLS LAST'
def test_sort_by_multiple_columns_preserves_order():
from src.utils.table_sql import sort_by_to_sql
result = sort_by_to_sql(
[
{"column_id": "dateNotification", "direction": "desc"},
{"column_id": "montant", "direction": "asc"},
],
SCHEMA,
)
assert result == '"dateNotification" DESC NULLS LAST, "montant" ASC NULLS LAST'
def test_sort_by_ignores_unknown_column():
from src.utils.table_sql import sort_by_to_sql
result = sort_by_to_sql([{"column_id": "fake", "direction": "asc"}], SCHEMA)
assert result == ""
Generated
+28 -10
View File
@@ -760,7 +760,7 @@ wheels = [
[[package]]
name = "decp-info"
version = "2.7.2"
version = "2.7.6"
source = { virtual = "." }
dependencies = [
{ name = "dash", extra = ["compress"] },
@@ -771,6 +771,7 @@ dependencies = [
{ name = "dash-leaflet", version = "1.1.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" },
{ name = "duckdb" },
{ name = "flask-caching" },
{ name = "flask-cors" },
{ name = "gunicorn" },
{ name = "httpx" },
{ name = "pandas", version = "2.3.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" },
@@ -783,7 +784,7 @@ dependencies = [
{ name = "xlsxwriter" },
]
[package.optional-dependencies]
[package.dev-dependencies]
dev = [
{ name = "dash", extra = ["testing"] },
{ name = "fastexcel" },
@@ -798,29 +799,33 @@ dev = [
requires-dist = [
{ name = "dash", specifier = "==3.4.0" },
{ name = "dash", extras = ["compress"] },
{ name = "dash", extras = ["testing"], marker = "extra == 'dev'" },
{ name = "dash-bootstrap-components" },
{ name = "dash-extensions" },
{ name = "dash-leaflet" },
{ name = "duckdb" },
{ name = "fastexcel", marker = "extra == 'dev'" },
{ name = "flask-caching" },
{ name = "flask-cors", specifier = ">=6.0.2" },
{ name = "gunicorn" },
{ name = "httpx" },
{ name = "pandas" },
{ name = "plotly", extras = ["express"] },
{ name = "polars" },
{ name = "pre-commit", marker = "extra == 'dev'" },
{ name = "pyarrow", specifier = ">=23.0.1" },
{ name = "pytest", marker = "extra == 'dev'" },
{ name = "pytest-env", marker = "extra == 'dev'" },
{ name = "python-dotenv" },
{ name = "selenium", marker = "extra == 'dev'" },
{ name = "unidecode" },
{ name = "webdriver-manager", marker = "extra == 'dev'" },
{ name = "xlsxwriter" },
]
provides-extras = ["dev"]
[package.metadata.requires-dev]
dev = [
{ name = "dash", extras = ["testing"] },
{ name = "fastexcel" },
{ name = "pre-commit" },
{ name = "pytest" },
{ name = "pytest-env" },
{ name = "selenium" },
{ name = "webdriver-manager" },
]
[[package]]
name = "dill"
@@ -975,6 +980,19 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/4c/0f/fe51e0b2301bbd429af44273a923ff92127b18d13abba5ae5a1d60e8e497/flask_compress-1.24-py3-none-any.whl", hash = "sha256:1e63668eb6e3242bd4f6ad98825a924e3984409be90c125477893d586007d00c", size = 11033 },
]
[[package]]
name = "flask-cors"
version = "6.0.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "flask" },
{ name = "werkzeug" },
]
sdist = { url = "https://files.pythonhosted.org/packages/70/74/0fc0fa68d62f21daef41017dafab19ef4b36551521260987eb3a5394c7ba/flask_cors-6.0.2.tar.gz", hash = "sha256:6e118f3698249ae33e429760db98ce032a8bf9913638d085ca0f4c5534ad2423", size = 13472 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/4f/af/72ad54402e599152de6d067324c46fe6a4f531c7c65baf7e96c63db55eaf/flask_cors-6.0.2-py3-none-any.whl", hash = "sha256:e57544d415dfd7da89a9564e1e3a9e515042df76e12130641ca6f3f2f03b699a", size = 13257 },
]
[[package]]
name = "geobuf"
version = "2.0.1"