diff --git a/CHANGELOG.md b/CHANGELOG.md index de1d145..ce56bf3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,9 @@ -#### 2.7.2 (19 avril 2026) +##### 2.7.3 (20 avril 2026) + +- Mise en cache des vues tableau par ensemble de filtres et de tris +- Résolution du bug d'écriture du fichier de vérouillage de la base de données + +##### 2.7.2 (19 avril 2026) - Chargement des données depuis une base DuckDB plutôt qu'en mémoire (plus de stabilité) ([#71](https://github.com/ColinMaudry/decp.info/issues/71)) - Mise en cache des vue sur l'observatoire pour un chargement plus rapide (remise à zéro quotidienne) @@ -6,7 +11,7 @@ - Quelques corrections de bugs d'affichage - Refactorisation des fonctions utilitaires (`utils.py` approchait des 1 000 lignes) -#### 2.7.1 (23 mars 2026) +##### 2.7.1 (23 mars 2026) - Correction du partage de données filtrées entre dashboard et vue des données diff --git a/CLAUDE.md b/CLAUDE.md index b5483c7..509783b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,8 +11,6 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ### Setup ```bash -python -m venv .venv && source .venv/bin/activate -pip install ".[dev]" cp template.env .env # then customize .env ``` @@ -25,14 +23,14 @@ uv run run.py # starts Dash with debug=True and hot reload ### Production ```bash -gunicorn app:server +uv run gunicorn app:server ``` ### Tests ```bash -uv run pytest # run all tests (Selenium-based integration tests) -uv run pytest tests/test_main.py::test_001_logo_and_search # run a single test +rtk uv run pytest # run all tests (Selenium-based integration tests) +rtk uv run pytest tests/test_main.py::test_001_logo_and_search # run a single test ``` Tests require a running Chrome/Chromium browser. They use `DashComposite` from `dash[testing]` with Selenium WebDriver. diff --git a/README.md b/README.md index 8ed961d..9a52e8f 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # decp.info -> v2.7.2 +> v2.7.3 > Outil d'exploration et de téléchargement des données essentielles de la commande publique. => [decp.info](https://decp.info) @@ -8,19 +8,15 @@ ## Installation et lancement ```shell -python -m venv .venv -source .venv/bin/activate -pip install . - # Copie et personnalisation du .env cp template.env .env nano .env # Pour la production -gunicorn app:server +uv run gunicorn app:server # Pour avoir le debuggage et le hot reload -python run.py +uv run run.py ``` ## Déploiement diff --git a/docs/superpowers/plans/2026-04-19-tableau-prepare-table-data-cache.md b/docs/superpowers/plans/2026-04-19-tableau-prepare-table-data-cache.md new file mode 100644 index 0000000..cc9d7ae --- /dev/null +++ b/docs/superpowers/plans/2026-04-19-tableau-prepare-table-data-cache.md @@ -0,0 +1,838 @@ +# Tableau prepare_table_data Cache Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make page navigation, sort changes, and repeated filter visits in the `/tableau` page near-instant by memoizing the expensive filter+sort+post-process pipeline inside `prepare_table_data`. + +**Architecture:** Extract a memoized inner function `_load_filter_sort_postprocess(filter_query, sort_by_key)` that performs the heavy work (load full data, filter, sort, collect, cast-to-string, fill-null, add HTML links, format values) and returns a fully post-processed Polars DataFrame. The outer `prepare_table_data` becomes a thin wrapper that handles non-deterministic side effects (`track_search`, `uuid.uuid4()` for cleanup trigger, `data_timestamp + 1`) and pagination. The memoized helper only runs when no `data` argument is passed (i.e., the Tableau path). Other callers (`acheteur`, `titulaire`, `observatoire`) keep the current uncached path because they pass an externally-provided LazyFrame that is not safely hashable for cache keys. + +**Tech Stack:** Polars (LazyFrame, DataFrame), Flask-Caching (`@cache.memoize()` on `FileSystemCache` already configured in `src/app.py:38`), pytest for unit tests. + +**Git**: the issue id is #72, add the reference in commit messages. + +--- + +## Background and constraints + +Read these before starting; they explain why the design takes the shape it does. + +1. **Cache infrastructure is already wired.** `src/cache.py` defines `cache = Cache()`. `src/app.py:38-48` initializes it with `FileSystemCache`, default 24h timeout, `CACHE_THRESHOLD=300`. The cache directory is wiped on every restart (`rmtree` at `src/app.py:36`), so cache always starts empty. + +2. **Existing pattern to mirror.** `src/pages/observatoire.py:650-660` already uses `@cache.memoize()` plus a `_normalize_filter_params` helper that converts a dict of filters into a hashable tuple. This plan applies the same idiom to `sort_by` (which is a `list[dict]` from Dash DataTable). + +3. **Non-deterministic outputs that MUST stay outside the memoized function:** + + - `data_timestamp + 1` (increments each call; would freeze if cached) + - `trigger_cleanup = str(uuid.uuid4())` (intentionally unique per call to fire the clientside filter-cleanup callback) + - `track_search(filter_query, source_table)` — Matomo HTTP POST, currently called inside `filter_table_data` at `src/utils/table.py:214`. Must fire on every user action including cache hits. + +4. **Tracking call site move.** `track_search` must move OUT of `filter_table_data` and into each caller, otherwise cache hits would silently skip Matomo tracking. Current callers of `filter_table_data` to update: + + - `src/utils/table.py:402` (inside `prepare_table_data`) + - `src/pages/tableau.py:325` (`download_data` callback) + - `src/pages/acheteur.py:427` (`download_data_acheteur` callback) + - `src/pages/titulaire.py:443` (`download_data_titulaire` callback) + +5. **Why Tableau-only caching.** `prepare_table_data` is also called from `acheteur.py`, `titulaire.py`, `observatoire.py`. Those callers pass a pre-filtered LazyFrame or list-of-dicts as `data`. Hashing arbitrary LazyFrames or large lists for memoization is impractical. The fix gates on `data is None` (the Tableau path) and leaves the other paths byte-for-byte identical. + +6. **Cache key composition.** The memoized function takes only `(filter_query, sort_by_key)`. `page_current` and `page_size` are intentionally NOT in the key — pagination happens in the outer wrapper after retrieving the cached, fully post-processed frame. This means every page click and page-size change is a cache hit (the whole point of the change). + +7. **Pickling.** Flask-Caching pickles arguments to form keys and pickles return values to disk. Polars `DataFrame` pickles cleanly. `LazyFrame` does not — so the memoized function must `.collect()` before returning. + +8. **File path expectations.** All paths below are relative to repo root `/home/colin/git/decp.info`. Run all commands from there. + +--- + +## File Structure + +- **Modify** `src/utils/table.py` — extract memoized helper, refactor `prepare_table_data`, remove `track_search` call from `filter_table_data`. +- **Modify** `src/pages/tableau.py` — add explicit `track_search` call in `download_data`. +- **Modify** `src/pages/acheteur.py` — add explicit `track_search` call in `download_data_acheteur`. +- **Modify** `src/pages/titulaire.py` — add explicit `track_search` call in `download_data_titulaire`. +- **Create** `tests/test_table.py` — unit tests for new helpers and refactored `prepare_table_data`. + +--- + +## Task 1: Set up unit tests for table.py + +**Files:** + +- Create: `tests/test_table.py` + +This task scaffolds a non-Selenium pytest module so subsequent tasks can do TDD without booting a Dash server. The conftest already writes a small `tests/test.parquet` fixture (see `tests/conftest.py:10`); reuse it. + +- [ ] **Step 1: Write the failing test** + +Create `tests/test_table.py` with: + +```python +import os + +import polars as pl +import pytest + + +@pytest.fixture +def sample_lff(): + """Small LazyFrame with the columns needed by add_links / format_values.""" + return pl.LazyFrame( + [ + { + "uid": "u1", + "id": "u1", + "acheteur_id": "12345678900011", + "acheteur_nom": "Mairie de Test", + "titulaire_id": "98765432100022", + "titulaire_nom": "Entreprise Test", + "titulaire_typeIdentifiant": "SIRET", + "objet": "Travaux divers", + "montant": 12500.0, + "dateNotification": "2025-03-15", + "codeCPV": "45000000", + "dureeRestanteMois": 6, + "titulaire_distance": 42.0, + } + ] + ) + + +def test_table_module_imports(): + from src.utils import table + + assert hasattr(table, "prepare_table_data") +``` + +- [ ] **Step 2: Run test to verify it passes (sanity check)** + +Run: `uv run pytest tests/test_table.py -v` +Expected: PASS for `test_table_module_imports`. (Selenium is not invoked because no `dash_duo` fixture is used.) + +- [ ] **Step 3: Commit** + +```bash +git add tests/test_table.py +git commit -m "test: scaffold unit tests for table utilities" +``` + +--- + +## Task 2: Move track_search out of filter_table_data + +**Files:** + +- Modify: `src/utils/table.py:210-274` (remove `track_search` import usage at line 214) +- Modify: `src/pages/tableau.py:317-334` (`download_data` callback) +- Modify: `src/pages/acheteur.py:425-430` area (`download_data_acheteur` callback) +- Modify: `src/pages/titulaire.py:441-446` area (`download_data_titulaire` callback) +- Modify: `tests/test_table.py` (add a test that confirms `filter_table_data` no longer calls Matomo) + +`track_search` must move out so that the soon-to-be-memoized helper does not swallow tracking on cache hits. We do this BEFORE introducing caching so that the diff is small and verifiable on its own. + +- [ ] **Step 1: Write the failing test** + +Append to `tests/test_table.py`: + +```python +def test_filter_table_data_does_not_call_track_search(monkeypatch, sample_lff): + from src.utils import table + + calls = [] + monkeypatch.setattr(table, "track_search", lambda *a, **kw: calls.append(a)) + + result = table.filter_table_data( + sample_lff, "{objet} icontains travaux", "tableau" + ).collect() + + assert calls == [] + assert result.height == 1 +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run pytest tests/test_table.py::test_filter_table_data_does_not_call_track_search -v` +Expected: FAIL (`assert calls == []` fails because `filter_table_data` currently calls `track_search` at line 214). + +- [ ] **Step 3: Remove the track_search call from filter_table_data** + +Edit `src/utils/table.py` — find this block: + +```python +def filter_table_data( + lff: pl.LazyFrame, filter_query: str, filter_source: str +) -> pl.LazyFrame: + _schema = lff.collect_schema() + track_search(filter_query, filter_source) + filtering_expressions = filter_query.split(" && ") +``` + +Remove the `track_search(filter_query, filter_source)` line. Result: + +```python +def filter_table_data( + lff: pl.LazyFrame, filter_query: str, filter_source: str +) -> pl.LazyFrame: + _schema = lff.collect_schema() + filtering_expressions = filter_query.split(" && ") +``` + +The `filter_source` parameter remains in the signature (avoids changing all callers in this task). It becomes unused; that is acceptable since callers will pass it again later if needed. Do NOT remove the `from src.utils.tracking import track_search` import yet — `prepare_table_data` will use it in Task 5. + +- [ ] **Step 4: Add explicit track_search calls in download callbacks** + +In `src/pages/tableau.py`, find: + +```python +def download_data(n_clicks, filter_query, sort_by, hidden_columns: list = None): + lff: pl.LazyFrame = query_marches().lazy() + + # Les colonnes masquées sont supprimées + if hidden_columns: + lff = lff.drop(hidden_columns) + + if filter_query: + lff = filter_table_data(lff, filter_query, "tab download") +``` + +Insert a `track_search` call so behavior is preserved. First add the import at the top of `src/pages/tableau.py` next to other `src.utils` imports: + +```python +from src.utils.tracking import track_search +``` + +Then change the body: + +```python +def download_data(n_clicks, filter_query, sort_by, hidden_columns: list = None): + lff: pl.LazyFrame = query_marches().lazy() + + # Les colonnes masquées sont supprimées + if hidden_columns: + lff = lff.drop(hidden_columns) + + if filter_query: + track_search(filter_query, "tab download") + lff = filter_table_data(lff, filter_query, "tab download") +``` + +Repeat the same pattern in `src/pages/acheteur.py` (search for `filter_table_data(lff, filter_query, "ach download")`): + +Add import: + +```python +from src.utils.tracking import track_search +``` + +Wrap the call: + +```python + if filter_query: + track_search(filter_query, "ach download") + lff = filter_table_data(lff, filter_query, "ach download") +``` + +Repeat in `src/pages/titulaire.py` (search for `filter_table_data(lff, filter_query, "titu download")`): + +Add import: + +```python +from src.utils.tracking import track_search +``` + +Wrap the call: + +```python + if filter_query: + track_search(filter_query, "titu download") + lff = filter_table_data(lff, filter_query, "titu download") +``` + +- [ ] **Step 5: Run test to verify it passes** + +Run: `uv run pytest tests/test_table.py::test_filter_table_data_does_not_call_track_search -v` +Expected: PASS. + +- [ ] **Step 6: Run full unit test file to verify no regressions** + +Run: `uv run pytest tests/test_table.py -v` +Expected: All tests in `test_table.py` PASS. + +- [ ] **Step 7: Commit** + +```bash +git add src/utils/table.py src/pages/tableau.py src/pages/acheteur.py src/pages/titulaire.py tests/test_table.py +git commit -m "refactor: move track_search out of filter_table_data into callers" +``` + +--- + +## Task 3: Add normalize_sort_by helper + +**Files:** + +- Modify: `src/utils/table.py` (add helper near other utility functions, e.g. after `dates_to_strings`) +- Modify: `tests/test_table.py` (add tests) + +A cache key must be hashable. Dash DataTable's `sort_by` is a `list[dict]` like `[{"column_id": "montant", "direction": "asc"}, ...]`, which is not hashable. We mirror the `_normalize_filter_params` idiom from `src/pages/observatoire.py:650-657`. + +- [ ] **Step 1: Write the failing tests** + +Append to `tests/test_table.py`: + +```python +def test_normalize_sort_by_handles_empty(): + from src.utils.table import normalize_sort_by + + assert normalize_sort_by(None) == () + assert normalize_sort_by([]) == () + + +def test_normalize_sort_by_returns_hashable_tuple(): + from src.utils.table import normalize_sort_by + + sort_by = [ + {"column_id": "montant", "direction": "desc"}, + {"column_id": "dateNotification", "direction": "asc"}, + ] + key = normalize_sort_by(sort_by) + + assert key == (("montant", "desc"), ("dateNotification", "asc")) + # Must be hashable so that flask-caching can build a cache key from it + hash(key) + + +def test_normalize_sort_by_preserves_order(): + """Order matters for sort: [A, B] != [B, A].""" + from src.utils.table import normalize_sort_by + + a_then_b = normalize_sort_by( + [{"column_id": "a", "direction": "asc"}, {"column_id": "b", "direction": "asc"}] + ) + b_then_a = normalize_sort_by( + [{"column_id": "b", "direction": "asc"}, {"column_id": "a", "direction": "asc"}] + ) + assert a_then_b != b_then_a +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `uv run pytest tests/test_table.py -v -k normalize_sort_by` +Expected: FAIL with `ImportError` for `normalize_sort_by`. + +- [ ] **Step 3: Implement normalize_sort_by** + +Edit `src/utils/table.py`. Add this function immediately after the `dates_to_strings` function (around line 148): + +```python +def normalize_sort_by(sort_by) -> tuple: + """Convert Dash DataTable sort_by (list[dict]) into a hashable tuple + suitable for use as a cache key. Order is preserved because it determines + sort precedence.""" + if not sort_by: + return () + return tuple((entry["column_id"], entry["direction"]) for entry in sort_by) +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `uv run pytest tests/test_table.py -v -k normalize_sort_by` +Expected: 3 PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/utils/table.py tests/test_table.py +git commit -m "feat: add normalize_sort_by hashable cache-key helper" +``` + +--- + +## Task 4: Extract memoized post-process helper + +**Files:** + +- Modify: `src/utils/table.py` (add `_load_filter_sort_postprocess`, decorate with `@cache.memoize()`, import `cache`) +- Modify: `tests/test_table.py` (add tests) + +Introduce the function whose result will live in the FileSystemCache. Inputs: `(filter_query, sort_by_key)`. Output: a fully post-processed, unpaginated Polars DataFrame ready to slice and convert to dicts. + +This task does NOT yet wire the helper into `prepare_table_data` — that happens in Task 5. Splitting these tasks keeps each diff small and testable. + +- [ ] **Step 1: Write the failing tests** + +Append to `tests/test_table.py`: + +```python +@pytest.fixture(autouse=True) +def reset_cache(): + """Ensure the flask-caching backend is empty between tests so that + cache-hit assertions are meaningful. Falls back to no-op when no + Flask app context is active (NullCache).""" + from utils.cache import cache + + try: + cache.clear() + except RuntimeError: + # No app context — cache is NullCache, nothing to clear + pass + yield + + +def test_load_filter_sort_postprocess_returns_dataframe(monkeypatch, sample_lff): + from src.utils import table + + monkeypatch.setattr( + table, "query_marches", lambda: sample_lff.collect() + ) + + df = table._load_filter_sort_postprocess(filter_query=None, sort_by_key=()) + + assert isinstance(df, pl.DataFrame) + assert df.height == 1 + # All values must be strings after post-processing + for col in df.columns: + assert df.schema[col] == pl.String + + +def test_load_filter_sort_postprocess_applies_filter(monkeypatch, sample_lff): + from src.utils import table + + monkeypatch.setattr( + table, "query_marches", lambda: sample_lff.collect() + ) + + df = table._load_filter_sort_postprocess( + filter_query="{objet} icontains travaux", sort_by_key=() + ) + assert df.height == 1 + + df_empty = table._load_filter_sort_postprocess( + filter_query="{objet} icontains nonexistent", sort_by_key=() + ) + assert df_empty.height == 0 + + +def test_load_filter_sort_postprocess_adds_links(monkeypatch, sample_lff): + from src.utils import table + + monkeypatch.setattr( + table, "query_marches", lambda: sample_lff.collect() + ) + + df = table._load_filter_sort_postprocess(filter_query=None, sort_by_key=()) + # add_links injects an wrapper around uid, acheteur_nom, titulaire_nom + assert " 0: + dff = format_values(dff) + + return dff +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `uv run pytest tests/test_table.py -v -k load_filter_sort_postprocess` +Expected: 3 PASS. + +- [ ] **Step 5: Run the full test_table.py to catch regressions** + +Run: `uv run pytest tests/test_table.py -v` +Expected: All PASS. + +- [ ] **Step 6: Commit** + +```bash +git add src/utils/table.py tests/test_table.py +git commit -m "feat: add memoized _load_filter_sort_postprocess helper" +``` + +--- + +## Task 5: Wire the memoized helper into prepare_table_data + +**Files:** + +- Modify: `src/utils/table.py` — replace the body of `prepare_table_data` so the Tableau path uses the cache +- Modify: `tests/test_table.py` — add tests covering the new flow + +The outer function keeps its signature unchanged so callers in `acheteur.py`, `titulaire.py`, `observatoire.py`, `tableau.py` need no updates. When `data is None` (the Tableau case), use the memoized helper; otherwise fall through to the original logic. + +- [ ] **Step 1: Write the failing tests** + +Append to `tests/test_table.py`: + +```python +def test_prepare_table_data_returns_expected_tuple(monkeypatch, sample_lff): + from src.utils import table + + monkeypatch.setattr( + table, "query_marches", lambda: sample_lff.collect() + ) + + result = table.prepare_table_data( + data=None, + data_timestamp=5, + filter_query=None, + page_current=0, + page_size=20, + sort_by=[], + source_table="tableau", + ) + + # Same arity as before: 9 outputs + assert len(result) == 9 + dicts, columns, tooltip, ts, nb_rows, dl_disabled, dl_text, dl_title, cleanup = result + assert isinstance(dicts, list) + assert ts == 6 # data_timestamp + 1 must still increment + assert "1 lignes" in nb_rows + + +def test_prepare_table_data_calls_track_search_on_filter(monkeypatch, sample_lff): + from src.utils import table + + calls = [] + monkeypatch.setattr( + table, "query_marches", lambda: sample_lff.collect() + ) + monkeypatch.setattr(table, "track_search", lambda *a, **kw: calls.append(a)) + + table.prepare_table_data( + data=None, + data_timestamp=0, + filter_query="{objet} icontains travaux", + page_current=0, + page_size=20, + sort_by=[], + source_table="tableau", + ) + + assert calls == [("{objet} icontains travaux", "tableau")] + + +def test_prepare_table_data_paginates_without_recomputing(monkeypatch, sample_lff): + """Two calls with same filter+sort but different pages must invoke + the inner heavy work only once.""" + from src.utils import table + + call_count = {"n": 0} + real_query = sample_lff.collect() + + def counting_query(): + call_count["n"] += 1 + return real_query + + monkeypatch.setattr(table, "query_marches", counting_query) + + # First call: cache miss + table.prepare_table_data( + data=None, + data_timestamp=0, + filter_query=None, + page_current=0, + page_size=10, + sort_by=[], + source_table="tableau", + ) + first_count = call_count["n"] + + # Second call, different page: cache hit, query_marches must NOT fire again + table.prepare_table_data( + data=None, + data_timestamp=0, + filter_query=None, + page_current=1, + page_size=10, + sort_by=[], + source_table="tableau", + ) + + assert call_count["n"] == first_count, ( + "query_marches was called again — pagination triggered cache miss" + ) + + +def test_prepare_table_data_cleanup_trigger_for_non_tableau(monkeypatch, sample_lff): + """Non-tableau pages still get a fresh uuid trigger, not no_update.""" + from dash import no_update + + from src.utils import table + + monkeypatch.setattr( + table, "query_marches", lambda: sample_lff.collect() + ) + + result = table.prepare_table_data( + data=None, + data_timestamp=0, + filter_query="{objet} icontains travaux", + page_current=0, + page_size=20, + sort_by=[], + source_table="acheteur", + ) + + cleanup = result[8] + assert cleanup is not no_update + assert isinstance(cleanup, str) + assert len(cleanup) >= 32 # uuid4 hex string + + +def test_prepare_table_data_with_external_data_does_not_use_cache( + monkeypatch, sample_lff +): + """When a caller passes data (acheteur/titulaire/observatoire path), + bypass the memoized helper entirely.""" + from src.utils import table + + sentinel = {"called": False} + + def should_not_be_called(*a, **kw): + sentinel["called"] = True + raise AssertionError("Memoized helper must not be called when data is provided") + + monkeypatch.setattr( + table, "_load_filter_sort_postprocess", should_not_be_called + ) + + table.prepare_table_data( + data=sample_lff, # external LazyFrame + data_timestamp=0, + filter_query=None, + page_current=0, + page_size=20, + sort_by=[], + source_table="acheteur", + ) + + assert sentinel["called"] is False +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `uv run pytest tests/test_table.py -v -k prepare_table_data` +Expected: At least the cache-hit (`paginates_without_recomputing`) and `track_search`-routing tests FAIL because the current `prepare_table_data` re-runs the full pipeline on every call and routes tracking through `filter_table_data` (which Task 2 already neutralized — so tracking would be lost without the new explicit call). + +- [ ] **Step 3: Refactor prepare_table_data** + +Edit `src/utils/table.py`. Replace the entire `prepare_table_data` function body with: + +```python +def prepare_table_data( + data, data_timestamp, filter_query, page_current, page_size, sort_by, source_table +): + """ + Préparation des données pour les datatables. + + Pour la page Tableau (data is None), le calcul lourd (chargement complet, + filtre, tri, post-traitement) est mémorisé via _load_filter_sort_postprocess. + Les changements de page deviennent ainsi des cache hits. + + Pour les autres pages (data fourni), le chemin original est conservé : la + LazyFrame externe n'est pas hashable et le coût de filtre/tri y est déjà + minime puisque les données sont pré-restreintes. + """ + logger.debug(" + + + + + + + + + + + + + + + + + + ") + + # Side effect non-cacheable : le tracking doit firer sur chaque action + # utilisateur, y compris sur cache hit. + if filter_query: + track_search(filter_query, source_table) + + # Trigger uuid pour les pages autres que tableau (clientside cleanup) + trigger_cleanup = ( + no_update if source_table == "tableau" else str(uuid.uuid4()) + ) + + if data is None: + # Tableau path : utilise le cache + sort_by_key = normalize_sort_by(sort_by) + dff: pl.DataFrame = _load_filter_sort_postprocess( + filter_query=filter_query, sort_by_key=sort_by_key + ) + else: + # acheteur / titulaire / observatoire path : code original, non caché + if isinstance(data, list): + lff: pl.LazyFrame = pl.LazyFrame( + data, strict=False, infer_schema_length=5000 + ) + elif isinstance(data, pl.LazyFrame): + lff = data + else: + lff = query_marches().lazy() + + if filter_query: + lff = filter_table_data(lff, filter_query, source_table) + + if sort_by and len(sort_by) > 0: + lff = sort_table_data(lff, sort_by) + + dff = lff.collect() + dff = dff.cast(pl.String) + dff = dff.fill_null("") + dff = add_links(dff) + if "sourceFile" in dff.columns: + dff = add_resource_link(dff) + if dff.height > 0: + dff = format_values(dff) + + height = dff.height + + if height > 0: + nb_rows = ( + f"{format_number(height)} lignes " + f"({format_number(dff.select('uid').unique().height)} marchés)" + ) + else: + nb_rows = "0 lignes (0 marchés)" + + # Pagination — toujours hors cache pour rester sur des cache hits + start_row = page_current * page_size + dff = dff.slice(start_row, page_size) + + table_columns, tooltip = setup_table_columns(dff) + + dicts = dff.to_dicts() + + download_disabled, download_text, download_title = get_button_properties(height) + + return ( + dicts, + table_columns, + tooltip, + data_timestamp + 1, + nb_rows, + download_disabled, + download_text, + download_title, + trigger_cleanup, + ) +``` + +Notes on what changed vs the original at `src/utils/table.py:372-458`: + +- `track_search` now called explicitly at the top, on every invocation (not via `filter_table_data`). +- `data is None` branch delegates the heavy work to the memoized helper. +- `data is not None` branch is functionally identical to the original (pagination still happens after collect+post-process). +- The post-processing (`cast`, `fill_null`, `add_links`, `add_resource_link`, `format_values`) is now done in BOTH branches before `nb_rows` calculation. In the cached branch this was already done inside `_load_filter_sort_postprocess`; in the uncached branch we keep doing it inline. This means `nb_rows` and `dff.select('uid').unique().height` operate on the post-processed frame in both branches, matching the original semantics. + +- [ ] **Step 4: Run all unit tests** + +Run: `uv run pytest tests/test_table.py -v` +Expected: All PASS, including `test_prepare_table_data_paginates_without_recomputing`. + +- [ ] **Step 5: Run the full repo test suite to catch regressions** + +Run: `uv run pytest -v` +Expected: All PASS. Selenium tests (`tests/test_main.py`) require Chrome/Chromium; if the executor lacks a browser, those tests will error/skip — note the failures and rerun in an environment with Chrome before declaring done. + +- [ ] **Step 6: Commit** + +```bash +git add src/utils/table.py tests/test_table.py +git commit -m "perf(tableau): memoize filter+sort+postprocess pipeline" +``` + +--- + +## Task 6: Manual smoke test in the browser + +**Files:** none modified. + +Type checks and unit tests cannot validate that page navigation actually feels faster. This task is explicitly a hands-on verification. + +- [ ] **Step 1: Start the dev server** + +Run: `uv run run.py` +Wait for `Dash is running on http://...`. + +- [ ] **Step 2: Open the Tableau page and warm the cache** + +1. Open `http://localhost:8050/tableau` (or whatever port the dev server prints). +2. With no filter applied, wait for the first page to load fully. This is the cold-cache load (slow expected). +3. Open the browser devtools Network panel. + +- [ ] **Step 3: Verify pagination is fast** + +1. Click "page 2" / "page 3" / "page 4" in the table footer in quick succession. +2. Each navigation should return data in well under 1 second (in the original code each took several seconds). +3. In the dev server logs, look for the line `Cache miss — recomputing for filter=...` from `_load_filter_sort_postprocess`. It should appear ONCE for the initial load and NOT appear again as you change pages. + +- [ ] **Step 4: Verify a new filter triggers exactly one cache miss** + +1. In the table, type a filter into one of the columns (e.g. `paris` in `acheteur_commune_nom`) and press Enter. +2. The dev log should show ONE new `Cache miss — recomputing` line. +3. Change page within the filtered view — no new cache miss line should appear. + +- [ ] **Step 5: Verify filter cleanup trigger still fires** + +1. Open `http://localhost:8050/acheteur?id=` (use any valid id from the dataset). +2. Apply a filter on the embedded table. +3. The clientside callback for filter cleanup (`src/assets/dash_clientside.js` `clean_filters`) should still rewrite the filter operators (e.g. `contains` → `icontains`). If it doesn't fire, the `trigger_cleanup` uuid is broken — investigate. + +- [ ] **Step 6: Verify download still works** + +1. On the Tableau page, click "Télécharger au format Excel" (the button must be enabled — apply a filter that brings the row count under 65,000). +2. The downloaded XLSX must open and contain the filtered rows. + +- [ ] **Step 7: Stop the dev server** + +Ctrl-C. + +- [ ] **Step 8: If all checks pass, this completes the implementation** + +No commit — this task is verification only. Report results to the user. diff --git a/pyproject.toml b/pyproject.toml index 3fbce4e..4c8748c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,7 +1,7 @@ [project] name = "decp.info" description = "Interface d'exploration et d'analyse des marchés publics français." -version = "2.7.2" +version = "2.7.3" requires-python = ">= 3.10" authors = [{ name = "Colin Maudry", email = "colin@colmo.tech" }] dependencies = [ @@ -23,7 +23,7 @@ dependencies = [ "pyarrow>=23.0.1", ] -[project.optional-dependencies] +[dependency-groups] dev = [ "pytest", "pytest-env", diff --git a/src/app.py b/src/app.py index acbba3a..d5ddb6f 100644 --- a/src/app.py +++ b/src/app.py @@ -7,8 +7,8 @@ from dash import Dash, Input, Output, State, dcc, html, page_container, page_reg from dotenv import load_dotenv from flask import Response -from src.cache import cache from src.utils import DEVELOPMENT +from src.utils.cache import cache load_dotenv() diff --git a/src/db.py b/src/db.py index 646fc25..9d96872 100644 --- a/src/db.py +++ b/src/db.py @@ -109,15 +109,8 @@ def build_database(db_path: Path, parquet_path: Path) -> None: logger.info(f"Base DuckDB construite : {db_path}") -def _resolve_db_path() -> Path: - parquet = os.getenv("DATA_FILE_PARQUET_PATH") - if not parquet: - raise RuntimeError("DATA_FILE_PARQUET_PATH is not set") - return Path(parquet).parent / "decp.duckdb" - - def _ensure_database() -> Path: - db_path = _resolve_db_path() + db_path = Path("./decp.duckdb") parquet_path = Path(os.getenv("DATA_FILE_PARQUET_PATH")) lock_path = db_path.with_suffix(".duckdb.lock") diff --git a/src/pages/acheteur.py b/src/pages/acheteur.py index 58e61db..e0e708a 100644 --- a/src/pages/acheteur.py +++ b/src/pages/acheteur.py @@ -35,6 +35,7 @@ from src.utils.table import ( prepare_table_data, sort_table_data, ) +from src.utils.tracking import track_search def get_title(acheteur_id: str | None = None) -> str: @@ -424,7 +425,8 @@ def download_filtered_acheteur_data( lff = lff.drop(hidden_columns) if filter_query: - lff = filter_table_data(lff, filter_query, "ach download") + track_search(filter_query, "ach download") + lff = filter_table_data(lff, filter_query) if len(sort_by) > 0: lff = sort_table_data(lff, sort_by) diff --git a/src/pages/observatoire.py b/src/pages/observatoire.py index b3afc84..2391a2a 100644 --- a/src/pages/observatoire.py +++ b/src/pages/observatoire.py @@ -16,7 +16,6 @@ from dash import ( register_page, ) -from src.cache import cache from src.db import query_marches, schema from src.figures import ( DataTable, @@ -31,6 +30,7 @@ from src.figures import ( make_donut, ) from src.utils import logger +from src.utils.cache import cache from src.utils.data import ( DEPARTEMENTS, DF_ACHETEURS, @@ -658,9 +658,11 @@ def _normalize_filter_params(filter_params: dict) -> tuple: @cache.memoize() -def _compute_dashboard_children(cache_key: tuple): +def _compute_dashboard_children(filter_params_normalized: tuple): logger.debug("Cache miss — computing dashboard") - filter_params = {k: (list(v) if isinstance(v, tuple) else v) for k, v in cache_key} + filter_params = { + k: (list(v) if isinstance(v, tuple) else v) for k, v in filter_params_normalized + } lff: pl.LazyFrame = query_marches().lazy() lff = prepare_dashboard_data(lff=lff, **filter_params) @@ -772,8 +774,8 @@ def update_dashboard_cards(*filter_values): ): filter_params[input_id] = value - cache_key = _normalize_filter_params(filter_params) - children = _compute_dashboard_children(cache_key) + filter_params_normalized = _normalize_filter_params(filter_params) + children = _compute_dashboard_children(filter_params_normalized) return dbc.Row(children=children), filter_params diff --git a/src/pages/tableau.py b/src/pages/tableau.py index 723db62..63d9927 100644 --- a/src/pages/tableau.py +++ b/src/pages/tableau.py @@ -31,6 +31,7 @@ from src.utils.table import ( prepare_table_data, sort_table_data, ) +from src.utils.tracking import track_search update_date_timestamp = os.path.getmtime(os.getenv("DATA_FILE_PARQUET_PATH")) update_date = datetime.fromtimestamp(update_date_timestamp).strftime("%d/%m/%Y") @@ -314,7 +315,7 @@ def update_table(href, page_current, page_size, filter_query, sort_by, data_time State("tableau_datatable", "hidden_columns"), prevent_initial_call=True, ) -def download_data(n_clicks, filter_query, sort_by, hidden_columns: list = None): +def download_data(n_clicks, filter_query, sort_by, hidden_columns: list | None = None): lff: pl.LazyFrame = query_marches().lazy() # Les colonnes masquées sont supprimées @@ -322,7 +323,8 @@ def download_data(n_clicks, filter_query, sort_by, hidden_columns: list = None): lff = lff.drop(hidden_columns) if filter_query: - lff = filter_table_data(lff, filter_query, "tab download") + track_search(filter_query, "tab download") + lff = filter_table_data(lff, filter_query) if sort_by and len(sort_by) > 0: lff = sort_table_data(lff, sort_by) diff --git a/src/pages/titulaire.py b/src/pages/titulaire.py index 3347c19..2a1005c 100644 --- a/src/pages/titulaire.py +++ b/src/pages/titulaire.py @@ -34,6 +34,7 @@ from src.utils.table import ( prepare_table_data, sort_table_data, ) +from src.utils.tracking import track_search def get_title(titulaire_id: str = None) -> str: @@ -429,7 +430,12 @@ def download_titulaire_data( prevent_initial_call=True, ) def download_filtered_titulaire_data( - data, n_clicks, titulaire_nom, filter_query, sort_by, hidden_columns: list = None + data, + n_clicks, + titulaire_nom, + filter_query, + sort_by, + hidden_columns: list | None = None, ): lff: pl.LazyFrame = pl.LazyFrame( data @@ -440,7 +446,8 @@ def download_filtered_titulaire_data( lff = lff.drop(hidden_columns) if filter_query: - lff = filter_table_data(lff, filter_query, "titu download") + track_search(filter_query, "titu download") + lff = filter_table_data(lff, filter_query) if len(sort_by) > 0: lff = sort_table_data(lff, sort_by) diff --git a/src/cache.py b/src/utils/cache.py similarity index 100% rename from src/cache.py rename to src/utils/cache.py diff --git a/src/utils/table.py b/src/utils/table.py index a0814f7..096acf5 100644 --- a/src/utils/table.py +++ b/src/utils/table.py @@ -7,6 +7,7 @@ from polars import selectors as cs from src.db import query_marches, schema from src.utils import logger +from src.utils.cache import cache from src.utils.data import DATA_SCHEMA from src.utils.frontend import get_button_properties from src.utils.tracking import track_search @@ -146,6 +147,12 @@ def dates_to_strings(lff: pl.LazyFrame, column: str) -> pl.LazyFrame: return lff +def normalize_sort_by(sort_by) -> tuple: + if not sort_by: + return () + return tuple((entry["column_id"], entry["direction"]) for entry in sort_by) + + def format_number(number) -> str: number = "{:,}".format(number).replace(",", " ") return number @@ -160,7 +167,7 @@ def unformat_montant(number: str) -> float: def format_values(dff: pl.DataFrame) -> pl.DataFrame: - def format_montant(expr, scale=None): + def format_montant(expr): # https://stackoverflow.com/a/78636786 expr = expr.cast(pl.String) expr = expr.str.splitn(".", 2) @@ -206,14 +213,13 @@ def format_values(dff: pl.DataFrame) -> pl.DataFrame: return dff -def filter_table_data( - lff: pl.LazyFrame, filter_query: str, filter_source: str -) -> pl.LazyFrame: +def filter_table_data(lff: pl.LazyFrame, filter_query: str) -> pl.LazyFrame: _schema = lff.collect_schema() - track_search(filter_query, filter_source) filtering_expressions = filter_query.split(" && ") for filter_part in filtering_expressions: col_name, operator, filter_value = split_filter_part(filter_part) + if not isinstance(col_name, str) or not isinstance(filter_value, str): + continue col_type = str(_schema[col_name]) # logger.debug("filter_value:", filter_value) # logger.debug("filter_value_type:", type(filter_value)) @@ -246,7 +252,7 @@ def filter_table_data( elif operator == "<=": lff = lff.filter(pl.col(col_name) <= filter_value) elif operator == "contains": - if col_type in ["String", "Date"]: + if col_type in ["String", "Date"] and isinstance(filter_value, str): filter_value = filter_value.strip('"') if filter_value.endswith("*"): lff = lff.filter( @@ -284,7 +290,9 @@ def sort_table_data(lff: pl.LazyFrame, sort_by: list) -> pl.LazyFrame: def setup_table_columns( - dff, hideable: bool = True, exclude: list = None, new_columns: list = None + dff, + hideable: bool = True, + exclude: list | None = None, ) -> tuple: # Liste finale de colonnes markdown_exceptions = ["montant", "titulaire_distance", "distance", "dureeMois"] @@ -368,6 +376,40 @@ def get_default_hidden_columns(page): return hidden_columns +@cache.memoize() +def _load_filter_sort_postprocess(filter_query, sort_by_key): + logger.debug( + f"Cache miss — recomputing for filter={filter_query!r} sort={sort_by_key!r}" + ) + + lff: pl.LazyFrame = query_marches().lazy() + + if filter_query: + lff = filter_table_data(lff, filter_query) + + if sort_by_key: + sort_by = [ + {"column_id": col, "direction": direction} for col, direction in sort_by_key + ] + lff = sort_table_data(lff, sort_by) + + dff = table_postprocess(lff) + + return dff + + +def table_postprocess(lff) -> pl.DataFrame: + lff = lff.cast(pl.String) + lff = lff.fill_null("") + dff: pl.DataFrame = lff.collect() + dff = add_links(dff) + if "sourceFile" in dff.columns: + dff = add_resource_link(dff) + if dff.height > 0: + dff = format_values(dff) + return dff + + def prepare_table_data( data, data_timestamp, filter_query, page_current, page_size, sort_by, source_table ): @@ -383,66 +425,53 @@ def prepare_table_data( :param source_table: :return: """ + logger.debug(" + + + + + + + + + + + + + + + + + + ") - if os.getenv("DEVELOPMENT").lower() == "true": - logger.debug(" + + + + + + + + + + + + + + + + + + ") - - trigger_cleanup = no_update - - # Récupération des données - if isinstance(data, list): - lff: pl.LazyFrame = pl.LazyFrame(data, strict=False, infer_schema_length=5000) - elif isinstance(data, pl.LazyFrame): - lff = data - else: - lff: pl.LazyFrame = query_marches().lazy() - - # Application des filtres if filter_query: - lff = filter_table_data(lff, filter_query, source_table) - trigger_cleanup = no_update if source_table == "tableau" else str(uuid.uuid4()) + track_search(filter_query, source_table) - # Application des tris - if sort_by and len(sort_by) > 0: - lff = sort_table_data(lff, sort_by) + trigger_cleanup = no_update if source_table == "tableau" else str(uuid.uuid4()) + + if data is None: + sort_by_key = normalize_sort_by(sort_by) + dff: pl.DataFrame = _load_filter_sort_postprocess( + filter_query=filter_query, sort_by_key=sort_by_key + ) + else: + if isinstance(data, list): + lff: pl.LazyFrame = pl.LazyFrame( + data, strict=False, infer_schema_length=5000 + ) + elif isinstance(data, pl.LazyFrame): + lff = data + else: + lff = query_marches().lazy() + + if filter_query: + lff = filter_table_data(lff, filter_query) + + if sort_by and len(sort_by) > 0: + lff = sort_table_data(lff, sort_by) + + dff: pl.DataFrame = table_postprocess(lff) - # Matérialisation des filtres - dff: pl.DataFrame = lff.collect() height = dff.height if height > 0: - nb_rows = f"{format_number(height)} lignes ({format_number(dff.select('uid').unique().height)} marchés)" + nb_rows = ( + f"{format_number(height)} lignes " + f"({format_number(dff.select('uid').unique().height)} marchés)" + ) else: nb_rows = "0 lignes (0 marchés)" - # Pagination des données start_row = page_current * page_size - # end_row = (page_current + 1) * page_size dff = dff.slice(start_row, page_size) - # Tout devient string - dff = dff.cast(pl.String) - - # Remplace les strings null par "", mais pas les numeric null - dff = dff.fill_null("") - - # Ajout des liens vers les pages de détails - dff = add_links(dff) - - # Ajout des liens vers les fichiers Open Data - if "sourceFile" in dff.columns: - dff = add_resource_link(dff) - - # Formatage des montants - if height > 0: - dff = format_values(dff) - - # Récupération des colonnes et tooltip table_columns, tooltip = setup_table_columns(dff) dicts = dff.to_dicts() - # Propriétés du bouton de téléchargement download_disabled, download_text, download_title = get_button_properties(height) return ( diff --git a/tests/test_table.py b/tests/test_table.py new file mode 100644 index 0000000..cc89386 --- /dev/null +++ b/tests/test_table.py @@ -0,0 +1,305 @@ +import polars as pl +import pytest + + +@pytest.fixture +def sample_lff(): + """Small LazyFrame with the columns needed by add_links / format_values.""" + return pl.LazyFrame( + [ + { + "uid": "u1", + "id": "u1", + "acheteur_id": "12345678900011", + "acheteur_nom": "Mairie de Test", + "titulaire_id": "98765432100022", + "titulaire_nom": "Entreprise Test", + "titulaire_typeIdentifiant": "SIRET", + "objet": "Travaux divers", + "montant": 12500.0, + "dateNotification": "2025-03-15", + "codeCPV": "45000000", + "dureeRestanteMois": 6, + "titulaire_distance": 42.0, + } + ] + ) + + +def test_table_module_imports(): + from src.utils import table + + assert hasattr(table, "prepare_table_data") + + +def test_filter_table_data_does_not_call_track_search(monkeypatch, sample_lff): + from src.utils import table + + calls = [] + monkeypatch.setattr(table, "track_search", lambda *a, **kw: calls.append(a)) + + result = table.filter_table_data(sample_lff, "{objet} icontains travaux").collect() + + assert calls == [] + assert result.height == 1 + + +def test_normalize_sort_by_handles_empty(): + from src.utils.table import normalize_sort_by + + assert normalize_sort_by(None) == () + assert normalize_sort_by([]) == () + + +def test_normalize_sort_by_returns_hashable_tuple(): + from src.utils.table import normalize_sort_by + + sort_by = [ + {"column_id": "montant", "direction": "desc"}, + {"column_id": "dateNotification", "direction": "asc"}, + ] + key = normalize_sort_by(sort_by) + + assert key == (("montant", "desc"), ("dateNotification", "asc")) + # Must be hashable so that flask-caching can build a cache key from it + hash(key) + + +def test_normalize_sort_by_preserves_order(): + """Order matters for sort: [A, B] != [B, A].""" + from src.utils.table import normalize_sort_by + + a_then_b = normalize_sort_by( + [{"column_id": "a", "direction": "asc"}, {"column_id": "b", "direction": "asc"}] + ) + b_then_a = normalize_sort_by( + [{"column_id": "b", "direction": "asc"}, {"column_id": "a", "direction": "asc"}] + ) + assert a_then_b != b_then_a + + +@pytest.fixture(scope="module") +def flask_app(): + """Minimal Flask app with SimpleCache so @cache.memoize() works in tests.""" + from flask import Flask + + from utils.cache import cache + + app = Flask(__name__) + cache.init_app(app, config={"CACHE_TYPE": "SimpleCache"}) + return app + + +@pytest.fixture(autouse=True) +def reset_cache(flask_app): + """Ensure the flask-caching backend is empty between tests so that + cache-hit assertions are meaningful. Falls back to no-op when no + Flask app context is active (NullCache).""" + from utils.cache import cache + + with flask_app.app_context(): + try: + cache.clear() + except (RuntimeError, AttributeError): + # No app context — cache is NullCache, nothing to clear + pass + yield + + +def test_load_filter_sort_postprocess_returns_dataframe( + flask_app, monkeypatch, sample_lff +): + from src.utils import table + + monkeypatch.setattr(table, "query_marches", lambda: sample_lff.collect()) + + with flask_app.app_context(): + df = table._load_filter_sort_postprocess(filter_query=None, sort_by_key=()) + + assert isinstance(df, pl.DataFrame) + assert df.height == 1 + # All values must be strings after post-processing + for col in df.columns: + assert df.schema[col] == pl.String + + +def test_load_filter_sort_postprocess_applies_filter( + flask_app, monkeypatch, sample_lff +): + from src.utils import table + + monkeypatch.setattr(table, "query_marches", lambda: sample_lff.collect()) + + with flask_app.app_context(): + df = table._load_filter_sort_postprocess( + filter_query="{objet} icontains travaux", sort_by_key=() + ) + assert df.height == 1 + + df_empty = table._load_filter_sort_postprocess( + filter_query="{objet} icontains nonexistent", sort_by_key=() + ) + assert df_empty.height == 0 + + +def test_load_filter_sort_postprocess_adds_links(flask_app, monkeypatch, sample_lff): + from src.utils import table + + monkeypatch.setattr(table, "query_marches", lambda: sample_lff.collect()) + + with flask_app.app_context(): + df = table._load_filter_sort_postprocess(filter_query=None, sort_by_key=()) + # add_links injects an wrapper around uid, acheteur_nom, titulaire_nom + assert "= 32 # uuid4 hex string + + +def test_prepare_table_data_with_external_data_does_not_use_cache( + monkeypatch, flask_app, sample_lff +): + """When a caller passes data (acheteur/titulaire/observatoire path), + bypass the memoized helper entirely.""" + from src.utils import table + + sentinel = {"called": False} + + def should_not_be_called(*a, **kw): + sentinel["called"] = True + raise AssertionError("Memoized helper must not be called when data is provided") + + monkeypatch.setattr(table, "_load_filter_sort_postprocess", should_not_be_called) + + with flask_app.app_context(): + table.prepare_table_data( + data=sample_lff, # external LazyFrame + data_timestamp=0, + filter_query=None, + page_current=0, + page_size=20, + sort_by=[], + source_table="acheteur", + ) + + assert sentinel["called"] is False diff --git a/uv.lock b/uv.lock index b2ced7c..2c14012 100644 --- a/uv.lock +++ b/uv.lock @@ -783,7 +783,7 @@ dependencies = [ { name = "xlsxwriter" }, ] -[package.optional-dependencies] +[package.dev-dependencies] dev = [ { name = "dash", extra = ["testing"] }, { name = "fastexcel" }, @@ -798,29 +798,32 @@ dev = [ requires-dist = [ { name = "dash", specifier = "==3.4.0" }, { name = "dash", extras = ["compress"] }, - { name = "dash", extras = ["testing"], marker = "extra == 'dev'" }, { name = "dash-bootstrap-components" }, { name = "dash-extensions" }, { name = "dash-leaflet" }, { name = "duckdb" }, - { name = "fastexcel", marker = "extra == 'dev'" }, { name = "flask-caching" }, { name = "gunicorn" }, { name = "httpx" }, { name = "pandas" }, { name = "plotly", extras = ["express"] }, { name = "polars" }, - { name = "pre-commit", marker = "extra == 'dev'" }, { name = "pyarrow", specifier = ">=23.0.1" }, - { name = "pytest", marker = "extra == 'dev'" }, - { name = "pytest-env", marker = "extra == 'dev'" }, { name = "python-dotenv" }, - { name = "selenium", marker = "extra == 'dev'" }, { name = "unidecode" }, - { name = "webdriver-manager", marker = "extra == 'dev'" }, { name = "xlsxwriter" }, ] -provides-extras = ["dev"] + +[package.metadata.requires-dev] +dev = [ + { name = "dash", extras = ["testing"] }, + { name = "fastexcel" }, + { name = "pre-commit" }, + { name = "pytest" }, + { name = "pytest-env" }, + { name = "selenium" }, + { name = "webdriver-manager" }, +] [[package]] name = "dill"