Vérification de l'existence de cache_dir

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

.gitignore

@@ -3,5 +3,11 @@
 __pycache__
 .idea
 .venv
+.worktrees
 build
 .env
+
+# DuckDB runtime artifacts (regenerated from decp_prod.parquet at startup)
+**/decp.duckdb
+**/decp.duckdb.tmp
+**/decp.duckdb.lock

CHANGELOG.md

@@ -1,3 +1,42 @@
+#### 2.7.2 (19 avril 2026)
+
+- Chargement des données depuis une base DuckDB plutôt qu'en mémoire (plus de stabilité) ([#71](https://github.com/ColinMaudry/decp.info/issues/71))
+- Mise en cache des vue sur l'observatoire pour un chargement plus rapide (remise à zéro quotidienne)
+- Correction de bug : la liste de colonnes par défaut est bien appliquée plutôt qu'afficher toutes les colonnes
+- Quelques corrections de bugs d'affichage
+- Refactorisation des fonctions utilitaires (`utils.py` approchait des 1 000 lignes)
+
+#### 2.7.1 (23 mars 2026)
+
+- Correction du partage de données filtrées entre dashboard et vue des données
+
+#### 2.7.0 (23 mars 2026)
+
+- Remplacement de la page Statistiques par l'observatoire
+- Généralisation de la grille dash (`dbc.Row`, `dbc.Col`)
+- Ajout de l'histogramme de distances aux pages acheteur et titulaire
+- Ajout de la colonne `acheteur_categorie` (commune, État, etc.)
+
+##### 2.6.2 (22 février 2026)
+
+- Correction du téléchargemnent buggé dans /tableau
+
+##### 2.6.1 (17 février 2026)
+
+- Corrections la création des liens canoniques (SEO)
+
+#### 2.6.0 (5 février 2026)
+
+- Suite de la refonte graphique
+- Persistence des filtres, des tris et des choix de colonnes sur toutes les pages
+- Joli tableau pour choisir les colonnes à afficher
+- Meilleure gestion des acheteurs et titulaires absents de la base SIRENE
+- Amélioration du SEO (liens canoniques)
+
+##### 2.5.1 (29 janvier 2026)
+
+- Mise en production un peu hâtive ([#67](https://github.com/ColinMaudry/decp.info/issues/67), [#68](https://github.com/ColinMaudry/decp.info/issues/68))
+
 #### 2.5.0 (29 janvier 2026)
 
 - Refonte graphique et amélioration des textes d'aide
@@ -151,7 +190,7 @@
 
 ### 1.0.0
 
-- publication sur https://decp.info
+- publication sur <https://decp.info>
 - ajout d'une vue équivalente au format DECP réglementaire
 - personnalisation de datasette
 - script de conversion quotidien basé sur [dataflows](https://github.com/datahq/dataflows)

CLAUDE.md

@@ -0,0 +1,89 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+**decp.info** is a French public procurement data explorer — a Dash (Python) web app for browsing, filtering, and visualizing _Données Essentielles de la Commande Publique_ (DECP). The UI is in French.
+
+## Commands
+
+### Setup
+
+```bash
+python -m venv .venv && source .venv/bin/activate
+pip install ".[dev]"
+cp template.env .env   # then customize .env
+```
+
+### Development
+
+```bash
+uv run run.py         # starts Dash with debug=True and hot reload
+```
+
+### Production
+
+```bash
+gunicorn app:server
+```
+
+### Tests
+
+```bash
+uv run pytest                  # run all tests (Selenium-based integration tests)
+uv run pytest tests/test_main.py::test_001_logo_and_search   # run a single test
+```
+
+Tests require a running Chrome/Chromium browser. They use `DashComposite` from `dash[testing]` with Selenium WebDriver.
+
+## Architecture
+
+### Multi-page Dash app
+
+- `src/app.py` — creates the Dash app instance, navbar, SEO endpoints (robots.txt, sitemap.xml), Matomo analytics
+- `src/pages/*.py` — each page registers itself with `@register_page()` and owns its own layout and callbacks
+- `run.py` — dev entry point; exports `server` (Flask) for gunicorn
+
+### Module imports
+
+- always import modules from the app starting with `src.` (e.g. `src.utils.`, `src.pages.recherche`, etc.)
+
+### Key pages
+
+| Page              | URL             | Purpose                                |
+| ----------------- | --------------- | -------------------------------------- |
+| `recherche.py`    | `/`             | Search homepage for buyers/contractors |
+| `acheteur.py`     | `/acheteur`     | Buyer detail with stats, charts, maps  |
+| `titulaire.py`    | `/titulaire`    | Contractor detail                      |
+| `tableau.py`      | `/tableau`      | Filterable data table with exports     |
+| `marche.py`       | `/marche`       | Individual contract detail             |
+| `observatoire.py` | `/observatoire` | An interactive analytics dashboard     |
+
+### Data layer
+
+- Data is stored as **Parquet** and loaded with **Polars** (fast columnar operations)
+- Path set via `DATA_FILE_PARQUET_PATH` env var; tests use `tests/test.parquet`
+- `src/utils.py` — filtering helpers, search (`search_org`), link generation, geographic data loading
+- `src/callbacks.py` — shared Dash callbacks (e.g. `get_top_org_table`)
+- `src/figures.py` — chart and map components (Plotly Express, Dash Leaflet with marker clustering)
+- a Parquet file with production data is located at `../decp-processing/decp_prod.parquet` (~ 1,5 million records)
+- the TableSchema of the dataset with the list of field and their definition is located at `../decp-processing/reference/base_schema.json`
+- `tests/test.parquet` is very small and may not contain all possible columns, only those necessary for testing
+
+### UI stack
+
+- **Dash 3.4** + **Dash Bootstrap Components** for layout
+- **Plotly Express** for charts
+- **Dash Leaflet** + **Dash Extensions** for interactive maps with clustering
+- Custom CSS in `src/assets/css/`
+
+### Environment
+
+- `DEVELOPMENT=true` enables debug logging and is set automatically during tests
+- `.env` file is required at runtime (copy from `template.env`)
+
+### Deployment
+
+- `main` branch → manual deploy to decp.info via GitHub Actions
+- `dev` branch → auto-deploy to test.decp.info via GitHub Actions

README.md

@@ -1,6 +1,6 @@
 # decp.info
 
-> v2.5.0
+> v2.7.2
 > Outil d'exploration et de téléchargement des données essentielles de la commande publique.
 
 => [decp.info](https://decp.info)

data/departements.json

@@ -391,8 +391,16 @@
     "departement": "La Réunion",
     "region": "La Réunion"
   },
+  "975": {
+    "departement": "Saint-Pierre-et-Miquelon",
+    "region": "Saint-Pierre-et-Miquelon"
+  },
   "976": {
     "departement": "Mayotte",
     "region": "Mayotte"
+  },
+  "977": {
+    "departement": "Saint-Barthelemy",
+    "region": "Saint-Barthelemy"
   }
 }

docs/superpowers/plans/2026-03-18-observatoire-link.md

@@ -0,0 +1,473 @@
+# Observatoire Link from Search & Tableau Results — Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Let users jump from search/tableau results to the observatoire page, pre-filtered for a given organization, via a 📊 link in the `_nom` columns.
+
+**Architecture:** Modify `add_links()` in `src/utils.py` to append an observatoire link to `_nom` columns. Add two callbacks to `src/pages/observatoire.py` for bidirectional URL ↔ filter sync using the existing `dcc.Location(id="dashboard_url")`. Add a share URL input and clipboard button to the observatoire layout.
+
+**Tech Stack:** Dash 3.4, Polars, `urllib.parse`, `dcc.Location`, `dcc.Clipboard`
+
+**Spec:** `docs/superpowers/specs/2026-03-18-observatoire-link-from-search-design.md`
+
+---
+
+### Task 1: Add observatoire link to `acheteur_nom` in `add_links()`
+
+**Files:**
+
+- Modify: `src/utils.py:82-91` (the `acheteur_` block inside `add_links()`)
+- Test: `tests/test_main.py`
+
+**Context:** The `add_links()` function loops over column names. The `if col.startswith("acheteur_")` block (lines 82-91) currently wraps both `acheteur_nom` and `acheteur_id` in a detail page link. We must only append the observatoire link when `col == "acheteur_nom"`.
+
+- [ ] **Step 1: Write a unit test for the observatoire link in acheteur_nom**
+
+In `tests/test_main.py`, add a test that calls `add_links()` on a minimal DataFrame and checks the `acheteur_nom` column contains both the detail link and the observatoire link, while `acheteur_id` does NOT contain the observatoire link.
+
+```python
+def test_004_add_links_observatoire_acheteur():
+    import polars as pl
+
+    from src.utils import add_links
+
+    dff = pl.DataFrame(
+        {
+            "acheteur_id": ["a1"],
+            "acheteur_nom": ["ACHETEUR 1"],
+        }
+    )
+    result = add_links(dff)
+    nom_value = result["acheteur_nom"][0]
+    id_value = result["acheteur_id"][0]
+
+    # acheteur_nom should contain detail link + observatoire link
+    assert "/acheteurs/a1" in nom_value
+    assert "ACHETEUR 1" in nom_value
+    assert '/observatoire?acheteur_id=a1' in nom_value
+    assert "📊" in nom_value
+
+    # acheteur_id should NOT contain observatoire link
+    assert "/observatoire" not in id_value
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `source .venv/bin/activate && pytest tests/test_main.py::test_004_add_links_observatoire_acheteur -v`
+Expected: FAIL — `'/observatoire?acheteur_id=a1'` not found in the output string.
+
+- [ ] **Step 3: Implement the observatoire link for acheteur_nom**
+
+In `src/utils.py`, modify the `if col.startswith("acheteur_")` block (lines 82-91). Gate the observatoire link append on `col == "acheteur_nom"`:
+
+```python
+            if col.startswith("acheteur_"):
+                detail_link = (
+                    '<a href = "/acheteurs/'
+                    + pl.col("acheteur_id")
+                    + '">'
+                    + pl.col(col)
+                    + "</a>"
+                )
+                if col == "acheteur_nom":
+                    detail_link = (
+                        detail_link
+                        + ' <a href="/observatoire?acheteur_id='
+                        + pl.col("acheteur_id")
+                        + '" title="Voir dans l\'observatoire">📊</a>'
+                    )
+                dff = dff.with_columns(detail_link.alias(col))
+```
+
+- [ ] **Step 4: Run test to verify it passes**
+
+Run: `source .venv/bin/activate && pytest tests/test_main.py::test_004_add_links_observatoire_acheteur -v`
+Expected: PASS
+
+- [ ] **Step 5: Update `test_001` to account for the new emoji in cell text**
+
+The existing `test_001` asserts `result_table.find_element(...).text == name` for `acheteur_nom`. The cell text now includes "📊" from the observatoire link. Update the assertion in `tests/test_main.py` to use `startswith` instead of exact match:
+
+```python
+        assert result_table.find_element(
+            by=By.CSS_SELECTOR, value=f'td[data-dash-column="{org_type}_nom"]'
+        ).text.startswith(
+            name
+        ), f"The search result should have the right {org_type} name"
+```
+
+- [ ] **Step 6: Run `test_001` to verify it still passes**
+
+Run: `source .venv/bin/activate && pytest tests/test_main.py::test_001_logo_and_search -v`
+Expected: PASS
+
+- [ ] **Step 7: Commit**
+
+```bash
+git add src/utils.py tests/test_main.py
+git commit -m "Ajout du lien observatoire dans acheteur_nom via add_links() #65"
+```
+
+---
+
+### Task 2: Add observatoire link to `titulaire_nom` in `add_links()`
+
+**Files:**
+
+- Modify: `src/utils.py:64-81` (the `titulaire_` block inside `add_links()`)
+- Test: `tests/test_main.py`
+
+**Context:** The `titulaire_` block (lines 64-81) uses a `pl.when().then().otherwise()` pattern because it guards on `titulaire_typeIdentifiant` being SIRET or null. The observatoire link must be appended inside the `.then()` branch, and only when `col == "titulaire_nom"`. Note: this block requires `titulaire_typeIdentifiant` to be present in the DataFrame.
+
+- [ ] **Step 1: Write a unit test for the observatoire link in titulaire_nom**
+
+```python
+def test_005_add_links_observatoire_titulaire():
+    import polars as pl
+
+    from src.utils import add_links
+
+    dff = pl.DataFrame(
+        {
+            "titulaire_id": ["t1"],
+            "titulaire_nom": ["TITULAIRE 1"],
+            "titulaire_typeIdentifiant": ["SIRET"],
+        }
+    )
+    result = add_links(dff)
+    nom_value = result["titulaire_nom"][0]
+    id_value = result["titulaire_id"][0]
+
+    # titulaire_nom should contain detail link + observatoire link
+    assert "/titulaires/t1" in nom_value
+    assert "TITULAIRE 1" in nom_value
+    assert '/observatoire?titulaire_id=t1' in nom_value
+    assert "📊" in nom_value
+
+    # titulaire_id should NOT contain observatoire link
+    assert "/observatoire" not in id_value
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `source .venv/bin/activate && pytest tests/test_main.py::test_005_add_links_observatoire_titulaire -v`
+Expected: FAIL — `'/observatoire?titulaire_id=t1'` not found.
+
+- [ ] **Step 3: Implement the observatoire link for titulaire_nom**
+
+In `src/utils.py`, modify the `if col.startswith("titulaire_")` block (lines 64-81). The `.then()` branch must build the link differently when `col == "titulaire_nom"`:
+
+```python
+            if col.startswith("titulaire_"):
+                detail_link = (
+                    '<a href = "/titulaires/'
+                    + pl.col("titulaire_id")
+                    + '">'
+                    + pl.col(col)
+                    + "</a>"
+                )
+                if col == "titulaire_nom":
+                    detail_link = (
+                        detail_link
+                        + ' <a href="/observatoire?titulaire_id='
+                        + pl.col("titulaire_id")
+                        + '" title="Voir dans l\'observatoire">📊</a>'
+                    )
+                dff = dff.with_columns(
+                    pl.when(
+                        pl.Expr.or_(
+                            pl.col("titulaire_typeIdentifiant").is_null(),
+                            pl.col("titulaire_typeIdentifiant") == "SIRET",
+                        )
+                    )
+                    .then(detail_link)
+                    .otherwise(pl.col(col))
+                    .alias(col)
+                )
+```
+
+- [ ] **Step 4: Run test to verify it passes**
+
+Run: `source .venv/bin/activate && pytest tests/test_main.py::test_005_add_links_observatoire_titulaire -v`
+Expected: PASS
+
+- [ ] **Step 5: Run all tests so far to check for regressions**
+
+Run: `source .venv/bin/activate && pytest tests/test_main.py::test_004_add_links_observatoire_acheteur tests/test_main.py::test_005_add_links_observatoire_titulaire -v`
+Expected: both PASS
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add src/utils.py tests/test_main.py
+git commit -m "Ajout du lien observatoire dans titulaire_nom via add_links() #65"
+```
+
+---
+
+### Task 3: Observatoire Callback A — URL → Inputs (page load)
+
+**Files:**
+
+- Modify: `src/pages/observatoire.py` (add import + new callback after line 281)
+- Test: `tests/test_main.py`
+
+**Context:** The existing `dcc.Location(id="dashboard_url")` is in the observatoire layout. A new callback reads `dashboard_url.search` on page load, parses query params, and sets `dashboard_acheteur_id.value` and/or `dashboard_titulaire_id.value`. It also clears `dashboard_url.search` to `""` to prevent re-triggering. Two imports must be added: `import urllib.parse` at the top of the file, and `no_update` to the existing `from dash import ...` line (currently: `from dash import ALL, Input, Output, State, callback, ctx, dcc, html, register_page` — add `no_update` to this).
+
+- [ ] **Step 1: Write a Selenium test for URL → Input sync**
+
+This test navigates to `/observatoire?acheteur_id=a1` and verifies the SIRET input gets populated.
+
+```python
+def test_006_observatoire_url_to_input(dash_duo: DashComposite):
+    from src.app import app
+
+    dash_duo.start_server(app)
+    dash_duo.wait_for_text_to_equal(".logo > h1", "decp.info", timeout=4)
+
+    # Navigate to observatoire with acheteur_id query param
+    dash_duo.wait_for_page(f"{dash_duo.server_url}/observatoire?acheteur_id=a1")
+    dash_duo.wait_for_element("#dashboard_acheteur_id", timeout=4)
+
+    acheteur_input = dash_duo.find_element("#dashboard_acheteur_id")
+    dash_duo.wait_for_text_to_equal(
+        "#dashboard_acheteur_id", "", timeout=4
+    )  # Wait for callback
+    import time
+    time.sleep(1)  # Allow callback chain to complete
+
+    assert acheteur_input.get_attribute("value") == "a1", (
+        "acheteur_id input should be populated from URL param"
+    )
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `source .venv/bin/activate && pytest tests/test_main.py::test_006_observatoire_url_to_input -v`
+Expected: FAIL — the input value is empty because no callback reads URL params yet.
+
+- [ ] **Step 3: Implement Callback A**
+
+Add `import urllib.parse` to the imports at the top of `src/pages/observatoire.py` (after line 1). Also add `no_update` to the existing dash import line:
+
+```python
+from dash import ALL, Input, Output, State, callback, ctx, dcc, html, no_update, register_page
+```
+
+Add the callback after the `layout` list ends, before existing callbacks:
+
+```python
+@callback(
+    Output("dashboard_acheteur_id", "value"),
+    Output("dashboard_titulaire_id", "value"),
+    Output("dashboard_url", "search"),
+    Input("dashboard_url", "search"),
+)
+def restore_filters_from_url(search):
+    if not search:
+        return no_update, no_update, no_update
+
+    params = urllib.parse.parse_qs(search.lstrip("?"))
+
+    acheteur_id = params.get("acheteur_id", [None])[0] or no_update
+    titulaire_id = params.get("titulaire_id", [None])[0] or no_update
+
+    return acheteur_id, titulaire_id, ""
+```
+
+- [ ] **Step 4: Run test to verify it passes**
+
+Run: `source .venv/bin/activate && pytest tests/test_main.py::test_006_observatoire_url_to_input -v`
+Expected: PASS
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add src/pages/observatoire.py tests/test_main.py
+git commit -m "Callback URL → filtres sur la page observatoire #65"
+```
+
+---
+
+### Task 4: Observatoire Callback B — Inputs → shareable URL + layout
+
+**Files:**
+
+- Modify: `src/pages/observatoire.py` (add layout components + new callback)
+- Test: `tests/test_main.py`
+
+**Context:** Following the tableau.py pattern (lines 237-238 for layout, lines 399-450 for callback), add a hidden `share-url` input and a `copy-container` div to the observatoire layout. The callback listens to the ID inputs and builds a shareable URL. Component IDs must be unique across the app, so use `observatoire-share-url` and `observatoire-copy-container` to avoid collisions with tableau's `share-url` and `copy-container`.
+
+- [ ] **Step 1: Write a test for the shareable URL generation**
+
+```python
+def test_007_observatoire_share_url(dash_duo: DashComposite):
+    from src.app import app
+
+    dash_duo.start_server(app)
+    dash_duo.wait_for_text_to_equal(".logo > h1", "decp.info", timeout=4)
+
+    # Navigate to observatoire with acheteur_id query param
+    dash_duo.wait_for_page(f"{dash_duo.server_url}/observatoire?acheteur_id=a1")
+    dash_duo.wait_for_element("#observatoire-share-url", timeout=4)
+
+    import time
+    time.sleep(1)  # Allow callback chain to complete
+
+    share_url_input = dash_duo.find_element("#observatoire-share-url")
+    share_url_value = share_url_input.get_attribute("value")
+
+    assert "acheteur_id=a1" in share_url_value, (
+        f"Share URL should contain acheteur_id param, got: {share_url_value}"
+    )
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `source .venv/bin/activate && pytest tests/test_main.py::test_007_observatoire_share_url -v`
+Expected: FAIL — `#observatoire-share-url` element does not exist yet.
+
+- [ ] **Step 3: Add layout components to observatoire**
+
+In `src/pages/observatoire.py`, add the share URL input and copy container inside the filters column (after the download button, before the closing `]` of the `id="filters"` children list, around line 264):
+
+```python
+                                    dcc.Input(
+                                        id="observatoire-share-url",
+                                        readOnly=True,
+                                        style={"display": "none"},
+                                    ),
+                                    html.Div(id="observatoire-copy-container"),
+```
+
+- [ ] **Step 4: Implement Callback B**
+
+Add after Callback A in `src/pages/observatoire.py`:
+
+```python
+@callback(
+    Output("observatoire-share-url", "value"),
+    Output("observatoire-copy-container", "children"),
+    Input("dashboard_acheteur_id", "value"),
+    Input("dashboard_titulaire_id", "value"),
+    State("dashboard_url", "href"),
+    prevent_initial_call=True,
+)
+def sync_observatoire_share_url(acheteur_id, titulaire_id, href):
+    if not href:
+        return no_update, no_update
+
+    base_url = href.split("?")[0]
+
+    params = {}
+    if acheteur_id:
+        params["acheteur_id"] = acheteur_id
+    if titulaire_id:
+        params["titulaire_id"] = titulaire_id
+
+    query_string = urllib.parse.urlencode(params)
+    full_url = f"{base_url}?{query_string}" if query_string else base_url
+
+    copy_button = dcc.Clipboard(
+        id="btn-copy-observatoire-url",
+        target_id="observatoire-share-url",
+        title="Copier l'URL de cette vue",
+        style={
+            "display": "inline-block",
+            "fontSize": 20,
+            "verticalAlign": "top",
+            "cursor": "pointer",
+        },
+        className="fa fa-link",
+        children=[
+            dbc.Button(
+                "Partager",
+                className="btn btn-primary mt-2",
+                title="Copier l'adresse de cette vue filtrée pour la partager.",
+            )
+        ],
+    )
+
+    return full_url, copy_button
+```
+
+- [ ] **Step 5: Run test to verify it passes**
+
+Run: `source .venv/bin/activate && pytest tests/test_main.py::test_007_observatoire_share_url -v`
+Expected: PASS
+
+- [ ] **Step 6: Run all tests to check for regressions**
+
+Run: `source .venv/bin/activate && pytest tests/test_main.py -v`
+Expected: all tests PASS
+
+- [ ] **Step 7: Commit**
+
+```bash
+git add src/pages/observatoire.py tests/test_main.py
+git commit -m "URL partageable pour la page observatoire #65"
+```
+
+---
+
+### Task 5: End-to-end integration test
+
+**Files:**
+
+- Test: `tests/test_main.py`
+
+**Context:** Verify the full flow: search for an organization on the homepage, see the 📊 link in results, click it, arrive on the observatoire with the correct input populated.
+
+- [ ] **Step 1: Write end-to-end test**
+
+```python
+def test_008_search_to_observatoire(dash_duo: DashComposite):
+    from src.app import app
+
+    dash_duo.start_server(app)
+    dash_duo.wait_for_text_to_equal(".logo > h1", "decp.info", timeout=4)
+
+    # Search for an acheteur
+    search_bar = dash_duo.find_element("#search")
+    search_bar.send_keys("ACHETEUR 1")
+    search_bar.send_keys(Keys.ENTER)
+
+    dash_duo.wait_for_element("#results_acheteur_datatable", timeout=2)
+
+    # Find the observatoire link in acheteur_nom column
+    observatoire_link = dash_duo.find_element(
+        '#results_acheteur_datatable td[data-dash-column="acheteur_nom"] a[href*="observatoire"]'
+    )
+    assert "📊" in observatoire_link.text
+
+    # Click the observatoire link
+    observatoire_link.click()
+
+    # Wait for observatoire page to load
+    dash_duo.wait_for_element("#dashboard_acheteur_id", timeout=4)
+
+    import time
+    time.sleep(1)  # Allow callback chain to complete
+
+    acheteur_input = dash_duo.find_element("#dashboard_acheteur_id")
+    assert acheteur_input.get_attribute("value") == "a1", (
+        "acheteur_id input should be populated after navigating from search"
+    )
+```
+
+- [ ] **Step 2: Run end-to-end test**
+
+Run: `source .venv/bin/activate && pytest tests/test_main.py::test_008_search_to_observatoire -v`
+Expected: PASS
+
+- [ ] **Step 3: Run the full test suite**
+
+Run: `source .venv/bin/activate && pytest tests/test_main.py -v`
+Expected: all tests PASS
+
+- [ ] **Step 4: Commit**
+
+```bash
+git add tests/test_main.py
+git commit -m "Test e2e : recherche → observatoire #65"
+```

docs/superpowers/specs/2026-03-18-distance-histogram-design.md

@@ -0,0 +1,64 @@
+# Distance Histogram — Design Spec
+
+**Date:** 2026-03-18
+**Branch:** feature/65_observatoire
+
+## Goal
+
+Display the distribution of distances (in km) between buyers and winning contractors, to help users assess whether a buyer or contractor tends to deal locally or at a national scale.
+
+## Data
+
+- Column: `titulaire_distance` (`Int64`, km)
+- Measured at address level — values are always > 0, no zero-handling needed
+- Already selected in the observatoire LazyFrame via `cs.starts_with("titulaire")`
+- Already available on acheteur and titulaire detail pages
+
+## Figure Function
+
+**Location:** `src/figures.py`
+
+**Signature:**
+
+```python
+def get_distance_histogram(lff: pl.LazyFrame) -> dcc.Graph:
+```
+
+**Behaviour:**
+
+- Collects `titulaire_distance` from the LazyFrame, drops nulls
+- If the resulting DataFrame is empty after dropping nulls, `px.histogram` produces a blank figure without errors — no guard logic needed. The order of operations must be: drop nulls → log-transform → histogram
+- Drop nulls first, then pre-log-transform the column (`pl.col("titulaire_distance").log(10)`) so bins are truly equal-width on a log scale. Use `px.histogram` with `nbins=50` on the transformed values
+- Set custom X-axis tick values at powers of 10 (1, 10, 100, 1000, 10000) with km labels, using `fig.update_xaxes(tickvals=[0,1,2,3,4], ticktext=["1","10","100","1 000","10 000"])`
+- Y axis: count of contracts
+- French axis labels: x = `"Distance (km)"`, y = `"Nombre de marchés"`
+- Returns a `dcc.Graph`
+
+## Integration
+
+### Observatoire (`src/pages/observatoire.py`)
+
+- `get_distance_histogram` imported and called inside `udpate_dashboard_cards`
+- Result wrapped in `make_card(title="Distance acheteur–titulaire", subtitle="en nombre de marchés, échelle logarithmique", fig=...)`
+- Card appended to the `cards` list alongside existing donuts and charts
+- No changes to the data pipeline — `titulaire_distance` is already in the LazyFrame
+
+### Acheteur page (`src/pages/acheteur.py`)
+
+The acheteur page uses a `dcc.Store` (`acheteur_data`) that holds serialised contract rows as a list of dicts. The integration follows the existing pattern used by other chart callbacks on this page:
+
+- Add a new `html.Div(id="acheteur-distance-histogram")` placeholder in the layout
+- Add a new callback with `Input("acheteur_data", "data")` that:
+  - Reconstructs `pl.LazyFrame(data)` from the store
+  - Calls `get_distance_histogram(lff)`
+  - Wraps the result in `make_card(...)` and returns it to the placeholder div
+
+### Titulaire page (`src/pages/titulaire.py`)
+
+Same pattern as acheteur: `dcc.Store` (`titulaire_data`) → new callback → `html.Div` placeholder.
+
+## Out of Scope
+
+- Filtering by distance range (could be a future filter on the observatoire page)
+- Showing distance on a map or as a trend over time
+- Bucket-based (named zone) grouping

docs/superpowers/specs/2026-03-18-observatoire-link-from-search-design.md

@@ -0,0 +1,78 @@
+# Observatoire Link from Search & Tableau Results
+
+## Problem
+
+Users searching for an organization (acheteur or titulaire) on the search page or browsing the tableau cannot jump directly to the observatoire page filtered for that organization. They must manually navigate and re-enter the identifier.
+
+## Solution
+
+Extend `add_links()` in `src/utils.py` to append an observatoire link (📊 emoji) to `_nom` columns, and add bidirectional URL parameter sync to the observatoire page.
+
+## Changes
+
+### 1. `src/utils.py` — `add_links()` modification
+
+The existing `add_links()` loop iterates over `["uid", "acheteur_nom", "titulaire_nom", "acheteur_id", "titulaire_id"]`. The `if col.startswith("acheteur_")` and `if col.startswith("titulaire_")` blocks match both `_nom` and `_id` columns. The observatoire link must only be appended to `_nom` columns, so it must be gated on `col == "acheteur_nom"` or `col == "titulaire_nom"` explicitly.
+
+For `acheteur_nom`, append an observatoire link after the existing detail page link:
+
+```
+Before: <a href="/acheteurs/12345678901234">Ville de Paris</a>
+After:  <a href="/acheteurs/12345678901234">Ville de Paris</a> <a href="/observatoire?acheteur_id=12345678901234" title="Voir dans l'observatoire">📊</a>
+```
+
+For `titulaire_nom`, same pattern but only when the existing `typeIdentifiant` guard passes (SIRET or null):
+
+```
+Before: <a href="/titulaires/12345678901234">Entreprise X</a>
+After:  <a href="/titulaires/12345678901234">Entreprise X</a> <a href="/observatoire?titulaire_id=12345678901234" title="Voir dans l'observatoire">📊</a>
+```
+
+The identifier used in the observatoire link (`acheteur_id` / `titulaire_id`) is the same `pl.col("acheteur_id")` / `pl.col("titulaire_id")` column value already used for the detail page link.
+
+The `_id` and `uid` columns are unchanged.
+
+### 2. `src/pages/observatoire.py` — URL parameter handling
+
+#### Callback A: URL → Inputs (page load)
+
+- Trigger: `Input("dashboard_url", "search")`
+- Outputs: `Output("dashboard_acheteur_id", "value")`, `Output("dashboard_titulaire_id", "value")`, `Output("dashboard_url", "search")` (to clear it)
+- `prevent_initial_call=False` (must fire on page load to read URL params)
+- If `search` is empty or None: return `no_update` for all outputs
+- Otherwise: parse query params with `urllib.parse.parse_qs`
+- Set `dashboard_acheteur_id` from `?acheteur_id=` param, or `no_update` if absent
+- Set `dashboard_titulaire_id` from `?titulaire_id=` param, or `no_update` if absent
+- Return `""` for `dashboard_url.search` to clear the URL and prevent re-triggering
+- No validation of param values — consistent with existing input handling in the observatoire callbacks
+
+#### Callback B: Inputs → shareable URL
+
+- Trigger: `Input("dashboard_acheteur_id", "value")`, `Input("dashboard_titulaire_id", "value")`
+- State: `State("dashboard_url", "href")` for base URL
+- `prevent_initial_call=True` (avoid generating URL on initial empty state)
+- Build query string with `urllib.parse.urlencode`, omitting empty values
+- Write full URL to a new `share-url` input component
+- Render a `dcc.Clipboard` + share button (same pattern as tableau.py)
+
+#### Callback chain
+
+When navigating from search with `?acheteur_id=123`: Callback A fires on page load, sets input values, clears URL search. The input value changes then trigger both the existing `udpate_dashboard_cards` callback and Callback B. Dash handles this chaining deterministically — no race condition.
+
+#### Layout additions
+
+- A `dcc.Input(id="share-url", ...)` (hidden or read-only) to hold the shareable URL
+- A `dcc.Clipboard` share/copy button near the filters
+
+### 3. Reuse of existing `dcc.Location`
+
+The existing `dcc.Location(id="dashboard_url")` component is reused — no new Location component needed.
+
+## Future extension
+
+The bidirectional URL sync pattern is designed to extend to all observatoire filters (year, categories, departments, market type, etc.) by adding more params to both callbacks.
+
+## Files touched
+
+- `src/utils.py` — modify `add_links()`
+- `src/pages/observatoire.py` — add 2 callbacks, add share-url + clipboard to layout

docs/superpowers/specs/2026-03-21-observatoire-full-url-sharing-design.md

@@ -0,0 +1,121 @@
+# Observatoire: Full URL Sharing for All Filters
+
+## Problem
+
+The "Partager" button on `/observatoire` currently only encodes `acheteur_id` and `titulaire_id` in the shareable URL. The other 15 filter parameters are lost, so a shared link does not reproduce the sender's filtered view.
+
+## Goal
+
+Extend URL sharing so that **all 17 filter parameters** are encoded in the URL and restored when a recipient opens it. The recipient sees exactly what the sender intended — URL params replace all local filter state.
+
+## Approach
+
+Flat query parameters with short, readable keys. Multi-value filters use repeated keys (native to `urllib.parse`). Only non-default values appear in the URL.
+
+## URL Parameter Mapping
+
+| Component ID                                       | URL key          | Type            | Default (omitted) |
+| -------------------------------------------------- | ---------------- | --------------- | ----------------- |
+| `dashboard_year`                                   | `annee`          | single          | `None`            |
+| `dashboard_acheteur_id`                            | `acheteur_id`    | single          | `None`            |
+| `dashboard_acheteur_categorie`                     | `acheteur_cat`   | single          | `None`            |
+| `dashboard_acheteur_departement_code`              | `acheteur_dept`  | multi           | `[]`/`None`       |
+| `dashboard_titulaire_id`                           | `titulaire_id`   | single          | `None`            |
+| `dashboard_titulaire_categorie`                    | `titulaire_cat`  | single          | `None`            |
+| `dashboard_titulaire_departement_code`             | `titulaire_dept` | multi           | `[]`/`None`       |
+| `dashboard_marche_type`                            | `type`           | single          | `None`            |
+| `dashboard_marche_objet`                           | `objet`          | single          | `None`            |
+| `dashboard_marche_code_cpv`                        | `cpv`            | single          | `None`            |
+| `dashboard_montant_min`                            | `montant_min`    | single (number) | `None`            |
+| `dashboard_montant_max`                            | `montant_max`    | single (number) | `None`            |
+| `dashboard_marche_techniques`                      | `techniques`     | multi           | `[]`/`None`       |
+| `dashboard_marche_innovant`                        | `innovant`       | single          | `"all"`           |
+| `dashboard_marche_sousTraitanceDeclaree`           | `sous_traitance` | single          | `"all"`           |
+| `dashboard_marche_considerationsSociales`          | `social`         | multi           | `[]`/`None`       |
+| `dashboard_marche_considerationsEnvironnementales` | `env`            | multi           | `[]`/`None`       |
+
+Example URL:
+
+```
+/observatoire?annee=2024&acheteur_id=12345678901234&acheteur_dept=75&acheteur_dept=13&montant_min=10000&innovant=oui
+```
+
+## Data Structure
+
+A list of tuples defines the mapping, used by both callbacks to avoid scattered string literals:
+
+```python
+FILTER_PARAMS = [
+    # (component_id, url_key, is_multi, default_value)
+    ("dashboard_year", "annee", False, None),
+    ("dashboard_acheteur_id", "acheteur_id", False, None),
+    ("dashboard_acheteur_categorie", "acheteur_cat", False, None),
+    ("dashboard_acheteur_departement_code", "acheteur_dept", True, None),
+    ("dashboard_titulaire_id", "titulaire_id", False, None),
+    ("dashboard_titulaire_categorie", "titulaire_cat", False, None),
+    ("dashboard_titulaire_departement_code", "titulaire_dept", True, None),
+    ("dashboard_marche_type", "type", False, None),
+    ("dashboard_marche_objet", "objet", False, None),
+    ("dashboard_marche_code_cpv", "cpv", False, None),
+    ("dashboard_montant_min", "montant_min", False, None),
+    ("dashboard_montant_max", "montant_max", False, None),
+    ("dashboard_marche_techniques", "techniques", True, None),
+    ("dashboard_marche_innovant", "innovant", False, "all"),
+    ("dashboard_marche_sousTraitanceDeclaree", "sous_traitance", False, "all"),
+    ("dashboard_marche_considerationsSociales", "social", True, None),
+    ("dashboard_marche_considerationsEnvironnementales", "env", True, None),
+]
+```
+
+## Callback Changes
+
+### 1. `sync_observatoire_share_url` (line 575)
+
+**Current:** Takes `acheteur_id` and `titulaire_id` as Inputs.
+
+**New:** Takes all 17 filter values as Inputs (same as `udpate_dashboard_cards`). Builds the URL using `FILTER_PARAMS`, skipping default values. Uses `urllib.parse.urlencode(params, doseq=True)` for multi-value params.
+
+### 2. `restore_filters` (line 539)
+
+**Current:** Extracts only `acheteur_id` and `titulaire_id` from URL.
+
+**New:**
+
+- Iterates over `FILTER_PARAMS` to extract all values from `parse_qs`
+- For multi-value params: reads the full list from `parse_qs` (returns lists natively)
+- For number params (`montant_min`, `montant_max`): casts to `float`
+- The guard condition changes from `if acheteur_id or titulaire_id` to "if any URL param is present" — this is necessary so URLs like `?annee=2024&montant_min=10000` (without an ID) work correctly
+- When **any** URL param is present: returns explicit values for all 17 outputs — the URL value for params present, `None`/default for params absent. This ensures "URL replaces all" semantics.
+- When **no** URL params are present: returns `(no_update,) * 17` (preserving local persistence)
+- Radio buttons (`innovant`, `sous_traitance`): value from URL if present, otherwise `"all"` (their default)
+
+### 3. Layout bug fix
+
+Remove the duplicate `dcc.Input(id="observatoire-share-url")` (lines 413-422 — two identical elements).
+
+## Backward Compatibility
+
+Old URLs with only `?acheteur_id=...` or `?titulaire_id=...` continue to work — the new `restore_filters` will read those keys and reset all others to defaults, which is the same effective behavior as before.
+
+Links generated by `add_links()` in `src/utils.py` (used on search results to link to `/observatoire?acheteur_id=...`) are unaffected.
+
+## Test Changes
+
+### Fix broken test `test_010_observatoire_montant_filter`
+
+This test imports `_apply_filters` from `pages.observatoire`, which no longer exists (replaced by `prepare_dashboard_data` in `src/utils.py`). Fix:
+
+- Replace import with `from src.utils import prepare_dashboard_data`
+- Update the call to match `prepare_dashboard_data`'s signature: rename `marche_type` keyword to `type`, and add missing params `objet`, `code_cpv`, `techniques`, `marche_innovant`, `sous_traitance_declaree` (all as `None`)
+
+### New test: multi-param URL round-trip
+
+Add a test that navigates to `/observatoire?annee=2024&acheteur_id=<test_id>&montant_min=10000` and verifies that:
+
+- `dashboard_year` dropdown shows "2024"
+- `dashboard_acheteur_id` input contains the test ID
+- `dashboard_montant_min` input contains "10000"
+
+### Update existing tests
+
+Tests `test_006` and `test_007` validate `acheteur_id` round-trip. These should continue to pass without changes since `acheteur_id` keeps the same URL key.

docs/superpowers/specs/2026-04-15-duckdb-migration-design.md

@@ -0,0 +1,196 @@
+# DuckDB migration — design spec
+
+**Date:** 2026-04-15
+**Branch:** dev
+**Status:** Approved, ready for planning
+
+## Goal
+
+Replace the global Polars dataframes that `src/utils.py` materializes at import time (`df` and the five derived frames, lines 891–913) with a DuckDB database on disk. The main table holds ~1.5M rows from `decp_prod.parquet`. Per-request queries pull only what each page needs, dramatically reducing steady-state RSS memory.
+
+Polars stays the primary API for small result sets and post-processing. DuckDB carries the heavy filtering, joining, and aggregation.
+
+## Approach summary
+
+- **Approach A — compatibility layer.** A new `src/db.py` module exposes a `query_marches(where_sql, params, columns, ...)` helper that runs SQL and returns a `pl.DataFrame`. Most existing `df.filter(pl.col(...) == x)` call sites translate mechanically to `query_marches("col = ?", (x,))`. The shape of downstream Polars code is unchanged.
+- **Two small helpers stay in memory.** `df_acheteurs` and `df_titulaires` (tens of thousands of rows, consumed by the autocomplete search on every keystroke) are kept as module-level Polars frames. They are populated from DuckDB at import time, not from Parquet.
+- **Four derived tables live in DuckDB**, built at startup alongside the main table: `acheteurs_marches`, `titulaires_marches`, `acheteurs_departement`, `titulaires_departement`.
+- **Connection model.** One read-only `duckdb.connect(..., read_only=True)` at module load, shared across the process. `conn.cursor()` per Dash callback for thread-safety. The read-write connection is short-lived and only used during the startup build phase.
+
+## Cache invalidation rule
+
+At startup, rebuild the DuckDB file if:
+
+1. **The DB file does not exist**, OR
+2. **`decp_prod.parquet.mtime > duckdb.mtime`**, **unless** `DEVELOPMENT=true` and `REBUILD_DUCKDB != true` — in which case the DB stays as-is (fast dev reloads).
+
+Production auto-rebuilds when the source Parquet is newer. Development keeps a stable DB across reloads unless the developer explicitly sets `REBUILD_DUCKDB=true` to force a rebuild.
+
+## Concurrency
+
+Multi-worker Gunicorn startup and crashed-mid-build scenarios are handled by a file lock, not by polling for the tmp file's existence:
+
+```python
+with open(DB_PATH.with_suffix(".duckdb.lock"), "w") as lock_fd:
+    fcntl.flock(lock_fd, fcntl.LOCK_EX)          # blocks if another worker is building
+    if should_rebuild(DB_PATH, PARQUET_PATH):
+        build_database(DB_PATH, PARQUET_PATH)
+conn = duckdb.connect(str(DB_PATH), read_only=True)
+```
+
+- Worker A acquires the lock, builds, atomically renames tmp → final, releases the lock.
+- Worker B blocks on `flock`, then re-checks `should_rebuild`, sees the fresh DB, skips building.
+- `fcntl.flock` is auto-released on process death, so a crash never deadlocks the next worker.
+- `build_database` unlinks any pre-existing tmp file before starting (safe because it holds the lock) — handles an abandoned tmp from a crashed previous build.
+
+## Build logic
+
+The build keeps **one source of truth** for transforms by reusing the existing Polars pipeline:
+
+```python
+def build_database(db_path, parquet_path):
+    tmp_path = db_path.with_suffix(".duckdb.tmp")
+    if tmp_path.exists():
+        tmp_path.unlink()
+    frame = get_decp_data()   # existing function in utils.py
+    with duckdb.connect(str(tmp_path)) as w:
+        w.register("frame", frame)
+        w.execute("CREATE TABLE decp AS SELECT * FROM frame")
+        w.execute("CREATE TABLE acheteurs_marches AS "
+                  "SELECT DISTINCT uid, objet, acheteur_id FROM decp "
+                  "ORDER BY acheteur_id")
+        w.execute("CREATE TABLE titulaires_marches AS "
+                  "SELECT DISTINCT uid, objet, titulaire_id FROM decp "
+                  "ORDER BY titulaire_id")
+        w.execute("CREATE TABLE acheteurs_departement AS "
+                  "SELECT DISTINCT acheteur_id, acheteur_nom, acheteur_departement_code "
+                  "FROM decp ORDER BY acheteur_nom")
+        w.execute("CREATE TABLE titulaires_departement AS "
+                  "SELECT DISTINCT titulaire_id, titulaire_nom, titulaire_departement_code "
+                  "FROM decp ORDER BY titulaire_nom")
+    os.replace(tmp_path, db_path)
+```
+
+Why Polars, not SQL, for the row-level transforms:
+
+- `booleans_to_strings` is not a simple cast — it replaces `true`/`false` with `"oui"`/`"non"` on every boolean column. Reimplementing in SQL risks drifting from the Polars version.
+- The null-name replacement (`acheteur_nom`, `titulaire_nom` → `"[Identifiant non reconnu dans la base INSEE]"`) is also easier to keep identical in Polars.
+- `w.register("frame", frame)` is zero-copy. The memory spike is one-time during build and released when the write connection closes.
+
+`os.replace` is atomic on POSIX — the read-only connection that opens next always sees a complete DB.
+
+## Module layout
+
+### New: `src/db.py`
+
+```python
+conn: duckdb.DuckDBPyConnection       # read-only, module-level
+schema: pl.Schema                     # from conn.execute("SELECT * FROM decp LIMIT 0").pl().schema
+
+def get_cursor() -> duckdb.DuckDBPyConnection: ...
+def query_marches(where_sql: str = "TRUE",
+                  params: tuple = (),
+                  columns: list[str] | None = None,
+                  order_by: str | None = None,
+                  limit: int | None = None) -> pl.DataFrame: ...
+def should_rebuild(db_path: Path, parquet_path: Path) -> bool: ...
+def build_database(db_path: Path, parquet_path: Path) -> None: ...
+```
+
+Only imports: `polars`, `duckdb`, `os`, `fcntl`, `pathlib`, `logging`. No app modules — prevents circular imports.
+
+### Changes to `src/utils.py`
+
+- `df: pl.DataFrame = get_decp_data()` — **removed** (after migration).
+- `df_acheteurs`, `df_titulaires` — **kept as Polars globals**, populated via DuckDB at import time. The query mirrors today's `get_org_data(df, org_type)`: select all columns whose name starts with `acheteur_` (or `titulaire_`) except the `_latitude` / `_longitude` pair, plus `COUNT(*) AS "Marchés"`, grouped by the same set. Implementation can either:
+
+  - enumerate the columns by filtering `schema.names()` at import time and build the `SELECT` / `GROUP BY` strings, or
+  - call `get_org_data()` once against a small Polars frame returned by `SELECT <org_ cols> FROM decp`.
+
+  Feeds `search_org` unchanged.
+
+- `df_acheteurs_marches`, `df_titulaires_marches`, `df_acheteurs_departement`, `df_titulaires_departement` — **removed** as Python globals. Call sites query the corresponding DuckDB tables.
+- `schema` — imported from `src/db.py` (stays a `pl.Schema` — so `schema.names()` and dtype lookups both work, no call-site changes beyond `acheteur.py:303`).
+- `columns` — replaced with `schema.names()`.
+- `get_decp_data()` — **kept** (used by `build_database`).
+- `get_org_data()` — can be removed once `df_acheteurs` / `df_titulaires` are populated from DuckDB directly.
+
+### Call-site translations
+
+| Before (Polars global)                                       | After                                                                             |
+| ------------------------------------------------------------ | --------------------------------------------------------------------------------- |
+| `df.filter(pl.col("acheteur_id") == aid)`                    | `query_marches("acheteur_id = ?", (aid,))`                                        |
+| `df.filter(pl.col("uid") == uid).row(0, named=True)`         | `query_marches("uid = ?", (uid,)).row(0, named=True)`                             |
+| `df.select("uid","objet","acheteur_id").filter(...)`         | `query_marches("...", (...), columns=["uid","objet","acheteur_id"])`              |
+| `df.columns`                                                 | `schema.names()`                                                                  |
+| `df_acheteurs_marches.filter(...)`                           | `get_cursor().execute("SELECT ... FROM acheteurs_marches WHERE ...", [...]).pl()` |
+| `pl.DataFrame(schema=df.collect_schema())` (acheteur.py:303) | `pl.DataFrame(schema=schema)`                                                     |
+
+Heavy dashboard aggregations (observatoire, tableau full-scan) use raw SQL via `get_cursor().execute(...).pl()` rather than the helper.
+
+## Configuration
+
+- **`DATA_FILE_PARQUET_PATH`** — unchanged.
+- **DuckDB file location** — computed: `Path(DATA_FILE_PARQUET_PATH).parent / "decp.duckdb"`. No new env var.
+- **`REBUILD_DUCKDB`** — new, optional, default `false`. In development, setting this to `true` forces a rebuild when the parquet is newer.
+- **`DEVELOPMENT`** — unchanged; now also gates the auto-rebuild behavior per the rule above.
+
+## Testing
+
+- `tests/conftest.py` (or a startup hook in `src/db.py`) ensures the test run builds the DuckDB in a temp directory derived from the parquet path — `tests/test.parquet` → `tests/decp.duckdb`. This file is added to `.gitignore`.
+- Tests already set `DEVELOPMENT=true`; they must also set `REBUILD_DUCKDB=true` on cold test runs to force a fresh build from the test parquet.
+- The existing Selenium suite exercises every page and is the primary acceptance signal.
+
+## Migration order
+
+Incremental — `df` global coexists with `src/db.py` until every page is migrated.
+
+1. **Add `src/db.py`** (build, lock, `query_marches`, `schema`). `df` global unchanged.
+2. **Migrate `marche.py`** — single-row lookup by `uid`, one call site.
+3. **Migrate `acheteur.py`, `titulaire.py`** — filter by id.
+4. **Migrate `arbre/departement.py`, `arbre/liste_marches_org.py`** — use the new derived DuckDB tables.
+5. **Migrate `tableau.py`** — may need raw SQL.
+6. **Migrate `observatoire.py`** — heaviest aggregations, most likely raw SQL.
+7. **Migrate `figures.py`** — uses `df` in chart generation.
+8. **Remove** `df`, `df_*_marches`, `df_*_departement` globals, `get_org_data()`, and the `df = get_decp_data()` call from `utils.py`. Move `schema` / `columns` exports to `src/db.py`.
+
+### Verification gates
+
+- `uv run pytest` green after every page migration.
+- Manual smoke test via `uv run run.py` of the migrated page before proceeding.
+- RSS memory measurement (`ps -o rss`) of a cold `gunicorn app:server` with the prod parquet, before and after, to confirm the memory reduction.
+
+## Out of scope
+
+- Changes to `src/cache.py` (flask-caching stays).
+- The in-progress observatoire-localstorage-filters work on `dev`.
+- Schema changes to the parquet.
+- SQL views beyond the four derived tables.
+- Multi-database or replication setups.
+
+## Risks and mitigations
+
+| Risk                                                                          | Mitigation                                                                                                                       |
+| ----------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
+| `booleans_to_strings` reimplemented in SQL and drifts from Polars version     | Transforms stay in Polars via `w.register("frame", frame)`. One source of truth.                                                 |
+| Two Gunicorn workers rebuild concurrently                                     | `fcntl.flock` serializes the build; second worker re-checks and skips.                                                           |
+| Crashed build leaves stale `.tmp` file                                        | Build unlinks any pre-existing tmp before starting (safe under lock).                                                            |
+| `schema` shape change breaks `acheteur.py:303`                                | `schema` stays a `pl.Schema` object, not a list. One call site (`collect_schema()` → module `schema`) updated.                   |
+| Test runs inherit a stale DuckDB from a previous run with a different parquet | Tests force `REBUILD_DUCKDB=true` on cold runs; test DB added to `.gitignore`.                                                   |
+| Read-only connection opened before build finishes in another worker           | Lock held across build + rename; read-only `connect` happens after lock release. Atomic `os.replace` guarantees a complete file. |
+
+## Outcome
+
+### Memory impact
+
+Memory measurement against the production parquet (`decp_prod.parquet`, ~1.5M rows) requires a running gunicorn process with access to the production data file. The measurement was deferred to the post-merge smoke test on the staging server (test.decp.info).
+
+**Expected reduction:** The removed globals (`df`, `df_acheteurs_departement`, `df_titulaires_departement`, `df_acheteurs_marches`, `df_titulaires_marches`) previously materialised the full 1.5M-row Parquet in memory as multiple Polars frames. At ~300 bytes/row × 5 frames, steady-state RSS reduction is estimated at **1–2 GB per worker**. The retained `df_acheteurs` and `df_titulaires` (autocomplete search) represent only the distinct-organisation subset (~tens of thousands of rows) and are negligible.
+
+**What remains in memory:**
+
+- `df_acheteurs` — distinct acheteurs with Marchés count (populated from DuckDB at startup)
+- `df_titulaires` — same for titulaires
+- DuckDB's own page cache (disk-backed, grows under load, evicted by OS)
+
+All per-request data is fetched from DuckDB and discarded after the callback returns.

pyproject.toml

@@ -1,11 +1,9 @@
 [project]
 name = "decp.info"
 description = "Interface d'exploration et d'analyse des marchés publics français."
-version = "2.5.0"
+version = "2.7.2"
 requires-python = ">= 3.10"
-authors = [
+authors = [{ name = "Colin Maudry", email = "colin@colmo.tech" }]
-    { name = "Colin Maudry", email = "colin@colmo.tech" }
-]
 dependencies = [
   "dash==3.4.0",
   "dash[compress]",
@@ -17,10 +15,31 @@ dependencies = [
   "plotly[express]",
   "httpx",
   "pandas",                    # utilisé pour la création de certains graphiques
-    "unidecode"
+  "unidecode",
+  "dash-leaflet",
+  "dash-extensions",
+  "duckdb",
+  "flask-caching",
+  "pyarrow>=23.0.1",
 ]
 
 [project.optional-dependencies]
 dev = [
+  "pytest",
+  "pytest-env",
   "pre-commit",
+  "selenium",
+  "webdriver-manager",
+  "dash[testing]",
+  "fastexcel",
 ]
+
+[tool.pytest.ini_options]
+pythonpath = ["src"]
+testpaths = ["tests"]
+env = [
+  "DATA_FILE_PARQUET_PATH=tests/test.parquet",
+  "DEVELOPMENT=true",
+  "DATA_SCHEMA_PATH=/home/colin/git/decp-processing/dist/schema.json",
+]
+addopts = "-p no:warnings"

src/app.py

@@ -1,5 +1,5 @@
-import logging
 import os
+from shutil import rmtree
 
 import dash_bootstrap_components as dbc
 import tomllib
@@ -7,11 +7,15 @@ from dash import Dash, Input, Output, State, dcc, html, page_container, page_reg
 from dotenv import load_dotenv
 from flask import Response
 
+from src.cache import cache
+from src.utils import DEVELOPMENT
+
 load_dotenv()
 
-development = os.getenv("DEVELOPMENT").lower() == "true"
+# if os.getenv("PYTEST_CURRENT_TEST"):
+#     os.environ["DATA_FILE_PARQUET_PATH"]
 
-meta_tags = [
+META_TAGS = [
     {"name": "viewport", "content": "width=device-width, initial-scale=1"},
     {
         "name": "keywords",
@@ -19,20 +23,32 @@ meta_tags = [
     },
 ]
 
-if development:
+if DEVELOPMENT:
-    meta_tags.append({"name": "robots", "content": "noindex"})
+    META_TAGS.append({"name": "robots", "content": "noindex"})
 
-app = Dash(
+app: Dash = Dash(
     title="decp.info",
     use_pages=True,
     compress=True,
-    meta_tags=meta_tags,
+    meta_tags=META_TAGS,
 )
 
-# COSMO (belle font, blue),
+cache_dir = os.getenv("CACHE_DIR", "/tmp/decp-cache")
-# UNITED (rouge, ubuntu font),
+
-# LUMEN (gros séparateur, blue clair),
+if os.path.exists(cache_dir):
-# SIMPLEX (rouge, séparateur)
+    rmtree(cache_dir)
+
+cache.init_app(
+    app.server,
+    config={
+        "CACHE_TYPE": "FileSystemCache",
+        "CACHE_DIR": cache_dir,
+        "CACHE_DEFAULT_TIMEOUT": int(
+            os.getenv("CACHE_DEFAULT_TIMEOUT", 3600 * 24)
+        ),  # 24h par défaut
+        "CACHE_THRESHOLD": 300,
+    },
+)
 
 
 # robots.txt
@@ -49,7 +65,7 @@ def sitemap():
     base_url = "https://decp.info"
     pages = [
         "/",
-        "/statistiques",
+        "/observatoire",
         "/tableau",
         "/a-propos",
     ]
@@ -63,13 +79,6 @@ def sitemap():
     return Response(xml, mimetype="text/xml")
 
 
-logger = logging.getLogger("decp.info")
-logging.basicConfig(
-    format="%(asctime)s %(levelname)-8s %(message)s",
-    level=logging.INFO,
-    datefmt="%Y-%m-%d %H:%M:%S",
-)
-
 with open("./pyproject.toml", "rb") as f:
     pyproject = tomllib.load(f)
     version = "v" + pyproject["project"]["version"]
@@ -83,6 +92,7 @@ app.index_string = """
         <title>{%title%}</title>
         {%favicon%}
         {%css%}
+        <!-- canonical link -->
     </head>
     <body>
         {%app_entry%}
@@ -155,7 +165,7 @@ navbar = dbc.Navbar(
                         )
                         for page in page_registry.values()
                         if page["name"]
-                        in ["Recherche", "À propos", "Tableau", "Statistiques"]
+                        in ["Recherche", "À propos", "Tableau", "Observatoire"]
                     ],
                     className="ms-auto",
                     navbar=True,

src/assets/css/style.css

@@ -27,18 +27,50 @@ b {
   font-weight: 600 !important;
 }
 
+h1,
+h2,
+h3,
+h4,
+h5 {
+  font-weight: 600;
+}
+
 h3 {
   margin: 36px 0 20px 0;
 }
 
-/* Base Button Styles */
+/* Base Button Styles
 button {
   font-weight: 400;
   background-color: #fff;
   border-radius: 3px;
   appearance: auto;
   border: solid var(--primary-color) 1px;
-  /* couleur thème foncée */
+}  */
+
+button.btn.btn-primary,
+button.show-hide {
+  display: block;
+  border-radius: 3px;
+  outline: 0;
+  color: #fff;
+  border: 0;
+  height: 30px;
+  padding-top: 2px;
+  background-image: linear-gradient(
+    rgb(209, 96, 73),
+    rgb(179, 56, 33) 26%,
+    rgb(159, 36, 22)
+  );
+}
+
+button.btn.btn-primary:hover,
+button.show-hide:hover {
+  background-image: linear-gradient(
+    rgb(239, 126, 103),
+    rgb(209, 86, 63) 26%,
+    rgb(189, 66, 52)
+  );
 }
 
 button[disabled] {
@@ -62,13 +94,6 @@ button:hover:not([disabled]) {
   padding: 28px 24px 0 24px;
 }
 
-.wrapper {
-  display: grid;
-  grid-gap: 10px;
-  margin-bottom: 50px;
-  justify-content: space-between;
-}
-
 #header > * {
   margin: 0 0 20px 0px;
 }
@@ -119,6 +144,10 @@ p.version > a {
   max-width: 900px;
 }
 
+.seeBorder {
+  border: dotted 1px green;
+}
+
 /* --- Search Page --- */
 .tagline {
   text-align: center;
@@ -144,14 +173,22 @@ p.version > a {
   margin-right: 12px;
 }
 
-.results_acheteur {
+/* --- Dashboard inputs --- */
-  grid-column: 1;
+
-  grid-row: 1;
+.Select--multi .Select-value {
+  color: var(--primary-color) !important;
+  background-color: rgba(255, 240, 240, 0.4) !important;
 }
 
-.results_titulaire {
+#filters .row > * {
-  grid-column: 2;
+  margin-bottom: 6px;
-  grid-row: 1;
+}
+
+#filters input[type="text"],
+#filters input[type="number"] {
+  border: 1px #ccc solid;
+  border-radius: 3px;
+  padding-left: 8px;
 }
 
 /* --- Tables (Dash & Custom) --- */
@@ -160,12 +197,13 @@ p.version > a {
 .table-menu {
   font-size: 16px;
   margin: 12px 0 12px 0;
-  height: 36px;
+  display: flex;
+  align-items: center;
+  flex-wrap: wrap;
 }
 
 .table-menu > * {
   margin: 8px 16px 8px 0;
-  float: left;
 }
 
 #source_table {
@@ -207,17 +245,21 @@ table.cell-table th {
 .dash-table-container
   .dash-spreadsheet-container
   .dash-spreadsheet-inner
-  th.dash-header {
+  th.dash-header,
+.dash-table-container
+  .dash-spreadsheet-container
+  .dash-spreadsheet-inner
+  th.dash-select-header {
   margin: 0;
   color: white;
   font-family: "Inter", sans-serif;
   text-align: left;
   font-weight: 600;
-  padding-right: 12px;
+  padding: 2px 12px 4px 2px;
   border: 1px solid rgb(179, 56, 33) !important;
   background-color: rgb(179, 56, 33);
   border-bottom: none !important;
-  height: 34px;
+  height: 32px;
 }
 
 .dash-table-container
@@ -225,7 +267,13 @@ table.cell-table th {
   .dash-spreadsheet-inner
   table.cell-table
   tr:first-of-type
-  th.dash-header:first-of-type {
+  th.dash-header:first-of-type,
+.dash-table-container
+  .dash-spreadsheet-container
+  .dash-spreadsheet-inner
+  table.cell-table
+  tr:first-of-type
+  th.dash-select-header:first-of-type {
   border-top-left-radius: 3px !important;
 }
 
@@ -258,6 +306,15 @@ table.cell-table th {
   vertical-align: center;
 }
 
+.dash-table-container
+  .dash-spreadsheet-container
+  .dash-spreadsheet-inner
+  .cell-table
+  .dash-filter
+  input[type="text"]::placeholder {
+  color: #999;
+}
+
 .dash-filter--case {
   display: none;
 }
@@ -307,22 +364,44 @@ table.cell-table th {
   display: none;
 }
 
-.show-hide {
+button.show-hide {
   position: relative;
   width: 180px;
   margin: 0 0 10px 0;
+  display: none;
 }
 
+/*
 .show-hide::before {
   background: inherit;
   content: "Colonnes affichées";
   position: absolute;
   left: 5px;
   right: 5px;
+
+
+#column_list .show-hide,
+#table .show-hide {
+  display: none;
 }
 
 .show-hide-menu-item > input {
   margin-right: 10px;
+} */
+
+#btn-copy-url:before {
+}
+
+/* Dropdowns */
+.Select-placeholder {
+  color: #333 !important;
+}
+
+/* Checkboxes */
+
+input[type="checkbox"] {
+  height: 17px;
+  width: 17px;
 }
 
 /* Tooltips */
@@ -345,40 +424,15 @@ table.cell-table th {
 }
 
 /* --- Organization Cards (Grid Items) --- */
-.org_title {
-  grid-column: 1 / 3;
-  grid-row: 1;
-}
 
-.org_year {
+#cards .card {
-  grid-column: 3;
+  margin-bottom: 16px;
-  grid-row: 1;
-}
-
-.org_infos {
-  grid-column: 1;
-  grid-row: 2;
 }
 
 .org_infos > p {
   margin: 8px 0;
 }
 
-.org_stats {
-  grid-column: 2;
-  grid-row: 2;
-}
-
-.org_map {
-  grid-column: 3;
-  grid-row: 2;
-}
-
-.org_top {
-  grid-column: 1/3;
-  grid-row: 3;
-}
-
 /* --- About Page (A Propos) --- */
 .a-propos-container {
   display: flex;
@@ -478,3 +532,13 @@ summary > h4 {
     display: none;
   }
 }
+
+input[type="number"]::-webkit-outer-spin-button,
+input[type="number"]::-webkit-inner-spin-button {
+  -webkit-appearance: none;
+  margin: 0;
+}
+
+input[type="number"] {
+  -moz-appearance: textfield;
+}

src/assets/dash_clientside.js

@@ -1,4 +1,31 @@
 window.dash_clientside = Object.assign({}, window.dash_clientside, {
+  leaflet: {
+    pointToLayer: function (feature, latlng, context) {
+      return L.circleMarker(latlng, {
+        radius: 5,
+        fillColor: feature.properties.marker_color,
+        color: "white",
+        weight: 1,
+        opacity: 1,
+        fillOpacity: 0.8,
+      }).bindTooltip(feature.properties.tooltip);
+    },
+    clusterToLayer: function (feature, latlng, index, context) {
+      console.log(feature);
+      console.log(index);
+      console.log(context);
+
+      const count = feature.properties.point_count;
+      const size = count < 100 ? 30 : count < 1000 ? 40 : 50;
+      const color = "#555"; // Default cluster color
+      const icon = L.divIcon({
+        html: `<div style="background-color: ${context.fillColor}; width: ${size}px; height: ${size}px; border-radius: 50%; display: flex; align-items:center; justify-content:center; color: white; border: 2px solid white; font-weight: bold;">${count}</div>`,
+        className: "marker-cluster",
+        iconSize: L.point(size, size),
+      });
+      return L.marker(latlng, { icon: icon });
+    },
+  },
   clientside: {
     clean_filters: function (trigger) {
       if (!trigger) {

src/cache.py

@@ -0,0 +1,4 @@
+from flask_caching import Cache
+
+# Isolé dans un fichier dédié pour éviter les imports circulaires
+cache = Cache()

src/callbacks.py

@@ -1,36 +0,0 @@
-import polars as pl
-from dash import html
-
-from src.figures import DataTable
-from utils import add_links_in_dict, format_values, setup_table_columns
-
-
-def get_top_org_table(data, org_type: str):
-    dff = pl.DataFrame(data, strict=False, infer_schema_length=5000)
-    if dff.height == 0:
-        return html.Div()
-
-    dff = dff.select(
-        ["uid", f"{org_type}_id", f"{org_type}_nom", "titulaire_distance", "montant"]
-    )
-    dff_nb = dff.group_by(
-        f"{org_type}_id", f"{org_type}_nom", "titulaire_distance"
-    ).agg(pl.len().alias("Attributions"), pl.sum("montant").alias("montant"))
-    dff_nb = dff_nb.sort(by="montant", descending=True, nulls_last=True)
-    dff_nb = dff_nb.cast(pl.String)
-    dff_nb = dff_nb.fill_null("")
-    dff_nb = format_values(dff_nb)
-    columns, tooltip = setup_table_columns(
-        dff_nb, hideable=False, exclude=[f"{org_type}_id"], new_columns=["Attributions"]
-    )
-    data = dff_nb.to_dicts()
-    data = add_links_in_dict(data, f"{org_type}")
-
-    return DataTable(
-        dtid=f"top10_{org_type}",
-        data=data,
-        page_action="native",
-        page_size=10,
-        columns=columns,
-        tooltip_header=tooltip,
-    )

src/db.py

@@ -0,0 +1,162 @@
+import fcntl
+import os
+from pathlib import Path
+from time import sleep
+
+import duckdb
+import polars as pl
+import polars.selectors as cs
+from polars.exceptions import ComputeError
+
+from src.utils import logger
+
+
+def should_rebuild(db_path: Path, parquet_path: Path) -> bool:
+    db_path = Path(db_path)
+    parquet_path = Path(parquet_path)
+    if not db_path.exists():
+        return True
+    dev = os.getenv("DEVELOPMENT", "False").lower() == "true"
+    force = os.getenv("REBUILD_DUCKDB", "False").lower() == "true"
+    if dev and not force:
+        return False
+    return parquet_path.stat().st_mtime > db_path.stat().st_mtime
+
+
+def _load_source_frame(parquet_path: Path) -> pl.DataFrame:
+    """Read the source parquet and apply the row-level transforms.
+
+    Kept here (not in utils.py) so src.db has no dependency on utils.
+    Mirrors the behavior previously in utils.get_decp_data().
+    """
+    try:
+        lff: pl.LazyFrame = pl.scan_parquet(str(parquet_path))
+    except ComputeError:
+        logger.info("Lecture du parquet échouée, nouvelle tentative dans 10s...")
+        sleep(10)
+        lff = pl.scan_parquet(str(parquet_path))
+
+    lff = lff.sort(by=["dateNotification", "uid"], descending=True, nulls_last=True)
+    lff = lff.filter(pl.col("donneesActuelles")).drop("donneesActuelles")
+
+    # booleans_to_strings: true → "oui", false → "non"
+    lff = lff.with_columns(
+        pl.col(cs.Boolean)
+        .cast(pl.String)
+        .str.replace("true", "oui")
+        .str.replace("false", "non")
+    )
+
+    for col in ["acheteur_nom", "titulaire_nom"]:
+        lff = lff.with_columns(
+            pl.when(pl.col(col).is_null())
+            .then(pl.lit("[Identifiant non reconnu dans la base INSEE]"))
+            .otherwise(pl.col(col))
+            .name.keep()
+        )
+
+    return lff.collect()
+
+
+def build_database(db_path: Path, parquet_path: Path) -> None:
+    """Build the DuckDB database atomically under an exclusive lock.
+
+    Caller MUST hold the fcntl.flock on the .lock file.
+    """
+    db_path = Path(db_path)
+    parquet_path = Path(parquet_path)
+    tmp_path = db_path.with_suffix(".duckdb.tmp")
+    staging_parquet = db_path.with_suffix(".staging.parquet")
+    if tmp_path.exists():
+        tmp_path.unlink()
+
+    logger.info(f"Construction de la base DuckDB à partir de {parquet_path}...")
+    frame = _load_source_frame(parquet_path)
+
+    # Write transformed frame as parquet so DuckDB can read it natively
+    # (avoids pyarrow dependency for the Polars→DuckDB handoff)
+    frame.write_parquet(str(staging_parquet))
+    try:
+        with duckdb.connect(str(tmp_path)) as w:
+            w.execute(
+                f"CREATE TABLE decp AS SELECT * FROM read_parquet('{staging_parquet}')"
+            )
+            w.execute(
+                "CREATE TABLE acheteurs_marches AS "
+                "SELECT DISTINCT uid, objet, acheteur_id FROM decp "
+                "ORDER BY acheteur_id"
+            )
+            w.execute(
+                "CREATE TABLE titulaires_marches AS "
+                "SELECT DISTINCT uid, objet, titulaire_id FROM decp "
+                "ORDER BY titulaire_id"
+            )
+            w.execute(
+                "CREATE TABLE acheteurs_departement AS "
+                "SELECT DISTINCT acheteur_id, acheteur_nom, acheteur_departement_code "
+                "FROM decp ORDER BY acheteur_nom"
+            )
+            w.execute(
+                "CREATE TABLE titulaires_departement AS "
+                "SELECT DISTINCT titulaire_id, titulaire_nom, titulaire_departement_code "
+                "FROM decp ORDER BY titulaire_nom"
+            )
+    finally:
+        if staging_parquet.exists():
+            staging_parquet.unlink()
+
+    os.replace(tmp_path, db_path)
+    logger.info(f"Base DuckDB construite : {db_path}")
+
+
+def _resolve_db_path() -> Path:
+    parquet = os.getenv("DATA_FILE_PARQUET_PATH")
+    if not parquet:
+        raise RuntimeError("DATA_FILE_PARQUET_PATH is not set")
+    return Path(parquet).parent / "decp.duckdb"
+
+
+def _ensure_database() -> Path:
+    db_path = _resolve_db_path()
+    parquet_path = Path(os.getenv("DATA_FILE_PARQUET_PATH"))
+    lock_path = db_path.with_suffix(".duckdb.lock")
+
+    with open(lock_path, "w") as lock_fd:
+        fcntl.flock(lock_fd, fcntl.LOCK_EX)
+        if should_rebuild(db_path, parquet_path):
+            build_database(db_path, parquet_path)
+        else:
+            logger.debug("Base de données déjà disponible et à jour.")
+    return db_path
+
+
+DB_PATH = _ensure_database()
+conn: duckdb.DuckDBPyConnection = duckdb.connect(str(DB_PATH), read_only=True)
+schema: pl.Schema = conn.execute("SELECT * FROM decp LIMIT 0").pl().schema
+
+
+def get_cursor() -> duckdb.DuckDBPyConnection:
+    """Return a per-request cursor that shares the process-wide connection."""
+    return conn.cursor()
+
+
+def query_marches(
+    where_sql: str = "TRUE",
+    params: tuple = (),
+    columns: list[str] | None = None,
+    order_by: str | None = None,
+    limit: int | None = None,
+) -> pl.DataFrame:
+    """Run a parameterized SELECT against the decp table and return Polars.
+
+    `where_sql` and `order_by` are trusted SQL fragments (callers are internal
+    code, never user input). `params` values are passed through DuckDB's
+    parameter binding.
+    """
+    cols = ", ".join(columns) if columns else "*"
+    sql = f"SELECT {cols} FROM decp WHERE {where_sql}"
+    if order_by:
+        sql += f" ORDER BY {order_by}"
+    if limit is not None:
+        sql += f" LIMIT {int(limit)}"
+    return get_cursor().execute(sql, list(params)).pl()

src/figures.py

@@ -1,62 +1,20 @@
-import json
+from datetime import datetime
 from typing import Literal
+from urllib.error import HTTPError, URLError
 
+import dash_bootstrap_components as dbc
+import dash_leaflet as dl
+import dash_leaflet.express as dlx
+import numpy as np
 import plotly.express as px
 import plotly.graph_objects as go
 import polars as pl
 from dash import dash_table, dcc, html
+from dash_extensions.javascript import Namespace
 
-from src.utils import format_number
+from src.db import schema
-
+from src.utils.data import DATA_SCHEMA, DEPARTEMENTS_GEOJSON
-
+from src.utils.table import add_links, format_number, setup_table_columns
-def get_map_count_marches(df: pl.DataFrame):
-    lf = df.lazy()
-    lf = lf.with_columns(
-        pl.col("lieuExecution_code").str.head(2).str.zfill(2).alias("Département")
-    )
-    lf = (
-        lf.select(["uid", "Département"])
-        .drop_nulls()
-        .unique(subset="uid")
-        .group_by("Département")
-        .len("uid")
-    )
-    # Suppression des infos pour les DOM/TOM pour l'instant
-    lf = lf.remove(pl.col("Département").is_in(["97", "98"]))
-
-    with open("./data/departements-1000m.geojson") as f:
-        departements = json.load(f)
-
-    # Ajout de feature.id
-    for f in departements["features"]:
-        f["id"] = f["properties"]["code"]
-
-    df = lf.collect(engine="streaming")
-
-    fig = px.choropleth(
-        df,
-        geojson=departements,
-        locations="Département",
-        color="uid",
-        color_continuous_scale="Reds",
-        title="Nombres de marchés attribués par département (lieu d'exécution)",
-        range_color=(df["uid"].min(), df["uid"].max()),
-        labels={"uid": "Marchés attribués"},
-        scope="europe",
-        width=900,
-        height=700,
-    )
-
-    fig.update_geos(fitbounds="locations", visible=False)
-    fig.update_layout(
-        mapbox={
-            "style": "carto-positron",
-            "center": {"lon": 10, "lat": 10},
-            "zoom": 1,
-            "domain": {"x": [0, 1], "y": [0, 1]},
-        }
-    )
-    return fig
 
 
 def get_yearly_statistics(statistics, today_str) -> html.Div:
@@ -77,11 +35,11 @@ def get_yearly_statistics(statistics, today_str) -> html.Div:
             }
         )
 
-    df = pl.DataFrame(data)
+    dff = pl.DataFrame(data)
 
     # Create Dash DataTable
     table = dash_table.DataTable(
-        data=df.to_dicts(),
+        data=dff.to_dicts(),
         columns=[
             {"name": "Année", "id": "Année"},
             {"name": "Marchés et accord-cadres", "id": "Marchés et accord-cadres"},
@@ -91,32 +49,27 @@ def get_yearly_statistics(statistics, today_str) -> html.Div:
         page_size=10,
         sort_action="none",
         filter_action="none",
-        style_cell={
+        style_header={"fontFamily": "Inter", "fontSize": "16px"},
-            "border": "solid 1px rgb(179, 56, 33)",
+        style_cell={"fontFamily": "Inter", "fontSize": "16px"},
-        },
-        style_header={
-            "border": "solid 1px rgb(179, 56, 33)",
-            "backgroundColor": "rgb(179, 56, 33)",
-            "color": "white",
-        },
     )
 
     return html.Div(children=table, className="marches_table")
 
 
-def get_barchart_sources(df_source: pl.DataFrame, type_date: str):
+def get_barchart_sources(lff: pl.LazyFrame, type_date: str):
-    lf = df_source.lazy()
     labels = {
         "dateNotification": "notification",
         "datePublicationDonnees": "publication des données",
     }
 
-    lf = lf.select("uid", type_date, "sourceDataset")
+    now_year = datetime.now().year
 
-    lf = lf.unique("uid")
+    lff = lff.select("uid", type_date, "sourceDataset")
+
+    lff = lff.unique("uid")
 
     # Rassemblement des datasets Atexo pour ne pas surcharger le graphique
-    lf = lf.with_columns(
+    lff = lff.with_columns(
         pl.when(pl.col("sourceDataset").str.starts_with("atexo"))
         .then(pl.lit("plateformes atexo"))
         .otherwise(pl.col("sourceDataset"))
@@ -124,38 +77,33 @@ def get_barchart_sources(df_source: pl.DataFrame, type_date: str):
     )
 
     # Rassemblement des datasets AWS pour ne pas surcharger le graphique
-    lf = lf.with_columns(
+    lff = lff.with_columns(
         pl.when(pl.col("sourceDataset").str.contains(r"aws|marches\-publics.info"))
         .then(pl.lit("aws"))
         .otherwise(pl.col("sourceDataset"))
         .alias("sourceDataset")
     )
 
-    lf = lf.with_columns(pl.col(type_date).dt.year().alias("annee"))
+    lff = lff.with_columns(pl.col(type_date).dt.year().alias("annee"))
-    lf = lf.filter(
+    lff = lff.filter(
-        pl.col(type_date).is_not_null() & pl.col("annee").is_between(2019, 2025)
+        pl.col(type_date).is_not_null() & pl.col("annee").is_between(2019, now_year)
     )
-    lf = lf.with_columns(pl.col(type_date).cast(pl.String).str.head(7))
+    lff = lff.with_columns(pl.col(type_date).cast(pl.String).str.head(7))
-    lf = (
+    lff = (
-        lf.group_by([type_date, "sourceDataset"])
+        lff.group_by([type_date, "sourceDataset"])
         .len()
         .sort(by=[type_date, "len"], descending=True)
     )
 
-    # lf = lf.with_columns(
+    lff = lff.sort(by=["sourceDataset"], descending=False)
-    #     pl.when(pl.col("sourceDataset").is_null()).then(
-    #         pl.lit("Source inconnue")).alias("sourceDataset")
-    #     )
 
-    lf = lf.sort(by=["sourceDataset"], descending=False)
+    dff: pl.DataFrame = lff.collect(engine="streaming")
-    df: pl.DataFrame = lf.collect(engine="streaming")
 
     fig = px.bar(
-        df,
+        dff,
         x=type_date,
         y="len",
         color="sourceDataset",
-        title=f"Nombre de marchés attribués par date de {labels[type_date]} et source de données",
         labels={
             "len": "Nombre de marchés",
             type_date: f"Mois de {labels[type_date]}",
@@ -163,12 +111,17 @@ def get_barchart_sources(df_source: pl.DataFrame, type_date: str):
         },
     )
 
-    return fig
+    graph = dcc.Graph(figure=fig)
+
+    return graph
 
 
 def get_sources_tables(source_path) -> html.Div:
-    df = pl.read_csv(source_path)
+    try:
-    df = df.with_columns(
+        dff = pl.read_csv(source_path)
+    except (URLError, HTTPError):
+        return html.Div("Erreur de connexion")
+    dff = dff.with_columns(
         (
             pl.lit('<a href = "')
             + pl.col("url")
@@ -177,8 +130,8 @@ def get_sources_tables(source_path) -> html.Div:
             + pl.lit("</a>")
         ).alias("nom")
     )
-    df = df.drop("url", "unique")
+    dff = dff.drop("url", "unique")
-    df = df.sort(by=["nb_marchés"], descending=True)
+    dff = dff.sort(by=["nb_marchés"], descending=True)
 
     columns = {
         "nom": "Nom de la source",
@@ -190,7 +143,7 @@ def get_sources_tables(source_path) -> html.Div:
 
     datatable = dash_table.DataTable(
         id="source_table",
-        data=df.to_dicts(),
+        data=dff.to_dicts(),
         columns=[
             {
                 "name": columns[i],
@@ -199,7 +152,7 @@ def get_sources_tables(source_path) -> html.Div:
                 "type": "text",
                 "format": {"nully": "N/A"},
             }
-            for i in df.schema.names()
+            for i in dff.schema.names()
         ],
         style_cell_conditional=[
             {
@@ -213,11 +166,8 @@ def get_sources_tables(source_path) -> html.Div:
         ],
         sort_action="native",
         markdown_options={"html": True},
-        style_header={
+        style_header={"fontFamily": "Inter", "fontSize": "16px"},
-            "border": "solid 1px rgb(179, 56, 33)",
+        style_cell={"fontFamily": "Inter", "fontSize": "16px"},
-            "backgroundColor": "rgb(179, 56, 33)",
-            "color": "white",
-        },
     )
 
     return html.Div(children=datatable)
@@ -253,6 +203,7 @@ def point_on_map(lat, lon):
     fig.update_layout(map_center={"lat": 46.6, "lon": 1.89}, map_zoom=4)
 
     graph = dcc.Graph(id="map", figure=fig)
+    graph = html.Div(style={"width": "400px"})
     return graph
 
 
@@ -267,10 +218,12 @@ class DataTable(dash_table.DataTable):
         page_action: Literal["native", "custom", "none"] = "native",
         sort_action: Literal["native", "custom", "none"] = "native",
         filter_action: Literal["native", "custom", "none"] = "native",
+        style_cell_conditional: list | None = None,
+        style_cell: dict | None = None,
         **kwargs,
     ):
         # Styles de base
-        style_cell_conditional = [
+        style_cell_conditional_common = [
             {
                 "if": {"column_id": "objet"},
                 "minWidth": "350px",
@@ -298,20 +251,29 @@ class DataTable(dash_table.DataTable):
                 "lineHeight": "18px",
                 "whiteSpace": "normal",
             },
-            {
-                "if": {"column_id": "montant"},
-                "textAlign": "right",
-            },
-            {
-                "if": {"column_id": "dureeMois"},
-                "textAlign": "right",
-            },
-            {
-                "if": {"column_id": "titulaire_distance"},
-                "textAlign": "right",
-            },
         ]
 
+        style_cell_common = {"fontFamily": "Inter", "fontSize": "16px"}
+
+        for key in DATA_SCHEMA.keys():
+            field = DATA_SCHEMA[key]
+            if field["type"] in ["number", "integer"]:
+                rule = {
+                    "if": {"column_id": field["name"]},
+                    "textAlign": "right",
+                    # "fontFamily": "Fira Code",
+                }
+                style_cell_conditional_common.append(rule)
+
+        style_cell_conditional = (
+            style_cell_conditional or []
+        ) + style_cell_conditional_common
+        if style_cell:
+            style_cell.update(style_cell_common)
+        else:
+            style_cell = style_cell_common
+        style_header = style_cell
+
         # Initialisation de la classe parente avec les arguments
         super().__init__(
             id=dtid,
@@ -323,16 +285,17 @@ class DataTable(dash_table.DataTable):
             page_action=page_action,
             filter_options={
                 "case": "insensitive",
-                "placeholder_text": "",
+                "placeholder_text": "Filtre de colonne...",
             },
             sort_action=sort_action,
             sort_mode="multi",
-            sort_by=[],
             row_deletable=False,
             page_current=0,
             style_cell_conditional=style_cell_conditional,
             data_timestamp=0,
             markdown_options={"html": True},
+            style_header=style_header,
+            style_cell=style_cell,
             tooltip_duration=8000,
             tooltip_delay=350,
             hidden_columns=hidden_columns,
@@ -340,40 +303,24 @@ class DataTable(dash_table.DataTable):
         )
 
 
-def get_duplicate_matrix() -> html.Div:
+def get_duplicate_matrix() -> dcc.Graph:
     """
     Fonction développée avec l'aide de la LLM Euria d'Infomaniak.
     :return:
     """
-    result_df = pl.read_parquet(
+    lff = pl.scan_parquet(
         "https://www.data.gouv.fr/api/1/datasets/r/a545bf6c-8b24-46ed-b49f-a32bf02eaffa"
     ).sort("sourceDataset")
-    result_df = result_df.select(
+    lff = lff.select(
-        ["sourceDataset", "unique"] + sorted(result_df.columns[2:])
+        ["sourceDataset", "unique"] + sorted(lff.collect_schema().names()[2:])
     )
 
-    description = dcc.Markdown("""
+    dff = lff.collect()
-    Ce graphique illustre les doublons de marchés publics entre sources, c'est-à-dire la proportion de marchés publiés par plus d'une source. Il s'appuie sur les identifiants `uid` qui sont pour chaque marché la concaténation du SIRET de l'acheteur et de l'identifiant interne du marché.
-
-    **Comment lire ce graphique ?**
-
-    On part des codes de sources de données en ordonnée. Ces jeux de données sont documentés dans [À propos](/a-propos#sources).
-
-    La première colonne (**unique**) représente le pourcentage de marchés fournis par cette source qui sont uniquement disponibles dans cette source. Plus le rouge est foncé, plus important est le pourcentage. Donc, à l'inverse, plus le rouge est clair dans la première colonne, plus la source en ordonnée a des marchés en commun avec d'autres sources, et donc plus on trouvera sur la même ligne d'autres cases plus ou moins foncées qui indiqueront avec quelles autres sources cette source partage des marchés.
-
-    Passez votre souris sur une case pour avoir les pourcentages exacts. À noter que ces statistiques sont produites avant le dédoublonnement qui a lieu avant la publication en Open Data et sur ce site.""")
-
-    # Assuming result_df is your DataFrame with structure:
-    # | sourceDataset | unique | dataset1 | dataset2 | dataset3 |
-    # |---------------|--------|----------|----------|----------|
-    # | dataset1      | 0.8    |          | 0.15     | 0.2      |
-    # | dataset2      | 0.75   | 0.15     |          | 0.12     |
-    # | dataset3      | 0.85   | 0.2      | 0.12     |          |
 
     # Extract data
-    z_data = result_df.select(pl.all().exclude("sourceDataset")).fill_null(0).to_numpy()
+    z_data = dff.select(pl.all().exclude("sourceDataset")).fill_null(0).to_numpy()
-    x_labels = result_df.columns[1:]  # columns after "sourceDataset"
+    x_labels = dff.columns[1:]  # columns after "sourceDataset"
-    y_labels = result_df["sourceDataset"].to_list()
+    y_labels = dff["sourceDataset"].to_list()
 
     # Create heatmap
     fig = go.Figure(
@@ -381,12 +328,6 @@ def get_duplicate_matrix() -> html.Div:
             z=z_data,
             x=x_labels,
             y=y_labels,
-            # colorscale=[
-            #     [0, "white"],  # 0% → white
-            #     [0.10, "lightblue"],  # 1% → light blue (soft start)
-            #     [0.50, "steelblue"],  # 50% → medium blue
-            #     [1, "darkblue"],  # 100% → dark blue
-            # ],
             colorscale=[
                 [0.0, "white"],  # 0% → white
                 [0.10, "lightsalmon"],  # 10% → light warm tone
@@ -394,12 +335,10 @@ def get_duplicate_matrix() -> html.Div:
             ],
             zmin=0,
             zmax=1,
-            # texttemplate="%{z:.0%}",  # Format as percentage
-            # textfont={"size": 10, "color": "black"},  # Smaller font
             hoverongaps=False,
             showscale=True,
             hovertemplate=(
-                "<b>%{z:.0%}</b> des marchés de <b>%{y}</b> sont également présents dans <b>%{x}</b>"
+                "<b>%{z:.0%}</b> des marchés présents dans <b>%{y}</b> sont également présents dans <b>%{x}</b>"
             ),
         )
     )
@@ -418,10 +357,500 @@ def get_duplicate_matrix() -> html.Div:
         margin=dict(l=100, r=50, t=80, b=100),  # Add margin for labels
     )
 
-    return html.Div(
+    return dcc.Graph(figure=fig)
-        children=[
+
-            html.H3("Doublons de marchés entre les sources"),
+
-            description,
+def get_geographic_maps(dff: pl.DataFrame) -> list[dbc.Col] | list:
-            dcc.Graph(figure=fig),
+    """
-        ]
+    Génère les cartes géographiques pour l'hexagone et les DOM-TOM.
+    """
+
+    regions: dict = {
+        "Hexagone": {
+            "coordinates": [46.6, 2.2],
+            "zoom_leaflet": 5,
+            "zoom_chloropleth": 1,
+            "name": "Hexagone",
+        },
+        "971": {
+            "coordinates": [16.23, -61.55],
+            "zoom_leaflet": 9,
+            "zoom_chloropleth": 1,
+            "name": "Guadeloupe",
+        },
+        "972": {
+            "coordinates": [14.64, -61.02],
+            "zoom_leaflet": 10,
+            "zoom_chloropleth": 1,
+            "name": "Martinique",
+        },
+        "973": {
+            "coordinates": [3.93, -53.12],
+            "zoom_leaflet": 7,
+            "zoom_chloropleth": 1,
+            "name": "Guyane",
+        },
+        "974": {
+            "coordinates": [-21.11, 55.53],
+            "zoom_leaflet": 9,
+            "zoom_chloropleth": 1,
+            "name": "La Réunion",
+        },
+        "976": {
+            "coordinates": [-12.82, 45.16],
+            "zoom_leaflet": 10,
+            "zoom_chloropleth": 1,
+            "name": "Mayotte",
+        },
+    }
+
+    def make_map_data(region_code: str) -> tuple[list, str | None]:
+        lff: pl.LazyFrame = dff.lazy()
+        if region_code == "Hexagone":
+            lff = lff.filter(
+                (pl.col("acheteur_departement_code").str.len_chars() == 2)
+                & (pl.col("titulaire_departement_code").str.len_chars() == 2)
+            )
+        else:
+            lff = lff.filter(
+                (pl.col("acheteur_departement_code") == code)
+                | (pl.col("titulaire_departement_code") == code)
+            )
+
+        nb_marches = lff.select("uid").collect()["uid"].n_unique()
+
+        if nb_marches == 0:
+            return [], None
+
+        dfs = []
+
+        if (code == "Hexagone" and nb_marches > 30000) or (
+            code != "Hexagone" and nb_marches > 10000
+        ):
+            _map_type: str = "chloropleth"
+
+            lff = lff.rename({"acheteur_departement_code": "Département"})
+            lff = (
+                lff.select(["uid", "Département"])
+                .drop_nulls()
+                .group_by("uid")
+                .agg(pl.col("Département").first())
+                .group_by("Département")
+                .len("uid")
+            )
+            dfs.append(lff.collect())
+        else:
+            _map_type: str = "clusters"
+            for org_type in ["acheteur", "titulaire"]:
+                lff_org = (
+                    lff.select(
+                        "uid",
+                        f"{org_type}_longitude",
+                        f"{org_type}_latitude",
+                        f"{org_type}_nom",
+                    )
+                    .group_by(
+                        f"{org_type}_longitude",
+                        f"{org_type}_latitude",
+                        f"{org_type}_nom",
+                    )
+                    .len("nb_marches")
+                    .filter(
+                        pl.col(f"{org_type}_latitude").is_not_null()
+                        & pl.col(f"{org_type}_longitude").is_not_null()
+                    )
+                )
+
+                markers = []
+
+                # Couleurs accessibles (Okabe-Ito)
+                colors = {
+                    "acheteur": "#E69F00",  # orange
+                    "titulaire": "#56B4E9",  # bleu ciel
+                }
+
+                for row in lff_org.collect().to_dicts():
+                    markers.append(
+                        {
+                            "lat": row[f"{org_type}_latitude"],
+                            "lon": row[f"{org_type}_longitude"],
+                            "tooltip": f"{row[f'{org_type}_nom']} ({row['nb_marches']} marchés)",
+                            "marker_color": colors[org_type],
+                        }
+                    )
+                dfs.append(markers)
+
+        return dfs, _map_type
+
+    cols = []
+
+    for code in regions.keys():
+        regions[code]["data"], map_type = make_map_data(code)
+
+        if map_type == "chloropleth":
+            map_graph = make_chloropleth_map(regions[code])
+        elif map_type == "clusters":
+            map_graph = make_clusters_map(regions[code])
+        elif map_type is None:
+            continue
+        else:
+            raise ValueError(f"Map type '{map_type}' not recognised")
+
+        lg, xl = (12, 8) if code == "Hexagone" else (6, 4)
+
+        col = make_card(regions[code]["name"], fig=map_graph, lg=lg, xl=xl)
+        cols.append(col)
+
+    return cols
+
+
+def make_chloropleth_map(region: dict) -> dcc.Graph:
+    df_map = region["data"][0]
+
+    fig = px.choropleth(
+        df_map,
+        geojson=DEPARTEMENTS_GEOJSON,
+        locations="Département",
+        color="uid",
+        color_continuous_scale="Reds",
+        range_color=(df_map["uid"].min(), df_map["uid"].max()),
+        labels={"uid": "Marchés attribués"},
+        scope="europe",
+    )
+
+    fig.update_geos(fitbounds="locations", visible=False)
+    fig.update_layout(
+        mapbox={
+            "style": "carto-positron",
+            "center": {"lon": 10, "lat": 10},
+            "zoom": 8,
+            "domain": {"x": [0, 1], "y": [0, 1]},
+        }
+    )
+
+    graph = dcc.Graph(figure=fig, config={"displayModeBar": False})
+    return graph
+
+
+def make_clusters_map(region: dict) -> dl.Map:
+    # JavaScript functions for styling
+    ns = Namespace("dash_clientside", "leaflet")
+    point_to_layer = ns("pointToLayer")
+    cluster_to_layer = ns("clusterToLayer")
+
+    name = region["name"]
+
+    # Données de la région
+    region_acheteurs = region["data"][0]
+    region_titulaires = region["data"][1]
+
+    # Couleurs
+    color_acheteur = region_acheteurs[0]["marker_color"]
+    color_titulaire = region_titulaires[0]["marker_color"]
+
+    acheteurs_geojson_data = dlx.dicts_to_geojson(region_acheteurs)
+    titulaires_geojson_data = dlx.dicts_to_geojson(region_titulaires)
+
+    center, zoom = region["coordinates"], region["zoom_leaflet"]
+    region_id = name.lower().replace(" ", "-")
+    leaflet_map = dl.Map(
+        [
+            dl.TileLayer(),
+            dl.GeoJSON(
+                data=titulaires_geojson_data,
+                cluster=True,
+                zoomToBoundsOnClick=True,
+                pointToLayer=point_to_layer,
+                clusterToLayer=cluster_to_layer,
+                id=f"geojson-{region_id}-titulaires",
+                options={"fillColor": color_titulaire},
+            ),
+            dl.GeoJSON(
+                data=acheteurs_geojson_data,
+                cluster=True,
+                zoomToBoundsOnClick=True,
+                pointToLayer=point_to_layer,
+                clusterToLayer=cluster_to_layer,
+                id=f"geojson-{region_id}-acheteurs",
+                options={"fillColor": color_acheteur},
+            ),
+        ],
+        center=center,
+        zoom=zoom,
+        style={
+            "width": "100%",
+            "height": "400px" if name == "Hexagone" else "300px",
+        },
+        id=f"map-{region_id}",
+    )
+    return leaflet_map
+
+
+def get_distance_histogram(lff: pl.LazyFrame) -> dcc.Graph:
+    if "titulaire_distance" not in lff.collect_schema().names():
+        dff = pl.DataFrame({"titulaire_distance": pl.Series([], dtype=pl.Float64)})
+    else:
+        dff = (
+            lff.select("titulaire_distance")
+            .drop_nulls()
+            .filter(pl.col("titulaire_distance") > 0)
+            .collect(engine="streaming")
+        )
+    log_distances = dff["titulaire_distance"].log(10).to_numpy()
+
+    fig = go.Figure()
+    if len(log_distances) > 0:
+        counts, bin_edges = np.histogram(log_distances, bins=25)
+        bin_centers = (bin_edges[:-1] + bin_edges[1:]) / 2
+        bin_widths = bin_edges[1:] - bin_edges[:-1]
+        bin_edges_km = 10.0**bin_edges
+
+        def fmt_km(km):
+            if km < 10:
+                return f"{km:.1f}"
+            elif km < 1000:
+                return f"{round(km)}"
+            else:
+                return f"{round(km):,}".replace(",", " ")
+
+        hover_texts = []
+        for i in range(len(counts)):
+            nb = f"{counts[i]:,}".replace(",", " ")
+            hover_texts.append(
+                f"Distance : {fmt_km(bin_edges_km[i])} – {fmt_km(bin_edges_km[i + 1])} km"
+                f"<br>Nombre de marchés : {nb}"
+            )
+
+        fig.add_trace(
+            go.Bar(
+                x=bin_centers,
+                y=counts,
+                width=bin_widths,
+                hovertext=hover_texts,
+                hoverinfo="text",
+            )
+        )
+        fig.update_layout(bargap=0)
+
+    fig.update_layout(margin=dict(r=10, t=10))
+    fig.update_xaxes(
+        tickvals=[0, 1, 2, 3, 4],
+        ticktext=["1", "10", "100", "1 000", "10 000"],
+        title_text="Distance (km)",
+    )
+    fig.update_yaxes(title_text="Nombre de marchés")
+    return dcc.Graph(figure=fig)
+
+
+def get_dashboard_summary_table(dff, dff_per_uid, nb_marches):
+    nb_acheteurs = dff.select("acheteur_id").n_unique()
+    nb_titulaires = dff.select("titulaire_id", "titulaire_typeIdentifiant").n_unique()
+    total_montant = int(dff_per_uid.select(pl.col("montant").sum()).item())
+    median_distance = dff.select(pl.median("titulaire_distance")).item()
+
+    summary_table = [
+        html.P(["Nombre de marchés : ", html.Strong(str(format_number(nb_marches)))]),
+        html.P(
+            [
+                "Nombre d'acheteurs uniques : ",
+                html.Strong(str(format_number(nb_acheteurs))),
+            ]
+        ),
+        html.P(
+            [
+                "Nombre de titulaires uniques : ",
+                html.Strong(str(format_number(nb_titulaires))),
+            ]
+        ),
+        html.P(
+            [
+                "Montant total (",
+                html.Span(
+                    "?",
+                    id={"type": "modal-trigger", "index": "montant"},
+                    style={"cursor": "pointer", "textDecoration": "underline dotted"},
+                ),
+                ") : ",
+                html.Strong(format_number(total_montant) + " €"),
+            ]
+        ),
+        html.P(
+            [
+                "Distance acheteur-titulaire médiane : ",
+                html.Strong(format_number(median_distance) + " km"),
+            ]
+        ),
+    ]
+
+    return summary_table
+
+
+def make_card(
+    title: str, subtitle=None, fig=None, paragraphs=None, lg=6, xl=4
+) -> dbc.Col:
+    children = []
+    if title:
+        children.append(html.H5(title, className="card-title"))
+    if subtitle:
+        children.append(html.H6(subtitle, className="card-subtitle mb-2 text-muted"))
+    if fig is not None:
+        children.append(fig)
+    if paragraphs:
+        for p in paragraphs:
+            p.className = "card-text"
+            children.append(p)
+
+    card = dbc.Col(
+        html.Div(html.Div(className="card-body", children=children), className="card"),
+        lg=lg,
+        xl=xl,
+        # width=width,
+        # className="mb-4",
+    )
+    return card
+
+
+def make_donut(
+    lff: pl.LazyFrame,
+    names_col,
+    per_uid: bool,
+    nulls="?",
+    potentially_many_names: bool = False,
+):
+    title = DATA_SCHEMA[names_col]["title"]
+    lff = lff.rename({names_col: title})
+    lff = lff.select("uid", title)
+
+    if per_uid:
+        lff = lff.group_by("uid").first()
+
+    lff = lff.group_by(title).len("Nombre")
+    lff = lff.with_columns(pl.col(title).replace(None, pl.lit(nulls)))
+    dff = lff.collect(engine="streaming")
+    nb_names = dff[title].n_unique()
+
+    sum_values = dff["Nombre"].sum()
+    dff = dff.with_columns(
+        pl.when((pl.col("Nombre") / sum_values) < 0.01)
+        .then(pl.lit("Autres"))
+        .otherwise(pl.col(title))
+        .alias(title)
+    )
+
+    dff = dff.with_columns(
+        pl.col("Nombre")
+        .map_elements(format_number, return_dtype=pl.String)
+        .alias("Nombre_fmt")
+    )
+    fig = px.pie(
+        dff,
+        values="Nombre",
+        names=title,
+        hole=0.4,
+        color_discrete_sequence=px.colors.qualitative.Safe,
+        custom_data=["Nombre_fmt"],
+    )
+    fig = fig.update_traces(
+        texttemplate="<b>%{label}</b><br><b>%{percent}</b>",
+        hovertemplate="<b>%{label}</b><br>%{customdata[0]}<extra></extra>",
+    )
+    fig = fig.update_layout(showlegend=False, font=dict(size=14))
+    graph = dcc.Graph(figure=fig)
+    if potentially_many_names:
+        return graph, nb_names
+    return graph
+
+
+def make_column_picker(page: str):
+    table_data = []
+    table_columns = [
+        {
+            "id": col,
+            "name": DATA_SCHEMA[col]["title"],
+            "description": DATA_SCHEMA[col]["description"],
+        }
+        for col in schema.names()
+    ]
+    for column in table_columns:
+        new_column = {
+            "id": column["id"],
+            "name": column["name"],
+            "description": DATA_SCHEMA[column["id"]]["description"],
+        }
+        table_data.append(new_column)
+
+    table = (
+        DataTable(
+            row_selectable="multi",
+            data=table_data,
+            filter_action="native",
+            sort_action="none",
+            style_cell={
+                "textAlign": "left",
+            },
+            columns=[
+                {
+                    "name": "Nom",
+                    "id": "name",
+                },
+                {
+                    "name": "Description",
+                    "id": "description",
+                },
+            ],
+            style_cell_conditional=[
+                {
+                    "if": {"column_id": "description"},
+                    "minWidth": "450px",
+                    "overflow": "hidden",
+                    "lineHeight": "18px",
+                    "whiteSpace": "normal",
+                }
+            ],
+            page_action="none",
+            dtid=f"{page}_column_list",
+        ),
+    )
+
+    return table
+
+
+def get_top_org_table(data, org_type: str, extra_columns: list, filters: bool = True):
+    if isinstance(data, pl.LazyFrame):
+        lff = data
+    else:
+        lff = pl.LazyFrame(data, strict=False, infer_schema_length=5000)
+
+    if org_type == "titulaire":
+        extra_columns.append("titulaire_typeIdentifiant")
+    columns = ["uid", f"{org_type}_id", f"{org_type}_nom"] + extra_columns
+
+    lff = lff.select(columns)
+    lff = lff.group_by([f"{org_type}_id", f"{org_type}_nom"] + extra_columns).agg(
+        pl.len().alias("Attributions")
+    )
+    lff = lff.sort(by="Attributions", descending=True, nulls_last=True)
+    lff = lff.cast(pl.String)
+    lff = lff.fill_null("")
+
+    dff: pl.DataFrame = lff.collect(engine="streaming")
+
+    if dff.height == 0:
+        return html.Div()
+
+    columns, tooltip = setup_table_columns(
+        dff, hideable=False, exclude=[f"{org_type}_id"], new_columns=["Attributions"]
+    )
+    dff = add_links(dff)
+    data = dff.to_dicts()
+    # data = add_links_in_dict(data, f"{org_type}")
+
+    return DataTable(
+        dtid=f"top10_{org_type}",
+        data=data,
+        page_action="native",
+        page_size=10,
+        columns=columns,
+        tooltip_header=tooltip,
+        filter_action="native" if filters else "none",
     )

src/pages/a-propos.py

@@ -3,9 +3,9 @@ import os
 from dash import dcc, html, register_page
 
 from src.figures import get_sources_tables
-from src.utils import meta_content
+from src.utils.seo import META_CONTENT
 
-name = "À propos"
+NAME = "À propos"
 
 register_page(
     __name__,
@@ -13,14 +13,14 @@ register_page(
     title="À propos | decp.info",
     name="À propos",
     description="En savoir plus sur decp.info, l'outil d'exploration des données essentielles de la commande publique.",
-    image_url=meta_content["image_url"],
+    image_url=META_CONTENT["image_url"],
     order=5,
 )
 
 layout = html.Div(
     className="container",
     children=[
-        html.H2(name),
+        html.H2(NAME),
         html.Div(
             className="a-propos-container",
             children=[
@@ -87,7 +87,7 @@ Vous pouvez consommer les données qui alimentent decp.info
                         dcc.Markdown(
                             """Les données visibles sur ce site proviennent exclusivement de la publication de données ouvertes par les acheteurs publics ou en leur nom, régie par [l'arrêté du 22 décembre 2022](https://www.legifrance.gouv.fr/jorf/id/JORFTEXT000046850496). Leur qualité est donc principalement liée à la qualité de leur saisie par les agents publics, parfois peu aidé·es par la qualité des outils à leur disposition. Je pense que l'analyse de marchés individuels et le comptage de marchés sur des critères autres que financiers sont plutôt fiables. En revanche, certains montants de marché estimés à des valeurs farfelues ([1 euro](https://decp.info/marches/432766947000192025S01301), [1 milliard](https://decp.info/marches/2459004280001320210000000271)) faussent les calculs par aggrégation (sommes, moyennes, médianes) et donc la production de statistiques financières fiables. Acheteurs, acheteuses : s'il vous plaît, essayez d'estimer les montants des marchés publics attribués de manière plus précise.
 
-Quant à l'exhaustivité, je consolide toutes les sources de données exploitables que j'ai pu identifier (voir [statistiques](/statistiques)). Certains profils d'acheteurs ne publient pas leurs données malgré l'obligation réglementaire :
+Quant à l'exhaustivité, je consolide toutes les sources de données exploitables que j'ai pu identifier (voir [ci-dessous](/a-propos#sources). Certains profils d'acheteurs ne publient pas leurs données malgré l'obligation réglementaire :
 
 - klekoon.fr (ils y travaillent)
 - safetender.com (Omnikles)

src/pages/acheteur.py

@@ -1,32 +1,49 @@
 import datetime
+from typing import Any
 
+import dash_bootstrap_components as dbc
 import polars as pl
-from dash import Input, Output, State, callback, dcc, html, register_page
+from dash import (
+    ClientsideFunction,
+    Input,
+    Output,
+    State,
+    callback,
+    clientside_callback,
+    dcc,
+    html,
+    register_page,
+)
 
-from src.callbacks import get_top_org_table
+from src.db import query_marches, schema
-from src.figures import DataTable, point_on_map
+from src.figures import (
-from src.utils import (
+    DataTable,
-    df,
+    get_distance_histogram,
-    df_acheteurs,
+    get_top_org_table,
+    make_card,
+    make_column_picker,
+    point_on_map,
+)
+from src.utils.data import DF_ACHETEURS, get_annuaire_data, get_departement_region
+from src.utils.frontend import get_button_properties
+from src.utils.seo import META_CONTENT
+from src.utils.table import (
+    COLUMNS,
     filter_table_data,
     format_number,
-    get_annuaire_data,
-    get_button_properties,
     get_default_hidden_columns,
-    get_departement_region,
-    meta_content,
     prepare_table_data,
     sort_table_data,
 )
 
 
-def get_title(acheteur_id: str = None) -> str:
+def get_title(acheteur_id: str | None = None) -> str:
-    acheteur_nom = (
+    acheteur_nom = DF_ACHETEURS.filter(pl.col("acheteur_id") == acheteur_id).select(
-        df_acheteurs.filter(pl.col("acheteur_id") == acheteur_id)
+        "acheteur_nom"
-        .select("acheteur_nom")
-        .item()
     )
-    return f"Marchés publics attribués par {acheteur_nom} | decp.info"
+    if acheteur_nom.height > 0:
+        return f"Marchés publics attribués par {acheteur_nom.item(0, 0)} | decp.info"
+    return "Marchés publics attribués | decp.info"
 
 
 register_page(
@@ -35,43 +52,53 @@ register_page(
     title=get_title,
     name="Acheteur",
     description="Consultez les marchés publics attribués par cet acheteur.",
-    image_url=meta_content["image_url"],
+    image_url=META_CONTENT["image_url"],
     order=5,
 )
 
-datatable = html.Div(
+DATATABLE = html.Div(
     className="marches_table",
     children=DataTable(
         dtid="acheteur_datatable",
+        persistence=True,
+        persistence_type="local",
+        persisted_props=["filter_query", "sort_by"],
         page_action="custom",
         filter_action="custom",
         sort_action="custom",
         page_size=10,
-        hidden_columns=get_default_hidden_columns(page="acheteur"),
+        hidden_columns=[],
+        columns=[{"id": col, "name": col} for col in schema.names()],
     ),
 )
 
 layout = [
     dcc.Store(id="acheteur_data", storage_type="memory"),
+    dcc.Store(id="acheteur-hidden-columns", storage_type="local"),
+    dcc.Store(id="filter-cleanup-trigger-acheteur"),
     dcc.Location(id="acheteur_url", refresh="callback-nav"),
     html.Div(
         children=[
             html.Div(
-                className="wrapper",
+                style={"marginBottom": "50px"},
                 children=[
+                    dbc.Row(
+                        className="mb-2",
+                        children=[
+                            dbc.Col(
                                 html.H2(
-                        className="org_title",
                                     children=[
                                         html.Span(id="acheteur_siret"),
                                         " - ",
                                         html.Span(id="acheteur_nom"),
                                     ],
                                 ),
-                    html.Div(
+                                width=8,
-                        className="org_year",
+                            ),
-                        children=dcc.Dropdown(
+                            dbc.Col(
+                                dcc.Dropdown(
                                     id="acheteur_year",
-                            options=["Toutes"]
+                                    options=["Toutes les années"]
                                     + [
                                         str(year)
                                         for year in range(
@@ -80,28 +107,40 @@ layout = [
                                     ],
                                     placeholder="Année",
                                 ),
+                                width=4,
                             ),
-                    html.Div(
+                        ],
+                    ),
+                    dbc.Row(
+                        className="mb-2",
+                        children=[
+                            dbc.Col(
                                 className="org_infos",
                                 children=[
                                     # TODO: ajouter le type d'acheteur : commune, CD, CR, etc.
-                            html.P(["Commune : ", html.Strong(id="acheteur_commune")]),
+                                    html.P(
+                                        [
+                                            "Commune : ",
+                                            html.Strong(id="acheteur_commune"),
+                                        ]
+                                    ),
                                     html.P(
                                         [
                                             "Département : ",
                                             html.Strong(id="acheteur_departement"),
                                         ]
                                     ),
-                            html.P(["Région : ", html.Strong(id="acheteur_region")]),
+                                    html.P(
+                                        ["Région : ", html.Strong(id="acheteur_region")]
+                                    ),
                                     html.A(
                                         id="acheteur_lien_annuaire",
                                         children="Plus de détails sur l'Annuaire des entreprises",
-                                target="_blank",
                                     ),
                                 ],
+                                width=4,
                             ),
-                    html.Div(
+                            dbc.Col(
-                        className="org_stats",
                                 children=[
                                     html.P(id="acheteur_titre_stats"),
                                     html.P(id="acheteur_marches_attribues"),
@@ -109,16 +148,26 @@ layout = [
                                     html.Button(
                                         "Téléchargement au format Excel",
                                         id="btn-download-data-acheteur",
+                                        className="btn btn-primary",
                                     ),
                                     dcc.Download(id="download-data-acheteur"),
                                 ],
+                                width=4,
                             ),
-                    html.Div(className="org_map", id="acheteur_map"),
+                            dbc.Col(
-                    html.Div(
+                                id="acheteur_map",
-                        className="org_top",
+                                width=4,
+                            ),
+                        ],
+                    ),
+                    dbc.Row(
                         children=[
-                            html.H3("Top titulaires"),
+                            dbc.Col(
-                            html.Div(className="marches_table", id="top10_titulaires"),
+                                className="marches_table",
+                                id="top10_titulaires",
+                                width=8,
+                            ),
+                            dbc.Col(id="acheteur-distance-histogram", width=4),
                         ],
                     ),
                 ],
@@ -132,17 +181,53 @@ layout = [
                 children=[
                     html.Div(
                         [
+                            # Bouton modal des colonnes affichées
+                            dbc.Button(
+                                "Colonnes affichées",
+                                id="acheteur_columns_open",
+                                className="column_list",
+                            ),
                             html.P("lignes", id="acheteur_nb_rows"),
                             html.Button(
                                 "Téléchargement désactivé au-delà de 65 000 lignes",
                                 id="btn-download-filtered-data-acheteur",
+                                className="btn btn-primary",
                                 disabled=True,
                             ),
                             dcc.Download(id="acheteur-download-filtered-data"),
+                            dbc.Button(
+                                "Remise à zéro",
+                                title="Supprime tous les filtres et les tris. Autrement ils sont conservés même si vous fermez la page.",
+                                id="btn-acheteur-reset",
+                            ),
                         ],
                         className="table-menu",
                     ),
-                    datatable,
+                    dbc.Modal(
+                        [
+                            dbc.ModalHeader(
+                                dbc.ModalTitle("Choix des colonnes à afficher")
+                            ),
+                            dbc.ModalBody(
+                                id="acheteur_columns_body",
+                                children=make_column_picker("acheteur"),
+                            ),
+                            dbc.ModalFooter(
+                                dbc.Button(
+                                    "Fermer",
+                                    id="acheteur_columns_close",
+                                    className="ms-auto",
+                                    n_clicks=0,
+                                )
+                            ),
+                        ],
+                        id="acheteur_columns",
+                        is_open=False,
+                        fullscreen="md-down",
+                        scrollable=True,
+                        size="xl",
+                    ),
+                    DATATABLE,
                 ],
             ),
         ],
@@ -162,12 +247,15 @@ layout = [
 )
 def update_acheteur_infos(url):
     acheteur_siret = url.split("/")[-1]
-    if len(acheteur_siret) != 14:
+    # if len(acheteur_siret) != 14:
-        acheteur_siret = (
+    #     acheteur_siret = (
-            f"Le SIRET renseigné doit faire 14 caractères ({acheteur_siret})"
+    #         f"Le SIRET renseigné doit faire 14 caractères ({acheteur_siret})"
-        )
+    #     )
     data = get_annuaire_data(acheteur_siret)
-    data_etablissement = data["matching_etablissements"][0]
+    data_etablissement = data.get("matching_etablissements") if data else None
+    if data_etablissement:
+        data_etablissement = data_etablissement[0]
+
         acheteur_map = point_on_map(
             data_etablissement["latitude"], data_etablissement["longitude"]
         )
@@ -178,10 +266,21 @@ def update_acheteur_infos(url):
         lien_annuaire = (
             f"https://annuaire-entreprises.data.gouv.fr/etablissement/{acheteur_siret}"
         )
+        raison_sociale = data["nom_raison_sociale"]
+        libelle_commune = data_etablissement["libelle_commune"]
+
+    else:
+        acheteur_map = html.Div()
+        code_departement, nom_departement, nom_region = "", "", ""
+        departement = ""
+        lien_annuaire = ""
+        raison_sociale = ""
+        libelle_commune = ""
+
     return (
         acheteur_siret,
-        data["nom_raison_sociale"],
+        raison_sociale,
-        data_etablissement["libelle_commune"],
+        libelle_commune,
         acheteur_map,
         departement,
         nom_region,
@@ -199,7 +298,7 @@ def update_acheteur_infos(url):
 def update_acheteur_stats(data):
     dff = pl.DataFrame(data, strict=False, infer_schema_length=5000)
     if dff.height == 0:
-        dff = pl.DataFrame(schema=df.collect_schema())
+        dff = pl.DataFrame(schema=schema)
     df_marches = dff.unique("id")
     nb_marches = format_number(df_marches.height)
     # somme_marches = format_number(int(df_marches.select(pl.sum("montant")).item()))
@@ -225,17 +324,15 @@ def update_acheteur_stats(data):
     Input(component_id="acheteur_url", component_property="pathname"),
     Input(component_id="acheteur_year", component_property="value"),
 )
-def get_acheteur_marches_data(url, acheteur_year: str) -> tuple:
+def get_acheteur_marches_data(url, ach_year: str) -> tuple:
     acheteur_siret = url.split("/")[-1]
-    lff = df.lazy()
+    lff = query_marches("acheteur_id = ?", (acheteur_siret,)).lazy()
-    lff = lff.filter(pl.col("acheteur_id") == acheteur_siret)
+    if ach_year and ach_year != "Toutes les années":
-    if acheteur_year and acheteur_year != "Toutes":
+        ach_year = int(ach_year)
-        acheteur_year = int(acheteur_year)
+        lff = lff.filter(pl.col("dateNotification").dt.year() == ach_year)
-        lff = lff.filter(pl.col("dateNotification").dt.year() == acheteur_year)
     lff = lff.sort(["dateNotification", "uid"], descending=True, nulls_last=True)
     dff: pl.DataFrame = lff.collect(engine="streaming")
     download_disabled, download_text, download_title = get_button_properties(dff.height)
-
     data = dff.to_dicts()
     return data, download_disabled, download_text, download_title
 
@@ -249,6 +346,8 @@ def get_acheteur_marches_data(url, acheteur_year: str) -> tuple:
     Output("btn-download-filtered-data-acheteur", "disabled"),
     Output("btn-download-filtered-data-acheteur", "children"),
     Output("btn-download-filtered-data-acheteur", "title"),
+    Output("filter-cleanup-trigger-acheteur", "data"),
+    Input("acheteur_url", "href"),
     Input("acheteur_data", "data"),
     Input("acheteur_datatable", "page_current"),
     Input("acheteur_datatable", "page_size"),
@@ -257,7 +356,7 @@ def get_acheteur_marches_data(url, acheteur_year: str) -> tuple:
     State("acheteur_datatable", "data_timestamp"),
 )
 def get_last_marches_data(
-    data, page_current, page_size, filter_query, sort_by, data_timestamp
+    href, data, page_current, page_size, filter_query, sort_by, data_timestamp
 ) -> tuple:
     return prepare_table_data(
         data, data_timestamp, filter_query, page_current, page_size, sort_by, "acheteur"
@@ -269,7 +368,8 @@ def get_last_marches_data(
     Input(component_id="acheteur_data", component_property="data"),
 )
 def get_top_titulaires(data):
-    return get_top_org_table(data, "titulaire")
+    table = get_top_org_table(data, "titulaire", ["titulaire_distance"])
+    return make_card(fig=table, title="Top titulaires", lg=12, xl=12)
 
 
 @callback(
@@ -282,7 +382,7 @@ def get_top_titulaires(data):
 )
 def download_acheteur_data(
     n_clicks,
-    data: [dict],
+    data: list[dict[str, Any]],
     acheteur_nom: str,
     annee: str,
 ):
@@ -290,7 +390,7 @@ def download_acheteur_data(
 
     def to_bytes(buffer):
         df_to_download.write_excel(
-            buffer, worksheet="DECP" if annee in ["Toutes", None] else annee
+            buffer, worksheet="DECP" if annee in ["Toutes les années", None] else annee
         )
 
     date = datetime.datetime.now().strftime("%Y-%m-%d_%H:%M:%S")
@@ -308,7 +408,12 @@ def download_acheteur_data(
     prevent_initial_call=True,
 )
 def download_filtered_acheteur_data(
-    data, n_clicks, acheteur_nom, filter_query, sort_by, hidden_columns: list = None
+    data,
+    n_clicks,
+    acheteur_nom,
+    filter_query,
+    sort_by,
+    hidden_columns: list | None = None,
 ):
     lff: pl.LazyFrame = pl.LazyFrame(
         data
@@ -331,3 +436,94 @@ def download_filtered_acheteur_data(
     return dcc.send_bytes(
         to_bytes, filename=f"decp_filtrées_{acheteur_nom}_{date}.xlsx"
     )
+
+
+# Pour nettoyer les icontains et i< des filtres
+# voir aussi src/assets/dash_clientside.js
+clientside_callback(
+    ClientsideFunction(
+        namespace="clientside",
+        function_name="clean_filters",
+    ),
+    Output("filter-cleanup-trigger-acheteur", "data", allow_duplicate=True),
+    Input("filter-cleanup-trigger-acheteur", "data"),
+    prevent_initial_call=True,
+)
+
+
+@callback(
+    Output("acheteur-hidden-columns", "data", allow_duplicate=True),
+    Input("acheteur_column_list", "selected_rows"),
+    prevent_initial_call=True,
+)
+def update_hidden_columns_from_checkboxes(selected_columns):
+    if selected_columns:
+        selected_columns = [COLUMNS[i] for i in selected_columns]
+        hidden_columns = [col for col in COLUMNS if col not in selected_columns]
+        return hidden_columns
+    else:
+        return []
+
+
+@callback(
+    Output("acheteur_datatable", "hidden_columns"),
+    Input(
+        "acheteur-hidden-columns",
+        "data",
+    ),
+)
+def store_hidden_columns(hidden_columns):
+    if hidden_columns is None:
+        hidden_columns = get_default_hidden_columns("acheteur")
+    return hidden_columns
+
+
+@callback(
+    Output("acheteur_column_list", "selected_rows"),
+    Input("acheteur_datatable", "hidden_columns"),
+    State("acheteur_column_list", "selected_rows"),  # pour éviter la boucle infinie
+)
+def update_checkboxes_from_hidden_columns(hidden_cols, current_checkboxes):
+    hidden_cols = hidden_cols or get_default_hidden_columns("acheteur")
+
+    # Show all columns that are NOT hidden
+    visible_cols = [COLUMNS.index(col) for col in COLUMNS if col not in hidden_cols]
+    return visible_cols
+
+
+@callback(
+    Output("acheteur_columns", "is_open"),
+    Input("acheteur_columns_open", "n_clicks"),
+    Input("acheteur_columns_close", "n_clicks"),
+    State("acheteur_columns", "is_open"),
+)
+def toggle_acheteur_columns(click_open, click_close, is_open):
+    if click_open or click_close:
+        return not is_open
+    return is_open
+
+
+@callback(
+    Output("acheteur_datatable", "filter_query", allow_duplicate=True),
+    Output("acheteur_datatable", "sort_by"),
+    Input("btn-acheteur-reset", "n_clicks"),
+    prevent_initial_call=True,
+)
+def reset_view(n_clicks):
+    return "", []
+
+
+@callback(
+    Output("acheteur-distance-histogram", "children"),
+    Input("acheteur_data", "data"),
+)
+def update_acheteur_distance_histogram(data):
+    lff = pl.LazyFrame(data, strict=False, infer_schema_length=5000)
+    fig = get_distance_histogram(lff)
+    return make_card(
+        title="Distance acheteur–titulaire",
+        subtitle="en nombre de marchés, échelle logarithmique",
+        fig=fig,
+        lg=12,
+        xl=12,
+    )

src/pages/arbre/departement.py

@@ -1,17 +1,17 @@
-import polars as pl
 from dash import Input, Output, callback, dcc, html, register_page
 
-from src.utils import departements, df_acheteurs_departement, df_titulaires_departement
+from src.db import get_cursor
+from src.utils.data import DEPARTEMENTS
 
-name = "Département"
+NAME = "Département"
 
 
 def get_title(code):
-    return f"Marchés publics de {departements[code]['departement']} | decp.info"
+    return f"Marchés publics de {DEPARTEMENTS[code]['departement']} | decp.info"
 
 
 def get_description(code):
-    return f"Marchés publics passés dans le département {departements[code]['departement']} | decp.info"
+    return f"Marchés publics passés dans le département {DEPARTEMENTS[code]['departement']} | decp.info"
 
 
 register_page(
@@ -20,7 +20,7 @@ register_page(
     title=get_title,
     description=get_description,
     order=50,
-    name=name,
+    name=NAME,
 )
 
 layout = html.Div(
@@ -39,29 +39,42 @@ def departement_marches(url):
     departement = url.split("/")[-1]
 
     def make_link_list(org_type) -> list:
-        link_list = []
+        table = (
-        if org_type == "acheteur":
+            "acheteurs_departement"
-            df = df_acheteurs_departement
+            if org_type == "acheteur"
-        elif org_type == "titulaire":
+            else "titulaires_departement"
-            df = df_titulaires_departement
+            if org_type == "titulaire"
-        else:
+            else None
+        )
+        if table is None:
             raise ValueError
+        col_prefix = org_type
+        rows = (
+            get_cursor()
+            .execute(
+                f"SELECT {col_prefix}_id, {col_prefix}_nom "
+                f"FROM {table} "
+                f"WHERE {col_prefix}_departement_code = ? "
+                f"ORDER BY {col_prefix}_nom",
+                [departement],
+            )
+            .fetchall()
+        )
 
-        df = df.filter(pl.col(f"{org_type}_departement_code") == departement)
+        link_list = []
-
+        for org_id, org_nom in rows:
-        for row in df.iter_rows(named=True):
             li = html.Li(
                 [
                     dcc.Link(
-                        row[f"{org_type}_nom"],
+                        org_nom,
-                        href=url + f"/{org_type}/{row[f'{org_type}_id']}",
+                        href=url + f"/{org_type}/{org_id}",
-                        title=f"Marchés publics de {row[f'{org_type}_nom']}",
+                        title=f"Marchés publics de {org_nom}",
                     ),
                     " ",
                     dcc.Link(
                         "(page dédiée)",
-                        href=f"/{org_type}s/{row[f'{org_type}_id']}",
+                        href=f"/{org_type}s/{org_id}",
-                        title=f"Page dédiée aux marchés publics de {row[f'{org_type}_nom']}",
+                        title=f"Page dédiée aux marchés publics de {org_nom}",
                     ),
                 ]
             )

src/pages/arbre/departements.py

@@ -1,8 +1,8 @@
 from dash import dcc, html, register_page
 
-from src.utils import departements
+from src.utils.data import DEPARTEMENTS
 
-name = "Départements"
+NAME = "Départements"
 
 register_page(
     __name__,
@@ -18,7 +18,7 @@ layout = html.Div(
         html.Ul(
             [
                 html.Li(dcc.Link(d["departement"], href=f"/departements/{k}"))
-                for k, d in departements.items()
+                for k, d in DEPARTEMENTS.items()
             ]
         ),
     ]

src/pages/arbre/liste_marches_org.py

@@ -1,28 +1,26 @@
 import polars as pl
 from dash import Input, Output, callback, dcc, html, register_page
 
-from src.utils import (
+from src.db import get_cursor
-    df_acheteurs,
+from src.utils.data import DF_ACHETEURS, DF_TITULAIRES
-    df_acheteurs_marches,
-    df_titulaires,
-    df_titulaires_marches,
-)
 
-name = "Liste des marchés publics"
+NAME = "Liste des marchés publics"
 
 
 def make_org_nom_verbe(org_type, org_id) -> tuple:
     if org_type == "titulaire":
-        df = df_titulaires
+        df = DF_TITULAIRES
         verbe = "remportés"
     elif org_type == "acheteur":
-        df = df_acheteurs
+        df = DF_ACHETEURS
         verbe = "attribués"
     else:
         raise ValueError
 
     org_nom = (
-        df.filter(pl.col(f"{org_type}_id") == org_id).select(f"{org_type}_nom").item()
+        df.filter(pl.col(f"{org_type}_id") == org_id)
+        .select(f"{org_type}_nom")
+        .item(0, 0)
     )
 
     return org_nom, verbe
@@ -46,7 +44,7 @@ register_page(
     title=get_title,
     description=get_description,
     order=40,
-    name=name,
+    name=NAME,
 )
 
 layout = html.Div(
@@ -66,28 +64,34 @@ def liste_marches(url):
     org_id = url.split("/")[-1]
 
     def make_link_list() -> list:
-        link_list = []
+        table = (
-        if org_type == "acheteur":
+            "acheteurs_marches"
-            df = df_acheteurs_marches
+            if org_type == "acheteur"
-        elif org_type == "titulaire":
+            else "titulaires_marches"
-            df = df_titulaires_marches
+            if org_type == "titulaire"
-        else:
+            else None
+        )
+        if table is None:
             raise ValueError
+        rows = (
+            get_cursor()
+            .execute(
+                f"SELECT uid, objet FROM {table} WHERE {org_type}_id = ?",
+                [org_id],
+            )
+            .fetchall()
+        )
 
-        df = df.filter(pl.col(f"{org_type}_id") == org_id)
+        return [
-
+            html.Li(
-        for row in df.iter_rows(named=True):
-            li = html.Li(
-                [
                 dcc.Link(
-                        row["objet"],
+                    objet,
-                        href=f"/marches/{row['uid']}",
+                    href=f"/marches/{uid}",
-                        title=f"Marchés public attribué : {row['objet']}",
+                    title=f"Marchés public attribué : {objet}",
                 )
+            )
+            for uid, objet in rows
         ]
-            )
-            link_list.append(li)
-        return link_list
 
     nom, verbe = make_org_nom_verbe(org_type, org_id)
 

src/pages/marche.py

@@ -2,18 +2,13 @@ import json
 from datetime import datetime
 
 import dash_bootstrap_components as dbc
-import polars as pl
 from dash import Input, Output, callback, dcc, html, register_page
 from polars import selectors as cs
 
-from src.utils import (
+from src.db import query_marches
-    data_schema,
+from src.utils.data import DATA_SCHEMA
-    df,
+from src.utils.seo import META_CONTENT, make_org_jsonld
-    format_values,
+from src.utils.table import format_values, unformat_montant
-    make_org_jsonld,
-    meta_content,
-    unformat_montant,
-)
 
 
 def get_title(uid: str = None) -> str:
@@ -26,7 +21,7 @@ register_page(
     title=get_title,
     name="Marché",
     description="Consultez les détails de ce marché public : montant, acheteur, titulaires, modifications, etc.",
-    image_url=meta_content["image_url"],
+    image_url=META_CONTENT["image_url"],
     order=7,
 )
 
@@ -88,19 +83,17 @@ layout = [
 def get_marche_data(url) -> tuple[dict, list]:
     marche_uid = url.split("/")[-1]
 
-    # Récupération des données du marché à partir du df global
+    # Filtre SQL côté DuckDB, puis Polars pour le post-traitement
+    dff_marche = query_marches("uid = ?", (marche_uid,))
+    if dff_marche.height == 0:
+        return {}, []
 
-    lff = df.lazy()
+    lff = dff_marche.lazy()
-    lff = lff.filter(pl.col("uid") == pl.lit(marche_uid))
-
-    # Données des titulaires du marché
     dff_titulaires = lff.select(cs.starts_with("titulaire")).collect(engine="streaming")
+    dff_marche_unique = lff.unique("uid").collect(engine="streaming")
+    dff_marche_unique = format_values(dff_marche_unique)
 
-    # Données du marché
+    return dff_marche_unique.to_dicts()[0], dff_titulaires.to_dicts()
-    dff_marche = lff.unique("uid").collect(engine="streaming")
-    dff_marche = format_values(dff_marche)
-
-    return dff_marche.to_dicts()[0], dff_titulaires.to_dicts()
 
 
 @callback(
@@ -113,7 +106,7 @@ def get_marche_data(url) -> tuple[dict, list]:
 )
 def update_marche_info(marche, titulaires):
     def make_parameter(col, bold=True):
-        column_object = data_schema.get(col)
+        column_object = DATA_SCHEMA.get(col)
         column_name = column_object.get("title") if column_object else col
 
         if marche[col]:
@@ -199,8 +192,8 @@ def update_marche_info(marche, titulaires):
     for titulaire in titulaires:
         if titulaire["titulaire_typeIdentifiant"] == "SIRET":
             categorie = titulaire.get("titulaire_categorie", "")
-            if titulaire.get("titulaire_categorie"):
+            if titulaire.get("titulaire_distance"):
-                distance = str(titulaire.get("titulaire_categorie")) + " km"
+                distance = str(titulaire.get("titulaire_distance")) + " km"
             else:
                 distance = ""
 

src/pages/observatoire.py

@@ -0,0 +1,942 @@
+import urllib.parse
+from datetime import datetime
+
+import dash_bootstrap_components as dbc
+import polars as pl
+from dash import (
+    ALL,
+    Input,
+    Output,
+    State,
+    callback,
+    ctx,
+    dcc,
+    html,
+    no_update,
+    register_page,
+)
+
+from src.cache import cache
+from src.db import query_marches, schema
+from src.figures import (
+    DataTable,
+    get_barchart_sources,
+    get_dashboard_summary_table,
+    get_distance_histogram,
+    get_duplicate_matrix,
+    get_geographic_maps,
+    get_top_org_table,
+    make_card,
+    make_column_picker,
+    make_donut,
+)
+from src.utils import logger
+from src.utils.data import (
+    DEPARTEMENTS,
+    DF_ACHETEURS,
+    DF_TITULAIRES,
+    prepare_dashboard_data,
+)
+from src.utils.frontend import get_enum_values_as_dict
+from src.utils.seo import META_CONTENT
+from src.utils.table import COLUMNS, get_default_hidden_columns, prepare_table_data
+
+NAME = "Observatoire"
+
+register_page(
+    __name__,
+    path="/observatoire",
+    title="Observatoire | decp.info",
+    name=NAME,
+    description="Visualisez l'état de la publication des données essentielles des marchés publics en France.",
+    image_url=META_CONTENT["image_url"],
+    order=3,
+)
+OPTIONS_YEARS = []
+for year in reversed(range(2017, datetime.now().year + 1)):
+    option_year = {
+        "label": str(year),
+        "value": year,
+    }
+    OPTIONS_YEARS.append(option_year)
+
+OPTIONS_DEPARTEMENTS = []
+for code in DEPARTEMENTS.keys():
+    departement = {
+        "label": f"{DEPARTEMENTS[code]['departement']} ({code})",
+        "value": code,
+    }
+    OPTIONS_DEPARTEMENTS.append(departement)
+
+OBSERVATOIRE_COLUMNS = [
+    col
+    for col in schema.names()
+    if col.startswith("acheteur")
+    or col.startswith("titulaire")
+    or col
+    in [
+        "uid",
+        "dateNotification",
+        "montant",
+        "considerationsSociales",
+        "considerationsEnvironnementales",
+        "marcheInnovant",
+        "sousTraitanceDeclaree",
+        "techniques",
+        "sourceDataset",
+        "type",
+        "codeCPV",
+    ]
+]
+
+layout = [
+    dcc.Location(id="dashboard_url", refresh="callback-nav"),
+    dcc.Store(id="observatoire-filters", storage_type="local"),
+    dcc.Store(id="observatoire-hidden-columns", storage_type="local"),
+    dcc.Store(
+        id="filter-cleanup-trigger-observatoire-preview"
+    ),  # utilisé juste pour ne pas avoir à adapter les données retournées de prepare_table data
+    dbc.Modal(
+        [
+            dbc.ModalHeader(dbc.ModalTitle("Montants")),
+            dbc.ModalBody(
+                [
+                    dcc.Markdown(
+                        """
+Les données saisies et publiées par les acheteurs comportent de nombreux montants farfelus qui sabotent les statistiques, au lieu de montants estimés avec rigueur. On parle de montants atteignant parfois les millions de milliards. Certains réutilisateurs des données mettent de côté ces marchés ou bien modifient les montants selon des règles fatalement arbitraires. J'ai fait le choix de ne quasiment pas modifier les données* afin de visibiliser le problème.
+
+Alors, on fait comment ?
+
+\\* Les montants composés de plus de 11 chiffres, sans les décimales, [sont ramenés](https://github.com/ColinMaudry/decp-processing/blob/main/src/tasks/clean.py#L63-L71) à 12 311 111 111, un nombre qui reste très élevé et qui est facilement reconnaissable.
+"""
+                    ),
+                ]
+            ),
+            dbc.ModalFooter(
+                dbc.Button("Fermer", id="montant-modal-close", className="ms-auto")
+            ),
+        ],
+        id="montant-modal",
+        is_open=False,
+    ),
+    html.Div(
+        className="container-fluid",
+        children=[
+            html.H2(children=[NAME], id="page_title"),
+            dcc.Loading(
+                overlay_style={"visibility": "visible", "filter": "blur(2px)"},
+                id="loading-statistques",
+                type="default",
+                children=[
+                    dbc.Row(
+                        [
+                            dbc.Col(
+                                xl=3,
+                                lg=4,
+                                id="filters",
+                                children=[
+                                    html.H5("Période d'attribution"),
+                                    dbc.Row(
+                                        dbc.Col(
+                                            dcc.Dropdown(
+                                                id="dashboard_year",
+                                                options=OPTIONS_YEARS,
+                                                placeholder="12 derniers mois",
+                                                persistence=True,
+                                                persistence_type="local",
+                                            ),
+                                        ),
+                                    ),
+                                    html.H5("Acheteur"),
+                                    dbc.Row(
+                                        dbc.Col(
+                                            dcc.Input(
+                                                id="dashboard_acheteur_id",
+                                                placeholder="SIRET",
+                                                debounce=True,
+                                                style={"width": "100%"},
+                                                persistence=True,
+                                                persistence_type="local",
+                                            ),
+                                        ),
+                                    ),
+                                    dbc.Row(
+                                        dbc.Col(
+                                            dcc.Dropdown(
+                                                id="dashboard_acheteur_categorie",
+                                                options=get_enum_values_as_dict(
+                                                    "acheteur_categorie"
+                                                ),
+                                                placeholder="Catégorie",
+                                                persistence=True,
+                                                persistence_type="local",
+                                            )
+                                        ),
+                                    ),
+                                    dbc.Row(
+                                        dbc.Col(
+                                            dcc.Dropdown(
+                                                id="dashboard_acheteur_departement_code",
+                                                searchable=True,
+                                                multi=True,
+                                                placeholder="Département",
+                                                options=OPTIONS_DEPARTEMENTS,
+                                                persistence=True,
+                                                persistence_type="local",
+                                            ),
+                                        ),
+                                    ),
+                                    html.H5("Titulaire"),
+                                    dbc.Row(
+                                        dbc.Col(
+                                            dcc.Input(
+                                                id="dashboard_titulaire_id",
+                                                placeholder="SIRET",
+                                                debounce=True,
+                                                style={"width": "100%"},
+                                                persistence=True,
+                                                persistence_type="local",
+                                            ),
+                                        ),
+                                    ),
+                                    dbc.Row(
+                                        dbc.Col(
+                                            dcc.Dropdown(
+                                                id="dashboard_titulaire_categorie",
+                                                placeholder="Catégorie",
+                                                options=get_enum_values_as_dict(
+                                                    "titulaire_categorie"
+                                                ),
+                                                persistence=True,
+                                                persistence_type="local",
+                                            ),
+                                        ),
+                                    ),
+                                    dbc.Row(
+                                        dbc.Col(
+                                            dcc.Dropdown(
+                                                id="dashboard_titulaire_departement_code",
+                                                searchable=True,
+                                                multi=True,
+                                                placeholder="Département",
+                                                options=OPTIONS_DEPARTEMENTS,
+                                                persistence=True,
+                                                persistence_type="local",
+                                            ),
+                                        ),
+                                    ),
+                                    html.H5("Marché"),
+                                    dbc.Row(
+                                        dbc.Col(
+                                            dcc.Dropdown(
+                                                id="dashboard_marche_type",
+                                                placeholder="Type",
+                                                options=get_enum_values_as_dict("type"),
+                                                persistence=True,
+                                                persistence_type="local",
+                                            ),
+                                        ),
+                                    ),
+                                    dbc.Row(
+                                        dbc.Col(
+                                            dcc.Input(
+                                                id="dashboard_marche_objet",
+                                                placeholder="Objet",
+                                                debounce=True,
+                                                style={"width": "100%"},
+                                                persistence=True,
+                                                persistence_type="local",
+                                            ),
+                                        ),
+                                    ),
+                                    dbc.Row(
+                                        [
+                                            dbc.Col(
+                                                dcc.Input(
+                                                    id="dashboard_marche_code_cpv",
+                                                    placeholder="Code CPV (début)",
+                                                    debounce=True,
+                                                    style={"width": "100%"},
+                                                    persistence=True,
+                                                    persistence_type="local",
+                                                ),
+                                                lg=8,
+                                            ),
+                                            dbc.Col(
+                                                html.A(
+                                                    "liste des codes",
+                                                    href="https://cpvcodes.eu/fr",
+                                                    target="_blank",
+                                                ),
+                                                lg=4,
+                                            ),
+                                        ]
+                                    ),
+                                    dbc.Row(
+                                        [
+                                            dbc.Col(
+                                                dcc.Input(
+                                                    id="dashboard_montant_min",
+                                                    placeholder="Montant min.",
+                                                    type="number",
+                                                    min=0,
+                                                    debounce=True,
+                                                    style={"width": "100%"},
+                                                    persistence=True,
+                                                    persistence_type="local",
+                                                ),
+                                                width=6,
+                                            ),
+                                            dbc.Col(
+                                                dcc.Input(
+                                                    id="dashboard_montant_max",
+                                                    placeholder="Montant max.",
+                                                    type="number",
+                                                    min=0,
+                                                    debounce=True,
+                                                    style={"width": "100%"},
+                                                    persistence=True,
+                                                    persistence_type="local",
+                                                ),
+                                                width=6,
+                                            ),
+                                        ]
+                                    ),
+                                    dbc.Row(
+                                        dbc.Col(
+                                            dcc.Dropdown(
+                                                id="dashboard_marche_techniques",
+                                                placeholder="Techniques d'achat",
+                                                options=get_enum_values_as_dict(
+                                                    "techniques"
+                                                ),
+                                                multi=True,
+                                                persistence=True,
+                                                persistence_type="local",
+                                            ),
+                                        ),
+                                    ),
+                                    dbc.Row(
+                                        [
+                                            dbc.Col("Sous-traitance :", lg=5),
+                                            dbc.Col(
+                                                dbc.RadioItems(
+                                                    id="dashboard_marche_sous_traitance_declaree",
+                                                    options=[
+                                                        {
+                                                            "label": "Tous",
+                                                            "value": "all",
+                                                        },
+                                                        {
+                                                            "label": "Oui",
+                                                            "value": "oui",
+                                                        },
+                                                        {
+                                                            "label": "Non",
+                                                            "value": "non",
+                                                        },
+                                                    ],
+                                                    value="all",
+                                                    inline=True,
+                                                    persistence=True,
+                                                    persistence_type="local",
+                                                ),
+                                                lg=7,
+                                            ),
+                                        ]
+                                    ),
+                                    dbc.Row(
+                                        [
+                                            dbc.Col("Marché innovant :", lg=5),
+                                            dbc.Col(
+                                                dbc.RadioItems(
+                                                    id="dashboard_marche_innovant",
+                                                    options=[
+                                                        {
+                                                            "label": "Tous",
+                                                            "value": "all",
+                                                        },
+                                                        {
+                                                            "label": "Oui",
+                                                            "value": "oui",
+                                                        },
+                                                        {
+                                                            "label": "Non",
+                                                            "value": "non",
+                                                        },
+                                                    ],
+                                                    value="all",
+                                                    inline=True,
+                                                    persistence=True,
+                                                    persistence_type="local",
+                                                ),
+                                                lg=7,
+                                            ),
+                                        ]
+                                    ),
+                                    dbc.Row(
+                                        dbc.Col(
+                                            dcc.Dropdown(
+                                                id="dashboard_marche_considerations_sociales",
+                                                placeholder="Considérations sociales",
+                                                options=get_enum_values_as_dict(
+                                                    "considerationsSociales"
+                                                ),
+                                                multi=True,
+                                                persistence=True,
+                                                persistence_type="local",
+                                            ),
+                                        ),
+                                    ),
+                                    dbc.Row(
+                                        dbc.Col(
+                                            dcc.Dropdown(
+                                                id="dashboard_marche_considerations_environnementales",
+                                                placeholder="Considérations environnementales",
+                                                multi=True,
+                                                options=get_enum_values_as_dict(
+                                                    "considerationsEnvironnementales"
+                                                ),
+                                                persistence=True,
+                                                persistence_type="local",
+                                            ),
+                                        ),
+                                    ),
+                                    dbc.Row(
+                                        [
+                                            dbc.Col(
+                                                [
+                                                    dcc.Download(
+                                                        id="download-observatoire"
+                                                    ),
+                                                    dbc.Button(
+                                                        "Voir les données",
+                                                        id="btn-observatoire-preview",
+                                                        className="btn btn-primary mt-2",
+                                                        color="primary",
+                                                        outline=True,
+                                                    ),
+                                                    dcc.Input(
+                                                        id="observatoire-share-url",
+                                                        readOnly=True,
+                                                        style={"display": "none"},
+                                                    ),
+                                                ],
+                                                lg=12,
+                                                xl=6,
+                                            ),
+                                            dbc.Col(
+                                                id="observatoire-copy-container",
+                                                lg=12,
+                                                xl=6,
+                                            ),
+                                        ]
+                                    ),
+                                ],
+                            ),
+                            dbc.Col(
+                                width=12,
+                                lg=8,
+                                xl=9,
+                                id="cards",
+                                children=[],
+                            ),
+                        ]
+                    )
+                ],
+            ),
+        ],
+    ),
+    dbc.Offcanvas(
+        id="observatoire-preview",
+        title="Prévisualisation des données",
+        placement="bottom",
+        is_open=False,
+        scrollable=True,
+        style={"height": "75vh"},
+        children=[
+            # Header row: title + "Colonnes affichées" button
+            dbc.Row(
+                [
+                    dbc.Col(
+                        html.Div(
+                            className="table-menu",
+                            children=[
+                                dbc.Button(
+                                    "Choisir les colonnes",
+                                    id="observatoire-preview-columns-open",
+                                    className="btn btn-primary",
+                                ),
+                                html.P(id="nb_rows_observatoire"),
+                                dbc.Button(
+                                    "Télécharger au format Excel",
+                                    id="btn-download-observatoire",
+                                    disabled=True,
+                                    className="btn btn-primary",
+                                    outline=True,
+                                ),
+                            ],
+                        ),
+                        width="auto",
+                    ),
+                ],
+                className="mb-2 align-items-center",
+            ),
+            # Column picker modal
+            dbc.Modal(
+                [
+                    dbc.ModalHeader(
+                        dbc.ModalTitle("Colonnes affichées dans la prévisualisation")
+                    ),
+                    dbc.ModalBody(
+                        id="observatoire-preview-columns-body",
+                        children=make_column_picker("observatoire_preview"),
+                    ),
+                    dbc.ModalFooter(
+                        dbc.Button(
+                            "Fermer",
+                            id="observatoire-preview-columns-close",
+                            className="ms-auto",
+                            n_clicks=0,
+                        )
+                    ),
+                ],
+                id="observatoire-preview-columns-modal",
+                is_open=False,
+                fullscreen="md-down",
+                scrollable=True,
+                size="xl",
+            ),
+            # DataTable
+            html.Div(
+                className="marches_table",
+                children=DataTable(
+                    dtid="observatoire-preview-table",
+                    page_size=5,
+                    page_action="custom",
+                    sort_action="custom",
+                    filter_action="custom",
+                    hidden_columns=[],
+                    columns=[{"id": col, "name": col} for col in OBSERVATOIRE_COLUMNS],
+                ),
+            ),
+        ],
+    ),
+]
+
+
+FILTER_PARAMS = [
+    # (component_id, url_key, is_multi, default_value)
+    ("dashboard_year", "annee", False, None),
+    ("dashboard_acheteur_id", "acheteur_id", False, None),
+    ("dashboard_acheteur_categorie", "acheteur_cat", False, None),
+    ("dashboard_acheteur_departement_code", "acheteur_dept", True, None),
+    ("dashboard_titulaire_id", "titulaire_id", False, None),
+    ("dashboard_titulaire_categorie", "titulaire_cat", False, None),
+    ("dashboard_titulaire_departement_code", "titulaire_dept", True, None),
+    ("dashboard_marche_type", "type", False, None),
+    ("dashboard_marche_objet", "objet", False, None),
+    ("dashboard_marche_code_cpv", "cpv", False, None),
+    ("dashboard_montant_min", "montant_min", False, None),
+    ("dashboard_montant_max", "montant_max", False, None),
+    ("dashboard_marche_techniques", "techniques", True, None),
+    ("dashboard_marche_innovant", "innovant", False, "all"),
+    ("dashboard_marche_sous_traitance_declaree", "sous_traitance", False, "all"),
+    ("dashboard_marche_considerations_sociales", "social", True, None),
+    ("dashboard_marche_considerations_environnementales", "env", True, None),
+]
+
+
+@callback(
+    *[Output(fp[0], "value") for fp in FILTER_PARAMS],
+    Input("dashboard_url", "search"),
+    Input("dashboard_url", "pathname"),
+    State("observatoire-filters", "data"),
+)
+def restore_filters(search, _pathname, stored_filters):
+    if search:
+        params = urllib.parse.parse_qs(search.lstrip("?"))
+        known_keys = {fp[1] for fp in FILTER_PARAMS}
+        if any(k in params for k in known_keys):
+            values = []
+            for _comp_id, url_key, is_multi, default in FILTER_PARAMS:
+                if url_key in params:
+                    if is_multi:
+                        values.append(params[url_key])
+                    else:
+                        raw = params[url_key][0]
+                        if url_key in ("montant_min", "montant_max"):
+                            try:
+                                raw = float(raw)
+                            except (ValueError, TypeError):
+                                raw = None
+                        values.append(raw)
+                else:
+                    values.append(default)
+            return tuple(values)
+    return (no_update,) * 17
+
+
+@callback(
+    Output("observatoire-share-url", "value"),
+    Output("observatoire-copy-container", "children"),
+    *[Input(fp[0], "value") for fp in FILTER_PARAMS],
+    Input("dashboard_url", "href"),
+)
+def sync_observatoire_share_url(*args):
+    # Last arg is href (State), rest are filter values
+    filter_values = args[:-1]
+    href = args[-1]
+
+    if not href:
+        return no_update, no_update
+
+    base_url = href.split("?")[0]
+
+    params = []
+    for (_, url_key, is_multi, default), value in zip(FILTER_PARAMS, filter_values):
+        if value is None or value == default or value == [] or value == "":
+            continue
+        if is_multi and isinstance(value, list):
+            for v in value:
+                params.append((url_key, v))
+        else:
+            params.append((url_key, value))
+
+    query_string = urllib.parse.urlencode(params)
+    full_url = f"{base_url}?{query_string}" if query_string else base_url
+
+    if params:
+        copy_button = dcc.Clipboard(
+            id="btn-copy-observatoire-url",
+            target_id="observatoire-share-url",
+            title="Copier l'URL de cette vue",
+            style={
+                "display": "inline-block",
+                "fontSize": 20,
+                "verticalAlign": "top",
+                "cursor": "pointer",
+            },
+            className="fa fa-link",
+            children=[
+                dbc.Button(
+                    "Partager cette vue",
+                    id="btn-copy-observatoire",
+                    className="btn btn-primary mt-2",
+                    title="Copier l'adresse de cette vue filtrée pour la partager.",
+                )
+            ],
+        )
+    else:
+        copy_button = html.Div()
+
+    return full_url, copy_button
+
+
+@callback(
+    Output("observatoire-copy-container", "children", allow_duplicate=True),
+    Input("btn-copy-observatoire", "n_clicks", allow_optional=True),
+    prevent_initial_call=True,
+)
+def show_confirmation(n_clicks):
+    if n_clicks:
+        return html.Span(
+            "Adresse de la vue copiée",
+            style={"color": "green", "fontWeight": "bold", "marginLeft": "10px"},
+        )
+    return no_update
+
+
+def _normalize_filter_params(filter_params: dict) -> tuple:
+    """Produce a deterministic, hashable key for caching."""
+    return tuple(
+        sorted(
+            (k, tuple(v) if isinstance(v, list) else v)
+            for k, v in filter_params.items()
+        )
+    )
+
+
+@cache.memoize()
+def _compute_dashboard_children(cache_key: tuple):
+    logger.debug("Cache miss — computing dashboard")
+    filter_params = {k: (list(v) if isinstance(v, tuple) else v) for k, v in cache_key}
+
+    lff: pl.LazyFrame = query_marches().lazy()
+    lff = prepare_dashboard_data(lff=lff, **filter_params)
+
+    dff = lff.collect(engine="streaming")
+
+    df_per_uid = (
+        dff.select("uid", "montant").group_by("uid").agg(pl.col("montant").first())
+    )
+    nb_marches = df_per_uid.height
+
+    cards = []
+    card_summary_table = get_dashboard_summary_table(dff, df_per_uid, nb_marches)
+    cards.append(make_card(title="Résumé", paragraphs=card_summary_table))
+
+    donut_acheteur_categorie, nb_acheteur_categories = make_donut(
+        lff,
+        "acheteur_categorie",
+        nulls="Autres",
+        per_uid=True,
+        potentially_many_names=True,
+    )
+    cards.append(
+        make_card(
+            title="Catégorie d'acheteur",
+            subtitle="en nombre de marchés attribués",
+            fig=donut_acheteur_categorie,
+            lg=12 if nb_acheteur_categories > 4 else 6,
+            xl=8 if nb_acheteur_categories > 4 else 4,
+        )
+    )
+
+    donut_titulaire_categorie = make_donut(
+        lff, "titulaire_categorie", per_uid=False, nulls="?"
+    )
+    cards.append(
+        make_card(
+            title="Catégorie d'entreprise",
+            subtitle="en nombre de titulaires",
+            fig=donut_titulaire_categorie,
+        )
+    )
+
+    donut_marche_type = make_donut(lff, "type", per_uid=True, nulls="?")
+    cards.append(
+        make_card(
+            title="Type d'achat",
+            subtitle="en nombre de marchés attribués",
+            fig=donut_marche_type,
+        )
+    )
+
+    distance_histogram = get_distance_histogram(lff)
+    cards.append(
+        make_card(
+            title="Distance acheteur–titulaire",
+            subtitle="en nombre de marchés, échelle logarithmique",
+            fig=distance_histogram,
+        )
+    )
+
+    top_acheteurs = get_top_org_table(
+        lff, org_type="acheteur", filters=False, extra_columns=[]
+    )
+    cards.append(make_card(title="Top acheteurs", fig=top_acheteurs, lg=12, xl=8))
+
+    top_titulaires = get_top_org_table(
+        lff, org_type="titulaire", filters=False, extra_columns=[]
+    )
+    cards.append(make_card(title="Top titulaires", fig=top_titulaires, lg=12, xl=8))
+
+    geographic_maps: list[dbc.Col] | None = get_geographic_maps(dff)
+
+    other_cards = []
+    sources_barchart = get_barchart_sources(lff, type_date="dateNotification")
+    other_cards.append(
+        make_card(
+            title="Sources de données",
+            subtitle="Nombre de marchés attribués par mois de notification et source de données",
+            fig=sources_barchart,
+            lg=12,
+            xl=8,
+        )
+    )
+
+    duplicate_matrix = get_duplicate_matrix()
+    other_cards.append(
+        make_card(
+            title="Matrice de doublons entre sources de données",
+            subtitle="Ce graphique illustre les doublons de marchés publics entre sources, c'est-à-dire la proportion de marchés publiés par plus d'une source.",
+            fig=duplicate_matrix,
+            lg=12,
+            xl=8,
+        )
+    )
+
+    return cards + geographic_maps + other_cards
+
+
+@callback(
+    Output("cards", "children"),
+    Output("observatoire-filters", "data"),
+    *[Input(fp[0], "value") for fp in FILTER_PARAMS],
+)
+def update_dashboard_cards(*filter_values):
+    filter_params = {}
+    for (input_id, _url_key, _is_multi, _default), value in zip(
+        FILTER_PARAMS, filter_values
+    ):
+        filter_params[input_id] = value
+
+    cache_key = _normalize_filter_params(filter_params)
+    children = _compute_dashboard_children(cache_key)
+
+    return dbc.Row(children=children), filter_params
+
+
+@callback(
+    Output("download-observatoire", "data"),
+    Input("btn-download-observatoire", "n_clicks"),
+    State("observatoire-filters", "data"),
+    State("observatoire-hidden-columns", "data"),
+    prevent_initial_call=True,
+)
+def download_observatoire(_n_clicks, filter_params, hidden_columns):
+    lff = prepare_dashboard_data(lff=query_marches().lazy(), **(filter_params or {}))
+
+    if hidden_columns:
+        lff = lff.drop(hidden_columns)
+
+    def to_bytes(buffer):
+        lff.collect(engine="streaming").write_excel(buffer, worksheet="DECP")
+
+    date = datetime.now().strftime("%Y-%m-%d_%H:%M:%S")
+    return dcc.send_bytes(to_bytes, filename=f"decp_observatoire_{date}.xlsx")
+
+
+@callback(
+    Output("montant-modal", "is_open"),
+    Input({"type": "modal-trigger", "index": ALL}, "n_clicks"),
+    Input("montant-modal-close", "n_clicks"),
+    prevent_initial_call=True,
+)
+def toggle_montant_modal(n_triggers, _close):
+    return isinstance(ctx.triggered_id, dict) and any(n_triggers)
+
+
+@callback(
+    Output("page_title", "children"),
+    Input("dashboard_acheteur_id", "value"),
+    Input("dashboard_titulaire_id", "value"),
+    prevent_initial_call=False,
+)
+def add_organization_name_in_title(acheteur_id, titulaire_id):
+    def lookup_nom(df_org, id_col, nom_col, org_id):
+        match = df_org.filter(pl.col(id_col) == org_id)
+        return match[nom_col].item(0) if match.height >= 1 else None
+
+    if acheteur_id and len(acheteur_id) == 14:
+        if nom := lookup_nom(DF_ACHETEURS, "acheteur_id", "acheteur_nom", acheteur_id):
+            return [
+                NAME,
+                html.Small(nom, className="text-muted d-block fw-normal fs-5"),
+            ]
+    elif titulaire_id and len(titulaire_id) == 14:
+        if nom := lookup_nom(
+            DF_TITULAIRES, "titulaire_id", "titulaire_nom", titulaire_id
+        ):
+            return [
+                NAME,
+                html.Small(nom, className="text-muted d-block fw-normal fs-5"),
+            ]
+    return NAME
+
+
+@callback(
+    Output("observatoire-preview", "is_open"),
+    Input("btn-observatoire-preview", "n_clicks"),
+    State("observatoire-preview", "is_open"),
+    prevent_initial_call=True,
+)
+def toggle_observatoire_preview(n_clicks, is_open):
+    return not is_open
+
+
+@callback(
+    Output("observatoire-preview-table", "data"),
+    Output("observatoire-preview-table", "columns"),
+    Output("observatoire-preview-table", "tooltip_header"),
+    Output("observatoire-preview-table", "data_timestamp"),
+    Output("nb_rows_observatoire", "children"),
+    Output("btn-download-observatoire", "disabled"),
+    Output("btn-download-observatoire", "children"),
+    Output("btn-download-observatoire", "title"),
+    Output("filter-cleanup-trigger-observatoire-preview", "data", allow_duplicate=True),
+    Input("observatoire-preview", "is_open"),
+    Input("observatoire-preview-table", "filter_query"),
+    Input("observatoire-preview-table", "page_current"),
+    Input("observatoire-preview-table", "page_size"),
+    Input("observatoire-preview-table", "sort_by"),
+    State("observatoire-preview-table", "data_timestamp"),
+    State("observatoire-filters", "data"),
+    prevent_initial_call=True,
+)
+def populate_preview_table(
+    is_open,
+    filter_query,
+    page_current,
+    page_size,
+    sort_by,
+    data_timestamp,
+    filter_params,
+):
+    if not is_open:
+        return (no_update,) * 9
+
+    lff = prepare_dashboard_data(lff=query_marches().lazy(), **(filter_params or {}))
+
+    return prepare_table_data(
+        lff,
+        data_timestamp,
+        filter_query,
+        page_current,
+        page_size,
+        sort_by,
+        "observatoire-preview",
+    )
+
+
+@callback(
+    Output("observatoire-hidden-columns", "data", allow_duplicate=True),
+    Input("observatoire_preview_column_list", "selected_rows"),
+    prevent_initial_call=True,
+)
+def update_hidden_columns_from_checkboxes(selected_columns):
+    if selected_columns:
+        selected_columns = [COLUMNS[i] for i in selected_columns]
+        hidden_columns = [col for col in COLUMNS if col not in selected_columns]
+        return hidden_columns
+    else:
+        return []
+
+
+@callback(
+    Output("observatoire-preview-table", "hidden_columns"),
+    Input(
+        "observatoire-hidden-columns",
+        "data",
+    ),
+)
+def store_hidden_columns(hidden_columns):
+    return hidden_columns
+
+
+@callback(
+    Output("observatoire_preview_column_list", "selected_rows"),
+    Input("observatoire-preview-table", "hidden_columns"),
+    State(
+        "observatoire_preview_column_list", "selected_rows"
+    ),  # pour éviter la boucle infinie
+)
+def update_checkboxes_from_hidden_columns(hidden_cols, current_checkboxes):
+    hidden_cols = hidden_cols or get_default_hidden_columns("tableau")
+
+    # Show all columns that are NOT hidden
+    visible_cols = [COLUMNS.index(col) for col in COLUMNS if col not in hidden_cols]
+    return visible_cols
+
+
+@callback(
+    Output("observatoire-preview-columns-modal", "is_open"),
+    Input("observatoire-preview-columns-open", "n_clicks"),
+    Input("observatoire-preview-columns-close", "n_clicks"),
+    State("observatoire-preview-columns-modal", "is_open"),
+)
+def toggle_tableau_columns(click_open, click_close, is_open):
+    if click_open or click_close:
+        return not is_open
+    return is_open

src/pages/recherche.py

@@ -1,23 +1,21 @@
+import dash_bootstrap_components as dbc
 from dash import Input, Output, State, callback, dcc, html, register_page
 
 from src.figures import DataTable
-from src.utils import (
+from src.utils.data import DF_ACHETEURS, DF_TITULAIRES
-    df_acheteurs,
+from src.utils.search import search_org
-    df_titulaires,
+from src.utils.seo import META_CONTENT
-    meta_content,
+from src.utils.table import setup_table_columns
-    search_org,
-    setup_table_columns,
-)
 
-name = "Recherche"
+NAME = "Recherche"
 
 register_page(
     __name__,
     path="/",
     title="Recherche de marchés publics | decp.info",
-    name=name,
+    name=NAME,
     description="Explorez et analysez les données des marchés publics français avec cet outil libre et gratuit. Pour une commande publique accessible à toutes et tous.",
-    image_url=meta_content["image_url"],
+    image_url=META_CONTENT["image_url"],
     order=0,
 )
 
@@ -46,20 +44,20 @@ layout = html.Div(
                         "width": "500px",
                         "border": "1px solid #ccc",
                         "borderRight": "none",
-                        "borderRadius": "4px 0 0 4px",
+                        "borderRadius": "3px 0 0 3px",
                         "padding": "5px 10px",
                         "outline": "none",
+                        "height": "34px",
                     },
                 ),
                 html.Button(
-                    "🔍",
+                    "=>",
                     id="search-button",
+                    className="btn btn-primary",
                     style={
                         "border": "1px solid #ccc",
-                        "borderRadius": "0 4px 4px 0",
+                        "borderRadius": "0 3px 3px 0",
                         "marginLeft": "0",
-                        "backgroundColor": "#f0f0f0",
-                        "cursor": "pointer",
                         "height": "auto",  # Ensure it matches input height if necessary, often relying on padding/line-height
                     },
                 ),
@@ -77,7 +75,7 @@ layout = html.Div(
         #     className="search_options",
         #     children=[dcc.RadioItems(options=["Acheteur(s)"])],
         # ),
-        html.Div(id="search_results", className="wrapper"),
+        dbc.Row(id="search_results"),
     ],
 )
 
@@ -92,13 +90,13 @@ layout = html.Div(
 )
 def update_search_results(n_submit, n_clicks, query):
     if query and len(query) >= 1:
-        content = []
+        cols = []
 
         for org_type in ["acheteur", "titulaire"]:
             if org_type == "acheteur":
-                dff = df_acheteurs
+                dff = DF_ACHETEURS
             elif org_type == "titulaire":
-                dff = df_titulaires
+                dff = DF_TITULAIRES
             else:
                 raise ValueError(f"{org_type} is not supported")
 
@@ -109,9 +107,8 @@ def update_search_results(n_submit, n_clicks, query):
             # Format output
             columns, tooltip = setup_table_columns(results, hideable=False)
 
-            org_content = [
+            col = (
-                html.Div(
+                dbc.Col(
-                    className=f"results_{org_type}",
                     children=[
                         html.H3(f"{org_type.title()}s : {count}"),
                         DataTable(
@@ -123,12 +120,13 @@ def update_search_results(n_submit, n_clicks, query):
                             filter_action="none",
                         ),
                     ],
+                    md=6,
                 )
                 if count > 0
-                else html.P(f"Aucun {org_type} trouvé."),
+                else html.P(f"Aucun {org_type} trouvé.")
-            ]
+            )
-            content.extend(org_content)
+            cols.append(col)
-            style = {"textAlign": "center", "display": "none"}
 
-        return content, style
+        style = {"textAlign": "center", "display": "none"}
+        return cols, style
     return html.P(""), {"textAlign": "center"}

src/pages/statistiques.py

@@ -1,85 +0,0 @@
-from datetime import datetime
-
-from dash import dcc, html, register_page
-
-from src.figures import (
-    get_barchart_sources,
-    get_duplicate_matrix,
-    get_map_count_marches,
-    get_yearly_statistics,
-)
-from src.utils import df, format_number, get_statistics, meta_content
-
-name = "Statistiques"
-
-register_page(
-    __name__,
-    path="/statistiques",
-    title="Statistiques | decp.info",
-    name=name,
-    description="Visualisez l'état de la publication des données essentielles des marchés publics en France.",
-    image_url=meta_content["image_url"],
-    order=3,
-)
-
-statistics: dict = get_statistics()
-today_str = datetime.fromisoformat(statistics["datetime"]).strftime("%d/%m/%Y")
-
-layout = [
-    html.Div(
-        className="container",
-        children=[
-            html.H2(name),
-            dcc.Loading(
-                overlay_style={"visibility": "visible", "filter": "blur(2px)"},
-                id="loading-statistques",
-                type="default",
-                children=[
-                    html.Div(
-                        children=[
-                            dcc.Markdown(f"""
-                            La publication de données essentielles de marchés publics (DECP) est souvent effectuée par
-                            les plateformes de marchés publics (profils d'acheteurs). Cependant, certaines plateformes ne publient pas,
-                            ou publient d'une manière qui rend la récupération des données compliquée. Les données présentées sur ce site
-                            ne représentent donc pas tous les marchés attribués en France, seulement une partie significative.
-
-                            L'ajout de nouvelles plateformes [est en cours](https://github.com/ColinMaudry/decp-processing/issues?q=is%3Aissue%20label%3A%22source%20de%20donn%C3%A9es%22),
-                            toutes les [contributions](/a-propos#contribuer) sont les bienvenues pour atteindre l'exhaustivité.
-
-                            Les statistiques publiées sur cette page ont été produites automatiquement à partir des données les plus récentes ({today_str}).
-                            """),
-                            html.H3(
-                                "Statistiques générales sur les marchés",
-                                id="marches",
-                            ),
-                            html.P(
-                                "À noter qu'une fois un marché attribué ses données essentielles peuvent malheureusement mettre plusieurs mois à être publiées par l'acheteur."
-                            ),
-                            html.H4("Statistiques cumulées"),
-                            dcc.Markdown(f"""
-                            - Nombre de marchés publics et accord-cadres : {format_number(statistics["nb_marches"])}
-                            - Nombre d'acheteurs publics (SIRET) : {format_number(statistics["nb_acheteurs_uniques"])}
-                            - Nombre de titulaires (SIRET) : {format_number(statistics["nb_titulaires_uniques"])}
-
-                            Je ne publie pas encore de statistiques sur les montants de marchés car je n'ai pas encore trouvé la bonne formule pour traiter les trop nombreux montants fantaisistes qui polluent les calculs.
-                                                        """),
-                            html.H4("Statistiques par année"),
-                            get_yearly_statistics(statistics, today_str),
-                            dcc.Graph(figure=get_map_count_marches(df)),
-                            get_duplicate_matrix(),
-                            html.H3("Nombre de marchés par source dans le temps"),
-                            dcc.Graph(
-                                figure=get_barchart_sources(df, "dateNotification")
-                            ),
-                            dcc.Graph(
-                                figure=get_barchart_sources(
-                                    df, "datePublicationDonnees"
-                                )
-                            ),
-                        ],
-                    )
-                ],
-            ),
-        ],
-    )
-]

src/pages/tableau.py

@@ -4,6 +4,7 @@ import urllib.parse
 import uuid
 from datetime import datetime
 
+import dash_bootstrap_components as dbc
 import polars as pl
 from dash import (
     ClientsideFunction,
@@ -18,51 +19,56 @@ from dash import (
     register_page,
 )
 
-from src.figures import DataTable
+from src.db import query_marches, schema
-from src.utils import (
+from src.figures import DataTable, make_column_picker
-    df,
+from src.utils import logger
+from src.utils.seo import META_CONTENT
+from src.utils.table import (
+    COLUMNS,
     filter_table_data,
     get_default_hidden_columns,
     invert_columns,
-    logger,
+    prepare_table_data,
-    meta_content,
-    schema,
     sort_table_data,
 )
-from utils import prepare_table_data
 
 update_date_timestamp = os.path.getmtime(os.getenv("DATA_FILE_PARQUET_PATH"))
 update_date = datetime.fromtimestamp(update_date_timestamp).strftime("%d/%m/%Y")
 update_date_iso = datetime.fromtimestamp(update_date_timestamp).isoformat()
 
 
-name = "Tableau"
+NAME = "Tableau"
 register_page(
     __name__,
     path="/tableau",
     title="Tableau des marchés publics | decp.info",
-    name=name,
+    name=NAME,
     description="Consultez, filtrez et exportez les données essentielles de la commande publique sous forme de tableau.",
-    image_url=meta_content["image_url"],
+    image_url=META_CONTENT["image_url"],
     order=1,
 )
 
-datatable = html.Div(
+DATATABLE = html.Div(
     className="marches_table",
     children=DataTable(
-        dtid="table",
+        dtid="tableau_datatable",
+        persisted_props=["filter_query", "sort_by"],
+        persistence_type="local",
+        persistence=True,
         page_size=20,
         page_action="custom",
         filter_action="custom",
         sort_action="custom",
-        hidden_columns=get_default_hidden_columns(None),
+        hidden_columns=[],
-        columns=[{"id": col, "name": col} for col in df.columns],
+        columns=[{"id": col, "name": col} for col in schema.names()],
     ),
 )
 
 layout = [
     dcc.Location(id="tableau_url", refresh=False),
-    dcc.Store(id="filter-cleanup-trigger"),
+    dcc.Store(id="filter-cleanup-trigger-tableau"),
+    dcc.Store(id="tableau-hidden-columns", storage_type="local"),
+    dcc.Store(id="tableau-table"),
     html.Script(
         type="application/ld+json",
         id="dataset_jsonld",
@@ -121,15 +127,26 @@ layout = [
         ],
     ),
     dcc.Markdown(
-        f"Ce tableau vous permet d'appliquer un filtre sur une ou plusieurs colonnes, et ainsi produire la liste de marchés dont vous avez besoin ([exemple de filtre](/tableau?filtres=%7Bacheteur_id%7D+icontains+24350013900189+%26%26+%7BdateNotification%7D+icontains+2025%2A+%26%26+%7Bmontant%7D+i%3C+40000+%26%26+%7Bobjet%7D+icontains+voirie&colonnes=uid%2Cacheteur_id%2Cacheteur_nom%2Ctitulaire_id%2Ctitulaire_nom%2Cobjet%2Cmontant%2CdureeMois%2CdateNotification%2Cacheteur_departement_code%2CsourceDataset)). Par défaut seules quelques colonnes sont affichées, mais vous pouvez en afficher jusqu'à {str(df.width)} en cliquant sur le bouton **Colonnes affichées**. Cet outil est assez puissant, je vous recommande de lire le mode d'emploi pour en tirer pleinement partie.",
+        f"Ce tableau contient tous les marchés attribués en France. Il vous permet d'appliquer un filtre sur une ou plusieurs colonnes, et ainsi produire la liste de marchés dont vous avez besoin (exemples : [marchés de voirie < 40 k€ en 2025](/tableau?filtres=%7Bacheteur_id%7D+icontains+24350013900189+%26%26+%7BdateNotification%7D+icontains+2025%2A+%26%26+%7Bmontant%7D+i%3C+40000+%26%26+%7Bobjet%7D+icontains+voirie&colonnes=uid%2Cacheteur_id%2Cacheteur_nom%2Ctitulaire_id%2Ctitulaire_nom%2Cobjet%2Cmontant%2CdureeMois%2CdateNotification%2Cacheteur_departement_code%2CsourceDataset), [marchés > 500 k€ avec clause sociale attribués à des PME à plus de 100 km dans le Var](/tableau?filtres=%7Btitulaire_categorie%7D+icontains+PME+%26%26+%7Btitulaire_distance%7D+i%3E+100+%26%26+%7Bmontant%7D+i%3E+500000+%26%26+%7Bacheteur_departement_code%7D+icontains+83+%26%26+%7BconsiderationsSociales%7D+icontains+clause&colonnes=uid%2Cacheteur_id%2Cacheteur_nom%2Ctitulaire_id%2Ctitulaire_nom%2Cobjet%2Cmontant%2CdureeMois%2CdateNotification%2CconsiderationsSociales%2Ctitulaire_distance%2Cacheteur_departement_code%2Ctitulaire_categorie%2CsourceDataset)). Par défaut seules quelques colonnes sont affichées, mais vous pouvez en afficher jusqu'à {len(schema.names())} en cliquant sur le bouton **Choisir les colonnes**. Cet outil est assez puissant, je vous recommande de lire le mode d'emploi pour en tirer pleinement partie.",
         style={"maxWidth": "1000px"},
     ),
     html.Div(
-        html.Details(
+        [],
-            children=[
+        id="header",
-                html.Summary(
-                    html.H4("Mode d'emploi", style={"textDecoration": "underline"}),
     ),
+    dcc.Loading(
+        overlay_style={"visibility": "visible", "filter": "blur(2px)"},
+        id="loading-home",
+        type="default",
+        children=[
+            html.Div(
+                [
+                    # Modal du mode d'emploi
+                    dbc.Button("Mode d'emploi", id="tableau_help_open"),
+                    dbc.Modal(
+                        [
+                            dbc.ModalHeader(dbc.ModalTitle("Mode d'emploi")),
+                            dbc.ModalBody(
                                 dcc.Markdown(
                                     dangerously_allow_html=True,
                                     children=f"""
@@ -137,6 +154,10 @@ layout = [
 
             Pour voir la définition d'une colonne, passez votre souris sur son en-tête.
 
+            ##### Vos réglages sont persistents
+
+            Les filtres, les tris et le choix de colonnes sont automatiquement enregistrés dans votre navigateur et persistent même si vous changez de page ou si vous fermez votre navigateur. À votre retour, vous retrouverez cette page comme vous l'avez laissée.
+
             ##### Appliquer des filtres
 
             Vous pouvez appliquer un filtre pour chaque colonne en entrant du texte sous le nom de la colonne, puis en tapant sur `Entrée`.
@@ -144,11 +165,9 @@ layout = [
             - Champs textuels : la recherche retourne les valeurs qui contiennent le texte recherché et n'est pas sensible à la casse (majuscules/minuscules).
                 - Exemple : `rennes` retourne "RENNES METROPOLE".
                 - Les guillemets simples (apostrophe du 4) doivent être prédédées d'une barre oblique (AltGr + 8). Exemple : `services d\\\'assurances`
-        - Lorsque vous ouvrez une URL de vue (voir "Partager une vue" plus bas), le format équivalent `icontains rennes` est utilisé. Mais dans vos filtres pas besoin de taper `icontains` !
             - Champs numériques (Durée en mois, Montant, ...) : vous pouvez...
                 - soit taper un nombre pour trouver les valeurs strictement égales. Exemple : `12` ne retourne que des 12
                 - soit le précéder de **>** ou **<** pour filtrer les valeurs supérieures ou inférieures. Exemple pour les offres reçues : `> 4` retourne les marchés ayant reçu plus de 4 offres.
-        - lorsque vous ouvrez une URL de vue (voir "Partager une vue" plus bas), le format équivalent `i<` ou `i>` est utilisé, mais c'est un bug : vous n'avez pas besoin de taper le `i` pour appliquer ce filtre.
             - Champs date (Date de notification, ...) : vous pouvez également utiliser **>** ou **<**. Exemples :
                 - `< 2024-01-31` pour "avant le 31 janvier 2024"
                 - `2024` pour "en 2024", `> 2022` pour "à partir de 2022".
@@ -156,7 +175,7 @@ layout = [
                 - pour chercher du texte qui **commence par** votre texte, entrez `texte*`. C'est par exemple utile pour filtrer des acheteurs ou titulaires par numéro SIREN (`123456789*`) ou les marchés sur une année en particulier (`2024*`)
                 - pour chercher du texte qui **finit par** votre texte, entrez `*texte`
 
-    Vous pouvez filtrer plusieurs colonnes à la fois. Vos filtres sont remis à zéro quand vous rafraîchissez la page.
+            Vous pouvez filtrer plusieurs colonnes à la fois.
 
             ##### Trier les données
 
@@ -168,13 +187,13 @@ layout = [
 
             ##### Afficher plus de colonnes
 
-    Par défaut, un nombre réduit de colonnes est affiché pour ne pas surcharger la page. Mais vous avez le choix parmi {str(df.width)} colonnes, ce serait dommage de vous limiter !
+            Par défaut, un nombre réduit de colonnes est affiché pour ne pas surcharger la page. Mais vous avez le choix parmi {len(schema.names())} colonnes, ce serait dommage de vous limiter !
 
-    Pour afficher plus de colonnes, cliquez sur le bouton **Colonnes affichées** et cochez les colonnes pour les afficher.
+            Pour afficher plus de colonnes, cliquez sur le bouton **Choisir les colonnes** et cochez les colonnes pour les afficher.
 
             ##### Partager une vue
 
-    Une vue est un ensemble de filtres, de tris et de choix de colonnes que vous avez appliqué. Cliquez sur l'icône <img src="/assets/copy.svg" alt="drawing" width="20"/> pour copier une adresse Web qui reproduit la vue courante à l'identique : en la collant dans la barre d'adresse d'un navigateur, vous ouvrez la vue Tableau avec les mêmes paramètres.
+            Une vue est un ensemble de filtres, de tris et de choix de colonnes que vous avez appliqués. Cliquez sur **Partager** pour copier une adresse Web qui reproduit la vue courante à l'identique : en la collant dans la barre d'adresse d'un navigateur, vous ouvrez la vue Tableau avec les mêmes paramètres.
 
             Pratique pour partager une vue avec un·e collègue, sur les réseaux sociaux, ou la sauvegarder pour plus tard.
 
@@ -189,28 +208,33 @@ layout = [
 
             """,
                                 ),
+                            ),
+                            dbc.ModalFooter(
+                                dbc.Button(
+                                    "Fermer",
+                                    id="tableau_help_close",
+                                    className="ms-auto",
+                                    n_clicks=0,
+                                )
+                            ),
                         ],
-            id="instructions",
+                        id="tableau_help",
+                        is_open=False,
+                        fullscreen="md-down",
+                        scrollable=True,
+                        size="lg",
                     ),
-        id="header",
+                    # Bouton modal des colonnes affichées
+                    dbc.Button(
+                        "Choisir les colonnes",
+                        id="tableau_columns_open",
+                        className="column_list",
+                        title="Choisir les colonnes à afficher et masquer",
                     ),
-    # html.Div(
-    #     [
-    #         "Recherche dans objet : ",
-    #         dcc.Input(id="search", value="", type="text"),
-    #     ]
-    # )]),
-    dcc.Loading(
-        overlay_style={"visibility": "visible", "filter": "blur(2px)"},
-        id="loading-home",
-        type="default",
-        children=[
-            html.Div(
-                [
                     html.P("lignes", id="nb_rows"),
                     html.Div(id="copy-container"),
                     dcc.Input(id="share-url", readOnly=True, style={"display": "none"}),
-                    html.Button(
+                    dbc.Button(
                         "Téléchargement désactivé au-delà de 65 000 lignes",
                         id="btn-download-data",
                         disabled=True,
@@ -218,31 +242,61 @@ layout = [
                     dcc.Download(id="download-data"),
                     dcc.Store(id="filtered_data", storage_type="memory"),
                     html.P("Données mises à jour le " + str(update_date)),
+                    dbc.Button(
+                        "Remettre à zéro",
+                        title="Supprime tous les filtres et les tris. Autrement ils sont conservés même si vous fermez la page.",
+                        id="btn-tableau-reset",
+                    ),
                 ],
                 className="table-menu",
             ),
-            datatable,
+            dbc.Modal(
+                [
+                    dbc.ModalHeader(dbc.ModalTitle("Choix des colonnes à afficher")),
+                    dbc.ModalBody(
+                        id="tableau_columns_body",
+                        children=make_column_picker("tableau"),
+                    ),
+                    dbc.ModalFooter(
+                        dbc.Button(
+                            "Fermer",
+                            id="tableau_columns_close",
+                            className="ms-auto",
+                            n_clicks=0,
+                        )
+                    ),
+                ],
+                id="tableau_columns",
+                is_open=False,
+                fullscreen="md-down",
+                scrollable=True,
+                size="xl",
+            ),
+            DATATABLE,
         ],
     ),
 ]
 
 
 @callback(
-    Output("table", "data"),
+    Output("tableau_datatable", "data"),
-    Output("table", "columns"),
+    Output("tableau_datatable", "columns"),
-    Output("table", "tooltip_header"),
+    Output("tableau_datatable", "tooltip_header"),
-    Output("table", "data_timestamp"),
+    Output("tableau_datatable", "data_timestamp"),
     Output("nb_rows", "children"),
     Output("btn-download-data", "disabled"),
     Output("btn-download-data", "children"),
     Output("btn-download-data", "title"),
-    Input("table", "page_current"),
+    Output("filter-cleanup-trigger-tableau", "data", allow_duplicate=True),
-    Input("table", "page_size"),
+    Input("tableau_url", "href"),
-    Input("table", "filter_query"),
+    Input("tableau_datatable", "page_current"),
-    Input("table", "sort_by"),
+    Input("tableau_datatable", "page_size"),
-    State("table", "data_timestamp"),
+    Input("tableau_datatable", "filter_query"),
+    Input("tableau_datatable", "sort_by"),
+    State("tableau_datatable", "data_timestamp"),
+    prevent_initial_call=True,
 )
-def update_table(page_current, page_size, filter_query, sort_by, data_timestamp):
+def update_table(href, page_current, page_size, filter_query, sort_by, data_timestamp):
     # if ctx.triggered_id != "url":
     #     search_params = None
     # else:
@@ -255,13 +309,13 @@ def update_table(page_current, page_size, filter_query, sort_by, data_timestamp)
 @callback(
     Output("download-data", "data"),
     Input("btn-download-data", "n_clicks"),
-    State("table", "filter_query"),
+    State("tableau_datatable", "filter_query"),
-    State("table", "sort_by"),
+    State("tableau_datatable", "sort_by"),
-    State("table", "hidden_columns"),
+    State("tableau_datatable", "hidden_columns"),
     prevent_initial_call=True,
 )
 def download_data(n_clicks, filter_query, sort_by, hidden_columns: list = None):
-    lff: pl.LazyFrame = df.lazy()  # start from the original data
+    lff: pl.LazyFrame = query_marches().lazy()
 
     # Les colonnes masquées sont supprimées
     if hidden_columns:
@@ -270,7 +324,7 @@ def download_data(n_clicks, filter_query, sort_by, hidden_columns: list = None):
     if filter_query:
         lff = filter_table_data(lff, filter_query, "tab download")
 
-    if len(sort_by) > 0:
+    if sort_by and len(sort_by) > 0:
         lff = sort_table_data(lff, sort_by)
 
     def to_bytes(buffer):
@@ -281,20 +335,21 @@ def download_data(n_clicks, filter_query, sort_by, hidden_columns: list = None):
 
 
 @callback(
-    Output("table", "filter_query"),
+    Output("tableau_datatable", "filter_query"),
-    Output("table", "sort_by"),
+    Output("tableau_datatable", "sort_by"),
-    Output("table", "hidden_columns"),
+    Output("tableau-hidden-columns", "data"),
-    Output("tableau_url", "search", allow_duplicate=True),
+    Output("tableau_url", "search"),
-    Output("filter-cleanup-trigger", "data"),
+    Output("filter-cleanup-trigger-tableau", "data"),
     Input("tableau_url", "search"),
-    prevent_initial_call=True,
+    State("tableau_datatable", "filter_query"),
+    State("tableau_datatable", "sort_by"),
 )
-def restore_view_from_url(search):
+def restore_view_from_url(search, stored_filters, stored_sort):
-    if not search:
+    if not search and not stored_filters:
         return no_update, no_update, no_update, no_update, no_update
 
-    params = urllib.parse.parse_qs(search.lstrip("?"))
+    params = urllib.parse.parse_qs(search.lstrip("?")) if search else {}
-    logger.debug("params", params)
+    logger.debug("params " + json.dumps(params, indent=2))
 
     filter_query = no_update
     sort_by = no_update
@@ -304,28 +359,37 @@ def restore_view_from_url(search):
     if "filtres" in params:
         filter_query = params["filtres"][0]
         trigger_cleanup = str(uuid.uuid4())
+    elif stored_filters:
+        filter_query = stored_filters
+        trigger_cleanup = str(uuid.uuid4())
 
     if "tris" in params:
         try:
             sort_by = json.loads(params["tris"][0])
         except json.JSONDecodeError:
             pass
+    elif stored_sort:
+        sort_by = stored_sort
 
     if "colonnes" in params:
-        columns = params["colonnes"][0].split(",")
+        table_columns = params["colonnes"][0].split(",")
-        verified_columns = [column for column in columns if column in schema.names()]
+        verified_columns = [
+            column for column in table_columns if column in schema.names()
+        ]
         hidden_columns = invert_columns(verified_columns)
 
     return filter_query, sort_by, hidden_columns, "", trigger_cleanup
 
 
+# Pour nettoyer les icontains et i< des filtres
+# voir aussi src/assets/dash_clientside.js
 clientside_callback(
     ClientsideFunction(
         namespace="clientside",
         function_name="clean_filters",
     ),
-    Output("filter-cleanup-trigger", "data", allow_duplicate=True),
+    Output("filter-cleanup-trigger-tableau", "data", allow_duplicate=True),
-    Input("filter-cleanup-trigger", "data"),
+    Input("filter-cleanup-trigger-tableau", "data"),
     prevent_initial_call=True,
 )
 
@@ -333,9 +397,9 @@ clientside_callback(
 @callback(
     Output("share-url", "value"),
     Output("copy-container", "children"),
-    Input("table", "filter_query"),
+    Input("tableau_datatable", "filter_query"),
-    Input("table", "sort_by"),
+    Input("tableau_datatable", "sort_by"),
-    Input("table", "hidden_columns"),
+    Input("tableau_datatable", "hidden_columns"),
     State("tableau_url", "href"),
     prevent_initial_call=True,
 )
@@ -354,9 +418,9 @@ def sync_url_and_reset_button(filter_query, sort_by, hidden_columns, href):
         params["tris"] = json.dumps(sort_by)
 
     if hidden_columns:
-        columns = invert_columns(hidden_columns)
+        table_columns = invert_columns(hidden_columns)
-        columns = ",".join(columns)
+        table_columns = ",".join(table_columns)
-        params["colonnes"] = columns
+        params["colonnes"] = table_columns
 
     query_string = urllib.parse.urlencode(params)
     full_url = f"{base_url}?{query_string}" if query_string else base_url
@@ -372,6 +436,13 @@ def sync_url_and_reset_button(filter_query, sort_by, hidden_columns, href):
             "cursor": "pointer",
         },
         className="fa fa-link",
+        children=[
+            dbc.Button(
+                "Partager la vue",
+                className="btn btn-primary",
+                title="Copier l'adresse de cette vue (filtres, tris, choix de colonnes) pour la partager.",
+            )
+        ],
     )
 
     return full_url, copy_button
@@ -389,3 +460,76 @@ def show_confirmation(n_clicks):
             style={"color": "green", "fontWeight": "bold", "marginLeft": "10px"},
         )
     return no_update
+
+
+@callback(
+    Output("tableau_help", "is_open"),
+    [Input("tableau_help_open", "n_clicks"), Input("tableau_help_close", "n_clicks")],
+    [State("tableau_help", "is_open")],
+)
+def toggle_tableau_help(click_open, click_close, is_open):
+    if click_open or click_close:
+        return not is_open
+    return is_open
+
+
+@callback(
+    Output("tableau-hidden-columns", "data", allow_duplicate=True),
+    Input("tableau_column_list", "selected_rows"),
+    prevent_initial_call=True,
+)
+def update_hidden_columns_from_checkboxes(selected_columns):
+    if selected_columns:
+        selected_columns = [COLUMNS[i] for i in selected_columns]
+        hidden_columns = [col for col in COLUMNS if col not in selected_columns]
+        return hidden_columns
+    else:
+        return []
+
+
+@callback(
+    Output("tableau_datatable", "hidden_columns"),
+    Input(
+        "tableau-hidden-columns",
+        "data",
+    ),
+)
+def store_hidden_columns(hidden_columns):
+    if hidden_columns is None:
+        hidden_columns = get_default_hidden_columns("tableau")
+    return hidden_columns
+
+
+@callback(
+    Output("tableau_column_list", "selected_rows"),
+    Input("tableau_datatable", "hidden_columns"),
+    State("tableau_column_list", "selected_rows"),  # pour éviter la boucle infinie
+)
+def update_checkboxes_from_hidden_columns(hidden_cols, current_checkboxes):
+    hidden_cols = hidden_cols or get_default_hidden_columns("tableau")
+
+    # Show all columns that are NOT hidden
+    visible_cols = [COLUMNS.index(col) for col in COLUMNS if col not in hidden_cols]
+    return visible_cols
+
+
+@callback(
+    Output("tableau_columns", "is_open"),
+    Input("tableau_columns_open", "n_clicks"),
+    Input("tableau_columns_close", "n_clicks"),
+    State("tableau_columns", "is_open"),
+)
+def toggle_tableau_columns(click_open, click_close, is_open):
+    if click_open or click_close:
+        return not is_open
+    return is_open
+
+
+@callback(
+    Output("tableau_datatable", "filter_query", allow_duplicate=True),
+    Output("tableau_datatable", "sort_by", allow_duplicate=True),
+    Input("btn-tableau-reset", "n_clicks"),
+    prevent_initial_call=True,
+)
+def reset_view(n_clicks):
+    return "", []

src/pages/titulaire.py

@@ -1,32 +1,48 @@
 import datetime
+from typing import Any
 
+import dash_bootstrap_components as dbc
 import polars as pl
-from dash import Input, Output, State, callback, dcc, html, register_page
+from dash import (
+    ClientsideFunction,
+    Input,
+    Output,
+    State,
+    callback,
+    clientside_callback,
+    dcc,
+    html,
+    register_page,
+)
 
-from src.callbacks import get_top_org_table
+from src.db import query_marches, schema
-from src.figures import DataTable, point_on_map
+from src.figures import (
-from src.utils import (
+    DataTable,
-    df,
+    get_distance_histogram,
-    df_titulaires,
+    get_top_org_table,
+    make_column_picker,
+    point_on_map,
+)
+from src.utils.data import DF_TITULAIRES, get_annuaire_data, get_departement_region
+from src.utils.frontend import get_button_properties
+from src.utils.seo import META_CONTENT
+from src.utils.table import (
+    COLUMNS,
     filter_table_data,
     format_number,
-    get_annuaire_data,
-    get_button_properties,
     get_default_hidden_columns,
-    get_departement_region,
-    meta_content,
     prepare_table_data,
     sort_table_data,
 )
 
 
 def get_title(titulaire_id: str = None) -> str:
-    titulaire_nom = (
+    titulaire_nom = DF_TITULAIRES.filter(pl.col("titulaire_id") == titulaire_id).select(
-        df_titulaires.filter(pl.col("titulaire_id") == titulaire_id)
+        "titulaire_nom"
-        .select("titulaire_nom")
-        .item()
     )
-    return f"Marchés publics remportés par {titulaire_nom} | decp.info"
+    if titulaire_nom.height > 0:
+        return f"Marchés publics remportés par {titulaire_nom.item(0, 0)} | decp.info"
+    return "Marchés publics remportés | decp.info"
 
 
 register_page(
@@ -35,43 +51,53 @@ register_page(
     title=get_title,
     name="Titulaire",
     description="Consultez les marchés publics remportés par ce titulaire.",
-    image_url=meta_content["image_url"],
+    image_url=META_CONTENT["image_url"],
     order=5,
 )
 
-datatable = html.Div(
+DATATABLE = html.Div(
     className="marches_table",
     children=DataTable(
         dtid="titulaire_datatable",
+        persistence=True,
+        persistence_type="local",
+        persisted_props=["filter_query", "sort_by"],
         page_action="custom",
         filter_action="custom",
         sort_action="custom",
         page_size=10,
-        hidden_columns=get_default_hidden_columns(page="titulaire"),
+        hidden_columns=[],
+        columns=[{"id": col, "name": col} for col in schema.names()],
     ),
 )
 
 layout = [
     dcc.Store(id="titulaire_data", storage_type="memory"),
+    dcc.Store(id="titulaire-hidden-columns", storage_type="local"),
+    dcc.Store(id="filter-cleanup-trigger-titulaire"),
     dcc.Location(id="titulaire_url", refresh="callback-nav"),
     html.Div(
         children=[
             html.Div(
-                className="wrapper",
+                style={"marginBottom": "50px"},
                 children=[
+                    dbc.Row(
+                        className="mb-2",
+                        children=[
+                            dbc.Col(
                                 html.H2(
-                        className="org_title",
                                     children=[
                                         html.Span(id="titulaire_siret"),
                                         " - ",
                                         html.Span(id="titulaire_nom"),
                                     ],
                                 ),
-                    html.Div(
+                                width=8,
-                        className="org_year",
+                            ),
-                        children=dcc.Dropdown(
+                            dbc.Col(
+                                dcc.Dropdown(
                                     id="titulaire_year",
-                            options=["Toutes"]
+                                    options=["Toutes les années"]
                                     + [
                                         str(year)
                                         for year in range(
@@ -80,28 +106,43 @@ layout = [
                                     ],
                                     placeholder="Année",
                                 ),
+                                width=4,
                             ),
-                    html.Div(
+                        ],
+                    ),
+                    dbc.Row(
+                        className="mb-2",
+                        children=[
+                            dbc.Col(
                                 className="org_infos",
                                 children=[
                                     # TODO: ajouter le type d'acheteur : commune, CD, CR, etc.
-                            html.P(["Commune : ", html.Strong(id="titulaire_commune")]),
+                                    html.P(
+                                        [
+                                            "Commune : ",
+                                            html.Strong(id="titulaire_commune"),
+                                        ]
+                                    ),
                                     html.P(
                                         [
                                             "Département : ",
                                             html.Strong(id="titulaire_departement"),
                                         ]
                                     ),
-                            html.P(["Région : ", html.Strong(id="titulaire_region")]),
+                                    html.P(
+                                        [
+                                            "Région : ",
+                                            html.Strong(id="titulaire_region"),
+                                        ]
+                                    ),
                                     html.A(
                                         id="titulaire_lien_annuaire",
                                         children="Plus de détails sur l'Annuaire des entreprises",
-                                target="_blank",
                                     ),
                                 ],
+                                width=4,
                             ),
-                    html.Div(
+                            dbc.Col(
-                        className="org_stats",
                                 children=[
                                     html.P(id="titulaire_titre_stats"),
                                     html.P(id="titulaire_marches_remportes"),
@@ -109,16 +150,33 @@ layout = [
                                     html.Button(
                                         "Téléchargement au format Excel",
                                         id="btn-download-data-titulaire",
+                                        className="btn btn-primary",
                                     ),
                                     dcc.Download(id="download-data-titulaire"),
                                 ],
+                                width=4,
                             ),
-                    html.Div(className="org_map", id="titulaire_map"),
+                            dbc.Col(
+                                id="titulaire_map",
+                                width=4,
+                            ),
+                        ],
+                    ),
+                    dbc.Row(
+                        children=[
+                            dbc.Col(
                                 html.Div(
-                        className="org_top",
                                     children=[
                                         html.H3("Top acheteurs"),
-                            html.Div(className="marches_table", id="top10_acheteurs"),
+                                        html.Div(
+                                            className="marches_table",
+                                            id="top10_acheteurs",
+                                        ),
+                                    ],
+                                ),
+                                width=8,
+                            ),
+                            dbc.Col(id="titulaire-distance-histogram", width=4),
                         ],
                     ),
                 ],
@@ -132,17 +190,54 @@ layout = [
                 children=[
                     html.Div(
                         [
+                            # Bouton modal des colonnes affichées
+                            dbc.Button(
+                                "Colonnes affichées",
+                                id="titulaire_columns_open",
+                                className="column_list",
+                            ),
                             html.P("lignes", id="titulaire_nb_rows"),
                             html.Button(
                                 "Téléchargement désactivé au-delà de 65 000 lignes",
                                 id="btn-download-filtered-data-titulaire",
                                 disabled=True,
+                                className="btn btn-primary",
                             ),
                             dcc.Download(id="titulaire-download-filtered-data"),
+                            dbc.Button(
+                                "Remise à zéro",
+                                title="Supprime tous les filtres et les tris. Autrement ils sont conservés même si vous fermez la page.",
+                                id="btn-titulaire-reset",
+                                className="btn btn-primary",
+                            ),
                         ],
                         className="table-menu",
                     ),
-                    datatable,
+                    dbc.Modal(
+                        [
+                            dbc.ModalHeader(
+                                dbc.ModalTitle("Choix des colonnes à afficher")
+                            ),
+                            dbc.ModalBody(
+                                id="titulaire_columns_body",
+                                children=make_column_picker("titulaire"),
+                            ),
+                            dbc.ModalFooter(
+                                dbc.Button(
+                                    "Fermer",
+                                    id="titulaire_columns_close",
+                                    className="ms-auto",
+                                    n_clicks=0,
+                                )
+                            ),
+                        ],
+                        id="titulaire_columns",
+                        is_open=False,
+                        fullscreen="md-down",
+                        scrollable=True,
+                        size="xl",
+                    ),
+                    DATATABLE,
                 ],
             ),
         ],
@@ -162,12 +257,11 @@ layout = [
 )
 def update_titulaire_infos(url):
     titulaire_siret = url.split("/")[-1]
-    if len(titulaire_siret) != 14:
-        titulaire_siret = (
-            f"Le SIRET renseigné doit faire 14 caractères ({titulaire_siret})"
-        )
     data = get_annuaire_data(titulaire_siret)
-    data_etablissement = data["matching_etablissements"][0]
+    data_etablissement = data.get("matching_etablissements") if data else None
+    if data_etablissement:
+        data_etablissement = data_etablissement[0]
+
         titulaire_map = point_on_map(
             data_etablissement["latitude"], data_etablissement["longitude"]
         )
@@ -178,10 +272,23 @@ def update_titulaire_infos(url):
         lien_annuaire = (
             f"https://annuaire-entreprises.data.gouv.fr/etablissement/{titulaire_siret}"
         )
+        raison_sociale = data["nom_raison_sociale"]
+        libelle_commune = data_etablissement["libelle_commune"]
+
+    else:
+        titulaire_map = html.Div()
+        code_departement, nom_departement, nom_region = "", "", ""
+        departement = ""
+        lien_annuaire = ""
+        raison_sociale = html.Span(
+            f"N° SIREN inconnu de l'INSEE ({titulaire_siret[:9]})"
+        )
+        libelle_commune = ""
+
     return (
         titulaire_siret,
-        data["nom_raison_sociale"],
+        raison_sociale,
-        data_etablissement["libelle_commune"],
+        libelle_commune,
         titulaire_map,
         departement,
         nom_region,
@@ -230,21 +337,18 @@ def update_titulaire_stats(data):
 )
 def get_titulaire_marches_data(url, titulaire_year: str) -> tuple:
     titulaire_siret = url.split("/")[-1]
-    lff = df.lazy()
+    lff = query_marches(
-    lff = lff.filter(
+        "titulaire_id = ? AND titulaire_typeIdentifiant = 'SIRET'",
-        (pl.col("titulaire_id") == titulaire_siret)
+        (titulaire_siret,),
-        & (pl.col("titulaire_typeIdentifiant") == "SIRET")
+    ).lazy()
-    )
+    if titulaire_year and titulaire_year != "Toutes les années":
-    if titulaire_year and titulaire_year != "Toutes":
         lff = lff.filter(
             pl.col("dateNotification").cast(pl.String).str.starts_with(titulaire_year)
         )
     lff = lff.sort(["dateNotification", "uid"], descending=True, nulls_last=True)
     lff = lff.fill_null("")
-
     dff: pl.DataFrame = lff.collect(engine="streaming")
     download_disabled, download_text, download_title = get_button_properties(dff.height)
-
     data = dff.to_dicts()
     return data, download_disabled, download_text, download_title
 
@@ -258,6 +362,8 @@ def get_titulaire_marches_data(url, titulaire_year: str) -> tuple:
     Output("btn-download-filtered-data-titulaire", "disabled"),
     Output("btn-download-filtered-data-titulaire", "children"),
     Output("btn-download-filtered-data-titulaire", "title"),
+    Output("filter-cleanup-trigger-titulaire", "data"),
+    Input(component_id="titulaire_url", component_property="href"),
     Input("titulaire_data", "data"),
     Input("titulaire_datatable", "page_current"),
     Input("titulaire_datatable", "page_size"),
@@ -266,7 +372,7 @@ def get_titulaire_marches_data(url, titulaire_year: str) -> tuple:
     State("titulaire_datatable", "data_timestamp"),
 )
 def get_last_marches_data(
-    data, page_current, page_size, filter_query, sort_by, data_timestamp
+    href, data, page_current, page_size, filter_query, sort_by, data_timestamp
 ) -> list[dict]:
     return prepare_table_data(
         data,
@@ -284,7 +390,7 @@ def get_last_marches_data(
     Input(component_id="titulaire_data", component_property="data"),
 )
 def get_top_acheteurs(data):
-    return get_top_org_table(data, "acheteur")
+    return get_top_org_table(data, "acheteur", ["titulaire_distance"])
 
 
 @callback(
@@ -297,7 +403,7 @@ def get_top_acheteurs(data):
 )
 def download_titulaire_data(
     n_clicks,
-    data: [dict],
+    data: list[dict[str, Any]],
     titulaire_nom: str,
     annee: str,
 ):
@@ -305,7 +411,7 @@ def download_titulaire_data(
 
     def to_bytes(buffer):
         df_to_download.write_excel(
-            buffer, worksheet="DECP" if annee in ["Toutes", None] else annee
+            buffer, worksheet="DECP" if annee in ["Toutes les années", None] else annee
         )
 
     date = datetime.datetime.now().strftime("%Y-%m-%d_%H:%M:%S")
@@ -346,3 +452,96 @@ def download_filtered_titulaire_data(
     return dcc.send_bytes(
         to_bytes, filename=f"decp_filtrées_{titulaire_nom}_{date}.xlsx"
     )
+
+
+# Pour nettoyer les icontains et i< des filtres
+# voir aussi src/assets/dash_clientside.js
+clientside_callback(
+    ClientsideFunction(
+        namespace="clientside",
+        function_name="clean_filters",
+    ),
+    Output("filter-cleanup-trigger-titulaire", "data", allow_duplicate=True),
+    Input("filter-cleanup-trigger-titulaire", "data"),
+    prevent_initial_call=True,
+)
+
+
+@callback(
+    Output("titulaire-hidden-columns", "data", allow_duplicate=True),
+    Input("titulaire_column_list", "selected_rows"),
+    prevent_initial_call=True,
+)
+def update_hidden_columns_from_checkboxes(selected_columns):
+    if selected_columns:
+        selected_columns = [COLUMNS[i] for i in selected_columns]
+        hidden_columns = [col for col in COLUMNS if col not in selected_columns]
+        return hidden_columns
+    else:
+        return []
+
+
+@callback(
+    Output("titulaire_datatable", "hidden_columns"),
+    Input(
+        "titulaire-hidden-columns",
+        "data",
+    ),
+)
+def store_hidden_columns(hidden_columns):
+    if hidden_columns is None:
+        hidden_columns = get_default_hidden_columns("titulaire")
+    return hidden_columns
+
+
+@callback(
+    Output("titulaire_column_list", "selected_rows"),
+    Input("titulaire_datatable", "hidden_columns"),
+    State("titulaire_column_list", "selected_rows"),  # pour éviter la boucle infinie
+)
+def update_checkboxes_from_hidden_columns(hidden_cols, current_checkboxes):
+    hidden_cols = hidden_cols or get_default_hidden_columns("titulaire")
+
+    # Show all columns that are NOT hidden
+    visible_cols = [COLUMNS.index(col) for col in COLUMNS if col not in hidden_cols]
+    return visible_cols
+
+
+@callback(
+    Output("titulaire_columns", "is_open"),
+    Input("titulaire_columns_open", "n_clicks"),
+    Input("titulaire_columns_close", "n_clicks"),
+    State("titulaire_columns", "is_open"),
+)
+def toggle_titulaire_columns(click_open, click_close, is_open):
+    if click_open or click_close:
+        return not is_open
+    return is_open
+
+
+@callback(
+    Output("titulaire_datatable", "filter_query", allow_duplicate=True),
+    Output("titulaire_datatable", "sort_by"),
+    Input("btn-titulaire-reset", "n_clicks"),
+    prevent_initial_call=True,
+)
+def reset_view(n_clicks):
+    return "", []
+
+
+@callback(
+    Output("titulaire-distance-histogram", "children"),
+    Input("titulaire_data", "data"),
+)
+def update_titulaire_distance_histogram(data):
+    lff = pl.LazyFrame(data)
+    if "titulaire_distance" in lff.collect_schema().names():
+        lff = lff.with_columns(
+            pl.col("titulaire_distance").cast(pl.Float64, strict=False)
+        )
+    fig = get_distance_histogram(lff)
+    return [
+        html.H3("Distance acheteur-titulaire"),
+        html.H6("par nombre de marchés", className="card-subtitle mb-2 text-muted"),
+        fig,
+    ]

src/utils/__init__.py

@@ -0,0 +1,19 @@
+import logging
+import os
+
+logging.basicConfig(
+    format="%(asctime)s %(levelname)-8s %(message)s",
+    level=logging.INFO,
+    datefmt="%Y-%m-%d %H:%M:%S",
+)
+DEVELOPMENT = os.getenv("DEVELOPMENT", "False").lower() == "true"
+logger = logging.getLogger("decp.info")
+
+if DEVELOPMENT:
+    logger.setLevel(logging.DEBUG)
+
+DOMAIN_NAME = (
+    "test.decp.info"
+    if os.getenv("DEVELOPMENT", "False").lower() == "true"
+    else "decp.info"
+)

src/utils/data.py

@@ -0,0 +1,214 @@
+import json
+import logging
+import os
+from collections import OrderedDict
+from datetime import datetime, timedelta
+
+import polars as pl
+from httpx import HTTPError, get
+
+from src.db import get_cursor, schema
+from src.utils import logger
+
+logging.getLogger("httpx").setLevel("WARNING")
+
+
+def get_annuaire_data(siret: str) -> dict:
+    url = f"https://recherche-entreprises.api.gouv.fr/search?q={siret}"
+    try:
+        response = get(url).raise_for_status()
+        response = response.json()["results"][0]
+    except (HTTPError, IndexError):
+        response = None
+        logger.warning("Could not fetch data from recherche-entreprises.api.")
+    return response
+
+
+def get_statistics() -> dict:
+    return (
+        get(
+            "https://www.data.gouv.fr/api/1/datasets/r/0ccf4a75-f3aa-4b46-8b6a-18aeb63e36df",
+            follow_redirects=True,
+        )
+        .raise_for_status()
+        .json()
+    )
+
+
+def get_departements() -> dict:
+    with open("data/departements.json", "rb") as f:
+        data = json.load(f)
+        return data
+
+
+def get_departements_geojson() -> dict:
+    with open("./data/departements-1000m.geojson") as f:
+        geojson = json.load(f)
+
+    # Ajout de feature.id
+    for f in geojson["features"]:
+        f["id"] = f["properties"]["code"]
+
+    return geojson
+
+
+def get_departement_region(code_postal):
+    if code_postal > "97000":
+        code_departement = code_postal[:3]
+    else:
+        code_departement = code_postal[:2]
+    nom_departement = DEPARTEMENTS[code_departement]["departement"]
+    nom_region = DEPARTEMENTS[code_departement]["region"]
+    return code_departement, nom_departement, nom_region
+
+
+def get_data_schema() -> dict:
+    # Récupération du schéma des données tabulaires
+    path = os.getenv("DATA_SCHEMA_PATH")
+    if path.startswith("http"):
+        original_schema: dict = get(
+            os.getenv("DATA_SCHEMA_PATH"), follow_redirects=True
+        ).json()
+    elif os.path.exists(path):
+        with open(path) as f:
+            original_schema: dict = json.load(f)
+    else:
+        raise Exception(f"Chemin vers le schéma invalide: {path}")
+
+    new_schema = OrderedDict()
+
+    for col in original_schema["fields"]:
+        new_schema[col["name"]] = col
+
+    return new_schema
+
+
+def prepare_dashboard_data(
+    lff: pl.LazyFrame,
+    dashboard_year=None,
+    dashboard_acheteur_id=None,
+    dashboard_acheteur_categorie=None,
+    dashboard_acheteur_departement_code=None,
+    dashboard_titulaire_id=None,
+    dashboard_titulaire_categorie=None,
+    dashboard_titulaire_departement_code=None,
+    dashboard_marche_type=None,
+    dashboard_marche_objet=None,
+    dashboard_marche_code_cpv=None,
+    dashboard_marche_considerations_sociales=None,
+    dashboard_marche_considerations_environnementales=None,
+    dashboard_marche_techniques=None,
+    dashboard_marche_innovant=None,
+    dashboard_marche_sous_traitance_declaree=None,
+    dashboard_montant_min=None,
+    dashboard_montant_max=None,
+) -> pl.LazyFrame:
+    if dashboard_year:
+        lff = lff.filter(pl.col("dateNotification").dt.year() == int(dashboard_year))
+    else:
+        lff = lff.filter(
+            pl.col("dateNotification") > (datetime.now() - timedelta(days=365))
+        )
+
+    if dashboard_acheteur_id:
+        lff = lff.filter(pl.col("acheteur_id").str.contains(dashboard_acheteur_id))
+    else:
+        if dashboard_acheteur_categorie:
+            lff = lff.filter(
+                pl.col("acheteur_categorie") == dashboard_acheteur_categorie
+            )
+        if dashboard_acheteur_departement_code:
+            lff = lff.filter(
+                pl.col("acheteur_departement_code").is_in(
+                    dashboard_acheteur_departement_code
+                )
+            )
+
+    if dashboard_titulaire_id:
+        lff = lff.filter(pl.col("titulaire_id").str.contains(dashboard_titulaire_id))
+    else:
+        if dashboard_titulaire_categorie:
+            lff = lff.filter(
+                pl.col("titulaire_categorie") == dashboard_titulaire_categorie
+            )
+        if dashboard_titulaire_departement_code:
+            lff = lff.filter(
+                pl.col("titulaire_departement_code").is_in(
+                    dashboard_titulaire_departement_code
+                )
+            )
+
+    if dashboard_marche_type:
+        lff = lff.filter(pl.col("type") == dashboard_marche_type)
+
+    if dashboard_marche_objet:
+        lff = lff.filter(pl.col("objet").str.contains(f"(?i){dashboard_marche_objet}"))
+
+    if dashboard_marche_code_cpv:
+        lff = lff.filter(pl.col("codeCPV").str.starts_with(dashboard_marche_code_cpv))
+
+    if dashboard_marche_innovant and dashboard_marche_innovant != "all":
+        lff = lff.filter(pl.col("marcheInnovant") == dashboard_marche_innovant)
+
+    if (
+        dashboard_marche_sous_traitance_declaree
+        and dashboard_marche_sous_traitance_declaree != "all"
+    ):
+        lff = lff.filter(
+            pl.col("sousTraitanceDeclaree") == dashboard_marche_sous_traitance_declaree
+        )
+
+    if dashboard_marche_techniques:
+        lff = lff.filter(
+            pl.col("techniques")
+            .str.split(", ")
+            .list.set_intersection(dashboard_marche_techniques)
+            .list.len()
+            > 0
+        )
+
+    if dashboard_marche_considerations_sociales:
+        lff = lff.filter(
+            pl.col("considerationsSociales")
+            .str.split(", ")
+            .list.set_intersection(dashboard_marche_considerations_sociales)
+            .list.len()
+            > 0
+        )
+
+    if dashboard_marche_considerations_environnementales:
+        lff = lff.filter(
+            pl.col("considerationsEnvironnementales")
+            .str.split(", ")
+            .list.set_intersection(dashboard_marche_considerations_environnementales)
+            .list.len()
+            > 0
+        )
+
+    if dashboard_montant_min is not None:
+        lff = lff.filter(pl.col("montant") >= dashboard_montant_min)
+
+    if dashboard_montant_max is not None:
+        lff = lff.filter(pl.col("montant") <= dashboard_montant_max)
+
+    return lff
+
+
+def build_org_frame(org_type: str) -> pl.DataFrame:
+    org_cols = [
+        c
+        for c in schema.names()
+        if c.startswith(f"{org_type}_")
+        and c not in (f"{org_type}_latitude", f"{org_type}_longitude")
+    ]
+    select_list = ", ".join(org_cols)
+    group_list = ", ".join(org_cols)
+    sql = f'SELECT {select_list}, COUNT(*) AS "Marchés" FROM decp GROUP BY {group_list}'
+    return get_cursor().execute(sql).pl()
+
+
+DF_ACHETEURS = build_org_frame("acheteur")
+DF_TITULAIRES = build_org_frame("titulaire")
+DEPARTEMENTS = get_departements()
+DEPARTEMENTS_GEOJSON = get_departements_geojson()
+DATA_SCHEMA = get_data_schema()

src/utils/frontend.py

@@ -0,0 +1,27 @@
+from src.utils.data import DATA_SCHEMA
+
+
+def get_button_properties(height):
+    if height > 65000:
+        download_disabled = True
+        download_text = "Téléchargement désactivé au-delà de 65 000 lignes"
+        download_title = " Ajoutez des filtres pour réduire le nombre de lignes, Excel ne supporte pas d'avoir plus de 65 000 URLs dans une même feuille de calcul."
+    elif height == 0:
+        download_disabled = True
+        download_text = "Pas de données à télécharger"
+        download_title = ""
+    else:
+        download_disabled = False
+        download_text = "Télécharger au format Excel"
+        download_title = "Télécharger les données telles qu'affichées au format Excel"
+    return download_disabled, download_text, download_title
+
+
+def get_enum_values_as_dict(column_name):
+    try:
+        options = {}
+        for value in DATA_SCHEMA[column_name]["enum"]:
+            options[value] = value
+        return options
+    except KeyError:
+        return {"not_found": "not found"}

src/utils/search.py

@@ -0,0 +1,84 @@
+import polars as pl
+from unidecode import unidecode
+
+from src.utils.table import add_links
+from src.utils.tracking import track_search
+
+
+def search_org(dff: pl.DataFrame, query: str, org_type: str) -> pl.DataFrame:
+    """
+    Search in either 'acheteur' or 'titulaire' DataFrame.
+
+    :param dff: Polars DataFrame with acheteur or titulaire columns
+    :param query: User search string
+    :param org_type: 'acheteur' or 'titulaire'
+    :return: Filtered DataFrame with 'matches' column
+    """
+    if not query.strip():
+        return dff.select(pl.lit(False).alias("matches"))
+
+    # Enregistrement des recherche dans Matomo
+    track_search(query, "home_page_search")
+
+    # Normalize query
+    normalized_query = unidecode(query.strip()).upper()
+    tokens = [" " + t.strip() for t in normalized_query.split() if t.strip()]
+
+    # Define columns based on entity type
+    cols = [
+        f"{org_type}_id",
+        f"{org_type}_nom",
+        f"{org_type}_departement_nom",
+        f"{org_type}_departement_code",
+        f"{org_type}_commune_nom",
+    ]
+
+    # Concatenate all fields into one string per row
+    org_str = pl.concat_str(pl.lit(" "), pl.col(cols), separator=" ").str.replace(
+        "-", " "
+    )
+
+    # For each token, create a boolean column: True if token is found
+    token_matches = []
+    for token in tokens:
+        token_match = org_str.str.contains(token).alias(f"token_{token}")
+        token_matches.append(token_match)
+
+    # Count how many tokens match per row
+    match_score = pl.sum_horizontal(token_matches).alias("match_score")
+
+    # For each token, create a boolean column: True if token is found
+    token_matches = []
+    for token in tokens:
+        token_match = org_str.str.contains(token).alias(f"token_{token}")
+        token_matches.append(token_match)
+
+    # Sélection des colonnes
+    if org_type == "acheteur":
+        dff = dff.select(cols + ["Marchés"])
+    if org_type == "titulaire":
+        dff = dff.select(cols + ["Marchés", "titulaire_typeIdentifiant"])
+
+    # Apply and filter
+    dff = (
+        dff.with_columns(token_matches + [match_score])
+        .filter(pl.col("match_score") == len(tokens))
+        .drop([f"token_{token}" for token in tokens])
+    )
+
+    # Format result
+    dff = add_links(dff)
+    dff = dff.with_columns(
+        pl.concat_str(
+            pl.col(f"{org_type}_departement_nom"),
+            pl.lit(" ("),
+            pl.col(f"{org_type}_departement_code"),
+            pl.lit(")"),
+        ).alias("Département")
+    )
+
+    dff = dff.select(f"{org_type}_id", f"{org_type}_nom", "Département", "Marchés")
+    dff = dff.group_by(f"{org_type}_id", f"{org_type}_nom", "Département").sum()
+    dff = dff.sort("Marchés", descending=True)
+
+    return dff

src/utils/seo.py

@@ -0,0 +1,52 @@
+from src.utils import DOMAIN_NAME
+from src.utils.data import get_annuaire_data
+
+
+def make_org_jsonld(org_id, org_type, org_name=None, type_org_id="SIRET") -> dict:
+    org_types = {"acheteur": "GovernmentOrganization", "titulaire": "Organization"}
+    address = None
+    if type_org_id.lower() == "siret" and len(org_id) == 14:
+        annuaire_data = get_annuaire_data(org_id)
+        annuaire_address = annuaire_data["matching_etablissements"][0]
+        code_postal = annuaire_address["code_postal"]
+        commune = annuaire_address["libelle_commune"]
+
+        address = (
+            {
+                "@type": "PostalAddress",
+                "streetAddress": annuaire_address.get("adresse", "")
+                .replace(code_postal, "")
+                .replace(commune, "")
+                .strip(),
+                "addressLocality": commune,
+                "postalCode": code_postal,
+                "addressCountry": "FR",
+            },
+        )
+
+    jsonld = {
+        "@type": org_types[org_type],
+        "name": org_name,
+        "url": f"https://decp.info/{org_type}s/{org_id}",
+        "sameAs": f"https://annuaire-entreprises.data.gouv.fr/etablissement/{org_id}",
+        "identifier": {
+            "@type": "PropertyValue",
+            "propertyID": type_org_id.lower(),
+            "value": org_id,
+        },
+    }
+
+    if address:
+        jsonld["address"] = address
+
+    return jsonld
+
+
+META_CONTENT = {
+    "image_url": f"https://{DOMAIN_NAME}/assets/decp.info.png",
+    "title": "decp.info - exploration des marchés publics français",
+    "description": (
+        "Explorez et analysez les données des marchés publics français avec cet outil libre et gratuit. "
+        "Pour une commande publique accessible à toutes et tous."
+    ),
+}

src/utils/table.py

@@ -1,26 +1,15 @@
-import json
-import logging
 import os
 import uuid
-from time import localtime, sleep
 
 import polars as pl
-import polars.selectors as cs
+from dash import no_update
-from httpx import get, post
+from polars import selectors as cs
-from polars.exceptions import ComputeError
-from unidecode import unidecode
 
-logging.basicConfig(
+from src.db import query_marches, schema
-    format="%(asctime)s %(levelname)-8s %(message)s",
+from src.utils import logger
-    level=logging.INFO,
+from src.utils.data import DATA_SCHEMA
-    datefmt="%Y-%m-%d %H:%M:%S",
+from src.utils.frontend import get_button_properties
-)
+from src.utils.tracking import track_search
-logger = logging.getLogger("decp.info")
-development = os.getenv("DEVELOPMENT", "False").lower() == "true"
-if development:
-    logger.setLevel(logging.DEBUG)
-
-logging.getLogger("httpx").setLevel("WARNING")
 
 
 def split_filter_part(filter_part):
@@ -32,14 +21,14 @@ def split_filter_part(filter_part):
         ["icontains", "contains"],
         # [" ", "contains"]
     ]
-    logger.debug("filter part", filter_part)
+    logger.debug("filter part " + filter_part)
     for operator_group in operators:
         if operator_group[0] in filter_part:
             name_part, value_part = filter_part.split(operator_group[0], 1)
             name_part = name_part.strip()
             value = value_part.strip()
             name = name_part[name_part.find("{") + 1 : name_part.rfind("}")]
-            logger.debug("=>", name, operator_group[1], value)
+            logger.debug("=> " + " ".join([name, operator_group[1], value]))
 
             return name, operator_group[1], value
 
@@ -56,10 +45,24 @@ def add_resource_link(dff: pl.DataFrame) -> pl.DataFrame:
     return dff
 
 
-def add_links(dff: pl.DataFrame, target: str = "_blank"):
+def add_links(dff: pl.DataFrame):
     for col in ["uid", "acheteur_nom", "titulaire_nom", "acheteur_id", "titulaire_id"]:
         if col in dff.columns:
             if col.startswith("titulaire_"):
+                detail_link = (
+                    '<a href = "/titulaires/'
+                    + pl.col("titulaire_id")
+                    + '">'
+                    + pl.col(col)
+                    + "</a>"
+                )
+                if col == "titulaire_nom":
+                    detail_link = (
+                        detail_link
+                        + ' <a href="/observatoire?titulaire_id='
+                        + pl.col("titulaire_id")
+                        + '" title="Voir dans l\'observatoire">📊</a>'
+                    )
                 dff = dff.with_columns(
                     pl.when(
                         pl.Expr.or_(
@@ -67,32 +70,32 @@ def add_links(dff: pl.DataFrame, target: str = "_blank"):
                             pl.col("titulaire_typeIdentifiant") == "SIRET",
                         )
                     )
-                    .then(
+                    .then(detail_link)
-                        '<a href = "/titulaires/'
-                        + pl.col("titulaire_id")
-                        + f'" target="{target}">'
-                        + pl.col(col)
-                        + "</a>"
-                    )
                     .otherwise(pl.col(col))
                     .alias(col)
                 )
             if col.startswith("acheteur_"):
-                dff = dff.with_columns(
+                detail_link = (
-                    (
                     '<a href = "/acheteurs/'
                     + pl.col("acheteur_id")
-                        + f'" target="{target}">'
+                    + '">'
                     + pl.col(col)
                     + "</a>"
-                    ).alias(col)
                 )
+                if col == "acheteur_nom":
+                    detail_link = (
+                        detail_link
+                        + ' <a href="/observatoire?acheteur_id='
+                        + pl.col("acheteur_id")
+                        + '" title="Voir dans l\'observatoire">📊</a>'
+                    )
+                dff = dff.with_columns(detail_link.alias(col))
             if col == "uid":
                 dff = dff.with_columns(
                     (
                         '<a href = "/marches/'
                         + pl.col("uid")
-                        + f'" target="{target}">'
+                        + '">'
                         + pl.col("uid")
                         + "</a>"
                     ).alias("uid")
@@ -203,88 +206,11 @@ def format_values(dff: pl.DataFrame) -> pl.DataFrame:
     return dff
 
 
-def get_annuaire_data(siret: str) -> dict:
-    url = f"https://recherche-entreprises.api.gouv.fr/search?q={siret}"
-    response = get(url)
-    return response.json()["results"][0]
-
-
-def get_decp_data() -> pl.DataFrame:
-    # Chargement du fichier parquet
-    # Le fichier est chargé en mémoire, ce qui est plus rapide qu'une base de données pour le moment.
-    # On utilise polars pour la rapidité et la facilité de manipulation des données.
-
-    try:
-        logger.info(
-            f"Lecture du fichier parquet ({os.getenv('DATA_FILE_PARQUET_PATH')})..."
-        )
-        lff: pl.LazyFrame = pl.scan_parquet(os.getenv("DATA_FILE_PARQUET_PATH"))
-    except ComputeError:
-        # Le fichier est probablement en cours de mise à jour
-        logger.info("Échec, nouvelle tentative dans 10s...")
-        sleep(10)
-        lff: pl.LazyFrame = pl.scan_parquet(os.getenv("DATA_FILE_PARQUET_PATH"))
-
-    # Tri des marchés par date de notification
-    lff = lff.sort(by=["dateNotification", "uid"], descending=True, nulls_last=True)
-
-    # Uniquement les données actuelles, pas les anciennes versions de marchés
-    lff = lff.filter(pl.col("donneesActuelles")).drop("donneesActuelles")
-
-    # Convertir les colonnes booléennes en chaînes de caractères
-    lff = booleans_to_strings(lff)
-
-    # Bizarrement je ne peux pas faire lff = lff.fill_null("") ici
-    # ça génère une erreur dans la page acheteur (acheteur_data.table) :
-    # AttributeError: partially initialized module 'pandas' has no attribute 'NaT' (most likely due to a circular import)
-
-    return lff.collect()
-
-
-def get_org_data(dff: pl.DataFrame, org_type: str) -> pl.DataFrame:
-    lff = dff.lazy()
-    lff = lff.select(
-        "uid",
-        cs.starts_with(org_type).exclude(
-            f"{org_type}_latitude", f"{org_type}_longitude"
-        ),
-    )
-    lff = lff.group_by(cs.starts_with(org_type)).len("Marchés")
-    return lff.collect()
-
-
-def get_statistics() -> dict:
-    return (
-        get(
-            "https://www.data.gouv.fr/api/1/datasets/r/0ccf4a75-f3aa-4b46-8b6a-18aeb63e36df",
-            follow_redirects=True,
-        )
-        .raise_for_status()
-        .json()
-    )
-
-
-def get_departements() -> dict:
-    with open("data/departements.json", "rb") as f:
-        data = json.load(f)
-        return data
-
-
-def get_departement_region(code_postal):
-    if code_postal > "97000":
-        code_departement = code_postal[:3]
-    else:
-        code_departement = code_postal[:2]
-    nom_departement = departements[code_departement]["departement"]
-    nom_region = departements[code_departement]["region"]
-    return code_departement, nom_departement, nom_region
-
-
 def filter_table_data(
     lff: pl.LazyFrame, filter_query: str, filter_source: str
 ) -> pl.LazyFrame:
     _schema = lff.collect_schema()
-    track_search(f"{filter_source}: {filter_query}")
+    track_search(filter_query, filter_source)
     filtering_expressions = filter_query.split(" && ")
     for filter_part in filtering_expressions:
         col_name, operator, filter_value = split_filter_part(filter_part)
@@ -367,7 +293,7 @@ def setup_table_columns(
     for column_id in dff.columns:
         if exclude and column_id in exclude:
             continue
-        column_object = data_schema.get(column_id)
+        column_object = DATA_SCHEMA.get(column_id)
         if column_object:
             column_name = column_object.get("title")
         else:
@@ -426,12 +352,11 @@ def get_default_hidden_columns(page):
             "codeCPV",
             "dureeRestanteMois",
         ]
+    elif page == "tableau":
+        displayed_columns = os.getenv("DISPLAYED_COLUMNS")
     else:
         displayed_columns = os.getenv("DISPLAYED_COLUMNS")
-        if displayed_columns is None:
+        logger.warning(f"Invalid page: {page}")
-            raise ValueError("DISPLAYED_COLUMNS n'est pas configuré")
-        else:
-            displayed_columns = displayed_columns.replace(" ", "").split(",")
 
     hidden_columns = []
 
@@ -443,134 +368,6 @@ def get_default_hidden_columns(page):
     return hidden_columns
 
 
-def get_data_schema() -> dict:
-    # Récupération du schéma des données tabulaires
-    path = os.getenv("DATA_SCHEMA_PATH")
-    if path.startswith("http"):
-        original_schema: dict = get(
-            os.getenv("DATA_SCHEMA_PATH"), follow_redirects=True
-        ).json()
-    elif os.path.exists(path):
-        with open(path) as f:
-            original_schema: dict = json.load(f)
-    else:
-        raise Exception(f"Chemin vers le schéma invalide: {path}")
-
-    new_schema = {}
-
-    for col in original_schema["fields"]:
-        new_schema[col["name"]] = col
-
-    return new_schema
-
-
-def track_search(query):
-    if (
-        len(query) >= 4
-        and os.getenv("DEVELOPMENT").lower != "true"
-        and os.getenv("MATOMO_DOMAIN")
-    ):
-        if os.getenv("DEVELOPMENT").lower() == "true":
-            url = "https://test.decp.info"
-        else:
-            url = "https://decp.info"
-        params = {
-            "idsite": os.getenv("MATOMO_ID_SITE"),
-            "url": url,
-            "rec": "1",
-            "action_name": "front_page_search",
-            "rand": uuid.uuid4().hex,
-            "apiv": "1",
-            "h": localtime().tm_hour,
-            "m": localtime().tm_min,
-            "s": localtime().tm_sec,
-            "search": query,
-            "token_auth": os.getenv("MATOMO_TOKEN"),
-        }
-        post(
-            url=f"https://{os.getenv('MATOMO_DOMAIN')}/matomo.php",
-            params=params,
-        ).raise_for_status()
-
-
-def search_org(dff: pl.DataFrame, query: str, org_type: str) -> pl.DataFrame:
-    """
-    Search in either 'acheteur' or 'titulaire' DataFrame.
-
-    :param dff: Polars DataFrame with acheteur or titulaire columns
-    :param query: User search string
-    :param org_type: 'acheteur' or 'titulaire'
-    :return: Filtered DataFrame with 'matches' column
-    """
-    if not query.strip():
-        return dff.select(pl.lit(False).alias("matches"))
-
-    # Enregistrement des recherche dans Matomo
-    track_search(query)
-
-    # Normalize query
-    normalized_query = unidecode(query.strip()).upper()
-    tokens = [" " + t.strip() for t in normalized_query.split() if t.strip()]
-
-    # Define columns based on entity type
-    cols = [
-        f"{org_type}_id",
-        f"{org_type}_nom",
-        f"{org_type}_departement_nom",
-        f"{org_type}_departement_code",
-        f"{org_type}_commune_nom",
-    ]
-
-    # Concatenate all fields into one string per row
-    org_str = pl.concat_str(pl.lit(" "), pl.col(cols), separator=" ").str.replace(
-        "-", " "
-    )
-
-    # For each token, create a boolean column: True if token is found
-    token_matches = []
-    for token in tokens:
-        token_match = org_str.str.contains(token).alias(f"token_{token}")
-        token_matches.append(token_match)
-
-    # Count how many tokens match per row
-    match_score = pl.sum_horizontal(token_matches).alias("match_score")
-
-    # For each token, create a boolean column: True if token is found
-    token_matches = []
-    for token in tokens:
-        token_match = org_str.str.contains(token).alias(f"token_{token}")
-        token_matches.append(token_match)
-
-    # Sélection des colonnes
-    if org_type == "acheteur":
-        dff = dff.select(cols + ["Marchés"])
-    if org_type == "titulaire":
-        dff = dff.select(cols + ["Marchés", "titulaire_typeIdentifiant"])
-
-    # Apply and filter
-    dff = (
-        dff.with_columns(token_matches + [match_score])
-        .filter(pl.col("match_score") == len(tokens))
-        .sort("Marchés", descending=True)
-        .drop([f"token_{token}" for token in tokens])
-    )
-
-    # Format result
-    dff = add_links(dff, target="")
-    dff = dff.with_columns(
-        pl.concat_str(
-            pl.col(f"{org_type}_departement_nom"),
-            pl.lit(" ("),
-            pl.col(f"{org_type}_departement_code"),
-            pl.lit(")"),
-        ).alias("Département")
-    )
-
-    dff = dff.select(f"{org_type}_id", f"{org_type}_nom", "Département", "Marchés")
-
-    return dff
-
-
 def prepare_table_data(
     data, data_timestamp, filter_query, page_current, page_size, sort_by, source_table
 ):
@@ -590,18 +387,23 @@ def prepare_table_data(
     if os.getenv("DEVELOPMENT").lower() == "true":
         logger.debug(" + + + + + + + + + + + + + + + + + + ")
 
+    trigger_cleanup = no_update
+
     # Récupération des données
     if isinstance(data, list):
         lff: pl.LazyFrame = pl.LazyFrame(data, strict=False, infer_schema_length=5000)
+    elif isinstance(data, pl.LazyFrame):
+        lff = data
     else:
-        lff: pl.LazyFrame = df.lazy()  # start from the original data
+        lff: pl.LazyFrame = query_marches().lazy()
 
     # Application des filtres
     if filter_query:
         lff = filter_table_data(lff, filter_query, source_table)
+        trigger_cleanup = no_update if source_table == "tableau" else str(uuid.uuid4())
 
     # Application des tris
-    if len(sort_by) > 0:
+    if sort_by and len(sort_by) > 0:
         lff = sort_table_data(lff, sort_by)
 
     # Matérialisation des filtres
@@ -624,7 +426,7 @@ def prepare_table_data(
     # Remplace les strings null par "", mais pas les numeric null
     dff = dff.fill_null("")
 
-    # Ajout des liens vers l'annuaire des entreprises
+    # Ajout des liens vers les pages de détails
     dff = add_links(dff)
 
     # Ajout des liens vers les fichiers Open Data
@@ -636,7 +438,7 @@ def prepare_table_data(
         dff = format_values(dff)
 
     # Récupération des colonnes et tooltip
-    columns, tooltip = setup_table_columns(dff)
+    table_columns, tooltip = setup_table_columns(dff)
 
     dicts = dff.to_dicts()
 
@@ -645,32 +447,17 @@ def prepare_table_data(
 
     return (
         dicts,
-        columns,
+        table_columns,
         tooltip,
         data_timestamp + 1,
         nb_rows,
         download_disabled,
         download_text,
         download_title,
+        trigger_cleanup,
     )
 
 
-def get_button_properties(height):
-    if height > 65000:
-        download_disabled = True
-        download_text = "Téléchargement désactivé au-delà de 65 000 lignes"
-        download_title = " Ajoutez des filtres pour réduire le nombre de lignes, Excel ne supporte pas d'avoir plus de 65 000 URLs dans une même feuille de calcul."
-    elif height == 0:
-        download_disabled = True
-        download_text = "Pas de données à télécharger"
-        download_title = ""
-    else:
-        download_disabled = False
-        download_text = "Télécharger au format Excel"
-        download_title = ""
-    return download_disabled, download_text, download_title
-
-
 def invert_columns(columns):
     """
     Renvoie les colonnes du schéma non spécifiées en paramètre. Utile pour passer d'une colonnes masquées à une liste de colonnes affichées, et vice versa.
@@ -685,80 +472,4 @@ def invert_columns(columns):
     return inverted_columns
 
 
-def make_org_jsonld(org_id, org_type, org_name=None, type_org_id="SIRET") -> dict:
+COLUMNS = schema.names()
-    org_types = {"acheteur": "GovernmentOrganization", "titulaire": "Organization"}
-    address = None
-    if type_org_id.lower() == "siret" and len(org_id) == 14:
-        annuaire_data = get_annuaire_data(org_id)
-        annuaire_address = annuaire_data["matching_etablissements"][0]
-        code_postal = annuaire_address["code_postal"]
-        commune = annuaire_address["libelle_commune"]
-
-        address = (
-            {
-                "@type": "PostalAddress",
-                "streetAddress": annuaire_address.get("adresse", "")
-                .replace(code_postal, "")
-                .replace(commune, "")
-                .strip(),
-                "addressLocality": commune,
-                "postalCode": code_postal,
-                "addressCountry": "FR",
-            },
-        )
-
-    jsonld = {
-        "@type": org_types[org_type],
-        "name": org_name,
-        "url": f"https://decp.info/{org_type}s/{org_id}",
-        "sameAs": f"https://annuaire-entreprises.data.gouv.fr/etablissement/{org_id}",
-        "identifier": {
-            "@type": "PropertyValue",
-            "propertyID": type_org_id.lower(),
-            "value": org_id,
-        },
-    }
-
-    if address:
-        jsonld["address"] = address
-
-    return jsonld
-
-
-df: pl.DataFrame = get_decp_data()
-schema = df.collect_schema()
-
-df_acheteurs = get_org_data(df, "acheteur")
-df_titulaires = get_org_data(df, "titulaire")
-df_acheteurs_departement: pl.DataFrame = (
-    df_acheteurs.select(["acheteur_id", "acheteur_nom", "acheteur_departement_code"])
-    .unique()
-    .sort("acheteur_nom")
-)
-df_titulaires_departement: pl.DataFrame = (
-    df_titulaires.select(
-        ["titulaire_id", "titulaire_nom", "titulaire_departement_code"]
-    )
-    .unique()
-    .sort("titulaire_nom")
-)
-df_acheteurs_marches: pl.DataFrame = (
-    df.select("uid", "objet", "acheteur_id").unique().sort("acheteur_id")
-)
-df_titulaires_marches: pl.DataFrame = (
-    df.select("uid", "objet", "titulaire_id").unique().sort("titulaire_id")
-)
-
-departements = get_departements()
-domain_name = (
-    "test.decp.info" if os.getenv("DEVELOPMENT").lower() == "true" else "decp.info"
-)
-meta_content = {
-    "image_url": f"https://{domain_name}/assets/decp.info.png",
-    "title": "decp.info - exploration des marchés publics français",
-    "description": (
-        "Explorez et analysez les données des marchés publics français avec cet outil libre et gratuit. "
-        "Pour une commande publique accessible à toutes et tous."
-    ),
-}
-data_schema = get_data_schema()

src/utils/tracking.py

@@ -0,0 +1,30 @@
+import os
+import uuid
+from time import localtime
+
+from httpx import post
+
+from src.utils import DEVELOPMENT
+
+
+def track_search(query, category):
+    if len(query) >= 4 and not DEVELOPMENT and os.getenv("MATOMO_DOMAIN"):
+        url = "https://decp.info"
+        params = {
+            "idsite": os.getenv("MATOMO_ID_SITE"),
+            "url": url,
+            "rec": "1",
+            "action_name": "search" if category == "home_page_search" else "filter",
+            "search_cat": category,
+            "rand": uuid.uuid4().hex,
+            "apiv": "1",
+            "h": localtime().tm_hour,
+            "m": localtime().tm_min,
+            "s": localtime().tm_sec,
+            "search": query,
+            "token_auth": os.getenv("MATOMO_TOKEN"),
+        }
+        post(
+            url=f"https://{os.getenv('MATOMO_DOMAIN')}/matomo.php",
+            params=params,
+        ).raise_for_status()

tests/conftest.py

@@ -0,0 +1,72 @@
+import datetime
+import os
+from pathlib import Path
+
+import polars as pl
+import pytest
+from selenium.webdriver.chrome.options import Options
+
+
+@pytest.fixture(scope="session", autouse=True)
+def test_data():
+    data = [
+        {
+            "uid": "1",
+            "id": "1",
+            "acheteur_nom": "ACHETEUR 1",
+            "acheteur_id": "123",
+            "titulaire_nom": "TITULAIRE 1",
+            "titulaire_id": "345",
+            "montant": 10,
+            "dateNotification": datetime.date(2025, 1, 1),
+            "codeCPV": "71600000",
+            "donneesActuelles": True,
+            "acheteur_departement_code": "75",
+            "acheteur_departement_nom": "Paris",
+            "acheteur_commune_nom": "Paris",
+            "titulaire_departement_code": "35",
+            "titulaire_departement_nom": "Ille-et-Vilaine",
+            "titulaire_commune_nom": "Rennes",
+            "titulaire_distance": 10,
+            "titulaire_typeIdentifiant": "SIRET",
+            "objet": "Objet test",
+            "dureeRestanteMois": 12,
+            "lieuExecution_code": "75001",
+            "sourceFile": "test.xml",
+            "sourceDataset": "test_dataset",
+            "datePublicationDonnees": datetime.date(2025, 1, 1),
+            "considerationsSociales": "",
+            "considerationsEnvironnementales": "",
+            "type": "Marché",
+            "acheteur_categorie": "Collectivité",
+            "titulaire_categorie": "PME",
+        }
+    ]
+    parquet_path = Path(os.path.abspath("tests/test.parquet"))
+    db_path = parquet_path.parent / "decp.duckdb"
+    print(f"Writing test data to: {parquet_path}")
+
+    pl.DataFrame(data).write_parquet(parquet_path)
+
+    # Remove any stale DuckDB from a previous run so src.db rebuilds from
+    # the freshly-written parquet at import time.
+    for artifact in (db_path, db_path.with_suffix(".duckdb.tmp")):
+        if artifact.exists():
+            artifact.unlink()
+
+    yield str(parquet_path)
+
+
+def pytest_setup_options():
+    options = Options()
+    options.add_argument("--window-size=1200,1200 ")
+    options.add_experimental_option(
+        "prefs",
+        {
+            "download.default_directory": "/home/colin/git/decp.info",
+            "download.prompt_for_download": False,
+            "download.directory_upgrade": True,
+            "safebrowsing.enabled": True,
+        },
+    )
+    return options

tests/test_db.py

@@ -0,0 +1,267 @@
+import datetime
+import os
+import time
+
+import polars as pl
+import pytest
+
+from src.db import should_rebuild
+
+
+@pytest.fixture
+def parquet_and_db(tmp_path, monkeypatch):
+    parquet = tmp_path / "source.parquet"
+    db = tmp_path / "decp.duckdb"
+    parquet.write_bytes(b"fake parquet content")
+    monkeypatch.delenv("REBUILD_DUCKDB", raising=False)
+    monkeypatch.delenv("DEVELOPMENT", raising=False)
+    return parquet, db
+
+
+def test_should_rebuild_when_db_missing(parquet_and_db):
+    parquet, db = parquet_and_db
+    assert should_rebuild(db, parquet) is True
+
+
+def test_should_rebuild_prod_when_parquet_newer(parquet_and_db, monkeypatch):
+    parquet, db = parquet_and_db
+    db.write_bytes(b"x")
+    parquet.touch()
+    now = time.time()
+    os.utime(db, (now, now))
+    os.utime(parquet, (now + 10, now + 10))
+    monkeypatch.setenv("DEVELOPMENT", "false")
+    assert should_rebuild(db, parquet) is True
+
+
+def test_should_not_rebuild_prod_when_parquet_older(parquet_and_db, monkeypatch):
+    parquet, db = parquet_and_db
+    parquet.touch()
+    db.write_bytes(b"x")
+    now = time.time()
+    os.utime(parquet, (now, now))
+    os.utime(db, (now + 10, now + 10))
+    monkeypatch.setenv("DEVELOPMENT", "false")
+    assert should_rebuild(db, parquet) is False
+
+
+def test_should_not_rebuild_dev_even_when_parquet_newer(parquet_and_db, monkeypatch):
+    parquet, db = parquet_and_db
+    db.write_bytes(b"x")
+    parquet.touch()
+    now = time.time()
+    os.utime(db, (now, now))
+    os.utime(parquet, (now + 10, now + 10))
+    monkeypatch.setenv("DEVELOPMENT", "true")
+    monkeypatch.delenv("REBUILD_DUCKDB", raising=False)
+    assert should_rebuild(db, parquet) is False
+
+
+def test_should_rebuild_dev_when_rebuild_forced(parquet_and_db, monkeypatch):
+    parquet, db = parquet_and_db
+    db.write_bytes(b"x")
+    parquet.touch()
+    now = time.time()
+    os.utime(db, (now, now))
+    os.utime(parquet, (now + 10, now + 10))
+    monkeypatch.setenv("DEVELOPMENT", "true")
+    monkeypatch.setenv("REBUILD_DUCKDB", "true")
+    assert should_rebuild(db, parquet) is True
+
+
+@pytest.fixture
+def built_db(tmp_path, monkeypatch):
+    """Build a DuckDB from a small Polars frame written as parquet."""
+    parquet_path = tmp_path / "source.parquet"
+    db_path = tmp_path / "decp.duckdb"
+
+    data = pl.DataFrame(
+        [
+            {
+                "uid": "1",
+                "id": "1",
+                "objet": "Travaux",
+                "acheteur_id": "123",
+                "acheteur_nom": "ACHETEUR 1",
+                "acheteur_departement_code": "75",
+                "acheteur_departement_nom": "Paris",
+                "acheteur_commune_nom": "Paris",
+                "titulaire_commune_nom": "Paris",
+                "titulaire_departement_nom": "Paris",
+                "titulaire_id": "345",
+                "titulaire_nom": "TITULAIRE 1",
+                "titulaire_departement_code": "35",
+                "titulaire_typeIdentifiant": "SIRET",
+                "montant": 1000.0,
+                "dateNotification": datetime.date(2025, 1, 1),
+                "donneesActuelles": True,
+                "marcheInnovant": True,
+            },
+            {
+                "uid": "2",
+                "id": "2",
+                "objet": "Études",
+                "acheteur_id": "123",
+                "acheteur_nom": "ACHETEUR 1",
+                "acheteur_departement_code": "75",
+                "acheteur_departement_nom": "Paris",
+                "acheteur_commune_nom": "Paris",
+                "titulaire_commune_nom": "Paris",
+                "titulaire_departement_nom": "Paris",
+                "titulaire_id": "567",
+                "titulaire_nom": None,
+                "titulaire_departement_code": "75",
+                "titulaire_typeIdentifiant": "SIRET",
+                "montant": 500.0,
+                "dateNotification": datetime.date(2024, 6, 1),
+                "donneesActuelles": True,
+                "marcheInnovant": False,
+            },
+            {
+                "uid": "3",
+                "id": "3",
+                "objet": "Ancien",
+                "acheteur_id": "A2",
+                "acheteur_nom": None,
+                "acheteur_departement_code": "13",
+                "acheteur_departement_nom": "Paris",
+                "acheteur_commune_nom": "Paris",
+                "titulaire_commune_nom": "Paris",
+                "titulaire_departement_nom": "Paris",
+                "titulaire_id": "T3",
+                "titulaire_nom": "Autre",
+                "titulaire_departement_code": "13",
+                "titulaire_typeIdentifiant": "SIRET",
+                "montant": 100.0,
+                "dateNotification": datetime.date(2023, 1, 1),
+                "donneesActuelles": False,  # must be filtered out
+                "marcheInnovant": False,
+            },
+        ]
+    )
+    data.write_parquet(parquet_path)
+    monkeypatch.setenv("DATA_FILE_PARQUET_PATH", str(parquet_path))
+
+    from src.db import build_database
+
+    build_database(db_path, parquet_path)
+    return db_path
+
+
+def test_build_filters_donnees_actuelles(built_db):
+    import duckdb
+
+    with duckdb.connect(str(built_db), read_only=True) as c:
+        rows = c.execute("SELECT uid FROM decp ORDER BY uid").fetchall()
+    assert [r[0] for r in rows] == ["1", "2"]
+
+
+def test_build_converts_booleans_to_oui_non(built_db):
+    import duckdb
+
+    with duckdb.connect(str(built_db), read_only=True) as c:
+        values = c.execute("SELECT marcheInnovant FROM decp ORDER BY uid").fetchall()
+    assert [v[0] for v in values] == ["oui", "non"]
+
+
+def test_build_replaces_null_org_names(built_db):
+    import duckdb
+
+    with duckdb.connect(str(built_db), read_only=True) as c:
+        titulaire_2 = c.execute(
+            "SELECT titulaire_nom FROM decp WHERE uid = '2'"
+        ).fetchone()
+    assert titulaire_2[0] == "[Identifiant non reconnu dans la base INSEE]"
+
+
+def test_build_creates_derived_tables(built_db):
+    import duckdb
+
+    with duckdb.connect(str(built_db), read_only=True) as c:
+        tables = {r[0] for r in c.execute("SHOW TABLES").fetchall()}
+    assert {
+        "decp",
+        "acheteurs_marches",
+        "titulaires_marches",
+        "acheteurs_departement",
+        "titulaires_departement",
+    } <= tables
+
+
+def test_query_marches_returns_polars_frame(built_db, monkeypatch):
+    monkeypatch.setenv(
+        "DATA_FILE_PARQUET_PATH", str(built_db.parent / "source.parquet")
+    )
+    # Force src.db to load pointing at this test DB.
+    import importlib
+
+    import src.db
+
+    importlib.reload(src.db)
+    from src.db import query_marches
+
+    frame = query_marches("acheteur_id = ?", ("123",))
+    assert isinstance(frame, pl.DataFrame)
+    assert frame.height == 2
+    assert set(frame["uid"].to_list()) == {"1", "2"}
+
+
+def test_concurrent_build_serialized(tmp_path):
+    """Multiple threads calling _ensure_database must serialize via flock.
+
+    Only one should actually build; others wait, see the fresh DB, and skip.
+    No tmp file should leak. No exceptions should occur.
+    """
+    import fcntl
+    import threading
+
+    import src.db as db
+
+    # Set up source parquet
+    parquet_path = tmp_path / "src.parquet"
+    df = pl.DataFrame(
+        {
+            "uid": ["A"],
+            "donneesActuelles": [True],
+            "dateNotification": ["2024-01-01"],
+            "objet": ["Test"],
+            "acheteur_id": ["a1"],
+            "acheteur_nom": ["A1"],
+            "titulaire_id": ["t1"],
+            "titulaire_nom": ["T1"],
+            "acheteur_departement_code": ["75"],
+            "titulaire_departement_code": ["75"],
+            "montant": [1000.0],
+            "dureeMois": [12],
+        }
+    )
+    df.write_parquet(parquet_path)
+
+    db_path = tmp_path / "decp.duckdb"
+    lock_path = db_path.with_suffix(".duckdb.lock")
+    tmp_path_artifact = db_path.with_suffix(".duckdb.tmp")
+
+    errors: list[BaseException] = []
+
+    def worker():
+        try:
+            # Mirror the locking logic in _ensure_database
+            with open(lock_path, "w") as lf:
+                fcntl.flock(lf.fileno(), fcntl.LOCK_EX)
+                try:
+                    if db.should_rebuild(db_path, parquet_path):
+                        db.build_database(db_path, parquet_path)
+                finally:
+                    fcntl.flock(lf.fileno(), fcntl.LOCK_UN)
+        except BaseException as exc:
+            errors.append(exc)
+
+    threads = [threading.Thread(target=worker) for _ in range(3)]
+    for t in threads:
+        t.start()
+    for t in threads:
+        t.join()
+
+    assert errors == []
+    assert db_path.exists()
+    assert not tmp_path_artifact.exists()

tests/test_main.py

@@ -0,0 +1,366 @@
+import polars as pl
+from dash.testing.composite import DashComposite
+from selenium.webdriver import Keys
+from selenium.webdriver.common.by import By
+from selenium.webdriver.remote.webelement import WebElement
+
+
+def test_001_logo_and_search(dash_duo: DashComposite):
+    from src.app import app
+
+    dash_duo.start_server(app)
+    dash_duo.wait_for_text_to_equal(".logo > h1", "decp.info", timeout=4)
+    assert dash_duo.find_element(".logo > h1").text == "decp.info"
+
+    for org_type in ["acheteur", "titulaire"]:
+        name = f"{org_type.upper()} 1"
+        search_bar: WebElement = dash_duo.find_element("#search")
+
+        dash_duo.clear_input(search_bar)
+
+        search_bar.send_keys(name)
+        search_bar.send_keys(Keys.ENTER)
+
+        dash_duo.wait_for_element(f"#results_{org_type}_datatable", timeout=2)
+        result_table: WebElement = dash_duo.find_element(
+            f"#results_{org_type}_datatable tbody"
+        )
+
+        assert len(result_table.find_elements(by=By.TAG_NAME, value="tr")) == 2, (
+            "The search should return only one result"
+        )  # header row + 1 result
+        assert result_table.find_element(
+            by=By.CSS_SELECTOR, value=f'td[data-dash-column="{org_type}_nom"]'
+        ).text.startswith(name), (
+            f"The search result should have the right {org_type} name"
+        )
+
+
+def test_002_filter_persistence(dash_duo: DashComposite):
+    from src.app import app
+
+    dash_duo.start_server(app)
+    dash_duo.wait_for_text_to_equal(".logo > h1", "decp.info", timeout=4)
+
+    def open_page_and_check_filter_input():
+        dash_duo.wait_for_page(f"{dash_duo.server_url}/{page}")
+        filter_input_selector = (
+            '.marches_table th[data-dash-column="uid"] input[type="text"]'
+        )
+        dash_duo.wait_for_element(filter_input_selector, timeout=2)
+        _filter_input: WebElement = dash_duo.find_element(filter_input_selector)
+        return _filter_input
+
+    for page in ["tableau", "acheteurs/123", "titulaires/345"]:
+        filter_input = open_page_and_check_filter_input()
+        filter_input.send_keys("11")  # a UID that doesn't exist
+        filter_input.send_keys(Keys.ENTER)
+        filter_input = open_page_and_check_filter_input()
+        assert filter_input.get_attribute("value") == "11"
+
+
+def test_003_tableau_download(dash_duo: DashComposite):
+    from pages.acheteur import download_acheteur_data
+    from pages.tableau import download_data
+    from pages.titulaire import download_titulaire_data
+    from src.app import app
+
+    # Juste pour instancier l'app
+    print(app.server.name)
+
+    dicts = pl.read_parquet("tests/test.parquet").to_dicts()
+
+    outputs = [
+        download_data(1, "", [], None),
+        download_acheteur_data(1, dicts, "123", "2025"),
+        download_titulaire_data(1, dicts, "345", "2025"),
+    ]
+    for output in outputs:
+        assert isinstance(output, dict)
+        for f in ["content", "filename", "type", "base64"]:
+            assert f in output
+        assert isinstance(output["content"], str) and len(output["content"]) > 100
+        assert isinstance(output["filename"], str) and output["filename"].startswith(
+            "decp_"
+        )
+        assert output["type"] is None
+        assert output["base64"] is True
+
+
+def test_004_add_links_observatoire_acheteur():
+    import polars as pl
+
+    from src.utils.table import add_links
+
+    dff = pl.DataFrame(
+        {
+            "acheteur_id": ["123"],
+            "acheteur_nom": ["ACHETEUR 1"],
+        }
+    )
+    result = add_links(dff)
+    nom_value = result["acheteur_nom"][0]
+    id_value = result["acheteur_id"][0]
+
+    # acheteur_nom should contain detail link + observatoire link
+    assert "/acheteurs/123" in nom_value
+    assert "ACHETEUR 1" in nom_value
+    assert "/observatoire?acheteur_id=123" in nom_value
+    assert "📊" in nom_value
+
+    # acheteur_id should NOT contain observatoire link
+    assert "/observatoire" not in id_value
+
+
+def test_005_add_links_observatoire_titulaire():
+    import polars as pl
+
+    from src.utils.table import add_links
+
+    dff = pl.DataFrame(
+        {
+            "titulaire_id": ["345"],
+            "titulaire_nom": ["TITULAIRE 1"],
+            "titulaire_typeIdentifiant": ["SIRET"],
+        }
+    )
+    result = add_links(dff)
+    nom_value = result["titulaire_nom"][0]
+    id_value = result["titulaire_id"][0]
+
+    # titulaire_nom should contain detail link + observatoire link
+    assert "/titulaires/345" in nom_value
+    assert "TITULAIRE 1" in nom_value
+    assert "/observatoire?titulaire_id=345" in nom_value
+    assert "📊" in nom_value
+
+    # titulaire_id should NOT contain observatoire link
+    assert "/observatoire" not in id_value
+
+
+def test_006_observatoire_url_to_input(dash_duo: DashComposite):
+    from src.app import app
+
+    dash_duo.start_server(app)
+    dash_duo.wait_for_text_to_equal(".logo > h1", "decp.info", timeout=4)
+
+    # Navigate to observatoire with acheteur_id query param
+    dash_duo.wait_for_page(f"{dash_duo.server_url}/observatoire?acheteur_id=123")
+    dash_duo.wait_for_element("#dashboard_acheteur_id", timeout=4)
+
+    import time
+
+    time.sleep(1)  # Allow callback chain to complete
+
+    acheteur_input = dash_duo.find_element("#dashboard_acheteur_id")
+    assert acheteur_input.get_attribute("value") == "123", (
+        "acheteur_id input should be populated from URL param"
+    )
+
+
+def test_007_observatoire_share_url(dash_duo: DashComposite):
+    from src.app import app
+
+    dash_duo.start_server(app)
+    dash_duo.wait_for_text_to_equal(".logo > h1", "decp.info", timeout=4)
+
+    # Navigate to observatoire with acheteur_id query param
+    dash_duo.wait_for_page(f"{dash_duo.server_url}/observatoire?acheteur_id=123")
+    dash_duo.wait_for_element("#observatoire-share-url", timeout=4)
+
+    import time
+
+    time.sleep(1)  # Allow callback chain to complete
+
+    share_url_input = dash_duo.find_element("#observatoire-share-url")
+    share_url_value = share_url_input.get_attribute("value")
+
+    assert "acheteur_id=123" in share_url_value, (
+        f"Share URL should contain acheteur_id param, got: {share_url_value}"
+    )
+
+
+def test_008_search_to_observatoire(dash_duo: DashComposite):
+    from src.app import app
+
+    dash_duo.start_server(app)
+    dash_duo.wait_for_text_to_equal(".logo > h1", "decp.info", timeout=4)
+
+    # Search for an acheteur
+    search_bar = dash_duo.find_element("#search")
+    search_bar.send_keys("ACHETEUR 1")
+    search_bar.send_keys(Keys.ENTER)
+
+    dash_duo.wait_for_element("#results_acheteur_datatable", timeout=2)
+
+    # Find the observatoire link in acheteur_nom column
+    observatoire_link = dash_duo.find_element(
+        '#results_acheteur_datatable td[data-dash-column="acheteur_nom"] a[href*="observatoire"]'
+    )
+    assert "📊" in observatoire_link.text
+
+    # Click the observatoire link
+    observatoire_link.click()
+
+    # Wait for observatoire page to load
+    dash_duo.wait_for_element("#dashboard_acheteur_id", timeout=4)
+
+    import time
+
+    time.sleep(1)  # Allow callback chain to complete
+
+    acheteur_input = dash_duo.find_element("#dashboard_acheteur_id")
+    assert acheteur_input.get_attribute("value") == "123", (
+        "acheteur_id input should be populated after navigating from search"
+    )
+
+
+def test_010_observatoire_montant_filter():
+    import datetime
+
+    from src.utils.data import prepare_dashboard_data
+
+    data = pl.DataFrame(
+        {
+            "uid": ["1", "2", "3"],
+            "montant": [100.0, 500.0, 1000.0],
+            "dateNotification": [datetime.date(2025, 1, 1)] * 3,
+        }
+    )
+
+    def apply(min_val=None, max_val=None):
+        return prepare_dashboard_data(
+            data.lazy(),
+            dashboard_year="2025",
+            dashboard_acheteur_id=None,
+            dashboard_acheteur_categorie=None,
+            dashboard_acheteur_departement_code=None,
+            dashboard_titulaire_id=None,
+            dashboard_titulaire_categorie=None,
+            dashboard_titulaire_departement_code=None,
+            dashboard_marche_type=None,
+            dashboard_marche_objet=None,
+            dashboard_marche_code_cpv=None,
+            dashboard_marche_considerations_sociales=None,
+            dashboard_marche_considerations_environnementales=None,
+            dashboard_marche_techniques=None,
+            dashboard_marche_innovant=None,
+            dashboard_marche_sous_traitance_declaree=None,
+            dashboard_montant_min=min_val,
+            dashboard_montant_max=max_val,
+        ).collect()
+
+    assert apply().height == 3
+    assert apply(min_val=400).height == 2  # 500, 1000
+    assert apply(max_val=500).height == 2  # 100, 500
+    assert apply(min_val=200, max_val=600).height == 1  # 500 only
+
+
+def test_009_observatoire_filter_persistence(dash_duo: DashComposite):
+    import time
+
+    from src.app import app
+
+    dash_duo.start_server(app)
+    dash_duo.wait_for_text_to_equal(".logo > h1", "decp.info", timeout=4)
+
+    # Clear localStorage to start from a clean state
+    dash_duo.driver.execute_script("localStorage.clear()")
+
+    # Navigate to observatoire without URL params
+    dash_duo.wait_for_page(f"{dash_duo.server_url}/observatoire")
+    dash_duo.wait_for_element("#dashboard_acheteur_id", timeout=4)
+
+    # Set the acheteur_id text input; press Enter to trigger the debounced save callback
+    acheteur_input = dash_duo.find_element("#dashboard_acheteur_id")
+    dash_duo.clear_input(acheteur_input)
+    acheteur_input.send_keys("123")
+    acheteur_input.send_keys(Keys.ENTER)
+
+    time.sleep(0.3)  # allow the save callback to write to localStorage
+
+    # Navigate away
+    dash_duo.wait_for_page(f"{dash_duo.server_url}/")
+
+    # Navigate back without URL params
+    dash_duo.wait_for_page(f"{dash_duo.server_url}/observatoire")
+    dash_duo.wait_for_element("#dashboard_acheteur_id", timeout=4)
+    time.sleep(0.5)  # allow restore callback chain to complete
+
+    acheteur_input = dash_duo.find_element("#dashboard_acheteur_id")
+    assert acheteur_input.get_attribute("value") == "123", (
+        "acheteur_id should be restored from localStorage after navigating back"
+    )
+
+    # Also verify URL params still override localStorage
+    dash_duo.wait_for_page(f"{dash_duo.server_url}/observatoire?acheteur_id=123")
+    dash_duo.wait_for_element("#dashboard_acheteur_id", timeout=4)
+    time.sleep(0.5)
+
+    acheteur_input = dash_duo.find_element("#dashboard_acheteur_id")
+    assert acheteur_input.get_attribute("value") == "123", (
+        "URL param acheteur_id should override the value stored in localStorage"
+    )
+
+
+def test_011_observatoire_multi_param_url(dash_duo: DashComposite):
+    import time
+
+    from src.app import app
+
+    dash_duo.start_server(app)
+    dash_duo.wait_for_text_to_equal(".logo > h1", "decp.info", timeout=4)
+
+    # Navigate with multiple filter params
+    dash_duo.wait_for_page(
+        f"{dash_duo.server_url}/observatoire?annee=2024&acheteur_id=12345678901234&montant_min=10000"
+    )
+    dash_duo.wait_for_element("#dashboard_acheteur_id", timeout=4)
+
+    time.sleep(1)  # Allow callback chain to complete
+
+    # Verify acheteur_id input
+    acheteur_input = dash_duo.find_element("#dashboard_acheteur_id")
+    assert acheteur_input.get_attribute("value") == "12345678901234", (
+        "acheteur_id input should be populated from URL param"
+    )
+
+    # Verify montant_min input
+    montant_input = dash_duo.find_element("#dashboard_montant_min")
+    montant_value = montant_input.get_attribute("value")
+    assert montant_value in ("10000", "10000.0"), (
+        f"montant_min input should be populated from URL param, got: {montant_value}"
+    )
+
+
+def test_get_distance_histogram_returns_graph():
+    import polars as pl
+    from dash import dcc
+
+    from src.figures import get_distance_histogram
+
+    lff = pl.LazyFrame({"titulaire_distance": [1, 10, 100, 500, 1000]})
+    result = get_distance_histogram(lff)
+    assert isinstance(result, dcc.Graph)
+
+
+def test_get_distance_histogram_handles_nulls():
+    import polars as pl
+    from dash import dcc
+
+    from src.figures import get_distance_histogram
+
+    lff = pl.LazyFrame({"titulaire_distance": [None, None, 50]})
+    result = get_distance_histogram(lff)
+    assert isinstance(result, dcc.Graph)
+
+
+def test_get_distance_histogram_all_nulls():
+    import polars as pl
+    from dash import dcc
+
+    from src.figures import get_distance_histogram
+
+    lff = pl.LazyFrame({"titulaire_distance": pl.Series([], dtype=pl.Int64)})
+    result = get_distance_histogram(lff)
+    assert isinstance(result, dcc.Graph)