feat(mes-aides): integration aides permis velo #296

hlecuyer · 2024-09-17T14:25:54Z

No description provided.

vmttn

1ère passe rapide :)

pipeline/defaults.env

pipeline/dbt/models/staging/decoupage_administratif/_decoupage_administratif__models.yml

pipeline/dbt/models/staging/sources/mes_aides/stg_mes_aides__permis_velo.sql

pipeline/dbt/models/intermediate/sources/mes_aides/garage/_mes_aides_garage__models.yml

...bt/models/intermediate/sources/mes_aides/permis_velo/int_mes_aides__services_permis_velo.sql

pipeline/dags/dag_utils/sources/__init__.py

...bt/models/intermediate/sources/mes_aides/permis_velo/int_mes_aides__services_permis_velo.sql

pipeline/dbt/models/staging/sources/mes_aides/stg_mes_aides__permis_velo.sql

.../models/intermediate/sources/mes_aides/permis_velo/int_mes_aides__structures_permis_velo.sql

vmttn

pas grand chose à dire, c'est propre ✨ juste quelques renommages + quelques tests à ajouter pour bien dormir 😴

vmttn · 2024-10-17T13:19:01Z

pipeline/dbt/models/staging/sources/mes_aides/stg_mes_aides__permis_velo.sql

+        STRING_TO_ARRAY(data #>> '{fields,Liaisons Besoins}', ', ')            AS "liaisons_besoins_mes_aides",
+        data #>> '{fields,Slug Thématiques}'                                   AS "slug_thematique_mes_aides",


Suggested change

STRING_TO_ARRAY(data #>> '{fields,Liaisons Besoins}', ', ') AS "liaisons_besoins_mes_aides",

data #>> '{fields,Slug Thématiques}' AS "slug_thematique_mes_aides",

STRING_TO_ARRAY(data #>> '{fields,Liaisons Besoins}', ', ') AS "liaisons_besoins",

data #>> '{fields,Slug Thématiques}' AS "slug_thematiques",

vmttn · 2024-10-17T13:19:31Z

pipeline/dbt/models/staging/sources/mes_aides/stg_mes_aides__permis_velo.sql

+        data #>> '{fields,SIRET}'                                              AS "siret_structure",
+        data #>> '{fields,Slug Organisme}'                                     AS "slug_organisme_structure",


Suggested change

data #>> '{fields,SIRET}' AS "siret_structure",

data #>> '{fields,Slug Organisme}' AS "slug_organisme_structure",

data #>> '{fields,SIRET}' AS "siret",

data #>> '{fields,Slug Organisme}' AS "slug_organisme",

vmttn · 2024-10-17T13:19:51Z

pipeline/dbt/models/staging/sources/mes_aides/stg_mes_aides__permis_velo.sql

+        data #>> '{fields,Nom Organisme}'                                      AS "nom_organisme_structure",
+        data #>> '{fields,Organisme Type}'                                     AS "typologie_structure",


Suggested change

data #>> '{fields,Nom Organisme}' AS "nom_organisme_structure",

data #>> '{fields,Organisme Type}' AS "typologie_structure",

data #>> '{fields,Nom Organisme}' AS "nom_organisme",

data #>> '{fields,Organisme Type}' AS "organisme_type",

vmttn · 2024-10-17T13:21:22Z

pipeline/dbt/models/staging/sources/mes_aides/stg_mes_aides__permis_velo.sql

+        _di_source_id                                                          AS "_di_source_id",
+        data #>> '{fields,ID}'                                                 AS "id",
+        TRIM(SUBSTRING(data #>> '{fields,Liaisons Villes}' FROM '^(.*) \('))   AS "liaisons_ville_nom",
+        TRIM(SUBSTRING(data #>> '{fields,Liaisons Villes}' FROM '\((\d+)\)$')) AS "liaisons_code_postal",


Suggested change

TRIM(SUBSTRING(data #>> '{fields,Liaisons Villes}' FROM '$(\d+)$$')) AS "liaisons_code_postal",

TRIM(SUBSTRING(data #>> '{fields,Liaisons Villes}' FROM '$(\d+)$$')) AS "liaisons_ville_code_postal",

vmttn · 2024-10-17T13:27:20Z

pipeline/dbt/models/staging/sources/mes_aides/stg_mes_aides__permis_velo.sql

reprends au maximum le nom des colonnes d'origine, ça facilite grandement la compréhension du mapping

vmttn · 2024-10-17T13:42:34Z

...t/models/intermediate/sources/mes_aides/permis_velo/int_mes_aides__permis_velo__services.sql

+            WHEN LENGTH(permis_velo.description) > 280 THEN SUBSTRING(permis_velo.description FROM 1 FOR 277) || '...'
+            ELSE permis_velo.description
+        END                                                                                                                                     AS "presentation_resume",
+        permis_velo.description || COALESCE(E'\n\n' || permis_velo.bon_a_savoir, '') || COALESCE(E'\n\n' || permis_velo.modalite_versement, '') AS "presentation_detail",


attention cette ligne est hyper longue et ça dégrade la visilibité en PR sur GH ^

vmttn · 2024-10-17T14:01:33Z

...models/intermediate/sources/mes_aides/permis_velo/int_mes_aides__permis_velo__structures.sql

+        NULL                                  AS "horaires_ouverture",
+        NULL                                  AS "accessibilite",
+        CASE
+            WHEN permis_velo.typologie_structure LIKE '%Mission locale%' THEN ARRAY['mission-locale']


insensible à la casse

Suggested change

WHEN permis_velo.typologie_structure LIKE '%Mission locale%' THEN ARRAY['mission-locale']

WHEN permis_velo.typologie_structure ~* 'mission locale' THEN ARRAY['mission-locale']

vmttn · 2024-10-17T14:11:16Z

pipeline/dbt/models/staging/sources/mes_aides/_mes_aides__models.yml

  - name: stg_mes_aides__garages
+  - name: stg_mes_aides__permis_velo


est-ce que tu pourrais ajouter quelques data_tests ? au minimum je dirais :

id : not_null, dbt_utils.not_empty_string, unique

date_maj : not_null, not_constant

mais je pense que ça serait bien de faire quelques checks sur les valeurs prédéfinies. Par exemple, le champ type contient Aide financière, Prestation et Accompagnement et ton mapping en dépend. Si une de ces valeurs est renommée, ça sera complément silencieux, il n'y aura juste plus de valeur mappée.

Pour tester ça simplement, je dirais qu'il faut faire comme pour fredo : dans models/staging/sources/mes_aides/ voir models/staging/sources/mes_aides/permis_velo tu crées des models supplémentaires où tu UNNEST : stg_mes_aides__permis_velo__types, etc. et là tu peux ajouter un data_test de type accepted_values.

(d'ailleurs je n'ai pas mis de tests pour les garages c'est dommage :))

...t/models/intermediate/sources/mes_aides/permis_velo/int_mes_aides__permis_velo__services.sql

…rmis-velo

hlecuyer force-pushed the hlecuyer/feat/mes-aides/integration-aides-permis-velo branch from 937e9ea to d0b9fa4 Compare September 17, 2024 14:27

hlecuyer marked this pull request as ready for review September 17, 2024 15:20

hlecuyer requested a review from vmttn as a code owner September 17, 2024 15:20

hlecuyer changed the title ~~Hlecuyer/feat/mes aides/integration aides permis velo~~ feat(mes-aides):integration aides permis velo Sep 17, 2024

hlecuyer marked this pull request as draft September 17, 2024 15:37

hlecuyer force-pushed the hlecuyer/feat/mes-aides/integration-aides-permis-velo branch from d0b9fa4 to 61806d9 Compare September 18, 2024 08:03

hlecuyer marked this pull request as ready for review September 18, 2024 08:36

hlecuyer temporarily deployed to staging September 18, 2024 08:40 — with GitHub Actions Inactive

hlecuyer had a problem deploying to prod September 18, 2024 08:40 — with GitHub Actions Error

vmttn requested changes Sep 18, 2024

View reviewed changes

vmttn changed the title ~~feat(mes-aides):integration aides permis velo~~ feat(mes-aides): integration aides permis velo Sep 18, 2024

vmttn reviewed Sep 18, 2024

View reviewed changes

pipeline/dbt/models/staging/sources/mes_aides/stg_mes_aides__permis_velo.sql Outdated Show resolved Hide resolved

vmttn reviewed Sep 18, 2024

View reviewed changes

.../models/intermediate/sources/mes_aides/permis_velo/int_mes_aides__structures_permis_velo.sql Outdated Show resolved Hide resolved

hlecuyer temporarily deployed to staging September 18, 2024 14:04 — with GitHub Actions Inactive

hlecuyer requested a deployment to prod September 18, 2024 14:04 — with GitHub Actions Waiting

hlecuyer temporarily deployed to staging September 18, 2024 14:42 — with GitHub Actions Inactive

hlecuyer had a problem deploying to prod September 18, 2024 14:42 — with GitHub Actions Failure

hlecuyer temporarily deployed to staging September 18, 2024 15:00 — with GitHub Actions Inactive

hlecuyer had a problem deploying to prod September 18, 2024 15:00 — with GitHub Actions Error

hlecuyer temporarily deployed to staging September 18, 2024 15:16 — with GitHub Actions Inactive

hlecuyer had a problem deploying to prod September 18, 2024 15:16 — with GitHub Actions Failure

hlecuyer force-pushed the hlecuyer/feat/mes-aides/integration-aides-permis-velo branch from 1c66198 to 23eb6c5 Compare September 18, 2024 15:33

hlecuyer temporarily deployed to staging September 18, 2024 15:40 — with GitHub Actions Inactive

hlecuyer had a problem deploying to prod September 18, 2024 15:40 — with GitHub Actions Failure

hlecuyer force-pushed the hlecuyer/feat/mes-aides/integration-aides-permis-velo branch from 23eb6c5 to 8cd30cc Compare September 18, 2024 15:58

hlecuyer temporarily deployed to staging September 18, 2024 16:01 — with GitHub Actions Inactive

hlecuyer had a problem deploying to prod September 18, 2024 16:01 — with GitHub Actions Failure

hlecuyer force-pushed the hlecuyer/feat/mes-aides/integration-aides-permis-velo branch from 8cd30cc to 5e5c762 Compare September 19, 2024 06:41

hlecuyer temporarily deployed to staging September 19, 2024 06:43 — with GitHub Actions Inactive

hlecuyer had a problem deploying to prod September 19, 2024 06:43 — with GitHub Actions Failure

hlecuyer temporarily deployed to staging October 8, 2024 09:00 — with GitHub Actions Inactive

hlecuyer had a problem deploying to prod October 8, 2024 09:01 — with GitHub Actions Failure

hlecuyer force-pushed the hlecuyer/feat/mes-aides/integration-aides-permis-velo branch from e382246 to f96279b Compare October 8, 2024 09:23

hlecuyer temporarily deployed to staging October 8, 2024 09:25 — with GitHub Actions Inactive

hlecuyer had a problem deploying to prod October 8, 2024 09:25 — with GitHub Actions Failure

hlecuyer force-pushed the hlecuyer/feat/mes-aides/integration-aides-permis-velo branch from f96279b to 4c52f8e Compare October 9, 2024 13:33

hlecuyer added the deploy-to-staging Permet d'activer le déploiement de la PR en staging. label Oct 9, 2024

hlecuyer had a problem deploying to staging October 9, 2024 13:37 — with GitHub Actions Failure

hlecuyer temporarily deployed to staging October 9, 2024 14:02 — with GitHub Actions Inactive

hlecuyer had a problem deploying to staging October 15, 2024 13:08 — with GitHub Actions Failure

hlecuyer temporarily deployed to staging October 15, 2024 13:42 — with GitHub Actions Inactive

vmttn removed the deploy-to-staging Permet d'activer le déploiement de la PR en staging. label Oct 15, 2024

vmttn requested changes Oct 17, 2024

View reviewed changes

hlecuyer force-pushed the hlecuyer/feat/mes-aides/integration-aides-permis-velo branch 3 times, most recently from 230ce5c to 0314344 Compare October 22, 2024 15:57

vmttn approved these changes Oct 22, 2024

View reviewed changes

hlecuyer commented Oct 22, 2024

View reviewed changes

...t/models/intermediate/sources/mes_aides/permis_velo/int_mes_aides__permis_velo__services.sql Show resolved Hide resolved

vmttn approved these changes Oct 24, 2024

View reviewed changes

hlecuyer added the deploy-to-staging Permet d'activer le déploiement de la PR en staging. label Oct 24, 2024

hlecuyer temporarily deployed to staging October 24, 2024 09:11 — with GitHub Actions Inactive

hlecuyer temporarily deployed to staging October 24, 2024 10:44 — with GitHub Actions Inactive

vmttn approved these changes Oct 24, 2024

View reviewed changes

hlecuyer temporarily deployed to staging October 24, 2024 11:34 — with GitHub Actions Inactive

feat(mes_aides): add permis velos from mes_aides source

56835c2

hlecuyer force-pushed the hlecuyer/feat/mes-aides/integration-aides-permis-velo branch from 5dbfca5 to 56835c2 Compare October 24, 2024 12:37

Merge branch 'main' into hlecuyer/feat/mes-aides/integration-aides-pe…

f1df21b

…rmis-velo

hlecuyer merged commit b6787ec into main Oct 24, 2024
6 checks passed

hlecuyer deleted the hlecuyer/feat/mes-aides/integration-aides-permis-velo branch October 24, 2024 12:40

hlecuyer temporarily deployed to staging October 24, 2024 12:45 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(mes-aides): integration aides permis velo #296

feat(mes-aides): integration aides permis velo #296

hlecuyer commented Sep 17, 2024

vmttn left a comment

vmttn left a comment

vmttn Oct 17, 2024

vmttn Oct 17, 2024

vmttn Oct 17, 2024

vmttn Oct 17, 2024

vmttn Oct 17, 2024

vmttn Oct 17, 2024

vmttn Oct 17, 2024

vmttn Oct 17, 2024 •

edited

Loading

		STRING_TO_ARRAY(data #>> '{fields,Liaisons Besoins}', ', ') AS "liaisons_besoins_mes_aides",
		data #>> '{fields,Slug Thématiques}' AS "slug_thematique_mes_aides",

		data #>> '{fields,SIRET}' AS "siret_structure",
		data #>> '{fields,Slug Organisme}' AS "slug_organisme_structure",

		data #>> '{fields,Nom Organisme}' AS "nom_organisme_structure",
		data #>> '{fields,Organisme Type}' AS "typologie_structure",

	TRIM(SUBSTRING(data #>> '{fields,Liaisons Villes}' FROM '\((\d+)\)$')) AS "liaisons_code_postal",
	TRIM(SUBSTRING(data #>> '{fields,Liaisons Villes}' FROM '\((\d+)\)$')) AS "liaisons_ville_code_postal",

	WHEN permis_velo.typologie_structure LIKE '%Mission locale%' THEN ARRAY['mission-locale']
	WHEN permis_velo.typologie_structure ~* 'mission locale' THEN ARRAY['mission-locale']

		- name: stg_mes_aides__garages
		- name: stg_mes_aides__permis_velo

feat(mes-aides): integration aides permis velo #296

feat(mes-aides): integration aides permis velo #296

Conversation

hlecuyer commented Sep 17, 2024

vmttn left a comment

Choose a reason for hiding this comment

vmttn left a comment

Choose a reason for hiding this comment

vmttn Oct 17, 2024

Choose a reason for hiding this comment

vmttn Oct 17, 2024

Choose a reason for hiding this comment

vmttn Oct 17, 2024

Choose a reason for hiding this comment

vmttn Oct 17, 2024

Choose a reason for hiding this comment

vmttn Oct 17, 2024

Choose a reason for hiding this comment

vmttn Oct 17, 2024

Choose a reason for hiding this comment

vmttn Oct 17, 2024

Choose a reason for hiding this comment

vmttn Oct 17, 2024 • edited Loading

Choose a reason for hiding this comment

vmttn Oct 17, 2024 •

edited

Loading