VACMS-16335 Add VAMC System police contact migration #16430

swirtSJW · 2023-12-14T16:07:13Z

Description

Relates to #16335

Testing done

Screenshots

QA steps

What needs to be checked to prove this works?
What needs to be checked to prove it didn't break any related things?
What variations of circumstances (users, actions, values) need to be checked?

As an admin

Go to the VA Police contact migration
- Validate the the row for "Node - VAMC System Police contact" shows 170 total items (the number of data rows in the CSV)
Click the Execute button for "vamc_system_police_contact" and Execute the migration with no additional options.
- Validate that 170 items were processed, Should show 139 created... which is a like it only updated, but it can't tell.

Go to the migrate messages list
- validate there are no messages for this migration.
Go to the content list filtered by va police
Toggle the updated date sort back and forth.
- validate the first item in the list and the last were both updated by the migration. (times may be a few minutes apart)
Go to content with extension
- validate that it has both a phone number and extension
- Validate it has a revision log message by cms migrator that shows the update of contact info.
Go to content with no extension
- validate that it has both a phone number but no extension

Definition of Done

Documentation has been updated, if applicable.
Tests have been added if necessary.
Automated tests have passed.
Code Quality Tests have passed.
Acceptance Criteria in related issue are met.
Manual Code Review Approved.
If there are field changes, front end output has been thoroughly checked.

Select Team for PR review

Is this PR blocked by another PR?

DO NOT MERGE

Does this PR need review from a Product Owner

Needs PO review

CMS user-facing announcement

Is an announcement needed to let editors know of this change?

Yes, and it's written in issue ____ and queued for publication.
- Merge and ping the UX writer so they are ready to publish after deployment
Yes, but it hasn't yet been written
- Don't merge yet -- ping the UX writer to write and queue content
No announcement is needed for this code change.
- Merge & carry on unburdened by announcements

swirtSJW · 2023-12-15T20:42:56Z

The node lookup is currently failing

edited 12/16/23: This is resolved.

swirtSJW · 2023-12-16T18:13:57Z

There is an upstream issue with token replacement

I will get it patched and fixed this weekend.

…ess plugins.

swirtSJW · 2023-12-18T02:18:24Z

This is largely working
I resolved the upstream bug #3409215: Process plugin create_default_paragraph_revision does not support values or tokens

I also created an upstream feature request and patched it to work out the phone number parsing #3409322: Add process plugin gate_comparator

I am only seeing 4 failures related to failed lookups. This is usually the result of data mismatches in the CSV. I am looking into those now

  Source ID(s)                                                Destination ID(s)   Level   Message                                                             
 ----------------------------------------------------------- ------------------- ------- -------------------------------------------------------------------- 
  16: MO":VA Fayetteville Arkansas health care                                    4       vamc_system_police_contact:nid: Entity lookup found no VAMC System  
                                                                                          for @vamc_system_label.                                             
  VA Greater Los Angeles health care:VA Greater LA HCS -                          4       vamc_system_police_contact:nid: Entity lookup found no police page  
  Sepulveda ACC:vha_691                                                                   for @vamc_system_label.                                             
  VA Greater Los Angeles health care:VA Greater LA HCS - W.                       4       vamc_system_police_contact:nid: Entity lookup found no police page  
  LA VAMC:vha_691                                                                         for @vamc_system_label.                                             
  17:"West Texas VA HCS - George H. O'Brien:VA West Texas                         4       vamc_system_police_contact:nid: Entity lookup found no VAMC System  
  health care                                                                             for @vamc_system_label.

swirtSJW · 2023-12-18T02:51:50Z

Solved the problem with the missing entity lookups. It was due to a mismatch in data enclosures. Switched to " and we are all good.

swirtSJW · 2023-12-18T03:10:12Z

16 are not showing as updated.... which is probably a data problem

Looking at it by the numbers
Rows: 170
skips: 31
Number of Police nodes 139
Expected updates based on rows 170-31 = 139

created and updated: 132
The updated count 9 plus the extra skipped (38ignored - 31expected = 7) Gives us the missing 16.

Now to figure out why.

swirtSJW · 2023-12-18T03:56:41Z

Troubleshooting log

The missing 16 were not skipped by 'skip' setting in CSV
The missing 16 were correctly labelled for the system
Difference between the updated and the ignored? nothing.
Forcing an update migration changes nothing.
There are no messages that would indicate a lookup could not be found.
ones not updated
- VA Washington DC health care
- VA White River Junction health care
- VA Walla Walla health care
- VA Syracuse health care
- VA Western New York health care
- VA Tuscaloosa health care
- VA West Texas health care
- VA Western Colorado health care
- VA Tampa health care
- VA West Palm Beach health care
- VA Wilmington health care
- VA Wilkes-Barre health care
- VA Tennessee Valley health care
- VA Wichita health care
- VA Tomah health care
- VA Texas Valley health care
Noticing these are all late in the alphabet.
They all the rows that appear after row 154 - St Louis.

swirtSJW · 2023-12-18T05:34:49Z

Solved, the collection of keys was not unique enough to make an id.
Now it is.

swirtSJW · 2023-12-18T13:08:37Z

composer.json

@@ -132,7 +132,7 @@
        "drupal/migrate_source_csv": "^3.4",
        "drupal/migrate_source_ui": "^1.0",
        "drupal/migrate_tools": "^6.0",
-        "drupal/migration_tools": "^2.7",
+        "drupal/migration_tools": "^2.8",


Update required to get

https://www.drupal.org/project/migration_tools/issues/3409215

https://www.drupal.org/project/migration_tools/issues/3409322

swirtSJW · 2023-12-18T13:09:54Z

config/sync/migrate_plus.migration.vamc_system_police_contact.yml

+      plugin: callback
+      callable: trim
+    -
+      plugin: gate_comparator


New process plugin https://www.drupal.org/project/migration_tools/issues/3409322

swirtSJW · 2023-12-18T13:11:45Z

config/sync/migrate_plus.migration.vamc_system_police_contact.yml

+  nid:
+    -
+      plugin: entity_lookup
+      access_check: false
+      ignore_case: true
+      entity_type: node
+      bundle_key: type
+      bundle: health_care_region_page
+      value_key: title
+      source: vamc_system_label
+    -
+      plugin: skip_on_empty
+      method: row
+      message: 'Entity lookup found no VAMC System for @vamc_system_label.'
+    -
+      plugin: entity_lookup
+      access_check: false
+      ignore_case: true
+      entity_type: node
+      bundle_key: type
+      bundle: vamc_system_va_police
+      value_key: field_office
+    -
+      plugin: skip_on_empty
+      method: row
+      message: 'Entity lookup found no police page for @vamc_system_label.'


This dance is needed to lookup the nid so that it can update an existing node rather than migrate in a new one.

omahane

With the exception of the

Click the Execute button for "vamc_system_police_contact" and Execute the migration with no additional options.

actually requiring me to choose the "update" option because the nodes had already been migrated, everything else was as described.

This migration looks pretty hairy. Your solution is really impressive.

va-cms-bot temporarily deployed to Tugboat December 14, 2023 16:07 Destroyed

swirtSJW self-assigned this Dec 14, 2023

swirtSJW mentioned this pull request Dec 14, 2023

Drupal: Update police page for MVP showing contact information and FAQs #16335

Closed

7 tasks

swirtSJW force-pushed the VACMS-16335-va-police-contact-migration branch from 6306bed to b151294 Compare December 15, 2023 20:41

va-cms-bot temporarily deployed to Tugboat December 15, 2023 20:41 Destroyed

swirtSJW added 5 commits December 17, 2023 20:55

VACMS-16335 Add VAMC System police contact migration

b30af06

VACMS-16335 Update migration to add skip.

f26a551

VACMS-16335 Update migration CSV to add skip.

026626b

VACMS-16335 Update police contact migration to properly set phone.

25fea76

VACMS-16335 Update migration_tools module to bring in updates to proc…

f83493c

…ess plugins.

swirtSJW force-pushed the VACMS-16335-va-police-contact-migration branch from b151294 to f83493c Compare December 18, 2023 02:02

va-cms-bot temporarily deployed to Tugboat December 18, 2023 02:03 Destroyed

VACMS-16335 Remove empry rows from CSV.

dfc24e8

VACMS-166335 Tweak data and parse wrapper.

b947279

va-cms-bot temporarily deployed to Tugboat December 18, 2023 02:47 Destroyed

VACMS-16335 Make migration id, more unique

4844aca

va-cms-bot temporarily deployed to Tugboat December 18, 2023 05:25 Destroyed

swirtSJW commented Dec 18, 2023

View reviewed changes

swirtSJW marked this pull request as ready for review December 18, 2023 14:37

swirtSJW requested review from a team as code owners December 18, 2023 14:37

swirtSJW requested a review from omahane December 18, 2023 14:37

omahane approved these changes Dec 18, 2023

View reviewed changes

dsasser approved these changes Dec 18, 2023

View reviewed changes

swirtSJW merged commit 430efdc into main Dec 18, 2023
29 checks passed

swirtSJW deleted the VACMS-16335-va-police-contact-migration branch December 18, 2023 16:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VACMS-16335 Add VAMC System police contact migration #16430

VACMS-16335 Add VAMC System police contact migration #16430

swirtSJW commented Dec 14, 2023 •

edited by omahane

Loading

swirtSJW commented Dec 15, 2023 •

edited

Loading

swirtSJW commented Dec 16, 2023

swirtSJW commented Dec 18, 2023

swirtSJW commented Dec 18, 2023

swirtSJW commented Dec 18, 2023 •

edited

Loading

swirtSJW commented Dec 18, 2023 •

edited

Loading

swirtSJW commented Dec 18, 2023

swirtSJW Dec 18, 2023

swirtSJW Dec 18, 2023

swirtSJW Dec 18, 2023

omahane left a comment •

edited

Loading

VACMS-16335 Add VAMC System police contact migration #16430

VACMS-16335 Add VAMC System police contact migration #16430

Conversation

swirtSJW commented Dec 14, 2023 • edited by omahane Loading

Description

Testing done

Screenshots

QA steps

Definition of Done

Select Team for PR review

Is this PR blocked by another PR?

Does this PR need review from a Product Owner

CMS user-facing announcement

swirtSJW commented Dec 15, 2023 • edited Loading

swirtSJW commented Dec 16, 2023

swirtSJW commented Dec 18, 2023

swirtSJW commented Dec 18, 2023

swirtSJW commented Dec 18, 2023 • edited Loading

swirtSJW commented Dec 18, 2023 • edited Loading

swirtSJW commented Dec 18, 2023

swirtSJW Dec 18, 2023

Choose a reason for hiding this comment

swirtSJW Dec 18, 2023

Choose a reason for hiding this comment

swirtSJW Dec 18, 2023

Choose a reason for hiding this comment

omahane left a comment • edited Loading

Choose a reason for hiding this comment

swirtSJW commented Dec 14, 2023 •

edited by omahane

Loading

swirtSJW commented Dec 15, 2023 •

edited

Loading

swirtSJW commented Dec 18, 2023 •

edited

Loading

swirtSJW commented Dec 18, 2023 •

edited

Loading

omahane left a comment •

edited

Loading