Update of the validation process step 1 #1785
Comments
|
@geoffj-FUG |
|
Anne
I had no preference. If you find it easier the other way round that is fine.
The important thing is that we will have 2 warning / error fields, so when an entry is propagated through a piece all the flags can be changed and the Validator does not have to keep looking at it. There can be a hundred or so entries to accept under the current system. We will be reducing that to one. (The biggest I ever encountered was almost 800. Luckily it was in the vld revalidation so I cleared them all together).
By having two fields we can do this. With just one field the system would not have known whether there were other problems in the record.
Geoff
From: Anne Vandervord ***@***.***>
Sent: Wednesday, June 12, 2024 12:34 AM
To: FreeUKGen/FreeCENMigration ***@***.***>
Cc: Geoff J ***@***.***>; Mention ***@***.***>
Subject: Re: [FreeUKGen/FreeCENMigration] Update of the validation process step 1 (Issue #1785)
@geoffj-FUG <https://github.com/geoffj-FUG>
Just wondering why the new warning/error field and utilisation of the existing warning/error field has been specified that way rather than using the new for the POB validation status (I'm probably missing something)?
—
Reply to this email directly, view it on GitHub <#1785 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKCPIFI5AZ3PH7NK2ROWDQ3ZG4DEPAVCNFSM6AAAAABI6BYFBKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRQHEZDGNBVGM> .
You are receiving this because you were mentioned. <https://github.com/notifications/beacon/AKCPIFKB5TU66NUHZYWY3PTZG4DEPA5CNFSM6AAAAABI6BYFBKWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUAZUJT2.gif> Message ID: ***@***.*** ***@***.***> >
|
|
@geoffj-FUG
|
|
Anne
1. Yes, we will need both columns in the validator’s download. At the moment we have just one but we will need one for each field.
I don’t think the extra action is necessary. It will be a case of all warnings being displayed in order based on the two fields instead of one.
2. For the VLD collection other than the rename revalidation will remain exactly as it is now. The rename is for consistency. For the csv collection prevalidation will be offered as part of the prepare for validation action.
Geoff
From: Anne Vandervord ***@***.***>
Sent: Thursday, June 13, 2024 6:40 PM
To: FreeUKGen/FreeCENMigration ***@***.***>
Cc: Geoff J ***@***.***>; Mention ***@***.***>
Subject: Re: [FreeUKGen/FreeCENMigration] Update of the validation process step 1 (Issue #1785)
@geoffj-FUG <https://github.com/geoffj-FUG>
Just starting to look at this now.
Some questions, there are bound to be more as I get into it.
1. Regarding the ‘POB valid ’ field:
Where will this be shown to the validator - in the file download?
Do you want an additional action button ‘View POB warnings’ where only records that have POB warnings are presented?
2. The VLD auto-validation process used by the data manager is to be renamed Pre-validation. Will all the other POB validation actions (‘Status’, List VLD entries and Manual Val) remain as is? Or are you referring to the Overall ‘POB Validation’ group heading?
—
Reply to this email directly, view it on GitHub <#1785 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKCPIFPETYQITE5SPITFFDTZHFLFZAVCNFSM6AAAAABI6BYFBKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRVGAYTANBYGA> .
You are receiving this because you were mentioned. <https://github.com/notifications/beacon/AKCPIFJGTG5O5ORFRQJ6M43ZHFLFZA5CNFSM6AAAAABI6BYFBKWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUBBNYDA.gif> Message ID: ***@***.*** ***@***.***> >
|
|
@geoffj-FUG
NOTE: Repurposing the current filed (named 'Record Valid') is not really on - so I'm likely to have to create two new fields unless the only place that you need to see the 'POB/NonPOB' Validity split is in the validator's download.
|
|
Anne
3. The only place that we need to see the value of the warning field is in the Validator’s download. If either of the fields is false we need to see the message of the issue that caused that condition in the information about the Warning – on the Warnings list and in the box at the top of the validator’s screen.
4. Fathers place of birth was shown but we do not intend to make it searchable. It is just a text field to be displayed to the researcher.
Geoff
From: Anne Vandervord ***@***.***>
Sent: Saturday, June 15, 2024 1:27 AM
To: FreeUKGen/FreeCENMigration ***@***.***>
Cc: Geoff J ***@***.***>; Mention ***@***.***>
Subject: Re: [FreeUKGen/FreeCENMigration] Update of the validation process step 1 (Issue #1785)
@geoffj-FUG <https://github.com/geoffj-FUG>
More Questions (continuing the numbering):
3. Will the 'POB/NonPOB Valid' field need to be shown on the Screens that show the CSV Record -see screenshot below
NOTE: Repurposing the current filed (named 'Record Valid') is not really on - so I'm likely to have to create two new fields unless the only place that you need to see the 'POB/NonPOB' Validity split is in the validator's download.
4. I notice that 1911 Channel Island Census appears to have a field for Father's POB but it is just a text string with no separate Father's County of Birth, so presumably this should not be considered as a 'place of birth' field in your specification.
—
Reply to this email directly, view it on GitHub <#1785 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKCPIFPXE6VOXX2UG7Z3PR3ZHMDUBAVCNFSM6AAAAABI6BYFBKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRYGI3TANZUHE> .
You are receiving this because you were mentioned. <https://github.com/notifications/beacon/AKCPIFLNAZ4J7BOHBW54PYDZHMDUBA5CNFSM6AAAAABI6BYFBKWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUBHUXZ2.gif> Message ID: ***@***.*** ***@***.***> >
|
|
@geoffj-FUG Is this what you want (see screenshot below) and also is it all actions in the 'POB Validation' group that should be available to Validator, Coordinator and Data Manager or just the 'Pre-Validation' action? |
|
Anne
It looks as if I got swamped by the detail in this one. Prevalidation should be the name, but it should only be available to the Coordinator and Data manager.
Geoff
From: Anne Vandervord ***@***.***>
Sent: Wednesday, June 19, 2024 2:35 AM
To: FreeUKGen/FreeCENMigration ***@***.***>
Cc: Geoff J ***@***.***>; Mention ***@***.***>
Subject: Re: [FreeUKGen/FreeCENMigration] Update of the validation process step 1 (Issue #1785)
@geoffj-FUG <https://github.com/geoffj-FUG>
5. RE: The auto-validation process used by the data manager is to be renamed Pre-validation. This term is to appear against each file in the vld collection. Permission to use it is to be changed to Validator, Coordinator and Data Manager.
Is this what you want (see screenshot below) and also is it all actions in the 'POB Validation' group that should be available to Validator, Coordinator and Data Manager or just the 'Pre-Validation' action?
—
Reply to this email directly, view it on GitHub <#1785 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKCPIFJBWOPKVZDUBOYCOGTZIBOTXAVCNFSM6AAAAABI6BYFBKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZWGUZDINZYGY> .
You are receiving this because you were mentioned. <https://github.com/notifications/beacon/AKCPIFIKZC6DNESIIQ7LS3LZIBOTXA5CNFSM6AAAAABI6BYFBKWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUBXMQ7E.gif> Message ID: ***@***.*** ***@***.***> >
|
|
Anne
Why closed? Is it live?
Geoff
From: Anne Vandervord ***@***.***>
Sent: Wednesday, June 19, 2024 8:52 PM
To: FreeUKGen/FreeCENMigration ***@***.***>
Cc: Geoff J ***@***.***>; Mention ***@***.***>
Subject: Re: [FreeUKGen/FreeCENMigration] Update of the validation process step 1 (Issue #1785)
Closed #1785 <#1785> as completed.
—
Reply to this email directly, view it on GitHub <#1785 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKCPIFO2JOCA3S5HUITWEOTZIFPDNAVCNFSM6AAAAABI6BYFBKVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJTGIYTINBQHA3TIMQ> .
You are receiving this because you were mentioned. <https://github.com/notifications/beacon/AKCPIFO6ACFTLURN5PXSLZ3ZIFPDNA5CNFSM6AAAAABI6BYFBKWGG33NNVSW45C7OR4XAZNWJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XKUY3PNVWWK3TUL5UWJTYAAAAAGE5D4ATA.gif> Message ID: ***@***.*** ***@***.***> >
|
|
@geoffj-FUG |
|
Anne
Do we have a copy of the revalidation data on test3? If so I will be able to revalidate automatically in test. If not I cant test it as there is no data to prevalidate from?
Geoff
From: Anne Vandervord ***@***.***>
Sent: Thursday, June 20, 2024 1:32 AM
To: FreeUKGen/FreeCENMigration ***@***.***>
Cc: Geoff J ***@***.***>; Mention ***@***.***>
Subject: Re: [FreeUKGen/FreeCENMigration] Update of the validation process step 1 (Issue #1785)
@geoffj-FUG <https://github.com/geoffj-FUG>
This is now ready for testing on Test3.
Regarding 'Create one new warning or error field. This field is only used when validation is in progress.
This field is a True/False field. It is set at true when there are no warnings or errors associated with non place of birth fields in a record.
The existing warning or error field is to be re-utilised as a place of birth warning or error field. It is set to true when the Place of birth fields are valid, or the entries are accepted by the validator.'
I actually added 2 new fields to the file download when validation is in progress (pop_valid and non_pob_valid) the record_valid field is still there. Validators can easily ignore it if they want to. I did this because record_valid is used throughout the code and I didn't want to have to review and potentially change code unless I absolutely had to.
—
Reply to this email directly, view it on GitHub <#1785 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKCPIFKJV2EQ7BP27TWIVTDZIGP6ZAVCNFSM6AAAAABI6BYFBKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZYHE4DMNJVGE> .
You are receiving this because you were mentioned. <https://github.com/notifications/beacon/AKCPIFNB4RYYEICXUZ4V6A3ZIGP6ZA5CNFSM6AAAAABI6BYFBKWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUB4CZDO.gif> Message ID: ***@***.*** ***@***.***> >
|
|
Hi Geoff,
Test3 has its own prevalidation data (not a copy from Production). You can add specific data to it testing by pre-validating a VLD file (perhaps one you have done in production all ready so know the data) and propagating changes in the usual way.
(An exact copy from Production won’t work because the Gazetteer is not exactly the same either.)
Anne
On 20 Jun 2024, at 08:26, Geoff J ***@***.***> wrote:
Anne
Do we have a copy of the revalidation data on test3? If so I will be able to revalidate automatically in test. If not I cant test it as there is no data to prevalidate from?
Geoff
From: Anne Vandervord ***@***.***>
Sent: Thursday, June 20, 2024 1:32 AM
To: FreeUKGen/FreeCENMigration ***@***.***>
Cc: Geoff J ***@***.***>; Mention ***@***.***>
Subject: Re: [FreeUKGen/FreeCENMigration] Update of the validation process step 1 (Issue #1785)
@geoffj-FUG <https://github.com/geoffj-FUG>
This is now ready for testing on Test3.
Regarding 'Create one new warning or error field. This field is only used when validation is in progress.
This field is a True/False field. It is set at true when there are no warnings or errors associated with non place of birth fields in a record.
The existing warning or error field is to be re-utilised as a place of birth warning or error field. It is set to true when the Place of birth fields are valid, or the entries are accepted by the validator.'
I actually added 2 new fields to the file download when validation is in progress (pop_valid and non_pob_valid) the record_valid field is still there. Validators can easily ignore it if they want to. I did this because record_valid is used throughout the code and I didn't want to have to review and potentially change code unless I absolutely had to.
—
Reply to this email directly, view it on GitHub <#1785 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKCPIFKJV2EQ7BP27TWIVTDZIGP6ZAVCNFSM6AAAAABI6BYFBKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZYHE4DMNJVGE> .
You are receiving this because you were mentioned. <https://github.com/notifications/beacon/AKCPIFNB4RYYEICXUZ4V6A3ZIGP6ZA5CNFSM6AAAAABI6BYFBKWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUB4CZDO.gif> Message ID: ***@***.*** ***@***.***> >
—
Reply to this email directly, view it on GitHub<#1785 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ARLANZ6SYLEZ4U2O5KXBNKLZIJ7ZLAVCNFSM6AAAAABI6BYFBKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZZHE4TQNBZGM>.
You are receiving this because you modified the open/close state.Message ID: ***@***.***>
|
|
Test3 has its own prevalidation data (not a copy from Production). You can add specific data to it for testing by pre-validating a VLD file (perhaps one you have done in production all ready so know the data) and propagating changes in the usual way. |
geoffj-FUG
added
Ready for Deployment
|
I have tested this on both csv and vld files and I am comfortable with the results. Geoff |
|
Geoff to inform the Coordinators |
geoffj-FUG
removed
the
Ready for Deployment
|
HO107_1925 is a file where validation is almost complete. Uploaded as a new file. It should reject as a file that is not currently being validated. Not uploaded and no email received. Does not appear in list of files. Problem, no email or response in this situation. |
|
Now tests OK. This is ready for Deployment but should be done with Step 2. Geoff |
This is part of the Epic #1783. Please read the document linked to #1783 when considering this story.
The initial work to complete this epic is:
Create one new warning or error field. This field is only used when validation is in progress.
This field is a True/False field. It is set at true when there are no warnings or errors associated with non place of birth fields in a record.
The existing warning or error field is to be re-utilised as a place of birth warning or error field. It is set to true when the Place of birth fields are valid, or the entries are accepted by the validator.
The auto-validation process used by the data manager is to be renamed Pre-validation. This term is to appear against each file in the vld collection. Permission to use it is to be changed to Validator, Coordinator and Data Manager.
The validation of a csv file is to be amended. When the Coordinator sets a piece to be validated they are to be asked whether they wish to pre-validate the file. If the response is yes then the existing auto-validation code is to be run against the csv file and any alternative POBs updated based on the database established during revalidation of the vld collection.
Once this is tested and accepted pre-validation can be deployed and used by Coordinators.
Geoff
The text was updated successfully, but these errors were encountered: