Merge pull request #39 from OasisLMF/release/3.1.0

* Add Platform testing to ODS-tools github actions (#31)

* Test remote trigger of platform unit tests

* force retest

* retest

* Update remote workflow ref

* Feature/fillempty (#32)

* introduce fill_empty to set default for empty cells

* fix read_csv only read needed line when reading header (#27)

* fix read_csv only read needed line when reading header
* set column types and empty val to be the same between parquet and csv
* empty string are converted to nan to be compatible with read_csv and the usage of fillna
* introduce fill_empty to set default for empty cells

---------

Co-authored-by: Sam Gamble <[email protected]>

* Fix Settings compatibility for older workers  (#36)

* Add settings validation to command line

* Fix CompatibilityMap - keep old key in settings file

* pep8

* Replace hardcoded packages, with find_packages (#40)

* Replace hardcoded packages, with findpackages

* misisng ,

* fix check category (#41)

* support empty OedSource (#43)

* support empty OedSource

* fixup

* Set package to version 3.1.0

* add option to manage unknown column when saving files (#46)

* add option to manage unknown column when saving files

* add check for conditional requirement (#44)

* add check for conditional requirement

* pep8

* Update schema to 3.0.4

* base conditional requirement on file type

* set schema to test release

* default val don't start a CR validation

* fixup

* Revert "set schema to test release"

This reverts commit 9fbdaa0.

---------

Co-authored-by: Sam Gamble <[email protected]>

* Update changelog

* Fix broken changelog

---------

Co-authored-by: Stephane Struzik <[email protected]>
Co-authored-by: awsbuild <[email protected]>

.github/workflows/test-platform.yml

@@ -0,0 +1,17 @@
+name: Platform Integration
+
+on:
+  pull_request:
+  workflow_dispatch:
+    inputs:
+      platform_branch:
+        description: "Platform branch to test with: [git ref]"
+        required: false
+
+jobs:
+  platform_testing:
+    uses: OasisLMF/OasisPlatform/.github/workflows/test-python.yml@develop
+    secrets: inherit
+    with:
+      ods_branch: ${{ github.ref }}
+      platform_branch: ${{ github.event_name != 'workflow_dispatch' && 'develop' ||  inputs.platform_branch }}

CHANGELOG.rst

@@ -1,6 +1,21 @@
 ODS_Tools Changelog
 ===================
 
+
+`3.1.0`_
+ ---------
+* [#33](https://github.com/OasisLMF/ODS_Tools/pull/32) - Consistant managment of blank values
+* [#35](https://github.com/OasisLMF/ODS_Tools/pull/36) - Bug in Analysis settings compatibility map
+* [#40](https://github.com/OasisLMF/ODS_Tools/pull/40) - Replace hardcoded packages, with find_packages
+* [#41](https://github.com/OasisLMF/ODS_Tools/pull/41) - fix check category
+* [#42](https://github.com/OasisLMF/ODS_Tools/pull/43) - ods_tools check crashes on empty location file
+* [#44](https://github.com/OasisLMF/ODS_Tools/pull/44) - add check for conditional requirement
+* [#46](https://github.com/OasisLMF/ODS_Tools/pull/46) - add option to manage unknown column when saving files
+* [#27](https://github.com/OasisLMF/ODS_Tools/pull/27) - fix read_csv only read needed line when reading header
+* [#30](https://github.com/OasisLMF/ODS_Tools/pull/30) - Release/3.0.6
+* [#31](https://github.com/OasisLMF/ODS_Tools/pull/31) - Add Platform testing to ODS-tools github actions
+.. _`3.1.0`:  https://github.com/OasisLMF/ODS_Tools/compare/3.0.7...3.1.0
+
 `3.0.6`_
  ---------
 * [#21](https://github.com/OasisLMF/ODS_Tools/pull/22) - LocPeril raise validation error when Blank but it should be allowed according to schema
@@ -12,20 +27,20 @@ ODS_Tools Changelog
 
 `3.0.5`_
  ---------
-* [#5](https://github.com/OasisLMF/ODS_Tools/pull/1) - Separate the ods-tools code from the data standard 
-* [#8](https://github.com/OasisLMF/ODS_Tools/pull/9) - Parquet OED ReinsScope incorrectly fail validation 
-* [#12](https://github.com/OasisLMF/ODS_Tools/pull/13) - Analysis Settings in the  OasisPlatform OpenAPI schema fails build check 
+* [#5](https://github.com/OasisLMF/ODS_Tools/pull/1) - Separate the ods-tools code from the data standard
+* [#8](https://github.com/OasisLMF/ODS_Tools/pull/9) - Parquet OED ReinsScope incorrectly fail validation
+* [#12](https://github.com/OasisLMF/ODS_Tools/pull/13) - Analysis Settings in the  OasisPlatform OpenAPI schema fails build check
 * [#15](https://github.com/OasisLMF/ODS_Tools/pull/15) - Release/3.0.4
-* [#17](https://github.com/OasisLMF/ODS_Tools/pull/18) - Loading blank fields from DataFrame creates string value 'None' 
+* [#17](https://github.com/OasisLMF/ODS_Tools/pull/18) - Loading blank fields from DataFrame creates string value 'None'
 * [#19](https://github.com/OasisLMF/ODS_Tools/pull/19) - support stream as Oed Source
 * [#20](https://github.com/OasisLMF/ODS_Tools/pull/20) - detect stream_obj type if not set
 * [#21](https://github.com/OasisLMF/ODS_Tools/pull/22) - LocPeril raise validation error when Blank but it should be allowed according to schema
 .. _`3.0.5`:  https://github.com/OasisLMF/ODS_Tools/compare/3.0.4...3.0.5
 
 `3.0.4`_
  ---------
-* [#5](https://github.com/OasisLMF/ODS_Tools/pull/1) - Separate the ods-tools code from the data standard 
-* [#12](https://github.com/OasisLMF/ODS_Tools/pull/13) - Analysis Settings in the  OasisPlatform OpenAPI schema fails build check 
-* [#8](https://github.com/OasisLMF/ODS_Tools/pull/9) - Parquet OED ReinsScope incorrectly fail validation 
+* [#5](https://github.com/OasisLMF/ODS_Tools/pull/1) - Separate the ods-tools code from the data standard
+* [#12](https://github.com/OasisLMF/ODS_Tools/pull/13) - Analysis Settings in the  OasisPlatform OpenAPI schema fails build check
+* [#8](https://github.com/OasisLMF/ODS_Tools/pull/9) - Parquet OED ReinsScope incorrectly fail validation
 .. _`3.0.4`:  https://github.com/OasisLMF/ODS_Tools/compare/3.0.3...3.0.4
 

ods_tools/__init__.py

@@ -1,4 +1,4 @@
-__version__ = '3.0.6'
+__version__ = '3.1.0'
 
 import logging
 

ods_tools/main.py

@@ -10,8 +10,13 @@
 import argparse
 import logging
 
-from ods_tools.oed import OedExposure, OdsException
 from ods_tools import logger
+from ods_tools.oed import (
+    OedExposure,
+    OdsException,
+    ModelSettingSchema,
+    AnalysisSettingSchema,
+)
 
 
 def get_oed_exposure(config_json=None, oed_dir=None, **kwargs):
@@ -34,8 +39,21 @@ def extract_exposure_args(kwargs):
 
 def check(**kwargs):
     """run the check command on Exposure"""
-    oed_exposure = get_oed_exposure(**extract_exposure_args(kwargs))
-    oed_exposure.check(**kwargs)
+    logger = logging.getLogger(__name__)
+    args_set = {k for k, v in kwargs.items() if v is not None}
+    args_exp = set(['location', 'account', 'ri_info', 'ri_scope'])
+
+    try:
+        if args_exp.intersection(set(args_set)):
+            oed_exposure = get_oed_exposure(**extract_exposure_args(kwargs))
+            oed_exposure.check()
+        if 'analysis_settings_json' in args_set:
+            AnalysisSettingSchema().validate_file(kwargs['analysis_settings_json'])
+        if 'model_settings_json' in args_set:
+            ModelSettingSchema().validate_file(kwargs['model_settings_json'])
+    except OdsException as e:
+        logger.error('Validation failed:')
+        logger.error(e)
 
 
 def convert(**kwargs):
@@ -70,7 +88,7 @@ def add_exposure_data_args(command):
  - by providing  the path to each OED source using `--location`, `--account`, `--ri-info`, `--ri-scope`.
  - by providing  an OED config json file using `--config-json`.
  - by providing  the path to the directory where the exposure is stored using `--oed-dir`.
- 
+
 if multiple options are use at the same time, --config-json will have the priority over --oed-dir
 specific paths (--location, --account, --ri-info, --ri-scope) will overwrite the path found in (--config-json or --oed-dir)
  """
@@ -98,6 +116,8 @@ def add_exposure_data_args(command):
 check_command = command_parser.add_parser('check', description=check_description + oed_exposure_creation,
                                           formatter_class=argparse.RawTextHelpFormatter)
 add_exposure_data_args(check_command)
+check_command.add_argument('--model-settings-json', help='Path to Model settings meta-data file to check', default=None)
+check_command.add_argument('--analysis-settings-json', help='Path to Analysis settings file to check', default=None)
 check_command.add_argument('-v', '--logging-level', help='logging level (debug:10, info:20, warning:30, error:40, critical:50)',
                            default=30, type=int)
 

ods_tools/oed/__init__.py

@@ -4,12 +4,14 @@
 from .source import OedSource
 from .common import (
     OdsException, PANDAS_COMPRESSION_MAP, PANDAS_DEFAULT_NULL_VALUES, USUAL_FILE_NAME, OED_TYPE_TO_NAME,
-    OED_NAME_TO_TYPE, OED_IDENTIFIER_FIELDS, VALIDATOR_ON_ERROR_ACTION, DEFAULT_VALIDATION_CONFIG, OED_PERIL_COLUMNS
+    OED_NAME_TO_TYPE, OED_IDENTIFIER_FIELDS, VALIDATOR_ON_ERROR_ACTION, DEFAULT_VALIDATION_CONFIG, OED_PERIL_COLUMNS, fill_empty,
+    UnknownColumnSaveOption
 )
 
 
 __all__ = [
     'OedExposure', 'OedSchema', 'OedSource', 'ModelSettingSchema', 'AnalysisSettingSchema',
     'OdsException', 'PANDAS_COMPRESSION_MAP', 'PANDAS_DEFAULT_NULL_VALUES', 'USUAL_FILE_NAME', 'OED_TYPE_TO_NAME',
-    'OED_NAME_TO_TYPE', 'OED_IDENTIFIER_FIELDS', 'VALIDATOR_ON_ERROR_ACTION', 'DEFAULT_VALIDATION_CONFIG', 'OED_PERIL_COLUMNS',
+    'OED_NAME_TO_TYPE', 'OED_IDENTIFIER_FIELDS', 'VALIDATOR_ON_ERROR_ACTION', 'DEFAULT_VALIDATION_CONFIG', 'OED_PERIL_COLUMNS', 'fill_empty',
+    'UnknownColumnSaveOption'
 ]

ods_tools/oed/common.py

@@ -4,6 +4,8 @@
 from urllib.parse import urlparse
 from pathlib import Path
 import numpy as np
+import pandas as pd
+from enum import Enum
 
 
 class OdsException(Exception):
@@ -87,9 +89,25 @@ def is_relative(filepath):
     {'name': 'occupancy_code', 'on_error': 'raise'},
     {'name': 'construction_code', 'on_error': 'raise'},
     {'name': 'country_and_area_code', 'on_error': 'raise'},
+    {'name': 'conditional_requirement', 'on_error': 'raise'},
 ]
 
 OED_PERIL_COLUMNS = ['AccPeril', 'PolPerilsCovered', 'PolPeril', 'CondPeril', 'LocPerilsCovered', 'LocPeril',
                      'ReinsPeril']
 
-BLANK_VALUES = {np.nan, '', None}
+BLANK_VALUES = {np.nan, '', None, pd.NA, pd.NaT}
+
+
+def fill_empty(df, columns, value):
+    if isinstance(columns, str):
+        columns = [columns]
+    for column in columns:
+        if df[column].dtypes.name == 'category' and value not in {None, np.nan}.union(df[column].cat.categories):
+            df[column] = df[column].cat.add_categories(value)
+        df.loc[df[column].isin(BLANK_VALUES), column] = value
+
+
+class UnknownColumnSaveOption(Enum):
+    IGNORE = 1
+    RENAME = 2
+    DELETE = 3

ods_tools/oed/exposure.py

@@ -9,7 +9,8 @@
 from pathlib import Path
 
 from .common import (PANDAS_COMPRESSION_MAP,
-                     USUAL_FILE_NAME, OED_TYPE_TO_NAME)
+                     USUAL_FILE_NAME, OED_TYPE_TO_NAME,
+                     UnknownColumnSaveOption)
 from .oed_schema import OedSchema
 from .source import OedSource
 from .validator import Validator
@@ -165,7 +166,7 @@ def save_config(self, filepath):
         with open(filepath, 'w') as info_file:
             json.dump(self.info, info_file, indent='  ', default=str)
 
-    def save(self, path, version_name=None, compression=None, save_config=False):
+    def save(self, path, version_name=None, compression=None, save_config=False, unknown_columns=UnknownColumnSaveOption.IGNORE):
         """
         helper function to save all OED data to a location with specific compression
         Args:
@@ -205,7 +206,8 @@ def save(self, path, version_name=None, compression=None, save_config=False):
             filepath = filepath.with_suffix(PANDAS_COMPRESSION_MAP[compression])
 
             oed_source.save(saved_version_name + '_' + f'{compression}',
-                            {'source_type': 'filepath', 'filepath': filepath, 'extension': compression})
+                            {'source_type': 'filepath', 'filepath': filepath, 'extension': compression},
+                            unknown_columns=unknown_columns)
         if save_config:
             self.save_config(Path(path, self.DEFAULT_EXPOSURE_CONFIG_NAME))
 

ods_tools/oed/oed_schema.py

@@ -1,9 +1,8 @@
 import json
 import os
 from pathlib import Path
-import pandas as pd
 
-from .common import OdsException
+from .common import OdsException, BLANK_VALUES
 
 
 class OedSchema:
@@ -164,7 +163,7 @@ def is_valid_value(value, valid_ranges, allow_blanks):
             True if value is in one of the range
         """
 
-        if pd.isna(value):
+        if value in BLANK_VALUES:
             return allow_blanks
         for valid_range in valid_ranges:
             if valid_range.get('min') is not None:

ods_tools/oed/setting_schema.py

@@ -105,7 +105,7 @@ def _remap_key(self, obj, key_new, key_old):
             return None  # base case exit
 
         if key_old in obj:
-            obj[key_new] = obj.pop(key_old)
+            obj[key_new] = obj[key_old]
 
         if isinstance(obj, dict):
             for k, v in obj.items():