Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validation crash after converting account file from csv to parquet #58

Closed
sambles opened this issue Sep 7, 2023 · 0 comments · Fixed by #59
Closed

Validation crash after converting account file from csv to parquet #58

sambles opened this issue Sep 7, 2023 · 0 comments · Fixed by #59
Labels
bug Something isn't working
Milestone

Comments

@sambles
Copy link
Contributor

sambles commented Sep 7, 2023

Steps to Reproduce (Bugs only)

  1. $ git clone [email protected]:OasisLMF/OasisAzureModels.git
  2. $ cd OasisAzureModels/test_data
  3. $ ods_tools check --account PF1_acc.csv --location PF1_loc_CAFL.csv (passes validation)
  4. $ ods_tools convert --account PF1_acc.csv -c parquet --output-dir . (works) creating a new file PF1_acc.parquet
  5. $ ods_tools check --account PF1_acc.parquet --location PF1_loc_CAFL.csv (fails with trace below)

Version / Environment information

ods_tools==3.1.0

Example data / logs

$ ods_tools check --account PF1_acc.parquet --location PF1_loc_CAFL.csv
Traceback (most recent call last):
  File "/home/sam/repos/venv/3py/bin/ods_tools", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/sam/repos/venv/3py/lib/python3.11/site-packages/ods_tools/main.py", line 135, in main
    command_action[kwargs.pop('command')](** kwargs)
  File "/home/sam/repos/venv/3py/lib/python3.11/site-packages/ods_tools/main.py", line 49, in check
    oed_exposure.check()
  File "/home/sam/repos/venv/3py/lib/python3.11/site-packages/ods_tools/oed/exposure.py", line 230, in check
    validator = Validator(self)
                ^^^^^^^^^^^^^^^
  File "/home/sam/repos/venv/3py/lib/python3.11/site-packages/ods_tools/oed/validator.py", line 28, in __init__
    self.column_to_field_maps[oed_source] = oed_source.get_column_to_field()
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sam/repos/venv/3py/lib/python3.11/site-packages/ods_tools/oed/source.py", line 390, in get_column_to_field
    self.dataframe.columns,
    ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/functools.py", line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/home/sam/repos/venv/3py/lib/python3.11/site-packages/ods_tools/oed/source.py", line 360, in dataframe
    df = self.load_dataframe()
         ^^^^^^^^^^^^^^^^^^^^^
  File "/home/sam/repos/venv/3py/lib/python3.11/site-packages/ods_tools/oed/source.py", line 416, in load_dataframe
    oed_df = self.as_oed_type(oed_df, column_to_field)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sam/repos/venv/3py/lib/python3.11/site-packages/ods_tools/oed/source.py", line 314, in as_oed_type
    oed_df.loc[oed_df[column].isin(BLANK_VALUES), column] = ''
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sam/repos/venv/3py/lib/python3.11/site-packages/pandas/core/indexing.py", line 849, in __setitem__
    iloc._setitem_with_indexer(indexer, value, self.name)
  File "/home/sam/repos/venv/3py/lib/python3.11/site-packages/pandas/core/indexing.py", line 1835, in _setitem_with_indexer
    self._setitem_with_indexer_split_path(indexer, value, name)
  File "/home/sam/repos/venv/3py/lib/python3.11/site-packages/pandas/core/indexing.py", line 1928, in _setitem_with_indexer_split_path
    self._setitem_single_column(loc, value, pi)
  File "/home/sam/repos/venv/3py/lib/python3.11/site-packages/pandas/core/indexing.py", line 2034, in _setitem_single_column
    self.obj._mgr.column_setitem(loc, plane_indexer, value)
  File "/home/sam/repos/venv/3py/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 1384, in column_setitem
    new_mgr = col_mgr.setitem((idx,), value)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sam/repos/venv/3py/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 394, in setitem
    return self.apply("setitem", indexer=indexer, value=value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sam/repos/venv/3py/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 352, in apply
    applied = getattr(b, f)(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sam/repos/venv/3py/lib/python3.11/site-packages/pandas/core/internals/blocks.py", line 1622, in setitem
    values[indexer] = value
    ~~~~~~^^^^^^^^^
  File "/home/sam/repos/venv/3py/lib/python3.11/site-packages/pandas/core/arrays/_mixins.py", line 250, in __setitem__
    self._ndarray[key] = value
    ~~~~~~~~~~~~~^^^^^
ValueError: assignment destination is read-only

File seems to load ok (outside of ods_tools)

In [1]: import pandas as pd
In [2]: PF1_acc = pd.read_parquet('PF1_acc.parquet')
In [3]: PF1_acc
Out[3]:
      PortNumber AccNumber AccCurrency  AccMaxDed5PD  ... PolPerilsCovered PolLimit6All PolLimitType6All  PolPeril
0      Portfolio     2-999         USD           0.0  ...              WEC  34163.04673                0       WEC
1      Portfolio     2-998         USD           0.0  ...              WEC  34163.04673                0       WEC
2      Portfolio     2-997         USD           0.0  ...              WEC  13665.21870                0       WEC
3      Portfolio     2-996         USD           0.0  ...              WEC  13665.21870                0       WEC
4      Portfolio     2-995         USD           0.0  ...              WEC  68326.09347                0       WEC
...          ...       ...         ...           ...  ...              ...          ...              ...       ...
88137  Portfolio       732         USD           0.0  ...              WEC  33791.29932                0       WEC
88138  Portfolio       398         USD           0.0  ...              WEC  34163.04673                0       WEC
88139  Portfolio       367         USD           0.0  ...              WEC  33791.29932                0       WEC
88140  Portfolio        32         USD           0.0  ...              WEC  34163.04673                0       WEC
88141  Portfolio         1         USD           0.0  ...              WEC  33791.29932                0       WEC

[88142 rows x 10 columns]

@sambles sambles added the bug Something isn't working label Sep 7, 2023
@sambles sambles linked a pull request Sep 12, 2023 that will close this issue
@awsbuild awsbuild modified the milestones: 3.1.1rc1, 3.1.1 Sep 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants