Skip to content

Commit

Permalink
Adding decoratory factory to validate_schema to make it work both as …
Browse files Browse the repository at this point in the history
…a function and as a decorator
  • Loading branch information
kunaljubce committed Mar 29, 2024
1 parent 4157a3c commit e642b86
Showing 1 changed file with 32 additions and 3 deletions.
35 changes: 32 additions & 3 deletions quinn/dataframe_validator.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,38 @@ def validate_presence_of_columns(df: DataFrame, required_col_names: list[str]) -
error_message = f"The {missing_col_names} columns are not included in the DataFrame with the following columns {all_col_names}"
if missing_col_names:
raise DataFrameMissingColumnError(error_message)


def validate_schema(

Check failure on line 39 in quinn/dataframe_validator.py

View workflow job for this annotation

GitHub Actions / ruff

Ruff (W293)

quinn/dataframe_validator.py:39:1: W293 Blank line contains whitespace
def validate_schema(required_schema: StructType, ignore_nullable=False, _func=None):

Check failure on line 40 in quinn/dataframe_validator.py

View workflow job for this annotation

GitHub Actions / ruff

Ruff (ANN201)

quinn/dataframe_validator.py:40:5: ANN201 Missing return type annotation for public function `validate_schema`

Check failure on line 40 in quinn/dataframe_validator.py

View workflow job for this annotation

GitHub Actions / ruff

Ruff (D103)

quinn/dataframe_validator.py:40:5: D103 Missing docstring in public function

Check failure on line 40 in quinn/dataframe_validator.py

View workflow job for this annotation

GitHub Actions / ruff

Ruff (ANN001)

quinn/dataframe_validator.py:40:50: ANN001 Missing type annotation for function argument `ignore_nullable`

Check failure on line 40 in quinn/dataframe_validator.py

View workflow job for this annotation

GitHub Actions / ruff

Ruff (ANN001)

quinn/dataframe_validator.py:40:73: ANN001 Missing type annotation for function argument `_func`
def decorator(func):

Check failure on line 41 in quinn/dataframe_validator.py

View workflow job for this annotation

GitHub Actions / ruff

Ruff (ANN202)

quinn/dataframe_validator.py:41:9: ANN202 Missing return type annotation for private function `decorator`

Check failure on line 41 in quinn/dataframe_validator.py

View workflow job for this annotation

GitHub Actions / ruff

Ruff (ANN001)

quinn/dataframe_validator.py:41:19: ANN001 Missing type annotation for function argument `func`
def wrapper(*args, **kwargs):

Check failure on line 42 in quinn/dataframe_validator.py

View workflow job for this annotation

GitHub Actions / ruff

Ruff (ANN202)

quinn/dataframe_validator.py:42:13: ANN202 Missing return type annotation for private function `wrapper`

Check failure on line 42 in quinn/dataframe_validator.py

View workflow job for this annotation

GitHub Actions / ruff

Ruff (ANN002)

quinn/dataframe_validator.py:42:22: ANN002 Missing type annotation for `*args`

Check failure on line 42 in quinn/dataframe_validator.py

View workflow job for this annotation

GitHub Actions / ruff

Ruff (ANN003)

quinn/dataframe_validator.py:42:30: ANN003 Missing type annotation for `**kwargs`
df = func(*args, **kwargs)
_all_struct_fields = copy.deepcopy(df.schema)
_required_schema = copy.deepcopy(required_schema)

if ignore_nullable:
for x in _all_struct_fields:
x.nullable = None

for x in _required_schema:
x.nullable = None

missing_struct_fields = [x for x in _required_schema if x not in _all_struct_fields]
error_message = f"The {missing_struct_fields} StructFields are not included in the DataFrame with the following StructFields {_all_struct_fields}"

if missing_struct_fields:
raise DataFrameMissingStructFieldError(error_message)
return df
return wrapper

if _func is None:
# This means the function is being used as a decorator
return decorator
else:
# This means the function is being called directly with a DataFrame
return decorator(lambda: _func)()


def x_validate_schema(
df: DataFrame,
required_schema: StructType,
ignore_nullable: bool = False,
Expand Down

0 comments on commit e642b86

Please sign in to comment.