Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

actual_range checking is not accurate for packed data #109

Open
Melsana opened this issue Jul 25, 2023 · 0 comments
Open

actual_range checking is not accurate for packed data #109

Melsana opened this issue Jul 25, 2023 · 0 comments

Comments

@Melsana
Copy link

Melsana commented Jul 25, 2023

if min_dv and actual_range[0] != min_dv:

I've run into an issue with packed data giving different actual_range values.

I calculate
pressure_msl:actual_range = 989.5993f, 1031.399f ;
in the code when creating the file, but then the checker gives me this error.

------------------
Checking variable: pressure_msl
------------------
ERROR: (2.5.1): First element of actual_range must equal minimum data value of variable after scale_factor/add_offset applied (989.56298828125)
ERROR: (2.5.1): Second element of actual_range must equal maximum data value of variable after scale_factor/add_offset applied (1031.401611328125)

As you can see, if you round these they are the same, but I think due to precision issues with packing and unpacking the data, the actual_range values don't match perfectly. So maybe this should be checked using an epsilon value or something like that. Here is an example where the unpacked values are very small numbers:

vorticity_low:actual_range = -2.90598e-05f, 2.69841e-05f ;
vs

------------------
Checking variable: vorticity_low
------------------
ERROR: (2.5.1): First element of actual_range must equal minimum data value of variable after scale_factor/add_offset applied (-2.9181901481933892e-05)
ERROR: (2.5.1): Second element of actual_range must equal maximum data value of variable after scale_factor/add_offset applied (2.7350401069270447e-05)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant