Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

change checking for size of dim/attr/header limit #130

Merged
merged 3 commits into from
Mar 13, 2024
Merged

Conversation

wkliao
Copy link
Member

@wkliao wkliao commented Mar 12, 2024

The maximal header extent for CDF 1 and 2 should be NC_MAX_INT , i,e,
the maximum value of a signed 4-byte integer, 2^31-1. Same for the
dimension size. This is because NetCDF file format defines them as
non-negative signed integers, OFFSET and NON_NEG respectively.

NetCDF file format specification:

netcdf_file  = header data
header       = magic numrecs dim_list gatt_list var_list
dim_list     = ABSENT | NC_DIMENSION nelems [dim ...]
dim          = name dim_length
var          = name nelems [dimid ...] vatt_list nc_type vsize begin
vsize        = NON_NEG
dim_length   = NON_NEG 
NON_NEG      = <non-negative INT>
begin        = OFFSET
OFFSET       = <non-negative INT> | <non-negative INT64>
INT          = <32-bit signed integer, Bigendian, two's complement>
  • begin is the file extent, i.e. starting file offset of the data section.
  • dim_length is the dimension size.

Note for variable size, the maximal is
2^31-3 for CDF-1
2^32-3 for CDF-2
2^63-3 for CDF-5
Variable size is calculated internally in PnetCDF. Therefore, the
variable size is limited to the space available in bytes. -3 is
due to the 4-byte upward alignment.
Although vsize is defined and stored in the file header, it is marked as a redundant item and is not used in PnetCDF.

According to NetCDF CDF 1 and 2 file format specification, the maximal
dimension size should be INT_MAX, i.e. 2^31-1, <non-negative INT>.

NetCDF file format specification:
  netcdf_file  = header data
  header       = magic numrecs dim_list gatt_list var_list
  dim_list     = ABSENT | NC_DIMENSION nelems [dim ...]
  dim          = name dim_length
  dim_length   = NON_NEG
  NON_NEG      = <non-negative INT>
  INT          = <32-bit signed integer, Bigendian, two's complement>

Note for variable size, the maximal is
  2^31-3  for CDF-1
  2^32-3  for CDF-2
  2^63-3  for CDF-5
Variable size is calculated internally in PnetCDF. Therefor, the
variable size is limited to the space available in bytes. -3 is
due to the 4-byte upward alignment.
Also, note vsize defined in the file header format is not used.
@wkliao wkliao changed the title change checking for dimension size and header size limit change checking for size of dim/attr/header limit Mar 13, 2024
@wkliao wkliao merged commit e6a6177 into master Mar 13, 2024
3 of 4 checks passed
@wkliao wkliao deleted the check_2GiB branch March 15, 2024 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant