diff --git a/doxygen/examples/FileImageOps.html b/doxygen/examples/FileImageOps.html index fda708cb54d..c8b07ee2f49 100644 --- a/doxygen/examples/FileImageOps.html +++ b/doxygen/examples/FileImageOps.html @@ -10,68 +10,106 @@ -

1. Introduction to HDF5 File Image Operations

- -

2. C API Call Syntax

-

2.1. Low-level C API Routines

- -

2.2. High-level C API Routine

- -

3. C API Call Semantics

-

3.1. File Image Callback Semantics

- -

3.2. Initial File Image Semantics

- -

4. Examples

-

4.1. Reading an In-memory HDF5 File Image
- 4.2. In-memory HDF5 File Image Construction
- 4.3. Using HDF5 to Construct and Read a Data Packet
- 4.4. Using a Template File

-

5. Java Signatures for File Image Operations API Calls

-

6. Fortran Signatures for File Image Operations API Calls

-

6.1. Low-level Fortran API Routines

- -

6.2. High-level Fortran API Routine

+
  • 1. Introduction to HDF5 File Image Operations
  • + + +
  • 2. C API Call Syntax
  • + +
  • 3. C API Call Semantics
  • + + +
  • 4. Examples
  • + + +
  • 5. Java Signatures for File Image Operations API Calls
  • +
  • 6. Fortran Signatures for File Image Operations API Calls
  • + + +

    1. Introduction to HDF5 File Image Operations

    +

    File image operations allow users to work with HDF5 files in memory in the same ways that users currently work with HDF5 files on disk. Disk I/O is not required when file images are opened, created, read from, or written to.

    An HDF5 file image is an HDF5 file that is held in a buffer in main memory. Setting up a file image in memory involves using either a buffer in the file access property list or a buffer in the Core (aka Memory) file driver.

    The advantage of working with a file in memory is faster access to the data.

    @@ -80,11 +118,11 @@

    1. Introduction to HDF5 File Image Operations

    1.1. File Image Operations Function Summary

    Functions used in file image operations are listed below.

    - +
    - - - + + + @@ -120,11 +158,10 @@

    1.1. File Image Operations Funct

    1.2. Abbreviations

    The following abbreviations are used in this document:

    -
    C FunctionPurposeSectionC FunctionPurposeSection
    H5Pset_file_image
    - +
    Table 1. Abbreviations
    - + @@ -147,14 +184,14 @@

    1.4. Resources

    See the following for more information.

    The “RFC: File Image Operations” is the primary source for the information in this document.

    The “Alternate File Storage Layouts and Low-level File Drivers” section is in “The HDF5 File” chapter of the HDF5 User’s Guide .

    -

    The H5P_SET_FAPL_CORE function call can be used to modify the file access property list so that the Memory virtual file driver, H5FD_ CORE, is used. The Memory file driver is also known as the Core file driver.

    +

    The H5P_SET_FAPL_CORE function call can be used to modify the file access property list so that the Memory virtual file driver, H5FD_CORE, is used. The Memory file driver is also known as the Core file driver.

    Refer to the Virtual File Layer for more detail. A list of VFL Functions is provided below.

    AbbreviationExplanationExplanation
    FAPL or fapl
    - + @@ -359,12 +396,12 @@

    2.1.3. H5Pset_file_image_callbacks

  • udata contains a pointer value, potentially to user-defined data, that will be passed to the image_malloc, image_memcpy, image_realloc, and image_free callbacks.
  • -The semantics of the values that can be set for the file_image_op parameter to the above callbacks are described in the table below: +The semantics of the values that can be set for the file_image_op parameter to the above callbacks are described in the table below:

    C FunctionPurposePurpose
    H5Pset_driver
    - + - - + + @@ -409,23 +446,24 @@

    2.1.4. H5Pget_file_image_callbacks

    The parameters of H5Pget_file_image_callbacks are defined as follows:

    + Upon successful return, the fields of *callbacks_ptr shall contain values as defined below:

    2.1.5. Virtual File Driver Feature Flags

    -Implementation of the H5Pget/set_file_image_callbacks() and H5Pget/set_file_image() function calls requires a pair of virtual file driver feature flags. The flags are H5FD_FEAT_ALLOW_FILE_IMAGE and H5FD_FEAT_CAN_USE_FILE_IMAGE_CALLBACKS. Both of these are defined in H5FDpublic.h.

    +Implementation of the H5Pget/set_file_image_callbacks() and H5Pget/set_file_image() function calls requires a pair of virtual file driver feature flags. The flags are H5FD_FEAT_ALLOW_FILE_IMAGE and H5FD_FEAT_CAN_USE_FILE_IMAGE_CALLBACKS. Both of these are defined in H5FDpublic.h.

    The first flag, H5FD_FEAT_ALLOW_FILE_IMAGE, allows a file driver to indicate whether or not it supports file images. A VFD that sets this flag when its ‘query’ callback is invoked indicates that the file image set in the FAPL will be used as the initial contents of a file. Support for setting an initial file image is designed primarily for use with the Core VFD. However, any VFD can indicate support for this feature by setting the flag and copying the image in an appropriate way for the VFD (possibly by writing the image to a file and then opening the file). However, such a VFD need not employ the file image after file open time. In such cases, the VFD will not make an in-memory copy of the file image and will not employ the file image callbacks.

    @@ -448,55 +486,88 @@

    2.1.6. H5Fget_file_image

    The signature of H5Fget_file_image is defined as follows:

    ssize_t H5Fget_file_image(hid_t file_id, void *buf_ptr, size_t buf_len)

    -The parameters of H5Fget_file_image are defined as follows:

    +The parameters of H5Fget_file_image are defined as follows: -

    file_id contains the ID of the target file. -buf_ptr contains a pointer to the buffer into which the image of the HDF5 file is to be copied. If buf_ptr is NULL, no data will be copied, but the return value will still indicate the buffer size required (or a negative value on error). -buf_len contains the size of the supplied buffer. -If the return value of H5Fget_file_image is a positive value, then the value will be the length of buffer required to store the file image (in other words, the length of the file). A negative value might be returned if the file is too large to store in the supplied buffer or on failure.

    + +

    If the return value of H5Fget_file_image is a positive value, then the value will be the length of buffer required to store the file image (in other words, the length of the file). A negative value might be returned if the file is too large to store in the supplied buffer or on failure.

    +

    The current file size can be obtained via a call to H5Fget_filesize(). Note that this function returns the value of the end of file (EOF) and not the end of address space (EOA). While these values are frequently the same, it is possible for the EOF to be larger than the EOA. Since H5Fget_file_image() will only obtain a copy of the file from the beginning of the superblock to the EOA, it will be best to use H5Fget_file_image() to determine the size of the buffer required to contain the image.

    -

    Other Design Considerations

    + +Other Design Considerations +

    Here are some other notes regarding the design and implementation of H5Fget_file_image.

    The H5Fget_file_image call should be part of the high-level library. However, a file driver agnostic implementation of the routine requires access to data structures that are hidden within the HDF5 Library. We chose to implement the call in the library proper rather than expose those data structures.

    There is no reason why the H5Fget_file_image() API call could not work on files opened with any file driver. However, the Family, Multi, and Split file drivers have issues that make the call problematic. At present, files opened with the Family file driver are marked as being created with that file driver in the superblock, and the HDF5 Library refuses to open files so marked with any other file driver. This negates the purpose of the H5Fget_file_image() call. While this mark can be removed from the image, the necessary code is not trivial.

    Thus we will not support the Family file driver in H5Fget_file_image() unless there is demand for it. Files created with the Multi and Split file drivers are also marked in the superblock. In addition, they typically use a very sparse address space. A sparse address space would require the use of an impractically large buffer for an image, and most of the buffer would be empty. So, we see no point in supporting the Multi and Split file drivers in H5Fget_file_image() under any foreseeable circumstances.

    2.2. High-level C API Routine

    +

    The H5LTopen_file_image high-level routine encapsulates the capabilities of routines in the main HDF5 Library with conveniently accessible abstractions.

    2.2.1. H5LTopen_file_image

    +

    The H5LTopen_file_image routine is designed to provide an easier way to open an initial file image with the Core VFD. Flags to H5LTopen_file_image allow for various file image buffer ownership policies to be requested. See the HDF5 Reference Manual for more information on high-level APIs.

    The signature of H5LTopen_file_image is defined as follows:

    +

    hid_t H5LTopen_file_image(void *buf_ptr, size_t buf_len, unsigned flags) -The parameters of H5LTopen_file_image are defined as follows:

    -

    buf_ptr contains a pointer to the supplied initial image. A NULL value is invalid and will cause H5LTopen_file_image to fail. -buf_len contains the size of the supplied buffer. A value of 0 is invalid and will cause H5LTopen_file_image to fail. -flags contains a set of flags indicating whether the image is to be opened read/write, whether HDF5 is to take control of the buffer, and how long the application promises to maintain the buffer. Possible flags are described in the table below: -Table 3. Flags for H5LTopen_file_image

    -

    H5LT_FILE_IMAGE_OPEN_RW Indicates that the HDF5 Library should open the image read/write instead of the default read-only. -H5LT_FILE_IMAGE_DONT_COPY -Indicates that the HDF5 Library should not copy the file image buffer provided, but should use it directly. The HDF5 Library will release the file image when finished. The supplied buffer must have been allocated via a call to the standard C library malloc() or calloc() routines. The HDF5 Library will call free() to release the buffer. In the absence of this flag, the HDF5 Library will copy the buffer provided. The H5LT_FILE_IMAGE_DONT_COPY flag provides an application with the ability to “give ownership” of a file image buffer to the HDF5 Library.

    -

    The HDF5 Library will modify the buffer on write if the image is opened read/write and the H5LT_FILE_IMAGE_DONT_COPY flag is set.

    -

    The H5LT_FILE_IMAGE_DONT_RELEASE flag, see below, is invalid unless the H5LT_FILE_IMAGE_DONT_COPY flag is set

    -

    H5LT_FILE_IMAGE_DONT_RELEASE -Indicates that the HDF5 Library should not attempt to release the buffer when the file is closed. This implies that the application will tend to this detail and that the application will not discard the buffer until after the file image is closed.

    -

    Since there is no way to return a changed buffer base address to the application, and since realloc can change this value, calls to realloc() must be barred when this flag is set. As a result, any write that requires an increased buffer size will fail.

    -

    This flag is invalid unless the H5LT_FILE_IMAGE_DONT_COPY flag, see above, is set.

    -

    If the H5LT_FILE_IMAGE_DONT_COPY flag is set and this flag is not set, the HDF5 Library will release the file image buffer after the file is closed using the standard C library free() routine.

    -

    Using this flag and the H5LT_FILE_IMAGE_DONT_COPY flag provides a way for the application to specify a buffer that the HDF5 Library can use for opening and accessing as a file image while letting the application retain ownership of the buffer.

    + +

    The parameters of H5LTopen_file_image are defined as follows:

    + + +
    Table 2. Values for the file_image_op parameterTable 2. Values for the file_image_op parameter
    ValueCommentsValueComments
    H5_FILE_IMAGE_OP_PROPERTY_LIST_SET
    + + + + + + + + + + + + + + + + + +
    Table 3. Flags for H5LTopen_file_image
    ValueComments
    H5LT_FILE_IMAGE_OPEN_RWIndicates that the HDF5 Library should open the image read/write instead of the default read-only.
    H5LT_FILE_IMAGE_DONT_COPY +
      +
    • Indicates that the HDF5 Library should not copy the file image buffer provided, but should use it directly. The HDF5 Library will release the file image when finished. The supplied buffer must have been allocated via a call to the standard C library malloc() or calloc() routines. The HDF5 Library will call free() to release the buffer. In the absence of this flag, the HDF5 Library will copy the buffer provided. The H5LT_FILE_IMAGE_DONT_COPY flag provides an application with the ability to “give ownership” of a file image buffer to the HDF5 Library.
    • +
    • The HDF5 Library will modify the buffer on write if the image is opened read/write and the H5LT_FILE_IMAGE_DONT_COPY flag is set.
    • +
    • The H5LT_FILE_IMAGE_DONT_RELEASE flag, see below, is invalid unless the H5LT_FILE_IMAGE_DONT_COPY flag is set
    • +
    +
    H5LT_FILE_IMAGE_DONT_RELEASE +
      +
    • Indicates that the HDF5 Library should not attempt to release the buffer when the file is closed. This implies that the application will tend to this detail and that the application will not discard the buffer until after the file image is closed.
    • +
    • Since there is no way to return a changed buffer base address to the application, and since realloc can change this value, calls to realloc() must be barred when this flag is set. As a result, any write that requires an increased buffer size will fail.
    • +
    • This flag is invalid unless the H5LT_FILE_IMAGE_DONT_COPY flag, see above, is set.
    • +
    • If the H5LT_FILE_IMAGE_DONT_COPY flag is set and this flag is not set, the HDF5 Library will release the file image buffer after the file is closed using the standard C library free() routine.
    • +
    • Using this flag and the H5LT_FILE_IMAGE_DONT_COPY flag provides a way for the application to specify a buffer that the HDF5 Library can use for opening and accessing as a file image while letting the application retain ownership of the buffer.
    • +
    +
    +

    The following table is intended to summarize the semantics of the H5LT_FILE_IMAGE_DONT_COPY and H5LT_FILE_IMAGE_DONT_RELEASE flags (shown as “Don’t Copy Flag” and “Don’t Release Flag” respectively in the table):

    - - +
    Table 4. Summary of Don’t Copy and Don’t Release Flag Actions
    + - - - - - - + + + + + + @@ -627,7 +698,7 @@

    4. Examples

    4.1. Reading an In-memory HDF5 File Image

    -

    The H5Pset_file_image() function call allows the Core file driver to be initialized from an application provided buffer. The following pseudo code illustrates its use: +

    The H5Pset_file_image() function call allows the Core file driver to be initialized from an application provided buffer. The following pseudo code illustrates its use:

    
     <allocate and initialize buf_len and buf>
    @@ -640,7 +711,8 @@ 

    4.1. Reading an In-memory HDF5 Fil <read and/or write file as desired, close>

    -Example 2. Using H5Pset_file_image to initialize the Core file driver

    +Example 2. Using H5Pset_file_image to initialize the Core file driver

    +

    This solution is easy to code, but the supplied buffer is duplicated twice. The first time is in the call to H5Pset_file_image() when the image is duplicated and the duplicate inserted into the property list. The second time is when the file is opened: the image is copied from the property list into the initial buffer allocated by the Core file driver. This is a non-issue for small images, but this could become a significant performance hit for large images.

    If we want to avoid the extra malloc and memcpycalls, we must decide whether the application should retain ownership of the buffer or pass ownership to the HDF5 Library.

    The following pseudo code illustrates opening the image read -only using the H5LTopen_file_image() routine. In this example, the application retains ownership of the buffer and avoids extra buffer allocations and memcpy calls.

    @@ -654,8 +726,9 @@

    4.1. Reading an In-memory HDF5 Fil <discard buf any time after this point> -Example 3. Using H5LTopen_file_image to open a read-only file image where the application retains ownership of the buffer -If the application wants to transfer ownership of the buffer to the HDF5 Library, and the standard C library routine free is an acceptable way of discarding it, the above example can be modified as follows:

    +Example 3. Using H5LTopen_file_image to open a read-only file image where the application retains ownership of the buffer + +

    If the application wants to transfer ownership of the buffer to the HDF5 Library, and the standard C library routine free is an acceptable way of discarding it, the above example can be modified as follows:

    
     <allocate and initialize buf_len and buf>
    @@ -665,10 +738,12 @@ 

    4.1. Reading an In-memory HDF5 Fil <read file as desired, and then close>

    -

    Example 4. Using H5LTopen_file_image to open a read-only file image where the application transfers ownership of the buffer -Again, file access is read-only. Read/write access can be obtained via the H5LTopen_file_image() call, but we will explore that in the section below.

    +Example 4. Using H5LTopen_file_image to open a read-only file image where the application transfers ownership of the buffer + +

    Again, file access is read-only. Read/write access can be obtained via the H5LTopen_file_image() call, but we will explore that in the section below.

    4.2. In-memory HDF5 File Image Construction

    +

    Before the implementation of file image operations, HDF5 supported construction of an image of an HDF5 file in memory with the Core file driver. The H5Fget_file_image() function call allows an application access to the file image without first writing it to disk. See the following code fragment:

    
    @@ -679,50 +754,57 @@ 

    4.2. In-memory HDF5 File Image C H5Fget_file_image(fid, buffer_ptr, size);

    -Example 5. Accessing the image of a file in memory +Example 5. Accessing the image of a file in memory

    The use of H5Fget_file_image() may be acceptable for small images. For large images, the cost of the malloc() and memcpy() operations may be excessive. To address this issue, the H5Pset_file_image_callbacks() call allows an application to manage dynamic memory allocation for file images and memory-based file drivers (only the Core file driver at present). The following code fragment illustrates its use. Note that most error checking is omitted for simplicity and that H5Pset_file_image is not used to set the initial file image.

    
    -struct udata_t {
    -void * image_ptr;
    -size_t image_size;
    +    struct udata_t {
    +    void * image_ptr;
    +    size_t image_size;
               } udata = {NULL, 0};
    +
     void *image_malloc(size_t size, H5_file_image_op_t file_image_op, void *udata)
     {
    -          ((struct udata_t *)udata)->image_size = size;
    -          return(malloc(size));
    +    ((struct udata_t *)udata)->image_size = size;
    +    return(malloc(size));
     }
    +
     void *image_memcpy)(void *dest, const void *src, size_t size,
      H5_file_image_op_t file_image_op, void *udata)
     {
    -assert(FALSE); /* Should never be invoked in this scenario. */
    -          return(NULL); /* always fails */
    -          }
    -void image_realloc(void *ptr, size_t size, H5_file_image_op_t file_image_op,
    -void *udata)
    +    assert(FALSE); /* Should never be invoked in this scenario. */
    +    return(NULL); /* always fails */
    +}
    +
    +void image_realloc(void *ptr, size_t size, H5_file_image_op_t file_image_op, void *udata)
     {
    -          ((struct udata_t *)udata)->image_size = size;
    -          return(realloc(ptr, size));
    +    ((struct udata_t *)udata)->image_size = size;
    +    return(realloc(ptr, size));
     }
    +
     herr_t image_free(void *ptr, H5_file_image_op_t file_image_op, void *udata)
     {
    -          assert(file_image_op == H5_FILE_IMAGE_OP_FILE_CLOSE);
    -          ((struct udata_t *)udata)->image_ptr = ptr;
    -          return(0); /* if we get here, we must have been successful */
    +    assert(file_image_op == H5_FILE_IMAGE_OP_FILE_CLOSE);
    +    ((struct udata_t *)udata)->image_ptr = ptr;
    +    return(0); /* if we get here, we must have been successful */
     }
    +
     void *udata_copy(void *udata)
     {
    -          return(udata);
    +    return(udata);
     }
    +
     herr_t udata_free(void *udata)
     {
    -          return(0);
    +    return(0);
     }
    +
     H5_file_image_callbacks_t callbacks = {image_malloc, image_memcpy,
                                            image_realloc, image_free,
                                            udata_copy, udata_free,
                                            (void *)(&udata)};
    +
     <allocate fapl_id>
     H5Pset_file_image_callbacks(fapl_id, &callbacks);
     <open core file using fapl_id, write file, close it>
    @@ -731,7 +813,7 @@ 

    4.2. In-memory HDF5 File Image C <use image of file, and then discard it via free()>

    -Example 6. Using H5Pset_file_image_callbacks to improve memory allocation

    +Example 6. Using H5Pset_file_image_callbacks to improve memory allocation

    The above code fragment gives the application full ownership of the buffer used by the Core file driver after the file is closed, and it notifies the application that the HDF5 Library is done with the buffer by setting udata.image_ptr to something other than NULL. If read access to the buffer is sufficient, the H5Fget_vfd_handle() call can be used as an alternate solution to get access to the base address of the Core file driver’s buffer.

    The above solution avoids some unnecessary mallocand memcpycalls and should be quite adequate if an image of an HDF5 file is constructed only occasionally. However, if an HDF5 file image must be constructed regularly, and if we can put a strong and tight upper bound on the size of the necessary buffer, then the following pseudo code demonstrates a method of avoiding memory allocation completely. The downside, however, is that buffer is allocated statically. Again, much error checking is omitted for clarity.

    @@ -744,43 +826,43 @@

    4.2. In-memory HDF5 File Image C size_t max_image_size; int ref_count; } udata = {(void *)(&(buf[0]), 0, BIG_ENOUGH, 0}; + void *image_malloc(size_t size, H5_file_image_op_t file_image_op, void *udata) { - assert(size <= ((struct udata_t *)udata)->max_image_size); - assert(((struct udata_t *)udata)->ref_count == 0); - ((struct udata_t *)udata)->image_size = size; - (((struct udata_t *)udata)->ref_count)++; - return((((struct udata_t *)udata)->image_ptr); + assert(size <= ((struct udata_t *)udata)->max_image_size); + assert(((struct udata_t *)udata)->ref_count == 0); + ((struct udata_t *)udata)->image_size = size; + (((struct udata_t *)udata)->ref_count)++; + return((((struct udata_t *)udata)->image_ptr); } -void *image_memcpy)(void *dest, const void *src, size_t size, - H5_file_image_op_t file_image_op, void *udata) +void *image_memcpy)(void *dest, const void *src, size_t size, H5_file_image_op_t file_image_op, void *udata) { -assert(FALSE); /* Should never be invoked in this scenario. */ - return(NULL); /* always fails */ - } + assert(FALSE); /* Should never be invoked in this scenario. */ + return(NULL); /* always fails */ +} void *image_realloc(void *ptr, size_t size, H5_file_image_op_t file_image_op, void *udata) { - assert(ptr == ((struct udata_t *)udata)->image_ptr); -assert(size <= ((struct udata_t *)udata)->max_image_size); -assert(((struct udata_t *)udata)->ref_count == 1); - ((struct udata_t *)udata)->image_size = size; -return((((struct udata_t *)udata)->image_ptr); + assert(ptr == ((struct udata_t *)udata)->image_ptr); + assert(size <= ((struct udata_t *)udata)->max_image_size); + assert(((struct udata_t *)udata)->ref_count == 1); + ((struct udata_t *)udata)->image_size = size; + return((((struct udata_t *)udata)->image_ptr); } herr_t image_free(void *ptr, H5_file_image_op_t file_image_op, void *udata) { - assert(file_image_op == H5_FILE_IMAGE_OP_FILE_CLOSE); - assert(ptr == ((struct udata_t *)udata)->image_ptr); -assert(((struct udata_t *)udata)->ref_count == 1); - (((struct udata_t *)udata)->ref_count)--; - return(0); /* if we get here, we must have been successful */ + assert(file_image_op == H5_FILE_IMAGE_OP_FILE_CLOSE); + assert(ptr == ((struct udata_t *)udata)->image_ptr); + assert(((struct udata_t *)udata)->ref_count == 1); + (((struct udata_t *)udata)->ref_count)--; + return(0); /* if we get here, we must have been successful */ } void *udata_copy(void *udata) { - return(udata); + return(udata); } herr_t udata_free(void *udata) { - return(0); + return(0); } H5_file_image_callbacks_t callbacks = {image_malloc, image_memcpy, image_realloc, image_free, @@ -790,11 +872,11 @@

    4.2. In-memory HDF5 File Image C <allocate fapl_id> H5Pset_file_image_callbacks(fapl_id, &callbacks); <open core file using fapl_id> - +<discard fapl any time after the open> <write the file, flush it, and then close it> assert(udata.ref_count == 0); /* udata now contains the base address and length of the final version of the core file */ - +<use the image of the file> <reinitialize udata, and repeat the above from the end of initialization onwards to write a new file image> @@ -803,60 +885,60 @@

    4.2. In-memory HDF5 File Image C

    If we can further arrange matters so that only the contents of the datasets in the HDF5 file image change, but not the structure of the file itself, we can optimize still further by re-using the image and changing only the contents of the datasets after the initial write to the buffer. The following pseudo code shows how this might be done. Note that the code assumes that buf already contains the image of the HDF5 file whose dataset contents are to be overwritten. Again, much error checking is omitted for clarity. Also, observe that the file image callbacks do not support the H5Pget_file_image() call.

    
    -
    -
    +<buf already defined and loaded with file image>
    +<udata already defined and initialized>
     void *image_malloc(size_t size, H5_file_image_op_t file_image_op, void *udata)
     {
    -          assert(size <= ((struct udata_t *)udata)->max_image_size);
    -          assert(size == ((struct udata_t *)udata)->image_size);
    -          assert(((struct udata_t *)udata)->ref_count >= 0);
    +    assert(size <= ((struct udata_t *)udata)->max_image_size);
    +    assert(size == ((struct udata_t *)udata)->image_size);
    +    assert(((struct udata_t *)udata)->ref_count >= 0);
               ((struct udata_t *)udata)->image_size = size;
               (((struct udata_t *)udata)->ref_count)++;
    -          return((((struct udata_t *)udata)->image_ptr);
    +    return((((struct udata_t *)udata)->image_ptr);
     }
     void *image_memcpy)(void *dest, const void *src, size_t size, H5_file_image_op_t file_image_op, void *udata)
     {
    -assert(dest == ((struct udata_t *)udata)->image_ptr);
    -assert(src == ((struct udata_t *)udata)->image_ptr);
    -assert(size <= ((struct udata_t *)udata)->max_image_size);
    -assert(size == ((struct udata_t *)udata)->image_size);
    -assert(((struct udata_t *)udata)->ref_count >= 1);
    -         return(dest); /* if we get here, we must have been successful */
    +    assert(dest == ((struct udata_t *)udata)->image_ptr);
    +    assert(src == ((struct udata_t *)udata)->image_ptr);
    +    assert(size <= ((struct udata_t *)udata)->max_image_size);
    +    assert(size == ((struct udata_t *)udata)->image_size);
    +    assert(((struct udata_t *)udata)->ref_count >= 1);
    +    return(dest); /* if we get here, we must have been successful */
     }
     void *image_realloc(void *ptr, size_t size, H5_file_image_op_t file_image_op, void *udata)
     {
    -         /* One would think that this function is not needed in this scenario, as
    -          *  only the contents of the HDF5 file is being changed, not its size or
    -          *  structure. However, the Core file driver calls realloc() just before
    -          *  close to clip the buffer to the size indicated by the end of the
    -          *  address space.
    -          *
    -          *  While this call must be supported in this case, the size of
    -          *  the image should never change. Hence the function can limit itself
    -          *  to performing sanity checks, and returning the base address of the
    -          *  statically allocated buffer.
    -          */
    -          assert(ptr == ((struct udata_t *)udata)->image_ptr);
    -assert(size <= ((struct udata_t *)udata)->max_image_size);
    -assert(((struct udata_t *)udata)->ref_count >= 1);
    -          assert(((struct udata_t *)udata)->image_size == size);
    -return((((struct udata_t *)udata)->image_ptr);
    +    /* One would think that this function is not needed in this scenario, as
    +     *  only the contents of the HDF5 file is being changed, not its size or
    +     *  structure. However, the Core file driver calls realloc() just before
    +     *  close to clip the buffer to the size indicated by the end of the
    +     *  address space.
    +     *
    +     *  While this call must be supported in this case, the size of
    +     *  the image should never change. Hence the function can limit itself
    +     *  to performing sanity checks, and returning the base address of the
    +     *  statically allocated buffer.
    +     */
    +    assert(ptr == ((struct udata_t *)udata)->image_ptr);
    +    assert(size <= ((struct udata_t *)udata)->max_image_size);
    +    assert(((struct udata_t *)udata)->ref_count >= 1);
    +    assert(((struct udata_t *)udata)->image_size == size);
    +    return((((struct udata_t *)udata)->image_ptr);
     }
     herr_t image_free(void *ptr, H5_file_image_op_t file_image_op, void *udata)
     {
    -         assert((file_image_op == H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE) ||
    -                   (file_image_op == H5_FILE_IMAGE_OP_FILE_CLOSE));
    -assert(((struct udata_t *)udata)->ref_count >= 1);
    +    assert((file_image_op == H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE) ||
    +           (file_image_op == H5_FILE_IMAGE_OP_FILE_CLOSE));
    +    assert(((struct udata_t *)udata)->ref_count >= 1);
               (((struct udata_t *)udata)->ref_count)--;
    -          return(0); /* if we get here, we must have been successful */
    +    return(0); /* if we get here, we must have been successful */
     }
     void *udata_copy(void *udata)
     {
    -          return(udata);
    +    return(udata);
     }
     herr_t udata_free(void *udata)
     {
    -          return(0);
    +    return(0);
     }
     H5_file_image_callbacks_t callbacks = {image_malloc, image_memcpy,
                                            image_realloc, image_free,
    @@ -892,124 +974,124 @@ 

    4.2. In-memory HDF5 File Image C <repeat the above from the end of initialization onwards to write new data to datasets in file image>

    -

    Example 9. Using H5LTopen_file_image where only the datasets change -The above pseudo code allows updates of a file image about as cheaply as possible. We assume the application has enough RAM for the image and that the HDF5 file structure is constant after the first write.

    +

    The above pseudo code allows updates of a file image about as cheaply as possible. We assume the application has enough RAM for the image and that the HDF5 file structure is constant after the first write.

    While the scenario above is plausible, we will finish this section with a more general scenario. In the pseudo code below, we assume sufficient RAM to retain the HDF5 file image between uses, but we do not assume that the HDF5 file structure remains constant or that we can place a hard pper bound on the image size.

    Since we must use malloc, realloc, and free in this example, and since realloc can change the base address of a buffer, we must maintain two of ptr, size, and ref_count triples in the udata structure. The first triple is for the property list (which will never change the buffer), and the second triple is for the file driver. As shall be seen, this complicates the file image callbacks considerably. Note also that while we do not use H5Pget_file_image() in this example, we do include support for it in the file image callbacks. As usual, much error checking is omitted in favor of clarity.

    
    -

    struct udata_t { -void * fapl_image_ptr; -size_t fapl_image_size; -int fapl_ref_count; -void * vfd_image_ptr; -size_t vfd_image_size; -nt vfd_ref_count; -} udata = {NULL, 0, 0, NULL, 0, 0}; +struct udata_t { + void * fapl_image_ptr; + size_t fapl_image_size; + int fapl_ref_count; + void * vfd_image_ptr; + size_t vfd_image_size; + nt vfd_ref_count; + } udata = {NULL, 0, 0, NULL, 0, 0}; boolean initial_file_open = TRUE; + void *image_malloc(size_t size, H5_file_image_op_t file_image_op, void *udata) { - void * return_value = NULL; - switch ( file_image_op ) { - case H5_FILE_IMAGE_OP_PROPERTY_LIST_SET: - case H5_FILE_IMAGE_OP_PROPERTY_LIST_COPY: - assert(((struct udata_t *)udata)->fapl_image_ptr != NULL); - assert(((struct udata_t *)udata)->fapl_image_size == size); - assert(((struct udata_t *)udata)->fapl_ref_count >= 0); - return_value = ((struct udata_t *)udata)->fapl_image_ptr; - (((struct udata_t *)udata)->fapl_ref_count)++; - break; - case H5_FILE_IMAGE_OP_PROPERTY_LIST_GET: - assert(((struct udata_t *)udata)->fapl_image_ptr != NULL); - assert(((struct udata_t *)udata)->vfd_image_size == size); - assert(((struct udata_t *)udata)->fapl_ref_count >= 1); - return_value = ((struct udata_t *)udata)->fapl_image_ptr; - /* don’t increment ref count */ - break; - case H5_FILE_IMAGE_OP_FILE_OPEN: - assert(((struct udata_t *)udata)->vfd_image_ptr == NULL); - assert(((struct udata_t *)udata)->vfd_image_size == 0); - assert(((struct udata_t *)udata)->vfd_ref_count == 0); -if (((struct udata_t *)udata)->fapl_image_ptr == NULL ) { - ((struct udata_t *)udata)->vfd_image_ptr = -malloc(size); - ((struct udata_t *)udata)->vfd_image_size = size; - } else { - assert(((struct udata_t *)udata)->fapl_image_size == -size); - assert(((struct udata_t *)udata)->fapl_ref_count >= -1); - ((struct udata_t *)udata)->vfd_image_ptr = -((struct udata_t *)udata)->fapl_image_ptr; - ((struct udata_t *)udata)->vfd_image_size = size; - } - return_value = ((struct udata_t *)udata)->vfd_image_ptr; - (((struct udata_t *)udata)->vfd_ref_count)++; - break; - default: - assert(FALSE); + void * return_value = NULL; + switch ( file_image_op ) { + case H5_FILE_IMAGE_OP_PROPERTY_LIST_SET: + case H5_FILE_IMAGE_OP_PROPERTY_LIST_COPY: + assert(((struct udata_t *)udata)->fapl_image_ptr != NULL); + assert(((struct udata_t *)udata)->fapl_image_size == size); + assert(((struct udata_t *)udata)->fapl_ref_count >= 0); + return_value = ((struct udata_t *)udata)->fapl_image_ptr; + (((struct udata_t *)udata)->fapl_ref_count)++; + break; + case H5_FILE_IMAGE_OP_PROPERTY_LIST_GET: + assert(((struct udata_t *)udata)->fapl_image_ptr != NULL); + assert(((struct udata_t *)udata)->vfd_image_size == size); + assert(((struct udata_t *)udata)->fapl_ref_count >= 1); + return_value = ((struct udata_t *)udata)->fapl_image_ptr; + /* don’t increment ref count */ + break; + case H5_FILE_IMAGE_OP_FILE_OPEN: + assert(((struct udata_t *)udata)->vfd_image_ptr == NULL); + assert(((struct udata_t *)udata)->vfd_image_size == 0); + assert(((struct udata_t *)udata)->vfd_ref_count == 0); + if (((struct udata_t *)udata)->fapl_image_ptr == NULL ) { + ((struct udata_t *)udata)->vfd_image_ptr = malloc(size); + ((struct udata_t *)udata)->vfd_image_size = size; + } else { + assert(((struct udata_t *)udata)->fapl_image_size == size); + assert(((struct udata_t *)udata)->fapl_ref_count >= 1); + ((struct udata_t *)udata)->vfd_image_ptr = ((struct udata_t *)udata)->fapl_image_ptr; + ((struct udata_t *)udata)->vfd_image_size = size; + } + return_value = ((struct udata_t *)udata)->vfd_image_ptr; + (((struct udata_t *)udata)->vfd_ref_count)++; + break; + default: + assert(FALSE); } - return(return_value); + return(return_value); } + void *image_memcpy)(void *dest, const void *src, size_t size, H5_file_image_op_t file_image_op, void *udata) { - switch(file_image_op) { - case H5_FILE_IMAGE_OP_PROPERTY_LIST_SET: - case H5_FILE_IMAGE_OP_PROPERTY_LIST_COPY: - case H5_FILE_IMAGE_OP_PROPERTY_LIST_GET: -assert(dest == ((struct udata_t *)udata)->fapl_image_ptr); -assert(src == ((struct udata_t *)udata)->fapl_image_ptr); -assert(size == ((struct udata_t *)udata)->fapl_image_size); -assert(((struct udata_t *)udata)->fapl_ref_count >= 1); -break; -case H5_FILE_IMAGE_OP_FILE_OPEN: -assert(dest == ((struct udata_t *)udata)->vfd_image_ptr); -assert(src == ((struct udata_t *)udata)->fapl_image_ptr); -assert(size == ((struct udata_t *)udata)->fapl_image_size); -assert(size == ((struct udata_t *)udata)->vfd_image_size); -assert(((struct udata_t *)udata)->fapl_ref_count >= 1); -assert(((struct udata_t *)udata)->vfd_ref_count == 1); -break; - default: - assert(FALSE); - break; - } - return(dest); /* if we get here, we must have been successful */ - } -void *image_realloc(void *ptr, size_t size, H5_file_image_op_t file_image_op, - void *udata) + switch(file_image_op) { + case H5_FILE_IMAGE_OP_PROPERTY_LIST_SET: + case H5_FILE_IMAGE_OP_PROPERTY_LIST_COPY: + case H5_FILE_IMAGE_OP_PROPERTY_LIST_GET: + assert(dest == ((struct udata_t *)udata)->fapl_image_ptr); + assert(src == ((struct udata_t *)udata)->fapl_image_ptr); + assert(size == ((struct udata_t *)udata)->fapl_image_size); + assert(((struct udata_t *)udata)->fapl_ref_count >= 1); + break; + case H5_FILE_IMAGE_OP_FILE_OPEN: + assert(dest == ((struct udata_t *)udata)->vfd_image_ptr); + assert(src == ((struct udata_t *)udata)->fapl_image_ptr); + assert(size == ((struct udata_t *)udata)->fapl_image_size); + assert(size == ((struct udata_t *)udata)->vfd_image_size); + assert(((struct udata_t *)udata)->fapl_ref_count >= 1); + assert(((struct udata_t *)udata)->vfd_ref_count == 1); + break; + default: + assert(FALSE); + break; + } + return(dest); /* if we get here, we must have been successful */ +} + +void *image_realloc(void *ptr, size_t size, H5_file_image_op_t file_image_op, void *udata) { - assert(ptr == ((struct udata_t *)udata)->vfd_image_ptr); | -assert(((struct udata_t *)udata)->vfd_ref_count == 1); -((struct udata_t *)udata)->vfd_image_ptr = realloc(ptr, size); - ((struct udata_t *)udata)->vfd_image_size = size; -return((((struct udata_t *)udata)->vfd_image_ptr); + assert(ptr == ((struct udata_t *)udata)->vfd_image_ptr); | + assert(((struct udata_t *)udata)->vfd_ref_count == 1); + ((struct udata_t *)udata)->vfd_image_ptr = realloc(ptr, size); + ((struct udata_t *)udata)->vfd_image_size = size; + return((((struct udata_t *)udata)->vfd_image_ptr); } + herr_t image_free(void *ptr, H5_file_image_op_t file_image_op, void *udata) { - switch(file_image_op) { - case H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE: - assert(ptr == ((struct udata_t *)udata)->fapl_image_ptr); - assert(((struct udata_t *)udata)->fapl_ref_count >= 1); - (((struct udata_t *)udata)->fapl_ref_count)--; - break; - case H5_FILE_IMAGE_OP_FILE_CLOSE: - assert(ptr == ((struct udata_t *)udata)->vfd_image_ptr); - assert(((struct udata_t *)udata)->vfd_ref_count == 1); - (((struct udata_t *)udata)->vfd_ref_count)--; - break; - default: - assert(FALSE); - break; - } - return(0); /* if we get here, we must have been successful */ + switch(file_image_op) { + case H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE: + assert(ptr == ((struct udata_t *)udata)->fapl_image_ptr); + assert(((struct udata_t *)udata)->fapl_ref_count >= 1); + (((struct udata_t *)udata)->fapl_ref_count)--; + break; + case H5_FILE_IMAGE_OP_FILE_CLOSE: + assert(ptr == ((struct udata_t *)udata)->vfd_image_ptr); + assert(((struct udata_t *)udata)->vfd_ref_count == 1); + (((struct udata_t *)udata)->vfd_ref_count)--; + break; + default: + assert(FALSE); + break; + } + return(0); /* if we get here, we must have been successful */ } + void *udata_copy(void *udata) { return(udata); } + herr_t udata_free(void *udata) { return(0); @@ -1022,18 +1104,19 @@

    4.2. In-memory HDF5 File Image C <allocate fapl_id> H5Pset_file_image_callbacks(fapl_id, &callbacks); if ( initial_file_open ) { - initial_file_open = FALSE; + initial_file_open = FALSE; } else { - assert(udata.vfd_image_ptr != NULL); - assert(udata.vfd_image_size > 0); - assert(udata.vfd_ref_count == 0); - assert(udata.fapl_ref_count == 0); - udata.fapl_image_ptr = udata.vfd_image_ptr; - udata.fapl_image_size = udata.vfd_image_size; - udata.vfd_image_ptr = NULL; - udata.vfd_image_size = 0; - H5Pset_file_image(fapl_id, udata.fapl_image_ptr, udata.fapl_image_size); + assert(udata.vfd_image_ptr != NULL); + assert(udata.vfd_image_size > 0); + assert(udata.vfd_ref_count == 0); + assert(udata.fapl_ref_count == 0); + udata.fapl_image_ptr = udata.vfd_image_ptr; + udata.fapl_image_size = udata.vfd_image_size; + udata.vfd_image_ptr = NULL; + udata.vfd_image_size = 0; + H5Pset_file_image(fapl_id, udata.fapl_image_ptr, udata.fapl_image_size); } + <open core file using fapl_id> <discard fapl any time after the open> <write/update the file, and then close it> @@ -1045,42 +1128,73 @@

    4.2. In-memory HDF5 File Image C <free the image when done>

    -Example 10. Using H5LTopen_file_image where only the datasets change and where the file structure and image size might not be constant

    +Example 10. Using H5LTopen_file_image where only the datasets change and where the file structure and image size might not be constant

    +

    The above pseudo code shows how a buffer can be passed back and forth between the application and the HDF5 Library. The code also shows the application having control of the actual allocation, reallocation, and freeing of the buffer.

    4.3. Using HDF5 to Construct and Read a Data Packet

    Using the file image operations described in this document, we can bundle up data in an image of an HDF5 file on one process, transmit the image to a second process, and then open and read the image on the second process without any mandatory file system I/O.

    We have already demonstrated the construction and reading of such buffers above, but it may be useful to offer an example of the full operation. We do so in the example below using as simple a set of calls as possible. The set of calls in the example has extra buffer allocations. To reduce extra buffer allocations, see the sections above.

    In the following example, we construct an HDF5 file image on process A and then transmit the image to process B where we then open the image and extract the desired data. Note that no file system I/O is performed: all the processing is done in memory with the Core file driver.

    -

    *** Process A *** -

    
    -<Open and construct the desired file with the Core file driver>
    -H5Fflush(fid);
    -size = H5Fget_file_image(fid, NULL, 0);
    -buffer_ptr = malloc(size);
    -H5Fget_file_image(fid, buffer_ptr, size);
    -<transmit size>
    -<transmit *buffer_ptr>
    -free(buffer_ptr);
    -<close core file>
    -*** Process B ***
    -hid_t file_id;

    -<receive size> -buffer_ptr = malloc(size) -<receive image in *buffer_ptr> -file_id = H5LTopen_file_image(buf, +

    Table 4. Summary of Don’t Copy and Don’t Release Flag Actions
    Don’t Copy FlagDon’t Release FlagMake Copy of User Supplied BufferPass User Supplied Buffer to File DriverRelease User Supplied Buffer When DonePermit realloc of Buffer Used by File DriverDon’t Copy FlagDon’t Release FlagMake Copy of User Supplied BufferPass User Supplied Buffer to File DriverRelease User Supplied Buffer When DonePermit realloc of Buffer Used by File Driver
    False
    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    *** Process A ****** Process B ***
    <Open and construct the desired file with the Core file driver>hid_t file_id;

    H5Fflush(fid);
    size = H5Fget_file_image(fid, NULL, 0);
    buffer_ptr = malloc(size);
    H5Fget_file_image(fid, buffer_ptr, size);
    <transmit size>
    <receive size>
    <transmit *buffer_ptr>buffer_ptr = malloc(size)
    free(buffer_ptr);<receive image in *buffer_ptr>
    <close core file>file_id = H5LTopen_file_image(buf, buf_len, - H5LT_FILE_IMAGE_DONT_COPY); -<read data from file, then close. note that the Core file driver will discard the buffer on close> - + H5LT_FILE_IMAGE_DONT_COPY);
    <read data from file, then close. note that the Core file driver will discard the buffer on close>
    -Example 11. Building and passing a file image from one process to another +Example 11. Building and passing a file image from one process to another

    4.4. Using a Template File

    +

    After the above examples, an example of the use of a template file might seem anti-climactic. A template file might be used to enforce consistency on file structure between files or in parallel HDF5 to avoid long sequences of collective operations to create the desired groups, datatypes, and possibly datasets. The following pseudo code outlines a potential use:

    -

    <allocate and initialize buf and buflen, with buf containing the desired initial image (which in turn contains the desired group, datatype, and dataset definitions), and buf_len containing the size of buf> + +

    
    +<allocate and initialize buf and buflen, with buf containing the desired initial image (which in turn contains the desired group, datatype, and dataset definitions), and buf_len containing the size of buf>
     <allocate fapl_id>
     <set fapl to use desired file driver that supports initial images>
     H5Pset_file_image(fapl_id, buf, buf_len);
    @@ -1090,7 +1204,8 @@ 

    4.4. Using a Template File

    <read and/or write file as desired, close>
    -Example 12. Using a template file

    +Example 12. Using a template file

    +

    Observe that the above pseudo code includes an unnecessary buffer allocation and copy in the call to H5Pset_file_image(). As we have already discussed ways of avoiding this, we will not address that issue here.

    What is interesting in this case is to consider why the application would find this use case attractive.

    In the serial case, at first glance there seems little reason to use the initial image facility at all. It is easy enough to use standard C calls to duplicate a template file, rename it as desired, and then open it as an HDF5 file.

    @@ -1101,14 +1216,18 @@

    4.4. Using a Template File

    In closing, we would like to consider one last point. In the parallel case, we would expect template files to be quite large. Parallel HDF5 requires eager space allocation for chunked datasets. For similar reasons, we would expect template files in this context to contain long sequences of zeros with a scattering of metadata here and there. Such files would compress well, and the compressed images would be cheap to distribute across the available processes if necessary. Once distributed, each process could uncompress the image and write to file those sections containing actual data that lay within the section of the file assigned to the process. This approach might be significantly faster than a simple copy as it would allow sparse writes, and thus it might provide a compelling use case for template files. However, this approach would require extending our current API to allow compressed images. We would also have to add the H5Pget/set_image_decompression_callback() API calls. We see no problem in doing this. However, it is beyond the scope of the current effort, and thus we will not pursue the matter further unless there is interest in our doing so.

    5. Java Signatures for File Image Operations API Calls

    +

    Potential Java function call signatures for the file image operation APIs are described in this section. These have not yet been implemented, and there are no immediate plans for implementation.

    Note that the H5LTopen_file_image() call is omitted. Our practice has been to not support high-level library calls in Java.

    -

    H5Pset_file_image

    -

    int H5Pset_file_image(int fapl_id, const byte[] buf_ptr); -H5Pget_file_image

    -

    herr_t H5Pget_file_image(hid_t fapl_id, byte[] buf_ptr_ptr); -H5_file_image_op_t

    -

    public static H5_file_image_op_t + +

    H5Pset_file_image

    +

    int H5Pset_file_image(int fapl_id, const byte[] buf_ptr);

    + +H5Pget_file_image

    +

    herr_t H5Pget_file_image(hid_t fapl_id, byte[] buf_ptr_ptr);

    + +H5_file_image_op_t

    +
    public static H5_file_image_op_t
     {
          H5_FILE_IMAGE_OP_PROPERTY_LIST_SET,
          H5_FILE_IMAGE_OP_PROPERTY_LIST_COPY,
    @@ -1117,34 +1236,41 @@ 

    5. Java Signatures H5_FILE_IMAGE_OP_FILE_OPEN, H5_FILE_IMAGE_OP_FILE_RESIZE, H5_FILE_IMAGE_OP_FILE_CLOSE -} -H5_file_image_malloc_cb

    -

    public interface H5_file_image_malloc_cb extends Callbacks { +}

    + +H5_file_image_malloc_cb

    +
    public interface H5_file_image_malloc_cb extends Callbacks {
          buf[] callback(H5_file_image_op_t file_image_op, CBuserdata udata);
    -}
    -H5_file_image_memcpy_cb

    -

    public interface H5_file_image_memcpy_cb extends Callbacks { +}

    + +H5_file_image_memcpy_cb

    +
    public interface H5_file_image_memcpy_cb extends Callbacks {
     buf[] callback(buf[] dest, const buf[] src, H5_file_image_op_t file_image_op, CBuserdata
     udata);
    -}
    -H5_file_image_realloc_cb

    -

    public interface H5_file_image_realloc_cb extends Callbacks { +}

    + +H5_file_image_realloc_cb

    +
    public interface H5_file_image_realloc_cb extends Callbacks {
          buf[] callback(buf[] ptr, H5_file_image_op_t file_image_op, CBuserdata udata);
    -}
    -H5_file_image_free_cb

    -

    public interface H5_file_image_free_cb extends Callbacks { +}

    + +H5_file_image_free_cb

    +
    public interface H5_file_image_free_cb extends Callbacks {
          void callback(buf[] ptr, H5_file_image_op_t file_image_op, CBuserdata udata);
    -}
    -H5_file_udata_copy_cb

    -

    public interface H5_file_udata_copy_cb extends Callbacks { +}

    + +H5_file_udata_copy_cb

    +
    public interface H5_file_udata_copy_cb extends Callbacks {
          buf[] callback(CBuserdata udata);
    -}
    -H5_file_udata_free_cb

    -

    public interface H5_file_udata_free_cb extends Callbacks { +}

    + +H5_file_udata_free_cb

    +
    public interface H5_file_udata_free_cb extends Callbacks {
          void callback(CBuserdata udata);
    -}
    -H5_file_image_callbacks_t

    -

    public abstract class H5_file_image_callbacks_t +}

    + +H5_file_image_callbacks_t

    +
    public abstract class H5_file_image_callbacks_t
     {
          H5_file_image_malloc_cb image_malloc;
          H5_file_image_memcpy_cb image_memcpy;
    @@ -1169,52 +1295,79 @@ 

    5. Java Signatures this.udata_free = udata_free; this.udata = udata; } -} -H5Pset_file_image_callbacks

    -

    int H5Pset_file_image_callbacks(int fapl_id, - H5_file_image_callbacks_t callbacks_ptr); -H5Pget_file_image_callbacks

    -

    int H5Pget_file_image_callbacks(int fapl_id, - H5_file_image_callbacks_t[] callbacks_ptr); -H5Fget_file_image

    -

    long H5Fget_file_image(int file_id, byte[] buf_ptr);

    +}

    + +H5Pset_file_image_callbacks

    +
    int H5Pset_file_image_callbacks(int fapl_id, H5_file_image_callbacks_t callbacks_ptr);
    + +H5Pget_file_image_callbacks

    +
    int H5Pget_file_image_callbacks(int fapl_id, H5_file_image_callbacks_t[] callbacks_ptr);
    + +H5Fget_file_image

    +
    long H5Fget_file_image(int file_id, byte[] buf_ptr);

    6. Fortran Signatures for File Image Operations API Calls

    +

    Potential Fortran function call signatures for the file image operation APIs are described in this section. These have not yet been implemented, and there are no immediate plans for implementation.

    -

    6.1. Low-level Fortran API Routines

    +

    6.1. Low-level Fortran API Routines

    +

    The Fortran low-level APIs make use of Fortran 2003’s ISO_C_BINDING module in order to achieve portable and standard conforming interoperability with the C APIs. The C pointer (C_PTR) and function pointer (C_FUN_PTR) types are returned from the intrinsic procedures C_LOC(X) and C_FUNLOC(X), respectively, defined in the ISO_C_BINDING module. The argument X is the data or function to which the C pointers point to and must have the TARGET attribute in the calling program. Note that the variable name lengths of the Fortran equivalent of the predefined C constants were shortened to less than 31 characters in order to be Fortran standard compliant.

    6.1.1. H5Pset_file_image_f

    +

    The signature of H5Pset_file_image_f is defined as follows:

    SUBROUTINE H5Pset_file_image_f(fapl_id, buf_ptr, buf_len, hdferr)

    The parameters of H5Pset_file_image are defined as follows:

    -

    INTEGER(hid_t), INTENT(IN):: fapl_id -Will contain the ID of the target file access property list.

    -

    TYPE(C_PTR), INTENT(IN):: buf_ptr -Will supply the C pointer to the initial file image or C_NULL_PTR if no initial file image is desired.

    -

    INTEGER(size_t), INTENT(IN):: buf_len -Will contain the size of the supplied buffer or 0 if no initial image is desired.

    -

    INTEGER, INTENT(OUT) :: hdferr -Will return the error status: 0 for success and -1 for failure.

    + + + + + + + + + + + + + + + + + + +
    INTEGER(hid_t), INTENT(IN):: fapl_idWill contain the ID of the target file access property list.
    TYPE(C_PTR), INTENT(IN):: buf_ptrWill supply the C pointer to the initial file image or C_NULL_PTR if no initial file image is desired.
    INTEGER(size_t), INTENT(IN):: buf_lenWill contain the size of the supplied buffer or 0 if no initial image is desired.
    INTEGER, INTENT(OUT) :: hdferrWill return the error status: 0 for success and -1 for failure.

    6.1.2. H5Pget_file_image_f

    The signature of H5Pget_file_image_f is defined as follows:

    SUBROUTINE H5Pget_file_image_f(fapl_id, buf_ptr, buf_len, hdferr)

    The parameters of H5Pget_file_image_f are defined as follows:

    -

    INTEGER(hid_t), INTENT(IN) :: fapl_id -Will contain the ID of the target file access property list

    -

    TYPE(C_PTR), INTENT(INOUT), VALUE :: buf_ptr -Will hold either a C_NULL_PTR or a scalar of type c_ptr. If buf_ptr is not C_NULL_PTR, on successful return, buf_ptr shall contain a C pointer to a copy of the initial image provided in the last call to H5Pset_file_image_f for the supplied fapl_id, or buf_ptr shall contain a C_NULL_PTR if there is no initial image set. The Fortran pointer can be obtained using the intrinsic C_F_POINTER.

    -

    INTEGER(size_t), INTENT(OUT) :: buf_len -Will contain the value of the buffer parameter for the initial image in the supplied fapl_id. The value will be 0 if no initial image is set.

    -

    INTEGER, INTENT(OUT) :: hdferr -Will return the error status: 0 for success and -1 for failure.

    + + + + + + + + + + + + + + + + + +
    INTEGER(hid_t), INTENT(IN) :: fapl_idWill contain the ID of the target file access property list
    TYPE(C_PTR), INTENT(INOUT), VALUE :: buf_ptrWill hold either a C_NULL_PTR or a scalar of type c_ptr. If buf_ptr is not C_NULL_PTR, on successful return, buf_ptr shall contain a C pointer to a copy of the initial image provided in the last call to H5Pset_file_image_f for the supplied fapl_id, or buf_ptr shall contain a C_NULL_PTR if there is no initial image set. The Fortran pointer can be obtained using the intrinsic C_F_POINTER.
    INTEGER(size_t), INTENT(OUT) :: buf_lenWill contain the value of the buffer parameter for the initial image in the supplied fapl_id. The value will be 0 if no initial image is set.
    INTEGER, INTENT(OUT) :: hdferrWill return the error status: 0 for success and -1 for failure.

    6.1.3. H5Pset_file_image_callbacks_f

    -

    The signature of H5Pset_file_image_callbacks_f is defined as follows:

    +

    The signature of H5Pset_file_image_callbacks_f is defined as follows: + +

    
     

    INTEGER :: H5_IMAGE_OP_PROPERTY_LIST_SET_F=0, H5_IMAGE_OP_PROPERTY_LIST_COPY_F=1, H5_IMAGE_OP_PROPERTY_LIST_GET_F=2, @@ -1232,82 +1385,163 @@

    6.1.3. H5Pset_file_image_callbacks_f

    TYPE(C_FUN_PTR), VALUE :: udata_free TYPE(C_PTR), VALUE :: udata END TYPE H5_file_image_callbacks_t +
    + The semantics of the above values will be the same as those defined in the C enum. See Section 2.1.3 for more information.

    Fortran Callback APIs

    The Fortran callback APIs are shown below.

    FUNCTION op_func(size, file_image_op, udata,) RESULT(image_malloc) -INTEGER(size_t) :: size -Will contain the size of the image buffer to allocate in bytes. -INTEGER :: file_image_op -Will be set to one of the values of H5_IMAGE_OP_* indicating the operation being performed on the file image when this callback is invoked. -TYPE(C_PTR), VALUE :: udata -Will be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f. -TYPE(C_FUN_PTR), VALUE :: image_malloc -Shall contain a pointer to a function with functionality identical to the standard C library memcpy() call.

    + + + + + + + + + + + + + + + + + + +
    INTEGER(size_t) :: sizeWill contain the size of the image buffer to allocate in bytes.
    INTEGER :: file_image_opWill be set to one of the values of H5_IMAGE_OP_* indicating the operation being performed on the file image when this callback is invoked.
    TYPE(C_PTR), VALUE :: udataWill be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f.
    TYPE(C_FUN_PTR), VALUE :: image_mallocShall contain a pointer to a function with functionality identical to the standard C library memcpy() call.

    FUNCTION op_func(dest, src, size, & file_image_op, udata) RESULT(image_memcpy) -TYPE(C_PTR), VALUE :: dest -Will contain the address of the buffer into which to copy. -TYPE(C_PTR), VALUE :: src -Will contain the address of the buffer from which to copy -INTEGER(size_t) :: size -Will contain the number of bytes to copy. -INTEGER :: file_image_op -Will be set to one of the values of H5_IMAGE_OP_* indicating the operation being performed on the file image when this callback is invoked. -TYPE(C_PTR), VALUE :: udata -Will be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f. -TYPE(C_FUN_PTR), VALUE :: image_memcpy -Shall contain a pointer to a function with functionality identical to the standard C library memcpy() call.

    + + + + + + + + + + + + + + + + + + + + + + + + + + +
    TYPE(C_PTR), VALUE :: destWill contain the address of the buffer into which to copy.
    TYPE(C_PTR), VALUE :: srcWill contain the address of the buffer from which to copy
    INTEGER(size_t) :: sizeWill contain the number of bytes to copy.
    INTEGER :: file_image_opWill be set to one of the values of H5_IMAGE_OP_* indicating the operation being performed on the file image when this callback is invoked.
    TYPE(C_PTR), VALUE :: udataWill be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f.
    TYPE(C_FUN_PTR), VALUE :: image_memcpyShall contain a pointer to a function with functionality identical to the standard C library memcpy() call.

    +

    FUNCTION op_func(ptr, size, & file_image_op, udata) RESULT(image_realloc) -TYPE(C_PTR), VALUE :: ptr -Will contain the pointer to the buffer being reallocated -INTEGER(size_t) :: size -Will contain the desired size of the buffer after realloc in bytes. -INTEGER :: file_image_op -Will be set to one of the values of H5_IMAGE_OP_* indicating the operation being performed on the file image when this callback is invoked. -TYPE(C_PTR), VALUE :: udata -Will be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f. -TYPE(C_FUN_PTR), VALUE :: image_realloc -Shall contain a pointer to a unction functionality identical to the standard C library realloc() call.

    + + + + + + + + + + + + + + + + + + + + + +
    TYPE(C_PTR), VALUE :: ptrWill contain the pointer to the buffer being reallocated
    INTEGER(size_t) :: sizeWill contain the desired size of the buffer after realloc in bytes.
    INTEGER :: file_image_opWill be set to one of the values of H5_IMAGE_OP_* indicating the operation being performed on the file image when this callback is invoked.
    TYPE(C_PTR), VALUE :: udataWill be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f.
    TYPE(C_FUN_PTR), VALUE :: image_reallocShall contain a pointer to a unction functionality identical to the standard C library realloc() call.

    +

    FUNCTION op_func(ptr, file_image_op, udata) RESULT(image_free) -TYPE(C_PTR), VALUE :: ptr -Will contain the pointer to the buffer being released. -INTEGER :: file_image_op -Will be set to one of the values of H5_IMAGE_OP_* indicating the operation being performed on the file image when this callback is invoked. -TYPE(C_PTR), VALUE :: udata -Will be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f. -TYPE(C_PTR), VALUE :: image_free -Shall contain a pointer to a function with functionality identical to the standard C library free() call

    + + + + + + + + + + + + + + + + + +
    TYPE(C_PTR), VALUE :: ptrWill contain the pointer to the buffer being released.
    INTEGER :: file_image_opWill be set to one of the values of H5_IMAGE_OP_* indicating the operation being performed on the file image when this callback is invoked.
    TYPE(C_PTR), VALUE :: udataWill be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f.
    TYPE(C_PTR), VALUE :: image_freeShall contain a pointer to a function with functionality identical to the standard C library free() call

    +

    FUNCTION op_func(udata) RESULT(udata_copy) -TYPE(C_PTR), VALUE :: udata -Will be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f. -TYPE(C_FUN_PTR), VALUE :: udata_copy -Shall contain a pointer to a function that will allocate a buffer of suitable size, copy the contents of the supplied udata into the new buffer, and return the address of the new buffer. The function will return C_NULL_PTR on failure.

    + + + + + + + + + +
    TYPE(C_PTR), VALUE :: udataWill be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f.
    TYPE(C_FUN_PTR), VALUE :: udata_copyShall contain a pointer to a function that will allocate a buffer of suitable size, copy the contents of the supplied udata into the new buffer, and return the address of the new buffer. The function will return C_NULL_PTR on failure.

    +

    FUNCTION op_func(udata) RESULT(udata_free) -TYPE(C_PTR), VALUE :: udata -Shall contain a pointer value, potentially to user-defined data, that will be passed to the image_malloc, image_memcpy, image_realloc, and image_free callbacks.

    + + + + + +
    TYPE(C_PTR), VALUE :: udataShall contain a pointer value, potentially to user-defined data, that will be passed to the image_malloc, image_memcpy, image_realloc, and image_free callbacks.

    +

    The signature of H5Pset_file_image_callbacks_f is defined as follows:

    SUBROUTINE H5Pset_file_image_callbacks_f(fapl_id, &callbacks_ptr, hdferr) The parameters are defined as follows:

    -

    INTEGER(hid_t), INTENT(IN) :: fapl_id -Will contain the ID of the target file access property list.

    -

    TYPE(H5_file_image_callbacks_t), INTENT(IN) :: callbacks_ptr -Will contain the callback derived type. callbacks_ptr shall contain a pointer to the Fortran function via the intrinsic functions C_LOC(X) and C_FUNLOC(X).

    -

    INTEGER, INTENT(OUT) :: hdferr -Will return the error status: 0 for success and -1 for failure.

    + + + + + + + + + + + + + +

    INTEGER(hid_t), INTENT(IN) :: fapl_id

    Will contain the ID of the target file access property list.

    TYPE(H5_file_image_callbacks_t), INTENT(IN) :: callbacks_ptr

    Will contain the callback derived type. callbacks_ptr shall contain a pointer to the Fortran function via the intrinsic functions C_LOC(X) and C_FUNLOC(X).

    INTEGER, INTENT(OUT) :: hdferr

    Will return the error status: 0 for success and -1 for failure.

    6.1.4. H5Pget_file_image_callbacks_f

    The H5Pget_file_image_callbacks_f routine is designed to obtain the current file image callbacks from a file access property list.

    The signature is defined as follows

    SUBROUTINE H5Pget_file_image_callbacks_f(fapl_id, callbacks_ptr, hdferr) The parameters are defined as follows:

    -

    INTEGER(hid_t), INTENT(IN) :: fapl_id -Will contain the ID of the target file access property list.

    -

    TYPE(H5_file_image_callbacks_t), INTENT(OUT) :: callbacks_ptr -Will contain the callback derived type. Each member of the derived type shall have the same meaning as its C counterpart. See section 2.1.4 for more information.

    -

    INTEGER, INTENT(OUT) :: hdferr -Will return the error status: 0 for success and -1 for failure.

    - -

    6.1.5. Fortran Virtual File Driver Feature Flags

    + + + + + + + + + + + + + +
    INTEGER(hid_t), INTENT(IN) :: fapl_idWill contain the ID of the target file access property list.

    TYPE(H5_file_image_callbacks_t), INTENT(OUT) :: callbacks_ptrWill contain the callback derived type. Each member of the derived type shall have the same meaning as its C counterpart. See section 2.1.4 for more information.

    INTEGER, INTENT(OUT) :: hdferrWill return the error status: 0 for success and -1 for failure.

    + +

    6.1.5. Fortran Virtual File Driver Feature Flags

    Implementation of the H5Pget/set_file_image_callbacks_f() and H5Pget/set_file_image_f() APIs requires a pair of new virtual file driver feature flags:

    H5FD_FEAT_LET_IMAGE_F H5FD_FEAT_LET_IMAGE_CALLBACK_F @@ -1317,31 +1551,58 @@

    6.1.6. H5Fget_file_image_f

    The signature of H5Fget_file_image_f shall be defined as follows:

    SUBROUTINE H5Fget_file_image_f(file_id, buf_ptr, buf_len, hdferr, buf_size) The parameters of H5Fget_file_image_f are defined as follows:

    -

    INTEGER(hid_t), INTENT(IN) :: file_id -Will contain the ID of the target file.

    -

    TYPE(C_PTR), INTENT(IN) :: buf_ptr -Will contain a C pointer to the buffer into which the image of the HDF5 file is to be copied. If buf_ptr is C_NULL_PTR, no data will be copied.

    -

    INTEGER(size_t), INTENT(IN) :: buf_len -Will contain the size in bytes of the supplied buffer.

    -

    INTEGER(ssizet_t), INTENT(OUT), OPTIONAL :: buf_size -Will indicate the buffer size required to store the file image (in other words, the length of the file). If only the buf_size is needed, then buf_ptr should be also be set to C_NULL_PTR

    -

    INTEGER, INTENT(OUT) :: hdferr -Returns the error status: 0 for success and -1 for failure.

    + + + + + + + + + + + + + + + + + + + +

    See the “H5Fget_file_image” section for more information.

    -

    6.2. High-level Fortran API Routine +

    +
    INTEGER(hid_t), INTENT(IN) :: file_idWill contain the ID of the target file.

    TYPE(C_PTR), INTENT(IN) :: buf_ptrWill contain a C pointer to the buffer into which the image of the HDF5 file is to be copied. If buf_ptr is C_NULL_PTR, no data will be copied.

    INTEGER(size_t), INTENT(IN) :: buf_lenWill contain the size in bytes of the supplied buffer.

    INTEGER(ssizet_t), INTENT(OUT), OPTIONAL :: buf_sizeWill indicate the buffer size required to store the file image (in other words, the length of the file). If only the buf_size is needed, then buf_ptr should be also be set to C_NULL_PTR

    INTEGER, INTENT(OUT) :: hdferrReturns the error status: 0 for success and -1 for failure.

    + +

    6.2. High-level Fortran API Routine

    + The new Fortran high-level routine H5LTopen_file_image_f will provide a wrapper for the high-level H5LTopen_file_image function. Consequently, the high-level Fortran API will not be implemented using low-level HDF5 Fortran APIs. -6.2.1. H5LTopen_file_image_f + +

    6.2.1. H5LTopen_file_image_f

    The signature of H5LTopen_file_image_f is defined as follows:

    SUBROUTINE H5LTopen_file_image_f(buf_ptr, buf_len, flags, file_id, hdferr) The parameters of H5LTopen_file_image_f are defined as follows:

    -

    TYPE(C_PTR), INTENT(IN), VALUE :: buf_ptr -Will contain a pointer to the supplied initial image. A C_NULL_PTR value is invalid and will cause H5LTopen_file_image_f to fail.

    -

    INTEGER(size_t), INTENT(IN) :: buf_len -Will contain the size of the supplied buffer. A value of 0 is invalid and will cause H5LTopen_file_image_f to fail.

    -

    INTEGER, INTENT(IN) :: flags -Will contain a set of flags indicating whether the image is to be opened read/write, whether HDF5 is to take control of the buffer, and how long the application promises to maintain the buffer. Possible flags are as follows: H5LT_IMAGE_OPEN_RW_F, H5LT_IMAGE_DONT_COPY_F, and H5LT_IMAGE_DONT_RELEASE_F. The C equivalent flags are defined in the “H5LTopen_file_image” section.

    -

    INTEGER(hid_t), INTENT(IN) :: file_id -Will be a file ID on success.

    -

    INTEGER, INTENT(OUT) :: hdferr -Returns the error status: 0 for success and -1 for failure.

    + + + + + + + + + + + + + + + + + + + + + +
    TYPE(C_PTR), INTENT(IN), VALUE :: buf_ptrWill contain a pointer to the supplied initial image. A C_NULL_PTR value is invalid and will cause H5LTopen_file_image_f to fail.
    INTEGER(size_t), INTENT(IN) :: buf_lenWill contain the size of the supplied buffer. A value of 0 is invalid and will cause H5LTopen_file_image_f to fail.
    INTEGER, INTENT(IN) :: flagsWill contain a set of flags indicating whether the image is to be opened read/write, whether HDF5 is to take control of the buffer, and how long the application promises to maintain the buffer. Possible flags are as follows: H5LT_IMAGE_OPEN_RW_F, H5LT_IMAGE_DONT_COPY_F, and H5LT_IMAGE_DONT_RELEASE_F. The C equivalent flags are defined in the “H5LTopen_file_image” section.
    INTEGER(hid_t), INTENT(IN) :: file_idWill be a file ID on success.
    INTEGER, INTENT(OUT) :: hdferrReturns the error status: 0 for success and -1 for failure.