Spaces: High level idea and API

(Authored by Naveen)

Base proposal:

• Subcategory:1 Introduce Space
  • Introduce Team-based SHEAP
  • Determine the Team/Context/Space relationships

Extended proposal:

• Subcategory:2 Support for different types of HBM, FAM (for host and device)
  • Adding extra traits in the space configuration
  • Add new memory order completion semantics (if required)
• Subcategory:3 Focused Support for FAM as external memory usage == OpenFAM
  • Introduce new type of space without any Team reqs (External Space)
  • Introduce new completion semantics (context options)
  • Comeback with actual use-case

Current state of the base proposal:

Intended Semantics:

• Enable using multiple spaces per team
• Allow accessing spaces of the parent team

API Changes:

shmem_space_config_t {
    long long        config_mask;
    int              size;
    shmem_space_sz_t page_size;
    shmem_space_pl_t alloc_policy;
    /* other traits will be added later */
    shmem_space_nd_t numa_domain;
};

// local
void shmem_space_create(shmem_space_config_t space_config, shmem_space_t *space);
void shmem_space_destroy(shmem_space_t space);

// collective
void shmem_space_attach(shmem_team_t team, shmem_space_t space);
void shmem_space_detach(shmem_team_t team, shmem_space_t space);

// adding space to have one-to-one mapping with ctx-space or else ctx is required
// to do lookup for space during RMA and AMO
shmem_ctx_opt_t {
    shmem_team_t  team;  // mandatory (kind-off)
    shmem_space_t space; // optional
};

void shmem_ext_ctx_create(long options, shmem_ctx_opt_t opt, shmem_ctx_t *ctx);

void *shmem_space_malloc(shmem_space_t space, size_t size);
void *shmem_space_calloc(shmem_space_t space, size_t count, size_t size);
void *shmem_space_align(shmem_space_t space, size_t alignment, size_t size);
void  shmem_space_free(shmem_space_t space, void *ptr);

Explanation:

• Space create and destroy are local operations
• Cannot use a space before attaching it to a team
• By default, contexts created on a team will inherit all the spaces attached to a team on which it is created
• Users can specifically select a space to a context if needed But, before that the space has to be attached to a relavant team
• Performance optimization
  • Destroying a space has to be done after detaching from team and any contexts associated with it - needs work

Feedback:

1. Discuss with users on the performance impact of the lookup involved in space identification
2. Bring FAM use-cases to understand sub-category(iii)
3. Difference between the semantics of the RMA/AMO and collectives - in Space identification