Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Teams API topic branch #886

Merged
merged 99 commits into from
Nov 19, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
99 commits
Select commit Hold shift + click to select a range
6d3a31b
Add Teams bare-bones API without any functionality
davidozog Apr 12, 2019
73439c9
Add a KVS of available teams (DO NOT MERGE)
davidozog Apr 23, 2019
99d2f97
Teams: remove uthash and make all strides linear
davidozog Apr 26, 2019
f00ac33
Add shmemx_tem_translate unit test
davidozog Apr 29, 2019
1526cf2
Fix global_exit seg fault with no backtrace method
davidozog Apr 29, 2019
2f2fb84
Return `#include "config.h"` to `shmem_env.h`
davidozog Apr 30, 2019
52339d8
Assure teams API routines are prepended w/ shmemx
davidozog Apr 30, 2019
0e9a662
Add implementation/test of shmem_team_translate_pe
davidozog Apr 30, 2019
1b53db4
Update to shmem_internal_team_t where needed
davidozog May 13, 2019
81884da
Implement shmemx_team_split_2D and add a unit test
davidozog May 14, 2019
23226ca
Cleanup team split code, remove post-split barrier
davidozog May 14, 2019
8e4f844
Add shmemx_sync/shmemx_team_sync with unit test
davidozog May 14, 2019
41feee7
Implement shmemx_broadcast with teams support
davidozog May 15, 2019
e587b63
Add teams collect, fcollect, alltoall, alltoalls
davidozog May 16, 2019
3332081
Add bindings for collectives, fix c11 shmemx tests
davidozog May 16, 2019
b01b801
Cleanup the shmemx tests, don't exit(0) on failure
davidozog May 16, 2019
18becf8
Point contexts to the appropriate team, add test
davidozog May 23, 2019
b7fd4fa
Add team PE indexing for ctx RMA/AMO operations
davidozog May 30, 2019
1ef41d7
Add C++/C11 teams API bindings and set psync bits
davidozog Jun 10, 2019
4c8fa5c
Rename some vars and fix destroy & op_to_all bugs
davidozog Jun 25, 2019
8c73f23
Add team field to portals and none transport ctxs
davidozog Jun 25, 2019
1598f51
Support reduce_local unsigned types, fix example
davidozog Jun 25, 2019
4a1391c
Use separate psync in non-barrier/sync collectives
davidozog Jun 26, 2019
568324b
Add preliminary support for SHMEMX_TEAM_SHARED
davidozog Jun 28, 2019
00bac21
Rewrite TEAM_SHARED stride check without a malloc
davidozog Jul 9, 2019
2665b71
Fix pWrk mistake, rename a variable in team init
davidozog Jul 9, 2019
63cfde3
Simplify search for local peer PEs in TEAM_SHARED
davidozog Jul 15, 2019
de5ba2c
Merge branch 'master' of github.com:davidozog/sandia-shmem into wip/t…
davidozog Aug 20, 2019
a4f0217
Cleaning pre-defined teams initialization
davidozog Aug 20, 2019
41109d2
Cleanup teams c11/cxx bindings and SHMEMX_* fixes
davidozog Sep 6, 2019
c71e028
Rename SHMEM_TEAM_NULL to SHMEM_TEAM_INVALID
davidozog Sep 6, 2019
e9d8043
Miscellaneous bug fixes & cleanup throughout teams
davidozog Sep 6, 2019
16cf053
Associate transport context array with team object
davidozog Sep 13, 2019
90d96f2
Destroy all teams during finalize
davidozog Sep 16, 2019
eca2f5c
team pool split update, add ctx opt w/no transport
davidozog Sep 17, 2019
33fc9fe
Merge branch 'master' of github.com:Sandia-OpenSHMEM/SOS into wip/teams
davidozog Sep 20, 2019
ff30922
Copyright cleanup in teams files
davidozog Sep 20, 2019
e26f0e9
Cleanup team shmemx tests, remove pWrk from reduce
davidozog Sep 20, 2019
f9a94e7
Remove HOST and LEADERS teams & cleanup teams init
davidozog Oct 8, 2019
48b0290
Rearrange shmem_team_t & raise error w/bad bit-set
davidozog Oct 8, 2019
e74ea03
Add 2 psyncs per team for back-to-back collectives
davidozog Oct 11, 2019
8cf7156
Generalize choose/release psync in team collective
davidozog Oct 14, 2019
b6cf4e2
Merge branch 'master' of github.com:Sandia-OpenSHMEM/SOS into wip/teams
davidozog Oct 14, 2019
1a7c7e7
Add back-to-back team-collectives shmemx tests
davidozog Oct 14, 2019
db70857
Fix function signature for shmem_team_destroy
davidozog Oct 16, 2019
bd7c010
Fix checks for valid team triplet & choosing psync
davidozog Oct 18, 2019
281739a
Add teams tests: reuse teams and shrink active set
davidozog Oct 18, 2019
f5c6cf1
Add C11 test to check teams-based collective types
davidozog Oct 21, 2019
cab66b6
Check collective team is valid, not the active set
davidozog Oct 21, 2019
674e8b7
Add teams reduction type coverage test & bug fixes
davidozog Oct 25, 2019
117e69a
Clean/clarify some teams tests
davidozog Oct 25, 2019
2fe2b61
Fix bug when more than MAX_TEAMS created
davidozog Oct 28, 2019
cd884dc
Remove C11 "extras" from team reduction bindings
davidozog Oct 28, 2019
4fa6681
Fully remove C11 extras from reduction bindings
davidozog Oct 29, 2019
743da92
Fix team bug w/ indexing sync/barrier psync region
davidozog Oct 30, 2019
f496282
Revert a portion of the previous commit (d83a7e6)
davidozog Oct 31, 2019
280fdd4
Return nonzero if max teams reached, update README
davidozog Oct 31, 2019
7d66d55
Fix team assignment/comparison bug, clean warnings
davidozog Nov 1, 2019
791d568
Remove "extra" C++ bindings in teams reductions
davidozog Nov 5, 2019
f6bdaa4
Merge branch 'master' of github.com:Sandia-OpenSHMEM/SOS into wip/teams
davidozog Nov 5, 2019
e8076c2
Cleanup teams docs, comments, and unit tests
davidozog Nov 7, 2019
d920b63
Cleanup teams bindings and fix unit test warnings
davidozog Nov 8, 2019
3ea2e5b
Rename TEAM_ONLY_SELF, fix type warns, unused vars
davidozog Nov 8, 2019
5853583
Missed an instance of 'const' team config types
davidozog Nov 8, 2019
a2f8c63
Review #2 - simpler fixes to push through testing
davidozog Nov 11, 2019
7a1518a
Make psync-availabile array 8 uchars, not a uint64
davidozog Nov 12, 2019
23be78e
Change pe_in_active_set to return -1 or PE index
davidozog Nov 12, 2019
2b0ec16
Add --enable-max-teams=NUMBER option to configury
davidozog Nov 12, 2019
f265b40
Revert change to setting max_teams in shmem_team.c
davidozog Nov 12, 2019
b86b413
Minor changes from teams review #3 for testing
davidozog Nov 14, 2019
1ba3da4
Fix type incongruence in shmemx_reduce test
Nov 15, 2019
61e8b44
Cleanup and remove unneeded dup in reuse_teams test
Nov 15, 2019
4c84ced
Fix numerous bugs in team_sync unit test
Nov 15, 2019
88f46fe
Remove extra team dup in collect active set test
Nov 15, 2019
fce3cb2
Mark internal function static
Nov 15, 2019
be9f86a
Update C11 test macro to eat the semicolon
Nov 15, 2019
f8b9ff6
Update choose_psync to return long*
Nov 15, 2019
aa8bd2f
Move accessors up to user bindings
Nov 15, 2019
87a44ea
Update to make unit test work with c99 compilers
Nov 15, 2019
9e2e095
Fix choose_psync comment
Nov 15, 2019
a296326
Merge pull request #7 from jdinan/wip/teams
davidozog Nov 15, 2019
314daa8
Use persistent assertion in team init
Nov 16, 2019
39c3694
Remove dead code from team init
Nov 16, 2019
1cc7b7e
Set team world on the default context
Nov 16, 2019
db112f5
Improve error message
Nov 16, 2019
daff427
Various teams review, round #4.
davidozog Nov 16, 2019
9044877
Merge pull request #8 from jdinan/wip/teams
davidozog Nov 16, 2019
595d1f0
Merge branch 'wip/teams' of github.com:davidozog/sandia-shmem into wi…
davidozog Nov 16, 2019
57d414b
Destroy the predefined teams in shmem_team_fini
davidozog Nov 16, 2019
7c57f6f
Destroy predefined teams outside portals as well
davidozog Nov 16, 2019
d40b74b
Remove redundant PE id from error message
Nov 18, 2019
1d2da2b
Improve error handling for SHMEM_TEAMS_MAX
Nov 18, 2019
b4e26ca
Only set bits corresponding to allocated psyncs
Nov 18, 2019
3b85c1a
Add psync avail bit string debugging output
Nov 18, 2019
24ca0a0
Add comment about SHMEM_CTX_DEFAULT destroy
Nov 18, 2019
90886c8
Remove sketchy spec tests
Nov 18, 2019
c76260f
Set my_pe field on new teams
Nov 18, 2019
0310b31
Fix bugs in spec tests
Nov 18, 2019
5294eb0
Update reductions to support size_t nreduce
Nov 18, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 11 additions & 27 deletions src/shmem_internal.h
Original file line number Diff line number Diff line change
Expand Up @@ -486,39 +486,33 @@ extern uint64_t (*shmem_internal_gettid_fn)(void);
extern void shmem_internal_register_gettid(uint64_t (*gettid_fn)(void));

static inline
void shmem_internal_bit_set(void * ptr, size_t size, size_t index)
void shmem_internal_bit_set(unsigned char *ptr, size_t size, size_t index)
jdinan marked this conversation as resolved.
Show resolved Hide resolved
{
shmem_internal_assert(size > 0 && (index < size * CHAR_BIT));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So ... A byte in C is not guaranteed to be an octet (hence CHAR_BIT) and sizeof returns bytes [1]. But, the fixed-width types (e.g. int64_t) are exactly that number of bits. So, there is an incongruence that could be problematic (at least in principle) if you use fixed-width types and access through a char pointer. :neckbeard: If ptr is going to correspond to a fixed-width type, we could introduce a configure check to ensure that CHAR_BIT evenly divides 64, or we can treat size as a number of octets and use uint8_t * instead of unsigned char *. Or we can ignore this problem (maybe just add a comment) since in practice it will never happen. Or (best option yet) make ptr an unsigned char * instead of void * and use an array of unsigned char for the bit array. I think I like this option best.

[1] https://stackoverflow.com/questions/11868211/does-sizeof-return-the-number-of-bytes-or-the-number-of-octets-of-a-type-in-c

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call! I think this is resolved, but please double check.


unsigned char *bytes = (unsigned char*) ptr;

size_t which_byte = index / size;
bytes[which_byte] |= (1UL << (index % CHAR_BIT));
ptr[which_byte] |= (1 << (index % CHAR_BIT));

return;
}

static inline
void shmem_internal_bit_clear(void * ptr, size_t size, size_t index)
void shmem_internal_bit_clear(unsigned char *ptr, size_t size, size_t index)
{
shmem_internal_assert(size > 0 && (index < size * CHAR_BIT));

unsigned char *bytes = (unsigned char*) ptr;

size_t which_byte = index / size;
bytes[which_byte] &= ~(1UL << (index % CHAR_BIT));
ptr[which_byte] &= ~(1 << (index % CHAR_BIT));

return;
}

static inline
size_t shmem_internal_bit_1st_nonzero(const void *ptr, const size_t size)
size_t shmem_internal_bit_1st_nonzero(const unsigned char *ptr, const size_t size)
{
unsigned char *bytes = (unsigned char*) ptr;

/* The following ignores endianess: */
for(size_t i = 0; i < size; i++) {
unsigned char bit_val = bytes[i];
unsigned char bit_val = ptr[i];
for (size_t j = 0; bit_val && j < CHAR_BIT; j++) {
if (bit_val & 1) return i * CHAR_BIT + j;
bit_val >>= 1;
Expand All @@ -528,26 +522,16 @@ size_t shmem_internal_bit_1st_nonzero(const void *ptr, const size_t size)
return -1;
}

/* Return true if `global_pe` is in the given active set (return 0 otherwise), and
* and set `local_pe` to the PE index within this set (-1 if the PE is not in the set)
* */
/* Return -1 if `global_pe` is in the given active set (return 0 otherwise).
* If `global_pe` is in the active set, return the PE index within this set. */
static inline
int shmem_internal_pe_in_active_set(int global_pe, int PE_start, int PE_stride, int PE_size, int *local_pe)
int shmem_internal_pe_in_active_set(int global_pe, int PE_start, int PE_stride, int PE_size)
{

if (local_pe)
*local_pe = -1;

if (global_pe < PE_start)
return 0;

int n = (global_pe - PE_start) / PE_stride;
if (global_pe < PE_start || (global_pe - PE_start) % PE_stride || n >= PE_size)
return 0;
return -1;
else {
if (local_pe)
*local_pe = n;
return 1;
return n;
}
}
davidozog marked this conversation as resolved.
Show resolved Hide resolved

Expand Down
15 changes: 7 additions & 8 deletions src/shmem_team.c
Original file line number Diff line number Diff line change
Expand Up @@ -226,8 +226,8 @@ int shmem_internal_team_translate_pe(shmem_internal_team_t *src_team, int src_pe
if (src_pe_world < src_team->start || src_pe_world >= shmem_internal_num_pes)
davidozog marked this conversation as resolved.
Show resolved Hide resolved
return -1;

shmem_internal_pe_in_active_set(src_pe_world, dest_team->start, dest_team->stride,
dest_team->size, &dest_pe);
dest_pe = shmem_internal_pe_in_active_set(src_pe_world, dest_team->start, dest_team->stride,
dest_team->size);

return dest_pe;
}
Expand Down Expand Up @@ -264,8 +264,9 @@ int shmem_internal_team_split_strided(shmem_internal_team_t *parent_team, int PE
myteam->contexts_len = 0;
myteam->psync_idx = 0;
davidozog marked this conversation as resolved.
Show resolved Hide resolved

if (shmem_internal_pe_in_active_set(shmem_internal_my_pe, myteam->start, PE_stride,
PE_size, &myteam->my_pe)) {
myteam->my_pe = shmem_internal_pe_in_active_set(shmem_internal_my_pe, myteam->start, PE_stride, PE_size);
davidozog marked this conversation as resolved.
Show resolved Hide resolved

if (myteam->my_pe != -1) {

size_t psync = shmem_internal_team_choose_psync(parent_team, shmem_internal_team_reduce_type);

Expand Down Expand Up @@ -326,8 +327,7 @@ int shmem_internal_team_split_2d(shmem_internal_team_t *parent_team, int xrange,
for (int i = 0; i < num_xteams; i++) {
int xsize = (i == num_xteams - 1 && parent_size % xrange) ? parent_size % xrange : xrange;

if (shmem_internal_pe_in_active_set(shmem_internal_my_pe, start,
parent_stride, xsize, NULL)) {
if (shmem_internal_pe_in_active_set(shmem_internal_my_pe, start, parent_stride, xsize) != -1) {
ret = shmem_internal_team_split_strided(parent_team, start, parent_stride,
xsize, xaxis_config, xaxis_mask, xaxis_team);
davidozog marked this conversation as resolved.
Show resolved Hide resolved
if (ret) {
Expand All @@ -344,8 +344,7 @@ int shmem_internal_team_split_2d(shmem_internal_team_t *parent_team, int xrange,
int yrange = parent_size / xrange;
int ysize = (remainder && i < remainder) ? yrange + 1 : yrange;

if (shmem_internal_pe_in_active_set(shmem_internal_my_pe, start,
xrange*parent_stride, ysize, NULL)) {
if (shmem_internal_pe_in_active_set(shmem_internal_my_pe, start, xrange*parent_stride, ysize) != -1) {
ret = shmem_internal_team_split_strided(parent_team, start, xrange*parent_stride,
ysize, yaxis_config, yaxis_mask, yaxis_team);
davidozog marked this conversation as resolved.
Show resolved Hide resolved
if (ret) {
Expand Down