RMA WG 08 08 2019

Attendees:

• Jim, Dave, Wasi (Intel)
• Keith Underwood, Naveen, Bob (Cray)
• Manju (Mellanox)
• Akhil (Nvidia)
• Nick

Non-blocking AMO update (Naveen)

• Topic 1: Text description

  • Text changed to remove saying that non-blocking AMOs completion is as a single atomic operation.
  • Wasiur, Nick: Are we limiting completion of non blocking AMOs, non-blocking gets to be known only with shmem_quiet? Can we use shmem_wait. In this regard, not changing anything as of now.
  • Jim: Generalize the statement about completion of non-blocking AMOs. It need not be shmem_quiet. It could be any subsequent operation that includes a quiet operations, for example, shmem_barrier, shmem_free. This needs to be fixed in other APIs as well. Naveen to do this as a separate PR.

• Topic 2: Deliver order of fetched values

  • Naveen: Can we order delivery of fetched value?
  • Jim: We want to only order atomic update and not returned value
  • This is true in SOS and Cray SHMEM
  • Nick: This is consistent with not having gets ordered by fence but having only puts ordered by fence

Memory Model Discussion

• Two options

1. shmem_fence as acquire_release operation
2. Shmem_fence as release operation. Have another acquire based shmem fence function. And also have shmem wait with acquire fence memory order.

• Jim: What are the performance implications of having shmem_fence as a release operation?
• Manju - With adaptive routing, shmem_fence will not a noop with IB. But adaptive routing is not a common case.
• Discussion to be continued...