Skip to content

Developer Documentation

Jump to bottom
David Ozog edited this page May 3, 2024 · 10 revisions

This section provides documentation for developers and those interested in the workings of the Sandia OpenSHMEM code.

Overall Software Design

SOS is comprised of several software layers:

  1. Generic language bindings (C11/C++)
  2. Language bindings (C/Fortran)
  3. Internal, transport-independent API (shmem_internal_*)
  4. Shared memory transport (XPMEM, CMA, memcpy)
  5. Networking transport (Portals, OFI)

Aggressive inlining is used to reduce code path lengths for critical path operations. Most operations collapse into a single SOS function call. The OFI library also supports a direct mode, where provider code is inlined into SOS, further reducing code path lengths when that transport is used. Code that implements the generic language bindings in inlined into the application and collapsed at compile time into a C routine invocation.

Language Bindings Generation

The language bindings (C, C11, C++, and Fortran) are generated using a combination of tools:

  1. Autoconf substitution (files ending with .in)
  2. m4 macros (files endings with 4, e.g. c4 or h4)
  3. C preprocessor

Autoconf substitution is used for simple substitution of values that are gathered/detected during configuration (e.g. sizes of pSync arrays, support for compiler features, etc.). Autoconf takes a file ending with .in and produces the same file with the .in removed from the filename.

Most of the generation work is done by m4.

Defining Environment Variables

Environment variables should be added to the src/shmem_env_defs.h file. Entries have the format:

SOS_ENV( name, kind, default, category, short description )

For example:

SHMEM_INTERNAL_ENV_DEF(INFO, bool, false, SHMEM_INTERNAL_ENV_CAT_OPENSHMEM,
                       "Print library information message at startup")

Info about the arguments:

  • name - The name of the environment variable (e.g. SHMEM_INFO name is INFO)
  • kind - The type of the variable (currently long, size, bool, or string)
  • default - The default value
  • category - The category, used for organizing the variables when printed (currently SHMEM_INTERNAL_ENV_CAT_OPENSHMEM, SHMEM_INTERNAL_ENV_CAT_COLLECTIVES, SHMEM_INTERNAL_ENV_CAT_TRANSPORT, SHMEM_INTERNAL_ENV_CAT_INTRANODE, SHMEM_INTERNAL_ENV_CAT_OTHER)
  • short description - A string with a brief description of the variable (printed at startup when info is enabled)

Within the SOS code, the value of the environment variable can be retrieved through the corresponding name field that is automatically added to the shmem_internal_params`` structure, e.g., shmem_internal_params.INFO```.

The environment variable management code is implemented using the C preprocessor. We first define the SHMEM_INTERNAL_ENV_DEF macro, next include the shmem_env_defs.h header, and then undefine the macro. And voila, a whole bunch of code is generated.

For details on kinds and categories, refer to src/shmem_env.h. Defining new kinds involves defining the C type, printf format, and parsing routines. Defining new categories will require updating code that traverses the params by category (e.g. shmem_internal_print_env).

Copyright Notices for New Contributions

When new development work requires a copyright notice, please include the notice as a comment at the top of all files that include the copyrighted material. Additionally, please include the notice towards the top of the LICENSE file. SOS is provided by a BSD style licenses, and other licenses (such as GPL or other copyleft variants) may not be approved due to legal complications.

Developer instructions for renaming "master" branch to "main"

On Feb 8, 2023, SOS renamed the default git branch from "master" to "main". Developers can update their local repositories accordingly with these commands (for the origin remote):

git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a

Affected developers may also want to rename the "master" branch on their downstream forks of SOS to "main" within the branch settings.

Pull Request Procedures

Initial Pull Request Creation

  1. Fork the Repository: Start by forking the SOS Github repository to your personal account or to an organization where you have permission to create repositories.
  2. Clone Your Fork: Clone your fork to your local machine, for example using git clone.
  3. Create a New Branch: Create a new branch for your changes, for example with git checkout -b your-branch-name.
  4. Make and Commit Changes: Make the necessary changes in your local repository and commit with meaningful commit messages, for example using git commit -m 'Your commit message'. Please follow best practices for Git commit messages.
  5. Push to Your Fork: Push your branch and changes to your fork on GitHub, for example with git push origin your-branch-name.
  6. Open a Pull Request (PR): Go to your repository on GitHub, and you'll see a prompt to open a pull request. Click on 'New Pull Request', choose your branch, and fill in the PR details.
  7. Request Reviews: After submitting the PR, request reviews from by either mentioning them in a comment or using the 'Reviewers' section on the right.

Review and Approval Process

  1. Reviewers Check the PR: The designated reviewers will review your pull request for any issues or required changes.
  2. Make Requested Changes: If changes are requested, make them in your local branch, commit them, and push the updates to your branch on GitHub.
  3. Re-request Reviews: After making changes, re-request reviews from the approvers by clicking on the re-review button next to their names in the 'Reviewers' section.
  4. Approval by Reviewers: Each reviewer will provide feedback or approve the pull request. You need at least two approvals for the PR to be eligible for merging.
  5. Ensure All Tests Pass: Examine the 'Checks' section for errors coming from the SOS continuous integration (CI) workflow and ensure everything passes. If they fail, fix the issues and push the updates.

Finalizing the Pull Request

  1. Additional Approvals for Changes: If you've made changes after initial approvals, you need at least one additional approval from the reviewers to ensure the changes are also reviewed.
  2. Merge the PR: Once you have the required approvals and all checks pass, the PR can be merged into the main branch. This can be done by you, a repository maintainer (@davidozog/@wrrobin), or someone with merge permissions.
  3. Delete the Branch: After the merge, you can delete the feature branch from your fork if it's no longer needed.

Notes

  • Always sync your fork with the original repository before starting work to ensure you're working with the latest code.
  • While SOS does not define any explicit C/C++ coding style requirements, please keep the style consistent with existing code. SOS tends to follow the Google C++ Style Guide.
  • If an SOS PR requires changes to the tests-sos test suite, then open a separate PR on the tests-sos repository with those changes. After that, update the SOS PR .gitmodules file to test with the branch on the tests-sos PR. After concluding the PR process and merge in tests-sos (the procedure is the same as defined in this document), then revert the SOS PR to point to the main branch of tests-sos and continue the PR procedure.
Clone this wiki locally