feature: support CXI in OFI for Slingshot-11 #3791
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This supports the CXI interface for Cassini (AKA Slingshot-11) within the OFI machine layer. No application level changes are required, but a variety of command line options are provided to configure the memory pool and the selection of cxi interfaces. Note: this does require the use of the memory pool in order to efficiently support the FI_MR_ENDPOINT mode of memory registration. As the time cost of registering individual messages would be otherwise be entirely too high. Charmrun has been configured to wrap srun and currently assumes PMI2 with cray extensions for launching. The build system has been set up to autodetect CXI and enable support for it accordingly. For compatibility purposes, it also supports the use of cxi on the build line, but that should not be necessary on most HPE systems with proper LMOD environments.
ericjbohm
added
feature
machine layers
Build & test automation
Zerocopy
Performance
Cray
OFI
ericjbohm
requested review from
rbuch,
kavithachandrasekar,
lvkale,
adityapb,
ZwFink and
evan-charmworks
stwhite91
reviewed
Mar 26, 2024
this allows xpmem to be supported in the build process for nonsmp targets. This has not yet proven beneficial for performance.
jcphill
reviewed
Apr 11, 2024
lvkale
reviewed
Apr 11, 2024
adityapb
reviewed
Apr 11, 2024
stwhite91
reviewed
Apr 11, 2024
enable thread diagnostics in debug case
stwhite91
reviewed
Apr 12, 2024
|
The last machine that I had access to with regular OFI was turned off last
year. I could try it over TCP/IP somewhere, but not sure how useful the
resulting information would be.
…On Fri, Apr 12, 2024 at 1:50 PM Sam White ***@***.***> wrote:
***@***.**** commented on this pull request.
Any documentation updates needed?
Has non-CXI OFI been tested and benchmarked for performance with the
changes to use the LRTS mempool and the ofi request cache?
------------------------------
In src/arch/ofi/conv-common.h
<#3791 (comment)>:
> /*
* Use Simple client-side implementation of PMI.
* Valid only for CMK_USE_PMI.
* Optional in an SLURM environment.
* See src/arch/util/proc_management/simple_pmi/
*/
-#define CMK_USE_SIMPLEPMI 1
+#define CMK_USE_SIMPLEPMI 0
Does this get set somewhere else now in the default OFI case?
—
Reply to this email directly, view it on GitHub
<#3791 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB3HFHYNFRL7VAFMZ56YGBTY5AUHJAVCNFSM6AAAAABFHVV5UGVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTSOJYGIZTAMBSGI>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
ritvikrao
approved these changes
Apr 21, 2024
RMeli
reviewed
Jul 4, 2024
Comment on lines
+371
to
+378
| # assume HPC installation | ||
| include(CMakePrintHelpers) | ||
| find_package(EnvModules REQUIRED) | ||
| find_package(PkgConfig REQUIRED) | ||
| if(EnvModules_FOUND) | ||
| #at least get libfabric loaded if it isn't already | ||
| env_module(load libfabric) | ||
| endif() |
There was a problem hiding this comment.
I think this assumption is too restrictive. It does not seem to work when building Charm++ with Spack, since libfabric is provided as a Spack package and not through modules.
There was a problem hiding this comment.
It makes sense for the standard case of building on a DOE machine, but could present a problem for Spack. Is there is a simple way to test for the Spack case and then extract the necessary information to accomplish the same smooth build this code accomplishes with LMOD and PkgConfig?
There was a problem hiding this comment.
I think the simplest way is to setup Charm++ as a develop build and introduce changes to the build system and Spack package until it works. I started working on it but got stuck and then other stuff took priority.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This supports the CXI interface for Cassini (AKA Slingshot-11) within the OFI machine layer. No application level changes are required, but a variety of command line options are provided to configure the memory pool and the selection of cxi interfaces.
Note: this does require the use of the memory pool in order to efficiently support the FI_MR_ENDPOINT mode of memory registration. As the time cost of registering individual messages would be otherwise be entirely too high.
Charmrun has been configured to wrap srun and currently assumes PMI2 with cray extensions for launching.
The build system has been set up to autodetect CXI and enable support for it accordingly. For compatibility purposes, it also supports the use of cxi on the build line, but that should not be necessary on most HPE systems with proper LMOD environments.