NCCL and NVSHMEM backends will not work on Alps, because these communication libraries only support execution on infiniband networks while Alps is based on libfabric+slingshot.
To run on more than one node, disable full autotuning and select one communication backend; this will disable autotuning of the backend.
Change from this:
! comm_backend = CUDECOMP_TRANSPOSE_COMM_MPI_P2P
comm_backend = 0 ! Enable full autotuning`
to this:
comm_backend = CUDECOMP_TRANSPOSE_COMM_MPI_P2P
!comm_backend = 0 ! Enable full autotuning
You can also set one of the following:
CUDECOMP_TRANSPOSE_COMM_MPI_P2P_PL
CUDECOMP_TRANSPOSE_COMM_MPI_A2A
NCCL and NVSHMEM backends will not work on Alps, because these communication libraries only support execution on infiniband networks while Alps is based on libfabric+slingshot.
To run on more than one node, disable full autotuning and select one communication backend; this will disable autotuning of the backend.
Change from this:
! comm_backend = CUDECOMP_TRANSPOSE_COMM_MPI_P2Pcomm_backend = 0 ! Enable full autotuning`
to this:
comm_backend = CUDECOMP_TRANSPOSE_COMM_MPI_P2P!comm_backend = 0 ! Enable full autotuningYou can also set one of the following:
CUDECOMP_TRANSPOSE_COMM_MPI_P2P_PLCUDECOMP_TRANSPOSE_COMM_MPI_A2A