mpirun ImportError: need more than 1 processor to run with mpi

[janeyapyx]

I am trying to run Phoebe 2.3.50 with SLURM.
I would like to try the mode of "multiprocessing, run under MPI" for emcee sampling.
I turned on MPI in Phoebe and run in SLURM, single node with mpirun --use-hwthread-cpus ...
The node has
Thread(s) per core: 2
Core(s) per socket: 24
Socket(s): 2
I assume mpirun can detect up to 96 threads but Pheobe seems not to get that.
Anyone has any idea perhaps I have done something wrong? Maybe I should disable MPI in Phoebe and let SLURM handles the MPI?

Traceback (most recent call last):
File "emcee_01.py", line 6, in
import phoebe; import json
File "/cluster/home/jane/.conda/envs/astr/lib/python3.7/site-packages/phoebe/init.py", line 217, in
mpi = MPI()
File "/cluster/home/jane/.conda/envs/astr/lib/python3.7/site-packages/phoebe/init.py", line 121, in init
raise ImportError("need more than 1 processor to run with mpi")
ImportError: need more than 1 processor to run with mpi

Thank you!

[kecnry]

You shouldn't have to disable MPI from within PHOEBE (if you do so - then you're responsible for all the parallelization). Have you used the same setup with any other code or test script successfully?

PHOEBE's logic is essentially as follows:

from mpi4py import MPI as mpi
nprocs = mpi.COMM_WORLD.Get_size()
if nprocs == 1:
    raise ImportError("need more than 1 processor to run with mpi")

It might be worth just trying to run this within mpi and then slurm and see if mpi4py is detecting the appropriate number of processors in each case.

[janeyapyx]

Indeed, if I use mpirun then mpi4py.COMM_WORLD.comm_size() returns 1.
However, changing it to srun --mpi=pmi2 seems to solve part of the problem.
While it can now run under mpi environment, it doesn't seem like it is doing multiprocessing at the same time.
Is Phoebe designed to do mpi and multiprocessing at the same time?
Thank you very much for the help!

[kecnry]

Phoebe should detect you're within MPI and not use multiprocessing. But the detection might fail for your setup since you already have to pass some custom flags. Can you try the following, within mpirun/srun? This is how phoebe attempts to detect if you're within MPI. It does seem this was evaluating as True for you originally, or else that error could never have been raised. But there's a chance your --mpi=pmi2 is bypassing this and that's why you're not seeing the error anymore and its falling back on multiprocessing.

import os
print('OMPI_COMM_WORLD_SIZE' in evars or 'MV2_COMM_WORLD_SIZE' in evars or 'PMI_SIZE' in os.environ.keys())

All of that said.... if you're attempting to use emcee, then just using multiprocessing instead of MPI might be the easiest thing to do.

[janeyapyx]

With mpirun, 'PMI_SIZE' in os.environ.keys()) returns False, which it shouldn't, I will report to the cluster admin about this.
(on the other hand, evars didn't work for me -- NameError: name 'evars' is not defined)
With srun, 'PMI_SIZE' in os.environ.keys()) returns False (script then runs in serial), but adding the flag --mpi=pmi2 gives True (script handles by mpi).

I am happy to use multiprocessing, I just had the naive assumption that it would run faster with both mpi and multiprocessing working (I am new to parallelisation).

I think I have the answer I need for the code to work for now, thanks a lot!

[kecnry]

I'm sorry, I tried to simplify the snippet into a single line but missed the evars. It checks to see if any of those strings are in the environment variables, so use the following instead:

import os
evars = os.environ.keys()
print('OMPI_COMM_WORLD_SIZE' in evars or 'MV2_COMM_WORLD_SIZE' in evars or 'PMI_SIZE' in evars)

But I'm glad to hear that multiprocessing is working for you - that should do what you need for emcee as long as you run on a single node and nwalkers <= nprocs (otherwise you'll have idle processors that won't get tasks).