Error accurred in parallel in cluster

Submitted by wangshuai on Mon, 03/23/2015 - 01:22

I have installed gcc4.7.0, openmpi-1.6.5 and LIGGGGHTS-3.1.0, OpenFOAM-2.3.1, CFDEM-2.3.1. When I test under OpenFOAM and under LIGGGHTS, it works well. However, when I want to test ErgunTestMPI under CFDEM, some errors occur like below:

Create time
Create mesh for time = 0
Reading field p

Reading physical velocity field U
Note: only if voidfraction at boundary is 1, U is superficial velocity!!!

Reading momentum exchange field Ksl

Reading voidfraction field voidfraction = (Vgas/Vparticle)

Creating dummy density field rho

Reading particle velocity field Us

Reading/calculating face flux field phi

Selecting incompressible transport model Newtonian
Selecting turbulence model type LESModel
Selecting LES turbulence model Smagorinsky
Selecting LES delta type cubeRootVol
SmagorinskyCoeffs
{
ce 1.048;
ck 0.094;
}

Reading g
Selecting locateModel engine
Selecting dataExchangeModel twoWayMPI
Starting up LIGGGHTS for first time execution
*** The MPI_Comm_rank() function was called before MPI_INIT was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[mnode:2654] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed!

I don't know how to deal with it, errors occurs after "Starting up LIGGGHTS for first time execution", namely, it can not execute "../DEM/in.liggghts_run", could anyone help me?

The executions are as follow:
in DEM, I execute "mpirun -np 4 ./lmp_fedora_fpic < in.liggghts.init", it works well;
then in CFD, I execute "blockMesh", "decomposePar", "cfdemSolverPiso", and then errors accur like above.

j-kerbl's picture

j-kerbl | Mon, 03/23/2015 - 11:05

Hi,

this line "mpirun -np 4 ./lmp_fedora_fpic < in.liggghts.init" confuses me a little bit.
Did you set a link to the executable or why is it in the same folder?

Does your liggghts executable path match the one you provided in the bashrc for the coupling?

Cheers
Josef

wangshuai | Mon, 03/23/2015 - 14:59

Hi,
I did not set the lliggghts excutable, and I copy the lmp_fedora_fpic file under the DEM, so I can type "mpirun -np 4 ./lmp_fedora_fpic < in.liggghts.init" and it worked well. However, I think the errors accurred becuase of the MPI. And I tried to find the method to deal with it, but failed. So, I install the software once again, tomorrow I will show you results. Finally, my bashrc file like below:

# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# User specific aliases and functions
export LANG=zh_CN.GB18030
export LD_LIBRARY_PATH=/home/aa/software/gmp/lib:/home/aa/software/mpfr/lib:/home/aa/software/mpc/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/home/aa/software/gcc/lib64:$LD_LIBRARY_PATH
export PATH=/home/aa/software/gcc/bin:$PATH
export PATH=/home/aa/software/cmake/bin:$PATH
source $HOME/OpenFOAM/OpenFOAM-2.1.1/etc/bashrc
export CFDEM_VERSION=PUBLIC
export CFDEM_PROJECT_DIR=$HOME/CFDEM/CFDEMcoupling-$CFDEM_VERSION-$WM_PROJECT_VERSION
export CFDEM_SRC_DIR=$CFDEM_PROJECT_DIR/src
export CFDEM_SOLVER_DIR=$CFDEM_PROJECT_DIR/applications/solvers
export CFDEM_DOC_DIR=$CFDEM_PROJECT_DIR/doc
export CFDEM_UT_DIR=$CFDEM_PROJECT_DIR/applications/utilities
export CFDEM_TUT_DIR=$CFDEM_PROJECT_DIR/tutorials
export CFDEM_PROJECT_USER_DIR=$HOME/CFDEM/$LOGNAME-$CFDEM_VERSION-$WM_PROJECT_VERSION
export CFDEM_bashrc=$CFDEM_SRC_DIR/lagrangian/cfdemParticle/etc/bashrc
export CFDEM_LIGGGHTS_SRC_DIR=$HOME/LIGGGHTS/LIGGGHTS-PUBLIC/src
export CFDEM_LIGGGHTS_MAKEFILE_NAME=fedora_fpic
export CFDEM_LPP_DIR=$HOME/LIGGGHTS/mylpp/src
export CFDEM_PIZZA_DIR=$HOME/LIGGGHTS/PIZZA/gran_pizza_17Aug10/src
. $CFDEM_bashrc

Best regards.
WANG Shuai

j-kerbl's picture

j-kerbl | Tue, 03/31/2015 - 10:47

Hi Wang!

so your LIGGGHTS actually is in $HOME/LIGGGHTS/LIGGGHTS-PUBLIC/src ?
Which mpi version are you using? Or more specific, did you use the libraries and the mpirun (mpiexec) executable from the same installation?

Cheers
Josef