Possible bug in twoWayMPI giveData / getData

Submitted by lavdwall on Fri, 04/10/2020 - 15:13

Dear,

I have noticed some weird behavior when transferring data between LIGGGHTS and OpenFOAM.

I have adjusted one of the solvers to modify some custom atom properties in OpenFOAM and then sending those modified properties back to LIGGGHTS to be transported there together with the particles. So basically, in every CFD time step, I do the following:

// getting data from LIGGGHTS
particleCloud_.dataExchangeM().getData(partFieldName_, "scalar-atom", partField_);
// modifications to partField
for(int index = 0; index < particleCloud_.numberOfParticles(); ++index) partField_[index][0] = ...;
// sending modified data back to LIGGGHTS
particleCloud_.dataExchangeM().giveData(partFieldName_, "scalar-atom", partField_);

However, the values that are received by LIGGGHTS are not the same as those that I send via giveData.
To make sure that this is not related to a bug in my own code, I also tested it by adjusting the commToDEM function in the LaEuScalarTemp forceModel. More specifically, I added a few giveData/getData commands immediately after each other and the first element of the heat flux is printed after each giveData/getData call:

particleCloud_.dataExchangeM().giveData(partHeatFluxName_, "scalar-atom", partHeatFlux_);
Info << "gave partHeatFlux_[0][0] = " << partHeatFlux_[0][0] << endl;
particleCloud_.dataExchangeM().getData(partHeatFluxName_, "scalar-atom", partHeatFlux_);
Info << "got partHeatFlux_[0][0] = " << partHeatFlux_[0][0] << endl;
particleCloud_.dataExchangeM().giveData(partHeatFluxName_, "scalar-atom", partHeatFlux_);
Info << "gave partHeatFlux_[0][0] = " << partHeatFlux_[0][0] << endl;
particleCloud_.dataExchangeM().getData(partHeatFluxName_, "scalar-atom", partHeatFlux_);
Info << "got partHeatFlux_[0][0] = " << partHeatFlux_[0][0] << endl;
particleCloud_.dataExchangeM().giveData(partHeatFluxName_, "scalar-atom", partHeatFlux_);
Info << "gave partHeatFlux_[0][0] = " << partHeatFlux_[0][0] << endl;
particleCloud_.dataExchangeM().getData(partHeatFluxName_, "scalar-atom", partHeatFlux_);
Info << "got partHeatFlux_[0][0] = " << partHeatFlux_[0][0] << endl;

By doing so, the output of the packedBedTemp tutorial in the first time step is as below. Clearly, although the first giveData seems to work properly, it messes up after 2 calls. I noticed that the values are often off by a factor equal to the number of processors used by the simulations, although this is not always the case for my own code.

gave partHeatFlux_[0][0] = -1.20515
got partHeatFlux_[0][0] = -1.20515
gave partHeatFlux_[0][0] = -1.20515
got partHeatFlux_[0][0] = -2.41031
gave partHeatFlux_[0][0] = -2.41031
got partHeatFlux_[0][0] = -4.82061
gave partHeatFlux_[0][0] = -4.82061
got partHeatFlux_[0][0] = -9.64123

Has somebody experienced the same issue? Is this a bug or am I doing something wrong?

Thanks,
Laurien

Daniel Queteschiner's picture

Daniel Queteschiner | Tue, 04/14/2020 - 16:32

Is this a bug or am I doing something wrong?
You are doing it wrong ;-)
To understand what is going on here, you have to look at the implementation of the data coupling in
https://github.com/CFDEMproject/LIGGGHTS-PUBLIC/blob/master/src/cfd_data...
Inspecting the pull_mpi/push_mpi functions reveals that data is collected using the MPI function MPI_Allreduce wit the operand MPI_SUM.
In other words, each time you call the data exchange functions the atom property is added to the old value.
In further consequence, this means that for non-cumulative data you need to clear the data array to zero between data exchange steps.

lavdwall | Wed, 04/15/2020 - 11:17

OK, thanks for the reply. I am a bit unfamiliar with those MPI commands, but I will look into it.

On the OpenFOAM side, I am already reallocating the array after every call... So, am I correct to understand that I should clear the LIGGGHTS atom data array (those assigned via fix property/atom) after every getData call? Can I clear those data arrays from within the CFD/OpenFOAM code (and if yes, how)? I prefer not to make any changes in the LIGGGHTS code, if that is possible..

Laurien

hoehnp | Sat, 09/12/2020 - 18:14

Hi Laurien,

sorry for hijacking this thread. I am also working with adding more properties to the data exchange and also face
problems. It would be nice if we could have a separate conversation to discuss with you my and your implementation
problems. If you are interested please drop me a mail under patrick.hoehn@tu-clausthal.de.

Patrick