Fix insert with 128 procs

Submitted by sbateman on Mon, 03/04/2013 - 18:39

I'm trying to run a simulation using fix insert/rate/region to generate a settled bed of >1 million particles. The inserted particles should be distributed evenly across all processors. When I tried to run with 128 processors on an HPC system, I got this error:
ERROR on proc 31: Failed to allocate -68719476576 bytes for array FixInsert::xnear (memory.cpp:45)

After a bit of debugging, I traced it back to this section of code from fix_insert.cpp, lines 661-670:

// remove fractions < 1%
// normalize fraction_local_all so sum across processors is 1

fraction_local_all_sum = 0.;
for(int iproc = 0; iproc < nprocs; iproc++)
{
if(fraction_local_all[iproc] < 0.01)
fraction_local_all[iproc] = 0.;
fraction_local_all_sum += fraction_local_all[iproc];
}

Since each processor has 1/128 of the total number of particles, the fractions are all <1%, and so the fraction_local_all_sum becomes 0, resulting in a later divide-by-zero.

Is there a reason for this restriction on insertion fraction <0.01? I am tempted to just set this to something like 0.00001 instead and get on with my simulation, but I want to know if there is a reason not to. :)

Thanks.

ckloss's picture

ckloss | Mon, 03/11/2013 - 21:39

Hi Sam,

thanks for letting us know!

The idea was to get around the problem when say 1 or 2 very narrow domains are involved. Since LIGGGHTS has parallelized the insertion itself (as opposed to LAMMPS) and inter-proc insertions are tedious to be checked for overlap, the idea was to omit such insertions with this criterion.

I was not thinking of such a configuration as in your case. I'll think about a way to patch this for the next release

>>I am tempted to just set this to something like 0.00001 instead and get on with my simulation,
>>but I want to know if there is a reason not to. :)
Actually not in your case, just feel free to do so.

Thanks, cheers
Christoph

ckloss's picture

ckloss | Mon, 03/25/2013 - 15:31

Hi Sam,

I will change the implementation to something like

if(fraction_local_all[iproc] < 0.01/nproc)

This will be released with 2.3.1. If you should still have troubles, please let me know!

Thanks and cheers
Christoph