Multisphere Particle that Extends across Multiple Subdomains

Submitted by estefan31 on Tue, 07/11/2017 - 08:28

I'm trying to model a structure sitting atop a granular assembly of multisphere particles. The structure is modeled as 1 large box-shaped multisphere particle containing about 300 constituent spheres. The structure as a whole extends across about 3 to 5 processor sub-domains in a simulation run with 80 or more cores. My problem is that when I insert this structure, the number of ghost atoms increases from about 100,000 to 1,000,000. The simulation becomes considerably slower, and the run time increases by a factor of about 20. I also noticed that the simulation involving the structure takes about the same amount of time whether I use 20 cores or 320 cores. When I don't insert a structure, I do get shorter run times with more cores as expected. Can someone recommend any settings I can play with to reduce the number of ghost atoms in my simulation so that my simulations will be faster when inserting such a large multisphere particle that extends across several sub-domains?

Forums:

estefan31 | Thu, 07/13/2017 - 20:18

Just to add more info, the structure is 87 mm long in the y-direction, 10 mm long in the z direction, and 26 mm long in the x-direction. The x-boundaries are periodic and also 26 mm apart. I restrict the structure from moving in the x-direction and from rotating about the y- and z- axes (so it's basically a plane-strain structure). I tried using the "communicate multi" command but it causes a significant portion of my structure to be deleted. Is there a way to manually adjust the size and locations of the sub-domains? At this point I'm convinced the issue has to do with the structure being one large multisphere that extends across too many subdomains, so I hope someone can lend any advice at all on how to work around this issue. I did get a warning that the bounding sphere is longer than the subdomain dimensions. There's not much I can do about the dimensions of my problem, but maybe there is something I can do about the processor communications?

aaigner's picture

aaigner | Sat, 07/22/2017 - 22:04

Hi!
You have somehow a systematic problem.
This one big multisphere particle is more or less the worst which can happen in a DEM simulation. This requires so much inefficient communication.
Can you achieve the same result with a different approach?
Stl-mesh or a region without integration, constant forces,... anything like that?
I cant give you a better advise since I do not know your exact set-up.

Regards
Andreas

estefan31 | Mon, 07/24/2017 - 05:45

We did consider the approaches you suggested but ultimately decided to keep our original approach. As it turns out, LIGGGHTS-Premium has the functionality we need (load balancing and 6 DOF rigid body motion). Our group has contacted CFDEM about pricing and are awaiting a response.

That being said, here is what worked out for me. Using "communicate multi" instead of "communicate single" significantly reduces the number of ghost atoms for each sub-domain and hence makes our simulations much faster. The problem is that when the boundaries of our subdomains strike through our large multisphere particle, that multisphere particle gets truncated along that boundary. Particles are erased and the multisphere particle is no longer usable. However, if our large multisphere particle is completely contained within one sub-domain at the start of our simulation, it remains intact. We had to find a way to implement load balancing so that we could adjust the sizes and locations of the sub-domains to ensure this. LAMMPS still has balance.cpp and balance.h files, and these files compiled successfully with LIGGGHTS 3.7. I've seen past forum users be warned to use these files at their own risk. Indeed, if I extend my simulations to more than 1 node, large numbers of particles are erased after using the load balancing routine. But on only 1 node with up to 24 cores, load balancing works for most cases and so far my simulations are running fine (albeit about 3x slower since I'm forced to use a fraction of the total number of processors I previously used).

This is only a partial solution. A full solution would require the load balancing routine that is part of the Premium package. We will keep contacting your group about getting a quote for LIGGGHTS-Premium :-)