Possible bug when calling lammp_open() with certain dump styles

Submitted by frankiebetancourt on Fri, 09/07/2018 - 23:19

Summary:
========
Using the lammmps_open() function on certain dump styles involving VTK causes a
deadlock.

More info:
==========
When starting liggghts with the library call lammps_open(), and using the
following dump styles,

custom/vtk
custom/vtm
local/gran/vtk
mesh/gran/vtk
mesh/vtm

causes a deadlock when the lammps_open() MPI_Comm parameter is set to anything
other than MPI_COMM_WORLD.

Software Versions:
==================
liggghts: 3.8.0, commit 28301df8853491784b1d8b90533ea89b8c6af1e8
Operating System: CentOS Linux 7
MPI: Open MPI 2.1.1
g++: 4.8.5

Possible cause:
===============
The issue appears to be caused by using the VTK library call
vtkMPIController::Initialize(), which calls InitializeCommunicator() with the
argument vtkMPICommunicator::GetWorldCommunicator(), which ends up
instantiating a new vtkMPICommunicator class with the MPI_Comm handle
MPI_COMM_WORLD. This results in a vtkMPICommunicator with the default handle
MPI_COMM_WORLD, instead of the communicator that gets passed in when calling
lammps_open().

Files Affected:
===============
All of the following files contain the improper VTK calls:

dump_custom_vtk.cpp
dump_custom_vtm.cpp
dump_local_gran_vtk.cpp
dump_mesh_vtk.cpp
dump_mesh_vtm.cpp

Fix:
====
Instead of creating a new vtkController and initializing it with
vtkMPIController->Initialize(), initialize a new vtkMPICommunicatorOpaqueComm
with the lmp->world communicator, and set the communicator of the new
vtkMPIController with SetCommunicator() before calling SetGlobalController().

For example, in all of the previously mentioned files, the
vtkMultiProcessController is set up in the following way,

```C++
vtkMPIController *vtkController = vtkMPIController::New();
vtkController->Initialize();
vtkMultiProcessController::SetGlobalController(vtkController);
```

In order to make VTK use the correct MPI communicator, the following
modifications were made:

```C++
vtkMPIController *vtkController = vtkMPIController::New();
vtkMPICommunicator *vtkComm = vtkMPICommunicator::New();
vtkMPICommunicatorOpaqueComm opaqueComm(&lmp->world);
vtkComm->InitializeExternal(&opaqueComm);
vtkController->SetCommunicator(vtkComm);
vtkMultiProcessController::SetGlobalController(vtkController);
```

Additionally, the following header files need to be included in in the
files in order to make use of vtkMPICommunicatorOpaqueComm and
vtkMPICommunicator classes:

vtkMPI.h
vtkMPICommunicator.h