CFDEM installation within an Ubuntu 16.04 container with OpenFOAM already installed and working

Submitted by AlexisEspinosa on Fri, 11/09/2018 - 07:53

Dear CFDEM & LIGGGTHS developers and guru users,

I’m failing to install correctly CFDEM, although my tests so far are limited to the LIGGGHTS step. And, more specifically, with a problem with VTK and MPI.
I’m installing CFDEM & LIGGGTHS within a container that runs on ubuntu:16.04 with OpenFOAM-4.x already installed in it. The OpenFOAM-4.x was installed with MPICH 3.1.4.

INTRO

Previously, I already had built an OpenFOAM-4.x with MPICH 3.1.4 container (The use of MPICH is a restriction for our systems. As we need to hook it to work with the MPICH already installed in Pawsey supercomputers. Shifter does this hooking in our systems instead of docker. This container already works fine in our systems.

This container can be pulled from here (tag:4.x):
https://hub.docker.com/r/alexisespinosa/openfoam
And the Dockerfile can be found here:
https://github.com/AlexisEspinosaGayosso/OpenFOAMContainers/tree/master/...

So far so good.

But now I was asked to also install CFDEM to the container. Basically I’m building another image from the originally working image:

FROM alexisespinosa/openfoam:4.x

I have made three different tries so far without success. First, second and third tries are described below. I would prefer to make the third try to work (if you can guide me with it) but, anyway, I will describe all of them in chronological order.

FIRST TRY

My first try was to follow your instructions in the documentation and install vtk and numpy with apt-get (as root in the container):

RUN apt-get install libvtk6.2 libvtk6-dev python-numpy

Unfortunately, the libvtk6 installation installed openmpi too, which I feel will create some messy stuff with the already installed (and needed) MPICH. Anyway, I just gave it a try:

WORKDIR $LIGGGHTSINSTDIR/LIGGGHTS-PUBLIC/src
RUN make auto; exit 0
RUN make clean-auto
RUN make auto

The complete Dockerfile (Dockerfile.apt) can be found here:
https://github.com/AlexisEspinosaGayosso/OpenFOAMContainers/tree/master/...

But this procedure gave me an error (The last part of the log is pasted here:)

mpicxx -lstdc++ -O2 dump_decomposition_vtk.o compute_erotate.o neigh_respa.o atom.o particleToInsert_multisphere.o sort_buffer.o fix_heat_gran_conduction.o compute_ke_atom.o fix_efield.o particleToInsertSuperquadric.o math_extra.o input_mesh_tri.o fix_cfd_coupling_convection.o run_time.o fix_multisphere.o reader_native.o fix_check_timestep_gran.o info.o atom_vec_charge.o fix_wall.o compute_com_molecule.o property_registry.o fix_heat_gran.o compute_rdf.o neigh_half_nsq.o fix_nve_sphere.o fix_sph_density_corr.o modify_liggghts.o write_dump.o fix_ave_histo.o fix_template_sphere.o input.o create_atoms.o fix_nve_asphere_noforce.o neigh_list.o region_plane.o fix_momentum.o fix_property_atom_tracer_stream.o mesh_module_stress.o fix_lineforce.o pair_sph_artvisc_tenscorr.o region_mesh_tet.o neigh_request.o improper_hybrid.o compute_ke.o pair_soft.o cfd_datacoupling_mpi.o tri_mesh.o library.o compute_atom_molecule.o image.o fix_insert.o fix_spring_self.o pair_hybrid.o dihedral.o angle_hybrid.o dump_movie.o atom_vec_line.o fix_contact_property_atom_wall.o compute_pressure.o compute_pe_atom.o compute_msd_molecule.o atom_vec_sph_var.o dihedral_hybrid.o dump_euler_vtk.o neigh_bond.o superquadric.o fix_template_multiplespheres.o dump_particle.o group.o region_prism.o fix_wall_gran.o procmap.o fix_move.o fix_wall_region.o fix_template_superquadric.o fix_enforce2d.o region_cone.o atom_vec_sphere_w.o compute_property_local.o fix_nve_asphere.o create_box.o dump_mesh.o irregular.o signal_handling.o fix_cfd_coupling.o special.o region_cylinder.o compute_pair_gran_local.o fix_check_timestep_sph.o fix_property_global.o fix_setforce.o atom_vec_sph.o fix_move_mesh.o granular_wall.o read_dump.o minimize.o fix_deform.o fix_read_restart.o compute_com.o dump_custom_vtk.o rotate.o reader.o dump_local_gran.o fix_region_variable.o read_restart.o compute_erotate_asphere.o fix_store.o compute_group_group.o fix_buoyancy.o atom_vec_molecular.o universe.o fix_cfd_coupling_force_implicit.o compute_centro_atom.o fix_box_relax.o change_box.o compute_reduce_region.o compute_pe.o fix_print.o fix_external.o fix_sph_density_continuity.o fix_insert_pack.o region_sphere.o region_intersect.o fix_deform_check.o output.o region_wedge.o compute.o memory.o fix_property_atom_region_tracer.o fix_planeforce.o fix_ave_correlate.o compute_coord_atom.o neigh_derive.o compute_slice.o dump_atom_vtk.o domain.o math_extra_liggghts_superquadric.o compute_reduce.o mesh_module_stress_servo.o compute_rigid.o fix_nve_superquadric.o fix_store_state.o region.o respa.o fix_rigid.o compute_cna_atom.o fix_property_atom_tracer.o input_multisphere.o fix_nve_sph.o region_union.o fix_gravity.o neighbor.o fix_massflow_mesh_sieve.o compute_gyration.o modified_andrew.o fix.o write_restart.o pair_gran.o fix_neighlist_mesh.o fix_contact_property_atom.o min_cg.o fix_nve_asphere_base.o compute_erotate_sphere.o compute_cluster_atom.o pair_sph.o cfd_datacoupling_file.o fix_nve.o fix_continuum_weighted.o compute_ke_multisphere.o dump_mesh_stl.o fix_multicontact_halfspace.o compute_erotate_superquadric.o delete_bonds.o force.o finish.o variable.o pair_line_lj.o dump_custom.o dump_custom_vtm.o velocity.o contact_models.o granular_pair_style.o bond.o displace_atoms.o improper.o fix_fiber_spring_simple.o reader_xyz.o fix_nve_sph_stationary.o min_linesearch.o fix_spring.o custom_value_tracker.o neigh_gran.o bond_hybrid.o compute_stress_atom.o fix_respa.o integrate.o modify.o tri_mesh_planar.o properties.o cfd_regionmodel_none.o fix_viscous.o dump_image.o compute_erotate_sphere_atom.o particleToInsert.o atom_vec_sphere.o compute_gyration_molecule.o compute_temp.o error.o fix_wall_region_sph.o set.o fix_property_atom.o compute_contact_atom_gran.o fix_nve_line.o neigh_stencil.o random_mars.o compute_contact_atom.o compute_displace_atom.o read_data.o fix_massflow_mesh.o multisphere.o dump.o compute_property_molecule.o main.o fix_base_liggghts.o fix_particledistribution_discrete.o fix_wall_sph.o compute_bond_local.o fix_cfd_coupling_force.o fix_cfd_coupling_convection_species.o thermo.o compute_erotate_multisphere.o update.o dump_local_gran_vtk.o fix_mesh.o lammps.o fix_dt_reset.o container_base.o citeme.o mesh_module.o random_park.o bounding_box.o fix_insert_rate_region.o fix_nve_noforce.o dump_local.o fix_mesh_surface.o atom_vec_tri.o delete_atoms.o write_data.o min.o tet_mesh.o fix_sph_pressure.o fix_scalar_transport_equation.o fix_store_force.o neigh_half_multi.o pair.o angle.o lattice.o fix_freeze.o fix_drag.o replicate.o pair_hybrid_overlay.o fix_sph.o cfd_datacoupling.o fix_addforce.o fix_cfd_coupling_convection_impl.o fix_ave_euler.o fix_ave_time.o compute_property_atom.o fix_lb_coupling_onetoone.o input_mesh_tet.o compute_msd.o fix_minimize.o atom_vec_full.o fix_template_multisphere.o fix_contact_history.o multisphere_parallel.o verlet.o dump_mesh_vtk.o kspace.o timer.o atom_vec_ellipsoid.o atom_vec_hybrid.o dump_vtk.o compute_inertia_molecule.o global_properties.o fix_sph_density_summation.o comm.o fix_multisphere_comm.o fix_multisphere_break.o compute_nparticles_tracer_region.o library_cfd_coupling.o fix_adapt.o fix_diam_max.o fix_contact_history_mesh.o run.o atom_vec_superquadric.o fix_ave_spatial.o mesh_mover_linear.o bond_harmonic.o math_extra_liggghts_nonspherical.o neigh_half_bin.o dump_mesh_vtm.o region_neighbor_list.o dump_xyz.o fix_nve_limit.o atom_vec_atomic.o fix_ave_atom.o atom_map.o random.o neigh_full.o fix_insert_stream.o atom_vec.o region_block.o mesh_module_liquidtransfer.o neigh_dummy.o mesh_mover_rotation.o fix_aveforce.o fix_spring_rg.o pair_gran_proxy.o -Wl,-rpath,/usr/lib/x86_64-linux-gnu/ -L/usr/lib/x86_64-linux-gnu/ -lmpich -lmpichcxx -lvtkCommonCore-6.2 -lvtkIOCore-6.2 -lvtkIOXML-6.2 -lvtkIOLegacy-6.2 -lvtkIOImage-6.2 -lvtkCommonDataModel-6.2 -lvtkParallelCore-6.2 -lvtkParallelMPI-6.2 -lvtkCommonExecutionModel-6.2 -lvtkFiltersCore-6.2 -lvtksys-6.2 -lvtkCommonMisc-6.2 -lvtkCommonTransforms-6.2 -lvtkCommonMath-6.2 -lvtkIOXMLParser-6.2 -lvtkCommonSystem-6.2 -lvtkDICOMParser-6.2 -lvtkmetaio-6.2 -lvtkIOParallelXML-6.2 -o ../lmp_auto
/usr/lib/libmpicxx.so: undefined reference to `MPIR_Op_set_cxx'
/usr/lib/libmpicxx.so: undefined reference to `MPIR_Keyval_set_proxy'
/usr/lib/libmpicxx.so: undefined reference to `MPIR_Errhandler_set_cxx'
collect2: error: ld returned 1 exit status
Makefile:1372: recipe for target '../lmp_auto' failed
make[1]: *** [../lmp_auto] Error 1
make[1]: Leaving directory '/MySoftware/LIGGGHTS/LIGGGHTS-PUBLIC/src/Obj_auto'
Makefile:106: recipe for target 'auto' failed
make: *** [auto] Error 2

SECOND TRY

As the use of the apt-get installed vtk was not my preferred choice (because of the possible mpi mess) I decided to download and install VTK and use it. VTK was not installed under /usr but into /MySoftware/VTK/VTK-install :

#...........
#Downloading VTK
ARG VTKBASEDIR=/MySoftware/VTK
WORKDIR $VTKBASEDIR
RUN git clone git://vtk.org/VTK.git \
&& cd VTK \
&& git fetch origin \
&& git rebase origin/master
.
.
.
#...........
#Compiling VTK from the downloaded VTK
ARG VTKBUILDDIR=$VTKBASEDIR/VTK-build
ARG VTKSOURCEDIR=${VTKBASEDIR}/VTK
ARG VTKINSTDIR=$VTKBASEDIR/VTK-install
ARG VTKMAKEOPTIONS="-j4"
WORKDIR $VTKBUILDDIR
RUN cmake ${VTKSOURCEDIR} -DBUILD_SHARED_LIBS=on -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=${VTKINSTDIR} \
&& make ${VTKMAKEOPTIONS} \
&& make install

Compilation and installation of VTK was successful. Obviosuly, I did not installed vtk with apt-get (and then no openmpi was installed), just numpy. But the compilation of LIGGGHTS also failed. Compilation this time was done pointing to the path where I installed VTK by modifying the Makefile.user file (VTK version is 8.2):

#...........
#Modifying the LIGGGHTS makefile
WORKDIR $LIGGGHTSINSTDIR/LIGGGHTS-PUBLIC/src
RUN make auto; exit 0
RUN sed -i $'s@^#VTK_INC_USR=-I/path/to/vtk/include@#VTK_INC_USR=-I/path/to/vtk/include\\\nVTK_INC_USR=-I'"${VTKINSTDIR}"'/include/vtk-8.2@' ./MAKE/Makefile.user \
&& sed -i $'s@^#VTK_LIB_USR=-L/path/to/vtk/lib@#VTK_LIB_USR=-L/path/to/vtk/lib\\\nVTK_LIB_USR=-I'"${VTKINSTDIR}"'/lib@' ./MAKE/Makefile.user \
&& make clean-auto
.
.
.
#...........
#Installing LIGGGTHS
RUN make auto

The complete Dockerfile (Dockerfile.vtk) can be found here:
https://github.com/AlexisEspinosaGayosso/OpenFOAMContainers/tree/master/...

But compilation of LIGGGHTS failed, and the error is:

Creating list of contact models completed.
make[1]: Entering directory '/MySoftware/LIGGGHTS/LIGGGHTS-PUBLIC/src/Obj_auto'
Makefile:940: *** 'Could not determine suitable appendix of VTK library with VTK_INC="-I/MySoftware/VTK/VTK-install/include/vtk-8.2", VTK_LIB="-I/MySoftware/VTK/VTK-install/lib" and VTK_APPENDIX="-8.2"'. Stop.
make[1]: Leaving directory '/MySoftware/LIGGGHTS/LIGGGHTS-PUBLIC/src/Obj_auto'
Makefile:106: recipe for target 'auto' failed
make: *** [auto] Error 2

THIRD TRY

Then, I tried to use the native VTK that comes with OpenFOAM-4.x. The installation of VTK is very similar to the previous test, but this time cmake uses the one under the Paraview directory, and again the installation is in /MySoftware/VTK/VTK-install:

#...........
#Compiling VTK from OF directory
ARG VTKBASEDIR=/MySoftware/VTK
ARG VTKBUILDDIR=$VTKBASEDIR/VTK-build
ARG VTKSOURCEDIR=${WM_THIRD_PARTY_DIR}/ParaView-${ParaView_VERSION}/VTK
ARG VTKINSTDIR=$VTKBASEDIR/VTK-install
ARG VTKMAKEOPTIONS="-j4"
WORKDIR $VTKBUILDDIR
RUN cmake ${VTKSOURCEDIR} -DBUILD_SHARED_LIBS=on -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=${VTKINSTDIR} \
&& make ${VTKMAKEOPTIONS} \
&& make install

Compilation and installation of VTK from the ThirdParty Paraview directory was successful.

The compilation of LIGGGHTS was tried in a similar way as previously (now VTK version is 7.1):
#...........
#...........
#Modifying the LIGGGHTS makefile
WORKDIR $LIGGGHTSINSTDIR/LIGGGHTS-PUBLIC/src
RUN make auto; exit 0
RUN sed -i $'s@^#VTK_INC_USR=-I/path/to/vtk/include@#VTK_INC_USR=-I/path/to/vtk/include\\\nVTK_INC_USR=-I'"${VTKINSTDIR}"'/include/vtk-7.1@' ./MAKE/Makefile.user \
&& sed -i $'s@^#VTK_LIB_USR=-L/path/to/vtk/lib@#VTK_LIB_USR=-L/path/to/vtk/lib\\\nVTK_LIB_USR=-I'"${VTKINSTDIR}"'/lib@' ./MAKE/Makefile.user \
&& make clean-auto
.
.
.
#...........
#Installing LIGGGTHS
RUN make auto

The Dockerfile (Dockerfile.paraview) can be found here:
https://github.com/AlexisEspinosaGayosso/OpenFOAMContainers/tree/master/...

But compilation of LIGGGHTS failed again. But now with a similar (but different) error. And the error is:

Creating list of contact models completed.
make[1]: Entering directory '/MySoftware/LIGGGHTS/LIGGGHTS-PUBLIC/src/Obj_auto'
Makefile:940: *** 'Could not determine suitable appendix of VTK library with VTK_INC=“-I/MySoftware/VTK/VTK-install/include/vtk-7.1", VTK_LIB=“-I/MySoftware/VTK/VTK-install/lib" and VTK_APPENDIX="-7.1"'. Stop.
make[1]: Leaving directory '/MySoftware/LIGGGHTS/LIGGGHTS-PUBLIC/src/Obj_auto'
Makefile:106: recipe for target 'auto' failed
make: *** [auto] Error 2

QUESTIONS

1. Can you guide me to install correctly CFDEM and LIGGGHTS within this “ubuntu-16.04 - OpenFOAM-4.x” container? I will follow what you suggest is best. Although the option I like the most is the “third option” where VTK from the ThirdParty Paraview is used. In this way I do not need to download any further VTK, all versions are the same, and no openmpi is installed. All using MPICH 3.4.1 and VTK 7.1 same as the OpenFOAM installation.

Many Thanks,

Alexis Espinosa
Pawsey Supercomputing Centre

Daniel Queteschiner | Fri, 11/09/2018 - 13:05

It seems to me that you are prefixing the library path VTK_LIB_USR with -I instead of -L marking it as an include directory of header files instead of a directory for library files.

AlexisEspinosa | Tue, 11/13/2018 - 06:32

Thanks a lot, that was a typo. It's no good to copy/paste without carefully check!
Just for possible readers, After correcting the typo, I was able to finish the installation by defining CFDEM_ADD_LIB_PATHS & CFDEM_ADD_LIBS as explained in:
https://www.cfdem.com/forums/cfdem-installation-cluster

Regards,
Alexis

AlexisEspinosa | Tue, 11/13/2018 - 06:37

I forgot to mention that I also needed to turn this variable on during the installation of VTK.
Cheers