EESSI-based GitHub Action for Continuous Integration

 

Description

This module sets up the European Environment for Scientific Software Installations (EESSI) for use in GitHub Workflows.

The European Environment for Scientific Software Installations (EESSI) is a collaboration between a number of academic and industrial partners in the HPC community to set up a shared stack of scientific software installations to avoid the installation and execution of sub-optimal applications on HPC resources. The software stack is intended to work on laptops, personal workstations, HPC clusters and in the cloud, which means the project will need to support different CPUs, networks, GPUs, and so on.

The EESSI project is supported by E-CAM, and forms the basis of the software stack used within the LearnHPC project (which is also supported by E-CAM).

EESSI can be leveraged in continuous integration (CI) workflows to easily provide the dependencies of an application. With this module we create a GitHub Action for EESSI so that it can be used within a projects CI on GitHub. By using the Action, you can use environment modules to resolve the dependencies of your application in highly predictable and reproducible way. This includes state-of-the-art compilers, MPI runtimes and mathematical libraries.

Documentation and source code

Documentation on our software repository here. See also the GitHub repository of the EESSI GitHub Action.

Share

March Module of the Month: DL_MESO (DPD) on Kokkos for enhanced performance portability

 

This work relates to the implementation of a performance portable version of DL_MESO (DPD) using the Kokkos library. It focuses on porting to DL_MESO (DPD) the first and second loops of the Verlet Velocity (VV) scheme for the time marching scheme. This allows to run DL_MESO on NVidia GPUs as well as on other GPUs or architectures (many-core hardware like KNL), allowing performance portability as well as separation of concern between computational science and HPC.

Description

The VV scheme is made of 3 steps:

  1. a first velocity and particle positions integration by Delta t/2,
  2. a force calculation, and
  3. a second velocity integration by Delta t/2.

Steps 1) and 2) are documented is the following two modules

DL_MESO (DPD) on Kokkos: Verlet Velocity step 1

DL_MESO (DPD) on Kokkos: Verlet Velocity step 2

Note: Kokkos is a C++ library, while DL_MESO (DPD) is in Fortran90 Language. The current implementation requires a transfer between Fortran to C++, due to the use of Fortran pointers not bound using the ISO_C_BINDING standard. This constraint will be removed in future versions of DL_MESO.

Practical application

With the advent of heterogeneous hardware, achieving performance portability across different architectures is one of the main challenges in HPC. In fact, while specific languages, like CUDA, can give best performance for the NVidia hardware, they cannot be used with different GPU vendors limiting the usage across supercomputers worldwide.

In this module we use Kokkos, developed at Sandia National Laboratories, which consist of several C++ templated libraries which provide the capability to offload a workload to several different architectures, taking care of the memory layout and transfer between host and device.

Documentation and source code

The modules documentation is available on our software repository here. The modules have also been pushed into DL_MESO git repository as explained in the modules documentation.

Share

March Module of the Month: n2p2 – Improved link to HPC MD software

 

This module improves the connection of n2p2 to HPC software, in particular to LAMMPS, by creating a pull request to the official LAMMPS repository. Furthermore, the build process for the n2p2 interface library is enhanced to allow for a selective activation of different interfaces. A first application is also supported: the user contribution of an n2p2 interface to CabanaMD which uses Kokkos to drive MD simulations with NNP support on GPUs.

Description

This module is based on n2p2 (NeuralNetworkPotentialPackage), a C++ code for generation and application of neural network potentials used in molecular dynamics simulations. The source code and documentation are located here:

Although n2p2 was already shipped with source files for patching LAMMPS before (see here) , the build process required manual intervention of users. To avoid this in future versions of LAMMPS a pull request was created to include the n2p2/LAMMPS interface by default as a user package. In order to conform with LAMMPS contribution guidelines multiple issues were resolved, triggering these changes/additions to LAMMPS and n2p2:

• Modify the CMake build process to search and include n2p2

• Create additional documentation about the build settings

• Create a suitable example which can be shipped with LAMMPS

• Adapt documentation of the LAMMPS “pair_style nnp” command

• Change n2p2 to conformwith LAMMPS “bigbig” settings

• Change the source files “pair_nnp.(cpp/h)” to conform with the LAMMPS coding style

Furthermore, the n2p2 build system was adapted to allow for multiple interfaces to other software packages, with an option to select only those of interest to the user. As a first application, the user contributed CabanaMD interface was integrated in the new build process. CabanaMD is an ECP proxy application which makes use of the Kokkos performance portability library and n2p2 to port neural network potentials in MD simulations to GPUs and other HPC hardware.

Practical applications

The integration of neural network potentials directly in LAMMPS via a user package with linkage to n2p2 will greatly enhance the visibility and user experience. User will also be able to retrieve information about the neural network potential method and its use directly on the LAMMPS documentation page. Modifying the n2p2 build process to allow for multiple interfacing software simplifies the development of CabanaMD. This contribution of n2p2 users can be viewed as a precursor of a Kokkos implementation of NNPs in LAMMPS. Ultimately, such an addition to n2p2/LAMMPS would be of great value for the community as it would allow for running molecular dynamics simulation with NNPs on GPUs.

A success story on the E-CAM developments on n2p2 is also available, which explains in more detail all the practical applications from this work: Implementation of High-Dimensional Neural Network Potentials .

Documentation and source Code

Module documentation is available on our software repository here.

Share

February Module of the Month: ALL library implementation in HemeLB, a CoE collaboration

 

This module describes the work done in E-CAM in cooperation with the HemeLB code from the CompBioMed Centre of Excellence.

HemeLB is a high performance lattice-Boltzmann solver optimised for simulating blood flow through sparse geometries, such as those found in the human vasculature. The code is used within the CompBioMed HPC Centre of Excellence H2020 project and is already highly optimised for HPC usage. Nevertheless, in an E-CAM workshop on the load balancing library ALL hosted at the Juelich Supercomputing Centre, a cooperation was set up in order to analyse and test whether the use of ALL could improve the existing scalability of the code.

ALL was designed to work with particle codes, therefore it was interesting to apply the library to a lattice-Boltzmann solver, which usually is not particle-based. The different grid points of the solution grid were designated as particles and since each of the grid-points already was assigned a workload, the sum of grid-point workloads could be used as domain work load.

As a result, it was demonstrated that the domain compositions provided by ALL show a better theoretical load distribution. Tests to check if this translates into better code performance are inconclusive as yet, due to hardware related issues on the testing platforms. However, these are currently under further investigation, and more definitive results about the performance of the ALL-provided domain decompositions can be expected in the near future. The results were part of an article about HemeLB, which was published in 2020[1] .

Documentation and source code

https://e-cam.readthedocs.io/en/latest/Meso-Multi-Scale-Modelling-Modules/modules/ALL_library/all_hemeLB_cooperation/hemeLBcooperation.html


References

Towards blood flow in the virtual human: efficient self-coupling of HemeLB
J. W. S. McCullough, R. A. Richardson, A. Patronis, R. Halver, R. Marshall, M. Ruefenacht, B. J. N. Wylie, T. Odaker, M. Wiedemann, B. Lloyd, E. Neufeld, G. Sutmann, A. Skjellum, D. Kranzlmüller and P. V. Coveney
Interface Focus 2020, 11: 20190119
DOI: http://dx.doi.org/10.1098/rsfs.2019.0119 (open access)

Share

January Module of the Month: MaZe, Mass-Zero Constrained Dynamics for Orbital Free Density Functional Theory

 

Description

The program performs Orbital-Free Density Functional Theory Molecular Dynamics (OF-DFT-MD) using the Mass-Zero (MaZe) constrained molecular dynamics approach described in [1].

This method enforces, at each time step, the Born-Oppenheimer condition that the system relaxes instantaneously to the ground state through the formalism of massless constraints. The adiabatic separation between the degrees of freedom is enforced rigorously, and the algorithm is symplectic and time-reversible in both physical and additional set of degrees of freedom.

The computation of the electronic density is carried out in reciprocal space through a plane-waves expansion so that the mass-zero degrees of freedom are associated to the Fourier coefficients of the electronic density field. The evolution of the ions is performed using Velocity-Verlet algorithm, while the SHAKE algorithm is used for computation of the additional degrees of freedom. The code can sample the NVE and the NVT ensemble, the latter through a Langevin thermostat.

The code was optimised to run on HPC machines, as explained in the software documentation. The proposed optimisations allow a reduction of the execution time by roughly 50% compared to the original version of the code.

Caption: MaZe optimisation of the electronic density at each nuclear step along an orbital-free DFT Born–Oppenheimer trajectory. Very high speed of convergence is achieved by interpreting the optimisation as a constraint solved via an original implementation of the SHAKE algorithm.  The number of iterations needed to converge the electronic density and the time per time step for MaZe (red) and standard conjugate gradients (blue) are compared for the indicated kinetic energy functionals (G_c is the energy cut-off).

Practical application

The code is intended for condensed matter physicists and for material scientists and it can be used for various purposes related to the subject. Even though some analysis tools are included in the package, the main goal of the software is to produce particles trajectories to be analysed in post-production by means of external software.

MaZe implements the orbital-free formulation of density functional theory, in which the optimisation of the energy functional is performed directly in terms of the electronic density without use of Kohn-Sham orbitals. This feature avoids the need for satisfying the orthonormality constraint among orbitals and allows the computational complexity of the code to scale linearly with the dimensionality of the system. The accuracy of the simulation relies on the choice of the kinetic energy functional, which has to be provided in terms of the electronic density alone.

Documentation and source code

The complete documentation is at this location. The source code is available from the E-CAM Gitlab under the MaZe project (software is under embargo until publication leveraging the developments is achieved. Contact code developers or info@e-cam2020.eu for more information.)

References

[1] Sara Bonella, Alessandro Coretti, Rodolphe Vuilleumier, Giovanni Ciccotti, “Adiabatic motion and statistical mechanics via mass-zero constrained dynamics”, Phys. Chem. Chem. Phys. 2020, 22, 10775-10785 DOI: 10.1039/D0CP00163E
Pre-print version (open access): https://arxiv.org/abs/2001.03556

Share

December Module of the Month: Load balancing for multi-GPU DL_MESO

 

Description

This module concerns the implementation of the E-CAM Load Balancing Library (ALL) in the multi-GPU version of DL_MESO_DPD code. The intention is to allow for better performance when modelling complex systems with DL_MESO_DPD, like large proteins or lipid bilayers, redistributing the work load across the GPUs.

ALL provides several schemes to find the ideal split of the work load : Tensor-Product method, Staggered Grid Method, Unstructured Mesh Method, Voronoi Mesh Method and Histogram-based Staggered Grid Method. Due to the orthogonal domain decomposition used in DL_MESO, the Tensor-Product scheme was used, which works well for non-staggered orthogonal meshes.

Practical application

A test case was implemented (see Figure 1 a), b) and c)) that reproduces 32k water beads initially scattered along a regular structure and then slowly agglomerating towards an unique large drop confined between two parallel surfaces. The system is divided across 8 GPUs and, for the purposes of the visualisation, we restrict ourselves to 32k particles. For a larger number of particles it would not be possible to simulate the system without load-balancing, since all the particles agglomerate to a subset of the available GPUs and one or more GPUs would run out of memory having to accommodate a large number of particles. Moreover, such a strong load imbalance drastically reduces the scalability of the application.

In Figure d) we see the time history of the load imbalance for each GPU when using the ALL library. Without load balancing the system would gradually diverge from the ideal value of 12.5%. You can find a video that shows the evolution of the load-balancing for this system in another software module.

Figure 1: Load imbalance in DL_MESO with ALL library for a water drop between two surfaces. Each colour represents different domain assigned to a different GPU: a) top view, b) perspective view, c) front view, d) load balance vs time

Source code

Further details on the implementation of ALL library in DL_MESO and the source code can be found in the E-CAM software repository here.

Share

November Module of the Month: PerGauss, Periodic Boundary Conditions for gaussian bases

 

Description

The module PerGauss (Per iodic Gauss ians) consists on an implementation of periodic boundary conditions for gaussian bases for the Quantics program package.

In quantum dynamics, the choice of coordinates is crucial to obtain meaningful results. While xyz or normal mode coordinates are linear and do not need a periodical treatment, particular angles, such as dihedrals, must be included to describe accurately the (photo-)chemistry of the system under consideration. In these cases, periodicity can be taken into account, since the value of the wave function and hamiltonian repeats itself after certain intervals.

Practical application

The module is expected to provide the quantum dynamics community with a more efficient way of treating large systems whose excited state driving forces involve periodic coordinates. When used on precomputed potentials (in G-MCTDH and vMCG), the model can improve the convergence since smaller grid sizes are needed. Used on-the-fly, it reduces considerably the amount of electronic structure computations needed compared to cartesian coordinates, since conformations that seemed far in the spanned space may be closer after applying a periodic transformation.

Source code

Currently PerGauss resides within the Quantics software package available upon request through gitlab. For more information see the PerGauss documentation here.

Share

Dask-traj

 

For analysis of molecular dynamics (MD) simulations MDTraj is a fast and commonly used analysis. However MDTraj has some restrictions such as (1) the whole trajectory needs to fit into memory, or gathering results becomes inconvenient; (2) the result of the computation also need to fit into memory, and (3) all processes need access to all the memory, preventing out-of-machine parallelisation and HPC scaling.

Dask-traj solves these restrictions by rewriting the MDTraj functions to work with Dask in order to achieve out-of-memory computations. Combined with dask-distributed this allows for out-of-machine parallelisation, essential for HPCs, and results in a (surprising) speed-up even on a single machine.

Source code

The source code for this module, and modules that build on it, is hosted at https://github.com/sroet/dask-traj

Share

CLstunfti: An extendable Python toolbox to compute scattering of electrons with a given kinetic energy in liquids and amorphous solids

 

Description

CLstunfti is an extendable Python toolbox to compute scattering of electrons with a given kinetic energy in liquids and amorphous solids. It uses a continuum trajectory model with differential ionization and scattering cross sections as input to simulate the motion of the electrons through the medium.

Originally, CLstunfti was developed to simulate two experiments: A measurement of the effective attenuation length (EAL) of photoelectrons in liquid water [1] and a measurement of the photoelectron angular distribution (PAD) of photoelectrons in liquid water [2]. These simulations were performed to determine the elastic mean free path (EMFP) and the inelastic mean free path (IMFP) of liquid water [3].

Practical application

The EMFP and IMFP are two central theoretical parameters of every simulation of electron scattering in liquids, but they are not directly accessible experimentally. As CLstunfti can be used to determine the EMFP and IMFP from experimental data, and as it can be easily extended to simulate other problems of particle scattering in liquids, it was decided to make the source code publicly available. For this purpose, within the E-CAM module, the necessary steps were taken to make CLstunfti a useful toolbox for other researchers by providing a documentation, examples, and also extensive inline documentation of the source code.

Source code

CLstunfti is available at https://gitlab.com/axelschild/CLstunfti .

 

References

[1] Suzuki, Nishizawa, Kurahashi, Suzuki, Effective attenuation length of an electron in liquid water between 10 and 600 eV, Phys. Rev. E 90, 010302 (2014)

[2] Thürmer, Seidel, Faubel, Eberhardt, Hemminger, Bradforth, Winter, Photoelectron Angular Distributions from Liquid Water: Effects of Electron Scattering, Phys. Rev. Lett. 111, 173005 (2013)

[3] Schild, Peper, Perry, Rattenbacher, Wörner, Alternative approach for the determination of mean free paths of electron scattering in liquid water based on experimental data, J. Phys. Chem. Lett., 11, 1128−1134 (2020)

Share

Minimal distance segment to segment with Karush-Kuhn-Tucker conditions

 

Description

The module minDist2segments_KKT returns the minimal distance between two line segments. It uses the Karush-Kuhn-Tucker conditions (KKT) for the minimization under constraints.

Practical application

We use the present module to avoid topology violations in an entangled polymer system. To preserve the topology in a system of entangled polymers we need to determine the minimal distance between two bonds. Once done we can apply either a soft or hard core potential to avoid the crossing of two bonds. Here, we propose to determine the minimal distance between two segments with the help of the Karush-Kuhn-Tucker conditions.

This module is a part of an E-CAM pilot project at the ENS Lyon, focused on the implementation of contact joint to resolve excluded volume constraints

Background information

A detailed derivation of the minimal distance between two segments using the Karush-Kuhn-Tucker conditions is available at  https://gitlab.e-cam2020.eu:10443/carrivain/mindist2segments_kkt/-/blob/master/minDist2segments_KKT.pdf

This module is used by other ongoing work, such as module velocities_resolve_EV, that resolves the excluded volume constraint  with a velocity formulation.

Source code

The source code and more information can be found at minDist2segments_KKT GitLab repository.

Share