Mesoscale simulation of billion atom complex systems using thousands of GPGPU’s, an industry success story


Dr. Jony Castagna, Science and Technology Facilities Council, United Kingdom


Abstract

Jony Castagna recounts his transition from industry scientist to research software developer at the STFC, his E-CAM rewrite of  DL_MESO allowing the simulation of billion atom systems on thousands of GPGPUs, and his latest role as Nvidia ambassador focused on machine learning.


Jony, can you tell us how you came to work on the E-CAM project and what you were doing before?

My background is in Computational Fluid Dynamic (CFD), and I worked for many years in London as a computational scientist for an Oil & Gas industry. I joined STFC – Hartree Centre in 2016 and E-CAM was my first project. E-CAM offered an opportunity to work in a new and more academic and fundamental research environment.  

What is your role in E-CAM?

My role is as research software developer, which consists mainly in supporting the E-CAM Postdoctoral Researchers in developing their software modules, benchmarking available codes and contribute to the deliverables of the several work-packages. This includes the work described here, in co-designing DL_MESO to run on GPUs.

What is DL_MESO and why was it important to port it to massively parallel computing platforms?

DL_MESO is a software package for mesoscale simulations developed by M. Seaton at the Hartree Centre [1, 2]. It is basically made of two software components: a Lattice Boltzmann method solver, which uses the Lattice Boltzmann equation discretize on a lattice (2D or 3D)  to simulate the fluid dynamic effects of complex multiphase systems; and a Dissipative Particle Dynamics (DPD) solver based on particle method where a soft potential, together with a coupled dissipation and stochastic forces, allows the use of Molecular Dynamics but with a larger time step.

The need to port DL_MESO to massively parallel computing platforms arose because  often real systems are made of millions of beads (each bead representing a group of molecules) and small clusters are usually not sufficient to obtain results in brief time. Moreover, with the advent of hybrid architectures, updating the code is becoming an important software engineering step to allow scientist to continue their work on such systems.

How well were you able to improve the scaling performance of DL_MESO with multiple GPGPU’s, and as a consequence, how large a system can you now treat?

The current multi-GPU version of DL_MESO scales with an 85% efficiency up to 2048 GPUs equivalent to about 10 petaflops of performance double precision (see Fig. 1 reproduced from E-CAM Deliverable 7.6[3]). This allows the simulation of very large systems like a phase mixture with 1.8 billion particles (Fig. 2). The performance has been obtained using the PRACE resource Piz Daint supercomputer from CSCS.


Figure 1. Strong scaling efficiency of DL_MESO versus the number of GPGPU for a simulation of a complex mixed phase system consisting of 1.8 billion atoms.  
Figure 2. Snapshot of the simulated system.

What are the sorts of practical problems that motivated these developments, and what is the interest from industry (in particular IBM and Unilever) ?

DPD has the intrinsic capability to conserve hydrodynamic behavior, which means it reproduces fluid dynamic effects when a large number of beads is used. The use of massively parallel computing allows the simulation of complex phenomena like shear banding in surfactants and ternary systems present in many personal care, nutrition, and hygiene products. DL_MESO has been used intensively by IBM Research UK and Unilever and there is a long history of collaboration with Hartree Centre still going on.   

Are there some examples of the power of DL_MESO to simulate continuum problems with difficult boundary conditions, etc., where standard continuum approaches fail?

Yes. One good example is the polymer melt simulation. Realistic polymers typically are notoriously very large macromolecules, and their modeling in industrial manufacturing processes, where fluid dynamic effects like extrusion exist, is a very challenging task. Traditional CFD solvers fail to describe well the complex interface and interactions between polymers.  DPD represents the ideal approach for such systems.

What were the particular challenges to porting DL_MESO to GPUs? You started by an implementation on a single GPU and only afterwards ported it to multi-GPUs. Was that needed?

The main challenge has been to adapt the numerical algorithm implemented in the serial version to the multithread GPU architecture. This required mainly a reorganization of the memory layout to guarantee coalescent access and take advantage of the extreme parallelism provided by the accelerator. The single GPU version was developed first, optimized and then extended to multi-GPU capability based on MPI library and a typical domain decomposition approach.

We know you are adding functionalities to the GPU version of DL_MESO, such as electrostatics and bond forces. Why is that important?

Electrostatic forces are very common in real systems, they allow the simulation of complex products where charges are distributed across the beads creating polarization effects like those in a molecule of water. However, these are long-range interactions and special methods like Ewald Summation and Smooth Particle Ewald Mesh are needed to fully compute their effects. They represent a challenge from numerical implementation due to their high computational cost and difficulties they present to parallelization.

Where can the reader find documentation about the software developments that you have been doing in DL_MESO?

Mainly on the E-CAM modules dedicated to DL_MESO that have been reported on Deliverables 4.4[4] and 7.6[3], and also on the E-CAM software repository here .

Did your work with E-CAM, on the porting of DL_MESO to GPUs, opened doors to you in some sense?

Yes. IBM Research UK has shown interest in the multi-GPU version of the code for their studies on multiphase systems and Formeric, a spin-off company of STFC, is planning to use it as the back end of their products for mesoscale simulations.

Recently, you have also been nominated as an NVidia Ambassador. How did that happen?

We have a regular collaboration with NVidia, not only through the Nvidia Deep Learning Institute (DLI) for dissemination and tutorials, but also for optimization in porting software to multi-GPU as well as Deep Learning applications applied mainly to computer vision industrial problems.  This is how I got the Nvidia DLI Ambassador status in October 2018. It is being a great experience and an exciting opportunity.

What would you like to do next?

The Nvidia Ambassador experience in Deep Learning opened a new exciting opportunity in the so-called Naive Science: the idea is to use neural networks for replacing traditional computational science solvers. A Neural Network can be trained using real or simulated data and then used to predict new properties of molecules or fluid dynamic behaviour in different systems. This will speed up the simulation by a couple of orders of magnitude as well as avoiding complex modeling based on the use of ad hoc parameters that are often difficult to determine.   

References

[1] http://www.cse.clrc.ac.uk/ccg/software/DL_MESO/
[2] M. A. Seaton, R. L. Anderson, S.Metz, and W. Smith, “DL_MESO: highly scalable mesoscale simulations,”Molecular Simulation, vol. 39, no. 10, pp. 796–821, Sep. 2013.
[3] Alan O’Cais, & Jony Castagna. (2019). E-CAM Software Porting and Benchmarking Data III (Version 1.0). Available in Zenodo: https://doi.org/10.5281/zenodo.2656216
[4] Silvia Chiacchiera, Jony Castagna, & Christian Krekeler. (2019). D4.4: Meso- and multi-scale modelling E-CAM modules III (Version 1.0). Available in Zenodo: https://doi.org/10.5281/zenodo.2555012


Share

Software vendor SMEs as a boost for technology transfer in industrial simulative pipelines

The E-CAM Scoping Workshop “Building the bridge between theories and software: SME as a boost for technology transfer in industrial simulative pipelines”, organised in May 2018 at the Fondazione Instituto Italiano di Tecnologia (IIT), Genoa, brought together top-level scientists of the E-CAM community with expertise in statistical mechanics, multi-scale modeling and electronic structure, and representatives of pharmaceutical and material industries, with the final objectives to identify the major gaps which still hamper a systematic exploitation of accurate computer simulations in industrial R&D. Special attention was given to the role of SMEs devoted to simulative software development, and several software vendor SMEs were present at the meeting.

It was clear from the meeting that software vendor SMEs may represent the missing link in the pipeline from-theory-to-software; as they can play an increasingly key role not only in translating the science developed in academia into a proper technological transfer process, but also in building a scientific bridge between the industry requirements in terms of automation and the new theories and algorithms developed at an academic level. There was also a consensus that EU funded Centers of Excellence for Computing Applications, such as E-CAM, can provide an opportunity to enhance the expertise and scope of software vendors SMEs.

Read the full report here.

Share

E-CAM Case Study: Mesoscale models for polarisable solvents: application to oil-water interfaces

Dr. Silvia Chiacchiera, Science and Technology Facilities Council, United Kingdom

Abstract

Water is a polar liquid and has a dielectric permittivity much higher than typical apolar liquids, such as light oils. This strong dielectric contrast at water-oil interfaces affects electrostatics and is important, for example, to include these effects to describe biomolecular processes and water-oil mixtures involving surfactants, as detergents. In this pilot project, developed in collaboration with Unilever and Manchester University, we have proposed and analysed a class of polarisable solvent models to be used in Dissipative Particle Dynamics (DPD), a coarse-grained particle-based simulation method commonly used in various industrial sectors. Related software modules for the DL_MESO package have also been developed.

Continue reading…

Share

The Curse of Dimensionality in Data-Intensive Modeling in Medicine, Biology, and Diagnostics

With Prof. Tim Conrad (TC), Free University of Berlin, and Dr. Donal Mackernan (DM), University College Dublin.

Abstract

Until recently the idea that methods rooted in statistical physics could be used to elucidate phenomena and underlying mechanisms in biology and medicine was widely considered to be a distant dream.  Elements of that dream are beginning to be realized, aided very considerably by machine learning and advances in measurement, exemplified by the development of large-scale biomedical data analysis for next-generation diagnostics. In this E-CAM interview of Tim Conrad,  the growing importance of diagnostics in medicine and biology is discussed. One difficulty faced by such developments and shared with particle-based simulation is the “curse of dimensionality”. It is manifest in problems such as: (a) the use of a very large number of order parameters when trying to identify reaction mechanisms, nucleation pathways, metastable states, reaction rates; polymorph recognition (b) machine learning  applied to electronic structure  problems – such as neural network based potentials need very high dimensional basis sets; (c) systematic coarse-graining would ideally start with a very high dimensional space and systematically reduce the dimension.  The opportunities and challenges for scientists engaging with industry are also discussed. Tim Conrad is Professor of “Medical Bioinformatics” at the Institute of Mathematics of the Free University of Berlin and head of MedLab, one of the four laboratories of the Modal research campus. MODAL is a public-private partnership project which conducts mathematical research on data-intensive modeling, simulation, and optimization of complex processes in the fields of energy, health, mobility, and communication.  Tim Conrad is also the founder of three successful start-up companies.

In this E-CAM interview with Prof. Tim Conrad, the growing importance of diagnostics in medicine and biology is discussed, including concepts rooted in signal analysis relevant to systematic dimensional reduction, and pattern recognition, and the possibilities of their application to systematic coarse-graining. The opportunities and challenges for scientists of start-up companies are also discussed based on experience.

 

Continue reading…

Share

E-CAM Case Study: Designing control pulses for superconducting qubit systems with local control theory

Dr. Momir Mališ, École Polytechnique Fédérale de Lausanne, Switzerland

 

Abstract

A quantum logic gate is one of the key components of the quantum computer, and designing an effective quantum universal gate is one of the major goals in the development of quantum computers. We have developed a software based on local control theory to design efficient state preparation control pulses for universal quantum gates which drive full population transfer between qubit states.

Continue reading…

Share

E-CAM Case Study: The implementation of a hierarchical equilibration strategy for polymer melts, to help studying the rheological properties of new composite materials

Dr. Hideki Kobayashi, Max-Planck-Institut für Polymerforschung, Germany

Abstract

The ability to accurately determine and predict properties of newly developed polymer materials is highly important to researchers and industry, but at the same time represents a significant theoretical and computational challenge. We have developed a novel multiscale simulation method based on the hierarchical equilibration strategy, which significantly decreases the equilibrium properties calculation time while satisfying the thermodynamic consistency. A number of E-CAM modules was developed and implemented in he ESPResSo++ software package.

Continue reading…

Share

The simulation of metal ions in protein-water systems using machine learning: An E-CAM case study and conversation

 

With Dr. Francesco Fracchia, Scuola Normale Superiore di Pisa

Interviewer: Dr. Donal Mackernan, University College Dublin

 

Abstract

One quarter to one third of all proteins require metals to function but the description of metal ions in standard force fields is still quite primitive. In this case study and interview an E-CAM project to develop a suitable parameterisation using machine learning is described.  The training scheme combines classical simulation with electronic structure calculations to produce a force field comprising standard classical force fields with additional terms for the metal ion-water and metal ion-protein interactions. The approach allows simulations to run as fast as standard molecular dynamics codes, and is suitable for efficient massive parallelism scale-up.

Continue reading…

Share

From Rational Design of Molecular Biosensors to Patent and potential Start-up

 

Dr. Donal Mackernan, University College Dublin

Abstract

The power of advanced simulation combined with statistical theory , experimental know-how and high performance computing is used to design a protein based molecular switch sensor with remarkable sensitivity and significant industry potential. The sensor technology has applications across commercial markets including diagnostics, immuno-chemistry, and therapeutics.

 

Continue reading…

Share

A Conversation on Neural Networks, from Polymorph Recognition to Acceleration of Quantum Simulations

 

With Prof. Christoph Dellago (CD), University of Vienna, and Dr. Donal Mackernan (DM), University College Dublin.

 

Abstract

Recently there has been a dramatic increase in the use of machine learning in physics and chemistry, including its use to accelerate simulations of systems at an ab-initio level of accuracy, as well as for pattern recognition. It is now clear that these developments will significantly increase the impact of simulations on large scale systems requiring a quantum level of treatment, both for ground and excited states. These developments also lend themselves to simulations on massively parallel computing platforms, in many cases using classical simulation engines for quantum systems.

 

Continue reading…

Share

Rare events, path sampling and the OpenPathSampling package

 

In the last few years, modelling of rare events has made tremendous progress and several computational methods have been put forward to study these events. Despite this effort, new approaches have not yet been included, with adequate efficiency and scalability, in common simulation packages. One objective of the Classical Dynamics Work Package of the project E-CAM is to close this gap. The present text is an easy-to-read article on the use of path sampling methods to study rare events, and the role of the OpenPathSampling package to performing these simulations. Practical applications of rare events sampling and scalabilities opportunities in OpenPathSampling are also discussed.

Continue reading…

Share