LUMI pilots: modelling weather and space environment

When fully operational, the LUMI supercomputer with a computing power of 552 petaflops will be one of the fastest supercomputers in the world. LUMI is located at CSC’s data center in Kajaani, Finland and funded by the EU and a consortium of ten European countries. As part of the LUMI consortium Belgium was provided early access to the supercomputer and could run several pilot projects on LUMI including the CS-MASK project (Royal Meteorological Institute, RMI and UGent) and the LIFTHRASIR project (KU Leuven). Both pilot users were very enthusiastic about LUMI and its performance. 

The CS-MASK project (Crowd-Sourced data for atmospheric Modelling At Sub-Kilometric scale) empowers high resolution modelling of weather forecasts.

–The horizontal resolution for weather forecast is usually 1.3 km but within this project we aimed at a resolution of 700 m. A thundercloud for example has a diameter of +- 1 km. If we have a resolution of 700 m it enables us to make more accurate forecasts. Another important application is accounting for the effect of the local environment in weather forecasts (e.g., the effect of a park in the middle of a city). To model these kinds of effects, a very high resolution is needed and thus more computing power is required so we participated in the LUMI pilot, said Dr. Daan Degrauwe from RMI.

Image: The urban heat island effect is clearly visible for Antwerp, Brussels, and to a lesser extent Ghent. The land breeze effect is also nicely visible (inverse of sea breeze), where cool air from the land is blown over the sea.

The other Belgian pilot project, LIFTHASIR, studied reconnection (process of energy conversion) in astrophysics. Professor Giovanni Lapenta from KULeuven described the project as follows:

– Our technological infrastructure is at great risk of solar and geomagnetic storms. Therefore, we need to have the best model possible of the space environment. For the past decades the fluid model has dominated. In this approach the particles that move through space are treated like a fluid but specific aspects of the simulation are lost in this way. With the new generation of supercomputers like LUMI it is possible to do a full description of the particles, treating both ions and electrons as particles, like they really are.

Image: The importance of going form hybrid to full particle models is illustrated in the figures above. Above you see a hybrid model and below a full particle model. The full particle model shows sharper and more intense transitions that produce much stronger energisation. Figure 2 identifies the regions of energisation using a quantity based on a special type of asymmetry of the electron particle distribution called agyrotropy. This physics process is present only in full particle models.

Advantages of LUMI

– LUMI was perfect for our project and not only because of the scale, but also because of its international character. Our weather model is developed by a consortium of 26 countries. The European context of LUMI makes the machine easily accessible for our partners which is very beneficial for the collaboration within the project, said Degrauwe from RMI.

– During the pilot phase it was interesting to experience the performance of LUMI and experience if there was a difference with other machines we were using in the past (e.g., the SuperMUC supercomputer based on Intel CPU’s & LUMI based on AMD). LUMI performed perfectly and the code worked perfectly well, Lapenta from KULeuven pointed out.

Looking forward to the GPU partition

The greatest computing power of LUMI lies in the GPU partition. It will be part of the system’s second installation phase. However, our pilot users are already anticipating this upcoming feature. The CS-MASK team is currently adapting their code, and LIFTHASIR has already a version of the code running on GPU.

– We now have a version of the code splitting the job between CPU and GPU. In the second pilot phase of LUMI we will be able to test this, Lapenta said.

Machine learning

LUMI can also be used for ‘on the fly’ data analysis using machine learning. The upcoming GPU partition will accelerate this technique.

– In our simulation we have a tremendous number of particles moving around (>100 billion). Every time step generates several terabytes of particle data and there is a huge number of time steps. Normally you save the data and then analyse it but, in this case, it would create a huge amount of data. Thanks to machine learning algorithms, the analysis can be done ‘on the fly’. This means we can select information from the simulation and store only that small amount of it that has been extracted by the machine learning tool, Lapenta explained.

– For weather forecasting it could be very interesting to replace some parts of the model by machine learning. LUMI provides the infrastructure to make this possible (not only thanks to the GPUs but also the data partition), Degrauwe continued.

LUMI meets expectations of pilot users

–The fact that the LUMI help desk is spread over different countries was also an added value, this ensured quick response and a wide variety of expertise, said Degrauwe.

Professor Lapenta agrees on this point:

– Since there were only few pilot projects running, we were able to establish a direct relationship with the team and problems or questions were solved quickly.

– It was a very nice experience working with LUMI. I am really looking forward to proceed in the second pilot phase, Lapenta continued.

–LUMI definitely met our expectations! We were able to do our forecasts with the resolution we had planned. Both from a scientific point of view and from a technical point of view, we took some important steps during this pilot phase, so we are very thankful to have been a part of it, Degrauwe concluded.

Authors: Jothi Blontrock, EuroCC Belgium & Tim Jaenen, FWO, Belgium