Python Exchange

With the rise of machine-learned interatomic potentials in the atomistic simulation community in Materials Science, the complexity of simulation workflows changed from directed acyclic graph (DAG) based simulation workflows with typically just a single simulation code to simulation workflows coupling simulation codes at different length and time scales and thousands of individual simulations. To orchestrate these workflows a number of simulation frameworks were developed. Still as part of the Exascale Computing Project we realized these were limited in their flexibility and scalability.

So, we developed executorlib[1] based on the concurrent futures Executor interface in the Python standard library and with the goal to distribute Python functions over hundreds of compute nodes. Internally, executorlib leverages the Simple Linux Utility for Resource Management (SLURM) and the flux framework from Lawrence Livermore National Laboratory[2] to up-scale simulation workflows from a workstation or traditional high-performance computers (HPC) to the latest generation of Exascale machines. In contrast to previous solutions, it does not require any daemon process or database but instead directly interfaces with the job manager to maximize computational efficiency. At the same time, it is designed with a focus on debugging capabilities to minimize the overhead of migrating workflows to the Exascale machines.

In this presentation I introduced executorlib, highlighting the lessons learned from development of the pyiron atomistic simulation suite which led to the development of executorlib as minimalistic workflow manager. Finally, I highlight the general applicability of executorlib to distribute python functions of any scientific domain on HPC clusters of all sizes.

[1]: Janssen et al., JOSS, 10(108), 7782, (2025).
[2]: Dong H. Ahn et al., Fut. Gen. Comp. Sys., 110, (2020).

Scikit-learn is one of the de facto libraries when it comes to predictive modeling with tabular data. For over a decade, it has provided traditional and reliable algorithms to address data science problems. While it excels at model fitting and prediction, these stages represent only a small portion of a data science project and are relatively well-defined. Many data scientists are familiar with the notion that 90% of their time is spent on preprocessing, while the modeling stage takes up only 10% of their efforts. Additionally, tracking and organizing experiments, as well as transitioning from experimentation to production, can be challenging.

This exchange aims to shed light on recent developments and efforts within the scikit-learn ecosystem. We will provide an overview of the following tools through a series of short notebook demos.