Past Python Exchanges

With the rise of machine-learned interatomic potentials in the atomistic simulation community in Materials Science, the complexity of simulation workflows changed from directed acyclic graph (DAG) based simulation workflows with typically just a single simulation code to simulation workflows coupling simulation codes at different length and time scales and thousands of individual simulations. To orchestrate these workflows a number of simulation frameworks were developed. Still as part of the Exascale Computing Project we realized these were limited in their flexibility and scalability.

So, we developed executorlib[1] based on the concurrent futures Executor interface in the Python standard library and with the goal to distribute Python functions over hundreds of compute nodes. Internally, executorlib leverages the Simple Linux Utility for Resource Management (SLURM) and the flux framework from Lawrence Livermore National Laboratory[2] to up-scale simulation workflows from a workstation or traditional high-performance computers (HPC) to the latest generation of Exascale machines. In contrast to previous solutions, it does not require any daemon process or database but instead directly interfaces with the job manager to maximize computational efficiency. At the same time, it is designed with a focus on debugging capabilities to minimize the overhead of migrating workflows to the Exascale machines.

In this presentation I introduced executorlib, highlighting the lessons learned from development of the pyiron atomistic simulation suite which led to the development of executorlib as minimalistic workflow manager. Finally, I highlight the general applicability of executorlib to distribute python functions of any scientific domain on HPC clusters of all sizes.

[1]: Janssen et al., JOSS, 10(108), 7782, (2025).
[2]: Dong H. Ahn et al., Fut. Gen. Comp. Sys., 110, (2020).

Scikit-learn is one of the de facto libraries when it comes to predictive modeling with tabular data. For over a decade, it has provided traditional and reliable algorithms to address data science problems. While it excels at model fitting and prediction, these stages represent only a small portion of a data science project and are relatively well-defined. Many data scientists are familiar with the notion that 90% of their time is spent on preprocessing, while the modeling stage takes up only 10% of their efforts. Additionally, tracking and organizing experiments, as well as transitioning from experimentation to production, can be challenging.

This exchange aims to shed light on recent developments and efforts within the scikit-learn ecosystem. We will provide an overview of the following tools through a series of short notebook demos.

While tabular data is central to all organizations, it seems left out of the AI discussion, which has focused on images, text and sound. Ineed, for data science, most of the excitement is in machine learning, but most of the work happens before. Tables often require extensive manual transformation or "data wrangling". I will discuss how we progressively rethought this process, building machine learning tool that require less wrangling. We are building a new library, skrub (https://skrub-data.org), that facilitates complex tabular-learning pipelines, writing as much as possible wrangling as high-level operations and automating them. A few lines of skrub can spare you dozens of wrangling lines!

We will discuss the future of the Jupyter project (from a features/technical standpoint), in which we present ongoing and future work on the project. (Collaborative editing, CAD, GIS, etc).

In this discussion, we will explore how Kubernetes, an open-source container orchestration platform, is used in data science and machine learning. As the demand for scalable and reproducible data science workflows grows, Kubernetes offers a robust solution for managing the complexities of deployment, scaling, and maintenance. We will begin by discussing the key challenges faced by data scientists and machine learning engineers, such as environment inconsistencies and infrastructure scalability, and how Kubernetes effectively addresses these issues.

The discussion will cover the basics of Kubernetes, including its core components and architecture, tailored specifically for a data science audience. We will delve into setting up data science environments on Kubernetes, from containerizing workflows to managing dependencies. Furthermore, the talk will highlight advanced topics like scaling data science workloads, orchestrating machine learning pipelines with tools like Kubeflow, and deploying machine learning models in production.

By the end of this session, attendees will have a clear understanding of how Kubernetes can enhance their data science and machine learning workflows, providing a scalable, efficient, and reproducible infrastructure. No Kubernetes knowledge is required for this presentation.

In my talk, I will present Top2Vec, an open source topic modeling project. I will begin by discussing the inspiration and motivations behind creating the algorithm and project. Following that I will explain how Top2vec differs from traditional topic modeling techniques, highlighting its distinctive features and advantages. To conclude, I will demonstrate some practical applications of Top2Vec.

In this exchange, I will present the FEniCS project (https://fenicsproject.org), an open-source toolkit for solving partial differential equations. The FEniCS Project is composed of several packages, written in either C++ or Python. I will delve into the structure of the project and its evolution over time. Specifically, I will explain our use of Python for generating C-code and how we interact with C++.

In this exchange, I will present VocalPy, the core package of a software community for researchers that study how animals communicate with sound. For this audience, I will focus on the domain-driven design approach I am taking to develop the package. This approach has resulted in features that are (arguably) not common in scientific Python packages. For example, domain-specific data types, and pipelines consisting of classes that use callbacks. I will contrast these features with the design of many core scientific Python packages, that typically place an emphasis on a purely functional approach operating mainly on numpy arrays. I will discuss the trade-offs involved; for example, the potential for increased readability and reproducibility, but along with that a potential increase in maintenance burden. You will walk away with a better understanding of bioacoustics and animal communication, and some food for thought about how we design scientific software.

We have seen innumerable advancements in the scientific community due to the shift toward open science. We've learned, as a community, we must work together in order to build the next generation of scientific innovation. The history of the scientific computing ecosystem is intricately tied to its open source initiatives. One cannot succeed without the other. In this talk, we will walk through where we've been, where we are going, and the lessons we've learned along the way.

Dr. Brown discusses his experience in developing two scientific Python packages, khmer and sourmash, designed for dealing with really large sequencing data sets. Topics include switching from C++ to Rust for the performance layer, integrating tests and documentation into your daily life, and the interplay between algorithmic novelty and effective implementations.

Our core program is a peer review of open source software which seeks to improve the quality and usability of python tools that scientists depend on to process, visualize and analyze their data. We also have been working on guides to help scientists better understand how to package code so others can use it.

In this session, Leah, provides an overview of our peer review process and packaging guide. She will also discuss how pyOpenSci provides support and visibility for maintainers who are creating scientific Python tools.

Talley presented the motivation and general design of magicgui, a python library that facilitates the autogeneration of GUIs based on type annotations. In its high level interface, magicgui attempts to solve this by mapping python type annotations to widgets (for those familiar with ipywidgets, the goal is similar to ipywidgets.interact).

At a lower level, magicgui provides a widget abstraction on top of GUI frameworks like Qt, or ipywidgets; this allows magicgui code to (mostly) work in both a jupyter-notebook environment, or a Qt-desktop environment (or anything for which a backend UI adapter exists: a textual UI adapter is in the works).

John and Leland discuss how UMAP As a Python Open Source Project became the way it is. They trek through the journey of developing UMAP going all the way back to PyCon 2016 with HDBSCAN package, and the rest is as they say, history.

Spack is an open source package management tool, written in Python, that simplifies the process of building and installing scientific software. It is used widely in the HPC community — by end users, HPC facility staff, and software developers who need to manage dependencies. Spack is very general; it is designed to allow packages to be built with many different versions, configurations, build options, and compiler flags, for CPU and GPU machines. This talk will give an overview of Spack, its community, and how enables users to be more productive.

This discussion will give an overview of Spack, its community, and how it enables users to be more productive.

Juan and Draga will discuss napari, a Python package for fast array visualization that is equally comfortable working with 2D, 3D, and higher-dimensional image data. Napari can overlay images with the results of downstream processing steps, enabling quality control as well as manual intervention at critical stages — a workflow that previously involved shuttling of data between disparate tools.

They will also discuss napari's plugin interface, which can be used to extend its functionality, and to distribute new tools and methods to collaborators (who may have less Python experience) and the broader scientific community.

With the first Exascale computers becoming available and with machine learning becoming a fundamental building block of many research projects, we have to rethink the way we manage our simulations and analysis. Moving away from shell scripts and a zoo of different utilities and simulation codes, pyiron[1,2], in analogy to an integrated development environment (IDE), provides a central interface for rapid prototyping and up-scaling simulation protocols. The whole simulation lifecycle is represented in pyiron by a class of generic objects – the pyiron objects – which connect to the job management, the data storage interface and the user interface. As a result, the individual objects can be combined like building blocks to construct complex simulation protocols and enable the automation of routine tasks.

For more information, visit the pyIron website: www.pyiron.org

The more widespread Scientific Python becomes, the more the distributed nature and multiple projects will make it hard for newcomers to understand the landscape. In particular, documentation for "All in one" proprietary and closed source solutions is easier to find. Now that the world is hyper-connected, computing is ubiquitous and CI virtually free, I believe we can have a much better documentation experience.

We discussed scaling Python on large-scale GPU systems, as well as how to manage those systems and promote the usage of those systems.