WG Large-Scale Computing

Simulating a physical phenomenon or training an artificial intelligence model requires immense computational power. Since the 1980s, supercomputers have provided this power by distributing computations across many servers connected by high-speed networks. While computing clusters have existed for decades, recent advancements in computing infrastructures present numerous challenges for parallel application developers. However, the increase in the number of compute nodes—from 8,192 cores in 2000 to 11 million in 2025—raises new challenges for developers, particularly in scalability, shared resource management, and performance optimization.

Improving scalability is a central issue to fully leverage these infrastructures. Applications must express sufficient parallelism to utilize the millions of available cores while minimizing network communications that introduce sequentiality. Contention for shared resources (memory, network, storage) can also degrade performance, especially when multiple applications use them simultaneously. To address this, a hierarchical structuring of resources and consideration of locality by algorithms and runtime systems are essential. Development tools, such as compilers and performance analyzers, also play a key role in optimizing applications and facilitating large-scale debugging. Additionally, fault tolerance becomes indispensable with the growing number of components, requiring mechanisms like checkpointing.

The heterogeneity of resources, particularly with the integration of accelerators like GPUs, adds another layer of complexity. These components offer increased computational power and better energy efficiency, but their optimal use requires major adaptations in applications. Traditional programming models, such as offloading (fork/join), introduce synchronizations that limit scalability. Alternatives, such as task-based models (OpenMP, Kokkos, StarPU), allow for different expressions of parallelism, but their large-scale use remains a challenge. Furthermore, mastering the hardware architecture of accelerators—both for performance and technological sovereignty—requires close collaboration between hardware and software layers.

Efficient data access is the third major challenge for supercomputers. While processor performance advances rapidly, memory and storage systems lag behind, creating a bottleneck. Checkpoints, essential for fault tolerance, generate massive data flows that saturate storage servers. Heterogeneous memories (HBM, non-volatile memory, CXL, etc.) and storage hierarchies (flash memory, hot/cold data servers) offer avenues for optimizing access. Runtime systems could also allocate frequently used data to fast memories, while technologies like unified memory (CPU/GPU) or memory disaggregation via CXL could redefine supercomputer architecture.

The “Large-Scale Computing” working group aims to identify the scientific challenges related to scaling up and contribute to animating the scientific community.

Head: François Trahay

Mailing list: c4p-gt-grande-echelle@groupes.renater.fr (subscribe to the working group on myGDR to join the mailing list)