[ONLINE] Introduction to Scalable Deep Learning @ JSC




In this course, we will cover machine learning and deep learning and how to achieve scaling to high performance computing systems. The course aims at covering all levels, from fundamental software design to specific compute environments and toolkits. We want to enable the participants to unlock the resource of machines like the JUWELS booster for their machine learning workflows. Different from previous years we assume that the participants have a background from a university level introductory course to machine learning. Suggested options for self-teaching are given below.

We will start the course with a presentation of high performance computing system architectures and the design paradigms for HPC software. In the tutorial, we familiarize the users with the environment. Furthermore, we give a recap of important machine learning concepts and algorithms and the participants will train and test a reference model. Afterwards, we introduce how deep learning algorithms can be parallelized for supercomputer usage with Horovod. Furthermore, we discuss best practicies and pitfalls in adopting deep learning algorithms on supercomputers and learn to test their function and performance. Finally we apply the gained expertise to large scale unsupervised learning, with a particular focus on Generative Adversarial Networks (GANs).


We assume that the participants are familiar with general concepts of machine learning and/or deep learning, such as widely used models, losses, regularization and basic model training / testing. Many excellent self-training resources are available such as:

Hands-on experience with ML/DL framework is required, first experience with HPC systems is helpful.

Learning outcome:

After this course, participants will be able to parallelize Tensorflow and Pytorch ML workflows on HPC machines, taking into account the HPC system architecture and circumventing typical pitfalls and bottlenecks.


14-18 March 2022, 9:00 - 13:00


Dr. Stefan Kesselheim, Dr. Jenia Jitsev, Roshni Kamath, Dr. Mehdi Cherti, Dr. Alexandre Strube, Jan Ebert, Jülich Supercomputing Centre

    • Day 1: Introduction to Supercomputers
      • 1
      • 2
        Lecture 1.1: Intro to supercomputers
      • 3
        Tutorial 1.1: First Steps on the Supercomputer
    • Day 1: Introduction to parallel programming
      • 4
        Lecture 1.2: Supercomputer architecture and MPI primer
      • 5
        Tutorial 1.2: Hello MPI World
    • Day 2: Deep Learning Basics Recap
      • 6
        Lecture 2.1: Motivation for Large Scale Deep Learning and Deep Learning Basics Recap
      • 7
        Tutorial 2.1: Deep Learning Basics Recap
    • Day 2: Distributed Training
      • 8
        Lecture 2.2: Distributed Training Schemes and Data Parallelism with Horovod
      • 9
        Tutorial 2.2: A first parallelization with Horovod
    • Day 3: Large Datasets and Scaling Distributed Training
      • 10
        Lecture 3.1: Large Datasets and Scaling Distributed Training
      • 11
        Tutorial 3.1: Distributed Training with ImageNet and Scaling Basics
    • Day 3: Performance analysis
      • 12
        Lecture 3.2: Is my code fast? Performance analysis
      • 13
        Tutorial 3.2: Tools for training performance analysis
    • Day 4: Combating Accuracy Loss
      • 14
        Lecture 4.1: Combating Accuracy Loss
      • 15
        Tutorial 4.1 Part I : Combating Accuracy Loss, Basics
    • Day 4: Outllook
      • 16
        Tutorial 4.1 Part II : Combating Accuracy Loss, Advanced
      • 17
        Lecture 4.2 : Outlook beyond the basics
    • Day 5: GANs I
      • 18
        Lecture 5.1: Generative models, GAN basics
      • 19
        Tutorial 5.1: Parallelizing a basic GAN: DCGAN architecture
    • Day 5: GANs II
      • 20
        Lecture 5.2: Advanced GANs
      • 21
        Tutorial 5.2: Parallelizing an advanced GAN: StyleGAN2 architecture
      • 22
        Concluding Remarks