An Introduction to Parallel Processing

About the course


The course is a modern introduction to Parallel Processing. The course covers theoretical parallel algorithms and practical tools such as: MPI, OpenMP, CilkPlus, parallel performance tools and jobs schedulers, code modernization and optimization, Intel's Parallel Studio, Xeon Phi, High-Throughput Computing (HTCondor).

The course is more practical than theoretical. It is oriented toward Scientific Computing. The educational cluster uses the Linux operating systems and it is required to have basic understanding of Linux.

Load: 10 Lectures, 2 labs, 4 home assignments and a final exam.

Meeting Date Topic Lecture Material Complementary Material
14/10/18 Begining of Semester
1 15/10/18 Introduction - Course Overview and requirements Administration
Lecutre:lecture01.pdf
If you experience downloading problems, please try to download the file from this alternative link (dropbox)
Parallel Computing Explained In 3 Minutes

Guy Tel-Zur lecture: Introduction to parallel Programming - Message Passing Interface (MPI)
2 22/10/18 Introduction to Parallel Processing. Basic Message Passing -MPI Administration
Message Passing Computing
local files: part 1 and part 2.

MPI
MPI Tutorials
mpi
3 29/10/18 Lab #1: Building 33, Room 330 new
Getting to know Linux
Hands-on practice withMPI,
Embarrassingly Parallel Computation

Parallel Profilers (Jumpshot, Scalasca)
Lecture

Example programs:
ring.c
cpi.c
cpi.c(local)
MPI reference card

Linux tutorials
http://www.ee.surrey.ac.uk/Teaching/Unix/
scalasca
4 5/11/18 Performance Tools
Jumpshot,Scalascaoverview(another one), tutorial

IPM

Torque-Open-Source Resource Manager

MPI Collective routines
Embarrassingly Parallel Computations

Home assignment #1
Tools and demos

The educational cluster

Embarrassingly Parallel Computations

adaptive computing
12/11/18 No class SC
5 19/11/18 Algorithms:


Partitioning and Divide and Conquer

- Demos: MPI on Windows OS
Lecture


Partitioning and Divide and Conquer Strategies

Synchronous Computations

Code: All to all example: c_ex07.c
Top500
More onIterative methods
6 26/11/18 Scalapack
some code examples

Home assignment #2
Lecture6

Lecture7

code:
oned.c
wave.c
dgemm_example.c
dgemm_demo.c
dgemm-mpi.c
pdgemv.c
mm_pblas.c
ScaLAPACK: a portable linear algebra library for distributed
memory computers - design issues and performance

scalapack
7 3/12/18 Parallel Programming inShared Memory
- Introduction
- OpenMP
Lecture

Programming with Shared Memory

Hands-On Introduction to OpenMP, Mattson and Meadows, from SC08 (a shorter version)

OpenMP home

Wikipedia

OpenMP quick reference card

OpenMP in Visual C/C++

"Reap the Benefits of Multithreading without All the Work"


Introduction to OpenMP and
Advanced OpenMP
from: "Programming Techniques for Supercomputers 2011" by Prof. Gerhard Wellein, RRZE HPC Services

openmp
8 10/12/18 OpenMP - Continued

Load Balancing and Termination Detection

Parallel Matlab -MatlabMPI(Engineering)

- Sorting Algorithms
Lecture

Hands-On Introduction to OpenMP, Mattson and Meadows, from SC08 (a shorter version)

Shared Memory(Intel)

Profiling OpenMP

Parallel Matlab

Sorting Algorithms
An Overview of OpenMP - Ruud van der Pas - Sun Microsystems

Shared Memory Programming with OpenMP

9 17/12/18 - OpenMP in Visual Studio
- Profiling OpenMP
- Hybrid MPI and OpenMP:
- CilkPlus
- Numerical Algorithms
Home assignment #3
OpenMP Tasking Explained

Hybrid MPI+OpenMP

CilkPlus
CilkPlus tutorial

Pipeline computations

Load Balancing and Termination Detection

Numerical Algorithms
CilkPlus
10 24/12/18 Code modernization and optimization
Intel's Xeon Phi

Lecture

Materials thanks to:Colfax Research

Introduction

N-body(slides)
youtube demo
Paper
Advanced OpenMP (version 4.5) and Vectorization
Xeon Phi


The Paradigm Shift beyond Exascale Computing by Thomas Sterling:
https://www.youtube.com/watch?v=m17pT8nrlNA
11 31/12/18 * Image Processing
* Introduction to Grid Computing
* HTCondor

Home assignment #4
Lecture

Image Processing

Introduction to Grid Computing

HTCondor and Pegasus

12 7/1/19 Lab #2 -
Building 33, room 330.


A very short introduction toHTCondor, Grid and Cloud Computing

Course summary
HTCondor Tutorial

HTCondorWeek 2013:
Basic Introduction to using HTCondor by Karen Miller, Center for High Throughput Computing

Preparation to the exam
11/1/19 End of Semester
21/1/19, 17:30 exam - מועד א Location: TBA
11/2/19, 17:30 exam - מועד ב Location: TBA