An Introduction to Parallel Processing

About the course

The course is a modern introduction to Parallel Processing. The course covers theoretical parallel algorithms and practical tools such as: MPI, OpenMP, CilkPlus, parallel performance tools and jobs schedulers, code modernization and optimization, Intel's Parallel Studio, Xeon Phi, High-Throughput Computing (HTCondor).

The course is more practical than theoretical. It is oriented toward Scientific Computing. The educational cluster uses the Linux operating systems and it is required to have basic understanding of Linux.

Load: 10 Lectures, 2 labs, 4 home assignments and a final exam.

Meeting Date Topic Lecture Material Complementary Material
22/10/17 Begining of Semester
1 23/10/17 Introduction  - Course Overview and requirements Administration
Lecutre: lecture01.pdf
Parallel Computing 101
2 30/10/17 Introduction to Parallel Processing. Basic Message Passing -  MPI Administration 
Message Passing Computing 
local files: part 1 and part 2.

MPI 
MPI Tutorials 
mpi  
3 6/11/17 Lab #1: Building 33, Room 330  new  
Getting to knowLinux 
Hands-on practice with MPI
Embarrassingly Parallel Computation 

Parallel Profiler (Jumpshot, Scalasca) 
Lecture 

Example programs: 
ring.c 
cpi.c 
cpi.c (local) 
MPI reference card 

Linux tutorials 
http://www.ee.surrey.ac.uk/Teaching/Unix/ 
4 13/11/17 Performance Tools 
Jumpshot,Scalasca overview(another one),  tutorial 

IPM 

Torque  - Open-Source Resource Manager 

MPI Collective routines 
Embarrassingly Parallel Computations 

Home assignment #1 
Tools and demos 

The educational cluster 

Embarrassingly Parallel Computations 

 
adaptive computing
5 20/11/17 Algorithms: 


Partitioning and Divide and Conquer 

- Demos: MPI on Windows OS 
Lecture 


Partitioning and Divide and Conquer Strategies 

Synchronous Computations

Code: All to all example: c_ex07.c
Top500  
More on Iterative methods 
27/11/17 No class
6 4/12/17 Scalapack 
some code examples 

Home assignment #2
Lecture6 

Lecture7 

code: 
oned.c 
wave.c 
dgemm_example.c 
dgemm_demo.c 
dgemm-mpi.c 
pdgemv.c 
mm_pblas.c 
ScaLAPACK: a portable linear algebra library for distributed 
memory computers - design issues and performance
 
scalapack
7 11/12/17 Parallel Programming inShared Memory 
- Introduction 
- OpenMP 
Lecture 

Programming with Shared Memory 

 Hands-On Introduction to OpenMP, Mattson and Meadows, from SC08 

OpenMP home 

Wikipedia 

OpenMP quick reference card 

OpenMP in Visual C/C++ 

"Reap the Benefits of Multithreading without All the Work" 


Introduction to OpenMP and 
Advanced OpenMP 
from: "Programming Techniques for Supercomputers 2011"  by Prof. Gerhard Wellein, RRZE HPC Services 

openmp
8 18/12/17 OpenMP - Continued 
 
Load Balancing and Termination Detection 

Parallel Matlab  -MatlabMPI(Engineering) 

- Sorting Algorithms 

Home assignment #3
Lecture 

Hands-On Introduction to OpenMP, Mattson and Meadows, from SC08 

Shared Memory (Intel) 

Profiling OpenMP 

Parallel Matlab 

Sorting Algorithms 
An Overview of OpenMP - Ruud van der Pas - Sun Microsystems 

Shared Memory Programming with OpenMP

9 25/12/17 - OpenMP in Visual Studio 
- Profiling OpenMP
- Hybrid MPI and OpenMP: 
- CilkPlus 
- Numerical Algorithms 
Lecture 

Pipeline computations

Load Balancing and  Termination Detection 

Hybrid MPI+OpenMP 

CilkPlus 
CilkPlus tutorial 

Numerical Algorithms 
  CilkPlus
10 1/1/18 Code modernization and optimization 
Intel's Xeon Phi 

Lecture 

Materials thanks to: Colfax Research 

Introduction 

N-body (slides
youtube demo 
Paper 
Advanced OpenMP (version 4.5) and OpenACC
  Xeon Phi


The Paradigm Shift beyond Exascale Computing by Thomas Sterling: 
https://www.youtube.com/watch?v=m17pT8nrlNA 
11 8/1/18 * Image Processing 
* Introduction to Grid Computing 
* HTCondor 

Home assignment #4
Lecture 

Image Processing 

Introduction to Grid Computing 

HTCondor  and Pegasus

12 15/1/18 Lab #2 - 
Building 33, room 330. 


A very short introduction toHTCondor, Grid and Cloud Computing 

Course summary
HTCondor Tutorial 

HTCondorWeek 2013: 
Basic Introduction to using HTCondor by Karen Miller, Center for High Throughput Computing

Preparation to the exam
 
19/1/18 End of Semester
TBA First term exam Building XX, Rooms: XX, XX
TBA Second term exam TBA