Parallel and Distributed Computing

Dr. Guy Tel-Zur

About the course (back to the top)

The course is a modern introduction to Parallel and Distributed Computing. The course covers theoretical parallel algorithms and practical tools such as: MPI, OpenMP, OpenCilk, and parallel profiling tools. Other topics include: Code optimization, and High-Throughput Computing -HTCondor.

The course is more practical than theoretical and it is oriented toward Scientific Computing.
Basic Linux and C skills are mandatory and expected to be self taught by lecture 3.

Connect (back to the top)

Schedule (back to the top)

Meeting	Topic. Course's slides are in the moodle	Complementary Material
	Begining of Semester
1	Introduction - Course Overview, administration, and requirements	Parallel Computing Explained In 3 Minutes Guy Tel-Zur lecture: Introduction to parallel Programming - Message Passing Interface (MPI)
2	Introduction - Course Overview (continued)	Parallel Computing Explained In 3 Minutes Guy Tel-Zur lecture: Introduction to parallel Programming - Message Passing Interface (MPI)
3	Parallel Programming in Distributed Memory - The Message Passing Interface - MPI	MPI MPI Tutorials
4	Hands-on & demos: - Getting to know Linux - Hands-on practice with MPI - Parallel Profilers (Jumpshot, Scalasca)	MPI reference card Linux tutorials in PDF
5	Algorithms 1 - Embarrassingly Parallel Computations - Partitioning and Divide and Conquer Home assignment #1
6	Algorithms 2 - Synchronous Computations	More onIterative methods
7	Algorithms 3 - Numerical Linear Algebra - MPI Topologies Home assignment #2	ScaLAPACK: a portable linear algebra library for distributed memory computers - design issues and performance
8	Algorithms 4 - Sorting Algorithms, - Pipeline - Load balancing - Image Processing
9	Parallel Programming in Shared Memory - Introduction - OpenMP Home assignment #3	OpenMP home OpenMP quick reference card
10	OpenMP - continued - OpenMP Tasking - Open Cilk - Hybrid MPI and OpenMP	Charles E. Leiserson receives 2014 ACM/IEEE Computer Society Ken Kennedy Award
11	Distributed Computing - High Throughput Computing - Introduction to Grid Computing - Introduction to HTCondor - Introduction to Pegasus Home assignment #4
12	Optimizations - Vectorization - The Roofline model - Aos and Soa - Loops optimizations
13	- Parallel Matlab - Introduction to OpenACC - Course summary

References (back to the top)

Textbooks

• Parallel Programming: Techniques and Applications Using Networked Workstations and Parallel Computers (2nd Edition). By Barry Wilkinson and Michael Allen, 2004. Available online at Archive.org
• The OpenMP Common Core - Making OpenMP Simple Again by Timothy G. Mattson, Yun (Helen) He and Alice E. Koniges
• High Performance Computing: Modern Systems and Practices. 2nd Edition by Thomas Sterling, Maciej Brodowicz, Matthew Anderson.

Other recommended textbooks

• An Introduction to Parallel Programming. By Peter Pacheco, 2011.
• Introduction to High Performance Computing for Scientists and Engineers (Chapman & Hall/CRC Computational Science). By Georg Hager and Gerhard Wellein, 2010.
• Introduction to Parallel Computing. By Ananth Grama, Anshul Gupta, George Karypis and Vipin Kumar, 2003.
  
• Designing and Building Parallel Programs, by Ian Foster (available online).
  
• The Art of HPC. By Victor Eijkhout.
  
• Using OpenMP: Portable Shared Memory Parallel Programming (Scientific and Engineering Computation). By Gabriele Jost and Ruud van van der Pas, 2007.
• Using MPI - 2nd Edition: Portable Parallel Programming with the Message Passing Interface (Scientific and Engineering Computation). By William Gropp, Ewing Lusk and Anthony Skjellum, 1999.
  
• The Sourcebook of Parallel Computing (The Morgan Kaufmann Series in Computer Architecture and Design). By Jack Dongarra, Ian Foster, Geoffrey C. Fox, William Gropp, Ken Kennedy, Linda Torczon and Andy White, 2002.
  

Tutorials and guides

Linux UNIX/Linux tutorial
UNIX/Linux from Scratch (a free book)EGEE and SEE-GRID Summer School on Grid Applicationpport

C programming language
C tutorial

Parallel Computing
LLNL Introduction to Parallel Computing
SC08 Parallel Computing 101

Message Passing Interface
LLNL MPI tutorial
LLNL Advanced MPI
MPI for Dummies
MPI reference card

OpenMP
LLNL tutoriial
OpenMP reference card

Parallel Processing Courses around the world

UCSB  CS240A: Applied Parallel Computing
Berkeley CS267/EngC233, Applications of Parallel Computers, Spring 2010
Indiana U., B673: Parallel Scientific Computing, Spring 2011
University of Washington, CSE 524, WI '08 - Parallel Algorithms
UCLA CS 239 Parallel Programming Languages Winter 2009
Par Lab Boot Camp - Short Course on Parallel Programming>