Parallel Programming (CS 4961)

Fall 2011

Schedule: Tues./Thur., 9:10-10:30 AM
Location: WEB L112
Instructor: Mary Hall
Office Hours: MEB 3466; Tuesdays, 10:45-11:15 AM; Wednesdays, 11:00-11:30 AM or by appointment
Mailing list: cs4961@list.eng.utah.edu
TA: Nikhil Mishrikoti Office hours: MEB 3115, Desk #12; Monday, 2-3 PM; lab hours on Tuesdays when announced

Jump to: [Background] [Grading] [Textbook] [Schedule] [Assignments] [Policies]

 Background and Description

This course is a comprehensive exploration of parallel programming paradigms, examining core concepts, focusing on a subset of widely used contemporary parallel programmingmodels, and providing context with a small set of parallel algorithms. In the last few years, this area has been the subject of significant interest due to a number of factors. Most significantly, the advent of multi-core microprocessors has made parallel computing available to the masses. At the high end, major vendors of large-scale parallel systems, including IBM, Cray and Sun, have recently introduced new parallel programming languages designed for applications that exploit tens of thousands of processors. Embedded devices can also be thought of as small multiprocessors. The convergence of these distinct markets offers an opportunity to finally provide application programmers with a productive way to express parallel computation.

The course will be structured as lectures, homeworks, programming assignments and a final project. Students will perform four programming projects to express algorithms using selected parallel programming models and measure their performance. The final project will consist of teams of 2-3 students who will implement codes by combining multiple programming models.

Prerequisites: CS 4400, or concurrent

 Grading

35%Programming projects(P1, P2, P3, P4)
25%Written homeworks
25%Quiz and Final
15%Final project

Note:Late homeworks are not allowed. Late projects will incur a 20% penalty per day.

 Textbook

An Introduction to Parallel Programming by Peter Pacheco (ISBN: 978-0-12-374260-5).

 Schedule (tentative)

The following schedule is subject to change with a week's notice, particularly as opportunities for guest lectures and conflicts arise. The readings listed should be completed ahead of class so that you can follow the lecture and respond to questions. Homeworks and programming assignments are due before class so that we can discuss them.

Date Topics Read Assign Notes
23 Aug Introduction (ppt) (pdf)
Importance of parallel programming 		Chapter 1 HW01 -
25 Aug Introduction to parallel algorithms and correctness (ppt) (pdf)
Concerns for parallelism correctness and performance 		Chapter 1 - -
30 Aug Parallel Computing Platforms and Models of Execution (ppt) (pdf)
A diversity of parallel architectures, taxonomy, and examples 		Chapter 2, 2.1-2.3, pgs. 15-46 - -
01 Sep Memory systems and interconnect (ppt) (pdf)
Memory system organization and coherence, interconnect topologies, models for applications 		Chapter 2, 2.1-2.3, pgs. 15-46 HW02 -
Parallel Programming Model Concepts
06 Sep Introduction to Shared Memory Programming (ppt) (pdf) Ch. 2.4-2.4.3 (pgs. 47-52), 4.1-4.2 (pgs. 151-159), 5.1 (pgs. 209-215) - -
08 Sep Data Parallelism in OpenMP(ppt) (pdf)
Introduction to OpenMP and Parallel Loops 		Chapter 5.2-5.7, 5.10 (pgs. 216-241, 256-258) P01 Sun Ultrasparc T2
13 Sep Reordering Transformations and Data Locality(ppt) (pdf)
Code restructuring techniques: permutation and tiling 		- - Dep Notes
15 Sep Data Locality, cont. (ppt) (pdf)
Code restructuring techniques: unroll-and-jam and scalar replacement 		- - Locality Notes
20 Sep Lab session for Project 1 (WEB L130)
- 		- - -
22 Sep Task Parallelism (ppt) (pdf)
Open MP 3.0, Task-Parallel Algorithms 		Chapter 5.8 (pgs. 241-450) HW03 -
27 Sep Introduction to SIMD (ppt) (pdf)
SIMD engines in multimedia and GPUs 		- - Loop Coalescing Notes
29 Sep More SIMD (ppt)
(pdf)
SIMD control flow 		- P02 -
04 Oct Advanced Synchronization (ppt)
(pdf)
Synchronization in Pthreads 		Chapter 4.4-4.9 (pgs. 162-190) - -
06 Oct -
Lab Day 		- - -
18 Oct -
Midterm Review 		- - -
20 Oct -
Midterm 		- - -
Parallel Programming for GPUs
25 Oct Introduction to GPUs and CUDA (ppt)
(pdf)
Architecture and programming constructs 		(Optional) CUDA Programming Guide - DDJ article
27 Oct CUDA, cont. (ppt) (pdf)
SIMT execution model, divergent branches, memory hierarchy 		- P03 -
01 Nov CUDA, cont. (ppt) (pdf)
More memory hierarchy and examples 		- - -
Message Passing and Distributed Memory
03 Nov Introduction to Message Passing (ppt)
(pdf)
What is MPI? Complexities of a distributed address space 		Chapter 3.1-3.2, 3.4, pgs. 83-96, 101-106 - -
08 Nov Sparse Algorithms, MPI Communication Operations (ppt) (pdf)
Sparse Matrices, Collectives 		Chapter 3.3,3.5-3.7 (pgs. 106-136) - -
10 Nov MPI Communication (ppt) (pdf)
Non-blocking communication, One-sided communication 		- P04 -
15 Nov Class cancelled
- 		- - -
17 Nov Guest Lecture, Ganesh Gopalakrishnan
- 		- - ISP
Parallel Algorithms
22 Nov Data Parallel Algorithms (ppt) (pdf)
N-body 		Chapter 6.1 - -
29 Nov Parallel Graph Algorithms (ppt) (pdf)
Tree Search, Traveling Salesperson Problem 		Chapter 6.2 - -
01 Dec Course Retrospective and Future Directions for Parallel Computing (ppt)
(pdf)
Where the field is going 		Chapter 7 - -
Project Presentations
06 Dec Project Presentations, Dry Run
- 		- - -
08 Dec Project Presentations Poster Session
- 		- - -
13 Dec -
- 		- - -
15 Dec Final exam
- 		- - -

 Assignments

All written homeworks are due on the Wednesday 8 days from when they are assigned. See the schedule above for due dates. Use handin , and submit PDF files. Programming assignments will sometimes be given more time if they require more depth. The final programming assignment will be a group project for teams of 1 to 3 students.

Written Homework

Programming Assignments Final Project The final project will be a group project with teams of 2 to 3 students (single-student projects may be approved by the instructor in some cases). You will combine two of the parallel programming models we learned in class and develop a non-trivial application.
  • Project description
  • Poster presentation
  • Poster dry run in class, Tues., Dec. 6
  • Presentation in class, Thurs., Dec. 8
  • 2-4 page report summarizing poster and project completion and software, due 11:59PM, Thurs., Dec. 15

    Final Exam Take-home exam final , due 11:59PM, Thurs., Dec. 15

  • cg-seq.f cg-mpi.f

     Policies

    Laptops in class: Laptops should not be used in class, unless you have an ADA exemption. (see ADA policy below)

    Cheating: Any assignment or exam that is handed in must be your own work. It must be in your own words, and based on your own understanding of the solution. However, talking with one another to understand the material better is strongly encouraged. When taking an exam, you must work independently. Any collaboration during an exam will be considered cheating. Any student who is caught cheating will be given an E in the course and referred to the University Student Behavior Committee.

    ADA: The University of Utah conforms to all standards of the Americans with Disabilities Act (ADA). If you wish to qualify for exemptions under this act, notify the Center for Disabled Students Services, 160 Union.

    College guidelines: Document concerning adding, dropping, etc. here.