Concurrent and Parallel Programming

5.

Parallel Programming Concepts

5.1.

Parallel Architectures

5.1.1.

Flynn's Taxonomy

5.1.1.1.

SISD Architecture

5.1.1.2.

SIMD Architecture

5.1.1.3.

MISD Architecture

5.1.1.4.

MIMD Architecture

5.1.2.

Shared Memory Architectures

5.1.2.1.

Uniform Memory Access

5.1.2.2.

Non-Uniform Memory Access

5.1.2.3.

Cache Coherence Protocols

5.1.3.

Distributed Memory Architectures

5.1.3.1.

Networked Nodes

5.1.3.2.

Message Passing in Distributed Systems

5.1.4.

Hybrid Architectures

5.1.4.1.

Combining Shared and Distributed Memory

5.1.4.2.

Clustered Multi-Core Systems

5.2.

Performance Metrics and Laws

5.2.1.

5.2.1.1.

Speedup Definition

5.2.1.2.

Speedup Calculation

5.2.2.

5.2.2.1.

Efficiency Definition

5.2.2.2.

Efficiency Calculation

5.2.3.

5.2.3.1.

5.2.3.2.

5.2.4.

5.2.4.1.

Theoretical Speedup Limit

5.2.4.2.

Identifying Serial Bottlenecks

5.2.4.3.

Implications for Parallelization

5.2.5.

Gustafson's Law

5.2.5.1.

5.2.5.2.

Implications for Large Problem Sizes

5.3.

Parallel Decomposition Strategies

5.3.1.

Task Parallelism

5.3.1.1.

Dividing Work by Function

5.3.1.2.

Dividing Work by Task

5.3.1.3.

Task Parallelism Use Cases

5.3.1.4.

Task Parallelism Examples

5.3.2.

Data Parallelism

5.3.2.1.

Dividing Data Among Processing Units

5.3.2.2.

Data Parallelism Use Cases

5.3.2.3.

Data Parallelism Examples

5.3.3.

Pipeline Parallelism

5.3.3.1.

Stages of Computation

5.3.3.2.

Throughput Considerations

5.3.3.3.

Latency Considerations

Previous

4. Concurrency Models and Patterns

Go to top

Next

6. Parallel Programming Patterns and Algorithms