Parallel and Distributed Computing

  1. Parallel Computing Fundamentals
    1. Parallel Computer Architectures
      1. Flynn's Taxonomy
        1. SISD Architecture
          1. SIMD Architecture
            1. MISD Architecture
              1. MIMD Architecture
              2. Memory Organization
                1. Shared Memory Systems
                  1. Uniform Memory Access
                    1. Non-Uniform Memory Access
                      1. Cache Coherence Protocols
                        1. Memory Consistency Models
                        2. Distributed Memory Systems
                          1. Message Passing Architecture
                            1. Hybrid Memory Systems
                          2. Processor Types
                            1. Multicore Processors
                              1. Many-core Processors
                                1. Graphics Processing Units
                                  1. Vector Processors
                                  2. Interconnection Networks
                                    1. Network Topologies
                                      1. Bus Networks
                                        1. Ring Networks
                                          1. Mesh Networks
                                            1. Torus Networks
                                              1. Hypercube Networks
                                                1. Tree Networks
                                                  1. Fat Tree Networks
                                                    1. Crossbar Networks
                                                    2. Routing Algorithms
                                                      1. Deterministic Routing
                                                        1. Adaptive Routing
                                                          1. Minimal vs Non-minimal Routing
                                                          2. Switching Techniques
                                                            1. Circuit Switching
                                                              1. Packet Switching
                                                                1. Wormhole Routing
                                                                  1. Virtual Cut-through
                                                              2. Parallel Algorithm Design
                                                                1. Decomposition Strategies
                                                                  1. Task Decomposition
                                                                    1. Functional Decomposition
                                                                      1. Domain Decomposition
                                                                        1. Recursive Decomposition
                                                                        2. Data Decomposition
                                                                          1. Block Distribution
                                                                            1. Cyclic Distribution
                                                                              1. Block-cyclic Distribution
                                                                                1. Random Distribution
                                                                              2. Granularity Considerations
                                                                                1. Fine-grained Parallelism
                                                                                  1. Coarse-grained Parallelism
                                                                                    1. Granularity Trade-offs
                                                                                    2. Load Balancing
                                                                                      1. Static Load Balancing
                                                                                        1. Dynamic Load Balancing
                                                                                          1. Work Stealing
                                                                                            1. Work Sharing
                                                                                            2. Communication Patterns
                                                                                              1. Point-to-point Communication
                                                                                                1. Collective Communication
                                                                                                  1. Broadcast Operations
                                                                                                    1. Reduction Operations
                                                                                                      1. Scatter and Gather Operations
                                                                                                      2. Synchronization Requirements
                                                                                                        1. Barrier Synchronization
                                                                                                          1. Producer-Consumer Synchronization
                                                                                                            1. Reader-Writer Synchronization
                                                                                                          2. Parallel Programming Models
                                                                                                            1. Shared Memory Programming
                                                                                                              1. Thread-based Programming
                                                                                                                1. Thread Creation and Management
                                                                                                                  1. Thread Synchronization
                                                                                                                    1. Thread Pools
                                                                                                                    2. Process-based Programming
                                                                                                                      1. Process Creation
                                                                                                                        1. Inter-process Communication
                                                                                                                        2. Synchronization Primitives
                                                                                                                          1. Mutexes and Locks
                                                                                                                            1. Semaphores
                                                                                                                              1. Condition Variables
                                                                                                                                1. Monitors
                                                                                                                                  1. Atomic Operations
                                                                                                                                  2. Memory Models
                                                                                                                                    1. Sequential Consistency
                                                                                                                                      1. Relaxed Memory Models
                                                                                                                                        1. Memory Barriers
                                                                                                                                      2. Message Passing Programming
                                                                                                                                        1. Point-to-point Communication
                                                                                                                                          1. Blocking Communication
                                                                                                                                            1. Non-blocking Communication
                                                                                                                                              1. Buffered Communication
                                                                                                                                              2. Collective Communication
                                                                                                                                                1. Broadcast
                                                                                                                                                  1. Scatter and Gather
                                                                                                                                                    1. All-to-all Communication
                                                                                                                                                      1. Reduction Operations
                                                                                                                                                      2. Communication Modes
                                                                                                                                                        1. Synchronous Communication
                                                                                                                                                          1. Asynchronous Communication
                                                                                                                                                            1. Ready Communication
                                                                                                                                                          2. Data Parallel Programming
                                                                                                                                                            1. SIMD Programming
                                                                                                                                                              1. Vector Operations
                                                                                                                                                                1. Parallel Loops
                                                                                                                                                                  1. Data Distribution
                                                                                                                                                                  2. Task Parallel Programming
                                                                                                                                                                    1. Fork-Join Model
                                                                                                                                                                      1. Master-Worker Model
                                                                                                                                                                        1. Pipeline Model
                                                                                                                                                                      2. Parallel Programming Languages and APIs
                                                                                                                                                                        1. OpenMP
                                                                                                                                                                          1. Compiler Directives
                                                                                                                                                                            1. Parallel Regions
                                                                                                                                                                              1. Work Sharing Constructs
                                                                                                                                                                                1. Synchronization Constructs
                                                                                                                                                                                  1. Data Environment Clauses
                                                                                                                                                                                    1. Task Parallelism
                                                                                                                                                                                    2. MPI
                                                                                                                                                                                      1. Point-to-point Communication
                                                                                                                                                                                        1. Collective Communication
                                                                                                                                                                                          1. Communicators and Groups
                                                                                                                                                                                            1. Derived Data Types
                                                                                                                                                                                              1. One-sided Communication
                                                                                                                                                                                                1. Parallel I/O
                                                                                                                                                                                                2. CUDA Programming
                                                                                                                                                                                                  1. GPU Architecture
                                                                                                                                                                                                    1. Kernel Functions
                                                                                                                                                                                                      1. Thread Hierarchy
                                                                                                                                                                                                        1. Memory Hierarchy
                                                                                                                                                                                                          1. Memory Management
                                                                                                                                                                                                            1. Performance Optimization
                                                                                                                                                                                                            2. OpenCL
                                                                                                                                                                                                              1. Platform Model
                                                                                                                                                                                                                1. Execution Model
                                                                                                                                                                                                                  1. Memory Model
                                                                                                                                                                                                                    1. Programming Model
                                                                                                                                                                                                                    2. Pthreads
                                                                                                                                                                                                                      1. Thread Management
                                                                                                                                                                                                                        1. Synchronization Primitives
                                                                                                                                                                                                                          1. Thread Attributes
                                                                                                                                                                                                                          2. Java Concurrency
                                                                                                                                                                                                                            1. Thread Class
                                                                                                                                                                                                                              1. Runnable Interface
                                                                                                                                                                                                                                1. Synchronization Keywords
                                                                                                                                                                                                                                  1. Concurrent Collections
                                                                                                                                                                                                                                    1. Executor Framework

                                                                                                                                                                                                                                Previous

                                                                                                                                                                                                                                1. Introduction to Parallel and Distributed Computing

                                                                                                                                                                                                                                Go to top

                                                                                                                                                                                                                                Next

                                                                                                                                                                                                                                3. Parallel Algorithms and Applications