Distributed Deep Learning Training

  1. Communication Optimization
    1. Computation-Communication Overlap
      1. Asynchronous Communication
        1. Pipeline Scheduling
          1. Gradient Bucketing
            1. Communication Hiding Techniques
            2. Gradient Compression
              1. Quantization Techniques
                1. Half-Precision Training
                  1. Mixed Precision Training
                    1. Integer Quantization
                      1. Dynamic Range Scaling
                      2. Sparsification Methods
                        1. Top-k Sparsification
                          1. Threshold-based Sparsification
                            1. Random Sparsification
                              1. Structured Sparsification
                              2. Error Compensation
                                1. Error Feedback Methods
                                  1. Momentum Correction
                                    1. Convergence Guarantees
                                  2. Communication Scheduling
                                    1. Gradient Accumulation
                                      1. Communication Frequency Control
                                        1. Adaptive Communication
                                          1. Priority-based Scheduling
                                          2. Memory Optimization
                                            1. Gradient Checkpointing
                                              1. Activation Recomputation
                                                1. Memory-Efficient Attention
                                                  1. Parameter Offloading

                                                Previous

                                                5. Communication in Distributed Training

                                                Go to top

                                                Next

                                                7. System and Hardware Considerations