Distributed Deep Learning Training

6.

Communication Optimization

6.1.

Computation-Communication Overlap

6.1.1.

Asynchronous Communication

6.1.2.

Pipeline Scheduling

6.1.3.

Gradient Bucketing

6.1.4.

Communication Hiding Techniques

6.2.

Gradient Compression

6.2.1.

Quantization Techniques

6.2.1.1.

Half-Precision Training

6.2.1.2.

Mixed Precision Training

6.2.1.3.

Integer Quantization

6.2.1.4.

Dynamic Range Scaling

6.2.2.

Sparsification Methods

6.2.2.1.

Top-k Sparsification

6.2.2.2.

Threshold-based Sparsification

6.2.2.3.

Random Sparsification

6.2.2.4.

Structured Sparsification

6.2.3.

Error Compensation

6.2.3.1.

Error Feedback Methods

6.2.3.2.

Momentum Correction

6.2.3.3.

Convergence Guarantees

6.3.

Communication Scheduling

6.3.1.

Gradient Accumulation

6.3.2.

Communication Frequency Control

6.3.3.

Adaptive Communication

6.3.4.

Priority-based Scheduling

6.4.

Memory Optimization

6.4.1.

Gradient Checkpointing

6.4.2.

Activation Recomputation

6.4.3.

Memory-Efficient Attention

6.4.4.

Parameter Offloading

Previous

5. Communication in Distributed Training

Go to top

Next

7. System and Hardware Considerations