Transformer deep learning architecture

  1. Advanced Attention Mechanisms
    1. Relative Position Encoding
      1. Relative Position Representations
        1. Shaw et al. Approach
          1. T5 Relative Position Bias
            1. RoPE (Rotary Position Embedding)
            2. Sparse Attention Patterns
              1. Local Attention Windows
                1. Strided Attention
                  1. Random Attention
                    1. Structured Sparsity
                    2. Multi-Scale Attention
                      1. Hierarchical Attention
                        1. Multi-Resolution Processing
                          1. Pyramid Attention
                          2. Cross-Modal Attention
                            1. Vision-Language Models
                              1. Audio-Text Alignment
                                1. Multimodal Fusion

                              Previous

                              10. Transformer Variants and Evolution

                              Go to top

                              Next

                              12. Applications and Adaptations