Transformer deep learning architecture

  1. Transformer Decoder
    1. Decoder Layer Architecture
      1. Sub-layer Organization
        1. Masked Multi-Head Self-Attention
          1. Encoder-Decoder Attention
            1. Position-wise Feed-Forward Network
              1. Residual Connections and Normalization
              2. Autoregressive Property
              3. Masked Multi-Head Self-Attention
                1. Masking Necessity
                  1. Preventing Future Information Access
                    1. Autoregressive Generation
                      1. Training vs Inference Consistency
                      2. Look-Ahead Mask Implementation
                        1. Mask Matrix Construction
                          1. Upper Triangular Masking
                            1. Attention Score Modification
                              1. Softmax Application
                              2. Causal Attention Pattern
                              3. Encoder-Decoder Attention (Cross-Attention)
                                1. Cross-Attention Mechanism
                                  1. Query Source (Decoder)
                                    1. Key and Value Source (Encoder)
                                      1. Information Integration
                                      2. Encoder-Decoder Alignment
                                        1. Source-Target Correspondence
                                          1. Dynamic Context Selection
                                          2. Attention Pattern Analysis
                                          3. Position-wise Feed-Forward Network
                                            1. Identical Structure to Encoder
                                              1. Independent Processing
                                              2. Residual Connections and Normalization
                                                1. Consistent Application
                                                  1. Gradient Flow Maintenance
                                                  2. Decoder Stack
                                                    1. Layer Stacking
                                                      1. Autoregressive Processing
                                                        1. Output Generation Preparation

                                                      Previous

                                                      3. Transformer Encoder

                                                      Go to top

                                                      Next

                                                      5. Output Generation and Decoding