Transformer deep learning architecture

  1. Output Generation and Decoding
    1. Final Linear Transformation
      1. Vocabulary Projection
        1. Hidden Dimension to Vocabulary Size
          1. Logit Computation
          2. Parameter Sharing
            1. Embedding Matrix Reuse
          3. Probability Distribution Generation
            1. Softmax Application
              1. Logit to Probability Conversion
                1. Temperature Scaling
                2. Next Token Prediction
                3. Decoding Strategies
                  1. Greedy Decoding
                    1. Highest Probability Selection
                      1. Deterministic Output
                        1. Local Optimality
                        2. Sampling-based Decoding
                          1. Random Sampling
                            1. Temperature-controlled Sampling
                              1. Top-k Sampling
                                1. Top-p (Nucleus) Sampling
                                2. Beam Search Decoding
                                  1. Multiple Hypothesis Tracking
                                    1. Beam Width Parameter
                                      1. Length Normalization
                                      2. Decoding Strategy Comparison

                                    Previous

                                    4. Transformer Decoder

                                    Go to top

                                    Next

                                    6. Training Methodology