Transformer deep learning architecture

2.

The Original Transformer Architecture

2.1.

Architectural Overview

2.1.1.

Motivation and Design Philosophy

2.1.1.1.

Moving Beyond Recurrence

2.1.1.2.

Parallelization Benefits

2.1.1.3.

Attention-Only Architecture

2.1.2.

High-Level Structure

2.1.2.1.

Encoder-Decoder Stack

2.1.2.2.

Layer Organization

2.1.2.3.

Information Flow

2.1.3.

Key Innovations

2.1.3.1.

Self-Attention Mechanism

2.1.3.2.

Multi-Head Attention

2.1.3.3.

Positional Encoding

2.1.3.4.

Layer Normalization Placement

2.2.

Input Representation

2.2.1.

Tokenization Strategies

2.2.1.1.

Word-level Tokenization

2.2.1.2.

Subword Tokenization

2.2.1.2.1.

Byte-Pair Encoding (BPE)

2.2.1.2.2.

2.2.1.2.3.

2.2.1.3.

Character-level Tokenization

2.2.1.4.

Vocabulary Construction

2.2.1.4.1.

Vocabulary Size Considerations

2.2.1.4.2.

Out-of-Vocabulary Handling

2.2.2.

Token Embeddings

2.2.2.1.

Embedding Layer

2.2.2.1.1.

Lookup Table Mechanism

2.2.2.1.2.

Embedding Dimension Selection

2.2.2.1.3.

Parameter Initialization

2.2.2.2.

Embedding Learning

2.2.2.2.1.

Gradient Updates

2.2.2.2.2.

Semantic Representation

2.2.3.

Positional Encoding

2.2.3.1.

Need for Position Information

2.2.3.1.1.

Permutation Invariance Problem

2.2.3.1.2.

Sequential Order Importance

2.2.3.2.

Sinusoidal Positional Encoding

2.2.3.2.1.

Mathematical Formulation

2.2.3.2.2.

Sine and Cosine Functions

2.2.3.2.3.

Frequency Variations

2.2.3.2.4.

Position-dependent Patterns

2.2.3.2.5.

Extrapolation Properties

2.2.3.3.

Learned Positional Embeddings

2.2.3.3.1.

Trainable Parameters

2.2.3.3.2.

Comparison with Sinusoidal

2.2.3.3.3.

Length Limitations

2.2.3.4.

Positional Encoding Addition

2.2.3.4.1.

Element-wise Addition

2.2.3.4.2.

Embedding Dimension Matching

Previous

1. Foundational Concepts and Predecessors

Go to top

Next

3. Transformer Encoder