PyTorch Library

  1. Automatic Differentiation
    1. Computational Graphs
      1. Graph Construction
        1. Dynamic Graph Building
          1. Forward Pass Graph Creation
          2. Graph Components
            1. Leaf Nodes
              1. Intermediate Nodes
                1. Function Nodes
                2. Graph Traversal
                  1. Topological Ordering
                    1. Backward Pass Execution
                  2. Gradient Computation Fundamentals
                    1. The Chain Rule
                      1. Mathematical Foundation
                        1. Computational Implementation
                        2. Jacobian Matrices
                          1. Definition and Properties
                            1. Vector-Jacobian Products
                            2. Gradient Accumulation
                              1. Multiple Backward Passes
                                1. Gradient Storage
                              2. The requires_grad System
                                1. Enabling Gradient Tracking
                                  1. Tensor-level Control
                                    1. Default Behaviors
                                    2. Gradient Propagation
                                      1. Automatic Propagation Rules
                                        1. Stopping Gradient Flow
                                        2. Leaf Tensor Concepts
                                          1. User-created Tensors
                                            1. Parameter Tensors
                                          2. Backward Pass Mechanics
                                            1. The backward() Method
                                              1. Scalar Output Backward
                                                1. Vector Output Backward
                                                  1. Gradient Arguments
                                                  2. Gradient Computation
                                                    1. Automatic Gradient Calculation
                                                      1. Custom Gradient Functions
                                                      2. Gradient Storage
                                                        1. The grad Attribute
                                                          1. Gradient Accumulation Behavior
                                                        2. Controlling Gradient Computation
                                                          1. Context Managers
                                                            1. torch.no_grad()
                                                              1. torch.enable_grad()
                                                                1. torch.set_grad_enabled()
                                                                2. Tensor Detachment
                                                                  1. detach() Method
                                                                    1. Breaking Gradient Flow
                                                                    2. In-place Operations
                                                                      1. Gradient Tracking with In-place Ops
                                                                        1. Potential Issues and Solutions
                                                                      2. Advanced Gradient Topics
                                                                        1. Higher-order Derivatives
                                                                          1. Second Derivatives
                                                                            1. create_graph Parameter
                                                                            2. Custom Autograd Functions
                                                                              1. Function Class Definition
                                                                                1. Forward and Backward Methods
                                                                                2. Gradient Checkpointing
                                                                                  1. Memory-efficient Training
                                                                                    1. Trade-offs and Implementation
                                                                                  2. Common Issues and Debugging
                                                                                    1. Gradient Flow Problems
                                                                                      1. Vanishing Gradients
                                                                                        1. Exploding Gradients
                                                                                        2. Runtime Errors
                                                                                          1. Leaf Tensor Modification
                                                                                            1. Graph Retention Issues
                                                                                            2. Debugging Techniques
                                                                                              1. Gradient Checking
                                                                                                1. Graph Visualization

                                                                                            Previous

                                                                                            3. Tensor Operations and Manipulation

                                                                                            Go to top

                                                                                            Next

                                                                                            5. Neural Network Construction