Apache Airflow

  1. Advanced DAG Authoring Techniques
    1. Dynamic DAG Generation
      1. Configuration-driven DAGs
        1. Creating DAGs from External Sources
          1. Using Loops to Generate Tasks
            1. Parameterizing DAGs
              1. DAG Factory Patterns
              2. Task Groups
                1. Task Group Concept
                  1. Creating Task Groups
                    1. Task Group Decorator
                      1. Nested Task Groups
                        1. Task Group Dependencies
                          1. UI Representation
                          2. SubDAGs (Legacy)
                            1. SubDAG Concept and Usage
                              1. SubDAG Limitations
                                1. Performance Considerations
                                  1. Migration to Task Groups
                                  2. Branching and Conditional Logic
                                    1. Conditional Task Execution
                                      1. BranchPythonOperator
                                        1. ShortCircuitOperator
                                          1. Skip Patterns
                                            1. Complex Branching Logic
                                            2. Parallel Processing
                                              1. Task Parallelism
                                                1. Dynamic Task Generation
                                                  1. Parallel Branches
                                                    1. Resource Management
                                                    2. Pools and Resource Management
                                                      1. Pool Concept
                                                        1. Limiting Task Concurrency
                                                          1. Configuring Pools
                                                            1. Assigning Tasks to Pools
                                                              1. Pool Monitoring
                                                              2. Queues and Worker Assignment
                                                                1. Queue Concept
                                                                  1. Assigning Tasks to Specific Workers
                                                                    1. Queue Configuration for CeleryExecutor
                                                                      1. Load Balancing Strategies
                                                                      2. Task Priority and Weight
                                                                        1. Priority Weight System
                                                                          1. Setting priority_weight
                                                                            1. Weight Rules
                                                                              1. Impact on Task Scheduling
                                                                                1. Priority Inheritance
                                                                                2. Error Handling and Recovery
                                                                                  1. Retry Configuration
                                                                                    1. Exponential Backoff
                                                                                      1. Failure Callbacks
                                                                                        1. Success Callbacks
                                                                                          1. SLA Configuration
                                                                                            1. Task Timeout Handling

                                                                                          Previous

                                                                                          9. Data Sharing and Communication

                                                                                          Go to top

                                                                                          Next

                                                                                          11. Airflow Providers and Extensibility