Apache Airflow

10.

Advanced DAG Authoring Techniques

10.1.

Dynamic DAG Generation

10.1.1.

Configuration-driven DAGs

10.1.2.

Creating DAGs from External Sources

10.1.3.

Using Loops to Generate Tasks

10.1.4.

Parameterizing DAGs

10.1.5.

DAG Factory Patterns

10.2.

10.2.1.

Task Group Concept

10.2.2.

Creating Task Groups

10.2.3.

Task Group Decorator

10.2.4.

Nested Task Groups

10.2.5.

Task Group Dependencies

10.2.6.

UI Representation

10.3.

SubDAGs (Legacy)

10.3.1.

SubDAG Concept and Usage

10.3.2.

SubDAG Limitations

10.3.3.

Performance Considerations

10.3.4.

Migration to Task Groups

10.4.

Branching and Conditional Logic

10.4.1.

Conditional Task Execution

10.4.2.

BranchPythonOperator

10.4.3.

ShortCircuitOperator

10.4.4.

10.4.5.

Complex Branching Logic

10.5.

Parallel Processing

10.5.1.

Task Parallelism

10.5.2.

Dynamic Task Generation

10.5.3.

Parallel Branches

10.5.4.

Resource Management

10.6.

Pools and Resource Management

10.6.1.

10.6.2.

Limiting Task Concurrency

10.6.3.

Configuring Pools

10.6.4.

Assigning Tasks to Pools

10.6.5.

Pool Monitoring

10.7.

Queues and Worker Assignment

10.7.1.

10.7.2.

Assigning Tasks to Specific Workers

10.7.3.

Queue Configuration for CeleryExecutor

10.7.4.

Load Balancing Strategies

10.8.

Task Priority and Weight

10.8.1.

Priority Weight System

10.8.2.

Setting priority_weight

10.8.3.

10.8.4.

Impact on Task Scheduling

10.8.5.

Priority Inheritance

10.9.

Error Handling and Recovery

10.9.1.

Retry Configuration

10.9.2.

Exponential Backoff

10.9.3.

Failure Callbacks

10.9.4.

Success Callbacks

10.9.5.

SLA Configuration

10.9.6.

Task Timeout Handling

Previous

9. Data Sharing and Communication

Go to top

Next

11. Airflow Providers and Extensibility