Apache Airflow

  1. Core Concepts of Airflow
    1. Directed Acyclic Graphs (DAGs)
      1. Definition and Properties of DAGs
        1. DAG as Code Paradigm
          1. DAG Structure and Components
            1. DAG Configuration Parameters
              1. DAG ID
                1. Description
                  1. Schedule Interval
                    1. Start Date
                      1. End Date
                        1. Default Arguments
                        2. DAG Runs
                          1. Scheduling and Execution of DAG Runs
                            1. DAG Run States
                              1. Manual vs Scheduled Runs
                              2. Task Instances
                                1. Mapping Tasks to DAG Runs
                                  1. Task Instance States
                                    1. Task Instance Lifecycle
                                  2. Operators
                                    1. Definition and Purpose
                                      1. Operator Classes and Types
                                        1. Instantiating Operators as Tasks
                                          1. Operator Parameters and Configuration
                                            1. Custom Operator Development
                                              1. Inheriting from BaseOperator
                                                1. Implementing Execute Method
                                                  1. Parameter Validation
                                                2. Tasks and Task Instances
                                                  1. The Basic Unit of Work
                                                    1. Task Definition vs Task Instance
                                                      1. Task Lifecycle
                                                        1. Task Creation
                                                          1. Task Scheduling
                                                            1. Task Execution
                                                              1. Task Completion
                                                              2. Task States
                                                                1. None
                                                                  1. Scheduled
                                                                    1. Queued
                                                                      1. Running
                                                                        1. Success
                                                                          1. Failed
                                                                            1. Up for Retry
                                                                              1. Up for Reschedule
                                                                                1. Upstream Failed
                                                                                  1. Skipped
                                                                                    1. Removed
                                                                                    2. Task Retries and Failure Handling
                                                                                    3. Relationships and Dependencies
                                                                                      1. Upstream and Downstream Concepts
                                                                                        1. Setting Dependencies with Bitshift Operators
                                                                                          1. Explicit Dependency Methods
                                                                                            1. set_upstream Method
                                                                                              1. set_downstream Method
                                                                                              2. Cross Dependencies
                                                                                                1. Dependency Visualization
                                                                                                  1. Complex Dependency Patterns
                                                                                                  2. Templating with Jinja
                                                                                                    1. Purpose and Use Cases
                                                                                                      1. Dynamic Content in DAGs
                                                                                                        1. Template Fields
                                                                                                          1. Pre-defined Variables
                                                                                                            1. Execution Date
                                                                                                              1. DAG Run ID
                                                                                                                1. Task Instance Key
                                                                                                                2. Macros and Custom Templates
                                                                                                                  1. Template Rendering Process

                                                                                                                Previous

                                                                                                                1. Introduction to Apache Airflow

                                                                                                                Go to top

                                                                                                                Next

                                                                                                                3. Airflow Architecture and Components