Apache Airflow

  1. Airflow Architecture and Components
    1. Overall Architecture Overview
      1. Multi-Component Architecture
        1. Communication Between Components
          1. Deployment Patterns
          2. The Metastore Database
            1. Role in State Management
              1. Storing DAG Metadata
                1. Storing Task and DAG Run States
                  1. Connection and Variable Storage
                    1. Supported Database Backends
                      1. SQLite
                        1. PostgreSQL
                          1. MySQL
                            1. SQL Server
                            2. Database Schema Overview
                            3. The Webserver
                              1. User Interface Overview
                                1. Web Framework and Technology Stack
                                  1. REST API Endpoints
                                    1. Authentication and Authorization
                                      1. Configuration Options
                                      2. The Scheduler
                                        1. Core Scheduling Logic
                                          1. Parsing and Loading DAGs
                                            1. DAG Bag Management
                                              1. Triggering DAG Runs
                                                1. Task Scheduling Logic
                                                  1. Handling Backfills
                                                    1. Scheduler Performance Considerations
                                                    2. Executors
                                                      1. Role in Task Execution
                                                        1. Executor Selection Criteria
                                                          1. Types of Executors
                                                            1. SequentialExecutor
                                                              1. LocalExecutor
                                                                1. CeleryExecutor
                                                                  1. KubernetesExecutor
                                                                    1. CeleryKubernetesExecutor
                                                                      1. DebugExecutor
                                                                      2. Executor Configuration
                                                                        1. Executor Comparison and Use Cases
                                                                        2. The Triggerer
                                                                          1. Asynchronous Task Execution Model
                                                                            1. Event-driven Task Handling
                                                                              1. Use Cases for Deferrable Operators
                                                                                1. Resource Efficiency Benefits
                                                                                  1. Configuration and Setup
                                                                                  2. Workers
                                                                                    1. Worker Processes in Different Executors
                                                                                      1. Worker Configuration
                                                                                        1. Worker Scaling Strategies

                                                                                      Previous

                                                                                      2. Core Concepts of Airflow

                                                                                      Go to top

                                                                                      Next

                                                                                      4. Setting Up an Airflow Environment