UsefulLinks
Computer Science
Big Data
Apache Airflow
1. Introduction to Apache Airflow
2. Core Concepts of Airflow
3. Airflow Architecture and Components
4. Setting Up an Airflow Environment
5. Authoring Your First DAG
6. Comprehensive Guide to Operators
7. Managing Workflows with the Airflow UI
8. Scheduling and Triggers
9. Data Sharing and Communication
10. Advanced DAG Authoring Techniques
11. Airflow Providers and Extensibility
12. Testing and Debugging Airflow DAGs
13. Operational Airflow Management
14. Scaling and Production Deployment
15. Airflow Best Practices and Patterns
3.
Airflow Architecture and Components
3.1.
Overall Architecture Overview
3.1.1.
Multi-Component Architecture
3.1.2.
Communication Between Components
3.1.3.
Deployment Patterns
3.2.
The Metastore Database
3.2.1.
Role in State Management
3.2.2.
Storing DAG Metadata
3.2.3.
Storing Task and DAG Run States
3.2.4.
Connection and Variable Storage
3.2.5.
Supported Database Backends
3.2.5.1.
SQLite
3.2.5.2.
PostgreSQL
3.2.5.3.
MySQL
3.2.5.4.
SQL Server
3.2.6.
Database Schema Overview
3.3.
The Webserver
3.3.1.
User Interface Overview
3.3.2.
Web Framework and Technology Stack
3.3.3.
Navigating the UI
3.3.4.
REST API Endpoints
3.3.5.
Authentication and Authorization
3.3.6.
Configuration Options
3.4.
The Scheduler
3.4.1.
Core Scheduling Logic
3.4.2.
Parsing and Loading DAGs
3.4.3.
DAG Bag Management
3.4.4.
Triggering DAG Runs
3.4.5.
Task Scheduling Logic
3.4.6.
Handling Backfills
3.4.7.
Scheduler Performance Considerations
3.5.
Executors
3.5.1.
Role in Task Execution
3.5.2.
Executor Selection Criteria
3.5.3.
Types of Executors
3.5.3.1.
SequentialExecutor
3.5.3.2.
LocalExecutor
3.5.3.3.
CeleryExecutor
3.5.3.4.
KubernetesExecutor
3.5.3.5.
CeleryKubernetesExecutor
3.5.3.6.
DebugExecutor
3.5.4.
Executor Configuration
3.5.5.
Executor Comparison and Use Cases
3.6.
The Triggerer
3.6.1.
Asynchronous Task Execution Model
3.6.2.
Event-driven Task Handling
3.6.3.
Use Cases for Deferrable Operators
3.6.4.
Resource Efficiency Benefits
3.6.5.
Configuration and Setup
3.7.
Workers
3.7.1.
Worker Processes in Different Executors
3.7.2.
Worker Configuration
3.7.3.
Worker Scaling Strategies
Previous
2. Core Concepts of Airflow
Go to top
Next
4. Setting Up an Airflow Environment