UsefulLinks
Computer Science
Big Data
Apache Airflow
1. Introduction to Apache Airflow
2. Core Concepts of Airflow
3. Airflow Architecture and Components
4. Setting Up an Airflow Environment
5. Authoring Your First DAG
6. Comprehensive Guide to Operators
7. Managing Workflows with the Airflow UI
8. Scheduling and Triggers
9. Data Sharing and Communication
10. Advanced DAG Authoring Techniques
11. Airflow Providers and Extensibility
12. Testing and Debugging Airflow DAGs
13. Operational Airflow Management
14. Scaling and Production Deployment
15. Airflow Best Practices and Patterns
4.
Setting Up an Airflow Environment
4.1.
Prerequisites and System Requirements
4.1.1.
Supported Python Versions
4.1.2.
Operating System Requirements
4.1.3.
Hardware Recommendations
4.1.4.
pip and Virtual Environments
4.1.5.
Database Backend Requirements
4.1.6.
System Dependencies
4.2.
Installation Methods
4.2.1.
Installing with pip
4.2.1.1.
Installing Airflow Core
4.2.1.2.
Installing Provider Packages
4.2.1.3.
Constraint Files
4.2.2.
Using the Official Docker Image
4.2.2.1.
Docker Compose Setup
4.2.2.2.
Customizing Docker Images
4.2.2.3.
Volume Mounting
4.2.3.
Installing from Source
4.2.3.1.
Development Setup
4.2.3.2.
Building from Git Repository
4.2.4.
Kubernetes Deployment
4.2.4.1.
Helm Charts
4.2.4.2.
Official Kubernetes Operator
4.3.
Initial Configuration
4.3.1.
The airflow.cfg File
4.3.1.1.
Configuration File Structure
4.3.1.2.
Key Configuration Sections
4.3.1.3.
Modifying Default Settings
4.3.2.
Setting the AIRFLOW_HOME Environment Variable
4.3.3.
Configuring Environment Variables
4.3.4.
Configuration Precedence
4.4.
Database Setup
4.4.1.
Initializing the Database
4.4.2.
Running airflow db init
4.4.3.
Database Migrations
4.4.4.
Database Connection Configuration
4.5.
User Management
4.5.1.
Creating an Admin User
4.5.2.
Using the CLI to Create Users
4.5.3.
User Roles and Permissions
4.5.4.
Authentication Backends
4.6.
Starting Airflow Services
4.6.1.
Starting the Webserver
4.6.2.
Starting the Scheduler
4.6.3.
Starting the Triggerer
4.6.4.
Starting Workers
4.6.5.
Service Management and Monitoring
4.7.
Verification and Testing
4.7.1.
Verifying Installation
4.7.2.
Running Test DAGs
4.7.3.
Troubleshooting Common Issues
Previous
3. Airflow Architecture and Components
Go to top
Next
5. Authoring Your First DAG