Apache Airflow

11.

Airflow Providers and Extensibility

11.1.

Provider System Overview

11.1.1.

Provider Architecture

11.1.2.

Provider Package Structure

11.1.3.

Provider Versioning

11.1.4.

Provider Installation

11.2.

Core Provider Packages

11.2.1.

Amazon (AWS) Provider

11.2.1.1.

S3 Operators and Sensors

11.2.1.2.

11.2.1.3.

11.2.1.4.

Lambda Operators

11.2.2.

Google Cloud (GCP) Provider

11.2.2.1.

BigQuery Operators

11.2.2.2.

Cloud Storage Operators

11.2.2.3.

Compute Engine Operators

11.2.2.4.

Dataflow Operators

11.2.3.

Microsoft Azure Provider

11.2.3.1.

Blob Storage Operators

11.2.3.2.

Data Factory Operators

11.2.3.3.

Container Instances

11.2.4.

Database Providers

11.2.4.1.

PostgreSQL Provider

11.2.4.2.

11.2.4.3.

SQLite Provider

11.2.5.

Other Popular Providers

11.2.5.1.

Snowflake Provider

11.2.5.2.

Databricks Provider

11.2.5.3.

Kubernetes Provider

11.2.5.4.

Docker Provider

11.3.

Creating Custom Components

11.3.1.

Custom Operators

11.3.1.1.

Inheriting from BaseOperator

11.3.1.2.

Implementing Execute Method

11.3.1.3.

Parameter Validation

11.3.1.4.

Template Fields

11.3.2.

11.3.2.1.

Hook Base Classes

11.3.2.2.

Connection Management

11.3.2.3.

API Integration Patterns

11.3.3.

11.3.3.1.

Sensor Implementation

11.3.3.2.

Poke vs Reschedule Logic

11.3.3.3.

Custom Sensor Patterns

11.3.4.

Custom Timetables

11.3.4.1.

Timetable Interface

11.3.4.2.

Custom Scheduling Logic

11.4.

Plugin Development

11.4.1.

Plugin Architecture

11.4.2.

Plugin Structure

11.4.3.

Installing Custom Plugins

11.4.4.

Plugin Distribution

11.5.

Community Contributions

11.5.1.

Contributing to Providers

11.5.2.

Provider Development Guidelines

11.5.3.

Testing Provider Components

Previous

10. Advanced DAG Authoring Techniques

Go to top

Next

12. Testing and Debugging Airflow DAGs