airflow

It's designed to programmatically define, schedule, and monitor complex data workflows.

Python45.7K17.2KUpdated 15d agoFeatured

Quick Overview

What is this?

Airflow allows developers to create data pipelines using Python code, defining tasks within Directed Acyclic Graphs (DAGs). It includes a built-in scheduler to run these DAGs at specified intervals and a web UI for visually tracking execution, progress, and logs. You'd typically write `.py` files containing DAG definitions and use commands like `airflow scheduler` and `airflow webserver` to get things running.

What problem does it solve?

It's designed to programmatically define, schedule, and monitor complex data workflows.

Who should use it?

Teams managing complex, scheduled data processing jobs or long-running operational workflows will find Airflow invaluable.

Setup difficulty:Medium

Pros

Pythonic Workflow Definitions: Define complex data pipelines entirely in Python code using DAGs.
Rich Ecosystem of Integrations: Features many built-in operators and sensors for common systems like cloud services, databases, and message queues.

Interactive Web Interface: Provides a dashboard to monitor, troubleshoot, and manage workflow execution, including task logs and retries.

Cons

Can Be Resource Intensive: Running a busy Airflow instance, particularly with many concurrent tasks or complex executors, often requires substantial CPU and memory.
Steep Learning Curve for Newcomers: Understanding DAGs, operators, and the overall system architecture can take time for those new to workflow orchestration concepts.

Scores

Trust Score

77

Star reputation (15%)93

Star velocity 7d (15%)0

Commit recency (15%)80

Fork ratio (10%)100

Issue ratio (10%)62

Contributor signal (10%)100

README quality (5%)100

License (5%)100

Homepage/demo (5%)100

Docs URL (5%)0

Topic count (5%)100

Maintenance

67

Commit frequency95

Issue management25

Documentation85

Popularity

69

Stars100

Forks100

Growth trend10

Star History

Snapshot Versions

Version	Commit	Size	Downloads	Date
latestLatest	HEAD	74.7 MB	3	1mo ago

Alternatives

tensorflow

80

An Open Source Machine Learning Framework for Everyone

Featured

C++195.4K75.3K15d ago

stable-diffusion-webui

AUTOMATIC1111

62

Stable Diffusion web UI

Python163.4K30.4K3mo ago

transformers

huggingface

79

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

Featured

Python161.2K33.4K15d ago