Dagster

Core capability for building, scaling, and observing AI and data pipelines.

Built-in catalog for data assets, tables, files, ML models, and notebooks.

Features for monitoring data quality, freshness, and asset health, including asset checks and metadata-bound checks.

Provides insights into the cost of materializing individual assets and longitudinal reporting on metadata.

Reusable components to focus on business logic and reduce boilerplate.

Built-in integrations with a wide range of tools (e.g., S3, Snowflake, PowerBI, dbt, Airbyte, Fivetran, Tableau, Looker, Sigma, Spark, Pandas, Atlan).

Includes features like role-based access management, component-level isolation, and SAML-based SSO.

Supports the creation and management of ETL/ELT data pipelines.

Optimized for building and orchestrating AI and machine learning workflows.

Tools to facilitate the modernization of data platforms.

Capabilities for building and shipping data products faster.

Allows developers to test pipelines locally in any dev stage.

Supports automatic deployment to staging environments for confident shipping.

Provides a unified view across teams for collaboration and governance.

Automatically tracks and documents the complete lineage of data assets.

Centralizes metadata with built-in observability and diagnostics.

Automatically tracks, documents, and audits datasets for integrity and compliance.

Uses new AutomationCondition APIs for automated materialization and pipeline control.

Enables teams to focus on business logic by providing reusable components.

Tools to automate, monitor, and optimize data pipelines.

Built with software engineering best practices for testing code in any stage, not just production.

Features for debugging runs, querying logs, and viewing Gantt charts for diagnostics.

Can trigger Slack or email notifications on run failure/success, schedule/sensor tick failure.

Manages the queuing and launching of runs and backfill jobs.

Manages schedules, sensors, and run queuing, and performs periodic tasks.

Creates runs based on external state changes (e.g., new file, external system issues).

Handles run worker failures.

Supports various execution engines like Kubernetes, Amazon ECS, and custom launchers.

Manages versions of code and data, with caching capabilities.

Specific checks associated with assets to ensure expectations are met.

Defines data assets and their computation through a declarative approach.

Strong typing system to validate inputs and outputs of each operation (op).

Facilitates a single, unified codebase for data platforms.

Offers a free, comprehensive seven-lesson course on ETL implementation with Dagster.

Available as an open-source library for building data systems.