Integration, orchestration & processing
Airbyte
Open EL platform with hundreds of ready-made connectors to data sources.
What is it
Airbyte is an open data integration platform with a large catalog of ready-made connectors to databases, APIs and SaaS services. It pulls data from sources and loads it into the warehouse on a schedule, supporting incremental synchronization and CDC. You can use the cloud version or deploy it self-hosted within your own perimeter.
Where and why it is used
I use Airbyte to quickly connect sources to the warehouse without writing connectors from scratch — CRMs, advertising accounts, databases, APIs. This speeds up populating the DWH with data and helps with migration, while the raw data is then transformed via dbt.
Used in my services
Alternatives
FivetranMeltanoStitchApache NiFiCustom-built connectors
Pros and cons
Pros
- Hundreds of ready-made connectors out of the box
- Open source, can be deployed self-hosted
- Incremental synchronization and CDC
- Fast onboarding of new sources
Cons
- – Connector quality is uneven
- – Self-hosted requires resources and maintenance
- – Loading only — transformations are separate (dbt)
- – Optimization is needed at large volumes
