Integration, orchestration & processing
Apache Kafka
Distributed platform for real-time event streaming.
What is it
Apache Kafka is a distributed platform for streaming and storing events. Source systems publish messages to topics, and consumers read them at their own pace; the data is durably stored and replicated. Kafka handles an enormous flow of events and acts as the «nervous system» for exchanging data between services in real time.
Where and why it is used
I use Kafka when data needs to land in the warehouse as a stream rather than in batches: website and application events, integrations between systems, CDC during a «hot» migration. It is the foundation for near-real-time analytics and resilient data exchange between sources.
Used in my services
Alternatives
RabbitMQRedpandaApache PulsarAWS KinesisNATS
Pros and cons
Pros
- High throughput and scalability
- Durable storage and replay of events
- Decouples data sources from consumers
- The standard for stream processing and CDC
Cons
- – Complex to deploy and operate
- – Overkill for small volumes and simple integrations
- – Requires a well-thought-out topic and partition design
- – Above-average maintenance cost
