README
About Karafka
Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework that:
- Has a built-in Web UI providing a convenient way to monitor and manage Karafka-based applications.
- Supports parallel processing in multiple threads (also for a single topic partition work) and processes.
- Automatically integrates with Ruby on Rails
- Has ActiveJob backend support (including ordered jobs)
- Has a seamless Dead Letter Queue functionality built-in
- Supports in-development code reloading
- Is powered by librdkafka (the Apache Kafka C/C++ client library)
- Has an out-of the box AppSignal and StatsD/DataDog monitoring with dashboard templates.
# Define what topics you want to consume with which consumers in karafka.rb
Karafka::App.routes.draw do
topic 'system_events' do
consumer EventsConsumer
end
end
# And create your consumers, within which your messages will be processed
class EventsConsumer < ApplicationConsumer
# Example that utilizes ActiveRecord#insert_all and Karafka batch processing