The easiest way to use Akka Streams with Kafka is by using the Alpakka Kafka Connector. The documentation page describes in detail how to use Akka Streams both as a Kafka producer and as a consumer.
Even though Kafka is agnostic regarding to the message format, the preferred message format is Avro and the preferred solution is using it in combination with a Schema Registry.
Avro has a schema so unlike when using JSON, it is known which fields the message contains. Normally the schema is included with the data itself with Avro. Including the schema with every message can have a significant overhead, but when the Schema Registry is used, the schema is registered in the Schema Registry and the message contains a reference to the schema.
You can configure backwards compatibility for your schema and messages with a new schema that isn’t backwards compatible will then be rejected. So you have the guarantee that consumers continue to be able to read messages even though they expect a lower schema version.
Since I couldn’t find a complete example on how to use Akka Streams with Kafka and a Schema Registry for messages in Avro format, I created a sample project which you can find at https://github.com/jaspervz/akkastreams-kafka-example.
The README.md of the project contains information on how to run the project.
Leave a Reply