using mqtt protocol with kafka as a message broker

Question

How can we use mqtt protocol with kafka as a message broker?

The clients(android/ios/desktop java apps etc) will be producing and consuming messages using mqtt phao client side libraries which are available in different languages using kafka as a message broker.

Any advice?

Solution 1

You can use a Kafka source connector which will stream data from an MQTT broker like Mosquitto into a Kafka cluster. See https://github.com/evokly/kafka-connect-mqtt

The simplest way to run the connector is in standalone mode, where a single instance will be running on the Kafka cluster on a single node. You can also run it in distributed mode (albeit with much more configuration) and this will distribute the connector across the cluster for greater throughput. In the distributed mode you can devise a topology that allows horizontal scaling, parallel throughput and high availability. Implementing additional guarantees requires additional load balancers, multiple MQTT brokers and last will and testament scenarios to deal with connectors crashing, but this is probably out of scope for this question.

Using the connector approach has the advantage of the Kafka cluster making sure your connector is alive and re-starting it if necessary. Distributed mode offers even more advantages.

Solution 2

This is not a good idea. A MQTT client usually is very light-weight with limited resources. Those devices or IoT has small memory/CPU power. A Kafka client generally is heavy-weight. For example, Kafka client has to keep track of the offset. It also requires interaction to Zookeepers. In short, Kafka is not suitable as MQTT broker. You are better off choosing a popular MQTT broker such as Mosquito.

Solution 3

AFAIK, there's no "official" MQTT connector for Kafka. But you can use MQTT-Kafka bridge. For inspiration look at https://github.com/km4rcus/mqttKafkaBridge (Please note there's a bug in this implementation: kafka topics can't contain "/" so you will probably want to replace them with "_" in messageArrived in Bridge.java file)

Note that this code is just a very simple solution, not scalable. It's a good idea probably to write your custom implementation to suit your expectations better. But you should keep it as simple as possible - it's a single point of failure. As long as you get your data into Kafka, you get some guarantees but you get no guarantees from MQTT broker. When the bridge crashes, you are simply loosing your data...