MQTT broker in Azure cloud

Question

I have a backend device with MQTT client connected to opensource MQTT broker (Mosquitto). On the other hand I have many frontend devices (PC, Tablet, Mobile) with GUI application also connected to the broker. So here Mosquitto works as a communication point between backend device and frontend devices and just forwards messages between them. Sometimes amount of data transferred can be quite high (e.g. 1 MB / min).

One backend device + many frontend devices is one installation. I need to prepare infrastructure for thousands of such installations working simultaneously. So my service needs to be very scallable. My company uses Azure cloud solutions, so I've started learning of this solution and I must admit that I am a little bit confused. I've read that I need to use IoT Hub, but it would need MQTT Gateway to be able to speak with MQTT devices. On the other hand, if I understand it well, the Gateway needs to be running on some VM, so here I lose scalability of my solution. Am I right? Now if I will need to support 100k or 500k devices then I will need another VM? One more thing is that I need to integrate all of this with some webservice (for management of backend and frontend devices), so I need some connection between webservice and the MQTT broker...

Before I started to play with Azure, I imagined, that I will simply start an MQTT broker service and magically it will be highly scallable and will be able to provide service for thousands of devices.

Can anybody explain me how to bite that?

Solution 1

Today there isn't an official support for MQTT protocol in Azure but only the public preview of IoT Hub that supports AMQP and HTTP. To connect MQTT devices to the IoT Hub, Microsoft provides a "framework" named IoT Protocol Gateway (https://github.com/Azure/azure-iot-protocol-gateway) that executes a protocol translation between MQTT and AMQP. The IoT Protocol Gateway can be installed on premise or in the cloud as an Azure Worker Role. In the second scenario you have the scalability offered by Azure and related to worker role instances. This solution is absolutely new due to the short life of IoT Hub (still in public preview) and the IoT Protocol Gateway itself.

Your first solution is based on using a third-party MQTT broker (like mosquitto) that you should install in a VM. AFAIK mosquitto doesn't support clustering like HiveMQ broker (see another reply here : Cluster forming with Mosquitto broker).

Last thing about the connection between your web service and the MQTT broker. In this case the web service should translate calls to him (from front end) to published message on the MQTT broker using an MQTT client that you need to include inside the web service itself.

Even if using AWS, the following link could be useful too : https://groups.google.com/forum/#!topic/mqtt/19jqofoPLro

Paolo.

Solution 2

Azure IoT Hub now talks MQTT natively. A protocol gateway is no longer required. https://azure.microsoft.com/en-us/documentation/articles/iot-hub-mqtt-support/

This is going to help you a lot if you've just spent the last hour trying to form the MQTT username and password: https://github.com/Azure/azure-content/blob/master/articles/iot-hub/iot-hub-devguide.md#example

Example:

Username (DeviceId is case sensitive): iothubname.azure-devices.net/DeviceId

Password (Generate SAS with Device Explorer): SharedAccessSignature sr=iothubname.azure-devices.net%2fdevices%2fDeviceId&sig=kPszxZZZZZZZZZZZZZZZZZAhLT%2bV7o%3d&se=1487709501

Tested with Paho and MQTT.fx on Windows. I could not make it authenticate with mosquitto, and i've put in reasonable effort, even tried using stunnel just in case mosquitto's TLS support wasn't cutting it. Mosquitto probably doesn't handle the long password correctly or something along those lines. It throws an authentication error. Escaping % and & didn't help.

If someone gets Mosquitto to work with Azure IoT Hub, please open my eyes.

...and someone did (Thank you Timothy in the comments)

Mosquitto_pub works, I verified by monitoring with Device Explorer Twin. Example:

mosquitto_pub -h IOTHubACMxxx.azure-devices.net
    -p 8883
    --cafile "C:\Users\jlaird\Documents\dev\azureca.crt"
    -t devices/eACM1/messages/events/
    -m "john says hello to azure from mosquitto"
    -i eACM1
    -u IOTHubACMxxx.azure-devices.net/eACM1/?api-version=2018-06-30
    -P "SharedAccessSignature sr=IOTHubACMxxx.azure-devices.net&sig=obfuscate&se=1593013589&skn=device"

Comments

[kappa]

Thank you, that was really helpful. It looks like Azure Worker Role is a solution for me. Additionally, if I understand it correctly, IoT Protocol Gateway will allow me to access some DB shared with my web service.

[Wagner Bertolini Junior]

Just adding: IoT Hub REQUIRES TLS. Not every device/firmware support it.

[Timothy John Laird]

Mosquitto_pub works, I verified by monitoring with Device Explorer Twin. Example: mosquitto_pub -h IOTHubACMxxx.azure-devices.net -p 8883 --cafile "C:\Users\jlaird\Documents\dev\azureca.crt" -t devices/eACM1/messages/events/ -m "john says hello to azure from mosquitto" -i eACM1 -u IOTHubACMxxx.azure-devices.net/eACM1/?api-version=2018-06-30 -P "SharedAccessSignature sr=IOTHubACMxxx.azure-devices.net&sig=obfuscate&se=159301358‌​9&skn=device"