Increasingly affordable micro controllers like
Raspberry Pi are enabling cheap devices
that measure sensor data and send it over the internet.
The goal of this document is to introduce
the lightweight protocol
MQTT and its capabilities to send data between devices and other systems,
and to demonstrate them by implementing two clients with
MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol.
MQTT was developed by Andy Stanford-Clark (IBM) and Arlen Nipper (Eurotech; now Cirrus Link)
in 1999 for the monitoring of an oil pipeline through the desert.
The goals were to have a protocol, designed as bandwidth-efficient and using little battery power,
because the devices were connected via satellite link and this was extremely expensive at that time.
The protocol uses a publish/subscribe architecture in contrast to HTTP with its request/response paradigm.
Publish/Subscribe is event-driven and enables messages to be pushed to clients.
The central communication point is the
MQTT broker, it is in charge of dispatching
all messages between the senders and the rightful receivers. Key elements of the communication are:
- Each client that publishes a message to the broker, includes a topic into the message.
- The topic is the routing information for the broker.
- Each client that wants to receive messages subscribes to a certain topic and the broker delivers all messages with the matching topic to the client.
The difference to HTTP is that a client doesn’t have to pull the information it needs,
but the broker pushes the information to the client, in the case there is something new.
MQTT client has a permanently open TCP connection to the broker.
If this connection is interrupted by any circumstances,
MQTT broker can buffer all messages and send them to the client when it is back online.
1.1 Subscription hierarchy
As mentioned before the central concept in
MQTT, to dispatch messages, is the topics.
A topic is a simple string that can have more hierarchy levels, which are separated by a slash.
A sample topic for sending temperature data of the living room could be house/living-room/temperature. On one hand the client can subscribe to the exact topic or on the other hand it can use a wildcard. The subscription to house/+/temperature would result in all message send to the previously mention topic house/living-room/temperature as well as any topic with an arbitrary value in the place of living room, for example house/kitchen/temperature.
The plus sign is a single level wild card and only allows arbitrary values for one hierarchy. If you need to subscribe to more than one level, for example to the entire subtree, there is also a multilevel wildcard (#). It allows to subscribe to all underlying hierarchy levels. For example house/# is subscribing to all topics beginning with house.
Messages in MQTT are published on topics. There is no need to configure a topic, publishing on it is enough. Topics are treated as a hierarchy, using a slash (/) as a separator. This allows sensible arrangement of common themes to be created, much in the same way as a filesystem. For example, multiple computers may all publish their hard drive temperature information on the following topic, with their own computer and hard drive name being replaced as appropriate:
Clients can receive messages by creating subscriptions. A subscription may be to an explicit topic, in which case only messages to that topic will be received, or it may include wildcards. Two wildcards are available, + or #. The + can be used as a wildcard for a single level of hierarchy. It could be used with the topic above to get information on all computers and hard drives as follows:
1.3 Quality of service
MQTT defines three levels of Quality of Service (QoS). The QoS defines how hard the broker/client will try to ensure that a message is received. Messages may be sent at any QoS level, and clients may attempt to subscribe to topics at any QoS level. This means that the client chooses the maximum QoS it will receive. For example, if a message is published at QoS 2 and a client is subscribed with QoS 0, the message will be delivered to that client with QoS 0. If a second client is also subscribed to the same topic, but with QoS 2, then it will receive the same message but with QoS 2. For a second example, if a client is subscribed with QoS 2 and a message is published on QoS 0, the client will receive it on QoS 0.
Higher levels of QoS are more reliable, but involve higher latency and have higher bandwidth requirements.
- 0: The broker/client will deliver the message once, with no confirmation.
- 1: The broker/client will deliver the message at least once, with confirmation required.
- 2: The broker/client will deliver the message exactly once by using a four step handshake.
2 Publish and Subscribe
MQTT client is connected to a broker, it can publish messages.
MQTT has a topic-based
filtering of the messages on the broker, so each
message must contain a topic, which will be used by the broker to forward the message
to interested clients. Each message typically has a payload which contains the actual
data to transmit in byte format.
MQTT is data-agnostic and it totally depends on the
use case how the payload is structured. It’s completely up to the sender if it wants
to send binary data, textual data or even full-fledged XML or JSON.
MQTT publish message also has some more attributes, which we’re going discuss in detail:
A simple string, which is hierarchically structured with forward slashes as delimiters.
A Quality of Service Level (QoS) for this message. The level (0,1 or 2) determines the guarantee of a message reaching the other end (client or broker).
This flag determines if the message will be saved by the broker for the specified topic as last known good value. New clients that subscribe to that topic will receive the last retained message on that topic instantly after subscribing. More on retained messages and best practices in one of the next posts..
This is the actual content of the message. MQTT is totally data-agnostic, it’s possible to send images, texts in any encoding, encrypted data and virtually every data in binary.
The packet identifier is a unique identifier between client and broker to identify a message in a message flow. This is only relevant for QoS greater than zero. Setting this MQTT internal identifier is the responsibility of the client library and/or the broker.
The duplicate flag indicates, that this message is a duplicate and is resent because the other end didn’t acknowledge the original message. This is only relevant for QoS greater than 0.
Publishing messages doesn’t make sense if no one ever receives the message, or, in other words,
if there are no clients subscribing to any topic.
A client needs to send a SUBSCRIBE message to the
MQTT broker in order to receive relevant messages.
A subscribe message is pretty simple, it just contains a unique packet identifier and a list of subscriptions.
3 Retained messages
When publishing MQTT messages, a publishing client has no guarantee that a message is actually received by a subscribing client. It can only make sure its message gets delivered safely to the broker. The same is true for a subscribing client. If a client is connecting and subscribing to topics it is interested in, there is no guarantee when the subscriber will get the first message, because this totally depends on a publisher on that topic.
It can take a few seconds, minutes or hours until the publisher sends a new message on that topic. Until then, the subscribing client is totally in the dark, about the current status. This is were retained messages come into play.
3.1 What are retained messages ?
When a subscriber connects to a broker it will only receive messages published after subscription. So it will not get any information till the publisher sends a message.
A retained message is a normal
MQTT message with the retained flag set to true.
The broker will store the last retained message and the corresponding QoS for
that topic Each client that subscribes to a topic pattern, which matches the
topic of the retained message, will receive the message immediately after subscribing.
For each topic only one retained message will be stored by the broker.
In other words a retained message on a topic is the last known good value, because it doesn’t have to be the last value, but it certainly is the last message with the retained flag set to true.
3.2 Send a retained message
Sending a retained message from the perspective of a developer is quite simple and straight-forward.
You just need to set the retained flag of a
MQTT publish message to true
3.3 Delete a retained message
There is also a very simple way for deleting a retained message on a topic: Just send a retained message with a zero byte payload on that topic where the previous retained message should be deleted. The broker deletes the retained message and all new subscribers won’t get a retained message for that topic anymore. Often deleting is not necessary, because each new retained message will overwrite the last one.
4 Use case
In order to make the subsequent code more understandable,
we will use the transferring of sensor data from a temperature
and brightness sensor to a control center over the internet as an example.
The sensors will be connected to a Raspberry Pi, which acts as gateway to the
which resides in the cloud.
On the other side is a second device, the control center,
that also has an
MQTT client and receives the data.
Additionally we will implement a notification, which alerts the control center if the sensor is disconnected.
Now it's time to write a simple sample. Few pre-requisites should be available:
- A cloud
MQTTbroker like Cloud MQTT
- A Raspberry Pi acting as
- A laptop acting as
We will link a cloud MQTT and 2 computers with the standard mosquitto command line tools, mosquitto_sub and mosquitto_pub.
|-d||Enable debug message|
|-m||Send a single message from the command line|
|-l||Send messages read from stdin, splitting separate lines into separate messages|
4.1.1 Setup MQTT broker
Configure a test account on CloudMQTT
4.1.2 Setup Laptop client (publisher)
Intall the mosquitto clients
$ sudo apt-get install mosquitto-clients
Now, suscribe to the MQTT topic. The client will keep waiting for messages.
$ mosquitto_sub -d -u ttvqonoe -P hT4DdZNIy_JO -h m21.cloudmqtt.com -p 15911 -t "test"
4.1.3 Setup Raspberry PI client
Intall the mosquitto clients
$ sudo apt-get install mosquitto-clients
Now, send a message.
$ mosquitto_pub -d -u ttvqonoe -P hT4DdZNIy_JO -h m21.cloudmqtt.com -p 15911 -t "test" -m "hello"
You should see the "hello" message on Laptop client console.
Now, we will send the temperature with a fixed interval with a simple Shell using a pipe.
#!/bin/bash while true do vcgencmd measure_temp | mosquitto_pub -d -u ttvqonoe -P hT4DdZNIy_JO -h m21.cloudmqtt.com -p 15911 -t "test" -l sleep 1 done