Connector MQTT API Reference

This guide shows how to enable MQTT (3.1.1) communication support for a West Connectivity Connector. The example code provided shows how a simulated device connects and activates with the West Connectivity MQTT endpoint.

Info

Your hardware & Library MUST be compatible with Connectivity TLS Specs., make sure you read it first.

Overview

West Connectivity MQTT offering supports bi-directional device communication. Devices can provision with a Connector, publish data to its resources, and receive updates about changes made to its resources.

West Connectivity MQTT communication support translates MQTT communication into native West Connectivity commands, allowing MQTT clients to connect and communicate with the Connector. While the translation layer provides most features of MQTT 3.1.1 there are some special considerations to make for West Connectivity integration.

Messaging Limitations

There is no specific bandwidth or message size limitation, however each individual resource size MUST respect the maximum size of 1048576 bytes

Connector supports standard MQTT messaging with a few exceptions. Those exceptions are detailed below. MQTT messages or message features not listed below are fully compliant to the MQTT 3.1.1 specification.

CONNECT message "Will Flag", "Will QoS" and "Will Retain" bits

Not supported. The primary purpose of communication from devices to Connector is the live reporting of (typically) sensory data. A last Will type of pre-stored message does not make much sense for such applications.

CONNECT message "Clean Session" flag

Ignored, acting as if set to 0. By definition, Connector will sync any resource value that had been updated while the device was offline, back to the device when it connects. This behavior essentially mimics that of when Clean Session is set to 0. Also, since subscriptions are also hardcoded by definition, the device will not have to resubscribe when it reconnects which is also inline with "Clean Session 0" behavior.

CONNACK message "Session Present" flag

Always set to 1. Because Connector devices are auto-subscribed and Connector will hold resource state for devices even before their initial connection and because "Clean Session" is always considered 0, the "Session Present" flag will always be set to 1.

PUBLISH message "Retain" flag

Ignored, acting as if set to 0. Device initiated PUBLISH messages are processed by Connector real time by applying the published value to the correspond device resource state. This will initiate an internal message to application Lua scripts running at the time for processing. Should new application scripts be launched later, the message will not be sent to them regardless of whether the device specified 0 or 1 for the "Retain" flag, essentially acting as if the device had set it to 0.

PUBLISH message "QoS" flags

Fully supports QoS-0 and QoS-1 but ignores QoS-2. If it is set to indicate QoS-2, Connector will act as if QoS-0 was specified.

PUBLISH message "DUP" flag

Ignored. Since it only has meaning for QoS-2 which Connector currently does not support, this flag is ignored by Connector. However, Connector will disconnect a device if it misuses the flag by setting it to 1 while specifying QoS-0.

PUBACK, PUBREC, PUBREL and PUBCOMP messages

Not supported. These messages are not supported because QoS-1 or QoS-2 are not supported. QoS-1 is supported device-to-Connector but not from Connector-to-device, hence PUBACK is supported Connector-to-device. If a device sends a one of these messages to Connector, it will disconnect the device immediately.

SUBSCRIBE message

Limited support. By definition, Connector will sync resource state changes back to devices. This does not require explicit subscription from the device. Connector will send the device a PUBLISH message with QoS-0 whenever there is a state change to one of its resources.

If a device sends a SUBSCRIBE message with a topic filter that it is entitled to, Connector will ack it by returning a SUBACK message with QoS-0, even if the device specifies a higher QoS level. The following are valid topics to subscribe to: "#", "$content", "$provision/{device id}", "$resource/#", "$resource/+" and "$resource/{alias}".

If a device sends a SUBSCRIBE message with a topic filter that does not exist or the device is not entitled to receive messages for, Connector will return a SUBACK with the return code 'failure'.

UNSUBSCRIBE message

Ignored. By definition, Connector will sync resource state changes back to devices using PUBLISH messages. Devices may not opt out of receiving these messages by unsubscribing. Hence, the UNSUBSCRIBE message is accepted, responded to by UNSUBACK but functionally ignored by Connector. That is, after an unsubscription message exchange, the device will continue to receive state updates from Connector.

Topics Support and Isolation

The Connector MQTT implementation is not a full MQTT broker. Rather, it is a protocol level implementation with some auto-subscribe functionality and topics support.

Device connectivity and data exchange from and to Connector, is implemented using standard MQTT messaging and using a predefined set of topics as follows: "$content", "$provision", "$provision/{device id}", "$resource", "$resource/{alias}" and "$resource.batch". Devices are not allowed to publish to other topics and doing so will result in the immediate disconnection of the device.

Anonymous access to the $provision topic is provided to facilitate activating an MQTT device. The processing of activation supplies the device with the credentials (i.e., MQTT password) it needs to authenticate future sessions with Connector.

Connector will use device authentication (token, certificate etc.) to identify a device and its state. It's recommended to leave MQTT ClientId empty because a mismatch with Connector device authentication will make the device unable to connect.

Topics are specially handled in Connector. Topics are not public. Access control isolates an activated device to publishing/subscribing only to that device’s topics even though multiple devices will have subscriptions to identically named topics. A device is not allowed to subscribe to another device's topics. A topic called "$resource/temperature", for example, will represent an isolated data stream for the authenticated device.

Anonymous clients, by contrast, can only publish to the $provision endpoint and can only subscribe to that endpoint’s activation reply topic, "$provision/{device id}", which is unique to each activation request. When connecting a device to a Connector for the first time, it must anonymously subscribe to the $provision topic in order to receive its MQTT password via the subscription reply. The password the device receives in the reply represents the "activation" of the client and is what enables successive future connections.

The "$content" topic is shared across devices belonging to the same Connector. That is, if two different devices of the same Connector publish to the "$content" topic, both will receive the same list of available contents to download. If, however, two devices belonging to two separate Connectors, publish to "$content", they will each receive a different list, unique to each Connector - despite the topic name being same.

MQTT APIs

Provision authentication credentials for a device

To provision a device identity to Connector, you have to send a PUBLISH message with the topic:

$provision/<device_id>

Device ID must conform to the identity format specification defined in the Settings tab of the Product UI.

Note: 1. This is only for token and password authentication types. 2. Only password authentication needs to provide the credentials (at least 20 characters) as the request payload. For token authentication, leave the payload empty since the token will be generated by Connector.

Response message with the secret token (When a Product is configured for token authentication):

eY4f3tyrE4Jqe3HsLGnPYf7cACKZlb0uvKVatFxX

Received Data

To retrieve all meta information (name, length, mime, url) for all contents, you have to send a PUBLISH message with the topic:

$content

Leave the request payload empty.

Response message with a list of JSON objects, each one containing the name, length, mime and url keys. For example::

[
  {
    "name": "logo.gif",
    "length": 16214,
    "mime": "image/gif",
    "url": "https://s3.something.something.logo.gif"
  }
]

The values for name, length, mime and url keys are:

name: The name of this content. Example: "name": "logo.gif"
length: The content length for this content in bytes. Example: "length": 16214
mime: The mime type for this content. Example: "mime": "image/gif"
url: The download url for this content. Example: "url": "https://s3.something.something.logo.gif"

Report data

to multiple resources

To report data, you have to send a PUBLISH message with the topic:

$resource/

The request payload should be a JSON object, containing the resource alias as key and value of the data point. For example:

{"state": "on", "temperature": 25}

Connector will assign a timestamp in microseconds.

to specific resource

To report data to a specific resource, you have to send a PUBLISH message with the topic:

$resource/<alias>

The request payload must follow the format specification defined in the Resources Settings. Connector will assign a timestamp in microseconds.

to historical timestamps

To report historical data to multiple resources, you have to send a PUBLISH message with the topic:

$resource.batch

The request payload must be a list of JSON objects, each one containing the timestamp and values keys. For example:

[{"timestamp": 1531111131679000, "values": {"state": "on", "temperature": 25}}, {"timestamp": 1531111131689665, "values": {"temperature": 30}}]

Accepted values for timestamp and values keys are:

timestamp: Timestamp in microseconds. Example: "timestamp": 1531111131679000
values: A JSON object containing the resource alias as key and value of the data point. Example: {"state": "on", "temperature": 25}

Getting Started

MQTT client libraries are readily available. Connector requires that the library supports TLS and requires that the TLS support is modern enough that it includes Server Name Indication (SNI). Please contact the West Connectivity development team if a preferred MQTT client library fails either criteria.

This tutorial will use Python and Eclipse Paho™ MQTT Python Client to connect to Connector.

Hardware Setup

A development computer or laptop.

Software Setup

To complete this guide, download and install the following on the development machine:

Python 3.5 from the Python website.
Install the Eclipse Paho™ MQTT Python Client via the following command:

$ pip install paho-mqtt

You must have a West Connectivity account and have created a Connector within it. For more information on how to create and account and a Connector in the account, visit the Create a Connector article for more information.

Setup A Sandbox

This guide will require files to be created on the development machine. Throughout this guide it will be assumed that all commands and files will be run from the following directory:

$ mkdir ~/mqtt-client
$ cd ~/mqtt-client

Configuration

Create a Resource

You can create a "temp" resource on the web UI:

Activate Your Device

The default Connector settings are such that devices are allowed to register their own identities. In some provisioning models it is required or advantageous to have a list of pre-authorized device identities registered with Connector. This is also known as whitelisting. At this point, make sure that the default setting is applied and saved. Navigate to the SETTINGS tab in your Connector web UI and verify that "Allow devices to register their own identity" is selected.

Save the following code into a file called ~/mqtt-client/activate.py:

from paho.mqtt import client as mqtt
import ssl
import logging

# logging.basicConfig(level=logging.DEBUG)

pid = input("Connector ID? ")
did = input("Device ID? ")
host = pid + ".m2.connect.westpharma.com"
open("IoT_Connector_id.txt", "w").write(pid)

def on_connect(client, userdata, flags, rc):
    provision_str = "$provision/" + did
    client.publish(provision_str, None, qos=0)

def on_message(client, userdata, msg):
    print("Activation succeeded!")
    token = msg.payload.decode()
    print("Token: ", token)
    open('token.txt', "w").write(token)
    client.disconnect()

def on_disconnect(client, userdata, rc):
    if rc != 0:
        print("Disconnected with error", rc)
        exit()

client = mqtt.Client(client_id="")
logger = logging.getLogger(__name__)
client.enable_logger(logger)

# see https://github.com/eclipse/paho.mqtt.python/blob/1.1/README.rst#tls_set
client.tls_set()

client.on_connect = on_connect
client.on_disconnect = on_disconnect
client.on_message = on_message

client.connect(host, 443)
client.loop_forever()

Now execute the above script by running the following:

$ python3 activate.py

Provide the Connector ID and a device identity of your choice when the activate.py script prompts you for them.

$ python activate.py
Connector ID? o33ara2ekbo800000
Device ID? 12345

Activation succeeded!
Token: b7b34f55e948b94841820ea50868a2490632d78f

A successful result, as shown above, activates the device, prints the credential, and saves it to a file called ~/mqtt-client/token.txt for subsequent sessions. Notice that in the Connector UI the device has been "activated". Save the device ID for later usage (e.g., 12345

NOTE: The client connected anonymously and then provisioned itself using the provided device identity (e.g., 12345) as the MQTT client ID.

Publish Data

Next, use the returned credentials to reconnect and publish data to the new device in the Connector. The data will be published to the temp resource defined a few steps above.

Save the following code into the file ~/mqtt-client/publish.py:

from paho.mqtt import client as mqtt
import ssl
import logging

logging.basicConfig(level=logging.DEBUG)

pid = open("IoT_Connector_id.txt").read()
print("Using Connector ID: ", pid)
host = pid + ".m2.connect.westpharma.com"

def on_connect(client, userdata, flags, rc):
    res = "temp"
    resource = "$resource/" + res
    print("Publishing to Resource: ", res)
    client.publish(resource, input("Value? "), qos=0)
    print("Press \"Ctrl+C\" to exit...")

def on_message(client, userdata, msg):
    print("Value set, previous was: ", msg.payload.decode())

def on_disconnect(client, userdata, rc):
    if rc != 0:
        print("Disconnected with error", rc)
        exit()

client = mqtt.Client(client_id="")
logger = logging.getLogger(__name__)
client.enable_logger(logger)

# see https://github.com/eclipse/paho.mqtt.python/blob/1.1/README.rst#tls_set
client.tls_set()

# read auth token from file
token = open("./token.txt").read()
print("Using Token: ", token)
client.username_pw_set("", token)

client.on_connect = on_connect
client.on_disconnect = on_disconnect
client.on_message = on_message

client.connect(host, 443)
client.loop_forever()

Next, execute the script with the following command:

$ python3 publish.py

The script prompts the user for the data to send. The device’s resources are represented as topics "$resource/" (e.g., $resource/temp”).

Below is some example output of the script prompting the user for data and then publishing:

$ python3 publish.py
Using Connector ID: o33ara2ekbo800000
Using Token: XzE3KU2Zhs9ZDl0cSz0Lf8Xp5Ez7rR0cUa1rO4qE
Publishing to Resource: temp
Value? 23
Press "Ctrl+c" to exit...

The device’s temp resource value will reflect the value published by the script.

You can check the value by either using the Connector web UI

Receiving data

Copy the following code to a file called ~/mqtt-client/subscribe.py:

from paho.mqtt import client as mqtt
import ssl
import logging

# logging.basicConfig(level=logging.DEBUG)

pid = open("IoT_Connector_id.txt", "r").read()
print("Using Connector ID: ", pid)
host = pid + ".m2.connect.westpharma.com"

def on_message(client, userdata, msg):
    print("Received: ", msg.payload.decode())

def on_connect(client, userdata, flags, rc):
    print("Waiting for messages")

def on_disconnect(client, userdata, rc):
    if rc != 0:
        print("Disconnected with error", rc)
        exit()

client = mqtt.Client(client_id="")
logger = logging.getLogger(__name__)
client.enable_logger(logger)

# see https://github.com/eclipse/paho.mqtt.python/blob/1.1/README.rst#tls_set
client.tls_set()

token = open("token.txt", "r").read()
print("Using Token: ", token)
client.username_pw_set("", token)

client.on_connect = on_connect
client.on_disconnect = on_disconnect
client.on_message = on_message

client.connect(host, 443)
client.loop_forever()

Executing the script (command provided below) will subscribe to all changes to the Connector device with the token recieved with the activate.py script. The subscribe.py script will print all messages sent to the device.

$ python3 subscribe.py

Now, using Connector web UI , change the state of the temp resource:

You can directly enter resource value on browser by enabling "Enable write from scripting and IoT Apps":

After entering the value, above, in a separate terminal, the subscribe.py script should print something like the following:

$ python3 subscribe.py
Using Product ID:  n110e3xbifmfk0000
Using Token:  eY4f3tyrE4Jqe3HsLGnPYf7cACKZlb0uvKVatFxX
Waiting for messages
Received:  32

Authentication Types

Field\Auth Type	Token	Password	Certificate
ClientID	Empty/DeviceID	Empty/DeviceID	Empty
Username	Empty/DeviceID	DeviceID	Empty
Password	Token Credential	Password Credential	Empty

Password Authentication

You need to change the Provisioning Authentication to Password first.

You can use the previous examples; you need to do minor changes.

Activation

Send the password (at least 20 characters) as the request body when you publish to $provision/DeviceID client.publish("$provision/device001", "12345678901234567890", qos=0)

pid = input("Connector ID? ")
# did = input("Device ID? ")
host = pid + ".m2.connect.westpharma.com"
open("IoT_Connector_id.txt", "w").write(pid)

def on_connect(client, userdata, flags, rc):
    # provision_str = "$provision/" + did
    client.publish("$provision/device001", "12345678901234567890", qos=0)
    client.disconnect()

def on_message(client, userdata, msg):
    print("Activation succeeded!")
    token = msg.payload.decode()
    print("Token: ", token)
    open('token.txt', "w").write(token)
    client.disconnect()

No token will be returned. Use the DeviceID as username and your password as the password for any further communication. client.username_pw_set("device001", "12345678901234567890")
The client ID should be set as empty

Read/publish

Use the DeviceID as username and your password as the password for any further communication.
The client ID does not need to be set

# read auth token from file
# token = open("./token.txt").read()
# print("Using Token: ", token)
client.username_pw_set("device001", "12345678901234567890")

Client Certificate

Change the provisioning/auth_type to TLS Client Certificate.

Activation

When setting up for TLS Client Certificate authentication, the client cert will be sent to the server during the activate request.
Any further communication needs to use the key/cert you used for the activation
The client ID does not need to be set.
Copy the code below to a file called certificate_activate.py
Create certs/ directory
Put the cert and the corresponding private key to a file called cert.pem and key.pem in the certs directory
Certificate common name will be used as device name by default
ex: self signed for devicea: openssl req -subj '/CN=devicea' -x509 -newkey rsa:4096 -nodes -keyout key.pem -out cert.pem -days 1500

from paho.mqtt import client as mqtt
import os
import ssl
import logging

# logging.basicConfig(level=logging.DEBUG)

pid = input("Connector ID? ")
host = pid + ".m2.connect.westpharma.com"
open("IoT_Connector_id.txt", "w").write(pid)

def on_connect(client, userdata, flags, rc):
    print("Activation succeeded!")
    client.disconnect()

def on_disconnect(client, userdata, rc):
    if rc != 0:
        print("Disconnected with error", rc)
        exit()

client = mqtt.Client(client_id="")
logger = logging.getLogger(__name__)
client.enable_logger(logger)

certfile = "./certs/" + "cert.pem"
keyfile  = "./certs/" + "key.pem"
# print("Current dir: " + os.getcwd() + " Certificate: " + certfile + ", Keyfile: " + keyfile)
client.tls_set(
    certfile=certfile,
    keyfile=keyfile,
    cert_reqs=ssl.CERT_REQUIRED
)

client.on_connect = on_connect
client.on_disconnect = on_disconnect

client.connect(host, 443)
client.loop_forever()

Now execute the above script by running the following:

$ python3 certificate_activate.py

Provide the Product ID of your choice when the certificate_activate.py script prompts you for them. The device ID does not need to be set; it will be set to the common name in the certificate automatically

$ python3 certificate_activate.py
Connector ID? x2lmj5npsktbuik9

Activation succeeded!

After the success message nothing need to be done you only need to use the same cert/key for any further communication.

Read/publish

Use the key/cert for any further communication
Copy the following snippet to a file called certificate_publish.py

from paho.mqtt import client as mqtt
import os
import ssl
import logging

# logging.basicConfig(level=logging.DEBUG)

pid = open("IoT_Connector_id.txt").read()
print("Using Connector ID: ", pid)
host = pid + ".m2.connect.westpharma.com"

def on_connect(client, userdata, flags, rc):
    resource = "$resource/" + input("Resource ID? ")
    client.publish(resource, input("Value? "), qos=0)
    client.disconnect()

def on_message(client, userdata, msg):
    print("Value set, previous was: ", msg.payload.decode())

def on_disconnect(client, userdata, rc):
    if rc != 0:
        print("DisConnected with error", rc)
        exit()

client = mqtt.Client(client_id="")
logger = logging.getLogger(__name__)
client.enable_logger(logger)

certfile = "./certs/" + "cert.pem"
keyfile  = "./certs/" + "key.pem"
# print("Current dir: " + os.getcwd() + " Certificate: " + certfile + ", Keyfile: " + keyfile)
client.tls_set(
    certfile=certfile,
    keyfile=keyfile,
    cert_reqs=ssl.CERT_REQUIRED
)

client.on_connect = on_connect
client.on_disconnect = on_disconnect

client.connect(host, 443)
client.loop_forever()

Next, execute the script with the following command:

$ python3 certificate_publish.py

The script prompts the user for the data to send. The device’s resources are represented as topics "$resource/" (e.g., $resource/temp”).

Below is some example output of the script prompting the user for data and then publishing:

$ python3 certificate_publish.py
Using Connector ID: d23kegyeoxb280000
Publishing to Resource: temp
Value? 23
Done. Disconnecting...

The device’s temp resource value will reflect the value published by the script.

Summary

This guide showed how to configure a Connector to use the MQTT internet protocol to provision and activate a simulated device with a Python script, as well as publishing to device resources and subscribing to changes to device resources.