ESP32 (2) – The development environment

After having introduced, in the previous post, the ESP32 chip and the official development board by Espressif, today I’m going to show you how to install and use the official development environment.

I’ll explain how to install the IDE under Windows because of it is the most complex procedure. The development environment its how-to-install guides are available also for Linux and MacOS.

The official development framework for the ESP32 chip is published by Expressif on Github, with an opensource license (Apache 2.0) and named Espressif IoT Development Framework (shortly idf).

To be able to use it under Windows, you need some GNU commands and toolchains (make, bash…). Espressif chose to offer a pre-configured version of MSYS2 that includes everything is required.

Download the ZIP archive from the Espressif website and unzip it in the root of your C drive; MSYS2 will be installed in the C:\msys32 subfolder:

esp32-07

Double-click on msys2_shell.cmd to run a bash shell:

esp32-08

We can now clone the idf Github repository to have a local copy of it. First we must choose a folder where it will be copied; to keep it simple let’s use the user’s home folder.

Move into that folder and run the clone command:

$ cd
$ git clone --recursive https://github.com/espressif/esp-idf.git

esp32-09

During the git clone command execution, you may get some error messages like: not a valid identifier; those errors are caused by a known bug and can be therefore ignored.

After the process is completed, the development framework is available in the esp32-idf subfolder:

esp32-10

To use if when compiling your projects, you have to export the path as an environment variable (IDF-PATH):

$ export IDF_PATH=~/esp-idf

Let’s connect the board!

You’re now ready to connect the development board to your computer.

If you already have the correct drivers, once connected it will appear as a new serial port:

esp32-12

On the contrary, the needed drivers are available on the Silicon Labs official website.

Hello World

As usual when learning a new language or framework, let’s start with a simple program that displays (using the serial-usb connection) the sentence Hello world.

The framework is distributed with some examples, among them you can find the 01_hello_world program. To compile it, move to its folder and run the make command:

$ cd
$ cd esp-idf/examples/01_hello_world/
$ make

A configuration menu will be displayed: we’ll talk more about it in a next post. Now you only need to tell the flasher which is the serial port your board is connected to.

Choose Serial flasher config, then /dev/ttyUSB0 (the default port):

esp32-14

Type the name of your serial port (as identified above), then confirm with OkSave and Exit:

esp32-16

When the compile process is complete, you can upload the program on the board with the following command:

make flash

esp32-17

Because of a bug, under Windows it may happen that the flash process is unsuccessful. The easiest solution is to keep the button boot pressed during all the flashing process.

If you now use a terminal emulator to connect to the serial port with the correct speed (115200 baud) you should see the program output:

esp32-18

(the program prints the sentence “Hello world!”, counts 10 seconds and then restart the chip…)

ESP32 (1) – Introduction

You probably already know the esp8266wifi chip, made by Espressif. It appeared on some Chinese webstores in the middle of 2014 and in the beginning it was used as a “bridge” to connect microcontrollers (Arduino…) to wifi networks thanks to its very low cost (about 5$ for a module).

esp32-01

Because of the original firmware was not well documented, it has some bugs and offered only “standard” features (via AT commands), the maker community developed some alternative firmwares  (the most famous of which is surely NodeMCU), to fully exploit the chip power and to build complete systems, without the need of external microcontrollers:

esp32-02

ESPToy, by Rayshobby

In September 2016, after a beta testing phase lasted some months, Espressif announced and made available the successor of esp8266, named ESP32.

The main features of the new chip are the following:

  • Tensilica LX6 dual core processor at 240 MHz
  • 520Kb of SRAM memory
  • Wifi 802.11 b/g/n connectivity with support for WEP, WPA/WPA2 PSK/Enterprise
  • Bluetooth connectivity (classic and LE)
  • 32 I/O pins with several builtin peripherals
  • hardware acceleration for security algorithms (AES, SHA2, RSA-4096)

With this post, I’m going to start a tutorial that will explore how to use this new chip to develop IoT systems and projects.

The ESP32 needs some additional components to work: a flash memory (to store firmware and data), a crystal (for the RTC), an antenna and some passive components. For this reason, you can find on sale ready-to-use modules:

esp32-03

On the left, the official module by Espressif (ESP-WROOM-32) while on the right the module made by Ai-Thinker (ESP-32S).

In addition, on some webstores you can start to find different development boards that include the ESP32 module, the power supply, an USB connector… everything you need to start building your project!

The esp32.net website keeps an updated list of the available boards; for my tutorials I chose the official Espressif development kit, named ESP32-DevKit3 or ESP32-CoreBoard. I brought mine from Olimex:

esp32-06 esp32-05

If you give a look to its schematics, you can find the ESP32 module, a voltage regulator (NCP1117) to lower the 5V coming from the USB bus to the 3.3V required by the chip, a CP2102 USB-to-serial adapter and two buttons. Moreover, all the most important PINs are available on the side connectors::

esp32-04

In the next post, you’ll learn how to install the development environment and how to write your first program!

MQTT – enc28j60 and Adafruit IO

Adafruit IO is the new cloud platform (at the moment still in beta) by Adafruit, designed to allow simple data connections using standard APIs and web-based dashboards.

This new platform exposes MQTT APIs, you can therefore apply what you learned in my previous posts to create a complete project that takes advantage of all its features.

The project

Goal of the project is to connect the Arduino to a dashboard hosted on Adafruit IO:

  • Arduino reads the room temperature (thanks to the DHT11 sensor already used in a previous tutorial) and sends its value every t seconds to Adafruit IO
  • Arduino is also able, using an external relay module, to turn on/off an heater
  • The dashboard on Adafruit IO displays the temperature values on a chart and has a toggle button to turn the heater connected to the Arduino on and off

mqtt-io1

Adafruit IO

Let’s see how to create the dashboard using Adafruit IO.

First, you need to subscribe to the open beta program using the button on the official website https://io.adafruit.com:

io-01

After having logged in with your Adafruit account (or having created a new account) you’ll be redirected to the default dashboard.

Click on My dashboards:

io-02

next click on Create dashboard:

io-03

give a name to the new dashboard, then click on Create dashboard:

io-04

you’ll see a completely empty dashboard. To add new elements (charts, buttons…) click on Create a new block:

io-05

choose line chart among the available elements and click on  Create:

io-06

you have to link the chart to a feed (= topic, in the MQTT language). Create a new feed named temperature:

io-07

once created, you can link it to the chart (choose), then click on next step:

io-08

you can change some chart settings (for example type °C as y-axis label), then complete the wizard with create block:

io-10

You’ll see a new block in your dashboard that represents the chart, empty for the moment. Using the drag’n’drop you can move the block in the page or resize it.

Now add the toggle button, again using Create a new block:

io-20

create a new feed named control:

io-21

and link it to the button:

io-22

give a label to the button and to the two possible values, then create the new block:

io-23

Your dashboard is now complete!

Feeds, topics and MQTT APIs

As explained above, each element of your dashboard can be linked to one or more feeds, that are data flows. For example you linked to the chart a feed named temperature, while the button is linked to another one named control.

If you use the MQTT APIs, each feed corresponds to a topic named as it follows:

<username>/f/<feedname>

where username is the name of your Adafruit account.

Hence, my two feeds correspond to the following two topics:

  • lucadentella/f/temperature
  • lucadentella/f/control

To connect to the Adafruit’s MQTT broker (address io.adafruit.com) you need to specify a username and a password.

The username is the name of your Adafruit account (lucadentella for me), while the password is your Adafruit IO key, that can be retrieved using a button in the dashboard:

io-31

io-32

You can test your dashboard using the mosquitto_sub command to subscribe to the topic linked to the toggle button. Every time you click on the button, the dashboard sends to that topic a message with the actual status of the button (ON or OFF):

mosquitto_sub.exe -h io.adafruit.com -u <username> -P <AIOkey> -t <username>/f/control

io-33

Arduino

The sketch for Arduino is available in my Github repository.

First, some constants are defined: the parameters for the Adafruit IO service (username, password, topics…) and the PINs the DHT11 sensor and the relay are connected to:

#define CLIENT_ID       "ArduinoMQTT"
#define USERNAME        "adafruit_username"
#define PASSWORD        "adafruit_io_key"
#define PUB_TOPIC       "username/f/temperature"
#define SUB_TOPIC       "username/f/control"
#define PUBLISH_DELAY   5000
#define DHTPIN          3
#define DHTTYPE         DHT11
#define RELAY_PIN       6

In the setup(),  the MQTT client is configured. Unlike the previous example, this time I also define the callback function that is called every time the client receives a new message from a topic that was subscribed:

// setup mqtt client
mqttClient.setClient(ethClient);
mqttClient.setServer("io.adafruit.com", 1883);
mqttClient.setCallback(mqttCallback);

Indeed, when the sketch connects to the broker it subscribes the topic linked to the button in the dashboard to be notified when a user clicks on it:

void mqttConnect() {
 
  while(!mqttClient.connected()) {
 
    if(mqttClient.connect(CLIENT_ID, USERNAME, PASSWORD)) {
 
      Serial.println(F("MQTT client connected"));
      mqttClient.subscribe(SUB_TOPIC);
      Serial.println(F("Topic subscribed"));
    } else {
      Serial.println(F("Unable to connect, retry in 5 seconds"));
      delay(5000);
    }
  }
}

The callback function checks if the message received is ON or OFF and changes the relay status accordingly:

void mqttCallback(char* topic, byte* payload, unsigned int length) {
 
  if(strncmp((const char*)payload, "ON", 2) == 0) {
    Serial.println("ON message received, turning relay ON");
    digitalWrite(RELAY_PIN, HIGH);
  } else {
    Serial.println("OFF message received, turning relay OFF");
    digitalWrite(RELAY_PIN, LOW);
  }
}

The following video shows the project in action:

MQTT – Security

In the IoT they often don’t give much importance to the security aspects of the communcation. Proof of this is that many of the latest DDOS (Distributed Denial of Service) attacks have been launched by hacked Internet-connected smart devices.

In the previous posts, you learned how to configure mosquitto to receive messages published by clients and to forward them to all the subscribers. Today I’m going to show you how to configure the application security of your MQTT broker.

This post is about the standard settings mosquitto offers. On the Internet you can find several plugins to extend its functionalities and to implement advanced security settings, like storing accounts in different backends or using json web tokens.

Authentication

The first step to make our broker secure is to require clients who are connecting to authenticate themselves. In this way, only authorized clients can send/receive messages.

Open the mosquitto.conf file and, first of all, disable the anonymous access setting to false the corresponding parameter.

mqtt-sec01

When you change a setting in the file, make sure that the hash (#) character is not present at the beginning of the line, otherwise the parameter is not considered

The list of the users authorized to connect to the broker is defined in a dedicated file. The name of this file is configured with the password_file parameter (in the example below, the file containing users and passwords is named “pwdfile”, stored in the same folder of mosquitto):

mqtt-sec02

Using the mosquitto_passwd.exe command you can create (-c) the file and add a new user (“luca”). The command will ask to type the user’s password twice:

mqtt-sec03

If you need to add additional users, you can run the command again, without the -c option:

mosquitto_passwd.exe pwdfile sara

Let’s try the new configuration. First run mosquitto specifying the configuration file (mosquitto.exe -c mosquitto.conf), then try to publish a message as you learned in the previous tutorials:

mqtt-sec04

You can see that the connection is refused because of the client is not authorized: we indeed didn’t specify any users and anonymous access was disabled.

Try to run the command again, this time specifying the correct user and password and you’ll find that the message will be correctly published:

mosquitto_pub.exe -u <utente> -P <password> -t home/bedroom/temp -m 19

Authorization

After having defined who can connect to the broker, it’s now time to define what a user can do once connected. Again, the configuration of the ACLs (access control lists) is in a dedicated file, whose name is configured with the acl_file parameter:

mqtt-sec05

The ACL file contains blocks with the following structure:

  • the first line, user <nomeutente>, that defines the user the following ACLs are applied to
  • one or more lines, topic <read|write|readwrite> <nometopic>, that define which actions the user above can do on the specified topic

Let’s see an example, a file in which we define a user (writeclient) that can send messages to the secure/data topic and a user (readclient) that can read those messages. Both the two users can in addition send an receive messages from all the open/# topic (for an explanation about the use of wildcards read my previous post):

mqtt-sec06

(lines starting with # are comments)

Let’s now try to send a message using the readclient user, who’s not allowed to do that:

mqtt-sec07

The client doesn’t get any error message, but in the mosquitto logs you can clearly see that the publish command was denied, consistent with the configured ACLs.

If instead the right users are specified, the message is correctly routed from the publishing client to the subscribing one:

mqtt-sec08

Conclusions

It’s very important to secure our broker, especially if it’s connected to a public network (for example if it’s available on the Internet). In today’s blog post you learned how to define users and grants in mosquitto.

ChipKIT based weather station using BME280 sensor module

P_20161109_153854-600

A comprehensive tutorial on making a chipKIT based weather station using BME280 sensor module from Embedded Lab:

BME280 is a fully integrated environmental unit from Bosch that combines sensors for pressure, humidity, and temperature in a tiny 8-pin metal-lid LGA package of size 2.5 x 2.5 x 0.93 mm³. Because of its compact size, ease of use (BME280 supports standard I2C and SPI interfaces), and availability of supporting open-source Arduino libraries, BME280 is very popular among weather enthusiasts. This project describes how to read barometric pressure, relative humidity, and temperature measurements from BME280 using chipKIT Uno32 to make a standalone weather station. The sensor readings are acquired over an I2C bus and are displayed on a Nokia 5110 LCD display.

More details at Embedded Lab blog.

MQTT – Topics

In the first part of this tutorial, I introduced the concept of topics. Every message sent to an MQTT broker must be published to a topic and clients connected to the broker decide which messages to receive subscribing to one or more topics.

Today you’ll learn the format of the topics and different ways to subscribe to them.

A topic is a string (UTF-8 format) and consists of one or more levels, separated by a forward slash.

Here are some examples of topics:

commands
home/temperature
garden/tomatoes/humidity

A topic must have at least one character and can contains spaces (even if it’s not recommended).

Note that topics are case sensitive, therefore the following topics are different: Home/Temperature and home/temperature

You’re not required to define/create a topic on the broker to use it: a client sending a message can specify any topics, provided it’s syntactically correct.

Wildcards

A client can subscribe one or more topics. The subscription can be precise, specifying the exact name of the topic to be subscribed. Alternatively, you can use two wildcards, that are special characters used to subscrive a group of topics.

The first wildcard is the character. This subscribes to a single level of topics.

Let’s see an example: if you subscribe to the topic home/+/temperature, you subscribe to all the topics that contain an arbitrary string instead of the wildcard, for example:

  • home/living/temperature
  • home/kitchen/temperature
  • home/terrace/temperature

Here are on the contrary some topics thare are not subscribed:

  • garage/temperature
  • home/living/humidity
  • home/terrace/dx/temperature

The last example explains that the + character is wildcard for a single level (it therefore does not correspond to the two levels, terrace/dx, of the example).

The multilevel wildcard character is #. This must be the last character in the topic and preceded by a forward slash, /.

For example, if you subscribe to the topic home/indoor/# you’ll get the messages of the following topics:

  • home/indoor/temperature
  • home/indoor/kitchen/temperature
  • home/indoor/kitchen/humidity

while you won’t get messages from the following topics:

  • home/outdoor/temperature
  • garage/humidity
If you subscribe to the # topic (without specifying any root string) you’ll receive all the messages sent to the broker

Let’s try!

We can verify what explained above thanks to mosquitto. After having executed the server, let’s subscribe to the topic home/+:

mosquitto_sub.exe -t home/+ -v

(we use the -v option to display, together with the message, also the topic the message was published to).

If now we send messages to different topics, we can verify which of them are received and which not:

mqtt-wild4

We can do a similar test for the wildcard #:

mqtt-wild3

MQTT – enc28j60 and Arduino (22)

Being a comics fan, I couldn’t resist publishing a crossover between my “famous” tutorial about using the enc28j60 driver with Arduino and the fresh new tutorial about the MQTT protocol.

So, after having introduced the MQTT protocol and explained how to use mosquitto, today I’ll show you how to send data to an MQTT broker using the enc28j60 ethernet controller.

Libraries

We’re going to use the PubSubClient library by Nick O’Leary to “talk” with our MQTT broker (mosquitto). You can install the library using the Library Manager in the Arduino IDE (Sketch menu – Include Library – Manage Libraries…):

mqtt-arduino01

The library I chose is based on the Ethernet Client API of Arduino, usually available if you use the “official” ethernet chip (W5100). I’ve already blogged about the UIPEthernet library that allows to use sketches and libraries designed for the W5100 chip also with the enc28j60 one. After having installed also this library in the IDE, you’re ready to write your first MQTT sketch…

For this tutorial I decided to connect to the Arduino a temperature (and humidity sensor): the DHT11. I’m going to use the DHT sensor library by Adafruit to read values from that sensor, this library too can be installed through the Library Manager.

mqtt-arduino03

Sketch

The full sketch is available in my Github’s repository.

Let’s see the most important lines of code. After having included the required libraries, the sketch defines some constants, like the topic to send data to, the identifier of the MQTT client, the interval between two transmissions and the configuration of the sensor:

#define CLIENT_ID       "ArduinoMQTT"
#define TOPIC           "temperature"
#define PUBLISH_DELAY   5000
#define DHTPIN          3
#define DHTTYPE         DHT11

then it configures the MAC address of the network interface and the IP address of the MQTT broker (= the IP address of the PC where mosquitto runs on):

uint8_t mac[6] = {0x00,0x01,0x02,0x03,0x04,0x05};
IPAddress mqttServer(192,168,1,4);

the MQTT client is configured passing the Ethernet client to be used and the server (and its port) to connect to:

mqttClient.setClient(ethClient);
mqttClient.setServer(mqttServer, 1883);
Serial.println(F("MQTT client configured"));

in the main loop() the sketch checks is the interval between two transmissions is passed; if so it reads the new temperature value and sends it to the broker:

if(millis() - previousMillis &gt; PUBLISH_DELAY) {
  sendData();
  previousMillis = millis();
}

it’s very important to call the loop() command of the MQTT Client frequently to let it process the different incoming packets and commands:

mqttClient.loop();
every tot seconds, the sketch reads the temperature, displays it on the serial terminal and sends the value to the broker. The library returns a float value, it’s therefore required to convert it in string before sending (using the dtostrf command):
float t = dht.readTemperature();
Serial.print("Temperature: ");
Serial.println(t);
 
if(mqttClient.connect(CLIENT_ID)) {
  mqttClient.publish(TOPIC, dtostrf(t, 6, 2, msgBuffer));
}

Test

Before running the sketch, first you have to start mosquitto on your PC. Open a second command prompt and run mosquitto_sub to subscribe the temperature topic:

mosquitto_sub.exe -t temperature

if you now run the sketch, you should be able to see the temperature values sent to the broker and, from this one, to the subscriber:

mqtt-arduino02

MQTT – Introduction

If two devices have to talk to each other, they must adopt the same protocol. In the past, I’ve already shown different types of communication, for example between Arduino and a C# program, Arduino and a website or Arduino and an NTP server.

All those communications are point-to-point ones, that is between two actors. Today I’m instead going to introduce you a quite special protocol: MQTT (MQ Telemetry Transport).

The MQTT protocol was invented in 1999 by two researchers (Andy Stanford-Clark of IBM and Arlen Nipper of Arcom) and its specs are public (IBM website). Its most important features, that make it very suitable for machine 2 machine communication, are:

  • it’s a publish/subscribe protocol
  • it’s very simple
  • it has a very low overhead (= amount of data to be sent for the protocol itself)

Publish/Subscribe

The main feature of MQTT is to be a publish/subscribe protocol.

In a traditional communication, the source of the data (for example a temperature sensor) sends it directly to the final user (for example the thermostat):

mqtt-00a

In a publish/subscribe communication, instead, the sensor sends (publishes) the data to a central system called broker. All the devices that need that data talk to the broker (they subscribe to it) and the broker sends them the data when available:

mqtt-00b

The main advantage of the pub/sub paradigm is to decouple the data source and the destination:

  • spatial decoupling: there’s no need of a direct connection between data source and destination;
  • temporal decoupling: the data source doesn’t need to be active at the same time of the destination;
  • synchronization decoupling: operations are not halted during publish or receiving.

From what above said, it’s easy to understand why the MQTT protocol and the pub/sub paradigm in general is very used in the Internet of Things: the simplest devices, often powered by batteries, can activate every tot seconds, send the data to the browser and return to power saving mode. It’s the broker, that is always active, to send the data to the end users (actuators, dashboards…) when it’s needed.

Topics

A broker can manage a huge number of messages, coming from different sources and intended for different users. How can it correctly route those messages? Thanks to the topics: each message is published to a specified topic (a temperature sensor for example can use temp as the topic for its data). When subscribing, the user has to specify one or more topics it wants the data of (in our example, to receive data from the temperature sensor, the thermostat should subscribe the temp topic):

mqtt-09

Later I’ll explain how topics are structured and how it’s possible to subscribe more topics using wildcards.

Mosquitto

After the introduction part, let’s see how to use the MQTT protocol. You now know that the most important element is the broker: one of the most used software as MQTT broker is mosquitto.

Mosquitto is available for almost all the operating systems (Windows, Linux, MacOS); today I’ll show you how to install and use it under Windows.

First, download the latest version from the official site and unzip the archive in a new folder.

We also need two external libraries, from OpenSSL and pThreads. Let’s start downloading the OpenSSL build for Windows (very important is to download a version from the 1.0.x tree):

mqtt-01

From the folder where OpenSSL was installed, copy libeay32.dll and ssleay32.dll to the mosquitto folder:

mqtt-02

Now download the pthreadVC2.dll library from SourceWare and copy this file too in the mosquitto folder

mqtt-03

This is the final content of the mosquitto folder after having copied the three libraries:

mqtt-04

First example

Now we can run mosquitto. Open 3 command prompts: one to run the broker, one to simulate the temperature sensor (publish) and one to simulate the thermostat (subscribe).

In the first prompt, run the broker, with the -v parameter to activate the verbose output :

mqtt-05

In the second prompt, use the mosquitto_sub to subscribe the temp topic:

mqtt-06

In the third prompt, use the mosquitto_pub command to publish the temperature value (21° for example) to the temp topic:

mqtt-07

If you go back to the prompt where the mosquitto_sub is running, you can see that the temperature value has correctly been routed and received:

mqtt-08

Conclusions

In this first article, I introduced the pub/sub paradigm and the MQTT protocol. I’ve also shown how to use mosquitto as a broker to send/receive messages. In the next articles, I’ll show you some real examples, with Arduino (and not only…).