ESP32 (31) – BLE, GAP

In my previous tutorials you learned how to use the wifi interface of the esp32 chip. Starting from this post, I’m going to explain you the second wireless technology the esp32 chip supports: bluetooth.

In particular, my tutorial will be about Bluetooth Low Energy (BLE), sometimes called also Bluetooth 4.0 or Bluetooth Smart:

ble-001

Bluetooth Low Energy

BLE is a technology to build personal wireless area networks (WPAN); that is it allows to put in communication different devices (computers, smartphones, smartwatches…) “close” to each other (a theoretical maximum distance of 100m). As the name suggests, version 4.0 of the Bluetooth standard was designed to reduce the power consumption of the devices connected to the network.

Devices are divided into two families:

  • central
  • peripheral

the first ones (central) are devices like PCs, tablets or smartphones with good processing power and memory. The second ones (peripheral) are instead sensors, tags… with less hardware resources and power. A central device can be connected to more peripheral devices at the same time, while it’s not true the opposite:

ble-002

BLE devices periodically report their presence by transmitting advertising packets. The advertising packet can contain up to 31 bytes of data and the transmission frequency can be chosen by the single device: reducing this frequency can indeed reduce energy consumption.

If a BLE device, after having received an avertising package, wants to obtain more information from the device that transmitted it, it can request a second packet of information (always for a maximum of 31 bytes), the scan response packet. The transmission of this second data package is optional:

ble-003

A BLE device can take advantage of advertising packages to send data in broadcast mode. In this case, this device is called a broadcaster, while the devices that receive the data are called observers.

What explained above is defined within a BLE specification called Generic Access Profile (GAP).

esp32

In this first tutorial you’ll learn how to develop a program that will periodically scan the air looking for BLE devices, that is a program which receives advertising packets and displays the data received in the serial console.

Before compiling a program which uses the Bluetooth controller, make sure (using menuconfig) that the controller is enabled (Component config -> Bluetooth):

ble-004

Start your program with the required header files:

#include "esp_bt.h"
#include "esp_bt_main.h"
#include "esp_gap_ble_api.h"

You also need to initialize the NVS partition, used by the Bluetooth driver:

ESP_ERROR_CHECK(nvs_flash_init());

the Bluetooth controller of the esp32 chip supports both the classic and the low energy mode. If one of this two modes is not required in your program, you can release the memory the framework normally allocates to manage it using the esp_bt_controller_mem_release() command. In this example you’re not going to use the classic mode, so:

ESP_ERROR_CHECK(esp_bt_controller_mem_release(ESP_BT_MODE_CLASSIC_BT));

Now you can configure (using the default settings) the controller in BLE mode:

esp_bt_controller_config_t bt_cfg = BT_CONTROLLER_INIT_CONFIG_DEFAULT();
esp_bt_controller_init(&bt_cfg);
esp_bt_controller_enable(ESP_BT_MODE_BLE);

The esp-idf framework esp-idf includes the Bluedroid bluetooth stack. This library was developed by Broadcom and used by Android since version 4.2 Bluedroid is initialized and enabled with the following commands:

esp_bluedroid_init();
esp_bluedroid_enable();

Now you’re ready to start scanning…

GAP, events

In a similar way to what you learned about the wifi driver, the bluetooth driver also runs in a thread separate from our program and communicates with it via events. In order to receive such events, you have to implement a callback function. Whenever the bluetooth driver has to notify an event, it will call that function.

The prototype of the callback function is:

static void esp_gap_cb(esp_gap_ble_cb_event_t event, esp_ble_gap_cb_param_t *param);

You tell the driver which callback function has to use with the esp_ble_gap_register_callback() method:

ESP_ERROR_CHECK(esp_ble_gap_register_callback(esp_gap_cb));

The Bluetooth driver handles several events, there are the ones related to the scan process:

ble-005

Before being able to start the scan process, you have to configure the scan parameters. The configuration is performed using the esp_ble_scan_params_t struct. It’s very important that the variable with the scan parameters is available during all the scan process; it’s therefore necessary to define it globally:

static esp_ble_scan_params_t ble_scan_params = {
  .scan_type              = BLE_SCAN_TYPE_ACTIVE,
  .own_addr_type          = BLE_ADDR_TYPE_PUBLIC,
  .scan_filter_policy     = BLE_SCAN_FILTER_ALLOW_ALL,
  .scan_interval          = 0x50,
  .scan_window            = 0x30
};

With the esp_ble_gap_set_scan_params() method you configure the scan process passing the struct defined above to the driver:

esp_ble_gap_set_scan_params(&ble_scan_params);

When the driver has finished the configuration, it calls the callback function with the event ESP_GAP_BLE_SCAN_PARAM_SET_COMPLETE_EVT. Depending on the event raised, the callback function also receives some parameters. The framework’s Programming Guide explains – for each event – the related parameters. For this event, it’s available the variable scan_param_cmpl that contains only the status parameter.

In the callback function you can use the switch statement to identify each event:

switch (event) {
  case ESP_GAP_BLE_SCAN_PARAM_SET_COMPLETE_EVT:
  [...]
  break;

and check if the configuration was successful with:

if(param->scan_param_cmpl.status == ESP_BT_STATUS_SUCCESS)

If so, you can start the scan process:

esp_ble_gap_start_scanning(10);

The parameter is the scan duration (in seconds).

Once the scan process has started, the driver raises the ESP_GAP_BLE_SCAN_START_COMPLETE_EVT event. For this event too it’s possible to verify the correct execution by reading the status parameter (pay attention: the name of the variable which contains the parameter changes!):

case ESP_GAP_BLE_SCAN_START_COMPLETE_EVT:
  if(param->scan_start_cmpl.status == ESP_BT_STATUS_SUCCESS)
    printf("Scan started\n\n");
  else 
    printf("Unable to start scan process");
  break;

GAP, scan process

During the scan process, for each advertising packet the chip receives the event ESP_GAP_BLE_SCAN_RESULT_EVT is raised.

This event contains some subevents. You can identify which subevent was raised reading the scan_rst.search_evt parameters. Two subevents are in particular interesting:

ble-006

the first tells you that a device was detected, while the second one that the scan process completed.

For each detected device, various information is available. For now let’s print its address in the console:

case ESP_GAP_BLE_SCAN_RESULT_EVT:
  if(param->scan_rst.search_evt == ESP_GAP_SEARCH_INQ_RES_EVT) {
    printf("Device found: ADDR=");
    for(int i = 0; i < ESP_BD_ADDR_LEN; i++) {
      printf("%02X", param->scan_rst.bda[i]);
      if(i != ESP_BD_ADDR_LEN -1) printf(":");
    }

The address is an uint8_t array, whose size is defined by the ESP_BD_ADDR_LEN constant. The address is normally displayed in hex form, with the bytes separated by :

ble-007

Device list management

As explained above, the ESP_GAP_BLE_SCAN_RESULT_EVT event is raised everytime a device sends an advertising packet. This means that a single device will be detected multiple times during the scan process.

It’s therefore necessary to maintain a list of the known devices. In my Github repository you can find a test program that scans the network and prints all the detected devices.

You can verify if it works correctly comparing what the program detects with the BLE devices listed by a smartphone… for Android for example you can use the very good nRF Connect application by Nordic.

Here’s what my program detected:

ble-008

and here’s the nRF Connect’s screenshot:

ble-009

 

ESP32 (30) – HTTP server in SoftAP mode

One of the most frequent questions I receive from my website’s contact form or from my Facebook page is whether it’s possible to publish an HTTP server when the esp32 chip is working in SoftAP mode, that is when it publishes its own wifi network.

In previous tutorials (18 – Access Point and 20 – Webserver) I’ve already blogged about the two subjects separately, today I’ll explain how to combine them.

Access Point

Let’s start defining the parameters of the TCP/IP network will be published by the esp32 chip. You have to choose an addressing plan, that is which IP addresses will belong to the network. You can use some of the IP addresses IANA assigned to private networks (RFC1918):

  • 10.0.0.0/8
  • 172.16.0.0/12
  • 192.168.0.0/16

The numbers /8/12/16 are, in short form, the network mask. For example/8 means that the network 10.0.0.0/8 contains all the addresses from 10.0.0.1 to 10.255.255.254, a total of 16.777.216 different addresses. Thanks to the subnetting you can slice a network in smaller subnetworks.

In this example I’m going to use the network 192.168.1.0 with netmask 255.255.255.0 (/24), that is a network with 254 usable addresses (192.168.1.1 – 192.168.1.254). You have to statically assign to the esp32 chip an address that belongs to that network; to make things easy I chose the first one (192.168.1.1):

tcpip_adapter_ip_info_t info;
memset(&info, 0, sizeof(info));
IP4_ADDR(&info.ip, 192, 168, 1, 1);
IP4_ADDR(&info.gw, 192, 168, 1, 1);
IP4_ADDR(&info.netmask, 255, 255, 255, 0);
ESP_ERROR_CHECK(tcpip_adapter_set_ip_info(TCPIP_ADAPTER_IF_AP, &info));

The other devices which will connect to the network will dynamically configure their IP address, thanks to the DHCP service. To use the DHCP server in your program, first you have to stop it before configuring the network settings and then start it:

ESP_ERROR_CHECK(tcpip_adapter_dhcps_stop(TCPIP_ADAPTER_IF_AP));
[...]
ESP_ERROR_CHECK(tcpip_adapter_dhcps_start(TCPIP_ADAPTER_IF_AP));

After having configured the TCP/IP network and the related services, you can set up the wifi interface of the esp32 chip and activate the SoftAP mode:

wifi_init_config_t wifi_init_config = WIFI_INIT_CONFIG_DEFAULT();
ESP_ERROR_CHECK(esp_wifi_init(&wifi_init_config));
ESP_ERROR_CHECK(esp_wifi_set_storage(WIFI_STORAGE_RAM));
ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_AP));

You have to prepare a wifi_config_t struct that contains all the parameters for the wifi network (SSID, authentication…) and pass it to the method:

ESP_ERROR_CHECK(esp_wifi_set_config(WIFI_IF_AP, &ap_config));

You can hardcode the parameters in the program, or make them configurable via menuconfig (as in this example) or Possiamo decidere di inserire i parametri hardcoded nel programma, di renderli configurabili tramite menuconfig (come nell’esempio) or even read them from a config file or from NVS variables.

Finally start the wifi interface with:

ESP_ERROR_CHECK(esp_wifi_start());

HTTP server

Now you can use the Netconn API of the lwip library to bind your program to the TCP/80 port (it’s the standard port used by the HTTP protocol) and to listen for incoming connections:

struct netconn *conn;
conn = netconn_new(NETCONN_TCP);
netconn_bind(conn, NULL, 80);
netconn_listen(conn);

The accept() method blocks the program until a new connection is accepted:

struct netconn *newconn;
netconn_accept(conn, &amp;newconn);

After having established the connection, your program must speak the same language (protocol) of the client; in this case the HTTP protocol used by web browsers.

To keep the example simple, it will publish a static website, whose content is stored in an SPIFFS partition (see the ESP32lights project for more information). Therefore, the program will answer requests in the form GET <resource> looking for the resource in the SPIFFS partition and, if found, sending its content using the netconn_write() method:

http-ap-001

Bonus, mDNS

To be able to connect to the HTTP server published by the esp32 chip, a client must know the IP address (192.168.1.1) assigned to the chip.

We can leverage the mDNS service (as explained here) to allow the client to connect using an alias (for example esp32web):

mdns_server_t* mDNS = NULL;
ESP_ERROR_CHECK(mdns_init(TCPIP_ADAPTER_IF_AP, &mDNS));
ESP_ERROR_CHECK(mdns_set_hostname(mDNS, "esp32web"));
ESP_ERROR_CHECK(mdns_set_instance(mDNS, "esp32 webserver"));

If the device you’re using supports the mDNS service, it will be possible to connect to the website using the address esp32web.local:

http-ap-002

Demo

The complete program is available in my Github repository. After having loaded it on your devboard, a new wifi network will be available:

http-ap-003

if you connect to that network, your device will be assigned an IP addess on the 192.168.1.0/24 network:

http-ap-004

and you’ll be able to point the browser to http://esp32web.local (or http://192.168.1.1) and display the website:

http-ap-005

BLE with Bluno Beetle

Some time ago my friend Mauro Alfieri showed me an interesting development board produced by DFRobot and called Bluno Beetle (now Beetle Ble). It seemed the perfect board to start “playing” with the Bluetooth Low Energy (BLE) technology; therefore I ordered  one board directly from the DFRobot store.

I expected to receive the usual anonymous parcel with the board inside an antistatic plastic bag; DFRobot instead sends its products in an elegant cardboard box, protecting them with foam:

bluno-011 bluno-012

Bluno Beetle is really small and therefore perfect for wearable projects:

bluno-013 bluno-014

But what is it? Simplifying is a board, Arduino Uno compatible (it hosts the ATmega328P microcontroller) to which has been added the CC2540 chip from Texas Instruments to act as USB and BLE controller. The two chips communicate via a serial interface:

bluno-001

The CC2540 chip is actually a real microcontroller that runs a firmware developed by DFRobot. This firmware can be configured using AT commands. Normally the firmware runs in transparent mode, that is it acts as a “bridge” between the USB/BLE interfaces and the ATmega microcontroller. If you then connect the Bluno to your PC and activate the serial monitor, each character you type is forwarded to the ATmega and viceversa.

To send AT commands, first you have to enter the AT mode of the firmware, sending the + character 3 times (without appending a line ending). The firmware confirms the new mode with the sentence “Enter AT Mode”:

bluno-006

Now you can send the commands, appending the Windows line terminator (CRLF). For example to display the firmware version:

bluno-007

To exit the AT mode and go back to transparent mode, you have to send the AT+EXIT command.

Drivers

It may happen that – if it’s the first time you connect the Bluno Beetle to your Windows PC – it is not correctly recognized:

bluno-003

The correct drivers are shipped with the Arduino IDE. You only need to do a manual installation specifying the path where you installed the IDE:

bluno-004

Windows will identify the new device as an Arduino Uno:

bluno-005

Arduino

As explained above, if the firmware running on the CC2540 chip is in transparent mode, using the USB connection you can talk directly to the ATmega328P microcontroller. This means that you can program the microcontroller using the Arduino IDE without any problems… just choose Arduino Uno as board and select the correct serial port:

bluno-008

BLE and transparent mode

In transparent mode Bluno transmits via BLE each byte it receives from Arduino (the ATmega microcontroller) and – viceversa – it sends to Arduino each byte it receives from BLE.

In this first post let’s explore the demo application DFRobot provides; in a future post I’ll explain how to develop your application to interact with Bluno Beetle via BLE.

If you have an Android smartphone, you can directly install the apk file for the application named BlunoBasicDemo (application of which the source code is also available). In the same Github repository you can also find the source code of the iOS application, you have to compile by yourself.

Compile and upload the following sketch on the board:

unsigned long previous_time = 0;
 
void setup() {
  Serial.begin(115200);
}
 
void loop() {
 
  if (Serial.available() > 0) {
    int incomingByte = Serial.read();
    Serial.print("New byte received: 0x");
    Serial.println(incomingByte, HEX);
  }
  unsigned long actual_time = millis();
  if(actual_time - previous_time > 10000) {
    Serial.println("Hello world!");
    previous_time = actual_time;
  }
}

The sketch reads the incoming bytes (coming from the app) and sends back to the app their hexadecimal value. Every 10 seconds moreover the sketch sends to the app the text Hello world!.

Launch the app. After having clicked on the Scan button, you can choose your Bluno board from the list of detected devices:

bluno-009

Every 10 seconds you should see a new HelloWorld! string appear. You can try to send a character (for example the letter “a”); you’ll receive an answer from Arduino (0x61 is indeed the hex code – in the ASCII table – for the letter “a”):

bluno-010

HID mode

Bluno also supports the HID (Human Interface Device) mode. When running in this mode, Bluno simulates an input peripheral (keyboard, mouse…) connected via BLE.

The AT command that enables this mode is:

  • AT+FSM=FSM_HID_USB_COM_BLE_AT

After having enabled the HID mode, you can send one or more “keys” with:

  • AT+KEY=

you can send up to 3 different keys at a time, concatenating their codes with the + character. The codes to be used, according to the type (page) of the HID device, are listed in the USB specification.

The AT+KEY command notifies the pressure of a key on the keyboard.  It is therefore necessary, after a few moments, to send the AT+KEY=0 command to indicate that the key has been released; otherwise on the PC associated with Bluno you’ll see the character appear repeatedly!

Debug mode

Using two different AT commands you can enable the debug mode of the firmware. This mode allows to receive – via the USB connection – a copy of all the data sent and received through the BLE connection.

The two commands are:

  • AT+BLUNODEBUG=ON (copies the messages sent by the ATmega)
  • AT+USBDEBUG=ON (copies the messages received from BLE)

bluno-002

By default the first debug mode is active, while the second is disabled. You can verify it if you upload the sketch listed above: in the serial monitor you’ll see the Hello World! sentences but not the characters sent by the app.

ESP32 (29) – Deep sleep

One of the major concerns for embedded devices is the power consumption. If the device you’re designing will be battery powered, it’s indeed important to reduce as much as possible its power consumption  to maximize the autonomy (= the working time before it’s necessary to replace or recharge the battery).

The esp32 chip offers 5 different power modes. The “normal” mode is named active mode; in this mode all the features of the chip are available. By starting to reduce the CPU speed and disabling some peripherals and cores, the chip switches to different power saving modes, as summarized in the following diagram:

sleep-001

In this first post about the esp32 power saving modes, I’ll explain the deep sleep mode.

Deep sleep

The esp-idf framework actually supports two power saving modes: light sleep and deep sleep. Between the two, the deep sleep mode is the one which offers greater energy savings. In this mode, are turned off:

  • both the CPUs
  • most of the RAM memory
  • all the peripherals

by default are instead kept active:

  • the RTC controller
  • the RTC peripherals, including the ULP coprocessor
  • the RTC memories (slowfast)

You can put the chip in deep sleep with the esp_deep_sleep_start() method, while it’s possible to wake up it via different events:

sleep-002

When the chip wakes up from deep sleep, a new boot sequence is performed. It’s therefore very important to understand that the execution of your program does not restart at the point where the esp_deep_sleep_start() method is called.

Let’s see how to configure and use two wake up events; in a future post I’ll write about touch pad and ULP.

Timer

The simplest wake up event is for sure the one which leverages a timer of the RTC controller. Thanks to the method:

esp_err_t esp_sleep_enable_timer_wakeup(uint64_t time_in_us)

you can wake up the esp32 chip after the specified number of milliseconds. The method must be called before entering the deep sleep mode:

// wakeup after 10 seconds
esp_sleep_enable_timer_wakeup(10000000);
esp_deep_sleep_start();

I/O triggers

In a previous post I’ve already blogged about the possibility to receive interrupts when a digital pin of the chip changes its status. We can leverage a similar functionality to wake up the chip from sleep.

With the method

esp_err_t esp_sleep_enable_ext0_wakeup(gpio_num_t gpio_num, int level)

you can enable the wake up if the specified pin (gpio_num) changes its status (level).

You can only use the pins with RTC function (0, 2, 4, 12-15, 25-27, 32-39) and the possible levels are 0 (= low) or 1 (high). If, for example, you want to wake up the chip from deep sleep if pin 4 has a low level, you’ll write:

esp_sleep_enable_ext0_wakeup(4, 0);

The framework also offers a method to monitor different pins:

esp_err_t esp_sleep_enable_ext1_wakeup(uint64_t mask, esp_sleep_ext1_wakeup_mode_t mode)

the pins (this method also accepts only the ones specified above) must be specified in a bitmask and  the wakeup modes are:

  • ESP_EXT1_WAKEUP_ALL_LOW = wakeup  when all the pins are low
  • ESP_EXT1_WAKEUP_ANY_HIGH = wakeup when at least one pin is high
When the chip wakes up from the sleep, the pins specified will be configured as RTC IO. To be able to use them again as normal digital pins, you have first to call the rtc_gpio_deinit(gpio_num) method. The ext0_wakeup method at the moment cannot be used together with touch pad or ULP events.

After the wake up…

If you configured more than one wake up event, you can know which specific event woke up the chip with:

esp_sleep_wakeup_cause_t esp_sleep_get_wakeup_cause()

The possible constants are:

sleep-003

For the ext1_wakeup event, a specific method is available to get the bitmask of the pins:

uint64_t esp_sleep_get_ext1_wakeup_status()

Memory

As explained above, in deep sleep mode the content of the RTC fast and RTC slow memories is preserved. You can therefore use those memory segments to store data that must be retained during the sleep.

To ask the compiler to store a variable in the RTC slow memory, you can use the RTC_DATA_ATTR attribute, or the RTC_RODATA_ATTR one if the variable is read only:

RTC_DATA_ATTR static time_t last;

Demo

I wrote a program (its source code is in my Github repository) to demonstrate the deep sleep mode and two different wake up events:

Parsing a packet using structs

When I was analyzing the source code of an opensource software, I found an interesting technique to parse incoming data in a very efficient way, using a struct.

This technique works if the received data has a fixed length and structure; for example it applies very well to data “packets”.

When receiving, the program normally stores incoming data in a buffer in the RAM memory. For example let’s assume that the packet has a length of 14 bytes:

char rx_buffer[14];

Your specific protocol defines the following fields in the packet:

  • address, 5 bytes
  • temperature value, 3 bytes
  • humidity value, 3 bytes
  • TX counter, 1 bytes
  • checksum, 2 bytes

You can define a struct that represents the structure of the packet:

typedef struct {
  uint8_t address[5];
  uint8_t temperature[3];
  uint8_t humidity[3];
  uint8_t tx;
  uint8_t checksum[2];
} my_packet;

The technique consists in applying, as if it were a template, the struct to the buffer:

mem-struct-001

in this way the received bytes are automatically divided into the different fields:

mem-struct-002

Therefore, to parse the data you only need to cast the buffer:

my_packet* parsed_data = (my_packet*)rx_buffer;

and access the struct fields:

parsed_data.address;
parsed_data.temperature;
[...]

Padding and packed

You still have a problem: to make it more efficient to access the memory, compliers normally align the variables based on their size. This means that – when a compiler allocates memory for your struct – some empty space (named padding) could be added between the different variables:

mem-struct-003

The presence of the padding space causes a misalignment if you apply the struct to the buffer with the received data (which are instead continous). Fortunately, it is possible to ask the compiler not to insert padding using the packed attribute:

typedef struct {
  uint8_t address[5];
  uint8_t temperature[3];
  uint8_t humidity[3];
  uint8_t tx;
  uint8_t checksum[2];
}__attribute__((packed)) my_packet;

If you remember to add this attribute, the technique works perfectly!

ESP32lights

Today’s project, ESP32lights, is a smart device based on the esp32 chip.

esp32lights-004 esp32lights-005

Thanks to ESP32lights you can turn a load on and off (I used it for my christmas lights)

  • manually
  • based on daily schedules
  • based on the light intensity

ESP32lights connects to your wifi network, can be configured and operated via a web browser and it’s optimized for mobile devices (responsive web interface based on jQuery Mobile).

Components

The heart of ESP32lights is the Lolin32 Lite devboard by Wemos. One of its digital pins is connected to a relay module, which controls the load. Two digital pins are assigned to the first i2c controller of the esp32 chip and are connected to a BH1750 light intensity sensor. All the elements are powered by an HLK-PM01 module by Hi-Link, which directly converts the mains’ 220V AC to 5V DC without the need of any external components:

esp32l-001

All the components are placed in a waterproof enclosure, to be able to use the device outdoor:

esp32lights-006 esp32lights-007

Programming

The firmware for the esp32 devboard is available in my Github repository.

In a following paragraph I’ll explain how it works. If you just want to build the device, you can program the firmware as it follows:

1) clone my repository in a local folder of your PC (you also have to install the development env esp-idf):

2) configure the correct settings for your wifi network and your timezone via menuconfig:

esp32lights-003

3) compile and flash the firmware:

make flash

4) store the image of the SPIFFS partition into the flash memory (replace the COM port of your devboard and the path where you saved the img file):

python $IDF_PATH/components/esptool_py/esptool/esptool.py --chip esp32 --port COM15
 --baud 115200 write_flash --flash_size detect 0x180000 /home/esp32lights.img

If everything is ok, when you connect to the serial console of the devboard (make monitor) you should see the following output:

esp32lights-025

Use

ESP32light publishes an HTTP interface you can use to set the schedules or the light intensity threshold or to manually turn the load on and off.

You can open the web interface connecting – through a PC or a smartphone -to the address http://<esp_ip> (the IP address of the board is displayed in the serial output as shown in the previous paragraph).

The interface has 3 tabs, one for each working mode:

esp32lights-001

The page footer displays the actual working mode and the relay status:

esp32lights-002

In this short video, you can see how the device works:

Software

I developed the firmware for ESP32lights leveraging what I explained in my previous tutorials about the esp32 chip. If you follow my blog, you probably understood that I really like the divide et impera method, that is divide a complex project into small, simpler tasks.

All the configuration settings of ESP32lights (actual working mode, start and stop time…) are stored in the NVS partition of the flash memory, as I explained in this tutorial.  In this way, it’s possible to keep them even if the chip is restarted:

nvs_handle my_handle;
int working_mode;
[...]
esp_err_t err = nvs_flash_init();
err = nvs_open("storage", NVS_READWRITE, &my_handle);
err = nvs_get_i32(my_handle, "mode", &working_mode);

The different elements of the web interface (html page, css style sheets…) are stored in an SPIFFS partition. In a previous tutorial you learned how to prepare the image and, in your program, get its content:

esp32l-002

In other tutorials I’ve also already explained you how to connect to a wifi network and how to use digital pins.

The setup phase is completed after having configured the BH1750 light intensity sensor. This sensor offers an i2c interface and therefore can be connected to one of the two i2c controllers of the esp32 chip as shown in this tutorial. In my program I included a driver developed by pcbreflux.

The main program runs two different tasks:

xTaskCreate(&http_server, "http_server", 20000, NULL, 5, NULL);
xTaskCreate(&monitoring_task, "monitoring_task", 2048, NULL, 5, NULL);

esp32l-003

The first one publishes the web interface, while the second one verifies – every second – if conditions exist (time or light intensity) to turn the load on or off:

if(working_mode == MODE_LIGHT && lux_valid) {
  int actual_light_value = get_light_value();
  if(actual_light_value < lux) {
    if(relay_status == false) {
      gpio_set_level(CONFIG_RELAY_PIN, 1);
      relay_status = true;
    }

Here’s in details how the http server fetches a static resource, stored in the SPIFFS partition.

First it adds to the resource path the root prefix for the SPIFFS partition (/spiffs):

sprintf(full_path, "/spiffs%s", resource);

then it checks if the resource exists in the partition:

if (stat(full_path, &st) == 0) {

if so, it opens the file in read mode:

FILE* f = fopen(full_path, "r");

and sends the content of the file to the client, reading blocks of 500 bytes:

char buffer[500];
while(fgets(buffer, 500, f)) {
  netconn_write(conn, buffer, strlen(buffer), NETCONN_NOCOPY);
}

Finally, this is how the web interface works. The interface is made by an html page (index.html) which uses jQuery to perform AJAX requests to the server and update the different page elements. You don’t need to enter the page name in the browser because of the http server automatically performs a redirect to it if the default page is requested:

if(strstr(request_line, "GET / "))
  spiffs_serve("/index.html", conn);

endpoints are published by the server and accessed using AJAX calls::

  • setConfig, to send a new configuration
  • getConfig, to read the actual configuration
  • getLight, to get the actual light intensity

When the page is loaded, it calls the getConfig endpoint to display the actual configuration; moreover it schedules every 5 seconds a call to the getLight endpoint to keep the light value updated:

refreshConfig();
setInterval("refreshLightLevel()", 5000);

When you click on the SET button, the page calls setConfig to send to the server the new configuration:

esp32l-004

All the information are sent using the JSON format. The esp-idf framework includes the cJSON library which makes it easy to create or parse a json message:

cJSON *root = cJSON_Parse(body);
cJSON *mode_item = cJSON_GetObjectItemCaseSensitive(root, "mode");
[...]
cJSON *root = cJSON_CreateObject();
cJSON_AddNumberToObject(root, "lux", light_value);
char *rendered = cJSON_Print(root);

Making of

I started the build of the device cutting a perfboard to the size of the enclosure:

esp32lights-008 esp32lights-009

The perfboard is screwed to the enclosure using two spacers:

esp32lights-010 esp32lights-011

I made two holes in one side of the enclosure for the main switch and for a status led:

esp32lights-012 esp32lights-013

I soldered all the different components on the perfboard and made the electric connections using wires:

esp32lights-014 esp32lights-015

To simplify the installation, all the external components (led, relay module…) are connected using jumpers:

esp32lights-016 esp32lights-017

First test:

esp32lights-018

I attached the light sensor to the top of the enclosure, after having made a hole to allow it to “see” the external light:

esp32lights-019 esp32lights-020

Finally I made the external connections, installing the main switch:

esp32lights-021 esp32lights-022

and connecting the output of the relay module to a wire with an universal plug at its end:

esp32lights-023 esp32lights-024

SIM800 GSM module

Today I received from  Banggood a GSM module based on the SIM800 chip by SIMCom. I’m going to use this module in a future Arduino project that will allow me to remotely control devices. In this article I’ll show you how to test the module.

The SIM800 is a quad-band (850/900/1800/1900MHz) chip and allows to trasmit/receive voice, SMS and data (using the GPRS network).

The small module (about 3x4cm) has the SIM800 chip and most of the components on one side, while on the other side you can find the sim card connector:

gsm-005 gsm-004

I chose to buy the module with an external antenna… in addition to the module you therefore receive the antenna and a short (about 20cm) cable to connect it:

gsm-006

The module is powered with 5V, offers a serial interface and you can control it using AT commands.

It is very important to connect the module to a power supply able to provide up to 2A, otherwise you may face problems on the stability of the GSM connection.

To verify if the module works you can therefore connect it to your PC using a simple USB – serial converter and send the correct AT commands as the manual explains. Be careful to invert the TX and RX pins: the TX pin of your adapter must be connected to the RX pin of the GSM module and viceversa:

gsm-008 gsm-007

The simplest test is to send and receive an SMS. For simplicity first configure the text mode (=1) with the command AT+CMGF=1. The module will answer with the command you send (without the starting AT) followed by OK.

Now you can specify the recipient of the message with the command AT+CMGS=”number. After having sent the “carriage return” character, the module will send the prompt, after which you can type the text of the message. End the message with the character 0x1A (CTRL-Z); the module will answer with the command you sent followed by the number of characters in the message (26):

gsm-001

When a new SMS is received, the module sends the +CMTI message. You can read the messages with the AT+CMGL command and you can also specify a filter (for example REC UNREAD, that is all the messages received but not read yet):

gsm-002

The filters available are listed in the manual:

gsm-009

After having verified that the module works, you can now realize a more complex project, for example interfacing it with your Arduino…

gsm-003

 

 

ESP32 (28) – MQTT and SSL

Security is a very important aspect for MQTT brokers. In a previous article you’ve already learned how to implement authentication and authorization. The weakness in that configuration was that credentials were transmitted in cleartext; it was therefore possible, for an attacker who can sniff the network traffic, to read and use them to impersonate a legitimate client.

Today I’ll show you how to encrypt the communication channel between client and broker using SSL certificates. I’ll also explain how to write a program for the esp32 chip to send data to the broker using the secure channel…

SSL certificate

To be able to encrypt the communication, mosquitto requires a server certificate.

First generate the private key (RSA with a length at least 2048 bits):

openssl genrsa -out mosquitto.key 2048

Then create the CSR file:

openssl req -new -out mosquitto.csr -key mosquitto.key

type the required information; the most important of which is the common name that will identify the server:

mq-ssl-001

Now sign the CSR file with your Certificate Authority (or send it to a public / corporate CA) to generate the certificate:

openssl ca -config openssl.cnf -extensions server_cert 
 -notext -in mosquitto.csr -out mosquitto.cer

mq-ssl-002

Create the ssl subfolder in the folder where you installed mosquitto and copy into that folder the certificate, its private key and the certificate of the CA:

mq-ssl-003

Configuration

Open the mosquitto.conf file and add the following lines:

mq-ssl-004

The first line changes the TCP port mosquitto is normally listening to (1883) to the default port for SSL connection, 8883.

The following 3 lines set the path for server and CA certificates and for the private key that corresponds to the server certificate. The last one, which is not compulsory, forces the use of the TLS v1.2 protocol, the most secure one at the moment of writing.

Once the server has been configured, you can start it (-v is to enable the verbose output):

mosquitto.exe -c mosquitto.conf -v

To be able to use the mosquitto_pub and mosquitto_sub tools, you now have to add new parameters:

mosquitto_sub.exe -p 8883 -t test --cafile .\ssl\ca.cer --insecure
mosquitto_pub.exe -p 8883 -m 20 -t test --cafile .\ssl\ca.cer --insecure

With -p you specify the TCP port of the server, with –cafile the path of the CA certificate which signed the server certificate mosquitto uses and finally with –insecure you configure the two clients not to verify that the certificate’s common name (in my example mymosquitto.local) corresponds to the server name.

You can avoid using the –insecure switch if you generate a certificate with a common name the exact name of the server/PC which runs mosquitto or – alternatively – if you add a DNS alias which resolves the common name of your certificate with the IP address of the server and run the clients with -h commonName

esp32

Tuan PM developed a library (espmqtt) for the esp-idf framework that implements a complete MQTT client. Moreover, the library does support secure connections, you can therefore use it to connect to an MQTT broker with TLS enabled.

mq-ssl-005

Copy the content of the Github repository in the components folder of your project and include the library’s header file in your source code:

#include "mqtt.h"

The MQTT client is configured using the mqtt_settings struct:

mqtt-001

The most important parameters are:

  • the server (host) that runs the MQTT broker (you can use the IP address or the DNS name)
  • the TCP port (port) the server is listening to (default is 1883 or 8883 if SSL is enabled)
  • username and password if the server requires authentication
  • one or more callback functions the espmqtt library will call when the corresponding event occurs
the espmqtt library does not copy the parameters in an internal struct. It’s therefore very important that the mqtt_settings variable has a global scope and it’s defined outside a specific function.

Your program can interact with the MQTT client implementing its callback functions.

For example the connection or disconnection from the MQTT server occurs as follows:

mqtt-002

The connect_cb and disconnect_cb functions perform the “real” connection and disconnection, while connected_cb and disconnected_cb functions are executed after the corresponding activity is completed (= the client successfully connected to the server). Your program usually doesn’t need to re-implement the main functions, but will implement the ones related to events, to execute actions (for example subscribe a topic) when a specific event occurs.

After having configured the client, you can run it with:

mqtt_start(&settings);

Once connected to the server (connected_cb callback function) you can subscribe or unsubscribe a topic with:

void mqtt_subscribe(mqtt_client *client, const char *topic, uint8_t qos);
void mqtt_unsubscribe(mqtt_client *client, const char *topic);

and publish data to a topic with:

void mqtt_publish(mqtt_client* client, const char *topic, 
  const char *data, int len, int qos, int retain);

Demo

I prepared an example to show my esp32 devboard sending data to a mosquitto server, with SSL enabled.

I connected to the devboard an HTU21D sensor as explained in a previous article and my program reads, every 5 seconds, the temperature and humidity values and sends them to the broker. I used a very handy opensource program, HelloIoT, to create a dashboard and display the received data.

The source code of the program and the configuration of the HelloIoT dashboard are available in my Github repository; here’s a short video of the demo:

Java, IllegalFormatConversionException when importing an SSL certificate

Today I faced a strange problem. I needed to import an SSL certificate in a java keystore, using the “classic” keytool command. I was sure that the format of the certificate was correct, but I always received the following error:

java.util.IllegalFormatConversionException: d != java.lang.String

java-keystore01

The reason of the error is that – in the latest versions of java – by default the keytool command uses the language of the system (in my case italian).

I had to add the parameter -J-Duser.language=en to the keytool command to made it work correctly:

java-keystore02

DCC, led accessory decoder

After having designed a shield to interface Arduino with a DCC bus, today I’ll show you how to realize a simple accessory decoder to control a couple of leds.

dccled-101 dccled-100

Accessory decoder

The DCC standard (specifically the document S-9.2.1 DCC Extended Packet Formats) defines different types of decoders, which are devices – connected to the DCC bus – that perform actions based on the command they receive.

An accessory decoder is “intended for control of a number of simple functions such as switch machine control or turning on and off lights“.

dcc2servo2relay

a DIY decoder by Paco

Addressing

Each decoder is assigned one or more addresses. The document S-9.2.1 linked above explains in details the format of a DCC packet sent to an accessory decoder (basic accessory decoder packet format):

dcc-001

9 bits of the packet (AAAAAAAAA) defines the address of the decoder. The address’ most significative bits (MSB) are the ones in the second bytes, while the least significative bits (LSB) are the ones in the first byte. To make things more complicated, the bits in the second byte are in ones’ complement representation.

Let’s understand it with an example:

dcc-003a

Most significative bits are 011. If you ones’ complement them, they become 100.

Least significative bits are 0010011. When you concatenate the two parts, you get 1000010011, therefore the decoder’s address is 267.

2 bit (PP) represents the address of a single port of the decoder. Conventionally, an accessory decoder has 4 ports, each one with a couple of outputs.

Bit O chooses, for the port represented by the PP bits, the output the command is related to. Finally, bit C is the action and defines if that output must be activated (1) or disactivated (0).

dcc-002

Some manufacturers prefert to use a “flat” addressing (not distinguishing between decoder address, port number and output). A good explaination about the different addressing methods (MADAPADA…) is available on the Rocrail wiki.

Arduino

Before developing the sketch for your accessory decoder, you have to install – using the Library Manager – the NmraDcc library in your Arduino IDE:

dcc-acc-001

Then include it in your sketch and create an instance of the NmraDcc object:

#include "NmraDcc.h"
NmraDcc Dcc;

In the setup() you have to configure and initialize the library. First, call the pin() method to tell the library which Arduino pin the DCC signal is connected to (ExtIntPinNum) and the corresponding interrupt number (ExtIntNum). In addition, you can enable or not the internal pullup resistor:

void pin(uint8_t ExtIntNum, uint8_t ExtIntPinNum, uint8_t EnablePullup);

The list of suitable pins for each Arduino board is available in the wiki and you can use the digitalPinToInterrupt() method to get the interrupt number that correspond to each pin.

If you’re using the shield I designed with an Arduino Uno, you can choose pin 2 with the pullup resistor enabled:

#define DCC_PIN 2
[...]
Dcc.pin(digitalPinToInterrupt(DCC_PIN), DCC_PIN, 1);

Let’s now configure the library, with the method:

void init(uint8_t ManufacturerId, uint8_t VersionId, uint8_t Flags, 
  uint8_t OpsModeAddressBaseCV);

The first two parameters are the manufacturer code and the decoder version; the DCC command station can get those values reading the corresponding CVs (configuration variables). For self-made decoders, the ManufacturedId is set to MAN_ID_DIY.

With the third parameter you can set, in OR, some flags:

dcc-acc-002

For accessory decoder, you have to specify at least the third flag (DCC_ACCESSORY_DECODER).

FLAG_MY_ADDRESS_ONLY

The first flag tells the library to filter the DCC messages it receives and to process only the ones sent to the decoder address. I usually do not set that flag, because I want my sketch to be able to “see” each incoming message. In my code I then decide which messages to process and which not.

When the library is correctly configured, in the loop() you have to call as frequently as possible the process() method to let the library handle the incoming messages:

void loop() {
  Dcc.process();
}

When the library receives a message for an accessory decoder, it can execute different callback functions, depending on what you defined in your sketch:

  • notifyDccAccState()
  • notifyDccAccTurnoutBoard()
  • notifyDccAccTurnoutOutput()
  • notifyDccSigState()

The first 3 methods are called for basic accessory decoder packets, while the last for extended accessory decoder control packets.

For my led decoder, I decided to implement the notifyDccAccState() method:

void notifyDccAccState(uint16_t Addr, uint16_t BoardAddr, 
  uint8_t OutputAddr, uint8_t State)

The method provides some variables. Let’s see their meaning (remember what you learned in the first part of this article about addressing):

  • BoardAddr is the decoder address (9 bits)
  • OutputAddr contains the 3 bits PP and O, that are the address for the port and the output
  • State is the action (1 = activate, 0 = deactivate)
  • Addr is the direct port address (PADA mode)

You can divide the port address (2 bits) from the output address (1 bit) with:

int portAddress = (OutputAddr >> 1) + 1;
int outAddress = OutputAddr & 0x01;

In the code I added 1 to the port address to be able to number the ports from 1 to 4 instead of from 0 to 3.

To keep it simple, the decoder developed for this article has hardcoded in the sketch both the board address and the port address (in a future article I’ll explain how to have the address configurable at runtime):

#define BOARD_ADDRESS   5
#define PORT_ADDRESS    1

In the sketch you can verify that the command received is for your decoder with:

if((BoardAddr == BOARD_ADDRESS) &amp;&amp; (portAddress == PORT_ADDRESS)) {

and change the output status of two digital pins (to which I connected the leds) based on the output number and the command:

if(outAddress == 0) {
  digitalWrite(RED_LED_PIN, HIGH);
  digitalWrite(GREEN_LED_PIN, LOW);
   Serial.println("! RED led activated");
} else {
  digitalWrite(RED_LED_PIN, LOW);
  digitalWrite(GREEN_LED_PIN, HIGH);
  Serial.println("! GREEN led activated");      
    }

The source code of the decoder is available in my Github repository, here’s a video that shows how it works: