ESP32 (31) – BLE, GAP

In my previous tutorials you learned how to use the wifi interface of the esp32 chip. Starting from this post, I’m going to explain you the second wireless technology the esp32 chip supports: bluetooth.

In particular, my tutorial will be about Bluetooth Low Energy (BLE), sometimes called also Bluetooth 4.0 or Bluetooth Smart:

ble-001

Bluetooth Low Energy

BLE is a technology to build personal wireless area networks (WPAN); that is it allows to put in communication different devices (computers, smartphones, smartwatches…) “close” to each other (a theoretical maximum distance of 100m). As the name suggests, version 4.0 of the Bluetooth standard was designed to reduce the power consumption of the devices connected to the network.

Devices are divided into two families:

  • central
  • peripheral

the first ones (central) are devices like PCs, tablets or smartphones with good processing power and memory. The second ones (peripheral) are instead sensors, tags… with less hardware resources and power. A central device can be connected to more peripheral devices at the same time, while it’s not true the opposite:

ble-002

BLE devices periodically report their presence by transmitting advertising packets. The advertising packet can contain up to 31 bytes of data and the transmission frequency can be chosen by the single device: reducing this frequency can indeed reduce energy consumption.

If a BLE device, after having received an avertising package, wants to obtain more information from the device that transmitted it, it can request a second packet of information (always for a maximum of 31 bytes), the scan response packet. The transmission of this second data package is optional:

ble-003

A BLE device can take advantage of advertising packages to send data in broadcast mode. In this case, this device is called a broadcaster, while the devices that receive the data are called observers.

What explained above is defined within a BLE specification called Generic Access Profile (GAP).

esp32

In this first tutorial you’ll learn how to develop a program that will periodically scan the air looking for BLE devices, that is a program which receives advertising packets and displays the data received in the serial console.

Before compiling a program which uses the Bluetooth controller, make sure (using menuconfig) that the controller is enabled (Component config -> Bluetooth):

ble-004

Start your program with the required header files:

#include "esp_bt.h"
#include "esp_bt_main.h"
#include "esp_gap_ble_api.h"

You also need to initialize the NVS partition, used by the Bluetooth driver:

ESP_ERROR_CHECK(nvs_flash_init());

the Bluetooth controller of the esp32 chip supports both the classic and the low energy mode. If one of this two modes is not required in your program, you can release the memory the framework normally allocates to manage it using the esp_bt_controller_mem_release() command. In this example you’re not going to use the classic mode, so:

ESP_ERROR_CHECK(esp_bt_controller_mem_release(ESP_BT_MODE_CLASSIC_BT));

Now you can configure (using the default settings) the controller in BLE mode:

esp_bt_controller_config_t bt_cfg = BT_CONTROLLER_INIT_CONFIG_DEFAULT();
esp_bt_controller_init(&bt_cfg);
esp_bt_controller_enable(ESP_BT_MODE_BLE);

The esp-idf framework esp-idf includes the Bluedroid bluetooth stack. This library was developed by Broadcom and used by Android since version 4.2 Bluedroid is initialized and enabled with the following commands:

esp_bluedroid_init();
esp_bluedroid_enable();

Now you’re ready to start scanning…

GAP, events

In a similar way to what you learned about the wifi driver, the bluetooth driver also runs in a thread separate from our program and communicates with it via events. In order to receive such events, you have to implement a callback function. Whenever the bluetooth driver has to notify an event, it will call that function.

The prototype of the callback function is:

static void esp_gap_cb(esp_gap_ble_cb_event_t event, esp_ble_gap_cb_param_t *param);

You tell the driver which callback function has to use with the esp_ble_gap_register_callback() method:

ESP_ERROR_CHECK(esp_ble_gap_register_callback(esp_gap_cb));

The Bluetooth driver handles several events, there are the ones related to the scan process:

ble-005

Before being able to start the scan process, you have to configure the scan parameters. The configuration is performed using the esp_ble_scan_params_t struct. It’s very important that the variable with the scan parameters is available during all the scan process; it’s therefore necessary to define it globally:

static esp_ble_scan_params_t ble_scan_params = {
  .scan_type              = BLE_SCAN_TYPE_ACTIVE,
  .own_addr_type          = BLE_ADDR_TYPE_PUBLIC,
  .scan_filter_policy     = BLE_SCAN_FILTER_ALLOW_ALL,
  .scan_interval          = 0x50,
  .scan_window            = 0x30
};

With the esp_ble_gap_set_scan_params() method you configure the scan process passing the struct defined above to the driver:

esp_ble_gap_set_scan_params(&ble_scan_params);

When the driver has finished the configuration, it calls the callback function with the event ESP_GAP_BLE_SCAN_PARAM_SET_COMPLETE_EVT. Depending on the event raised, the callback function also receives some parameters. The framework’s Programming Guide explains – for each event – the related parameters. For this event, it’s available the variable scan_param_cmpl that contains only the status parameter.

In the callback function you can use the switch statement to identify each event:

switch (event) {
  case ESP_GAP_BLE_SCAN_PARAM_SET_COMPLETE_EVT:
  [...]
  break;

and check if the configuration was successful with:

if(param->scan_param_cmpl.status == ESP_BT_STATUS_SUCCESS)

If so, you can start the scan process:

esp_ble_gap_start_scanning(10);

The parameter is the scan duration (in seconds).

Once the scan process has started, the driver raises the ESP_GAP_BLE_SCAN_START_COMPLETE_EVT event. For this event too it’s possible to verify the correct execution by reading the status parameter (pay attention: the name of the variable which contains the parameter changes!):

case ESP_GAP_BLE_SCAN_START_COMPLETE_EVT:
  if(param->scan_start_cmpl.status == ESP_BT_STATUS_SUCCESS)
    printf("Scan started\n\n");
  else 
    printf("Unable to start scan process");
  break;

GAP, scan process

During the scan process, for each advertising packet the chip receives the event ESP_GAP_BLE_SCAN_RESULT_EVT is raised.

This event contains some subevents. You can identify which subevent was raised reading the scan_rst.search_evt parameters. Two subevents are in particular interesting:

ble-006

the first tells you that a device was detected, while the second one that the scan process completed.

For each detected device, various information is available. For now let’s print its address in the console:

case ESP_GAP_BLE_SCAN_RESULT_EVT:
  if(param->scan_rst.search_evt == ESP_GAP_SEARCH_INQ_RES_EVT) {
    printf("Device found: ADDR=");
    for(int i = 0; i < ESP_BD_ADDR_LEN; i++) {
      printf("%02X", param->scan_rst.bda[i]);
      if(i != ESP_BD_ADDR_LEN -1) printf(":");
    }

The address is an uint8_t array, whose size is defined by the ESP_BD_ADDR_LEN constant. The address is normally displayed in hex form, with the bytes separated by :

ble-007

Device list management

As explained above, the ESP_GAP_BLE_SCAN_RESULT_EVT event is raised everytime a device sends an advertising packet. This means that a single device will be detected multiple times during the scan process.

It’s therefore necessary to maintain a list of the known devices. In my Github repository you can find a test program that scans the network and prints all the detected devices.

You can verify if it works correctly comparing what the program detects with the BLE devices listed by a smartphone… for Android for example you can use the very good nRF Connect application by Nordic.

Here’s what my program detected:

ble-008

and here’s the nRF Connect’s screenshot:

ble-009

 

ESP32 (30) – HTTP server in SoftAP mode

One of the most frequent questions I receive from my website’s contact form or from my Facebook page is whether it’s possible to publish an HTTP server when the esp32 chip is working in SoftAP mode, that is when it publishes its own wifi network.

In previous tutorials (18 – Access Point and 20 – Webserver) I’ve already blogged about the two subjects separately, today I’ll explain how to combine them.

Access Point

Let’s start defining the parameters of the TCP/IP network will be published by the esp32 chip. You have to choose an addressing plan, that is which IP addresses will belong to the network. You can use some of the IP addresses IANA assigned to private networks (RFC1918):

  • 10.0.0.0/8
  • 172.16.0.0/12
  • 192.168.0.0/16

The numbers /8/12/16 are, in short form, the network mask. For example/8 means that the network 10.0.0.0/8 contains all the addresses from 10.0.0.1 to 10.255.255.254, a total of 16.777.216 different addresses. Thanks to the subnetting you can slice a network in smaller subnetworks.

In this example I’m going to use the network 192.168.1.0 with netmask 255.255.255.0 (/24), that is a network with 254 usable addresses (192.168.1.1 – 192.168.1.254). You have to statically assign to the esp32 chip an address that belongs to that network; to make things easy I chose the first one (192.168.1.1):

tcpip_adapter_ip_info_t info;
memset(&info, 0, sizeof(info));
IP4_ADDR(&info.ip, 192, 168, 1, 1);
IP4_ADDR(&info.gw, 192, 168, 1, 1);
IP4_ADDR(&info.netmask, 255, 255, 255, 0);
ESP_ERROR_CHECK(tcpip_adapter_set_ip_info(TCPIP_ADAPTER_IF_AP, &info));

The other devices which will connect to the network will dynamically configure their IP address, thanks to the DHCP service. To use the DHCP server in your program, first you have to stop it before configuring the network settings and then start it:

ESP_ERROR_CHECK(tcpip_adapter_dhcps_stop(TCPIP_ADAPTER_IF_AP));
[...]
ESP_ERROR_CHECK(tcpip_adapter_dhcps_start(TCPIP_ADAPTER_IF_AP));

After having configured the TCP/IP network and the related services, you can set up the wifi interface of the esp32 chip and activate the SoftAP mode:

wifi_init_config_t wifi_init_config = WIFI_INIT_CONFIG_DEFAULT();
ESP_ERROR_CHECK(esp_wifi_init(&wifi_init_config));
ESP_ERROR_CHECK(esp_wifi_set_storage(WIFI_STORAGE_RAM));
ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_AP));

You have to prepare a wifi_config_t struct that contains all the parameters for the wifi network (SSID, authentication…) and pass it to the method:

ESP_ERROR_CHECK(esp_wifi_set_config(WIFI_IF_AP, &ap_config));

You can hardcode the parameters in the program, or make them configurable via menuconfig (as in this example) or Possiamo decidere di inserire i parametri hardcoded nel programma, di renderli configurabili tramite menuconfig (come nell’esempio) or even read them from a config file or from NVS variables.

Finally start the wifi interface with:

ESP_ERROR_CHECK(esp_wifi_start());

HTTP server

Now you can use the Netconn API of the lwip library to bind your program to the TCP/80 port (it’s the standard port used by the HTTP protocol) and to listen for incoming connections:

struct netconn *conn;
conn = netconn_new(NETCONN_TCP);
netconn_bind(conn, NULL, 80);
netconn_listen(conn);

The accept() method blocks the program until a new connection is accepted:

struct netconn *newconn;
netconn_accept(conn, &amp;newconn);

After having established the connection, your program must speak the same language (protocol) of the client; in this case the HTTP protocol used by web browsers.

To keep the example simple, it will publish a static website, whose content is stored in an SPIFFS partition (see the ESP32lights project for more information). Therefore, the program will answer requests in the form GET <resource> looking for the resource in the SPIFFS partition and, if found, sending its content using the netconn_write() method:

http-ap-001

Bonus, mDNS

To be able to connect to the HTTP server published by the esp32 chip, a client must know the IP address (192.168.1.1) assigned to the chip.

We can leverage the mDNS service (as explained here) to allow the client to connect using an alias (for example esp32web):

mdns_server_t* mDNS = NULL;
ESP_ERROR_CHECK(mdns_init(TCPIP_ADAPTER_IF_AP, &mDNS));
ESP_ERROR_CHECK(mdns_set_hostname(mDNS, "esp32web"));
ESP_ERROR_CHECK(mdns_set_instance(mDNS, "esp32 webserver"));

If the device you’re using supports the mDNS service, it will be possible to connect to the website using the address esp32web.local:

http-ap-002

Demo

The complete program is available in my Github repository. After having loaded it on your devboard, a new wifi network will be available:

http-ap-003

if you connect to that network, your device will be assigned an IP addess on the 192.168.1.0/24 network:

http-ap-004

and you’ll be able to point the browser to http://esp32web.local (or http://192.168.1.1) and display the website:

http-ap-005

BLE with Bluno Beetle

Some time ago my friend Mauro Alfieri showed me an interesting development board produced by DFRobot and called Bluno Beetle (now Beetle Ble). It seemed the perfect board to start “playing” with the Bluetooth Low Energy (BLE) technology; therefore I ordered  one board directly from the DFRobot store.

I expected to receive the usual anonymous parcel with the board inside an antistatic plastic bag; DFRobot instead sends its products in an elegant cardboard box, protecting them with foam:

bluno-011 bluno-012

Bluno Beetle is really small and therefore perfect for wearable projects:

bluno-013 bluno-014

But what is it? Simplifying is a board, Arduino Uno compatible (it hosts the ATmega328P microcontroller) to which has been added the CC2540 chip from Texas Instruments to act as USB and BLE controller. The two chips communicate via a serial interface:

bluno-001

The CC2540 chip is actually a real microcontroller that runs a firmware developed by DFRobot. This firmware can be configured using AT commands. Normally the firmware runs in transparent mode, that is it acts as a “bridge” between the USB/BLE interfaces and the ATmega microcontroller. If you then connect the Bluno to your PC and activate the serial monitor, each character you type is forwarded to the ATmega and viceversa.

To send AT commands, first you have to enter the AT mode of the firmware, sending the + character 3 times (without appending a line ending). The firmware confirms the new mode with the sentence “Enter AT Mode”:

bluno-006

Now you can send the commands, appending the Windows line terminator (CRLF). For example to display the firmware version:

bluno-007

To exit the AT mode and go back to transparent mode, you have to send the AT+EXIT command.

Drivers

It may happen that – if it’s the first time you connect the Bluno Beetle to your Windows PC – it is not correctly recognized:

bluno-003

The correct drivers are shipped with the Arduino IDE. You only need to do a manual installation specifying the path where you installed the IDE:

bluno-004

Windows will identify the new device as an Arduino Uno:

bluno-005

Arduino

As explained above, if the firmware running on the CC2540 chip is in transparent mode, using the USB connection you can talk directly to the ATmega328P microcontroller. This means that you can program the microcontroller using the Arduino IDE without any problems… just choose Arduino Uno as board and select the correct serial port:

bluno-008

BLE and transparent mode

In transparent mode Bluno transmits via BLE each byte it receives from Arduino (the ATmega microcontroller) and – viceversa – it sends to Arduino each byte it receives from BLE.

In this first post let’s explore the demo application DFRobot provides; in a future post I’ll explain how to develop your application to interact with Bluno Beetle via BLE.

If you have an Android smartphone, you can directly install the apk file for the application named BlunoBasicDemo (application of which the source code is also available). In the same Github repository you can also find the source code of the iOS application, you have to compile by yourself.

Compile and upload the following sketch on the board:

unsigned long previous_time = 0;
 
void setup() {
  Serial.begin(115200);
}
 
void loop() {
 
  if (Serial.available() > 0) {
    int incomingByte = Serial.read();
    Serial.print("New byte received: 0x");
    Serial.println(incomingByte, HEX);
  }
  unsigned long actual_time = millis();
  if(actual_time - previous_time > 10000) {
    Serial.println("Hello world!");
    previous_time = actual_time;
  }
}

The sketch reads the incoming bytes (coming from the app) and sends back to the app their hexadecimal value. Every 10 seconds moreover the sketch sends to the app the text Hello world!.

Launch the app. After having clicked on the Scan button, you can choose your Bluno board from the list of detected devices:

bluno-009

Every 10 seconds you should see a new HelloWorld! string appear. You can try to send a character (for example the letter “a”); you’ll receive an answer from Arduino (0x61 is indeed the hex code – in the ASCII table – for the letter “a”):

bluno-010

HID mode

Bluno also supports the HID (Human Interface Device) mode. When running in this mode, Bluno simulates an input peripheral (keyboard, mouse…) connected via BLE.

The AT command that enables this mode is:

  • AT+FSM=FSM_HID_USB_COM_BLE_AT

After having enabled the HID mode, you can send one or more “keys” with:

  • AT+KEY=

you can send up to 3 different keys at a time, concatenating their codes with the + character. The codes to be used, according to the type (page) of the HID device, are listed in the USB specification.

The AT+KEY command notifies the pressure of a key on the keyboard.  It is therefore necessary, after a few moments, to send the AT+KEY=0 command to indicate that the key has been released; otherwise on the PC associated with Bluno you’ll see the character appear repeatedly!

Debug mode

Using two different AT commands you can enable the debug mode of the firmware. This mode allows to receive – via the USB connection – a copy of all the data sent and received through the BLE connection.

The two commands are:

  • AT+BLUNODEBUG=ON (copies the messages sent by the ATmega)
  • AT+USBDEBUG=ON (copies the messages received from BLE)

bluno-002

By default the first debug mode is active, while the second is disabled. You can verify it if you upload the sketch listed above: in the serial monitor you’ll see the Hello World! sentences but not the characters sent by the app.

ESP32 (29) – Deep sleep

One of the major concerns for embedded devices is the power consumption. If the device you’re designing will be battery powered, it’s indeed important to reduce as much as possible its power consumption  to maximize the autonomy (= the working time before it’s necessary to replace or recharge the battery).

The esp32 chip offers 5 different power modes. The “normal” mode is named active mode; in this mode all the features of the chip are available. By starting to reduce the CPU speed and disabling some peripherals and cores, the chip switches to different power saving modes, as summarized in the following diagram:

sleep-001

In this first post about the esp32 power saving modes, I’ll explain the deep sleep mode.

Deep sleep

The esp-idf framework actually supports two power saving modes: light sleep and deep sleep. Between the two, the deep sleep mode is the one which offers greater energy savings. In this mode, are turned off:

  • both the CPUs
  • most of the RAM memory
  • all the peripherals

by default are instead kept active:

  • the RTC controller
  • the RTC peripherals, including the ULP coprocessor
  • the RTC memories (slowfast)

You can put the chip in deep sleep with the esp_deep_sleep_start() method, while it’s possible to wake up it via different events:

sleep-002

When the chip wakes up from deep sleep, a new boot sequence is performed. It’s therefore very important to understand that the execution of your program does not restart at the point where the esp_deep_sleep_start() method is called.

Let’s see how to configure and use two wake up events; in a future post I’ll write about touch pad and ULP.

Timer

The simplest wake up event is for sure the one which leverages a timer of the RTC controller. Thanks to the method:

esp_err_t esp_sleep_enable_timer_wakeup(uint64_t time_in_us)

you can wake up the esp32 chip after the specified number of milliseconds. The method must be called before entering the deep sleep mode:

// wakeup after 10 seconds
esp_sleep_enable_timer_wakeup(10000000);
esp_deep_sleep_start();

I/O triggers

In a previous post I’ve already blogged about the possibility to receive interrupts when a digital pin of the chip changes its status. We can leverage a similar functionality to wake up the chip from sleep.

With the method

esp_err_t esp_sleep_enable_ext0_wakeup(gpio_num_t gpio_num, int level)

you can enable the wake up if the specified pin (gpio_num) changes its status (level).

You can only use the pins with RTC function (0, 2, 4, 12-15, 25-27, 32-39) and the possible levels are 0 (= low) or 1 (high). If, for example, you want to wake up the chip from deep sleep if pin 4 has a low level, you’ll write:

esp_sleep_enable_ext0_wakeup(4, 0);

The framework also offers a method to monitor different pins:

esp_err_t esp_sleep_enable_ext1_wakeup(uint64_t mask, esp_sleep_ext1_wakeup_mode_t mode)

the pins (this method also accepts only the ones specified above) must be specified in a bitmask and  the wakeup modes are:

  • ESP_EXT1_WAKEUP_ALL_LOW = wakeup  when all the pins are low
  • ESP_EXT1_WAKEUP_ANY_HIGH = wakeup when at least one pin is high
When the chip wakes up from the sleep, the pins specified will be configured as RTC IO. To be able to use them again as normal digital pins, you have first to call the rtc_gpio_deinit(gpio_num) method. The ext0_wakeup method at the moment cannot be used together with touch pad or ULP events.

After the wake up…

If you configured more than one wake up event, you can know which specific event woke up the chip with:

esp_sleep_wakeup_cause_t esp_sleep_get_wakeup_cause()

The possible constants are:

sleep-003

For the ext1_wakeup event, a specific method is available to get the bitmask of the pins:

uint64_t esp_sleep_get_ext1_wakeup_status()

Memory

As explained above, in deep sleep mode the content of the RTC fast and RTC slow memories is preserved. You can therefore use those memory segments to store data that must be retained during the sleep.

To ask the compiler to store a variable in the RTC slow memory, you can use the RTC_DATA_ATTR attribute, or the RTC_RODATA_ATTR one if the variable is read only:

RTC_DATA_ATTR static time_t last;

Demo

I wrote a program (its source code is in my Github repository) to demonstrate the deep sleep mode and two different wake up events:

ESP32lights

Today’s project, ESP32lights, is a smart device based on the esp32 chip.

esp32lights-004 esp32lights-005

Thanks to ESP32lights you can turn a load on and off (I used it for my christmas lights)

  • manually
  • based on daily schedules
  • based on the light intensity

ESP32lights connects to your wifi network, can be configured and operated via a web browser and it’s optimized for mobile devices (responsive web interface based on jQuery Mobile).

Components

The heart of ESP32lights is the Lolin32 Lite devboard by Wemos. One of its digital pins is connected to a relay module, which controls the load. Two digital pins are assigned to the first i2c controller of the esp32 chip and are connected to a BH1750 light intensity sensor. All the elements are powered by an HLK-PM01 module by Hi-Link, which directly converts the mains’ 220V AC to 5V DC without the need of any external components:

esp32l-001

All the components are placed in a waterproof enclosure, to be able to use the device outdoor:

esp32lights-006 esp32lights-007

Programming

The firmware for the esp32 devboard is available in my Github repository.

In a following paragraph I’ll explain how it works. If you just want to build the device, you can program the firmware as it follows:

1) clone my repository in a local folder of your PC (you also have to install the development env esp-idf):

2) configure the correct settings for your wifi network and your timezone via menuconfig:

esp32lights-003

3) compile and flash the firmware:

make flash

4) store the image of the SPIFFS partition into the flash memory (replace the COM port of your devboard and the path where you saved the img file):

python $IDF_PATH/components/esptool_py/esptool/esptool.py --chip esp32 --port COM15
 --baud 115200 write_flash --flash_size detect 0x180000 /home/esp32lights.img

If everything is ok, when you connect to the serial console of the devboard (make monitor) you should see the following output:

esp32lights-025

Use

ESP32light publishes an HTTP interface you can use to set the schedules or the light intensity threshold or to manually turn the load on and off.

You can open the web interface connecting – through a PC or a smartphone -to the address http://<esp_ip> (the IP address of the board is displayed in the serial output as shown in the previous paragraph).

The interface has 3 tabs, one for each working mode:

esp32lights-001

The page footer displays the actual working mode and the relay status:

esp32lights-002

In this short video, you can see how the device works:

Software

I developed the firmware for ESP32lights leveraging what I explained in my previous tutorials about the esp32 chip. If you follow my blog, you probably understood that I really like the divide et impera method, that is divide a complex project into small, simpler tasks.

All the configuration settings of ESP32lights (actual working mode, start and stop time…) are stored in the NVS partition of the flash memory, as I explained in this tutorial.  In this way, it’s possible to keep them even if the chip is restarted:

nvs_handle my_handle;
int working_mode;
[...]
esp_err_t err = nvs_flash_init();
err = nvs_open("storage", NVS_READWRITE, &my_handle);
err = nvs_get_i32(my_handle, "mode", &working_mode);

The different elements of the web interface (html page, css style sheets…) are stored in an SPIFFS partition. In a previous tutorial you learned how to prepare the image and, in your program, get its content:

esp32l-002

In other tutorials I’ve also already explained you how to connect to a wifi network and how to use digital pins.

The setup phase is completed after having configured the BH1750 light intensity sensor. This sensor offers an i2c interface and therefore can be connected to one of the two i2c controllers of the esp32 chip as shown in this tutorial. In my program I included a driver developed by pcbreflux.

The main program runs two different tasks:

xTaskCreate(&http_server, "http_server", 20000, NULL, 5, NULL);
xTaskCreate(&monitoring_task, "monitoring_task", 2048, NULL, 5, NULL);

esp32l-003

The first one publishes the web interface, while the second one verifies – every second – if conditions exist (time or light intensity) to turn the load on or off:

if(working_mode == MODE_LIGHT && lux_valid) {
  int actual_light_value = get_light_value();
  if(actual_light_value < lux) {
    if(relay_status == false) {
      gpio_set_level(CONFIG_RELAY_PIN, 1);
      relay_status = true;
    }

Here’s in details how the http server fetches a static resource, stored in the SPIFFS partition.

First it adds to the resource path the root prefix for the SPIFFS partition (/spiffs):

sprintf(full_path, "/spiffs%s", resource);

then it checks if the resource exists in the partition:

if (stat(full_path, &st) == 0) {

if so, it opens the file in read mode:

FILE* f = fopen(full_path, "r");

and sends the content of the file to the client, reading blocks of 500 bytes:

char buffer[500];
while(fgets(buffer, 500, f)) {
  netconn_write(conn, buffer, strlen(buffer), NETCONN_NOCOPY);
}

Finally, this is how the web interface works. The interface is made by an html page (index.html) which uses jQuery to perform AJAX requests to the server and update the different page elements. You don’t need to enter the page name in the browser because of the http server automatically performs a redirect to it if the default page is requested:

if(strstr(request_line, "GET / "))
  spiffs_serve("/index.html", conn);

endpoints are published by the server and accessed using AJAX calls::

  • setConfig, to send a new configuration
  • getConfig, to read the actual configuration
  • getLight, to get the actual light intensity

When the page is loaded, it calls the getConfig endpoint to display the actual configuration; moreover it schedules every 5 seconds a call to the getLight endpoint to keep the light value updated:

refreshConfig();
setInterval("refreshLightLevel()", 5000);

When you click on the SET button, the page calls setConfig to send to the server the new configuration:

esp32l-004

All the information are sent using the JSON format. The esp-idf framework includes the cJSON library which makes it easy to create or parse a json message:

cJSON *root = cJSON_Parse(body);
cJSON *mode_item = cJSON_GetObjectItemCaseSensitive(root, "mode");
[...]
cJSON *root = cJSON_CreateObject();
cJSON_AddNumberToObject(root, "lux", light_value);
char *rendered = cJSON_Print(root);

Making of

I started the build of the device cutting a perfboard to the size of the enclosure:

esp32lights-008 esp32lights-009

The perfboard is screwed to the enclosure using two spacers:

esp32lights-010 esp32lights-011

I made two holes in one side of the enclosure for the main switch and for a status led:

esp32lights-012 esp32lights-013

I soldered all the different components on the perfboard and made the electric connections using wires:

esp32lights-014 esp32lights-015

To simplify the installation, all the external components (led, relay module…) are connected using jumpers:

esp32lights-016 esp32lights-017

First test:

esp32lights-018

I attached the light sensor to the top of the enclosure, after having made a hole to allow it to “see” the external light:

esp32lights-019 esp32lights-020

Finally I made the external connections, installing the main switch:

esp32lights-021 esp32lights-022

and connecting the output of the relay module to a wire with an universal plug at its end:

esp32lights-023 esp32lights-024

SIM800 GSM module

Today I received from  Banggood a GSM module based on the SIM800 chip by SIMCom. I’m going to use this module in a future Arduino project that will allow me to remotely control devices. In this article I’ll show you how to test the module.

The SIM800 is a quad-band (850/900/1800/1900MHz) chip and allows to trasmit/receive voice, SMS and data (using the GPRS network).

The small module (about 3x4cm) has the SIM800 chip and most of the components on one side, while on the other side you can find the sim card connector:

gsm-005 gsm-004

I chose to buy the module with an external antenna… in addition to the module you therefore receive the antenna and a short (about 20cm) cable to connect it:

gsm-006

The module is powered with 5V, offers a serial interface and you can control it using AT commands.

It is very important to connect the module to a power supply able to provide up to 2A, otherwise you may face problems on the stability of the GSM connection.

To verify if the module works you can therefore connect it to your PC using a simple USB – serial converter and send the correct AT commands as the manual explains. Be careful to invert the TX and RX pins: the TX pin of your adapter must be connected to the RX pin of the GSM module and viceversa:

gsm-008 gsm-007

The simplest test is to send and receive an SMS. For simplicity first configure the text mode (=1) with the command AT+CMGF=1. The module will answer with the command you send (without the starting AT) followed by OK.

Now you can specify the recipient of the message with the command AT+CMGS=”number. After having sent the “carriage return” character, the module will send the prompt, after which you can type the text of the message. End the message with the character 0x1A (CTRL-Z); the module will answer with the command you sent followed by the number of characters in the message (26):

gsm-001

When a new SMS is received, the module sends the +CMTI message. You can read the messages with the AT+CMGL command and you can also specify a filter (for example REC UNREAD, that is all the messages received but not read yet):

gsm-002

The filters available are listed in the manual:

gsm-009

After having verified that the module works, you can now realize a more complex project, for example interfacing it with your Arduino…

gsm-003

 

 

ESP32 (28) – MQTT and SSL

Security is a very important aspect for MQTT brokers. In a previous article you’ve already learned how to implement authentication and authorization. The weakness in that configuration was that credentials were transmitted in cleartext; it was therefore possible, for an attacker who can sniff the network traffic, to read and use them to impersonate a legitimate client.

Today I’ll show you how to encrypt the communication channel between client and broker using SSL certificates. I’ll also explain how to write a program for the esp32 chip to send data to the broker using the secure channel…

SSL certificate

To be able to encrypt the communication, mosquitto requires a server certificate.

First generate the private key (RSA with a length at least 2048 bits):

openssl genrsa -out mosquitto.key 2048

Then create the CSR file:

openssl req -new -out mosquitto.csr -key mosquitto.key

type the required information; the most important of which is the common name that will identify the server:

mq-ssl-001

Now sign the CSR file with your Certificate Authority (or send it to a public / corporate CA) to generate the certificate:

openssl ca -config openssl.cnf -extensions server_cert 
 -notext -in mosquitto.csr -out mosquitto.cer

mq-ssl-002

Create the ssl subfolder in the folder where you installed mosquitto and copy into that folder the certificate, its private key and the certificate of the CA:

mq-ssl-003

Configuration

Open the mosquitto.conf file and add the following lines:

mq-ssl-004

The first line changes the TCP port mosquitto is normally listening to (1883) to the default port for SSL connection, 8883.

The following 3 lines set the path for server and CA certificates and for the private key that corresponds to the server certificate. The last one, which is not compulsory, forces the use of the TLS v1.2 protocol, the most secure one at the moment of writing.

Once the server has been configured, you can start it (-v is to enable the verbose output):

mosquitto.exe -c mosquitto.conf -v

To be able to use the mosquitto_pub and mosquitto_sub tools, you now have to add new parameters:

mosquitto_sub.exe -p 8883 -t test --cafile .\ssl\ca.cer --insecure
mosquitto_pub.exe -p 8883 -m 20 -t test --cafile .\ssl\ca.cer --insecure

With -p you specify the TCP port of the server, with –cafile the path of the CA certificate which signed the server certificate mosquitto uses and finally with –insecure you configure the two clients not to verify that the certificate’s common name (in my example mymosquitto.local) corresponds to the server name.

You can avoid using the –insecure switch if you generate a certificate with a common name the exact name of the server/PC which runs mosquitto or – alternatively – if you add a DNS alias which resolves the common name of your certificate with the IP address of the server and run the clients with -h commonName

esp32

Tuan PM developed a library (espmqtt) for the esp-idf framework that implements a complete MQTT client. Moreover, the library does support secure connections, you can therefore use it to connect to an MQTT broker with TLS enabled.

mq-ssl-005

Copy the content of the Github repository in the components folder of your project and include the library’s header file in your source code:

#include "mqtt.h"

The MQTT client is configured using the mqtt_settings struct:

mqtt-001

The most important parameters are:

  • the server (host) that runs the MQTT broker (you can use the IP address or the DNS name)
  • the TCP port (port) the server is listening to (default is 1883 or 8883 if SSL is enabled)
  • username and password if the server requires authentication
  • one or more callback functions the espmqtt library will call when the corresponding event occurs
the espmqtt library does not copy the parameters in an internal struct. It’s therefore very important that the mqtt_settings variable has a global scope and it’s defined outside a specific function.

Your program can interact with the MQTT client implementing its callback functions.

For example the connection or disconnection from the MQTT server occurs as follows:

mqtt-002

The connect_cb and disconnect_cb functions perform the “real” connection and disconnection, while connected_cb and disconnected_cb functions are executed after the corresponding activity is completed (= the client successfully connected to the server). Your program usually doesn’t need to re-implement the main functions, but will implement the ones related to events, to execute actions (for example subscribe a topic) when a specific event occurs.

After having configured the client, you can run it with:

mqtt_start(&settings);

Once connected to the server (connected_cb callback function) you can subscribe or unsubscribe a topic with:

void mqtt_subscribe(mqtt_client *client, const char *topic, uint8_t qos);
void mqtt_unsubscribe(mqtt_client *client, const char *topic);

and publish data to a topic with:

void mqtt_publish(mqtt_client* client, const char *topic, 
  const char *data, int len, int qos, int retain);

Demo

I prepared an example to show my esp32 devboard sending data to a mosquitto server, with SSL enabled.

I connected to the devboard an HTU21D sensor as explained in a previous article and my program reads, every 5 seconds, the temperature and humidity values and sends them to the broker. I used a very handy opensource program, HelloIoT, to create a dashboard and display the received data.

The source code of the program and the configuration of the HelloIoT dashboard are available in my Github repository; here’s a short video of the demo:

ESP32, the new Wemos Lolin32 LITE

On my blog I’ve already posted about the development board Wemos designed for the esp32 chip and that, on many webstores, you can find boards that “copy” the Lolin32 name but are instead clones of other opensource projects.

Few weeks ago Wemos published on its official site and make available on its store on Aliexpress a new revision of the board, now named Lolin32 LITE. Thanks to my friend Mauro Alfieri I was able to test a copy:

lolin-lite-001 lolin-lite-002

The main features of the board are the same as the old one:

  • esp32 chip
  • PH-2 (2mm) connector for a LiPo 1S battery
  • charging circuit via USB (max 500mA)

At a visual comparison it is immediately noticeable that the LITE is slightly smaller than the previous, being 5mm long compared to the 5.8mm of the Lolin32:

lolin-lite-003

The smaller length forced Wemos to reduce the number of available pins (19 instead of 26) and to abandon the use of the ESP-WROOM-32 module soldering the esp32 chip and flash memory directly onto the PCB, which also accomodates the antenna.

Speaking of components, for the LITE version Wemos decided to change the chip used to perform the USB-serial conversion and adopted the CH340G IC (which I also used in this project). Finally, the card in my possession uses the latest revision of the esp32 chip (labeled 302017):

lolin-lite-004

 

esp32, chip revisions

In the same way that software programs have bugs, even hardware chips may have problems due to errors in their design. Producers resolve these errors by producing a new chip revision and keeping updated a document with the known bugs for the different revisions and the possible workarounds.

Espressif manufactured two different revisions of the esp32 chip to date:

lolin-lite-005

If you want to know the revision of a chip, you can use the espefuse.py utility included in the esptool package:

lolin-lite-008

specify the serial port (-p) your board is connected to and the summary command.

Observing the values of CHIP_VERSION and CHIP_PACKAGE you can determine the revision of the chip, for example my Lolin32 board has a chip with revision 0:

lolin-lite-006

while the new Lolin32 LITE a chip with revision 1:

lolin-lite-007

ESP32 (27) – GPS

In today’s post I’ll show you how to interface the esp32 chip to a GPS receiver to receive the actual position, speed and more…

GPS receiver

On the Internet and in electronic stores you can find different GPS receivers… thanks to the spread of navigators, smartphones and multicopters you can now buy one for few euros. For this tutorial, I used a GPS receiver sold by Banggood and based on the u-blox NEO-M8 chip:

gps-010 gps-011

Almost all the GPS receivers offer a serial interface, you can therefore connect them to an esp32 chip using one of its UART controllers, as I explained in a previous article. Sometimes it’s not easy to identify the various pins and it may be necessary – as in my case – to open the plastic case that contains the chip to read the PCB silk screen:

gps-001 gps-002

From the photos above you can understand the pinout of my receiver:

  • GND (ground) -> black cable
  • VCC (power supply) -> red cable
  • TXD (transmit) -> green  cable
  • RXD (receive) -> yellow cable

I decided to use the UART1 controller of my esp32 chip, with pins 4 (TX) and 16 (RX). I therefore connected all the cables, paying attention to connect the TX pin of the receiver to the RX pin of the chip and viceversa:

gps-003

NMEA

GPS receivers send data following the NMEA 0183 standard (often called just NMEA).

NMEA is based on strings (sentences) composed by a maximum of 80 characters with an ending CRLF. Each string has the following format:

$PREFIX,data1,data2 ... dataN-1,datoN*CHECKSUM

The prefix (5 chars) defines the type of device (for GPS receiver is GP) and the type of sentence (the following 3 chars). Each sentence includes a checksum (XOR) that allows the receiver to validate the data received.

The NMEA standard defines many types of sentences and each manufacturer can add proprietary types to communicate specific data. A complete list of NMEA sentences is available at this website. Let’s analyze one GGA (Global Positioning System Fix Data) sentence:

$GPGGA,183619,3877.038,N,07702.110,W,1,08,0.9,545.4,M,46.9,M,,*47

The receiver is sending the actual position (latitude 38°77.038′ NORTH and longitude 77°02.110′ WEST), obtained at 18:36:19 thanks to 8 satellites. The fix (position) quality is GPS Fix (1).

To be able to get data from the GPS receiver your program must therefore “understand” the NMEA sentences it receives. I found a very good library, minmea, developed in C and able to parse the most important NMEA sentences.

You only need to copy the two source files (minmea.c e minmea.h) in a subfolder of the components folder of your project and to create a component.mk file as it follows:

gps-001

I had to add the highlighted flag because of the esp framework does not include the timegm() function and therefore, as explained also in the README of the library, you need to tell the compiler to use the more common mktime() instead.

Program

In my Github repository you can find the program I developed for this tutorial. Let’s analyse the most important pieces of code:

To use the minmea library, include its header file:

#include "minmea.h"

Read a line from the UART1 controller and pass it to the library to be parsed:

char *line = read_line(UART_NUM_1);
switch (minmea_sentence_id(line, false)) {
  case MINMEA_SENTENCE_RMC:
  [...]
  case MINMEA_SENTENCE_GGA:
  [...]
}

Finally verify if the new parameters are different from the ones already saved and, if so, print them on the screen:

float new_latitude = minmea_tocoord(&frame.latitude);
if((new_latitude != NAN) && (abs(new_latitude - latitude) > 0.001)) {
  latitude = new_latitude;
  printf("New latitude: %f\n", latitude);
}

I added a threshold value (0.001) to avoid printing a lot of small updates, due to disturbances or oscillations on received data.

Here’s the program running on my laptop:

gps-010

ESP32 (26) – UART

UART (Universal Asynchronous Receiver-Transmitter) is an hardware peripheral which allows serial, asynchronous communication with configurable data format and speed. The UART interface usually works at logic level: the electric signals are generated by an external circuit, following the standards of the communication bus you chose.

For example the classical “serial port” of many personal computers is based on the EIA RS-232 standard, which defines how – at the physical layer – signals are generated on the communication medium. There are dedicated chips (the most famous of which is surely the MAX232 by Maxim Integrated) to convert the logic levels of a UART peripheral to the physical signals of the EIA RS232 standard:

uart-001

The esp32 chip offers 3 UART controllers. These controllers are connected to the GPIO matrix; this allows to assign them most of the digital pins of the chip:

uart-002

The esp-idf framework includes a driver (uart.c) to simplify the configuration and the use of the controllers; to use it, include its header file in your program:

#include "driver/uart.h"

For the driver, the 3 controllers are named as follows:

uart-003

In this first post, I’ll explain the basic use of a controller; events and interrupts will be covered in a future article.

The first thing to do is to configure the controller using the uart_config_t struct:

uart_config_t uart_config = {
  .baud_rate = 115200,
  .data_bits = UART_DATA_8_BITS,
  .parity = UART_PARITY_DISABLE,
  .stop_bits = UART_STOP_BITS_1,
  .flow_ctrl = UART_HW_FLOWCTRL_DISABLE
};
  • baud_rate is the transmission speed
  • data_bits, and stop_bits are the number of bits for each “word” and the number of stop bits
  • parity defines if the controller must send the parity bit or not
  • flow_ctrl is the type of flow control (hardwaresoftware or disabled)

There are two additional parameters (rx_flow_ctrl_thresh and use_ref_tick) you can use to set the threshold for the RTS signal if hardware flow control is selected and to enable the REF_TICK signal as clock for the UART controller.

The constant values for data_bitsstop_bits… are declared in the uart.h file.

Often the parameters required to communicate with a serial device are expressed in a “condensed” format, for example if you read 9600,8N1 it means:

  • speed 9600 baud
  • “word” of 8 bits
  • No parity
  • 1 stop bit

Configure the controller with the method:

uart_param_config(uart_port_t uart_num, const uart_config_t *uart_config);

passing as parameters the number of the controller (uart_num) and the struct with the configuration previously defined (uart_config).

Conclude the configuration setting the pins the controller have to use for the different signals:

uart_set_pin(uart_port_t uart_num, int tx_io_num, int rx_io_num, 
  int rts_io_num, int cts_io_num);

You can use the UART_PIN_NO_CHANGE constant if that specific signal is not used or if you want to keep the default pin.

For example to map the controller to pins 4 and 5 without using the RTS and CTS signals:

uart_set_pin(UART_NUM_0, 4, 5, UART_PIN_NO_CHANGE, UART_PIN_NO_CHANGE);

Now you can install the driver with:

uart_driver_install(uart_port_t uart_num, int rx_buffer_size, int tx_buffer_size, 
  int queue_size, QueueHandle_t* uart_queue, int intr_alloc_flags);

In addition to the controller number, you have to pass the size for the receive and transmit buffers. The parameters about queue and interrupts will be discussed in a future article.

The two buffers must have a size bigger than the corresponding hardware buffers (128). Only for the transmit buffer you can specify size = 0; in this case, the driver will be blocking, that is the execution of the task will be halted until the transmission ends.

Let’s now learn how to send data. To better understand the differences between the available commands, you have to understand that there are two buffers: one hardware, included in the UART controller, and one software, implemented in the driver:

uart-004

The first command to send data – to be used only if the software trasmit buffer is disabled – is uart_tx_chars():

int uart_tx_chars(uart_port_t uart_num, const char* buffer, uint32_t len);

This command sends len bytes from buffer. As you’re not using a software buffer, it may happens that the command cannot send all the bytes because the hardware buffer is full; the uart_tx_chars method therefore returns the number of bytes actually sent.

To use the software buffer, the uart_write_bytes() is available:

int uart_write_bytes(uart_port_t uart_num, const char* src, size_t size);

This command copies size bytes from the src array to the driver’s buffer: the driver will take care of filling the hardware buffer of the controller until all the data is transmitted. The uart_write_bytes() also retuns the number of bytes actually copied in the tx buffer.

When receiving, you can use the uart_read_bytes() command:

int uart_read_bytes(uart_port_t uart_num, uint8_t* buf, uint32_t length, 
  TickType_t ticks_to_wait);

The command reads a maximum of length bytes from the receive buffer and copies them in the buf array. The command waits for data the number of specified ticks, then returns the number of bytes actually read.

You can know the number of bytes at each moment available in the receive buffer with:

uart_get_buffered_data_len(uart_port_t uart_num, size_t* size);

UART and stdio

The framework allows to use a UART controller as a peripheral for standard I/O. Unix streams stdin, stdout and stderr are indeed linked to RX and TX operations on that controller: you can therefore use standard C functions like printf()scanf()… to write and read from the UART controller.

Via menuconfig you can specify which controller to use, the parameters for the controller and you can also disable this feature at all:

uart-005

Demo

In the following video I’ll show how to work with UART controllers connecting one of them to a USB->serial converter. Both the converter and the esp32 devboard are connected to my laptop: in this way I can send data from the laptop to UART1 via the converter, read it and send it back to my laptop using the UART0 controller.