ESP32 (21) – Mutual authentication

After having published my post about how to implement a webserver on the esp32 chip, some readers correctly warned me that everyone, once connected to the wifi network, could control the relay and asked me how to limit access to the webpage.

A classic solution (that I also used in this previous tutorial) is to ask the user to enter a password. It’s very easy to implement, but has a negative aspect, that is you have to type the password everytime you need to switch the relay.

You can avoid asking the password everytime the user access the page if you implement a login page and keep the user logged in using a session cookie as I’ll explain in a future tutorial!

Today I’ll show you how to protect your website using a feature of the SSL/TLS protocol: the mutual authentication.

Mutual authentication

Everytime you connect to a website using the https protocol, the server which hosts the site sends to your browser the SSL certificate of the site. Thanks to this certificate, you can verify the identity of the server and establish a protected connection to it:

ssl-02

The TLS protocol also offers the possibility for the server to require a certificate from the client: this means a two-way authentication, that is server and client authenticates each other using SSL certificates:

ssl-01

With this authentication method, a user has only to install on his device the client certificate (and the corresponding private key) and choose which certificate to send to the server if more than one is present:

ssl-03

The authentication with SSL certificates is therefore based on the principle of something I own (for example, a key) and not of something I know (for example, a password).

Certificates

Generally speaking, SSL certificates can be:

  • self-signed
  • signed by a Certification Authority

The first ones are perfect for internal use or for testing, while the second ones, normally signed by trusted CAs, are widely used in production environments or on the Internet. Sometimes you probably received this warning message when surfing on the Internet:

ssl-04

that means that your browser doesn’t trust the certificate sent by the server because it’s self-signed or signed by a non trusted CA.

For this tutorial, I’m going to use OpenSSL as a certification authority to create the certificates we need. If you’re using the toolchain provided by Espressif, you don’t need to install anything else because OpenSSL is included.

First create a folder that will contains all the files required by OpenSSL to operate as a CA in your home directory:

cd 
mkdir myCA

Now move into that folder and create some empty files:

cd myCA
mkdir csr certs crl newcerts private
touch index.txt
echo 1000 > serial

sslca-01

Copy the certification authority‘s configuration file (openssl.cnf) from my Github repository into the myCA folder. Open the file and change the main path accordingly to your system:

sslca-02

Now you have to generate the private key and the certificate for the CA. Let’s start with the key (its length will be 2048bit, enough to ensure good security):

winpty openssl genrsa -aes256 -out private/ca.key 2048

(all the commands must start with winpty only if you’re using Windows!)

You’re prompted for a password; it’s very important to remember it because you’ll have to type it everytime you use the CA:

sslca-03

Now generate the CA certificate (with a duration of 3650 days, that is 10 years):

winpty openssl req -config openssl.cnf -key private/ca.key -new -x509 -days 3650 -sha256 -extensions ca_cert -out certs/ca.cer

You have to enter some informations… the only mandatory is the common name which is the name of the CA, that will be included in all the certificates the CA will sign. At the end of the process you can open the file (ca.cer) and verify name and duration:

sslca-04

Let’s now generate the server and client certificates. The steps are the same for both:

  • generate a new private key – openssl genrsa
  • generate the CSR (Certificate Signing Request) file – openssl req
  • sign the CSR file with the CA to obtain the final certificate – openssl ca

Server certificate:

winpty openssl genrsa -out private/esp-server.key 2048
winpty openssl req -config openssl.cnf -key private/esp-server.key -new -sha256 -out csr/esp-server.csr
winpty openssl ca -config openssl.cnf -extensions server_cert -days 365 -notext -md sha256 -in csr/esp-server.csr -out certs/esp-server.cer

Client certificate:

winpty openssl genrsa -out private/esp-user.key 2048
winpty openssl req -config openssl.cnf -key private/esp-user.key -new -sha256 -out csr/esp-user.csr
winpty openssl ca -config openssl.cnf -extensions usr_cert -days 365 -notext -md sha256 -in csr/esp-user.csr -out certs/esp-user.cer

As said before, the client device must own both the certificate and the private key. You can merge certificate and key in a single p12 (or pfx) file with the following command:

winpty openssl pkcs12 -export -out esp-user.pfx -inkey private/esp-user.key -in certs/esp-user.cer

The way you install the P12 file on your device depends on the specific operating system it runs: on Windows you only need to double-click the file and follow the import wizard.

SSL webserver

The source code for the program running on the esp32 devboard is available in my Github repository.

First you have to copy in the prioject folder the certificates (CA and server ones) and the private key for the server certificate and include them – as text data – in the program (I’ve already explained how to include binary files in my previous post):

esp-ssl01

To implement the SSL protocol I’m using the mbedTLS library, included in the esp framework.

First include all the required header files:

#include "mbedtls/platform.h"
#include "mbedtls/net.h"
#include "mbedtls/ssl.h"
#include "mbedtls/entropy.h"
#include "mbedtls/ctr_drbg.h"
#include "mbedtls/debug.h"
#include "mbedtls/error.h"

Before being able to accept SSL connections, you have to define and initialize all the variables you’re going to use:

// mbed TLS variables
mbedtls_ssl_config conf;
mbedtls_ssl_context ssl;
mbedtls_net_context listen_fd, client_fd;
mbedtls_entropy_context entropy;
mbedtls_ctr_drbg_context ctr_drbg;
mbedtls_x509_crt srvcert;
mbedtls_x509_crt cachain;
mbedtls_pk_context pkey;
 
[...]
 
// initialize mbedTLS components
mbedtls_net_init(&listen_fd);
mbedtls_net_init(&client_fd);
mbedtls_ssl_config_init(&conf);
mbedtls_ctr_drbg_init(&ctr_drbg);
mbedtls_entropy_init(&entropy);	
mbedtls_x509_crt_init(&srvcert);
mbedtls_x509_crt_init(&cachain);
mbedtls_pk_init(&pkey);

Now parse the certificates and the private key and store them in the variables:

mbedtls_x509_crt_parse(&cachain, ca_cer_start, ca_cer_end - ca_cer_start);
mbedtls_x509_crt_parse(&srvcert, espserver_cer_start, espserver_cer_end - espserver_cer_start);
mbedtls_pk_parse_key(&pkey, espserver_key_start, espserver_key_end - espserver_key_start, NULL, 0);

Tell the library which variables to use for CA chain and own certificate and require the mutual authentication (MBEDTLS_SSL_VERIFY_REQUIRED):

mbedtls_ssl_conf_ca_chain(&conf, &cachain, NULL);
mbedtls_ssl_conf_own_cert(&conf, &srvcert, &pkey);
mbedtls_ssl_conf_authmode(&conf, MBEDTLS_SSL_VERIFY_REQUIRED);

After having configured other elements of the library (the random number generator and the debug function) you can bind it to the standard TCP port for the https protocol (443) and accept new incoming connections:

mbedtls_net_bind(&listen_fd, NULL, "443", MBEDTLS_NET_PROTO_TCP);
mbedtls_net_accept(&listen_fd, &client_fd, NULL, 0, NULL);

After having accepted a connection, you only have to call the function:

mbedtls_ssl_handshake(&ssl);

during the handshake process the library sends to the client its own certificate and require the one from the client, verifying that this is signed by the specified CA: the mutual authentication is implemented!

After the handshake, the program is similar to the previous example, you only have to use mbedTLS specific send/receive functions (mbedtls_ssl_read e mbedtls_ssl_write).

Test

ESP32, Wemos or not to Wemos

Wemos is a chinese manufacturer of IoT products, well known for its D1 Mini family, that includes boards based on the ESP-8266EX chip and expansion shields.

Some weeks ago, Wemos announced a board based on the new ESP32 chip, named LOLIN32. This board hosts an ESP-WROOM-32 module and can be powered by a single cell (1S) LiPo battery. The board can also recharge the battery thanks to a circuit based on the TP4054 chip.

I brought a LOLIN32 board from Wemos’s official store on Aliexpress. The board is shipped inside an anti-static bag, with a label with your name:

lolin-01

The bag also include some pin headers:

lolin-02 lolin-03

Curiously I noticed that other Aliexpress stores sell boards named “lolin”. I brought one of them:

lolin-04 lolin-05

Although this board has the “wemos” logo printed on it, it actually looks like a clone of the D-duino-32 project by Travis Lin: unlike the LOLIN32 indeed this board does not include the LiPo battery connector and the charging circuit while it adds a 0.96″ OLED display.

Here’s a visual comparison between the two boards:

lolin-06 lolin-07

The advantage of purchasing an original Wemos card is the availability of the electrical schematic, some tutorials and a support forum… all that is missing for the clone board (how is the OLED display connected?). My suggestion is therefore that if you want to buy a Wemos board, you should get it from the original store.

If instead you are interested in a board with a OLED display, why not buy it directly from the designer’s tindie page, to support its development?

ESP32 (20) – Webserver

One of the most popular projects among the ones included in my tutorial about the enc28j60 chip is fore sure WebRelay. This project allows to control an output pin of Arduino using a simple web page, designed to be accessed also using your smartphone. Today I’m going to show you how to implement a similar project with the esp32 chip; it’s also the opportunity to teach how to write a TCP server, especially a web server.

Netconn API

As you know, the esp-idf framework uses the lwip library to manage network communication. This library offers different abstraction levels: the programmer can decide to work on the raw packets or to leverage pre-build components.

To develop my TCP server, I decided to use one of those pre-build components, the Netconn API.

Using Netconn API it’s very easy to implement a server and the required steps are described below:

webrelay-01

The netconn_new() method creates a new connection and returns a a pointer to struct netconn which represents the connection:

struct netconn *conn;
conn = netconn_new(NETCONN_TCP);

The parameter defines the connection type… the most common ones are NETCONN_TCP for a connection using the TCP protocol and NETCONN_UDP for a connection using the UDP protocol.

To use the connection in server mode, you have then to associate (bind) it to a specific port… for example a webserver normally listens on port 80 (443 if HTTPS):

netconn_bind(conn, NULL, 80);

The second parameter (NULL above) allows to bind the connection to a specific IP address and may be useful if the device has more than a network interface. If you pass NULL (or the equivalent IP_ADDR_ANY) you ask the library to bind the connection to any available interfaces.

Now you can start listening on that port with:

netconn_listen(conn);

Working with a client connection

With the netconn_accept() method, your program can accept a new incoming connection:

struct netconn *newconn;
netconn_accept(conn, &newconn);

The method returns a pointer to a new struct netconn that represents the connection established with the client. This method is blocking: the program is stopped until a client makes a connection request.

Once the connection is established, you can use the netconn_recv() and netconn_write() methods to receive or send data to the client:

netconn_recv(struct netconn* aNetConn, struct netbuf** aNetBuf );
netconn_write(struct netconn* aNetConn, const void* aData, size_t aSize, u8_t aApiFlags);

To optimize RAM memory usage, the netconn_recv() method handles data using an internal buffer (zero-copy mode). To access the received data you therefore have to:

  • declare a new variable as a pointer to struct netbuf
  • pass the pointer address as the second parameter of the recv method
  • use the netbuf_data() method to obtain a pointer to the data within the netbuffer structure and its length

webrelay-02

struct netbuf *inbuf;
char *buf;
u16_t buflen;
netconn_recv(conn, &inbuf);
netbuf_data(inbuf, (void**)&buf, &buflen);

Likewise, the netconn_write() method accepts, as last parameter, a flag to indicate whether or not to copy the buffer content before sending it. To save memory you can therefore, if you’re sure that the buffer won’t be modified by other theads, use NETCONN_NOCOPY as flag:

netconn_write(conn, outbuff, sizeof(outbuff), NETCONN_NOCOPY);

When the communication with the client is complete, you can close the connection and free the buffer:

netconn_close(conn);
netbuf_delete(inbuf);

HTTP server

What you learned so far can be applied to every TCP server. If you want to communicate with an Internet browser, you have to “talk” the same language, that is the HTTP protocol.

In the example program (available on Github) I implemented a very minimal version of that protocol. When you type an address in your browser (for example www.google.com), the browser connects to Google server and sends a request with the form:

GET <resource>
[...]

The request can have different fields, but the first line always contains the name of the requested resource (html page, image…). In particular if you access the homepage of the website the request will simply be GET /.

The website published by the esp32 chip to control the relay consists of only two pages:

  • off.html, displayed when the relay is off
  • on.html, displayed when the relay is on

Each page contains a label (“Relay is ON|OFF“) and an image. The image itself is a link to the other page and when you click on it, the relay status is also changed:

webrelay-03

The program identifies the resource to be sent analyzing the first line of the request with strstr():

char *first_line = strtok(buf, "\n");
if(strstr(first_line, "GET / ")) [...]
else if(strstr(first_line, "GET /on.html ")) [...]
else if(strstr(first_line, "GET /on.png ")) [...]

An HTTP server responds to the browser by first indicating the result of the request. If ok, the return code is 200:

HTTP/1.1 200 OK

then it returns the media type of the resource and finally it sends the resource file. In this example, the possible media types are:

  • text/html for HTML pages
  • image/png for images

Static content

A webserver it normally stores the resouces for the website it publishes on an external device (memory card, hard drive…). For simple projects, you may consider to include all the content inside your program.

For example in my code HTML pages and HTTP protocol messages are defined as static arrays:

const static char http_html_hdr[] = "HTTP/1.1 200 OK\nContent-type: text/html\n\n";
const static char http_png_hdr[] = "HTTP/1.1 200 OK\nContent-type: image/png\n\n";
const static char http_off_hml[] = "";

To be able to include also the images, I used a feature of the framework (embedding binary data). You can indeed specify in the component.mk file the binary resources to be included:

webrelay-04

In your program you can access the content of the embedded files using the following pointers:

extern const uint8_t on_png_start[] asm("_binary_on_png_start");
extern const uint8_t on_png_end[]   asm("_binary_on_png_end");
extern const uint8_t off_png_start[] asm("_binary_off_png_start");
extern const uint8_t off_png_end[]   asm("_binary_off_png_end");

The syntax is _binary_filename_start|end, replace “.” with “_” in the filename. Having pointers to both the start and the end of the resource content, it’s easy to send it with the netconn_write() method:

netconn_write(conn, on_png_start, on_png_end - on_png_start, NETCONN_NOCOPY);

Demo

ESP32 (19) – NVS

In today’s tutorial you’ll learn how to permanently store information, so that they are preserved even if the esp32 chip is reset or power is removed.

NVS

NVS (Non Volatile Storage) is a library included in the esp-idf framework that allows to store information (in the form of key/value) in the flash memory, the content of which is not erased when you reset the chip or remove the power.

If you remember, in a previous post I explained how the external flash memory is organized. The main purpose of that memory is for sure to store the program that will be executed by the esp32 chip. However, it’s possible to divide the flash memory in partitions: the framework offers some ready-to-use partition tables but you’re free to define custom ones.

If you use the default partition table (“Single factory app, no OTA”), you may notice that it contains a partition of type data and subtype nvs:

nvs-01

The default size for that partition is 24Kbyte.

Thanks to the NVS library, you can store custom data into that partition. Information is organized in key/value pairs; a label (= key) with a maximum length of 15 characters is assigned to each value:

nvs-02

You can store different types of data: from numeric values to text strings and byte sequences (blobbinary large object). You’ll learn that the library provides specific methods based on the type of data you want to store or retrieve.

To be able to use the library in your code, first include the following headers:

#include "esp_partition.h"
#include "esp_err.h"
#include "nvs_flash.h"
#include "nvs.h"

Initialization

The first step to use the nvs partition is to initialize the library, with the command:

esp_err_t err = nvs_flash_init();

The command returns ESP_OK if successful; on the contrary it returns one of the error codes defined in the nvs.h file (see below). In particular, if the partition was resized or changed, you may get the ESP_ERR_NVS_NO_FREE_PAGES error code. This error can be resolved “formatting” the partition.

First you have to identify the nvs partition in the flash memory:

const esp_partition_t* nvs_partition = 
 esp_partition_find_first(ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_DATA_NVS, NULL);      
if(!nvs_partition) printf("FATAL ERROR: No NVS partition found\n");

then you can format it with the esp_partition_erase_range() command:

err = (esp_partition_erase_range(nvs_partition, 0, nvs_partition->size));
if(err != ESP_OK) printf("FATAL ERROR: Unable to erase the partition\n");

The last step is to open the partition, you can do it in READONLY o READWRITE mode:

nvs_handle my_handle;
err = nvs_open("storage", NVS_READWRITE, &my_handle);
if (err != ESP_OK) printf("FATAL ERROR: Unable to open NVS\n");

Set – Get

Once the partition is open, you can store (set) new values or retrieve (get) existing ones:

There are different methods, based on the data type (i8 stands for integer with 8bits, u8 per unsigned integer 8bit…):

  • nvs_set_i8(), nvs_set_u8(), nvs_set_i16(), nvs_set_u16()…
  • nvs_set_str()
  • nvs_set_blob()

All the set methods require as parameters the partition handler, the key name and the value to be stored:

esp_err_t nvs_set_i8(nvs_handle handle, const char* key, int8_t value);

The only exception is the nvs_set_blob() method which requires as additional parameter the length of the byte array to be stored:

esp_err_t nvs_set_blob(nvs_handle handle, const char* key, const void* value, size_t length);

After having called a set method, you have to commit the change with the method:

esp_err_t nvs_commit(nvs_handle handle);

To get data from the flash the library offers similar methods:

  • nvs_get_i8(), nvs_get_u8(), nvs_get_i16(), nvs_get_u16()…
  • nvs_get_str()
  • nvs_get_blob()

The parameters for those methods are the partition handler, the key name and a pointer to the variable that has to be updated with the value retrieved:

esp_err_t nvs_get_i8(nvs_handle handle, const char* key, int8_t* out_value);

Because of you cannot know a priori the size for string or blob values, you can do a “trick”: first call the nvs_get_string() method passing NULL as pointer to get the length of the value, then allocate a variable with the correct size and finally call again the method passing that variable to get the value:

size_t string_size;
esp_err_t err = nvs_get_str(my_handle, parameter, NULL, &string_size);
char* value = malloc(string_size);
err = nvs_get_str(my_handle, parameter, value, &string_size);

Erase

The library also include a couple of methods to erase the partition content:

esp_err_t nvs_erase_key(nvs_handle handle, const char* key);
esp_err_t nvs_erase_all(nvs_handle handle);

The first method is more selective and allows you to delete a single key while the second deletes the entire memory content.

Both the methods must be followed by the nvs_commit() command as explained before.

Error handling

All the methods in the nvs library return an error code (esp_err_t variable).

The possible errors are listed in the nvs.h file:

nvs-03

For example, if you’re calling a get method with a key that is not present in the flash, you’ll get the ESP_ERR_NOT_FOUND error. You should handle the different errors in your program:

esp_err_t err = nvs_get_i32(my_handle, parameter, &value);
if(err == ESP_ERR_NVS_NOT_FOUND) printf("\nKey %s not found\n", parameter);

Demo

For this tutorial I wrote a program that allows, using a simple command line, to store and retrieve information from the nvs partition. The source code is available on Github, here’s a video that shows how it works:

ESP32 (18) – Access Point

In all the examples so far, we used the esp32 chip in STAtion Mode, that is as a client which connects to an existing wifi network. In a previous article I explained in details how the different components of the esp-idf framework interact to establish the connection and how to use events to syncronize the tasks of your program.

Today you’ll learn how to configure the esp32 chip to publish its own wifi network, similar to an access point. This mode is indeed named SoftAP.

IP addressing

Every device connected to an IP network must have its unique address. You can statically assign the address or leverage a network service (DHCP, Dynamic Host Configuration Protocol) that dynamically assigns them. The main advantage in using the DHCP service is that you don’t need to know in advance – when connecting to a network – the correct settings to apply (IP address, netmask, gateway…); this is the reason why normally the device which manages the network also offers a DHCP server:

ap-05

The esp-idf framework does include a DHCP server. To use it in the network you’re going to create, you have to:

  • stop the service if it’s already running – tcpip_adapter_dhcps_stop()
  • statically configure the network interface of the esp32 chip – tcpip_adapter_set_ip_info()
  • start the service – tcpip_adapter_dhcps_start()

The esp-idf framework uses two different interfaces depending if the esp32 chip is in station mode or in SoftAP mode (TCPIP_ADAPTER_IF_STA e TCPIP_ADAPTER_IF_AP). Make sure you’re configuring the correct one based on the mode you’re planning to use.

For example, to assign the address 192.168.10.1/24 (/24 corresponds to 255.255.255.0 netmask) and run the DHCP server:

ESP_ERROR_CHECK(tcpip_adapter_dhcps_stop(TCPIP_ADAPTER_IF_AP));
tcpip_adapter_ip_info_t info;
memset(&info, 0, sizeof(info));
IP4_ADDR(&info.ip, 192, 168, 10, 1);
IP4_ADDR(&info.gw, 192, 168, 10, 1);
IP4_ADDR(&info.netmask, 255, 255, 255, 0);
ESP_ERROR_CHECK(tcpip_adapter_set_ip_info(TCPIP_ADAPTER_IF_AP, &info));
ESP_ERROR_CHECK(tcpip_adapter_dhcps_start(TCPIP_ADAPTER_IF_AP));

SoftAP

Now you can configure the SoftAP mode for the esp32 chip.

First initialize the event handler (later I’ll list the events that are available when working in this mode):

ESP_ERROR_CHECK(esp_event_loop_init(event_handler, NULL));

Then configure the wifi stack in access point mode, with the default settings and RAM storage:

wifi_init_config_t wifi_init_config = WIFI_INIT_CONFIG_DEFAULT();
ESP_ERROR_CHECK(esp_wifi_init(&wifi_init_config));
ESP_ERROR_CHECK(esp_wifi_set_storage(WIFI_STORAGE_RAM));
ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_AP));

The different settings for the AP mode are available in the wifi_ap_config_t struct:

ap-01

  • ssid is the name (Service Set Identifier) of the network
  • password is the password you have to specify to connect to the network (if the authentication mode requires it, see below)
  • ssid_len is the length (in characters) of the ssid string – you can set it to zero if the string is null-terminated
  • channel is the transmission channel
  • authmode is the authentication mode, see paragraph below
  • ssid_hidden allows to “hide” the network not broadcasting its SSID
  • max_connection is the maximum number of concurrent connections (max 4)
  • beacon_intereval is the transmission interval for beacon frames

Once configured, you have to apply struct using the esp_wifi_set_config() method. In the example I published on Github, all the parameters can be configured via menuconfig:

ap-02

wifi_config_t ap_config = {
	.ap = {
		.ssid = CONFIG_AP_SSID,
		.password = CONFIG_AP_PASSWORD,
		.ssid_len = 0,
		.channel = CONFIG_AP_CHANNEL,
		.authmode = CONFIG_AP_AUTHMODE,
		.ssid_hidden = CONFIG_AP_SSID_HIDDEN,
		.max_connection = CONFIG_AP_MAX_CONNECTIONS,
		.beacon_interval = CONFIG_AP_BEACON_INTERVAL,			
	},
};
ESP_ERROR_CHECK(esp_wifi_set_config(WIFI_IF_AP, &ap_config));

Finally start the wifi stack with:

ESP_ERROR_CHECK(esp_wifi_start());

Events

Three are the main events available when working in access point mode:

  • SYSTEM_EVENT_AP_START, when the stack completed the start process (after esp_wifi_start)
  • SYSTEM_EVENT_AP_STACONNECTED, when a new device connects to the access point
  • SYSTEM_EVENT_AP_STADISCONNECTED, when a device disconnects

ap-03

In the example, the events are notified using different event bits:

case SYSTEM_EVENT_AP_START:
	printf("Access point started\n");
	break;
 
case SYSTEM_EVENT_AP_STACONNECTED:
	xEventGroupSetBits(event_group, STA_CONNECTED_BIT);
	break;
 
case SYSTEM_EVENT_AP_STADISCONNECTED:
	xEventGroupSetBits(event_group, STA_DISCONNECTED_BIT);
	break;

Authentication modes

In SoftAP mode, the esp32 chip supports different authentication modes:

ap-04

open configures a wifi network with no authentication, while all the other modes offer some security, from the “weak” ones (WEP, now easily hackable) to the strongest ones (WPA2). Personally, I always use the WPA2_PSK mode, which offers high security and authentication, using a shared password (PSK = PreShared Key).

Some authentication modes require a minimum password length. In particular, WPA and WPA2 require a password at least 8 characters long; if you specify a password too short, the esp_wifi_set_config method will return an error.

List the connected devices

You can get the list of the devices connected to the access point with the esp_wifi_ap_get_sta_list() method, which updates a wifi_sta_list_t struct:

wifi_sta_list_t wifi_sta_list;
memset(&amp;wifi_sta_list, 0, sizeof(wifi_sta_list));
ESP_ERROR_CHECK(esp_wifi_ap_get_sta_list(&amp;wifi_sta_list));

To obtain the IP settings (for example the address) of the different stations, you can then use the tcpip_adapter_get_sta_list() method:

tcpip_adapter_sta_list_t adapter_sta_list;
memset(&adapter_sta_list, 0, sizeof(adapter_sta_list));
ESP_ERROR_CHECK(tcpip_adapter_get_sta_list(&wifi_sta_list, &adapter_sta_list));
for(int i = 0; i < adapter_sta_list.num; i++) {
	tcpip_adapter_sta_info_t station = adapter_sta_list.sta[i];
	printf("%d - mac: %.2x:%.2x:%.2x:%.2x:%.2x:%.2x - IP: %s\n", i + 1,
		station.mac[0], station.mac[1], station.mac[2],
		station.mac[3], station.mac[4], station.mac[5],
		ip4addr_ntoa(&(station.ip)));
}

Demo

ESP32 (17) – SNTP

In embedded applications it’s often necessary a clock, as accurate as possible. For example, imagine a device that has to activate an output at a given time or a logger that has to store values (temperature…) at precise intervals.

A solution widely adopted is the use of an RTC (Real Time Clock) chip. The problem becomes how you configure the RTC chip with the current time and how you keep it synchronized. Some time ago, I published a project (RTCSetup) to configure an RTC connected to an Arduino using the time of your PC; today I’ll show you how to use an Internet time service with the esp32 chip.

SNTP

SNTP (Simple Network Time Protocol) is a protocol designed to synchronize the clock of devices connected to the Internet. The latest version of the protocol (SNTPv4) is defined in the RFC 4330. The word “simple” in its name is because of SNTP is a reduced and simplified version of the NTP (Network Time Protocol) protocol; protocol which, to be able to guarantee high accuracy, is quite complex.

The basic working principle of SNTP is as follows:

- the client device connects to the server using the UDP protocol on port 123

- the server answers with different fields, among which is the timestamp value, 64bit long, where:

  • the first 32 bits are the number of seconds from 01/01/1990
  • the remaining 32 bits are the fraction of seconds

sntp-00

esp-idf

In a previous post I’ve already introduced the lwip library, chosen by the esp-idf framework to implement network communcation. The library includes an app (sntp.c) which implements a SNTP client.

Let’s see how to use it in your program (the full source of the test program is available in my Github repository).

First you have to include its header file:

#include "apps/sntp/sntp.h"

Then configure the SNTP client in pool mode, to query the servers every n seconds:

sntp_setoperatingmode(SNTP_OPMODE_POLL);

Alternatively, you can configure the client in listen only mode (SNTP_OPMODE_LISTENONLY). In this configuration, the client will listen for broadcast updates, without actively query any servers.

sntp_opts.h

Some configuration settings of the client, including the update delay (in milliseconds), can be changed in the sntp_opts.h file:

sntp-01

Now you have to tell the SNTP client which server it has to use. A common choice is to use the cluster of servers from pool.ntp.org. Alternatively, because of I live in Italy, I normally use the SNTP server offered by the Istituto Nazionale di Ricerca Meterologica (ntp1.inrim.it) that has an atomic clock as its time source.

In the example provided with this post, the server name can be configured via menuconfig:

sntp_setservername(0, CONFIG_SNTP_SERVER);

After having configured the client, you can start it with:

sntp_init();

the client will be executed in parallel with the main program.

To get the actual time from the system clock, use the time() method which updates a time_t variable:

time_t now;
time(&now);

The time_t variable usually represents the time as the number of seconds from a date called Epoch (01/01/1970). To split the variable into the different time values (year, month, day…) you can use the localtime_r method which updates a tm struct:

struct tm timeinfo;
localtime_r(&now, &timeinfo);

The fields of the tm struct are defined in the time.h file:

sntp-02

We can verify if the first sync with the SNTP server was perfomed with success for example checking the year field:

while(timeinfo.tm_year < (2016 - 1900)) {
  printf("Time not set, waiting...\n");
  vTaskDelay(5000 / portTICK_PERIOD_MS);
  time(&now);
  localtime_r(&now, &timeinfo);
}
Many date and time functions have two versions, one “normal” and one restartable (es. localtime and localtime_r). In a multithread environment like the FreeRTOS operating system is always suggested to use the functions with the _r suffix because they can be executed in parallel (reentrancy) as they don’t use common buffers.

If the struct tm is available, you can then use the strftime() method to format a string with the pattern you prefer. Many placeholders are available: they are described in the manpage of the method. Here are some examples:

char buffer[100];
// es. 08/05/2017 15:10:34
strftime(buffer, sizeof(buffer), "%d/%m/%Y %H:%M:%S", &timeinfo);
// es. Monday, 08 May 2017 
strftime(buffer, sizeof(buffer), "%A, %d %B %Y", &timeinfo);
// es. Today is day 128 of year 2017 
strftime(buffer, sizeof(buffer), "Today is day %j %Y", &timeinfo);

Time zone

SNTP servers keep the time with the UTC (Coordinated universal time) time zone. You can change the time zone modifying the TZ environmental value with the method:

int setenv(const char *name, const char *value, int overwrite);

name is the name of the variable and value the value for it. If overwrite is > 0 the method updates the variable if already exists.

Once the value for the TZ variable is set, you can initialize the timezone conversion routine with the tzset() command.

The TZ variable can have two different kind of values, depending on whether you want to handle daylight saving time or not.

In the simplest form (no daylight saving time) the format is std offset, where std is the name for the time zone (3 or more characters long) and offset specifies the time value you must add to the local time to get a UTC value. For example Italy is in the CET (Central European Time) timezone, which is one hour ahead from UTC (therefore offset -1). The command to use in Italy is:

setenv("TZ", "CET-1", 1);

If you want to handle the daylight saving time, you can use the format std offset dst [offset],start[/time],end[/time]std and the first offset have the same meaning as above. dst and the second offset specify the name and offset for the corresponding daylight saving time zone (if the second offset is omitted, the default is 1 hour ahead of standard time). Finally, start and end are the day and time when daylight saving time starts and ends.

For Italy, in the daylight saving time we use the CEST (Central European Summer Time) time zone, active (direttiva 2008/84/CE del Parlamento Europeo) from 02:00 of the last Sunday of March to 03:00 of the last Sunday of October.

The command is therefore:

setenv("TZ", "CET-1CEST-2,M3.5.0/02:00:00,M10.5.0/03:00:00", 1);

M3.5.0 specify March (3), last week (5), Sunday (0).

On the Internet you can find websites that lists the TZ values for different countries.

In conclusion, if you want to convert the time to a given timezone the steps are:

  • configure the TZ  variable (setenv())
  • inizialize the timezone conversion routine (tzset())
  • update the tm struct (localtime_r())

In the example program:

// print the actual time in Italy
printf("Actual time in Italy:\n");
localtime_r(&now, &timeinfo);
strftime(buffer, sizeof(buffer), "%d/%m/%Y %H:%M:%S", &timeinfo);
printf("%s\n", buffer);

sntp-03

ESP32 (16) – IFTTT

After having published my tutorial about sending SMS with the ESP32, I received some comments via the contact form of my website asking a way to send notifications for free. Today I’ll show you a way to do it; it will be also the occasion to write about IFTTT service and about developing custom components for the esp-idf framework.

IFTTT

IFTTT (IF This Then That) it’s a web service that allows to create applets connecting different services with the pattern if this happens, then do that action.

ifttt-12

To better understand what you can do with IFTTT, here are some pre-built applets:

  • send me an email if rain is expected tomorrow
  • automatically post on Facebook the photos I share on Instagram
  • turn on silence mode on my smartphone when I arrive at work

The main advantage of using IFTTT is indeed the availability of many services ready to use: starting from the ones which query weather forecast providers, to services which connect to famous social networks and even including services that can talk to smart devices like FitBit bracelets or Nest thermostats.

Besides the website, IFTTT is also available on Android and iOS smartphones; for this tutorial I’m going to use indeed a feature of this app, the notifications.

Maker Webhooks

Among the many services IFTTT offers, one is very handy to integrate IFTTT with IoT devices: the Maker Webhooks service. As the name suggests, this service allows applets to interface with the Web. If you add it as a This service (trigger, source of the event that activates the applet) Maker Webhooks allows to send an event to the applet using a web request (GET or POST), while if you add it as a That service (action that the applet performs), Maker Webhooks allows the applet to make a web call:

ifttt-13

The Maker Webhooks service is the first element of our applet: thanks to it we’ll be able to activate the applet with a web request performed by the ESP32 chip and we’ll be also able to pass optional values to the applet itself.

Notifications

The second service we’ll be using is called Notifications: using this service the applet can send notifications to our smartphone. The only requirement is that our smartphone is running the IFTTT app and that we log in with the same account used to create the applet.

Applet

Let’s see how to build the applet. After having registered to the website, click on New Applet.

First click on this:

ifttt-01

Type some letters of “Maker Webhooks” to search the service, then click on its icon:

ifttt-02

Select the only trigger the service offers:

ifttt-03

Type the name of the event that will activate the applet. You can choose the name we prefer; the same name will be used when making the web request:

ifttt-04

Now click on that:

ifttt-05

Search and select the Notifications service:

ifttt-06

Click on the only action the service offers:

ifttt-07

Customize the message that will be displayed as notification. You can also include some custom values that will be received by the applet via the web call, together with the event name:

ifttt-08

Complete the wizard with a click on Finish:

ifttt-09

Personal key

To be able to send a request to the Maker Webhooks service, you must specify your personal key. Connect to the service’s homepage and click on Settings:

ifttt-10

Copy the key, that is the string after /use/ in the service URL:

ifttt-11

esp32_ifttt_maker component

To simplify the interaction with the Maker Webhooks service, I developed a component that is ready to use with the esp-idf framework. You can compare components to libraries, reusable code elements for your programs. You only need to copy the component’s folder within the components folder of your project to be able to use it:

ifttt-14

My esp32_ifttt_maker component is available in a dedicated Github repository and I explained how to use it in details in a dedicated webpage of my website.

Demo

I prepared a demo program that sends notifications using IFTTT applets; its source code is on Github.

Here’s a video that shows how it works (italian subtitles are available):

ESP32 (15) – mDNS

When you surf on the Internet, the DNS (Domain Name System) service has the job of “translating” (resolve) the hostnames to the corresponding IP addresses.

If, for example, you type www.google.com in your favorite browser, your computer queries the DNS server – usually the one managed by your provider – and gets from it the IP address of (one of) the servers that host Google website:

mdns-05

A home network usually doesn’t include a DNS server, so if you need to communicate with a device you must know its IP address. It may be easy to find the address of a device if such address is statically assigned or if the device itself has a display that can show it:

mdns-01

On the contrary, if the IP address is dynamically assigned (for example by the DHCP server running on your router) and the device can’t display it, it may be hard to find which is the address of the device. It may help the Multicast DNS (mDNS) protocol; let’s learn how to use it with our esp32 chip.

mDNS

mDNS is a network protocol, defined in RFC6762, which allows to resolve hostnames without the need of a DNS server in the network.

mDNS is the protocol we use also when connecting to a Raspberry through the hostname raspberrypi.local as explained for example in my tutorial about the connection to a Raspberry Pi Zero.

Its behavior is very simple. When a device needs to know the IP address of another device on the network, it sends a multicast UDP packet with the request. Since it is a multicast packet, it reaches all the devices connected to the network. The response is again sent out using a multicast packet, so all the devices can receive it and update their resolving table:

mdns-02

By default, mDNS resolves only hostnames with the .local suffix.

ESP32

The esp-idf framework esp-idf includes a component that implements the mDNS protocol; in this tutorial I’ll explain how to use it to answer mDNS queries. The complete source code is available in my Github repository; let’s analyse the most important part.

First of all, using the menuconfig you can configure two parameters:

mdns-03

MDNS_HOSTNAME is the hostname that will be resolved by the esp32 chip (for example “esp32“) while MDNS_INSTANCE is a description of the device (es. “ESP32 Demo Board“).

It’s very easy to setup the mDNS server that answers incoming queries. After having established the connection to the wifi network (this part of the program is based on a previous example) you have only to create a new instance of the server and configure it with the parameters defined above:

// mDNS server instance
mdns_server_t* mDNS = NULL;
[...]
// create and configure the mDNS server
ESP_ERROR_CHECK(mdns_init(TCPIP_ADAPTER_IF_STA, &amp;mDNS));
ESP_ERROR_CHECK(mdns_set_hostname(mDNS, CONFIG_MDNS_HOSTNAME));
ESP_ERROR_CHECK(mdns_set_instance(mDNS, CONFIG_MDNS_INSTANCE));

Do not forget to include the component’s header at the top of your code:

#include "mdns.h"

That’s all! When you run the program, the mDNS server waits for incoming requests and, if those requests are for the name MDNS_HOSTNAME.local, it answers with the IP address of your board.

mDNS for Windows and Linux

To be able to test the program, your computer has to “speak” the mDNS protocol.

If you’re running Windows, you can install the Bonjour Print Services tool from Apple. Even if it was designed to discovery and configure network printers (hence its name), this service does support the full mDNS protocol and it’s therefore the best solution with Microsoft OSes.

mdns-04

Under Linux, you can use the Avahi daemon, that is usually already installed on the most common distributions.

After having configured your PC, you can test the program with the command ping esp32.local (change the name with the one you configured):

mdns-06

Conclusion

Using the mDNS component, you can design devices, based on the esp32 chip, that can be easily identified on the network, without the need to add external components like a display or a serial connection. Moreover it’s simpler, specially for non-expert users, to use a standard name (“esp32.local”) then looking for a “strange number” (the IP address) that can also change over time.

In this tutorial I’ve only explained how to write a program that answers mDNS queries: this is the most common use of the protocol but it’s not the only one: the component included in the framework also allows to send queries or to broadcast different services… Espressif published a sample program that leverages also some of these features.

ESP32 (14) – esp-idf v2 is here and how to manage different versions

On 6th of April, Espressif published version 2.0 of the esp-idf framework.

idf2-01

The complete list of the new features and the bugs fixed is available on Github, let’s see the most important ones (in my opinion ;)):

  • they included or made stable new drivers for peripherals like I2C, I2S, SPI Master, SDMMC
  • you can now run a serial monitor using build commands (make monitor)
  • both the two cores (CPU e APP) are now enabled by default
  • the examples have been revised and expanded

It’s very important to notice – if you’re using a dedicated toolchain to build your projects – that the new framework does require version 5 of GCC compiler; you therefore need to download the updated toolchain Espressif prepared (I’ve already updated my tutorial about how to install and configure it).

Git repositories

The framework‘s source code is published on Github, that is a projects web repository based on the version control software Git. To explain how you can download and use the different versions of the framework, I have to do a brief introduction of the main concepts and commands of the tool.

What follows is not meant to be a complete tutorial about Git but only a way to give some information to better understand the commands used afterwards. I strongly suggest the tutorials by Atlassian if you want to explore the complex world of Git.

In Git, the “container” for a project or a set of files is named repository. All the information Git requires to manage a repository are stored in the .git subfolder within the main folder of your project:

idf2-02

To create a local repository, you only need to enter the folder of your project and run the command:

git init

If instead you want to copy on your local PC a remote repository (for example the repository – hosted on Github – that contains the esp-idf framework) you have to use the clone command:

git clone <url>
Notice that the clone command creates a local copy of the whole repository, including all the different versions of the files it contains.

It may happen that, when developing a project, you need to use code from other projects (for example libraries developed by others). In Git, thanks to submodules, you can “include” a Git repository as a folder of another repository. The esp-idf framework uses submodules to include the Wifi and Bluetooth libraries – at the moment for those libraries you cannot access the source code.

Browsing the repository, you can identify submodules thanks to a different icon:

idf2-03

If you click on it, you’re redirected to the Github repository which contains the module:

idf2-04

If a Git repository contains one or more submodules and you want to clone it including those submodules you have to add the –recursive option. The command to have a full local copy of the esp-idf repository is therefore:

git clone --recursive https://github.com/espressif/esp-idf.git

You can keep the local copy aligned to the remote repository with the command git pull. If the repository contains submodules, you have also to run the command git submodule update to update them as well:

idf2-05

 Branches and tags

Every version control system allows to branch your project, for example to perform parallel developments or to add new features and at the same time do some bugfixing on the actual version.

Moreover, Git allows to “take a snapshot” of the repository in a point in time and to label that snapshot with a tag.

On the Github website, you can select a specific branch or tag using the drop-down menu on the top left corner:

idf2-06

When Espressif releases a new version of the framework – both for stables and RC ones – it creates a specific tag:

idf2-07

We can therefore align the local copy of the repository to a specifig tag (for example version 2.0) with the commands:

git checkout v2.0
git submodule update --init

Working with different versions

Sometimes you may need to have different versions of the framework at the same time on your PC; for example because one of your projects is not compatible with version 2.0, version you’re instead using for a new development.

All the framework’s build process is based on the folder declared in the IDF_PATH env variable:

idf2-08

You can create different folders on your PC and use the git commands explained above to clone in those folders different versions of the repository. It only needs to change the IDF_PATH variable to let you choose which version you’re going to use when you run the make command.