ESP32 (37) – https OTA

In the previous post of this tutorial, I explained how it is possible to update your board Over-The-Air thanks to a feature of the Freshen IoT dashboard.

Today I’ll show you how to update the firmware running on an esp32 chip using only components included in the esp-idf framework, without the need of any external tools or platforms.

OTA API

The esp-idf framework offers a set of native functions to implement, in your program, the ability to be updated over the air.

Those functions are grouped in the app_update component and to use them in your program you have to include the corresponding header file:

#include "esp_ota_ops.h"

Altough the use of the native functions is not very difficult (on Github you can find an example program), Espressif developers have added a component to the framework that makes it even easier the over the air update if the new firmware is located on a web site.

The component is named esp_https_ota.

esp_https_ota

The esp_https_ota component uses the OTA API to update the firmware of your board, downloading the binary file that contains the new firmware from a web site. As the name suggests, the only requirement (for security reason) is that the web site supports the secure version of the protocol (HTTPS).

The component is able to automatically identify an OTA partition in the flash memory that is not in use and to save the new firmware in that partition. It then configures the chip to boot from that partition:

https_ota_001

The use is very simple. First create an esp_http_client_config_t struct to configure the URL of the file with the new firmware and the SSL certificate of the server (or the certificate of the CA that signed it):

esp_http_client_config_t ota_client_config = {
  .url = "https://mysite.com/newfirmware.bin",
  .cert_pem = server_cert_pem_start,
};

You have to provide the certificate in PEM format. To store the certificate in your program, you can leverage the embedding binary files functionality of the firmware, as I already explained in a previous tutorial.

Then you only have to call the function:

esp_err_t ret = esp_https_ota(&ota_client_config);

to start the update process. If – when the process is complete – the ret variable contains a positive result (ESP_OK), you can reboot the chip to run the new firmware:

esp_restart();

A real application would probably need to periodically check if a new firmware is available and, only in that case, to start the update process. How can it be done?

In the program I wrote for this post and that is explained in the video below, I’m going to show a way widely used also in commercial products… enjoy the show ;)

as usual, the source code of the program is available in my Github repository

ESP32 (36) – OTA with Freshen

In one of the first posts of this tutorial, I wrote about the bootloader and about how the flash memory is organized. That article says:

This allows to implement an over-the-air (OTA) application update process: you send the new version of your application to the esp32 chip; the version is stored in a new app partition

There are different ways to implement OTA updates for your application… today I’ll show you how to do it in an easy way, thanks to a cloud service named Freshen.

Freshen

Freshen is an IoT backend, that is a cloud dashboard (published on the Internet) to manage IoT devices.

After having connected your devices to Frashen (I’ll show you later how to do it) you can:

  • display the list of the devices and their status
  • send a command to a device
  • manage the files stored in a device
  • update the firmware over-the-air (OTA)

Freshen is developed by Cesanta, the company that also develops the Mongoose library and the MongooseOS, very used in the embedded world and compatible with the esp32 chip. The service has different fees, including a free one:

freshen-001s

To use Freshen, first you have to register, using your Github or Google account:

freshen-002

Client library

Cesanta offers a client library you can use to connect your IoT project based on the esp32 chip to Freshen. The library consists only in a header file (freshen.h), and is available on the official site.

The library is fully compatible with the esp-idf framework and supports all the functionalities of the dashboard:

freshen-003

It’s also very easy to use. First copy the freshen.h file in the main folder of your project:

freshen-009

Then include the file in your program:

#include "freshen.h"

To let the library communicate with the Freshen platform in the cloud, in your program you have to periodically call the freshen_loop() function:

You can do it in a dedicated task:

void freshen_task(void *pvParameter) {
  while(1) {	
    freshen_loop(FIRMWARE_VERSION, ACCESS_TOKEN);
    vTaskDelay(2000 / portTICK_RATE_MS);
  }
}

This task calls the function every 2 seconds. Create the task in your app_main() with:

// start the freshen update task
xTaskCreate(&freshen_task, "freshen_task", 10000, NULL, 5, NULL);

The freshen_loop() function requires two parameters: the firmware version (it’s a text string at your choice, for example “v1.0″) and the access token, a code that is generated by the platform when you register a device.

Connect to the dashboard and click on Add new device:

freshen-010

A new device will be added (My Device #n). You can click on its name to display the details.

The access token is hidden… Click on click to copy to save it in the clipboard on your computer; you can then paste it in your program:

freshen-011

The device details page also allows to change the name of your device.

If you run the example program (you can download it from my Github repository) you’ll notice that, after a couple of seconds, the status of your device changes to online, to indicate that it’s correctly sending data to the dashboard:

freshen-012

In the details page, you can now list the functions that you can remotely call:

freshen-013

For example, if you call the Sys.GetInfo function you can retrieve the information about firmware version, architecture, compile date…:

freshen-014

OTA

To be able to update the firmware of your device over the air, you have to select an appropriate partition layout. For example you can choose the “OTA” layout that is already included in the framework:

freshen-006

Let’s now modify the firmware version, to be “1.1”:

freshen-015

then compile the program but don’t use the flash command. Note the path of the binary (compiled) file:

freshen-016

Select your device in the dashboard and click on OTA update selected:

freshen-017

 

Choose the .bin file that contains the new version of your firmware.

After a few moments, you’ll probably see the device going offline and return online. If the update process was successful, call the Sys.GetInfo function again and you’ll see the new version:

freshen-018

 

SeeedStudio, 3 PCBs for 1$

Here’s a new promo for the Fusion PCB service by SeeedStudio: from today untill September 21st, you can order 3 PCBs for 1$ (excluding shipping costs)

seeed-1dollar

To take advantage of the offer, simply enter the order from the dedicated page of their website.

The offer includes PCBs with the following characteristics:

  • 1/2 layers
  • max size of 10x10cm
  • thickness 0.6 -> 1.6mm
  • one of the 6 available colors (green/red/yellow/blue/white/black)

If you want to learn how to use the Fusion PCB service, here’s the link to my tutorial!

 

The smallest esp32 module (so far…)

Some months ago, Espressif announced the production of a new chip, named ESP32-PICO-D4.

esp-pico-001

It’s a complete SiP (System in Package), that is a chip which integrates the esp32 microcontroller, a 4Mbit flash memory, a crystal oscillator, filter capacitors and RF matching links. The chip datasheet is available on the official website.

Using this chip, it’s possible to create very small modules. I recently received one of those from Aliexpress:

esp-pico-002

esp-pico-003

To better understand how small it is, let’s compare its size with a “classic” ESP-WROOM-32 module and with a 1 euro coin:

esp-pico-004

The module includes a chip antenna; it’s also possible to connect an external antenna thanks to the presence of an I-PEX connector.

In conclusion, the availability of the ESP32-PICO-D4 SiP makes it possible to use the esp32 chip in applications where the available space is very small…

ESP32, PlatformIO

PlatformIO is an opensource ecosystem (as it’s defined in the homepage of the project) to develop IoT projects.

The heart of the platform is a software component named PlatformIO Core. This component includes:

  • a cross-platform compiler
  • a libraries and dependences manager
  • a serial monitor

PlatformIO Core is developed in Python and therefore it can run on different operating systems (Windows, Linux, MacOS).

Although you can use the Core component directly, PlatformIO’s strength lies in its IDE, which allows the development of multi-platform projects and integrates with the Core itself.

In this article I’m going to show you how to use PlatformIO to develop projects running on the esp32 chip.

Installation

PlatformIO IDE is provided as a plugin for two different development tools: Atom and VisualStudio Code. I tried both solutions and I preferred VSCode: both the installation and the use are simpler and more immediate.

Install VSCode after having downloaded the package from Microsoft’s website (the installer is available for Windows, Linux and MacOS).

Open the Package Manager:

pio-001

search the PlatformIO IDE package, then click on Install:

pio-002

wait until the installation is complete:

pio-003

Hello world

Now it’s time to develop your first program, which traditionally will display the sentence Hello world! on the terminal.

If it doesn’t show up automatically, open the PlatformIO’s homepage:

pio-004

then click on New Project:

pio-005

give a name to the project and choose a devboard based on the esp32 chip (in this example I’ll use a Lolin32 board by Wemos). PlatformIO supports both the esp-idf framework and the arduino-esp32 one. All my tutorials are based on the first one:

pio-006

PlatformIO automatically creates some folders for your project. Choose the src folder (it stands for source, that is the folder which will contain the source code) and create a new file:

pio-007

name the file main.c and type the simple program as it follows:

pio-008

run the compiler by clicking on the corresponding button in the bottom bar:

pio-009

the editor displays an error… indeed your code is using the printf() function without having included the library:

pio-010

add the missing line, now you should be able to complile the code without errors:

pio-011

PlatformIO can also upload the compiled program to your board. Thanks to its auto-detect feature, you usually don’t need to specify the serial port the board is connected to:

pio-012

Serial monitor

PlatformIO also includes a serial monitor you can use to test your program. By default, this monitor connects to the serial port with a speed of 9600 baud. The esp32 chip instead has a default speed of 115200 baud; you have therefore to change the platformio.ini file included in your folder as it follows:

pio-013

Now open the serial monitor; you should see the correct output of your program:

pio-014

Conclusions

I found the use of PlatformIO really immediate: after a few minutes I was able to develop, compile, load and test a program. Try it and leave a comment with your impressions!

 

My new third hand

I recently brought from Banggood a support for having a third hand during soldering operations…

hand-001 hand-002

the support is made by an aluminum base and three arms which end with a crocodile clamp with insulating sheath:

hand-003 hand-004

The assembly is very simple, the arms must be screwed on the base:

hand-005

In general this support is very usefulthe arms are very well articulated and their length allows to “grasp” even medium-sized cards:

hand-007 hand-006

The only defect is the position of the feet, which makes the base not too stable:

hand-008

 

JQ6500, “flashdata error”

After having published my previous posts about the JQ6500 module, some readers wrote me about an error the MusicDownload.exe application shows when they click on the upload button to store the mp3 files in the flash memory.

The error is

flashdata error, have not define program mode in flashdata.ini!!

jq6500-err-001

It seems that the application is not fully compatible with Windows 8 or 10.

To solve the problem, you only have to enable the Windows 7 compatibility mode:

jq6500-err-002

With this setting, the upload process runs fine:

jq6500-err-003

In the following video, you can see all the required steps:

JQ6500 module recovery

In a previous post I presented the JQ6500 audio module. When you connect it to your computer, it’s detected as a CDROM drive.

I brought some modules from different webstores. A couple of them gave me the following error when I tried to browse the content of the CD:

Unable to access the disk. The disk may be damaged.

jq6500-rec-000

It seems that some manufacturers do not program the flash memory of the chip and therefore it doesn’t contain the MusicDownload.exe application.

Luckily, thanks to the work by Reinhard Max and Nikolai Radke, it’s now possible to recover those modules. From Nikolai’s Github repository it’s possible to download an ISO image with a minimal Linux distribution and the recovery toolkit:

jq6500-rec-100

After having downloaded the image, you can burn it to a CD (or use software like UNetbootin to transfer it to a bootable USB drive) and restart your PC with that image.

However, there is an alternative way: the use of VMware Workstation Player, a free virtualization software (for non-commercial use) by VMware. Using this application it’s possible to start a virtual machine on your PC that can run the recovery ISO image.

At the end of this post you can find a video that shows all the following steps…

Let’s see how to do it. After having installed the application, create a new virtual machine:

jq6500-rec-008

Choose I will install the OS later:

jq6500-rec-009

Select Linux as operating system:

jq6500-rec-010

Give the VM a name and select the folder where it will be saved:

jq6500-rec-011

You don’t really need an hard disk, so you can specify a small size for it (for example 1Gb):

jq6500-rec-012

Now you have to configure VMware Player to start the VM using the ISO image. Click on Customize Hardware:

jq6500-rec-013

Select the CD/DVD drive and connect it to the ISO image downloaded from Github:

jq6500-rec-014

Before starting the VM, verify that your module is not connected to the computer:

jq6500-rec-015

Wait for the boot process to complete. The toolkit (started automatically) will warn you that it wasn’t able to identify any modules:

jq6500-rec-001

Open the Removable Devices menu and note down all the listed devices:

jq6500-rec-002

Now connect the module to the USB port. If you open the menu again, you should be able to identify which device corresponds to your module:

jq6500-rec-003

Connect it to the virtual machine:

jq6500-rec-004

Perform a new scan typing the command. If everything is ok, now the toolkit should be able to identify the module:

jq6500-rec-005

Depending on the size of the flash memory of your module, choose the correct menu item. Most modules are sold with a 16Mbit or 2Mbyte memory chip:

jq6500-rec-006

At the end of the recovery process, you can shut down the VM and verify that your computer is now able to display the disk content:

jq6500-rec-007

Video

Here’s a video that shows the recovery process:

First look at JQ6500 modules

For some time, on different chinese webstores (for example Banggood) there is a module called JQ6500 for sale:

jq6500-001 jq6500-002

it’s often described as a voice sound module or as an MP3 player sound module.

Actually JQ6500 is the name of the main chip hosted on the module:

jq6500-003

The chip is manufactured by a Chinese company named JQ. A datasheet for the chip is also available, unfortunately only in Chinese (but Google Translate can help to understand what it contains).

On the other side of the PCB, the module houses two additional integrated circuits:

  • a 16Mbit flash memory (25L1606E)
  • a 3W audio amplifier (HXJ 8002)

When you connect the module to your computer via USB, it is detected as a CDROM drive. If you browse the content of the CD, you can find the MusicDownload.exe application that allows to upload audio files in the flash memory:

jq6500-rec-007

The software is in Chinese but its use is very simple: by moving to the second tab you can select the MP3 files to be uploaded. If you now move back to the initial tab, you can start the upload process clicking on the only available button. In the video at the end of this post you can see how it works…

You can control the JQ6500 chip in different ways. The easiest one is using external buttons connected to pins K1-2-3-4-5:

jq6500-004

When you press a button, the chip plays the corresponding audio file. For example if you press the button connected to pin K1, the chip plays the audio file named 001.mp3.

The onboard amplifier (HXJ 8002) is a mono IC and its output is connected to pin SPK+ and SPK-. You can therefore connect to those pins a small speaker. If you want a stereo audio, you can instead use pins ADL_L (left channel) and ADC_R (right channel) and connect them to an external amplifier.

Conclusions

This module is an excellent and inexpensive solution to add audio to your projects. The use of an internal flash memory has the advantage of not requiring SD cards or other media to store your audio files; in contrast its capacity (16Mbit = 2MByte) makes it more suitable to reproduce sound effects / guide voices than to make a music player.

In the next articles I will show you how to interface the module with Arduino … meanwhile here is a video showing my first tests:

 

 

 

ESP32 (35) – BLE, scan response

In the previous posts I explained how to receive and send advertising packets based on the Bluetooth LE standard.

The payload (that is the amount of “useful” data) of those packets is at most 31 bytes. It isn’t much: if – for example – you want to include the device name, little place remains for other data.

The BLE standard allows peripherals to send additional data using the scan request – scan response process.

When a device receives an advertising packet, it can contact the transmitter by sending a scan request packet to request further information. When receiving a scan request package, the peripheral can respond with a scan response packet:

scan-response-001

Advertising and scan request packets have the same format; it’s therefore possible to transfer, using scan response, additional 31 bytes of data.

esp32

The esp framework offers two modes for configuring the content of a scan response packet: using the esp_ble_adv_data_t struct or creating a byte array (raw mode). These modes are similar to the ones used to configure advertising packets you learned in previous articles (struct and raw mode).

In the first case, you have to declare a second struct, in addition to the one related to the advertising packet, to define the content of the scan response packet:

static uint8_t manufacturer_data[6] = {0xE5,0x02,0x01,0x01,0x01,0x01};
static esp_ble_adv_data_t scan_rsp_data = {
  .set_scan_rsp = true,
  .manufacturer_len = 6,
  .p_manufacturer_data = manufacturer_data,
};

Very important is set to true the set_scan_rsp parameter. It’s indeed this parameter what tells the driver that this struct is related to the scan response packet.

You can then pass the new struct to the driver, with the same function used previously:

esp_ble_gap_config_adv_data(&scan_rsp_data);

The driver will call the callback function twice: one to indicate the successful configuration of the advertising packet and one for the configuration of the scan response one. The two events are different:

case ESP_GAP_BLE_ADV_DATA_SET_COMPLETE_EVT:
  [...]
case ESP_GAP_BLE_SCAN_RSP_DATA_SET_COMPLETE_EVT:
  [...]

You have to wait until both the events have triggered before starting the advertising process. In my example program (you can download the source code from my Github repository) I use two boolean variables:

bool adv_data_set = false;
bool scan_rsp_data_set = false;
[...]
case ESP_GAP_BLE_ADV_DATA_SET_COMPLETE_EVT:
  adv_data_set = true;
  if(scan_rsp_data_set) esp_ble_gap_start_advertising(&ble_adv_params); break;
 
case ESP_GAP_BLE_SCAN_RSP_DATA_SET_COMPLETE_EVT:
  scan_rsp_data_set = true;
  if(adv_data_set) esp_ble_gap_start_advertising(&ble_adv_params); break;

If you want to use the raw mode instead, you have to declare a byte array and fill it with the content of the payload of the packet. Then you can use a specific function of the framework to pass the array to the driver:

static uint8_t scan_rsp_raw_data[8] = {0x07,0xFF,0xE5,0x02,0x01,0x01,0x01,0x01};
[...]
esp_ble_gap_config_scan_rsp_data_raw(scan_rsp_raw_data, 8);

did you notice that the content of the scan response packet is the same in the two examples?

The driver will confirm the configuration of the packet with a dedicated event. Also in this case you have to wait for the end of both configurations (advertising and scan response):

case ESP_GAP_BLE_SCAN_RSP_DATA_RAW_SET_COMPLETE_EVT:
  scan_rsp_data_set = true;
  if(adv_data_set) esp_ble_gap_start_advertising(&ble_adv_params); break;
You can also mix the two modes in your program. For example you can configure the advertising packet using the struct and configure the scan response one using the raw mode.

Now with the nRF Connect app you can verify that your scan response packet is correctly received by your smartphone:

scan-response-002

In the following video I explain how I built the payload of the packet and how the program works: