ESP32 (37) – https OTA

In the previous post of this tutorial, I explained how it is possible to update your board Over-The-Air thanks to a feature of the Freshen IoT dashboard.

Today I’ll show you how to update the firmware running on an esp32 chip using only components included in the esp-idf framework, without the need of any external tools or platforms.

OTA API

The esp-idf framework offers a set of native functions to implement, in your program, the ability to be updated over the air.

Those functions are grouped in the app_update component and to use them in your program you have to include the corresponding header file:

#include "esp_ota_ops.h"

Altough the use of the native functions is not very difficult (on Github you can find an example program), Espressif developers have added a component to the framework that makes it even easier the over the air update if the new firmware is located on a web site.

The component is named esp_https_ota.

esp_https_ota

The esp_https_ota component uses the OTA API to update the firmware of your board, downloading the binary file that contains the new firmware from a web site. As the name suggests, the only requirement (for security reason) is that the web site supports the secure version of the protocol (HTTPS).

The component is able to automatically identify an OTA partition in the flash memory that is not in use and to save the new firmware in that partition. It then configures the chip to boot from that partition:

https_ota_001

The use is very simple. First create an esp_http_client_config_t struct to configure the URL of the file with the new firmware and the SSL certificate of the server (or the certificate of the CA that signed it):

esp_http_client_config_t ota_client_config = {
  .url = "https://mysite.com/newfirmware.bin",
  .cert_pem = server_cert_pem_start,
};

You have to provide the certificate in PEM format. To store the certificate in your program, you can leverage the embedding binary files functionality of the firmware, as I already explained in a previous tutorial.

Then you only have to call the function:

esp_err_t ret = esp_https_ota(&ota_client_config);

to start the update process. If – when the process is complete – the ret variable contains a positive result (ESP_OK), you can reboot the chip to run the new firmware:

esp_restart();

A real application would probably need to periodically check if a new firmware is available and, only in that case, to start the update process. How can it be done?

In the program I wrote for this post and that is explained in the video below, I’m going to show a way widely used also in commercial products… enjoy the show ;)

as usual, the source code of the program is available in my Github repository

ESP32 (36) – OTA with Freshen

In one of the first posts of this tutorial, I wrote about the bootloader and about how the flash memory is organized. That article says:

This allows to implement an over-the-air (OTA) application update process: you send the new version of your application to the esp32 chip; the version is stored in a new app partition

There are different ways to implement OTA updates for your application… today I’ll show you how to do it in an easy way, thanks to a cloud service named Freshen.

Freshen

Freshen is an IoT backend, that is a cloud dashboard (published on the Internet) to manage IoT devices.

After having connected your devices to Frashen (I’ll show you later how to do it) you can:

  • display the list of the devices and their status
  • send a command to a device
  • manage the files stored in a device
  • update the firmware over-the-air (OTA)

Freshen is developed by Cesanta, the company that also develops the Mongoose library and the MongooseOS, very used in the embedded world and compatible with the esp32 chip. The service has different fees, including a free one:

freshen-001s

To use Freshen, first you have to register, using your Github or Google account:

freshen-002

Client library

Cesanta offers a client library you can use to connect your IoT project based on the esp32 chip to Freshen. The library consists only in a header file (freshen.h), and is available on the official site.

The library is fully compatible with the esp-idf framework and supports all the functionalities of the dashboard:

freshen-003

It’s also very easy to use. First copy the freshen.h file in the main folder of your project:

freshen-009

Then include the file in your program:

#include "freshen.h"

To let the library communicate with the Freshen platform in the cloud, in your program you have to periodically call the freshen_loop() function:

You can do it in a dedicated task:

void freshen_task(void *pvParameter) {
  while(1) {	
    freshen_loop(FIRMWARE_VERSION, ACCESS_TOKEN);
    vTaskDelay(2000 / portTICK_RATE_MS);
  }
}

This task calls the function every 2 seconds. Create the task in your app_main() with:

// start the freshen update task
xTaskCreate(&freshen_task, "freshen_task", 10000, NULL, 5, NULL);

The freshen_loop() function requires two parameters: the firmware version (it’s a text string at your choice, for example “v1.0″) and the access token, a code that is generated by the platform when you register a device.

Connect to the dashboard and click on Add new device:

freshen-010

A new device will be added (My Device #n). You can click on its name to display the details.

The access token is hidden… Click on click to copy to save it in the clipboard on your computer; you can then paste it in your program:

freshen-011

The device details page also allows to change the name of your device.

If you run the example program (you can download it from my Github repository) you’ll notice that, after a couple of seconds, the status of your device changes to online, to indicate that it’s correctly sending data to the dashboard:

freshen-012

In the details page, you can now list the functions that you can remotely call:

freshen-013

For example, if you call the Sys.GetInfo function you can retrieve the information about firmware version, architecture, compile date…:

freshen-014

OTA

To be able to update the firmware of your device over the air, you have to select an appropriate partition layout. For example you can choose the “OTA” layout that is already included in the framework:

freshen-006

Let’s now modify the firmware version, to be “1.1”:

freshen-015

then compile the program but don’t use the flash command. Note the path of the binary (compiled) file:

freshen-016

Select your device in the dashboard and click on OTA update selected:

freshen-017

 

Choose the .bin file that contains the new version of your firmware.

After a few moments, you’ll probably see the device going offline and return online. If the update process was successful, call the Sys.GetInfo function again and you’ll see the new version:

freshen-018

 

MIUI and tethering

My smartphone is a Xiaomi Redmi Note 3 Pro, running the “Global” MIUI ROM. Some days ago I brought a SIM card from a new italian operator, ho-mobile, which allows to use the tethering feature of the phone to connect external devices to the Internet.

I faced some problems before being able to use that feature… this blog post is to describe how I solved them.

Tethering was not working due to the incorrect configuration of a property of the Android operating system, called

tether_dun_required

As explained in Android source code, this property tells the O.S. if it’s necessary to use a DUN (dial-up networkAPN for the tethering funcion. Actually, as well explained on Taming the Droid:

This refers to an outdated method for using your phone to emulate a dial-up modem and is not the way that modern smartphones do tethering anymore. There should be no need to use this value.

To set the property to 0 (= not required), you have to send a command in the smartphone’s shell.

Requirements

You need to:

Procedure

In the Developer Options menu, enable the following 3 items:

  • USB debugging
  • Install via USB
  • USB debugging (Security settings)

x-teth001

You’ll be prompted, if not already done, to login with your MI account.

It may happen that, when you enable an item, you get the following error:

x-teth002

The error means that your phone cannot contact Xiaomi servers, probably because of the Great Firewall of China.

You can solve the problem thanks to an app that establishes a VPN with a chinese server, that is behind the firewall. Install, from Play Store, the Flit VPN app and run it.

Choose Start without registration, then click on Connection Settings and choose one of the Jangsu server, the one with best performances:

x-teth003

Click on Connect and wait until the connection is established:

x-teth004Now you should be able to enable all the debugging items without any problems. When enabled, you can disconnect the VPN and uninstall the app.

Connect your smartphone to the computer via USB and run Minimal ADB and Fastboot.

Type the command adb shell to open a console connection with the phone.

Using the settings get global tether_dun_required command you can read the current value of the parameter. If you read null or 1 the setting is wrong:

mi-teth002

Use the settings put global tether_dun_required 0 command to set the value to zero. You can test if the command was successful re-entering the get command:

mi-teth003

NB: if you see a permission denied error, it means that not all the debugging features have been correctly enabled:

mi-teth001

Now you can use the tethering of your phone, happy surfing!

SeeedStudio, 3 PCBs for 1$

Here’s a new promo for the Fusion PCB service by SeeedStudio: from today untill September 21st, you can order 3 PCBs for 1$ (excluding shipping costs)

seeed-1dollar

To take advantage of the offer, simply enter the order from the dedicated page of their website.

The offer includes PCBs with the following characteristics:

  • 1/2 layers
  • max size of 10x10cm
  • thickness 0.6 -> 1.6mm
  • one of the 6 available colors (green/red/yellow/blue/white/black)

If you want to learn how to use the Fusion PCB service, here’s the link to my tutorial!

 

The smallest esp32 module (so far…)

Some months ago, Espressif announced the production of a new chip, named ESP32-PICO-D4.

esp-pico-001

It’s a complete SiP (System in Package), that is a chip which integrates the esp32 microcontroller, a 4Mbit flash memory, a crystal oscillator, filter capacitors and RF matching links. The chip datasheet is available on the official website.

Using this chip, it’s possible to create very small modules. I recently received one of those from Aliexpress:

esp-pico-002

esp-pico-003

To better understand how small it is, let’s compare its size with a “classic” ESP-WROOM-32 module and with a 1 euro coin:

esp-pico-004

The module includes a chip antenna; it’s also possible to connect an external antenna thanks to the presence of an I-PEX connector.

In conclusion, the availability of the ESP32-PICO-D4 SiP makes it possible to use the esp32 chip in applications where the available space is very small…

ESP32, PlatformIO

PlatformIO is an opensource ecosystem (as it’s defined in the homepage of the project) to develop IoT projects.

The heart of the platform is a software component named PlatformIO Core. This component includes:

  • a cross-platform compiler
  • a libraries and dependences manager
  • a serial monitor

PlatformIO Core is developed in Python and therefore it can run on different operating systems (Windows, Linux, MacOS).

Although you can use the Core component directly, PlatformIO’s strength lies in its IDE, which allows the development of multi-platform projects and integrates with the Core itself.

In this article I’m going to show you how to use PlatformIO to develop projects running on the esp32 chip.

Installation

PlatformIO IDE is provided as a plugin for two different development tools: Atom and VisualStudio Code. I tried both solutions and I preferred VSCode: both the installation and the use are simpler and more immediate.

Install VSCode after having downloaded the package from Microsoft’s website (the installer is available for Windows, Linux and MacOS).

Open the Package Manager:

pio-001

search the PlatformIO IDE package, then click on Install:

pio-002

wait until the installation is complete:

pio-003

Hello world

Now it’s time to develop your first program, which traditionally will display the sentence Hello world! on the terminal.

If it doesn’t show up automatically, open the PlatformIO’s homepage:

pio-004

then click on New Project:

pio-005

give a name to the project and choose a devboard based on the esp32 chip (in this example I’ll use a Lolin32 board by Wemos). PlatformIO supports both the esp-idf framework and the arduino-esp32 one. All my tutorials are based on the first one:

pio-006

PlatformIO automatically creates some folders for your project. Choose the src folder (it stands for source, that is the folder which will contain the source code) and create a new file:

pio-007

name the file main.c and type the simple program as it follows:

pio-008

run the compiler by clicking on the corresponding button in the bottom bar:

pio-009

the editor displays an error… indeed your code is using the printf() function without having included the library:

pio-010

add the missing line, now you should be able to complile the code without errors:

pio-011

PlatformIO can also upload the compiled program to your board. Thanks to its auto-detect feature, you usually don’t need to specify the serial port the board is connected to:

pio-012

Serial monitor

PlatformIO also includes a serial monitor you can use to test your program. By default, this monitor connects to the serial port with a speed of 9600 baud. The esp32 chip instead has a default speed of 115200 baud; you have therefore to change the platformio.ini file included in your folder as it follows:

pio-013

Now open the serial monitor; you should see the correct output of your program:

pio-014

Conclusions

I found the use of PlatformIO really immediate: after a few minutes I was able to develop, compile, load and test a program. Try it and leave a comment with your impressions!

 

My new third hand

I recently brought from Banggood a support for having a third hand during soldering operations…

hand-001 hand-002

the support is made by an aluminum base and three arms which end with a crocodile clamp with insulating sheath:

hand-003 hand-004

The assembly is very simple, the arms must be screwed on the base:

hand-005

In general this support is very usefulthe arms are very well articulated and their length allows to “grasp” even medium-sized cards:

hand-007 hand-006

The only defect is the position of the feet, which makes the base not too stable:

hand-008

 

IIS, mutual authentication using SSL certificates

Not everyone knows that IIS (Internet Information Services), the webserver included in Windows Server, offers the possibility to perform mutual authentication using SSL certificates.

You probably saw that, within the site’s SSL configuration, you can require an SSL client certificate to the client that is connecting:

iis-ssl-001

in this case, IIS only verifies that the submitted certificate is valid, i.e. signed by a trusted CA.

The mutual authentication allows you to go one step further: based on the certificate that the client sends to IIS, the server maps it to a local or domain user. It is therefore possible to identify a user without requiring username and password, but based on the certificate it owns.

IIS allows two different mappings: one-to-one (each certificate corresponds to a specific user) or many-to-one (multiple certificates correspond to a single user). In the first case you have to load the whole certificate in IIS, while in the second one you indicate to IIS the characteristics that the different certificates must have (for example the value of CN):

iis-ssl-002

Configuration

You cannot configure the Certificate Mapping via graphical interface (IIS Manager); the configuration is possible only directly modifying the IIS configuration.

First verify that the following role services are installed on your server:

iis-ssl-005

the first one (Client Certificate Mapping) is used for mapping clients to domain accounts (Active Directory), while the second one (IIS Client Certificate Mapping) for local accounts.

Open the configuration editor of the website you want to modify:

iis-ssl-003

The configuration of both modules is under security – authentication:

iis-ssl-004

For example let’s configure the mapping on local accounts. First you have to enable the module and choose if you want one-to-one or many-to-one mapping (you can enable both):

iis-ssl-006

To add a one-to-one mapping, click on :

iis-ssl-007

then click on Add and type the requested information (certificate, username and password):

iis-ssl-008

you have to provide the certificate in base64, after having removed the lines —BEGIN CERTIFICATE— and —END CERTIFICATE— and the new line characters so that all the content is on a single line:

iis-ssl-010

The steps to add a many-to-one mapping are similar… instead of loading the certificate you have to define one or more rules. If the client certificate matches a rule, the mapping is performed:

iis-ssl-011

As last step, remember to disable the other authentication methods:

iis-ssl-012

JQ6500, “flashdata error”

After having published my previous posts about the JQ6500 module, some readers wrote me about an error the MusicDownload.exe application shows when they click on the upload button to store the mp3 files in the flash memory.

The error is

flashdata error, have not define program mode in flashdata.ini!!

jq6500-err-001

It seems that the application is not fully compatible with Windows 8 or 10.

To solve the problem, you only have to enable the Windows 7 compatibility mode:

jq6500-err-002

With this setting, the upload process runs fine:

jq6500-err-003

In the following video, you can see all the required steps:

JQ6500 module recovery

In a previous post I presented the JQ6500 audio module. When you connect it to your computer, it’s detected as a CDROM drive.

I brought some modules from different webstores. A couple of them gave me the following error when I tried to browse the content of the CD:

Unable to access the disk. The disk may be damaged.

jq6500-rec-000

It seems that some manufacturers do not program the flash memory of the chip and therefore it doesn’t contain the MusicDownload.exe application.

Luckily, thanks to the work by Reinhard Max and Nikolai Radke, it’s now possible to recover those modules. From Nikolai’s Github repository it’s possible to download an ISO image with a minimal Linux distribution and the recovery toolkit:

jq6500-rec-100

After having downloaded the image, you can burn it to a CD (or use software like UNetbootin to transfer it to a bootable USB drive) and restart your PC with that image.

However, there is an alternative way: the use of VMware Workstation Player, a free virtualization software (for non-commercial use) by VMware. Using this application it’s possible to start a virtual machine on your PC that can run the recovery ISO image.

At the end of this post you can find a video that shows all the following steps…

Let’s see how to do it. After having installed the application, create a new virtual machine:

jq6500-rec-008

Choose I will install the OS later:

jq6500-rec-009

Select Linux as operating system:

jq6500-rec-010

Give the VM a name and select the folder where it will be saved:

jq6500-rec-011

You don’t really need an hard disk, so you can specify a small size for it (for example 1Gb):

jq6500-rec-012

Now you have to configure VMware Player to start the VM using the ISO image. Click on Customize Hardware:

jq6500-rec-013

Select the CD/DVD drive and connect it to the ISO image downloaded from Github:

jq6500-rec-014

Before starting the VM, verify that your module is not connected to the computer:

jq6500-rec-015

Wait for the boot process to complete. The toolkit (started automatically) will warn you that it wasn’t able to identify any modules:

jq6500-rec-001

Open the Removable Devices menu and note down all the listed devices:

jq6500-rec-002

Now connect the module to the USB port. If you open the menu again, you should be able to identify which device corresponds to your module:

jq6500-rec-003

Connect it to the virtual machine:

jq6500-rec-004

Perform a new scan typing the command. If everything is ok, now the toolkit should be able to identify the module:

jq6500-rec-005

Depending on the size of the flash memory of your module, choose the correct menu item. Most modules are sold with a 16Mbit or 2Mbyte memory chip:

jq6500-rec-006

At the end of the recovery process, you can shut down the VM and verify that your computer is now able to display the disk content:

jq6500-rec-007

Video

Here’s a video that shows the recovery process: