BusPirate.com: Compile PIC and ARM firmware on a cheap server

Buspiratecom-head

The Bus Pirate project currently has four firmware builds (v3/v4/v5/vNG1) under two toolchains (PIC C/ARM C). To make this more manageable, we use a cheap VPS to check for new code in our git repo and compile the firmware automatically. Fresh compiles are available for everyone immediately, without any intervention from developers or friendly forum members.

We wanted extremely tight integration with BusPirate.com so we rolled our own script instead of using a tool like Jenkins. This post covers:

  • Installing MPLABX and PIC XC16 compiler for automated builds
  • Installing ARM GCC and libopencm3 for automated builds
  • A simple build service to periodically update repos and compile firmware

Is there new code to compile?

Creating builds for a bunch of platforms is boring, let’s automate it! First we find out if there’s new code to compile. With git we can check the log before and after a ‘pull’ command.

#git log --pretty=format:'%H' -n 1
cba17f0d1cf0f7a2b5e8ad5954390feae08b6ba3

Check the git log for the long hash of the current local repository.

git pull

Pull the latest commits and update the local repository.

git log --pretty=format:'%H' -n 1
42f05e5fe033f37d029a0483c967a01a81222c7c

Check the long hash again. The hash changed! Let’s compile some code!

make clean && make

Compile the latest code and then do something useful with the output.

Server setup

A build server can definitely run in your basement or at the hackerspace, but a $5/month VPS from Vultr or DigitalOcean is a super slick option. We run Ubuntu 14.04 LTS 64bit because some of libraries we need aren’t available for newest versions of Ubuntu.

Install instructions for both toolchains are in shell script .sh format. You shouldn’t run these! Defaults and version numbers change. Some user interaction is needed. Open a shell terminal and paste the lines one at a time. Notes in the .sh file explain each step.

Setup PIC MPLABX and XC16 compiler for automated builds

PIC firmware can be compiled on a headless Linux server since Microchip released MPLABX and the XC compilers. Installation is a bit tedious, but the toolchain works great. bp-install-mplabx.sh documents our setup. Credit to this solution.

The Bus Pirate MPLABX project needs a little preparation before compiling on the server. There’s three active hardware versions to compile for the PIC-based Bus Pirate. Each hardware version needs a separate MPLABX project, the project configuration passes the hardware version #define via a compiler flag.

Setup ARM GCC compiler and libopencm3 for automated builds

ARM GCC is easier to setup than the PIC toolchain. Install the compiler, pull the source, then compile libopencm3 and the Bus Pirate NG firmware. bp-install-armgcc.sh documents our install.

Install and configure automated build service

buildv2.py runs ‘git pull’ every 10 minutes. If there’s new commits, the source is compiled and the firmware is uploaded to a server for further processing. Follow the steps in bp-install-autobuild.sh to install and configure the build service.

buildv2_tasks.py has settings for the repositories and builds. Each repository can have multiple build tasks. The PIC-based Bus Pirate repo is pulled once, then compiled for three hardware configurations (v3/4/5).

The service can be started from the command line with:

/etc/init.d/bp-build start

And stopped with:

/etc/init.d/bp-build stop

If the service is installed in /etc/init.d it will start with the operating system. The script has been stable for months at a time, but we play it safe and use monit to restart the service if it crashes.

Backend processing

buspiratecom-CDreleases

The firmware and build logs are packaged into a JSON file and uploaded to an API at BusPirate.com. Here’s an example of the JSON output, the important variables are described in the table below.

‘error’ 0 if make executed, 1 if make returned system error (build failed with errors).
‘timestamp’ start/stop build timestamp
‘firmware’/’hardware’ Identifying info
‘starthashlong’/’endhashlong’ Commit hash before/after git pull command
‘gitoutput’ Output from the git pull’ command
‘makeoutput’ Output from the make command
‘apikey’ Identifying info
‘firmware_type’ The file extension of the firmware. Added to automate naming in the backend. PIC uses .HEX, ARM uses .BIN
‘base64encbin’ Base 64 encoded firmware file

BusPirate.com serves up the fresh build and notifies anyone subscribed to the mailing list. We also grab all the commit notes and change history from GitHub so it’s easy to see what’s in the build. You can check it out here.

Up next

For the rest of this week we’ll be testing a new version of Dirty Cables at DirtyCables.com.

Later next week we’ll receive a few hand-assembled Bus Pirate v5 prototypes. We ruined another Bus Pirate v5 prototype by tearing the USB Micro B connector and traces off the PCB. Instead of building another board in-house, we sent the parts to a Chinese PCBA that does two-off hand-assembled prototypes. Let’s see how they turn out!

BusPirate.com: Automated documentation updates

wx_camera_1523965080542

Up-to-date documentation makes a project easy to learn about, but its a really boring job that takes a lot of time. Even great documentation eventually has outdated examples and screenshots that don’t quite match the latest version.

BusPirate.com has a hacked together toolchain to keep the documentation fresh.

buspiratecom-refmanual-600

It’s a three part process:

  • Test scripts run demos on actual Bus Pirate hardware
  • Results files are uploaded to BusPirate.com
  • Tutorial templates and the results files are merged to make updated docs

Hardware and firmware specific docs

buspiratecom-refmanualpdficon

Bus Pirate documentation is full of version caveats like “Hardware v4+ only” and “Firmware v5.1+”. BusPirate.com docs are versioned, and just shows the stuff that matters for your version. Choose a hardware and firmware from the menu to see the docs for that specific combination.

Automated updates

Developers still need to add new feature information manually, but the huge task of “refreshing” everything after a firmware release can be automated. We took a three step approach: test scripts that run commands on actual Bus Pirate hardware, results files that capture the test output, and templates that merge with result files to create version-specific documentation.

Test scripts

buspiratecom-testeditor

Test scripts are just a list of commands to run on the Bus Pirate, for example running a self-test.  A test has multiple steps, and each step has one or more Bus Pirate commands. We build the demos with a simple editor, then dump them to JSON in a file. Here’s the test script that runs all the commands shown in the reference manual.

Results files

buspiratecom-pipepy

pipe.py sends commands in the test script to a Bus Pirate. It also records the terminal output to a results file. Result files are uploaded to BusPirate.com. Here’s the results file from the reference manual test script.

Templates

buspiratecom-template-entry

Demos and docs are blade templates. A template merges with a results file to show the tutorial exactly as it appears on a hardware and firmware combination. It’s not super elegant, the version specific stuff is done with a bunch of PHP if statements.

Taking it further

Command Reference, Self-test guide, Pull-up resistors guide, and Number entry guide are up now on the new site. More will come soon.

Ideally the update process will be triggered by an automated firmware release. An RPi in the workshop will bootload the new firmware into the Bus Pirate, run the test scripts, and upload the results without any intervention.

test-rig

A test rig with a bunch of devices might be a cool way to do release testing.  Scripts could test various firmware features on real devices.  Comparing the scripted test results with a previous release would highlight things that may be broken. Our goal is to hack together something that does comprehensive release testing with no manual effort.

Thursday we’ll document the automated build server that’s kicking out freshies every time there’s new code in the git repo.

 

 

BusPirate.com: A dedicated website for docs, firmware, tutorials

buspiratecom-refmanual-close

Bus Pirate documentation and demos are all buried in a huge wiki and around the blog, this is less than ideal for such an expansive project. We’ve been working on a new site just for Bus Pirate stuff.

This site is full of hacks that automate boring development stuff.

  • Documentation and tutorials are updated through test scripts and a templating system
  • The latest code in the git repo is compiled continuously and always ready to download
  • Firmware releases are packaged and announced automatically

There’s a broad overview of the site features below, we’ll follow up with detailed posts later this week.

Big obvious menu options

buspiratecom-menu

It’s a simple thing, but it makes a big difference. Obvious menu options dedicated to the Bus Pirate are a nicer experience than digging through a wiki with stuff about dozens of (partially) completed prototypes.

Versioned tutorials you can download in PDF

buspiratecom-refmanualpdficon

Bus Pirate documentation is full of version caveats like “Hardware v4+ only” and “Firmware v5.1+”. BusPirate.com docs are versioned, they only show info that applies to a specific hardware and firmware combination.

buspiratecom-pdf

Docs can be downloaded as a PDF. We tried to use PDF specific stylesheets so the docs look as good printed as they do in a browser. wkhtmltopdf handles the conversion to PDF, it’s by far the best conversion tool we’ve worked with.

Automated firmware builds and release packages

buspiratecom-CDreleases

An automated build server compiles new firmware every time there’s a commit to the git repo. This is a huge improvement for everyone. Users get the latest bug fixes without compiling from source, and developers don’t have to make interim releases for a bunch of platforms.

Currently the build server compiles firmware for four hardware versions (v3/4/5/NG1) under two different compilers (MPLABX/XC16, ARM GCC).  We’ll document the build server on Thursday.

buspiratecom-firmware

Firmware releases are packaged and announced automatically. The release is a tidy archive with the latest firmware, update tools, readme.txt and updated history.txt.

Notification of new tutorials, releases, and firmware builds

buspiratecom-releaseNotification

Subscribe to be notified of new developments. Firmware builds and releases mailing lists can be limited to a specific hardware version to cut down on noise.

Taking it further

Later today we’ll post about the system to automate updates of the Bus Pirate tutorials and documentation. Thursday we’ll show how we setup an update automated build server to compile PIC and ARM firmware on a $5/month virtual private server.

Pocket Pi

img_20180912_190447

A Raspberry pi zero w pocket terminal project from Facelesstech, that is available on Github:

So if you have been following my blog lately you may have noticed me rambling on about trying to get a Xbox 360 chat pad and an ps3 keypad working with a raspberry pi to make a portable terminal. I have finally finished my quest so join me below to see how I did it

More details  on Facelesstech blog.

Check out the video after the break.

App note: Recommendations to avoid short pulse width issues in HVIC gate driver applications

an_on_AN8102

Application note from ON Semiconductors discussing possible abnormalities on High voltage gate driver when operating on short pulses. Link here (PDF)

The High−Voltage Integrated Circuit (HVIC) gate driver family is designed to drive an N−channel MOSFET or IGBT up to 600 V. One of the most common methods to supply power to the high−side gate drive circuitry of the high−voltage gate drive IC is the bootstrap power supply. This bootstrap power supply technique has the advantage of being simple and low cost. However, duty cycle is limited by the requirement to charge the bootstrap capacitor and serious problems occur when extremely short pulse width is used in the application system. This application note explains the features of HVIC gate drivers and provides recommendations to avoid short pulse−width issues in the application.

App note: High voltage inverting buck reduces complexity and board space

an_on_AND9785

App note from ON semiconductors about repurposing a buck converter to produce negative voltages. Link here (PDF)

Applications in the electronics industry ranging from sensor−based designs to power amplifiers are periodically faced with the requirement to generate a negative voltage rail. Although many transformer−based designs, charge pumps and other methods have been used to meet such a requirement, the inverting buck−boost topology stands out as simple to design and can save on power and board space too.

With power budgets in many applications already stretched, and PCB real estate limited due to the high levels of functionality incorporated in many new products, power devices that use an inverting buck−boost topology can prove extremely valuable to systems designers.

A Buck regulator can be reconfigured to generate a negative output voltage from a positive input voltage using the inverting buck−boost topology. Unlike a buck regulator, the Inverting buck−boost transfers energy to the output through the output diode during the ’Off’ time. For this reason, users must keep in mind that the average output current is always less than the average inductor current.

Ultra MegaMax Dominator (UMMD) 3D printer

new extruder carriage

has a nice write-up about building his Ultra MegaMax Dominator (aka UMMD) 3D printer:

Ultra MegaMax Dominator (aka UMMD) is my third 3D printer design/build.  I used much of what I learned from its predecessors, and tried a few experiments, resulting in a very high quality machine that produces very high quality prints.  Time will tell if it meets my reliability goals.

See the full post on Mark Rehorst’s blog.

PROTOTYPE: Bus Pirate Next Gen v1

NG1-cover-shot-big

Sjaak, fresh off his ARM adventures at SMDprutser, hacked together an STM32-based Bus Pirate prototype we’ve been calling NG1 (Next Gen version 1). NG1 uses a modern ARM chip with completely open tools and libraries, and sports an integrated logic analyzer . The firmware is very usable, but largely untested. Let’s have a look at the biggest changes.

STM32 ARM-based design

bpng1-board_001

NG1 is built around an STM32F103CBT6, a modern ARM chip with 128K flash and 20K RAM running at 72MIPS.

All the extra resources let us use a proper printf function for terminal display, instead of the crappy obscure way of storing strings we invented for the v3.x hardware. The current firmware uses about half of the flash (78 of 128K). If we fill the chip, like we did with the Bus Pirate v3.x, STM32 has a full line of compatible chips with up to 512K of flash.

ARM has completely open toolchains, while the Microchip PIC used in previous Bus Pirates is still straggling with free-but-not-open-not-distributableand-crippled compilers and libraries. The ARM GCC compiler is open source and freely available for multiple operating systems. Several active open source libraries support the chip’s features, such as multiple USB endpoints. NG1 uses an open source peripheral library, libopencm3, so the complete source and libraries can be redistributed under the GPL.

v5-vng1-compare

Like most chips, STM32 doesn’t have a Peripheral Pin Select feature. PPS on the PIC24FJ64GA002 is what makes the original Bus Pirate v3.x design so elegant (left layout). Most major hardware modules like UART, SPI, timers, PWM, etc can be assigned to almost any pin using PPS – just one PIC pin is needed for each of the 5 Bus Pirate IO connections.

STM32 hardware modules are fixed to specific pins. Each Bus Pirate IO pin is connected to up to five STM32 pins to provide the necessary features. This kind of design uses a lot more pins (28 vs 64), and the PCB is a lot more complicated (right layout).

On-board Logic Analyzer

bpng1-logicanalysis

Frequently we use the Bus Pirate along side a logic analyzer like the Logic Pirate. NG1 has an SRAM-based logic analyzer directly on-board. The logic analyzer records bus activity every time the Bus Pirate writes or reads commands.

This adds an additional layer of debugging when things go wrong, without any extra cables or connections. Talk to a chip through the terminal, then check the logic analyzer to verify what actually happened on the bus. Terminal mode and the logic analyzer work at the same time because they each have a dedicated USB endpoint in the STM32. Microchip 23LC1024 SRAMs provide 256K samples per channel at 20MSPS on the current prototype, but we’ve found an alternate SRAM chip with over 10M samples per channel.

Currently NG1 uses a 74LVC573 single direction latch between the logic analyzer and bus. The next revision uses a 74LVC245 bidirectional latch[link] so the logic analyzer can also work as a signal generator without any additional hardware. Maybe we could toss on a resistor ladder to make an analog signal generator too.

So far we’ve focused on capturing bus activity while the Bus Pirate is writing and reading commands, but the hardware should work as a stand-alone logic analyzer as well. The high latency of STM32 pin interrupts could contribute to really sloppy capture triggers, especially at high capture speeds, but there’s still a lot to learn about this chip.

Selectable 3.3volt, 5volt, external pull-up voltage

pu-menu

The Bus Pirate has on-board pull-up resistors (10K) to use with 1Wire, I2C, and any other situation where an open drain bus is used. The on-board pull-up resistors can be connected to one of three voltage sources straight from the Bus Pirate terminal. Select from the on-board 3.3volt or 5volt power supplies, or use an external supply through the Vpull-up pin. Selecting the pull-up voltage from the terminal is really convenient new feature.

ng1-la-cct

Two BL1551 analog switches are chained together so that three input sources (3.3volt, 5volt, Vpu pin) are controlled by two pins. If this seems familiar, it’s the same switch hardware we tested on the v3.x update.

Integrated USB and USB Micro B connector

STM32F103CBT6 has integrated USB hardware with actively developed open source driver libraries. Not only is the USB interface faster than a USB-to-serial converter chip, it supports multiple USB endpoints at the same time. Currently we’ve implemented one “low speed” USB endpoint for the terminal and DFU bootloader updates, and a second “high speed” USB endpoint for the on-board logic analyzer.

USB Micro B is by far the most common USB connector at the moment. NG1 uses a Micro B connector so it works with the phone cable that’s probably already on your desk.

However, we don’t like the Micro B connector as much as the Mini B. Every hand-soldered USB Micro B connector eventually broke off of the PCB. After re-soldering it numerous times the pads and tracks get to the point of no repair.

On the most recent hardware revision we removed the cheap “standard” Chinese Micro B connector and used a much more expensive Molex 47589-001 USB connector with through-hole tabs for reinforcement.

1x10pin keyed locking connector

NG1-IOheader-compare

Previous Bus Pirates use a 2x5pin IDC connector. These are easy to use with 0.1” jumper wires from your parts box, but there’s not many options for probe cables using an ICD connector.

NG1 uses a 1x10pin 2543/TJC8S-10AW (equivalent to Molex 70553-0044)connector, a common 0.1” header inside a keyed/locking shroud. The pins are 0.1” pitch, so jumper wires still work great, but now we can make high quality keyed/locking probe cables with tangle-free silicone wire.

Buttons

The NG1 has a hardware reset button and a user configurable button. The user configurable button is currently only used for activating the bootloader during power-up. We haven’t decided yet on a good secondary purpose for it, but we are open for suggestions.

Second UART

A hardware UART is exposed for debugging without using USB. The code supports debugging via the UART, though we started debugging through the second USB CDC port after USB was up and running.

Taking it further

A logic analyzer, selectable pull-up resistor voltage supply, more memory and a totally open toolchain are probably the four biggest improvements to Bus Pirate NG1. The next revision will also support a hardware signal generator.

Hardware files are available here. The hardware is not final, but the current version addresses all known bugs. Quick order the latest PCB version at DirtyPCBs, or wait a week and we’ll have some to give away for free.

Firmware repository is here, it builds with the ARM GCC compiler. Use the command ‘git clone –recursive https://github.com/DangerousPrototypes/bus_pirate_ng‘ to clone our git repository. That command should also take care of getting the right libopencm3 version. Be sure to run make in the ‘/libopencm3/‘ folder first, then switch to the ‘/source/‘ folder and run make again to built the firmware. We’ll walk through the toolchain setup next week.

Weekly Roundup #57: New Maker Products // News

It’s pretty quiet in this week’s Weekly Roundup, especially on the Crowd Funding side, but we certainly do have wiffeys. [mmYouTube VideoID=”WLL9t9TLIFc” Title=”Weekly Roundup #57: New Maker Products // News”] uStepper S line Tachyon Exen Proto CCPROG PAT Zymkey Neutis Continue reading Weekly Roundup #57: New Maker Products // News

The post Weekly Roundup #57: New Maker Products // News appeared first on MickMake.