@madpilot rants

Garage Door Opener – Modifying the ESP8266 Over-the-Air update code

Now that I have the proof of concept code running, it’s time to modify the built in Arduino core library that handles OTA updates.

The existing OTA library takes a binary object and an optional MD5 hash (to verify the upload), stores it in flash memory, then swaps the old binary out of the new binary and reboots the device.

To do verification via digital signatures, we need three additional pieces of information: the developers certificate (used to decrypt the hash), the encrypted hash, and the Certificate Authority certificate used to verify the developers signature.

The CA needs to be compiled in to the source code – there is little point sending that along with our payload.

The developer certificate and encrypted hash on the other hand, need to be supplied with the binary . One option is to upload the three files separately, but this would require extensive reworking of the updater API, and of the OTA libraries.

A better option would be to somehow bundle all three files in one package, which is the path I am looking to go down.

So, the first thing to do is work out what the file format looks like.

The binary blob is of an arbitrary size, and starts with the magic byte: 0xE9, which I assume is an instruction that is required to start the boot process.

Our certificate is also an arbitrary size. The signature will be fixed size, but dependent on the algorithm we use. Clearly we need some way of instructing the updater code where the boundaries for each file are.

We could pack them at the beginning, and set the byte before the file with the expected length – ie if our signature was four bytes, and certificate was 6 bytes it might look like this:

[ 4 | s | s | s | s | 6 | c | c | c | c | c | c | … ]

but that would mean we’d have to move the data around, as the bootloader would be looking for the magic number in the position 0. I’ve decided to do it the other way around – I’m going to use the last two bits to signify the lengths, and then count backwards. ie:

[ … | c | c | c | c | c | c | s | s | s | s | 6 | 4 ]

I wrote a quick little c program that packages everything up.

#include <stdio.h>
#include <stdlib.h>

unsigned char *bundle;

uint32_t size;
uint32_t certificate_size;
uint32_t signature_size;

int main() {
  FILE *f1 = fopen("WebUpdater.ino.bin", "rb");
  if(f1) {
    fseek(f1, 0, SEEK_END);
    size = ftell(f1);
    rewind(f1);
    printf("Binary file size: %i\n", size);
  } else {
    printf("Unable to open WebUpdater.ino.bin\n");
    return -1;
  }

  FILE *f2 = fopen("developer.crt.der", "rb");
  if(f2) {
    fseek(f2, 0, SEEK_END);
    certificate_size = ftell(f2);
    rewind(f2);
    printf("Certificate size: %i\n", certificate_size);
  } else {
    printf("Unable to open developer.crt.der\n");
    return -1;
  }

  FILE *f3 = fopen("WebUpdater.ino.sig", "rb");
  if(f3) {
    fseek(f3, 0, SEEK_END);
    signature_size = ftell(f3);
    rewind(f3);
    printf("Signature size: %i\n", signature_size);
  } else {
    printf("Unable to open WebUpdater.ino.sig\n");
    return -1;
  }

  printf("Signature size: 0x%x\n", signature_size);
  uint32_t bundle_size = size + certificate_size + signature_size + (2 * sizeof(uint32_t));
  bundle = (unsigned char *)malloc(bundle_size);

  for(int i = 0; i < bundle_size; i++) {
    bundle[i] = 0;
  }

  fread(bundle, size, 1, f1);
  fread(bundle + size, certificate_size, 1, f2);
  fread(bundle + size + certificate_size, signature_size, 1, f3);

  bundle[bundle_size - 4] = signature_size & 0xFF;
  bundle[bundle_size - 3] = (signature_size >> 8) & 0xFF;
  bundle[bundle_size - 2] = (signature_size >> 16) & 0xFF;
  bundle[bundle_size - 1] = (signature_size >> 24) & 0xFF;

  bundle[bundle_size - 8] = certificate_size & 0xFF;
  bundle[bundle_size - 7] = (certificate_size >> 8) & 0xFF;
  bundle[bundle_size - 6] = (certificate_size >> 16) & 0xFF;
  bundle[bundle_size - 5] = (certificate_size >> 24) & 0xFF;

  FILE *f4 = fopen("Bundle.bin", "wb");
  if(f4) {
    fwrite(bundle, bundle_size, 1, f4);
    printf("Bundle size: %i\n", bundle_size);
  } else {
    printf("Unable to save Bundle.bin");
  }

  return 0;
}

This produces a Bundle.bin file that can be uploaded.So far, I’ve managed to decode the lengths, and find where the two files I’m interested are live. Next I need to pull the files out, and do the verification. I think I’ll sign the binary using MD5 for the moment, as the updater class already has that function built in, so I effectively get it for free.

Garage Door Opener – Signing a binary using axTLS

Comparing the SHA256 of a file after it has been uploaded allows us to check that it hasn’t changed. This doesn’t tell us if the file has been tampered with though – it would be easy enough for a someone to change the binary, and then change the hash so it matches.

To check the file was created by the person who said it was created by, we need to verify a cryptographic signature. The steps are fairly simple:

  1. We upload the new binary, our public key and the signature file.
  2. We check that the public key has been signed by a trusted certificate authority – if this fails, the CA can’t vouch for the person signing it, so we shouldn’t trust it.
  3. We decrypt the signature file using the public key. This is the original SHA256 hash of the binary. If we can’t decrypt it, we can’t compare the hashes
  4. We SHA256 the binary ourselves
  5. We compare the hash we computed with the file that was uploaded. If the two hashes match, then the binary hasn’t been tampered with, and we can trust it.

I took the previous POC code, and extended it to do just that.

I’ve covered generating a certificate authority before, as well as generating a certificate. The last bit to do is to sign out binary. Again, using OpenSSL:

openssl dgst -sha256 -sign cert/developer.key.pem -out data/sig256 data/data.txt

Garage Door Opener – Signing Over-the-Air updates

The garage door opener has been running pretty well for the past couple of months, but I still have some work to do. I haven’t built out the configuration interface yet, and it turns out that if Home Assistant restarts, it forgets the last open state, so with out opening and closing the door again, I don’t know the state of the door.

This means I need to update the firmware.

The ESP8266 has facilities to do Over-the-Air (OTA) updates, however it doesn’t verify that the uploaded binary has been compiled by the person the device thinks it has. The easiest way to do this is to create a digest hash of the file and sign it. Then the device can verify the hash and check the signature matches.

There is an issue to implement this on the ESP8266 Github page, so I thought I would have a look at implementing something.

The first step is to be able to compare a hash. I decided to use the AxTLS library, as it has already been used for the SSL encryption on the device. After a google search, I found this page that outlines has to verify a SHA1 + RSA signature.

I simply pulled the sha1.c file (renamed it sha1.cpp), and created a sha1.h file that defines the functions in the cpp file. Next I created a test file, and hashed it using openssl:

openssl dgst -sha1 -binary -out hash data.txt

I then uploaded the files to the ESP8266 SPIFFS filesystem, and wrote some quick POC code.

The computed hash matches the supplied hash. Step 1 complete!

The next step will be to generate a signed digest, and decrypt that.

Garage Door Opener – Connecting the ESP8266 with TLS

While I want to do full CA verification, I’m waiting on some of the bugs to get ironed out of the ESP8266 Arduino library, so I’ll take a shortcut for the moment, and use fingerprinting to verify the server certificate (It should be pretty easy to move to CA verification down the track).

To do this, we need three pieces of information:

  1. Client certificate
  2. Client secret key
  3. Server fingerprint

We have already generated the certificate and the secret key, so let”s generate a fingerprint for the server certificate. We can use OpenSSL to do this:

openssl x509 -in mosquitto/etc/certs/mqtt.local.crt.pem -sha1 -noout -fingerprint

This will generate a 160bit sha1 string, that may look a little like this:

SHA1 Fingerprint=DA:39:A3:EE:5E:6B:4B:0D:32:55:BF:EF:95:60:18:90:AF:D8:07:09

Convert PEM to DER

I was reading through the docs, and while the axTLS says it supports PEM, I found that I needed to convert the PEM format certificate (which is ASCII based) to DER which is binary.

Again, OpenSSL to the rescue:

openssl x509 -outform der -in garage.local.csr.pem -out garage.local.csr.der
openssl x509 -outform der -in garage.local.key.pem -out garage.local.key.der

Loading the certificates

Down the track, I’ll be able to upload the certificate and key via the administration console (certificates expire, and need to be updated), which means storing them in the code makes little sense. Instead I’m going to store them on the flash as a file.

Thankfully there is a tool that makes uploading files to the SPIFFS filesystem super simple. Gratuitously stolen from the Filesystem Read Me:

ESP8266FS is a tool which integrates into the Arduino IDE. It adds a menu item to Tools menu for uploading the contents of sketch data directory into ESP8266 flash file system.

  • Download the tool: https://github.com/esp8266/arduino-esp8266fs-plugin/releases/download/0.2.0/ESP8266FS-0.2.0.zip.
  • In your Arduino sketchbook directory, create tools directory if it doesn’t exist yet
  • Unpack the tool into tools directory (the path will look like<home_dir>/Arduino/tools/ESP8266FS/tool/esp8266fs.jar)
  • Restart Arduino IDE
  • Open a sketch (or create a new one and save it)
  • Go to sketch directory (choose Sketch > Show Sketch Folder)
  • Create a directory named data and any files you want in the file system there
  • Make sure you have selected a board, port, and closed Serial Monitor
  • Select Tools > ESP8266 Sketch Data Upload. This should start uploading the files into ESP8266 flash file system. When done, IDE status bar will display SPIFFS Image Uploaded message.

Copy the two der files in to the data directory, and run the data upload function (I renamed them client.key.der and client.crt.der because that kind of makes more sense in this context). To load the files, the code looks a little like this:

// We need to use WiFiCLientSecure to the encryption works
WiFiClientSecure wifiClient

SPIFFS.begin();
File ca = SPIFFS.open("/client.crt.der", "r");
if(!ca) {
  Serial.println("Couldn't load cert");
  return;  
}

if(wifiClient.loadCertificate(ca)) {
  Serial.println("Loaded Cert");
} else {
  Serial.println("Didn't load cert");
  return;
}
  
File key = SPIFFS.open("/client.key.der", "r");
if(!key) {
  Serial.println("Couldn't load key");
  return;  
}

if(wifiClient.loadPrivateKey(key)) {
  Serial.println("Loaded Key");
} else {
  Serial.println("Didn't load Key");
}

The device will connect at this point, and the mosquitto server will be happy. It will use the name on the certificate as the client name, and the connection will be encrypted.

However, I not verifing the server yet. I will do that using the verify function

PubSubClient client("mqtt.local", 8883, callback, wifiClient); //set  MQTT port number to 8883 as per //standard

wifiClient.connect("mqtt.local", 8883);
  
if(wifiClient.verify("DA 39 A3 EE 5E 6B 4B 0D 32 55 BF EF 95 60 18 90 AF D8 07 09", "mqtt.local")) {
  Serial.println("connection checks out");
  wifiClient.stop();

  if(client.connect("garage")) {
    Serial.println("Connected");
    client.subscribe("test");
    client.publish("test", "hello world");    
  } else {
    Serial.println("Not connected");
  }
} else {
  wifiClient.stop();
  Serial.println("connection doesn't check out");
}

Notice that all of the colons have been replace with spaces.

Because of the way the PubSubClient works, I need to make a (pre?)connection to the server to do the verification. It’s pretty simple – connect, verify and disconnect, then continue with the PubSub connection.

There is a problem here – the mDNS name won’t actually resolve at this point. Because the DNS resolver doesn’t support it.

Ok, that isn’t strictly true – every DNS resolver CAN resolve mDNS, but you need to hit a special DNS server and port (224.0.0.251 on port 5353), which I haven’t worked out how to do with the Arduino library yet…

Garage Door Opener – So, the ESP8266 does support TLSv1.2

I was digging around the ESP8266 github page, as there was an announcement that it now supports CA verification (it ALMOST does – there was a regression bug that means it’s still not working), but I noticed that the last release (2.3.0 at time of writing) is actually quite old – it was released in June.

I was going through recent commit messages, and noticed there had been quite a bit of work around integrating axTLS, which is a tiny TLS library specifically designed for computers with small memory footprints (AKA microcontrollers).

This piqued my interest.

It has been said that the ESP8266 doesn’t support TLSv1.2, because of a buggy implementation in the firmware, which I had verified earlier in testing and hence configuring MQTT to force TLSv1.1.

By including axTLS as part of the Arduino library, and not relying on the ESP8266 API, I wondered if TLS1.2 was now supported.

Turns out it is!

It does mean we’ll have to use a “unstable” version of ESP8266 library. This means other stuff might not work, we need to install it using git, and it’s up to us to keep everything up to day.

Thankfully, it isn’t that difficult to install the library using git.

First, remove the existing ESP8266 library using the board manager – find it using the search function, then hit remove.

Now, you can manually install – Instructions are here. Don’t forget to re-select the right board and to set the CPU frequency to 160Mhz!

Now, that regression bug. It stops client certificates being sent for verification, which is bad for us, so we’ll need to roll back a couple of commits. To do that:

cd ~/Arduino/hardware/esp8266com/esp8266
git checkout -b pre-axtls-2 d6e38f0abd2e1bf796a32e8b1a24d37fdc7daaf8

This creates a new branch based on an older revision that seems to still work ok.

Modify mosquitto to use TLSv1.2

This one’s pretty easy: Remove the tls_version line

#tls_version tlsv1.1

Then remove

"--tls-version", "tlsv1.1", 
from both mosquitto-client/Dockerfile.pub and mosquitto-client/Dockerfile.sub.

Run

docker-compose build
and start the server with
docker-compose up

You should see “Hello world” in there somewhere. Next up, the Arduino side of things…

Garage Door Opener – How can I make this garage door opener secure?

One of the design goals for this project was a secure system – this device can open my garage, and I don’t really want just any person to be able to do that!

Disclaimer: I’m not a security expert! If you find holes in my logic, please let me know!

The most obvious way to interface with the controller was to setup some sort of server that a controller (eg an iPhone app) talks to. The problem with this is that an open network port is an attack vector – if a good guy can connect to a network port, so can a bad guy. While I could set up a username or password, having a port open potentially means buffer overflows, and since I’m not really a C/C++ guy this a big problem for me – I mean a lot of really smart people avoid writing servers in C…

After some research, I found this paper (PDF) talking about the use of the publish/subscriber pattern (AKA pub/sub), where a device connects to as server and subscribes to event notifications. This means there is no port open on the device, so even if an attacker found it’s IP address, there is no way for them to connect to it. It seems MQTT is the IOT pub/sub system of choice (it drives a lot of public IOT platforms).

This of course is not without other issues (not exhaustive – please suggest more if you can think of others):

  1. Problem: If an attacker got on the network, they could just connect to the MQTT server and send a “open” command.

    Solution: Access control lists (ACL): Only allow certain MQTT clients send certain commands. We would need some way for the server to verify who the client is…

  2. Problem: What if an attacker managed to spoof the IP or MAC address (perhaps through ARP poisoning) they could become a legitimate MQTT server, allowing them to send an open command.

    Solution: Verify the identity of the server. TLS certificates can help with that. Using a certificate signed by a custom Certificate Authority (CA) means we can guarantee the server is who it says it is. This also fixes the client verification problem from point 1.

After a quick search, I found the mosquitto project – an open source MQTT server which looked pretty easy to setup, supports ACLs and TLS.

Docker all the things

I decided to set it up a test environment using docker, as it makes dependency management much easier, and also makes the setup scriptable, so when it comes time to deploy to “production” I know exactly what I need to do.

You can clone my git repository to get started (I’ve tagged each of the steps so you can play at home – we’ll start with mqtt-1):

git clone https://github.com/madpilot/home-automation
cd home-automation
git checkout -b mqtt-1 mqtt-1
docker-compose build
docker-compose up

Congratulations! You now have a working MQTT server running on port 1883. Because it hasn’t configured it yet, the server has no authentication or authorisation – that’s fine for the moment – let’s get the baseline working, then we can add the security layers later.The easiest way to test the server is to use the mosquitto command line client. I’ve included these as part of the docker-compose cluster – in fact, when you ran docker-compose, a subscriber was automatically created (subscribed to the “test” channel) and publisher publishes the first message.

Look through the output on the screen. If you see:

mosquitto-subscriber_1  | Hello World

Everything has worked!You can try it out yourself – run the following commandm*:

docker-compose run mosquitto-publisher -t "test" -m "Hello again"

And you should see

mosquitto-subscriber_1  | Hello again

* I’ve been a bit tricky by using a docker entry point to hide away the full command to make sending messages much easier. The full command that gets called is

mosquitto_pub -h mosquitto -t test -m "Hello again"

Which tells mosquitto_pub to connect to the server with a host name of mosquitto (Which is set by docker), publish to the “test” channel, and send the message “Hello again”.Now that we have a working MQTT server and a way of testing things, we can start looking at the Arduino side of things.